Move 25.2.0 changes to v25_STABLE - #365
Closed
mmols wants to merge 49 commits into
Closed
Conversation
…ble platform builds - Introduce two build modes, stable and current - "stable" builds produce a platform build with all stable components with the latest CLI - "current" mode additionally produces spock daily builds (from in-development main or specified branch) - Publish each build to its appropriate S3 location within the pgedge-devel repo - Adjust S3 lifecycle policies dynamically based on the build mode Corresponding stable/current workflows in the CLI project now call this script with the appropriate mode and switches
…l.sh changes for stable/current modes handling - Update env.sh to define removeComponentFromOut (that is used in make_tgz.sh) for stripping the Spock (unstable) component in stable mode builds - Enhance make_tgz.sh to support stable/current modes - Align with build_to_devel.sh for a consistent mode based build process
- Naming them appropriately - Making necessary adjustments (switches) to account for the changes in build scripts.
builds along with platform packages - Introduces AMD8/ARM9 workflows for daily 'current' builds containing: * Stable platform components + latest Spock (e.g. spock50) * Default from main branch but configurable in manual mode - Supports both scheduled (defaults) and manual (custom) triggers - Publishes to pgedge-devel in REPO/current/ with 7-day retention - Captures build logs and Spock binaries/sources as artifacts
- Changing default cli branch to main for current workflow - Adjusting time slightly to not produce builds in the next day MMDD in s3
* CLI feature -datadir to pass absolute path for data directory to pgedge setup and cluster commands * adjustments based on review (PR#324) * more adjustments based on review (PR#324) * more adjustments based on review (PR#324) * done some cleanup (PR#324) * ensure fallback for pg_data * fix helptext in setup --------- Co-authored-by: Matthew Mols <matt@pgedge.com>
* Added build-mtree to build a merkle tree for a table * Fix node_list population * Also minimise critical section in compare_checksums * Use same block ranges on all nodes * Breakthrough: extending a merkle tree on new inserts works * Sweeping optimisations in table-diff * Use an optimised sql query instead of get_pkey_offsets * Fix pkey offsets query * Added feature to merge blocks on large deletes * Handle blocks splits and merges inside ACE * Fix some rebalancing bugs * Integrate merkle trees into ACE cli * Update mtree only splits blocks * Merges happen only when --rebalance=true is passed * Adding mtree diff * First version of mtree diff ready * Add progress bar for splits and merges * Use always trigger for tracking dirty blocks * Update cryptography to address CVE * No longer using OrderedSet for comparisons * Handle mismatching tree levels * Use generic trigger functions * Initialise mtree objects once per DB node * More cleanup * Tweaks to get all base tests to pass * Use SQL composables; fix rebalance issues * Unify pkey offset computations * Add support for specifying a ranges file * Remove node len check for mtree build * Update leaf hash during mtree update * Temp fix for range boundary issue * Address boundary issues using a lookup table * Use prepared statements * Sunset batches option and async rerun * Add block boundary and repset-diff tests * Remove explicit stmt.close() * Add support for composite keys; use stmt triggers * SQL cleanup * Add mtree init, teardown; fix block size usage * Added merkle tree tests * Use mogrify during repairs * Prior use of executemany would internally call execute multiple times, thereby making repairs slow because of statement triggers from mtrees * Fix write-ranges to use str by default * Add support for non-numeric datatypes in tracking triggers * Fix conn establishment in cleanup * Address codacy issues * Addressed more codacy issues * String literal fix * Codacy fix #4 * Codacy fix #5 * Use nosemgrep * More nosemgreps * Add mtree cli helptext * Fix metadata task type * Add metadata tracking for mtree modules * Move error codes out of config file * Use a separate consts file * Revamp ACE CLI invocation * Group mtree cmds into a sub-cmd * Treat each of table-diff, -repair, etc. as a top-level sub-cmd * Update tests * Fix table names in mtree test * Minor fixes * Help texts mostly fixed * fix fire.py helptext generation for mtree submodule * generate help for ace mtree submodule * Backward compatibility fixes * Rename 'override-block-size' to 'skip-block-size-check' to free up -o * Add back 'block_rows' as an alias of 'block_size' * Add back 'behavior' in rerun and mark as deprecated * Improve merkle tree help text * Fix tests * update generated helptext * Ensure pgcrypto is present * Fix spock diff --------- Co-authored-by: Matthew Mols <matt@pgedge.com>
Spock 5 integration
Pg17 is new default version
…ons to the guc_value portion of the ALTER SYSTEM SET SQL in order to support spaced, quoted guc values \n PLAT-44
spock5 issue with regression test
Adds an extra set of quotations to the guc_value and escapes / or quotations it may hold to allow for spaced and quoted parameters to be passed PLAT-44
json-create allows creation of a node or subnode with either ip address or hostname through the use of socket.gethostbyname PLAT-131
pg1-host issue resolved
In the case that a hostname or ip address cannot be resolved by gethostbyname, socket error will be caught, appended to validation_errors and reported with the specific node from which it arose from. PLAT-131
fix: db guc set supports quoted param
* [BR-86]: Updates copy-to-devel.sh script so that it takes two more inputs i.e. mode (stable/current) and the s3 subdirectory within stable/current
…ostname json-create supports hostnames node config
* resolve error with spock set-readonly * remove spock node-alter-location * add missing dependency
* major spock version upgrade command (draft) * successfully upgraded to spock5 using um upgrade-spock * spock 5 upgrade support * Update um.py Upgrade-spock function hide * code refactored * additional test case added for spock 5 upgrade comp check * additional test case added for spock 5 upgrade comp check * additional test case added for spock 5 upgrade comp check * warning banner removed from case 3 * final adjustment to logging output --------- Co-authored-by: Matthew Mols <matt@pgedge.com>
* Use conservative datatype handling * Add configurable option for using repeatable read while updating Merkle trees
Build script changes to produce spock60 builds from the main branch. Updated versioning (6.0.0-devel) and metadata. Updated 'current' builds workflow to now produce spock60 from main. Although, spock60 is hidden in the um list output, it can be installed in the current builds (only) by passing --spock_ver=6.0.0 in the pgedge setup command.
…-space The fix now passes the data-directory exactly once—wrapped in quotes but without backslashes—so paths like “/data/foo bar” work naturally. All shell interactions funnel paths through shlex.quote, removing our old hand-built escapes and closing the command-injection hole Codacy flagged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.