chore(setup): overhaul build system for true incremental builds#17135
chore(setup): overhaul build system for true incremental builds#17135
Conversation
P403n1x87
added
the
changelog/no-changelog
Codeowners resolved as |
Performance SLOsComparing candidate chore/setup-separate-download-build-location (8d2bb46) with baseline main (15b3c2f) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 104.416µs (SLO: <130.000µs 📉 -19.7%) vs baseline: +2.5% Memory: ✅ 44.004MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 102.332µs (SLO: <130.000µs 📉 -21.3%) vs baseline: +0.1% Memory: ✅ 44.087MB (SLO: <46.000MB -4.2%) vs baseline: +4.8% ✅ add_inplace_noaspectTime: ✅ 28.242µs (SLO: <40.000µs 📉 -29.4%) vs baseline: ~same Memory: ✅ 44.024MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ add_noaspectTime: ✅ 49.006µs (SLO: <70.000µs 📉 -30.0%) vs baseline: -0.5% Memory: ✅ 44.015MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ bytearray_aspectTime: ✅ 254.925µs (SLO: <400.000µs 📉 -36.3%) vs baseline: -0.3% Memory: ✅ 43.970MB (SLO: <46.000MB -4.4%) vs baseline: +4.6% ✅ bytearray_extend_aspectTime: ✅ 653.935µs (SLO: <800.000µs 📉 -18.3%) vs baseline: +1.9% Memory: ✅ 43.961MB (SLO: <46.000MB -4.4%) vs baseline: +4.7% ✅ bytearray_extend_noaspectTime: ✅ 262.381µs (SLO: <400.000µs 📉 -34.4%) vs baseline: ~same Memory: ✅ 44.052MB (SLO: <46.000MB -4.2%) vs baseline: +4.9% ✅ bytearray_noaspectTime: ✅ 141.002µs (SLO: <300.000µs 📉 -53.0%) vs baseline: -0.7% Memory: ✅ 44.080MB (SLO: <46.000MB -4.2%) vs baseline: +5.1% ✅ bytes_aspectTime: ✅ 215.527µs (SLO: <300.000µs 📉 -28.2%) vs baseline: +0.5% Memory: ✅ 44.024MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ bytes_noaspectTime: ✅ 134.037µs (SLO: <200.000µs 📉 -33.0%) vs baseline: +0.3% Memory: ✅ 44.098MB (SLO: <46.000MB -4.1%) vs baseline: +5.1% ✅ bytesio_aspectTime: ✅ 3.885ms (SLO: <5.000ms 📉 -22.3%) vs baseline: +0.4% Memory: ✅ 44.018MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ bytesio_noaspectTime: ✅ 323.225µs (SLO: <420.000µs 📉 -23.0%) vs baseline: -0.2% Memory: ✅ 44.017MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ capitalize_aspectTime: ✅ 88.776µs (SLO: <300.000µs 📉 -70.4%) vs baseline: -0.2% Memory: ✅ 44.039MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ capitalize_noaspectTime: ✅ 277.437µs (SLO: <300.000µs -7.5%) vs baseline: +6.0% Memory: ✅ 44.057MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ casefold_aspectTime: ✅ 87.999µs (SLO: <500.000µs 📉 -82.4%) vs baseline: -1.3% Memory: ✅ 43.930MB (SLO: <46.000MB -4.5%) vs baseline: +4.5% ✅ casefold_noaspectTime: ✅ 319.524µs (SLO: <500.000µs 📉 -36.1%) vs baseline: +1.3% Memory: ✅ 44.015MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ decode_aspectTime: ✅ 86.516µs (SLO: <100.000µs 📉 -13.5%) vs baseline: -0.3% Memory: ✅ 43.996MB (SLO: <46.000MB -4.4%) vs baseline: +4.6% ✅ decode_noaspectTime: ✅ 156.221µs (SLO: <210.000µs 📉 -25.6%) vs baseline: -1.1% Memory: ✅ 43.920MB (SLO: <46.000MB -4.5%) vs baseline: +4.5% ✅ encode_aspectTime: ✅ 84.377µs (SLO: <200.000µs 📉 -57.8%) vs baseline: -0.1% Memory: ✅ 44.070MB (SLO: <46.000MB -4.2%) vs baseline: +5.1% ✅ encode_noaspectTime: ✅ 144.025µs (SLO: <200.000µs 📉 -28.0%) vs baseline: -1.6% Memory: ✅ 44.016MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ format_aspectTime: ✅ 13.844ms (SLO: <19.200ms 📉 -27.9%) vs baseline: +1.1% Memory: ✅ 44.195MB (SLO: <46.000MB -3.9%) vs baseline: +4.6% ✅ format_map_aspectTime: ✅ 15.468ms (SLO: <21.500ms 📉 -28.1%) vs baseline: +0.2% Memory: ✅ 44.222MB (SLO: <46.000MB -3.9%) vs baseline: +4.8% ✅ format_map_noaspectTime: ✅ 380.304µs (SLO: <500.000µs 📉 -23.9%) vs baseline: +1.1% Memory: ✅ 44.074MB (SLO: <46.000MB -4.2%) vs baseline: +5.1% ✅ format_noaspectTime: ✅ 317.448µs (SLO: <500.000µs 📉 -36.5%) vs baseline: +0.4% Memory: ✅ 44.017MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ index_aspectTime: ✅ 132.004µs (SLO: <300.000µs 📉 -56.0%) vs baseline: +7.6% Memory: ✅ 44.019MB (SLO: <46.000MB -4.3%) vs baseline: +4.7% ✅ index_noaspectTime: ✅ 40.946µs (SLO: <300.000µs 📉 -86.4%) vs baseline: -0.4% Memory: ✅ 44.058MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ join_aspectTime: ✅ 219.133µs (SLO: <300.000µs 📉 -27.0%) vs baseline: -1.0% Memory: ✅ 43.999MB (SLO: <46.000MB -4.4%) vs baseline: +4.9% ✅ join_noaspectTime: ✅ 141.810µs (SLO: <300.000µs 📉 -52.7%) vs baseline: +0.4% Memory: ✅ 43.994MB (SLO: <46.000MB -4.4%) vs baseline: +4.7% ✅ ljust_aspectTime: ✅ 519.030µs (SLO: <700.000µs 📉 -25.9%) vs baseline: +0.1% Memory: ✅ 43.996MB (SLO: <46.000MB -4.4%) vs baseline: +4.5% ✅ ljust_noaspectTime: ✅ 293.597µs (SLO: <300.000µs -2.1%) vs baseline: 📈 +10.5% Memory: ✅ 44.073MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 303.712µs (SLO: <500.000µs 📉 -39.3%) vs baseline: -1.2% Memory: ✅ 44.083MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ lower_noaspectTime: ✅ 251.236µs (SLO: <300.000µs 📉 -16.3%) vs baseline: +2.1% Memory: ✅ 44.034MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ lstrip_aspectTime: ✅ 0.279ms (SLO: <3.000ms 📉 -90.7%) vs baseline: +0.3% Memory: ✅ 44.026MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 0.177ms (SLO: <3.000ms 📉 -94.1%) vs baseline: -1.3% Memory: ✅ 44.014MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ modulo_aspectTime: ✅ 13.380ms (SLO: <18.750ms 📉 -28.6%) vs baseline: +0.2% Memory: ✅ 44.197MB (SLO: <46.000MB -3.9%) vs baseline: +4.9% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 13.815ms (SLO: <19.350ms 📉 -28.6%) vs baseline: +0.3% Memory: ✅ 44.119MB (SLO: <46.000MB -4.1%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 13.440ms (SLO: <18.900ms 📉 -28.9%) vs baseline: +0.4% Memory: ✅ 44.144MB (SLO: <46.000MB -4.0%) vs baseline: +4.8% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 13.715ms (SLO: <19.150ms 📉 -28.4%) vs baseline: +0.5% Memory: ✅ 44.077MB (SLO: <46.000MB -4.2%) vs baseline: +4.7% ✅ modulo_noaspectTime: ✅ 0.372ms (SLO: <3.000ms 📉 -87.6%) vs baseline: +1.0% Memory: ✅ 44.054MB (SLO: <46.000MB -4.2%) vs baseline: +4.9% ✅ replace_aspectTime: ✅ 17.407ms (SLO: <24.000ms 📉 -27.5%) vs baseline: ~same Memory: ✅ 44.109MB (SLO: <46.000MB -4.1%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 284.109µs (SLO: <400.000µs 📉 -29.0%) vs baseline: +1.1% Memory: ✅ 43.958MB (SLO: <46.000MB -4.4%) vs baseline: +4.6% ✅ repr_aspectTime: ✅ 310.382µs (SLO: <420.000µs 📉 -26.1%) vs baseline: -1.3% Memory: ✅ 43.940MB (SLO: <46.000MB -4.5%) vs baseline: +4.6% ✅ repr_noaspectTime: ✅ 46.674µs (SLO: <90.000µs 📉 -48.1%) vs baseline: -0.6% Memory: ✅ 44.057MB (SLO: <46.000MB -4.2%) vs baseline: +5.1% ✅ rstrip_aspectTime: ✅ 389.448µs (SLO: <500.000µs 📉 -22.1%) vs baseline: -0.2% Memory: ✅ 44.030MB (SLO: <46.000MB -4.3%) vs baseline: +5.1% ✅ rstrip_noaspectTime: ✅ 185.359µs (SLO: <300.000µs 📉 -38.2%) vs baseline: -1.1% Memory: ✅ 43.998MB (SLO: <46.000MB -4.4%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 177.602µs (SLO: <300.000µs 📉 -40.8%) vs baseline: +0.6% Memory: ✅ 44.118MB (SLO: <46.000MB -4.1%) vs baseline: +5.2% ✅ slice_noaspectTime: ✅ 53.780µs (SLO: <90.000µs 📉 -40.2%) vs baseline: -0.3% Memory: ✅ 44.041MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 3.911ms (SLO: <5.000ms 📉 -21.8%) vs baseline: +0.4% Memory: ✅ 43.974MB (SLO: <46.000MB -4.4%) vs baseline: +4.7% ✅ stringio_noaspectTime: ✅ 352.671µs (SLO: <500.000µs 📉 -29.5%) vs baseline: +0.2% Memory: ✅ 44.053MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ strip_aspectTime: ✅ 276.679µs (SLO: <350.000µs 📉 -20.9%) vs baseline: -0.4% Memory: ✅ 44.025MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 178.724µs (SLO: <240.000µs 📉 -25.5%) vs baseline: -0.7% Memory: ✅ 44.035MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ swapcase_aspectTime: ✅ 342.741µs (SLO: <500.000µs 📉 -31.5%) vs baseline: +0.5% Memory: ✅ 44.047MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ swapcase_noaspectTime: ✅ 302.930µs (SLO: <400.000µs 📉 -24.3%) vs baseline: +7.1% Memory: ✅ 44.019MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 327.643µs (SLO: <500.000µs 📉 -34.5%) vs baseline: -0.6% Memory: ✅ 43.999MB (SLO: <46.000MB -4.3%) vs baseline: +4.9% ✅ title_noaspectTime: ✅ 269.964µs (SLO: <400.000µs 📉 -32.5%) vs baseline: -0.8% Memory: ✅ 43.981MB (SLO: <46.000MB -4.4%) vs baseline: +4.8% ✅ translate_aspectTime: ✅ 503.123µs (SLO: <700.000µs 📉 -28.1%) vs baseline: +1.2% Memory: ✅ 44.062MB (SLO: <46.000MB -4.2%) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 437.859µs (SLO: <500.000µs 📉 -12.4%) vs baseline: -0.6% Memory: ✅ 43.976MB (SLO: <46.000MB -4.4%) vs baseline: +4.8% ✅ upper_aspectTime: ✅ 309.413µs (SLO: <500.000µs 📉 -38.1%) vs baseline: ~same Memory: ✅ 44.040MB (SLO: <46.000MB -4.3%) vs baseline: +4.8% ✅ upper_noaspectTime: ✅ 252.579µs (SLO: <400.000µs 📉 -36.9%) vs baseline: +2.9% Memory: ✅ 44.054MB (SLO: <46.000MB -4.2%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 534.916µs (SLO: <700.000µs 📉 -23.6%) vs baseline: 📈 +18.8% Memory: ✅ 44.052MB (SLO: <46.000MB -4.2%) vs baseline: +5.4% ✅ ospathbasename_noaspectTime: ✅ 451.240µs (SLO: <700.000µs 📉 -35.5%) vs baseline: ~same Memory: ✅ 43.785MB (SLO: <46.000MB -4.8%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 643.076µs (SLO: <700.000µs -8.1%) vs baseline: +0.6% Memory: ✅ 44.102MB (SLO: <46.000MB -4.1%) vs baseline: +5.5% ✅ ospathjoin_noaspectTime: ✅ 652.781µs (SLO: <700.000µs -6.7%) vs baseline: +0.4% Memory: ✅ 43.834MB (SLO: <46.000MB -4.7%) vs baseline: +4.6% ✅ ospathnormcase_aspectTime: ✅ 368.487µs (SLO: <700.000µs 📉 -47.4%) vs baseline: -1.4% Memory: ✅ 44.106MB (SLO: <46.000MB -4.1%) vs baseline: +5.6% ✅ ospathnormcase_noaspectTime: ✅ 374.549µs (SLO: <700.000µs 📉 -46.5%) vs baseline: -0.7% Memory: ✅ 43.844MB (SLO: <46.000MB -4.7%) vs baseline: +4.8% ✅ ospathsplit_aspectTime: ✅ 506.349µs (SLO: <700.000µs 📉 -27.7%) vs baseline: -0.9% Memory: ✅ 44.085MB (SLO: <46.000MB -4.2%) vs baseline: +5.7% ✅ ospathsplit_noaspectTime: ✅ 512.742µs (SLO: <700.000µs 📉 -26.8%) vs baseline: -0.9% Memory: ✅ 43.806MB (SLO: <46.000MB -4.8%) vs baseline: +4.2% ✅ ospathsplitdrive_aspectTime: ✅ 386.655µs (SLO: <700.000µs 📉 -44.8%) vs baseline: -0.1% Memory: ✅ 44.124MB (SLO: <46.000MB -4.1%) vs baseline: +5.1% ✅ ospathsplitdrive_noaspectTime: ✅ 73.018µs (SLO: <700.000µs 📉 -89.6%) vs baseline: -1.9% Memory: ✅ 43.726MB (SLO: <46.000MB -4.9%) vs baseline: +3.9% ✅ ospathsplitext_aspectTime: ✅ 478.792µs (SLO: <700.000µs 📉 -31.6%) vs baseline: +1.1% Memory: ✅ 44.052MB (SLO: <46.000MB -4.2%) vs baseline: +5.4% ✅ ospathsplitext_noaspectTime: ✅ 484.046µs (SLO: <700.000µs 📉 -30.9%) vs baseline: +0.3% Memory: ✅ 43.785MB (SLO: <46.000MB -4.8%) vs baseline: +4.7% 📈 iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 176.733µs (SLO: <250.000µs 📉 -29.3%) vs baseline: 📈 +11.0% Memory: ✅ 44.065MB (SLO: <46.000MB -4.2%) vs baseline: +5.6% ✅ rsplit_noaspectTime: ✅ 168.873µs (SLO: <250.000µs 📉 -32.5%) vs baseline: -0.2% Memory: ✅ 43.778MB (SLO: <46.000MB -4.8%) vs baseline: +4.4% ✅ split_aspectTime: ✅ 162.604µs (SLO: <250.000µs 📉 -35.0%) vs baseline: ~same Memory: ✅ 44.018MB (SLO: <46.000MB -4.3%) vs baseline: +5.0% ✅ split_noaspectTime: ✅ 171.966µs (SLO: <250.000µs 📉 -31.2%) vs baseline: +1.7% Memory: ✅ 43.754MB (SLO: <46.000MB -4.9%) vs baseline: +4.9% ✅ splitlines_aspectTime: ✅ 153.193µs (SLO: <250.000µs 📉 -38.7%) vs baseline: -1.9% Memory: ✅ 43.804MB (SLO: <46.000MB -4.8%) vs baseline: +4.4% ✅ splitlines_noaspectTime: ✅ 162.205µs (SLO: <250.000µs 📉 -35.1%) vs baseline: ~same Memory: ✅ 43.984MB (SLO: <46.000MB -4.4%) vs baseline: +5.3% 🟡 Near SLO Breach (2 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 19.775ms (SLO: <22.300ms 📉 -11.3%) vs baseline: +0.7% Memory: ✅ 68.646MB (SLO: <73.500MB -6.6%) vs baseline: +4.9% ✅ exception-replay-enabledTime: ✅ 1.398ms (SLO: <1.450ms -3.6%) vs baseline: -0.5% Memory: ✅ 66.749MB (SLO: <71.500MB -6.6%) vs baseline: +4.8% ✅ iastTime: ✅ 19.681ms (SLO: <22.250ms 📉 -11.5%) vs baseline: +0.2% Memory: ✅ 68.657MB (SLO: <75.000MB -8.5%) vs baseline: +4.9% ✅ profilerTime: ✅ 15.184ms (SLO: <16.550ms -8.3%) vs baseline: ~same Memory: ✅ 60.410MB (SLO: <61.000MB 🟡 -1.0%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 19.693ms (SLO: <21.750ms -9.5%) vs baseline: ~same Memory: ✅ 68.677MB (SLO: <73.500MB -6.6%) vs baseline: +4.9% ✅ span-code-originTime: ✅ 20.238ms (SLO: <28.200ms 📉 -28.2%) vs baseline: +1.1% Memory: ✅ 68.663MB (SLO: <75.000MB -8.4%) vs baseline: +4.9% ✅ tracerTime: ✅ 19.750ms (SLO: <21.750ms -9.2%) vs baseline: +0.5% Memory: ✅ 68.597MB (SLO: <75.000MB -8.5%) vs baseline: +4.8% ✅ tracer-and-profilerTime: ✅ 21.046ms (SLO: <23.500ms 📉 -10.4%) vs baseline: ~same Memory: ✅ 70.612MB (SLO: <75.000MB -5.9%) vs baseline: +4.8% ✅ tracer-dont-create-db-spansTime: ✅ 19.823ms (SLO: <21.500ms -7.8%) vs baseline: +0.4% Memory: ✅ 68.737MB (SLO: <75.000MB -8.4%) vs baseline: +5.1% ✅ tracer-minimalTime: ✅ 16.860ms (SLO: <17.500ms -3.7%) vs baseline: +0.2% Memory: ✅ 68.653MB (SLO: <75.000MB -8.5%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 19.638ms (SLO: <21.750ms -9.7%) vs baseline: +0.4% Memory: ✅ 68.597MB (SLO: <72.500MB -5.4%) vs baseline: +4.8% ✅ tracer-no-cachesTime: ✅ 17.655ms (SLO: <19.650ms 📉 -10.2%) vs baseline: +0.2% Memory: ✅ 68.666MB (SLO: <75.000MB -8.4%) vs baseline: +4.9% ✅ tracer-no-databasesTime: ✅ 19.429ms (SLO: <20.100ms -3.3%) vs baseline: +0.7% Memory: ✅ 68.597MB (SLO: <75.000MB -8.5%) vs baseline: +4.8% ✅ tracer-no-middlewareTime: ✅ 19.364ms (SLO: <21.500ms -9.9%) vs baseline: -0.5% Memory: ✅ 68.654MB (SLO: <75.000MB -8.5%) vs baseline: +4.8% ✅ tracer-no-templatesTime: ✅ 19.573ms (SLO: <22.000ms 📉 -11.0%) vs baseline: +0.5% Memory: ✅ 68.724MB (SLO: <73.500MB -6.5%) vs baseline: +5.1% 🟡 recursivecomputation - 8/8✅ deepTime: ✅ 311.397ms (SLO: <320.950ms -3.0%) vs baseline: +0.2% Memory: ✅ 37.473MB (SLO: <38.750MB -3.3%) vs baseline: +4.9% ✅ deep-profiledTime: ✅ 332.367ms (SLO: <359.150ms -7.5%) vs baseline: ~same Memory: ✅ 43.922MB (SLO: <46.000MB -4.5%) vs baseline: +4.9% ✅ mediumTime: ✅ 7.299ms (SLO: <7.400ms 🟡 -1.4%) vs baseline: ~same Memory: ✅ 36.255MB (SLO: <38.000MB -4.6%) vs baseline: +4.5% ✅ shallowTime: ✅ 1.020ms (SLO: <1.050ms -2.9%) vs baseline: +1.6% Memory: ✅ 36.372MB (SLO: <38.000MB -4.3%) vs baseline: +5.3%
|
P403n1x87
force-pushed
the
chore/setup-separate-download-build-location
branch
from
fd8d7bf to
4c13c41
Compare
P403n1x87
changed the title
P403n1x87
force-pushed
the
chore/setup-separate-download-build-location
branch
2 times, most recently
from
9b5dd26 to
650a9c8
Compare
We separate the build site from the download cache to avoid separate builds to conflict with each other, causing incremental builds to fail to trigger. We also enhance the incremental build support and build the cmake shared dependencies once. We also include these in the caching for the test runner to avoid rebuilding when unnecessary.
P403n1x87
force-pushed
the
chore/setup-separate-download-build-location
branch
from
650a9c8 to
db42a71
Compare
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🔗 Commit SHA: 8d2bb46 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db42a71bcc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
taegyunkim
left a comment
taegyunkim
left a comment
There was a problem hiding this comment.
I like the idea to share abseil between appsec and profiling, but I'm just a tad bit hesitant to remove CMakeLists.txt files. As that would mean we no longer have a way to build only that modules if i understand it correctly.
Given that the build time is skyrocketing and tests are passing on this PR as is, I would not request changes, but just saying. 😅
Is this something useful to others? If so we could consider exposing a setuptools command that can be used to compile single extensions? I feel that after paying the price of a full compile, subsequent ones would be fast enough (20s) that one would not be concerned with recompiling a single extension. |
P403n1x87
force-pushed
the
chore/setup-separate-download-build-location
branch
from
bb2573f to
8d2bb46
Compare
|
Would you mind splitting this into a a few more focused PRs? I appreciate the care put into this PR but my concern is that this PR bundles several independent improvements together, as you listed in the description. At least
Also, I noticed that a narrower approach, #17202, just skipping |
2 participants
Problem
Running
pip install -e .repeatedly (as riot does on every test run) caused unnecessary full rebuilds of native extensions:pip-build-env-XXXXXtemp dir each run, giving Cython's.pxdheaders "now" timestamps → Cython re-generates.cfiles → setuptools sees the fresh.cand recompiles the extension.build_libtemp dir, so cmake's object files and built.sowere discarded after every run → full C++ recompilation on everypip install -e .._native,_memalloc,_ddup,_stack), wasting time and disk space on redundant identical builds.remove_native_extensions()was called on every build, deleting the inplace.sofiles that would otherwise have let setuptools skip the rebuild.Solution
Persistent build artifacts anchored to the source tree
All build artifacts are now stored in platform-tagged directories inside the source tree's
build/folder, which survives across pip invocations:build/temp.<plat>/— compiler object files (.o)build/cmake.<plat>/— cmake build dirs for cmake extensionsbuild/cmake_deps.<plat>.<mode>/— pre-built shared dependencies (Abseil)finalize_optionspinsbuild_tempto this persistent location so object files are reused for partial recompilation when only some.cppfiles change.Incremental skip for setuptools extensions
_build_extension_incrementaluses the inplace.soin the source tree as a stable freshness anchor. Instead of comparing the generated.cfile (which Cython always regenerates due to pip's fresh.pxdtimestamps), it compares the original.pyxsource. If the.pyxhasn't changed, the extension is skipped. After building, the inplace copy is kept in sync so future checks are correct.The skip logic is shared between setuptools and cmake extensions via a new
_skip_if_up_to_date(ext_name, anchor_so, sources, force)helper.Shared Abseil pre-build
A new
FETCH_CONTENT_DEPSlist insetup.pydeclares all cmake shared dependencies (currently Abseil)._prebuild_cmake_depsbuilds them once into a platform-tagged install prefix and passesCMAKE_PREFIX_PATH+FETCHCONTENT_TRY_FIND_PACKAGE_MODE=ALWAYSto all cmake extensions, so they resolve Abseil viafind_packageinstead of downloading and building their own copy. The build now fails fast if this step fails — there is no silent fallback to per-extension inline builds.IAST
_nativeand_memallocmigrated to plain setuptools extensionsBoth extensions were previously built via cmake (
CMakeExtension), which forced a cmake configure + build + install cycle on every run. They are now standardExtensionsubclasses:AbslExtension— injects Abseil include dirs and static libs at build time via aprepare(build_cmd)hook called frombuild_extension.MemallocExtension(AbslExtension)— additionally injects the Rust-generated C headers and thelibdd_wrapperlink arg.This makes them fully incremental via the standard setuptools mechanism plus the persistent
build_tempdirectory.Generated cmake deps CMakeLists.txt
The static
cmake_deps/CMakeLists.txtis replaced by a generator:_generate_cmake_deps_filerendersFETCH_CONTENT_DEPSinto aCMakeLists.txtwritten tobuild/cmake_deps.<plat>.<mode>/src/at build time. Adding or bumping a dependency is now a single-line change insetup.py.DONT_COMPILE_ABSEILmachinery removedThe
DD_COMPILE_ABSEIL=0env var, theDONT_COMPILE_ABSEILpreprocessor define, and all related conditional paths are removed. Abseil is now always built and always used in release builds. TheNDEBUGguards in_memalloc_heap.cppandtaint_range.hremain (debug builds fall back tostd::containers naturally), but the redundant!defined(DONT_COMPILE_ABSEIL)clauses are dropped.CI caching extended to cover Abseil
scripts/ext_cache.pynow supports directory-tree artifacts alongside single-file artifacts: a trailing/in the#EXTHASHtarget path signals thatshutil.copytreeshould be used instead ofshutil.copy.ExtensionHashes.runemits a__cmake_deps__entry keyed on the fullFETCH_CONTENT_DEPScontent, platform tag, and compile mode. When the hash matches, the entire Abseil install prefix is restored from cache and_prebuild_cmake_depsfast-paths immediately — Abseil is never recompiled.Files changed
setup.pyscripts/ext_cache.pyddtrace/profiling/collector/CMakeLists.txtMemallocExtension)ddtrace/appsec/_iast/_taint_tracking/CMakeLists.txtAbslExtension)ddtrace/appsec/_iast/_taint_tracking/tests/CMakeLists.txtddtrace/appsec/_iast/_ast/CMakeLists.txtcmake_deps/CMakeLists.txt(entire dir)_generate_cmake_deps_file)ddtrace/profiling/collector/_memalloc_heap.cpp#if defined(NDEBUG)guardddtrace/appsec/_iast/_taint_tracking/taint_tracking/taint_range.h#if defined(NDEBUG)guardCompatibility with
ddtest/ DockerThe incremental caching is safe when alternating between local builds and
ddtest-driven Docker runs because all artifact directories are tagged with the platform string (e.g.linux-x86_64vsmacosx-15.4-arm64) and the Python extension suffix encodes the platform in the.sofilename. Concretely:build/cmake_deps.linux-x86_64.RelWithDebInfo/(Docker) vsbuild/cmake_deps.macosx-15.4-arm64.RelWithDebInfo/(host) — different directories, no cross-contamination.build/cmake.<plat>/..sofiles:_encoding.cpython-311-x86_64-linux-gnu.so(Docker) vs_encoding.cpython-313-darwin.so(host) — different filenames, coexist safely..download_cache/_cmake_deps/) lives inside the project directory, whichddtestmounts at/home/bits/project, so it is shared and persistent across Docker runs.No changes to
ddtestordocker-compose.ymlare required.Testing
pip install -e .skips all unchanged extensions on second run_ddup,_stack) also skip correctlysetup.pyfixed as a drive-by cleanup