merging in patch by tanyastickles · Pull Request #63 · HubSpot/hadoop

tanyastickles · 2026-02-11T20:46:23Z

Issue

We want to pull in this PR from apache/hadoop to unblock Java 25.

… of invalidated blocks

* Build Hadoop for arm64 * Rely on buildpack for docker usage * Use Java 11 * try without maven cache * Compute env vars inside build.sh * use buildpack maven support * back to master dependencies Co-authored-by: Charles Connell <cconnell@hubspot.com>

Co-authored-by: Charles Connell <cconnell@hubspot.com>

) Co-authored-by: Charles Connell <cconnell@hubspot.com>

Co-authored-by: Charles Connell <cconnell@hubspot.com>

…fallback (#26) Co-authored-by: Charles Connell <cconnell@hubspot.com>

Hadoop release process involves creating a new branch for every release, so we can't easily follow the expected process of having a branch per major release. Instead we need to have a branch per release as well, and cherry-pick our changes over to each new release. Update our build script and rpm spec to account.

Co-authored-by: Charles Connell <cconnell@hubspot.com>

* Update .blazar.yaml * Update .blazar.yaml * Update .blazar.yaml * Update .blazar.yaml

- Update build versioning to be based around patch versions - Use single-module builds to ease dependency graph issues - Force HBase1 building to work around activeByDefault behavior

…ble with node.js 12 to avoid compilation error. (apache#5827). Contributed by Masatake Iwasaki Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> (cherry picked from commit a822a3c)

#32) * SaslPropertiesResolver should not allow mutation of its internal state * Keep SASL properties map sorted --------- Co-authored-by: Charles Connell <cconnell@hubspot.com> (cherry picked from commit 6761522)

* Persist build env variables withour build rc file * Add clarifying comment * Cleanup HADOOP_VERSION environment variable * Copy changes to all blazar yaml files * Revert "Copy changes to all blazar yaml files" This reverts commit 5f4457e. --------- Co-authored-by: Samuel McCann <smccann@hubspot.com>

…utStream (#41)

…ddTransferRateMetric(). (apache#6290). Contributed by Xing Lin. Reviewed-by: Ravindra Dingankar <rdingankar@linkedin.com> Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com> Reviewed-by: Tao Li <tomscut@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

- Change build order to avoid deploy plugin defect - Remove build exclusion - Restore original location for argument, remove weird skip config - Move the deployAtEnd to a different spot

- Leave config in same place - Disable deployAtEnd - Increase build timeout for Hadoop

* Use new rpm buildpack * Update .blazar.yaml --------- Co-authored-by: Scott Williams <srwilliams@hubspot.com>

Co-authored-by: Charles Connell <cconnell@hubspot.com>

…twork-distance FileSystem metrics (#51)

…its in a single thread (apache#5706) (#52) Contributed by Moditha Hewasinghage Co-authored-by: Moditha Hewasinghage <33624668+modithah@users.noreply.github.com>

…15865 (apache#6223) (#53) Co-authored-by: Hiroaki Segawa <goto@stonedot.com>

This will help us use a performant job submitter in SKIR

* log the timestamp difference instead of throwing exception. * keep trailing slash for s3 filesystems use. * revert to original * add check if the cluster is a S3 cluster * null check * more universal check

…pache#5256) (#58) Contributed by Sebastian Klemke Co-authored-by: Sebastian Klemke <3669903+packet23@users.noreply.github.com>

…lease renewal interval (#59)

* Diagnose build issue * Different config * use bp branch * Remove BP branch --------- Co-authored-by: Jared Stehler <jstehler@hubspot.com>

* use offstack buildpack for module * apply patch (#66) Co-authored-by: Tanya Stickles <tstickles@hubspot.com> --------- Co-authored-by: Tanya Stickles <tanyastickles@gmail.com> Co-authored-by: Tanya Stickles <tstickles@hubspot.com>

rmdmattingly

This backport of apache#7886 is missing two of the three call-site migrations in UserGroupInformation.java. The upstream PR replaces all three uses of the deprecated Subject APIs, but this PR only migrates getCurrentUser() and leaves the two doAs() overloads untouched.

Both still call Subject.doAs() directly, which is deprecated-for-removal alongside Subject.getSubject() and will throw UnsupportedOperationException on Java 25:

UserGroupInformation.java ~L1540 — doAs(PrivilegedAction<T>):

- return Subject.doAs(subject, action);
+ return SubjectUtil.doAs(subject, action);

UserGroupInformation.java ~L1556 — doAs(PrivilegedExceptionAction<T>):

- return Subject.doAs(subject, action);
+ return SubjectUtil.doAs(subject, action);

These are heavily used code paths — UGI.doAs() is the standard way to execute code as a given user. Without these changes the Java 25 goal is only partially addressed.

bbeaudreault and others added 30 commits July 18, 2023 16:33

Add RPM build

84e4824

HubSpot Backport HDFS-16261 Configurable grace period around deletion…

4f7b758

… of invalidated blocks

Fix build by avoiding building modules that require node.js

818d5fb

Give hadoop-rpm a name in Blazar

d2da4e7

Save build time by not generating Javadocs

31ed7c8

Build for CentOS 8

d0dd4d8

Pin build to JDK 11

182d14a

Tweak how YUM repo is specified (#9)

a1a23e6

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Fix where MAIN_BRANCH env var is defined (#10)

140f8c3

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Leverage arm64 hardware for faster builds (#11)

63254fc

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Build with less memory (#15)

3506898

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Build RPM with zstd for speed (#16)

76f8582

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Add a migration mode for block access tokens (#17)

8de4415

dont enable token auth method during migration mode (#19)

84a6ca1

Check NameNode principal pattern in GetJournalEditServlet (#20)

646ddcc

Co-authored-by: Charles Connell <cconnell@hubspot.com>

New setting unsafe.dfs.encrypt.data.transfer.plaintext.fallback (#21)

f31e4e8

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Accept encrypted or authenticated data transfer connections, take 3 (#24

62f5236

) Co-authored-by: Charles Connell <cconnell@hubspot.com>

Supply SASL properties when using dfs.data.transfer.accept.sasl (#25)

2d8d558

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Accept any QOP when using unsafe.dfs.encrypt.data.transfer.plaintext.…

d6159ad

…fallback (#26) Co-authored-by: Charles Connell <cconnell@hubspot.com>

Remove legacy RPM dependencies: chkconfig, service, and init-functions

251a311

Fix NameNode principal check from #20 per upstream changes

56e914b

Build Hadoop into maven (#13)

957853d

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Add HubSpot specific Dispatcher changes (#28)

63e093a

Update build configuration to not require write to RC file (#31)

c1ce269

* Update .blazar.yaml * Update .blazar.yaml * Update .blazar.yaml * Update .blazar.yaml

Preparing for 3.3.6 upgrade

414e189

- Update build versioning to be based around patch versions - Use single-module builds to ease dependency graph issues - Force HBase1 building to work around activeByDefault behavior

bbeaudreault and others added 25 commits January 31, 2024 11:31

Configurably use WeakReferencedElasticByteBufferPool in DFSStripedInp…

0b5c2f4

…utStream (#41)

Use MAVEN_BUILD_ARGS instead of MAVEN_ARGS (#43)

d948d97

Update Maven deploy plugin to fix deployAtEnd issue (#42)

5c11924

- Change build order to avoid deploy plugin defect - Remove build exclusion - Restore original location for argument, remove weird skip config - Move the deployAtEnd to a different spot

Use Java native calls for permissions control (#44)

819b6f6

Handle direct file copy correctly for S3 use cases (#45)

93091a3

- Leave config in same place - Disable deployAtEnd - Increase build timeout for Hadoop

Use new rpm buildpack (#46)

343dfc3

* Use new rpm buildpack * Update .blazar.yaml --------- Co-authored-by: Scott Williams <srwilliams@hubspot.com>

Relax the dir name check, username can contain .. (#47)

e5e6bbe

upload RPMs to AnyLinuxVersion (#48)

2aaabe1

Co-authored-by: Charles Connell <cconnell@hubspot.com>

Add backreference to prevent bad finalize behavior (#49)

2224a32

HubSpot Backport: MAPREDUCE-6660 Add MR Counters for bytes-read-by-ne…

56a07b7

…twork-distance FileSystem metrics (#51)

HubSpot Backport: HADOOP-18757. S3A Committer only finalizes the comm…

c4c25b0

…its in a single thread (apache#5706) (#52) Contributed by Moditha Hewasinghage Co-authored-by: Moditha Hewasinghage <33624668+modithah@users.noreply.github.com>

HubSpot Backport: HDFS-17024. Potential data race introduced by HDFS-…

c7949ba

…15865 (apache#6223) (#53) Co-authored-by: Hiroaki Segawa <goto@stonedot.com>

Adds a post copy files hook for configuring the job (#54)

b68bc7d

This will help us use a performant job submitter in SKIR

Add unresolved local service address recovery (#55)

da7a7b2

Remove Timestamp Check for S3 (#56)

f3312a2

* log the timestamp difference instead of throwing exception. * keep trailing slash for s3 filesystems use. * revert to original * add check if the cluster is a S3 cluster * null check * more universal check

Acknowledge that builds run on Alma Linux 9 (#57)

bf6b696

HubSpot Backport: HADOOP-18583. Fix loading of OpenSSL 3.x symbols (a…

c3e0486

…pache#5256) (#58) Contributed by Sebastian Klemke Co-authored-by: Sebastian Klemke <3669903+packet23@users.noreply.github.com>

HubSpot Backport: HDFS-17685: Option to explicitly choose DFS client …

0ff46e9

…lease renewal interval (#59)

Remove tracking URL log line (#60)

7a8b0d5

Fix Hadoop Build (#61)

6f10ed5

* Diagnose build issue * Different config * use bp branch * Remove BP branch --------- Co-authored-by: Jared Stehler <jstehler@hubspot.com>

merging in patch

64868d0

delete test for now

54eff57

use offstack buildpack for module (#62)

f991a5d

* use offstack buildpack for module * apply patch (#66) Co-authored-by: Tanya Stickles <tstickles@hubspot.com> --------- Co-authored-by: Tanya Stickles <tanyastickles@gmail.com> Co-authored-by: Tanya Stickles <tstickles@hubspot.com>

Merge branch 'hubspot-3.3.6' into ts_3.3.6-with-patch

35c671b

rmdmattingly requested changes Feb 18, 2026

View reviewed changes

This was referenced Feb 18, 2026

Backport HADOOP-19212: Hadoop UGI compatible with Java 25 #67

Closed

Backport HADOOP-19212: Hadoop UGI compatible with Java 25 #68

Merged

charlesconnell force-pushed the hubspot-3.3.6 branch 2 times, most recently from 0057a70 to c1f38bb Compare July 29, 2026 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

merging in patch - #63

merging in patch#63
tanyastickles wants to merge 66 commits into
hubspot-3.3.6from
ts_3.3.6-with-patch

tanyastickles commented Feb 11, 2026 •

edited

Loading

Uh oh!

rmdmattingly left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

tanyastickles commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Uh oh!

rmdmattingly left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

tanyastickles commented Feb 11, 2026 •

edited

Loading