Store the content address on Version instead of parsing full_name

[jenshenny]

rubygems#6674

Problem

Every component of full_name has its own column (rubygems.name, versions.number, versions.platform) — except the content address, which only existed embedded in the string. That forced two workarounds: the compact index paths re-extracted it with a hex-format heuristic (gated on ABI + platform to avoid mistaking date-style version numbers like 20260101 for addresses), and Version#content_address re-ran the DB uniqueness probe on every call. There was also no database-level guard against two versions of a gem computing the same identity — unique lower(full_name) indexes are impossible on this table (97 grandfathered case-collision groups from 2009–2010, most still live).

Solution

Store the address in versions.content_address:

• content_addressify! (a before_validation alongside full_nameify! / gem_full_nameify!) runs the uniqueness probe once at push time and stores the result; full_name is derived from the column
• /versions and /info select the column directly — the extraction heuristic is deleted
• Format validation (/\A[0-9a-f]{8,64}\z/, allow_nil) keeps junk addresses out
• A partial unique index enforces the real invariant, mirroring the classic platform uniqueness:

Tophat

Pushes three real .gem files through the pipeline (flag enabled), then verifies the column against the digest and full_name, the /versions + /info output, and the unique index.

Run from the repo root with the server on :3000 — paste the whole block into your console:

Tophat script (single paste)

bash <<'TOPHAT'
set -uo pipefail

BASE_URL="${BASE_URL:-http://localhost:3000}"
GEM="catophat$(date +%s)"
PLATFORM="x86_64-linux-musl"
BUILD_DIR=$(mktemp -d)
PASS=0 FAIL=0

step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
ok()   { PASS=$((PASS+1)); printf '  \033[32mPASS\033[0m %s\n' "$*"; }
bad()  { FAIL=$((FAIL+1)); printf '  \033[31mFAIL\033[0m %s\n' "$*"; }

push() { # push <file> <expected_body_substring>
  local file="$1" want_body="$2" resp code body
  resp=$(curl -s -w $'\n%{http_code}' -X POST "$BASE_URL/api/v1/gems" \
              -H "Authorization: $KEY" -H "Content-Type: application/octet-stream" \
              --data-binary "@$file")
  code=${resp##*$'\n'}
  body=${resp%$'\n'*}
  if [[ "$code" == "200" && "$body" == *"$want_body"* ]]; then
    ok "pushed → $code: ${body:0:110}"
  else
    bad "push → got $code; body: ${body:0:150}"
  fi
}

step "Preflight: server on :3000"
curl -sf "$BASE_URL/" > /dev/null || { echo "Server not running — start it with ./.agents/skills/run-rubygems-org/smoke.sh"; exit 1; }
ok "server is up"

step "Setup: user + push API key + feature flag"
RUNNER_LOG=$(mktemp)
KEY=$(bin/rails runner "
  user = User.find_or_create_by!(handle: \"catophat\") do |u|
    u.email = \"catophat@rubygems-test.org\"
    u.password = SecureRandom.hex(16)
    u.email_confirmed = true
  end
  FeatureFlag.enable_for_actor(FeatureFlag::CONTENT_ADDRESSABLE_GEM_PUSHES, user)
  raw = SecureRandom.hex(24)
  user.api_keys.create!(name: \"catophat-#{SecureRandom.hex(4)}\",
                        hashed_key: Digest::SHA256.hexdigest(raw),
                        scopes: %w[push_rubygem])
  puts \"KEY=#{raw}\"
" 2>"$RUNNER_LOG" | grep "^KEY=" | cut -d= -f2)
if [[ -n "$KEY" ]]; then
  ok "user + key ready, CONTENT_ADDRESSABLE_GEM_PUSHES enabled"
else
  bad "setup failed — bin/rails runner output:"
  tail -15 "$RUNNER_LOG" | sed 's/^/    /'
  echo "    (common causes: wrong ruby active — .ruby-version wants $(cat .ruby-version), you have $(ruby -v 2>/dev/null | cut -d' ' -f2); or pending migrations: bin/rails db:migrate)"
  exit 1
fi

step "Build: skinny 3.2 / skinny 3.4 / multi-ABI .gem files (same number + platform)"
ruby -rrubygems/package -e '
  gem_name, platform, build_dir = ARGV
  [["skinny32", "~> 3.2.0"], ["skinny34", "~> 3.4.0"], ["multi", ">= 3.2"]].each do |label, req|
    dir = File.join(build_dir, label)
    Dir.mkdir(dir)
    spec = Gem::Specification.new do |s|
      s.name        = gem_name
      s.version     = "1.0.0"
      s.platform    = platform
      s.summary     = "content address tophat (#{label})"
      s.authors     = ["tophat"]
      s.files       = []
      s.required_ruby_version = req
    end
    Dir.chdir(dir) { Gem::Package.build(spec) }
  end
' "$GEM" "$PLATFORM" "$BUILD_DIR" > /dev/null 2>&1 \
  && ok "built in $BUILD_DIR" || { bad "gem build failed"; exit 1; }

step "Push: through the real pipeline"
push "$BUILD_DIR/skinny32/$GEM-1.0.0-$PLATFORM.gem" "Ruby ABI 3.2"
push "$BUILD_DIR/skinny34/$GEM-1.0.0-$PLATFORM.gem" "Ruby ABI 3.4"
push "$BUILD_DIR/multi/$GEM-1.0.0-$PLATFORM.gem"    "Successfully registered gem: $GEM (1.0.0-$PLATFORM)"

step "Column: content_address stored, consistent with sha256 and full_name"
CHECK=$(bin/rails runner "
  r = Rubygem.find_by!(name: \"$GEM\")
  abi32 = r.versions.find_by!(ruby_abi: \"3.2\")
  abi34 = r.versions.find_by!(ruby_abi: \"3.4\")
  multi = r.versions.find_by!(ruby_abi: nil, platform: \"$PLATFORM\")

  checks = []
  checks << (abi32.content_address == abi32.sha256_hex.first(8))
  checks << (abi32.full_name == \"#{r.name}-1.0.0-#{abi32.content_address}\")
  checks << (abi34.content_address == abi34.sha256_hex.first(8))
  checks << (multi.content_address.nil?)
  checks << (multi.full_name == \"#{r.name}-1.0.0-$PLATFORM\")
  puts \"CHECK=#{checks.all? ? 'ok' : checks.inspect}\"
  puts \"ADDR32=#{abi32.content_address}\"
  puts \"ADDR34=#{abi34.content_address}\"
" 2>/dev/null | grep -E "^(CHECK|ADDR3[24])=")
eval "$(echo "$CHECK" | grep -E '^ADDR')"
[[ "$(echo "$CHECK" | grep '^CHECK=' | cut -d= -f2)" == "ok" ]] \
  && ok "column matches digest prefix and full_name derives from it ($ADDR32 / $ADDR34); multi is NULL" \
  || bad "column state wrong: $CHECK"

step "Index: /versions and /info read the column"
VERSIONS=$(curl -s "$BASE_URL/versions")
[[ "$VERSIONS" == *"$GEM 1.0.0-$ADDR32"* || "$VERSIONS" == *", 1.0.0-$ADDR32"* || "$VERSIONS" == *"1.0.0-$ADDR32,"* ]] \
  && ok "/versions contains the 1.0.0-$ADDR32 token" \
  || bad "/versions missing token: $(echo "$VERSIONS" | grep "^$GEM " || echo 'gem line absent')"
INFO=$(curl -s "$BASE_URL/info/$GEM")
[[ "$INFO" == *"1.0.0-$ADDR32 "* && "$INFO" == *"1.0.0-$ADDR34 "* && "$INFO" == *"platform:= $PLATFORM"* && "$INFO" == *"rubygems:>= 4.1.0.beta1"* ]] \
  && ok "/info lists both content-addressed lines with platform:= and normalized rubygems floor" \
  || bad "/info unexpected: ${INFO:0:250}"

step "Unique index: duplicate (rubygem, number, content_address) rejected at the database level"
DUP=$(bin/rails runner "
  r = Rubygem.find_by!(name: \"$GEM\")
  abi32 = r.versions.find_by!(ruby_abi: \"3.2\")
  dup = r.versions.build(number: \"1.0.0\", platform: \"arm64-darwin-25\", gem_platform: \"arm64-darwin-25\",
                         authors: [\"tophat\"], description: \"dup\", ruby_abi: \"3.2\",
                         sha256: abi32.sha256, spec_sha256: abi32.spec_sha256, size: 1,
                         full_name: \"#{r.name}-1.0.0-dup\", gem_full_name: \"#{r.name}-1.0.0-dup\",
                         content_address: abi32.content_address, canonical_number: \"1.0.0\")
  begin
    dup.save!(validate: false)
    dup.destroy
    puts \"DUP=inserted\"
  rescue ActiveRecord::RecordNotUnique
    puts \"DUP=rejected\"
  end
" 2>/dev/null | grep "^DUP=" | cut -d= -f2)
[[ "$DUP" == "rejected" ]] && ok "RecordNotUnique raised for duplicate content address" \
                           || bad "duplicate insert result: $DUP"

rm -rf "$BUILD_DIR"
printf '\n\033[1m%d passed, %d failed\033[0m\n' "$PASS" "$FAIL"
[[ $FAIL -eq 0 ]]
TOPHAT

Output (10 passed, 0 failed)

== Preflight: server on :3000
  PASS server is up

== Setup: user + push API key + feature flag
  PASS user + key ready, CONTENT_ADDRESSABLE_GEM_PUSHES enabled

== Build: skinny 3.2 / skinny 3.4 / multi-ABI .gem files (same number + platform)
  PASS built

== Push: through the real pipeline
  PASS pushed → 200: Successfully registered gem: catophat (1.0.0-fffb03fb, Platform: x86_64-linux-musl, Ruby ABI 3.2)
  PASS pushed → 200: Successfully registered gem: catophat (1.0.0-c9685ae9, Platform: x86_64-linux-musl, Ruby ABI 3.4)
  PASS pushed → 200: Successfully registered gem: catophat (1.0.0-x86_64-linux-musl)

== Column: content_address stored, consistent with sha256 and full_name
  PASS column matches digest prefix and full_name derives from it (fffb03fb / c9685ae9); multi is NULL

== Index: /versions and /info read the column
  PASS /versions contains the 1.0.0-fffb03fb token
  PASS /info lists both content-addressed lines with platform:= and normalized rubygems floor

== Unique index: duplicate (rubygem, number, content_address) rejected at the database level
  PASS RecordNotUnique raised for duplicate content address

10 passed, 0 failed

[girachawda]

Tophat looks good! And the code changes in general make sense!

QQ about something I noticed in how we validate content-addressable versions. In the Version model, a version is only considered content-addressable when it is platformed and has a ruby_abi. But in the compact index token rendering (lib/compact_index/gem_version.rb:6), we switch to the content-addressed token whenever content_address.present?.

That means an inconsistent row like platform: "x86_64-linux", ruby_abi: nil, content_address: "abcdef12" would not be considered content-addressable by the model, but would still be emitted in the compact index as 1.0.0-abcdef12.

How likely is it for that case to happen? And should we make these rules consistent?

[jenshenny]

@girachawda yes good point. Right now, we are setting a content address before validation if there's a Ruby ABI set, but we aren't checking if content address is nil if Ruby ABI is set. I added a validation to make sure Ruby ABI and content address is consistent.

[girachawda]

Thanks, this addresses my concern. I reran the tophat locally and also checked the validation path with a gem that had ruby_abi as nil. It wasn't considered content addressable which is expected ✅