add github-org-* tools, dm-fork-sync, dm-github-policy, CI mock + live-audit#3
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f42c5c804
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| local owner="$1" is_source="$2" | ||
| local page=1 result body status | ||
| while : ; do | ||
| result="$(gh_api GET "/orgs/${owner}/repos?per_page=100&page=${page}&type=all")" |
There was a problem hiding this comment.
Use account-aware repo listing endpoint
The new sync script documents assisted-by-ai as a bot account, but list_repos hardcodes the org endpoint (/orgs/${owner}/repos). For a user account this returns 404, so the initial mirror repo lookup fails and the script exits before any fork/configuration work happens. Please branch on account type (or probe once) and use /users/{owner}/repos for user accounts.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 045c0c8. detect_account_type() probes /users/{owner} (which works for both User and Organization accounts) and reads .type, then list_repos routes to /users/{owner}/repos or /orgs/{owner}/repos accordingly. Result is cached per owner. create_fork similarly omits the organization body field when the mirror is a User.
Generated by Claude Code
| ## remote names, rather than duplicating info. | ||
| set_batch_current_package_remote_list() { | ||
| batch_current_package_remote_list=( "github-${batch_current_project_name,,}" ) | ||
| batch_current_package_remote_list=( "github-${batch_current_project_name,,}" "assisted-by-ai" ) |
There was a problem hiding this comment.
Don’t push assisted-by-ai for unsupported source orgs
This change adds assisted-by-ai to every package’s push/fetch remote list, including the special derivative-maker repo, but dm-fork-sync only provisions mirrors from Kicksecure and Whonix. When a derivative-maker mirror branch is missing, pkg_git_push_remotes now attempts diffs/pushes against nonexistent refs/remotes/assisted-by-ai/*, causing background git failures and aborting at wait_for_git_processes. Gate this remote by project/source org (or mirror derivative-maker as well).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Skipping — the premise doesn't hold for this repo layout:
derivative-makerexists asKicksecure/derivative-makerandWhonix/derivative-maker, sodm-fork-syncdoes mirror it (the first source org wins on the name collision inassisted-by-ai, which is the intended behavior).- The "missing remote-tracking ref" case is already handled by
internal_git_diffatusr/bin/dm-packaging-helper-script:678-700: it catches git's exit 128, pauses for human inspection (read -r _), then deliberately falls through and returns 0 so the caller attemptsgit push, which fails visibly with the same git error rather than silently skipping. That's the documented design — see the comment block at line 681-687.
So the worst case after a fresh remote add is a visible push failure (and only until the first dm-fork-sync run + pkg_git_fetch_remotes populate the tracking refs), not the silent abort the comment describes. Gating per source org would add complexity for no practical gain.
Generated by Claude Code
Codex review P1 (PR #3): list_repos hardcoded /orgs/{owner}/repos, which 404s for User accounts. assisted-by-ai is currently a User account, so the mirror lookup failed before any fork ran. Add detect_account_type() that probes /users/{owner} (returns .type "User" or "Organization") and caches the result. list_repos now selects /users/{owner}/repos or /orgs/{owner}/repos accordingly, including the right query parameters per endpoint. create_fork also adapted: the forks endpoint takes `organization` only when targeting an org. For a User mirror, omit it - the fork goes to the authenticated user, which must therefore be MIRROR_ORG.
accf1ab to
049d68e
Compare
bc4ddf0 to
f1e84ce
Compare
Three coordinated additions:
1. Generic GitHub-org tooling under usr/bin (github-org-clone,
github-org-fork, github-org-push, github-org-set-fork-approval)
plus the shared library github-org-lib.bsh.
2. Project-policy wrappers dm-fork-sync and dm-github-policy.
3. CI mock-API test suite under ci/, plus two GitHub Actions
workflows (mock-tests, live audit).
Tools (usr/bin/github-org-*):
- github-org-clone <source-org> [<dest-dir>]
Clone or fast-forward every public, non-archived, non-fork repo
from the source into <dest-dir>/<repo-name>/. Idempotent.
- github-org-fork <source-owner> <target-owner>
Fork every selected source repo into the target. Source/target
may each be a User or an Organization. Optional flags configure
target settings (--disable-issues/wiki/projects, --actions
{enable|disable|leave}, --workflow-perms {read|write|leave}).
- github-org-push <source-dir> <target-owner>
Push HEAD (and optionally tags) from each clone under source-dir
to a same-named target repo. Refuses to push if missing.
- github-org-set-fork-approval <org> [<policy>]
Set the org-level fork-PR contributor approval policy. Default
policy: all_external_contributors.
Common defaults: skip private/archived/fork repos on the source
side; 4 parallel jobs where applicable; --dry-run, -v/--verbose,
--include/--exclude REGEX, --ssh|--https. Auth via GITHUB_TOKEN env
or chmod-600 ~/.config/github-token. No gh CLI dependency.
Shared library (usr/libexec/developer-meta-files/github-org-lib.bsh):
- Token resolution from env or chmod-600 token file. Refuses
symlinks (defeats stat-then-read perm-check).
- ghorg_api: token via curl --config from stdin so it never appears
in argv (no ps aux leak). Pinned to https with --proto / --proto-
redir; bounded by --connect-timeout / --max-time; capped at
--max-filesize 10 MB.
- Rate-limit retry on HTTP 429, 5xx, and 403-rate-limit body.
Honors Retry-After and X-RateLimit-Reset; falls back to
exponential backoff. GHORG_MAX_RETRIES, GHORG_MAX_BACKOFF_SECONDS.
- Mock mode: GHORG_MOCK=1 GHORG_MOCK_DIR=path serves canned
responses from local fixtures. Missing fixture returns HTTP 599
so tests fail loudly.
- Token-handling functions wrap with `local -; set +x` so an
enclosing set -x cannot leak the secret.
- Account-type probe (User vs Organization) with caching;
paginated listing routed to the right endpoint. For User owners
with --include-private, switches to /user/repos?affiliation=owner
and refuses if the listed user is not the auth user.
- ghorg_repo_lookup: 0 = exists, 1 = 404, 2 = other failure.
- ghorg_authenticated_user (cached) + ghorg_repo_parent (for
cross-source name-collision detection).
- ghorg_validate_name: every API-supplied identifier must match
^[A-Za-z0-9._-]+$, length-capped, no leading dash, no embedded
"..", not equal to . / .. / .git. Allows .github / .gitignore.
- Safe display via sanitize-string for any error message that
includes API-derived bytes.
Project-policy wrappers (usr/bin/):
- dm-fork-sync: thin wrapper around github-org-fork. Mirrors
source orgs into the bot account.
- dm-github-policy: applies project-wide GitHub security policy.
Modes: default --apply, --dry-run, --audit (read-only). End-of-
run summary with ok / warn / dry-run / skip counts.
Settings applied at the org level:
- Fork-PR contributor approval = all_external_contributors
- Default workflow GITHUB_TOKEN permissions = read; cannot
create or approve pull requests
- Allowed Actions = github-owned + verified creators
- Member policy: default-perm=read, no member create
- Code Security Configurations API: a single org-level
configuration enables secret scanning, push protection,
Dependabot alerts and security updates, dependency graph,
explicitly disables CodeQL default setup and private
vulnerability reporting; attached to all existing repos and
set as the default for new repos. Replaces the deprecated
PATCH /orgs/{org} security-defaults fields and the
/orgs/{org}/{product}/enable_all family.
- Default-branch ruleset: block force-push, block deletion,
require signed commits, bypass_actors=[] (no admin override).
Deliberately does not include the pull_request rule
(maintainers push directly to master).
- Tag ruleset: block deletion, block re-pointing, require
signed tags, bypass_actors=[].
Per-repo settings (no org-wide endpoint exists):
- has_wiki=false, has_issues=true.
SKIP+notify with UI URL pointers for: 2FA enforcement, PAT
policies, App/OAuth policies. Verified against the GitHub
OpenAPI spec to be UI-only.
--audit reports current state of every applied setting plus
members lacking 2FA, existing rulesets, fine-grained PAT
activity, installed GitHub Apps, org-level webhooks lacking a
secret (Scorecard "Webhooks" Critical-risk gap), and
.github/dependabot.yml file presence per repo (Scorecard
"Dependency-Update-Tool" file-based detection).
Help text documents the OSSF Scorecard mapping per check
(Branch-Protection ceiling = Tier 1 deliberate; Code-Review,
CodeQL, Maintained = out of scope; etc.).
Test-only configuration: ORGS in dm-github-policy and SOURCE_ORGS
in dm-fork-sync currently target ('org-ai-assisted'). The
production list ('Kicksecure', 'Whonix') is preserved as a
single-line comment above the active line for easy revert.
dm-packaging-helper-script: same comment-out pattern; the
batch_current_package_remote_list mirror is org-ai-assisted (was
assisted-by-ai); the corresponding remote-add line is added in
pkg_git_remotes_add.
CI infrastructure (ci/ + .github/workflows/):
- ci/fixtures/: 32 canned responses for the test org, the auth
user, /user/repos, every org-policy GET that audit and dry-run
paths hit, plus webhooks list and per-repo dependabot.yml
contents lookups.
- ci/tests/test_*.sh: 10 mock-API tests covering github-org-*
arg parsing, dry-run flow, --get/--audit, the validator's
attack-pattern rejection, the rate-limit-wait header parser,
and a shellcheck-clean enforcement test that subjects every
authored file to project-wide .shellcheckrc rules.
- ci/test-github-org-tools.sh: runs every test_*.sh; quiet on
success, prints captured logs on failure. Refuses to run
outside CI=true.
- ci/install-genmkfile.sh + ci/install-helper-scripts.sh:
standalone install scripts for CI; refuse to run outside
CI=true.
- ci/probe-live-unauth.sh: best-effort live unauth smoke against
the real REST API; exits 0 with a skip message if the rate-
limit bucket is exhausted.
- .github/workflows/test-github-org-tools.yml: runs the suite
on push to master/claude/** and on every PR. Container
debian:trixie. paths: filter so it only runs on github-org-*,
dm-* tools, github-org-lib, ci/, or the workflow itself.
permissions: contents: read only.
- .github/workflows/audit-live.yml: manual-trigger
(workflow_dispatch only) live audit using a repo secret
GHORG_AUDIT_TOKEN (read-only fine-grained PAT). Guarded by
github.repository check so forks cannot use the secret;
pre-flight step verifies the secret is present and exits 1
otherwise.
Strict shell options on every authored script:
set -o errexit / nounset / pipefail / errtrace
shopt -s inherit_errexit / shift_verbose
Long option names everywhere (--quiet, --fixed-strings,
--ignore-case, --extended-regexp, --invert-match, --count,
--line-number, --lines=N). End-of-options separator (--) on
grep, head, sed, find, basename, sanitize-string calls that
take patterns or potentially untrusted strings.
Header on every authored file:
## Copyright (C) 2026 - 2026 ENCRYPTED SUPPORT LLC <...>
## See the file COPYING for copying conditions.
## AI-Assisted
The "AI-Assisted" marker discloses substantial AI involvement
without naming any specific tool or person, so the disclosure
does not invite per-file fame creep on minor contributions.
Defensive permissions on the workflow files: actions: write
intentionally not granted. actions/upload-artifact@v7 was tested
empirically against contents: read alone (run id 24967508066);
the artifact backend uses the workflow run's own identity, not
the GITHUB_TOKEN's permission scope.
521e080 to
d5a8747
Compare
Summary
Single-commit PR adding GitHub-org tooling, project-policy wrappers, and a mock-API CI test suite to developer-meta-files. The code is ready to apply project security policy at the org level, with a deliberate testing detour through
org-ai-assistedbefore flipping back toKicksecure+Whonix.Tools (
usr/bin/github-org-*)github-org-clone <source-org> [<dest-dir>]-- clone or fast-forward every public, non-archived, non-fork repo into<dest-dir>/<repo-name>/. Idempotent.github-org-fork <source-owner> <target-owner>-- fork every selected source repo into the target. Both may be User or Organization. Optional flags configure target settings.github-org-push <source-dir> <target-owner>-- push HEAD (and optionally tags) from each clone undersource-dirto a same-named target repo. Refuses if missing.github-org-set-fork-approval <org> [<policy>]-- set the org-level fork-PR contributor approval policy. Defaultall_external_contributors.Common across all four:
--dry-run,-v/--verbose,--include/--exclude REGEX,--ssh|--https,--jobs N. Auth viaGITHUB_TOKENenv or chmod-600~/.config/github-token. NoghCLI dependency.Shared library (
usr/libexec/developer-meta-files/github-org-lib.bsh)ghorg_api: token viacurl --config -from stdin (no argv leak), pinned tohttps, bounded by--connect-timeout/--max-time/--max-filesize. Token-handling functions wrap withlocal -; set +xso an enclosingset -xcannot leak the secret.Retry-AfterandX-RateLimit-Reset. Falls back to exponential backoff. Caps viaGHORG_MAX_RETRIESandGHORG_MAX_BACKOFF_SECONDS.GHORG_MOCK=1 GHORG_MOCK_DIR=pathserves canned fixture files. Missing fixture = HTTP 599 (loud failure).--include-privatefor User owners switches to/user/repos?affiliation=ownerand refuses if listed user is not the auth user.ghorg_validate_name:^[A-Za-z0-9._-]+$, length-capped, no leading dash, no embedded... Allows.github/.gitignore. Reserved names (.,..,.git) explicitly rejected.sanitize-stringfrom helper-scripts.Project-policy wrappers (
usr/bin/)dm-fork-sync-- thin wrapper aroundgithub-org-fork. Mirrors source orgs into the bot account.dm-github-policy-- applies project security policy. Modes: default--apply,--dry-run,--audit(read-only). End-of-run summary withok/warn/dry-run/skipcounts; non-zero exit on anywarn.Settings applied at org level:
all_external_contributorsread; cannot create or approve PRsdefault_repository_permission=read,members_can_create_repositories=falsePATCH /orgs/{org}fields and the deprecated/orgs/{org}/{product}/enable_allfamily): one configuration enables secret scanning + push protection + Dependabot alerts/updates + dependency graph; CodeQL default-setup and private vulnerability reporting explicitly disabled. Attached to all existing repos and set as default for new repos.bypass_actors=[]-- nobody, including org owners, can override.bypass_actors=[].Per-repo settings (no org-wide endpoint):
has_wiki=false,has_issues=true.SKIP+notify with UI URL pointers for: 2FA enforcement, PAT policies, App/OAuth policies. Verified against the OpenAPI spec to be UI-only at this time.
Audit-only sections (--audit): members lacking 2FA, existing rulesets, fine-grained PAT activity, installed GitHub Apps, org-level webhooks lacking a secret (Scorecard "Webhooks" Critical-risk gap), and
.github/dependabot.ymlfile presence per repo (Scorecard "Dependency-Update-Tool" file-based detection).Help text documents the OSSF Scorecard mapping per check (Branch-Protection ceiling = Tier 1 deliberate; Code-Review, CodeQL, Maintained = out of scope).
Testing-only configuration (currently active on this branch)
dm-github-policy::ORGS,dm-fork-sync::SOURCE_ORGS, anddm-packaging-helper-script::batch_current_package_remote_listall currently targetorg-ai-assistedonly. The production list (Kicksecure,Whonix, plus theassisted-by-aiuser mirror) is preserved as a single-line comment above each active line for one-keystroke revert.pkg_git_remotes_adddefines remotes for bothassisted-by-aiandorg-ai-assistedso either can be activated without further edits.CI infrastructure (
ci/+.github/workflows/)ci/fixtures/-- 32 canned API responses for the test org, the auth user,/user/repos, every org-policy GET that audit and dry-run paths hit, plus webhooks list and per-repo dependabot.yml content lookups. All shapes verified against the current OpenAPI spec atgithub/rest-api-description@main.ci/tests/test_*.sh-- 10 mock-API tests covering arg parsing, dry-run flow,--get/--audit, validator attack-pattern rejection, rate-limit-wait header parsing, and a shellcheck-clean enforcement test.ci/test-github-org-tools.sh-- runs everytest_*.sh. Refuses to run outsideCI=true.ci/install-genmkfile.sh+ci/install-helper-scripts.sh-- standalone CI install scripts using thegenmkfile installflow (no ad-hoc symlinks).ci/probe-live-unauth.sh-- best-effort live unauth smoke that skips when the rate-limit bucket is exhausted..github/workflows/test-github-org-tools.yml-- runs the mock-API suite on push/PR.paths:filter so it only runs when github-related files change. Containerdebian:trixie.permissions: contents: readonly..github/workflows/audit-live.yml-- manual-trigger (workflow_dispatchonly) live audit using a repo secretGHORG_AUDIT_TOKEN(read-only fine-grained PAT). Guarded bygithub.repositorycheck so forks cannot use the secret; pre-flight step verifies the secret is present and exits 1 otherwise.Coding standards used
errexit nounset pipefail errtrace inherit_errexit shift_verbose.--quiet,--fixed-strings,--ignore-case,--extended-regexp,--invert-match,--count,--line-number,--lines=N).--ongrep,head,sed,find,basename,sanitize-stringcalls that take patterns or potentially untrusted strings.## AI-Assistedafter the standard copyright + COPYING header. Neutral disclosure, no name, no fame-creep on minor contributions.Live audit setup (post-merge)
assisted-by-aiuser account, resource-owner =org-ai-assisted, read-only scopes: Repository (Administration, Contents, Metadata, Webhooks) + Organization (Administration, Members, Webhooks). Expiration 7 days.GHORG_AUDIT_TOKEN.Test plan
audit-live.ymlafter merge; review report.org-ai-assistedtest config back toKicksecure+Whonix(uncomment the production line in three files:dm-github-policy,dm-fork-sync,dm-packaging-helper-script).dm-github-policy --dry-run, then real--apply, then manual UI flips for the SKIP-flagged settings (2FA, PAT policies, App/OAuth policies).dm-packaging-helper-scriptpush flow.https://claude.ai/code