Skip to content

Resolve the Hugging Face token from HF_TOKEN or a stored login - #87

Open
kobihikri wants to merge 1 commit into
google-research:mainfrom
kobihikri:fix/hf-token-from-env
Open

Resolve the Hugging Face token from HF_TOKEN or a stored login#87
kobihikri wants to merge 1 commit into
google-research:mainfrom
kobihikri:fix/hf-token-from-env

Conversation

@kobihikri

Copy link
Copy Markdown

While reading the Hugging Face upload utility I noticed that --token is mandatory whenever --repo_id is set:

if not FLAGS.token:
raise ValueError("Hugging Face token is required when repo_id is provided.")
from huggingface_hub import HfApi # pylint: disable=g-import-not-at-top
api = HfApi(token=FLAGS.token)

So the only way to run an upload today is to put a write-scoped Hugging Face token into the command line. On Linux /proc/<pid>/cmdline is world-readable, which means any other user on the machine can read that token out of ps for as long as the run lasts, and it is left behind in shell history. A conversion followed by an upload is not a quick command, so the window is not a small one.

huggingface_hub already handles this. Its exported get_token() resolves HF_TOKEN, then the token saved by hf auth login, and in CI an OIDC token via Trusted Publishers. It is present in the pinned huggingface-hub==1.21.0.

This change falls back to get_token() only when --token is not passed, so anyone already passing the flag sees no change — the flag still wins. The error message now names all three ways to supply a credential, and the flag's help text explains why the environment variable is the better default.

What I checked, and what I could not: the file parses, and get_token is exported by huggingface_hub 1.21.0. I could not exercise the upload end to end, because that needs write access to google/tabfm-1.0.0-pytorch.

If you would rather keep the hard requirement on the flag, I am happy to close this — it is your call.

Disclosure: I used an AI assistant to help find and prepare this change. I have reviewed and verified it myself.

@google-cla

google-cla Bot commented Aug 9, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

convert_and_upload.py requires --token whenever --repo_id is given, so today
the only way to upload is to put a write-scoped Hugging Face credential into
argv. On Linux /proc/<pid>/cmdline is world-readable, so that token is visible
to any other user on the machine through `ps` for as long as the conversion
and upload run, and it is left behind in shell history afterwards.

huggingface_hub already resolves credentials from HF_TOKEN and then from the
token saved by `hf auth login`, via the exported get_token(). This falls back
to that resolution when --token is not passed, and keeps the flag working
exactly as before for anyone who prefers it. The error message now names all
three options.
@kobihikri
kobihikri force-pushed the fix/hf-token-from-env branch from 5f91688 to 662e0ce Compare August 9, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant