Secrets Triage and Remediation: Revoke First, Clean Up Later
It is 09:14 on a Tuesday, and GitHub has just raised secret scanning alert 17 on billing-api, your private billing repository. The finding is an Azure Storage account key inside a connection string in src/, and although the file name says development, the key opens the production account that holds your invoices. It arrived with a commit on a feature branch 11 days ago, and the branch was merged to main 9 days ago. “Push protection” went on only yesterday evening, so it never saw the commit. The key has been readable by 31 people with access for 11 days, and a contractor’s fork may hold it too. Your first instinct may be to delete the line, force push and close the alert, and each of those steps would leave the key working. What do you do first?
For a lead with only a few minutes, the method fits in the four boxes below, and their links lead to the sections that explain each step.
What counts as a secret
The bearer test
RFC 6750 of 2012 describes a “bearer token” as one that any party in possession of it can use, with no further proof. Our “bearer test” applies the same idea: if a stranger held this value and nothing else, could they do something that they could not do before? If so, it is a secret, whatever it looks like and wherever it sits. API keys, personal access tokens, connection strings, cloud keys and private keys all pass, and so do webhook secrets, cookies, refresh tokens, SAS URLs, kubeconfigs and signing keys, and a signing key often has no shape at all.
| Kind | Sample shape | What the holder can do |
|---|---|---|
| API key | sk_live_EXAMPLE000000000000000000 |
Spend as your Stripe account |
| Personal access token | ghp_EXAMPLE00000000000000000000000000000 |
Act as the developer who created it |
| Connection string | Server= |
Log in to the database as that user |
| Cloud key | AKIAIOSFODNN7EXAMPLE, AWS’s public sample |
Act as that IAM user, with its secret key |
| Private key | -----BEGIN PRIVATE KEY----- |
Log in over SSH, or pose as your site |
| Signing key | Jwt__SigningKey=, often no shape at all |
Mint tokens that your services accept |
| Webhook secret | whsec_EXAMPLE00000000000000000000 |
Forge events that your service trusts |
| Cookie, refresh token | ghr_EXAMPLE00000000000000000000000000000 |
Carry on as a logged in user |
| SAS URL | ...?sp=rl&sig=EXAMPLE, a shared access signature |
Use what the URL grants until it expires |
| Kubeconfig | client-key-data: EXAMPLE, token: EXAMPLE |
Reach the cluster API as that user |
Note. The cloud key, kubeconfig and webhook secret rows rest on AWS’s page on managing access keys for IAM users, the Kubernetes kubeconfig v1 reference and Stripe’s page on webhooks.
Shape is a poor guide in either direction. The address below reads like configuration, yet its holder can post into your channel as your app, and Slack says that “Your webhook URL contains a secret.” By contrast, a -----BEGIN PUBLIC KEY----- file looks alarming and fails the test, because its holder can only check a signature, unless the same file hides a private key further down.
# In docs/alerts.md. Nothing here reads like a key, and it is still a secret
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX...
The address passes the bearer test because Slack’s own example request carries no other credential. So for the one in docs/alerts.md, replace the webhook, and treat the old address as leaked.
Other values deserve care and still fail the test. An internal host name or an AWS account number may help an attacker plan, but neither lets anyone act, and AWS says that account IDs are not considered secret. A bare AWS key ID fails too, although it stays a lead, because its secret key may be close by. Keep the rest out of your secrets queue, where they only bury the keys that matter. Some values are public by design, such as a Stripe pk_live_ key, and Is it a credential at all shows how to tell, and when that changes.
Static secrets and sprawl
The worst trouble comes from “static secrets”, created once and never changed, which live for years, copy easily and often outlast their owners. GitGuardian calls the result “secret sprawl”, and in its vendor figures on public code, more than 64 percent of the secrets that it found valid in 2022 still worked in January 2026. The same report, which GitGuardian also summarizes on its blog, counted 28,649,024 new secrets in public GitHub commits in 2025. Beyond public code, Verizon’s 2026 Data Breach Investigations Report puts credential abuse at 13 percent of initial access, and at 39 percent across a whole breach. Picture a key for the IAM user ci-deploy, created in 2023, which by 2026 sits on a laptop, in an Actions secret, a deleted script, a wiki, a chat and Terraform state. The team cannot say which of those copies a running service depends on, so the key is never revoked.
Generally, a secret’s life runs along one route: it is issued, stored, delivered, used, rotated and finally revoked. Copies branch off as it goes, to a laptop and a CI log where the value is delivered, and to a backup and a chat where it is used. A copy that leaves the route never rejoins it, so it works until someone revokes the value at the issuer.
Assess, fix, prove
At 717 DEV, we run a secrets response as a loop of three parts. Assess means that you find every secret, wherever it hides, with scanners and gates, and decide which ones are working credentials and how far they traveled. Fix means that you contain and end the credential, hunt for abuse, clean up what it left and replace it with something better, everywhere. Prove means that you show that the fix held with evidence that someone else could check, and keep it fixed.
The parts run in this order because each one consumes what the last produced, so you cannot choose between revoking now and rotating with overlap until you know what the value is and who uses it. It is a loop because proof often starts the next assessment, when a hunt turns up a second key or a record shows a missing log. On a bad day, though, a key that has gone public goes straight to revocation, and the rest of the assessment catches up beside containment.
Throughout, we follow one invented case, Example Systems, which keeps billing-api on Azure App Service. Every credential shown contains EXAMPLE, is built from zeros and x characters, or is a public value that its vendor prints, and output in a listing is a sample unless the text says that we ran it.
The first hour
GitHub’s advice starts from one assumption: “Once a secret has been committed to a repository, you should consider the secret compromised.” Treat it as compromised whether or not anyone has used it, and as a working credential until you can show that it is not. Removing the commit leaves the credential exactly as it was, because only the issuer can end the key, by “revocation”, and until it does, every clone, fork and backup holds a working copy.
- Stop. Do not delete anything yet: not the file, the branch, the alert or a log.
- Write it down, and tell. Record what leaked, start the log exports without waiting on them, and once an incident trigger applies, tell the owner and repository admins during or after the revocation, then security, legal and privacy.
- Choose. Is it public, or do the issuer’s logs show use by someone you cannot name? If so, revoke now: do step 5 first, take the failed calls, then come back to step 4. If not, you may use an “overlap rotation”, in the order below.
- Issue the replacement, and move every consumer onto it.
- Revoke the old one at the issuer, inside the response clock, since nothing else ends it.
- Hunt. Search the issuer’s logs across the whole “window of exposure”.
- Keep the right people told, from the owner to legal and privacy.
The full order, in Contain before you clean, runs past the hunt to cleanup and then proof. Revoking includes the sessions and tokens that the key spawned, and the hunt includes removing anything an attacker added. At step 3, a test, development or inactive key in a private repository, or a credential that cannot have two valid values at once, takes another move.
Step 3 is the one test of the first hour, and if either answer is yes, you revoke now. Answer the second from a first look at the issuer’s logs, never from silence. If both are no, an overlap rotation issues the replacement, moves every consumer and only then kills the old value, so that consumers see no outage. It needs an owner and a short window, because the leaked value keeps working throughout. Before you kill the old value, confirm that nothing still presents it. The rotation is complete only at that revocation, since the deploy alone does nothing for a copy in a chat or a backup.
Three moves feel responsible and do harm. Do not force push first, since a rewrite cannot reach forks, clones or pull request refs, and it changes the SHAs that your hunt depends on, so it comes last, if at all, as History comes last explains. Do not paste the secret into a ticket, a chat or an alert comment, each one more copy, and use the key ID or a token’s SHA-256 instead, never the hash of a weak password, which can be guessed. A GitHub resolution comment is permanent, since it cannot be edited or deleted; editing a pull request comment leaves the value in an edit history open to anyone with read access, and deleting that revision does not undo the leak. And do not close the alert, since removing the line changes nothing at the issuer. Flare found that when Docker Hub leakers removed a file, “in most cases, the associated credential was not revoked.” A finding closes on proof that the old key fails, as Prove the fix shows.
Twelve false assumptions
A missing control looks like a gap, and someone may eventually file a ticket for it. A false assumption looks like a finished job, so it is rarely checked again. Each of the twelve beliefs in this section is something a capable developer might say in a review, and each one is true of something, which is why it survives. A CI store does encrypt, a mask does hide, and a private repository does keep strangers out. The mistake is to stop there, since a secret that can be used has to leave the layer that protects it.
The CI/CD secret store
The first belief, that a secret in the CI/CD store is safe, needs the most care, because the usual rebuttal overshoots. Your CI secret store is rarely plain text, but it is rarely a boundary either. On some systems the value or its key sits on the server, and on all of them, anyone who can change what a pipeline runs can read what that pipeline can read. A pipeline can only use a secret in plain text, so the store protects it until a job starts. The warning that CI secrets sit unencrypted holds for a defined set of systems, so we state it in three tiers, from literally unencrypted to false as worded.
| Tier | What it means | Systems |
|---|---|---|
| 1. Literally unencrypted | Readable with no decrypt step | CodeBuild PLAINTEXT, Kubernetes Secrets in default etcd, the Azure agent key file on Linux and macOS, Jenkins plugins named in advisories, Argo CD Redis manifests with injected secrets, TeamCity uploaded SSH keys per its Security Notes, though its SSH keys page says encrypted |
| 2. Encrypted, key beside it | Whoever holds the server holds the secret | Jenkins, self managed GitLab, TeamCity by default, Bamboo with a split key |
| 3. False as worded | Same practical exposure | GitHub Actions, GitLab.com, Azure Pipelines, CircleCI, Bitbucket |
Note. Bamboo sits in tier 2 per its 9.0 documentation, and EKS on Kubernetes 1.28 or later encrypts Secrets by default.
Tier 1 is narrower than the warning suggests. On CodeBuild, PLAINTEXT is the default type, and upstream Kubernetes keeps Secrets in etcd unencrypted by default. A self hosted Azure agent keeps its RSA key in .credentials_rsaparams with no further encryption on Linux and macOS, as On the server sets out.
Once a job decrypts a secret, everything after that point is plain text, starting in the job environment, which every action in the job can reach. From there it runs out to every step, third party actions, child processes, the log, workspace and temp files, and caches and artifacts, all within reach of anyone on the runner host, and each stage has a wider audience than the settings page.
On the server
Tier 2 is easiest to see on a Jenkins controller, where, as Jenkins documents, the master key “is stored unencrypted on the Jenkins controller filesystem”, beside the values that it protects. The Javadoc of hudson.util.Secret warns that it “is not meant as a protection against code running in the same VM, nor against an attacker who has local file system access on Jenkins master.” Encryption at rest did not save CircleCI in December 2022 either, when malware on a laptop stole a session cookie and, as its January 2023 incident report puts it, “the third party extracted encryption keys from a running process”. In the twelve publicly reported CI incidents that we reviewed, the value was read at run time or the key went with it. Encryption at rest is worth having, but it does nothing to protect a value that a running job can already read.
On Jenkins, a backup that reaches the master key reaches the values as well, so keep master.key out of backups and move it with the jenkins. property. You can check the three files by name alone, as the sample output below shows, so nothing secret lands in your terminal.
# On your own controller. Names only, so this prints no secret.
$ ls -1 $JENKINS_HOME/credentials.xml $JENKINS_HOME/secrets/{hudson.util.Secret,master.key}
/var/lib/jenkins/credentials.xml # stored credentials, encrypted
/var/lib/jenkins/secrets/hudson.util.Secret # the key for generic secrets, itself protected
/var/lib/jenkins/secrets/master.key # protects the keys here, stored unencrypted
OWASP says it directly: “Often, maintainers of a GitLab/GitHub project can see or otherwise extract its secrets.” Across CI servers, five more places hold a value or its key, each with its own change to make.
| Where | What is there, and what to change |
|---|---|
Jenkins job config.xml |
Plugins, not core. Gogs Plugin “stored credentials unencrypted in job config.xml files”, and so did Aqua Security Scanner Plugin 3.2.8 and earlier. Read the advisories for what you run. |
/etc/ |
Holds db_key_base, which encrypts CI/CD variables in the database. Restrict it, and never store it beside a database backup. |
.credentials_rsaparams |
The agent’s RSA private key, mode 600 with no further encryption on Linux and macOS, DPAPI on Windows. Treat a self hosted agent as a credential. |
| TeamCity data directory | Its pages disagree on a “fixed key” or one “unique for each TeamCity installation”, but file or database access retrieves values. Hold TEAMCITY_ENCRYPTION_KEYS elsewhere. |
<workspace>@tmp/ |
File bindings, mode 0700, deleted on unbind. Until then, “any other build running concurrently on the same node will be able to read the contents of this file.” |
Platform by platform
Read the row for your own platform, and give what stays on disk, at the end of each row, the attention that it rarely gets. The tier beside each name follows the three tiers above. Note who can read Secrets in the Kubernetes row, and see Kubernetes under Replace it everywhere else for what to change there.
| System | Where it lives, and the key | Who reads it back, and what stays on disk |
|---|---|---|
| GitHub Actions, tier 3 | GitHub, encrypted at rest. Cipher not published. Key: GitHub, which decrypts to inject at run time. | Not readable in the UI or REST. Anyone with write access, through a workflow. On disk: the actions/checkout token, from v6 on, in plain text in $RUNNER_TEMP, until cleanup. |
| GitLab, tier 2, GitLab.com tier 3 | Database, with aes-256-cbc. Key: db_key_base, in the secrets file on the application host. |
Maintainers, unless masked and hidden. Anyone who can run a job. On disk: file type variables, in <builds_dir>/. |
| Azure Pipelines, tier 3 | Azure DevOps, under a 2048 bit RSA key. Key: Microsoft, on Azure DevOps Services. | Anyone who can alter the pipeline, since env: maps a value to a script. On disk: the agent key, on a self hosted agent, and secure files, which go when the job ends. |
| Jenkins, tier 2 | credentials.xml, encrypted. Key: master.key, unencrypted, protects hudson.util.Secret. |
Anyone who can read $JENKINS_HOME, or any job bound to the credential. On disk: file bindings, in <workspace>@tmp/, until unbind. |
| TeamCity, tier 2 | Data directory or database, scrambled. Key: a default key, or a custom one held elsewhere. | Any project administrator, or anyone who can change build code, or reach the files or database. On disk: uploaded SSH keys, plain text per its Security Notes and encrypted per its SSH keys page. |
| CircleCI, tier 3 | CircleCI, encrypted at rest. Cipher not published. Key: not published. In December 2022, keys were taken from a running process. | Anyone who can run a job or use SSH debug. On disk: not published for cloud. |
| Bitbucket, tier 3 | Secured variables. Method not published. Key: not published. | Anyone who can change the pipeline. The UI only replaces or deletes. On disk: not published. |
| CodeBuild, tier 1 by default | Project definition. PLAINTEXT is the default type. Key: none for PLAINTEXT. |
Anyone with codebuild:BatchGetProjects, in the console or CLI. On disk: not published. Builds run as root unless run-as is set. |
| Bamboo, tier 2, per its 9.0 docs | Database, AES 256, by keyword in the variable name. Key, split: the database, and BAMBOO-HOME/. |
Whoever holds both parts. A variable named TOKEN misses the rule. On disk: not published. Jobs get values as bash shell variables. |
| Kubernetes, tier 1 upstream | etcd, unencrypted by default. EKS on 1.28 or later encrypts. Key: none until EncryptionConfiguration is set. |
Anyone with get or list on Secrets, or who can create a Pod in the namespace. Mounted Secrets, in tmpfs. Argo CD manifests, in Redis. |
Note. “Not published” means that, as of November 2026, we could not find the answer in a primary source.
From the pipeline
The second route skips the server. A pipeline hands the secret to repository code, so anyone who can change that code controls the value, which OWASP calls “poisoned pipeline execution”. GitHub says it plainly: “Any user with write access to your repository has read access to all secrets configured in your repository.” GitLab adds that “malicious code can compromise both masked and protected variables”, and on GitHub, protecting main does not close the route, because a run uses “the version of the workflow” in its own commit. We will not print the attack, which GitLab illustrates in its own CI/CD variable security documentation, so the useful view here is the reviewer’s.
# Pull request 482, .github/workflows/ci.yml. Stop this one
+ - name: Update coverage badge
+ run: ./scripts/badge.sh # a script this same pull request adds
+ env:
+ DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} # a badge needs a number, not a deploy key
A reviewer should stop this diff, since the log will only show ***. Generally, four changes deserve a second reader: a new secrets. reference, a run: that points at a file in the same pull request, an unpinned uses:, and a wider permissions: block. For a new secret reference, ask whether the step needs it at all, because whatever it runs can read it. Dependencies count too: repointed tags in tj-actions/changed-files exposed secrets in logs in March 2025, and TanStack lost an OpenID Connect, or OIDC, token from runner memory in May 2026.
What closes the route differs by platform. On GitHub Actions, it is OIDC federation, environment secrets behind required reviewers, least GITHUB_TOKEN scope, CODEOWNERS on .github/workflows and SHA pins. As of November 2026, though, reviewers on Free, Pro and Team plans “are only available for public repositories”, and admins can bypass them by default. GitHub will also block pull_request_target by default in public repositories from December 2, 2026, after “brownouts” on November 9 and 16. On GitLab, it is protected variables, which reach only protected branches and tags, id_tokens for external secrets and a review of every .gitlab-ci.yml change, since the “Masked and hidden” setting guards only the UI and does nothing inside a job.
Azure Pipelines offers “workload identity federation”, with “Grant access permission to all pipelines” left off, CircleCI offers OIDC tokens with restricted contexts, and Bitbucket offers oidc: true on a step. On Jenkins, it is item scoped credentials, no executors on the built in node and a tight hold on Job/Configure. On TeamCity, it is remote secret parameters, for HashiCorp Vault only as of November 2026, “disposable, one-off build agents” and a tight hold on who can change build code, since that reaches “the values of the password fields”. On Bamboo, it is Plan Edit held tightly, since it reaches “the configuration of the plan and its jobs”, variable names with password or secret, so that encryption applies, and ephemeral agents. On CodeBuild, moving a value to PARAMETER_STORE or SECRETS_MANAGER changes who else can read it, while a build still reads it as before. Every platform gains from runners that take one job and disappear, and from rotation on suspicion, since logs show nothing. Federation has a limit worth stating: OIDC removes the stored secret, but you still have to control what the pipeline runs.
Check your own pipeline in 30 minutes
Each command below prints names and metadata, and never a value. For the same reason, the Kubernetes lines print the provider lines alone and never the file, the Jenkins line on credentials.xml matches only the scope and id tags, and the TeamCity line lists its files without printing them. Keep every --query and jq filter as written, since the CodeBuild query and the GitLab filter are what keep values out of your terminal.
# The 30-minute check, names and metadata only
# GitHub, from a clone. Listing collaborators needs write access itself.
R=repos/example-systems/billing-api; grep -n workflows .github/CODEOWNERS
gh api --paginate "$R/collaborators" --jq '.[] | select(.permissions.push) | .login'
gh api "$R/branches/main" --jq .protected; gh api "$R/rules/branches/main" --jq '.[].type'
gh secret list; gh api "$R/actions/organization-secrets" --jq '.secrets[].name'
gh api "$R/environments" --jq '.environments[]|{name,protection_rules,deployment_branch_policy}'
grep -rnoE 'secrets\.[A-Za-z0-9_]+|secrets: inherit' .github/workflows | sort -u
# CodeBuild. The --query keeps each value out of the output, so never drop it.
aws codebuild list-projects --query projects --output text | xargs -n 100 \
aws codebuild batch-get-projects --output json --query "projects[].[name,
environment.environmentVariables[?type!='SECRETS_MANAGER' && type!='PARAMETER_STORE'].name]" \
--names | jq -r '.[] | .[0] + " " + .[1][]?' | grep -iE ' .*(key|token|secret|pass|conn|cred)'
# Kubernetes, on a control plane you can see. Print the provider lines alone, never the file.
kubectl -n kube-system get pods -o yaml | grep -o 'encryption-provider-config=[^ "]*'
grep -oE '(identity|aescbc|aesgcm|secretbox|kms):' CONFIG_FILE_PRINTED_ABOVE # in order
# Jenkins, on the controller. Match these tags alone, since credentials.xml holds the values.
cd "$JENKINS_HOME"; grep -E '<(scope|id)>' credentials.xml
grep -E 'authorizationStrategy class|Item\.Configure|<numExecutors>' config.xml
# GitLab. The API returns each variable's value, so keep the jq filter.
P=projects/:fullpath
glab api --paginate $P/variables | jq -r '.[] | [.key, .protected, .masked, .hidden] | @tsv'
glab api --paginate $P/members/all | jq -r '.[] | select(.access_level >= 40) | .username'
sudo find / -name 'gitlab-secrets.json*' -printf '%m %u %p\n' 2>/dev/null # on the host
# TeamCity, in ~/.BuildServer/config by default. List these files, and never print them.
ls -l database.properties encryption-config.xml projects/*/pluginData/ssh_keys/*
| Check | What the result means |
|---|---|
| GitHub | Each login printed can read every repository secret from any branch, since a run uses “the version of the workflow” in its commit. No CODEOWNERS line for workflows, and no broader one such as /.github/, means that no owner is asked to review workflow edits, and a false together with an empty rule list means that nothing protects main at all. Empty protection_rules and a null policy gate nothing, and secrets: inherit passes all. |
| CodeBuild | Each line, such as nightly-export DB_PASSWORD, is a value that anyone with codebuild:BatchGetProjects can read in plain text. Move it to SECRETS_MANAGER and rotate it, since it was never hidden. |
| Kubernetes | No flag, or identity first, means “you do not have encryption at rest enabled”. An encrypting provider “does not tell you” if older Secrets were migrated, so see Kubernetes under Replace it everywhere else. |
| Jenkins | GLOBAL reaches every job, and SYSTEM stays with Jenkins. Item.Configure names who can change what a build runs, and numExecutors above 0 gives builds “the same level of access to the controller file system”. |
| GitLab | false after the name means that unprotected branches get it too, and each username can read or replace it. A secrets file beside a backup “defeats the purpose of using encryption in the first place”. |
| TeamCity | No encryption-config.xml and no TEAMCITY_ENCRYPTION_KEYS means the default key. That file holds the keys themselves, and JetBrains says to move database.properties credentials to environment variables. |
A false assumption looks like a finished job, so it is rarely checked again.
The other eleven
The other eleven beliefs follow the same pattern, each true of one layer and false as a conclusion. For the third, every person, fork and token on a private repository can hold a copy that keeps working, and one stolen laptop or token reads everything its owner can read. The third belief has numbers behind it: GitGuardian’s 2026 report found at least one secret in 32.2 percent of internal repositories, against 5.6 percent of public ones. Private also costs you help: a GitHub token pushed to a public repository is revoked automatically, while a match in a private repository goes to your admins and the committer, and the revocation action there is still a public preview as of November 2026. For the fifth, AWS notes that temporary credentials “cannot be revoked”, and for the seventh, Kubernetes states that “Base64 encoding is not an encryption method, it provides no additional confidentiality over plain text.”
| No. | What people say | What is true instead |
|---|---|---|
| 01 | “It is in the CI/CD secret store, so it is safe.” | Anyone who can edit a pipeline can read the secrets it uses. |
| 02 | “It is masked in the logs, so nobody can see it.” | Masking applies only to log text. |
| 03 | “The repository is private, so the key is private.” | Everyone with access to the repository can read the key. |
| 04 | “We deleted it in the next commit.” | The old commit holds it, and the key still works. |
| 05 | “We rotated it, so we are done.” | Sessions and forged tokens outlive the old key. |
| 06 | “.gitignore keeps it out of Git.” |
A rule does nothing for a file Git already tracks. |
| 07 | “It is encoded, so it is protected.” | Base64 is a format, and reversing it needs no key. |
| 08 | “Environment variables are the secure way.” | Better than source, and still readable at run time. |
| 09 | “The scanner came back clean.” | It means only that no rule matched what it could read. |
| 10 | “It is only a test key.” | Its permissions decide what it can reach, whatever its name. |
| 11 | “It is in the vault, so the problem is solved.” | Then the question is who can read the vault. |
| 12 | “Terraform marks it sensitive.” | That redacts output. State keeps the plain value. |
Note. If a secret has just leaked, 01, 04 and 05 are the three that change what you do in the first hour, and What Survives a Rotation, further down, expands 05.
The second belief, masking, follows directly below. Elsewhere, Real or not takes up the third, seventh and tenth, while the fourth runs through In history, not in the tree and Clean up Git history, and the fifth through What survives a rotation and Rotate and revoke, credential by credential. The sixth belongs to What .gitignore does not cover, the eighth and eleventh to Replace it with something better, the ninth to What a scanner cannot see and the twelfth to Terraform and OpenTofu.
Masking hides log text and nothing more
Masking is a filter on log text, applied after the fact. It never governs what a job can do with a value, and the vendors themselves say that it can fail. Generally, a runner swaps any value it knows for a placeholder such as *** in each line of output, so masking acts on the log alone. The common warning that Base64 gets past GitHub’s masking is wrong as a blanket claim, though, since actions/runner v2.338.2 registers Base64 forms of each secret.
Other systems document their own limits. CodeBuild says that “Masking matches the exact value stored”, so a Base64 form is not masked, GitLab masks only single line values of 8 characters or more, and Azure Pipelines says, “We never mask substrings of secrets.” The vendors also disown masking, CircleCI saying that it “can be bypassed in many ways, either accidentally or maliciously”, Jenkins that it “could of course be trivially circumvented”, and GitHub that its own redaction is not assured, because a secret value can be transformed in multiple ways. A masked log may therefore still hold the value in a form that the mask does not match.
Structure works against masking too. GitHub’s advice is “do not use a blob of JSON, XML, or YAML” around a secret, because redaction relies on an exact match, so give each sensitive value a secret of its own.
# Before: one blob, registered only as one exact string
env:
BILLING_CONFIG: ${{ secrets.BILLING_CONFIG }}
# After: one secret per sensitive value, each registered on its own
env:
SQL_PASSWORD: ${{ secrets.SQL_PASSWORD }}
FORECAST_API_KEY: ${{ secrets.FORECAST_API_KEY }}
Going forward, keep debug tracing such as GitLab’s CI_DEBUG_TRACE away from jobs that hold secrets. A value derived from a secret, such as a signed token, “should be formally registered as a secret” too, as GitHub puts it. If a value does reach a log, GitHub’s advice is that “you should delete the log and rotate the secret”, because anyone who read it still has it. Rotate it first, then scan the run’s log, as TruffleHog in Finding secrets shows, and delete the logs through GitHub’s workflow runs API.
# Delete one run's logs, once you have scanned them
gh api --method DELETE \
repos/example-systems/billing-api/actions/runs/RUN_ID/logs
Where secrets hide
A secret rarely stays in the file where it started. It moves into history, builds, laptops and the tools around the code, and each of those places keeps a copy of its own, so search where the value has been, since the tree shows only today. CI logs belong to Masking hides log text and nothing more, and pull request comments to The first hour.
In the code you can see
Start with the files that developers edit every day, since a value typed into one of them reaches every clone on the next push. A file name proves nothing, since alert 17 turns on a production key in appsettings., and infrastructure as code is no safer, as one study counted 7,849 hard coded passwords across 50,323 Ansible and Chef scripts. In review, stop on the four shapes below, from settings copied in “for now” to keys in a Terraform provider block.
# Four shapes to stop on in review
# src/Api/appsettings.json, production settings copied in "for now"
"Billing": "Server=tcp:sql-example-prod.database.windows.net;User ID=app;Password=EXAMPLE"
# infra/providers.tf, credentials typed into the provider block
provider "aws" {
access_key = "AKIAIOSFODNN7EXAMPLE" # AWS's public sample value
secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
# tests/StorageFixture.cs, a fixture that points at a real account
const string Conn = "AccountName=stexample;AccountKey=EXAMPLE00000000000000000000==;";
# src/Mail/InvoiceMailer.cs, a comment left after a quick fix
// old SendGrid key, keep until the new one is proven: SG.EXAMPLE00000000.EXAMPLE00000000
What .gitignore does not cover
A .gitignore rule has no effect on a file that is already tracked, on a forced add, or on any tool that is not Git. Git’s documentation says it in one line, “Files already tracked by Git are not affected”, so a .env committed once stays tracked after you add the rule, and every later change goes in with it. A rule is also narrower than it looks, since .env does not match .env.production, and git add --force walks straight past it. The rule stops at Git too, since docker build reads .dockerignore, so COPY . . can carry an ignored file into an image. Below, check-ignore says nothing about a tracked file, and ls-files tells the truth.
# Shell, sample output. The rule exists, and check-ignore says nothing
$ cat .gitignore
.env
$ git check-ignore -v .env
$ git ls-files -ci --exclude-standard
.env
$ git log --oneline -- .env
5af250b Update local config
77e1228 Add local config
Stop tracking the file, then treat every value it ever held as leaked, since this changes only the tree and the history still holds each value. Going forward, write the rule as .env* with an exception for .env.example, commit that example with names and no values, and add **/.env* to .dockerignore. Fail CI when the ls-files check prints anything, since the rule filters only what Git has not yet tracked.
# Stop tracking .env, then a CI line that fails on any ignored file still tracked
git rm --cached .env # the file stays on disk
git commit -m "Stop tracking .env"
test -z "$(git ls-files -ci --exclude-standard)" # CI: fail on a match
# .gitignore, a block to copy
# No effect on a tracked file. Run git rm --cached on it, and if it was ever committed, rotate.
.env
.env.*
!.env.example
appsettings.Development.json
*.pem
*.key
*.tfstate
*.tfstate.*
*.tfvars
gitleaks-report.json
findings.json
In history, not in the tree
In git log, -S finds commits that change how many times a string appears, which suits a string that you already know, and -G finds commits whose changed lines match a regular expression, which suits a shape. With --all, the search covers every ref, and the oldest line of the output is when the exposure began. Our rule is to search on something that is not secret, such as a key ID, a setting name or a token shape, and never on the value or any part of it. Searching this way keeps the secret half out of your shell history, and the key ID below is AWS’s documented sample.
# Shell, sample output. The tip is clean, and history is not
$ grep -rn 'AKIA' config/
$ git log --all -m --format='%h %s' -S 'AKIAIOSFODNN7EXAMPLE' | uniq
bd132c9 Remove key from export config
aa8a87c Add export job config
$ git grep -c 'AKIA' aa8a87c -- config/ # a count, so no value prints
aa8a87c:config/export.env:1
A stash, which lives only in your clone, is a merge commit under refs/stash, so a plain -S or -G sees only its staged part and skips any value that first appeared in a merge commit, since a merge only matches with -m. That flag prints a line per parent, which is why the search above ends in uniq. To list a stash’s files by name, run git stash show --name-only. Deleting a branch deletes its own reflog too, but the HEAD reflog still lists any of its commits that were checked out in this clone, and since they are not reachable from the commit that HEAD now points to, those entries last 30 days by default, under gc.reflogExpireUnreachable. Step 5 under A look without a scanner finds them with git fsck. Once the branch has no ref, --all misses the commits that only it reached, while --reflog still finds them, which is why step 4 of that look adds --reflog to --all. On the server, a normal clone skips refs/pull/, which git clone --mirror fetches, and neither fetches a commit that a force push left “dangling”, although it may stay reachable by its SHA, so no scan of a clone can see it. TruffleHog’s github-experimental source, alpha as of November 2026, tries to reach such commits.
Run every history scan on a full clone, since a shallow one holds only recent commits, and add --text for binary files, which -G otherwise skips. We ran the commands below on Git 2.54, in bash and zsh, against a repository of sample values, and the output is from that run.
# Shell, in the clone you work in, with the output of our run
$ git log --all --date=short --format='%h %ad %s' -G'ghp_[A-Za-z0-9]{36}'
e613ce3 2026-03-09 Remove .env
e47084a 2026-03-09 Add .env for local runs
$ git show e47084a:.env | grep -oE '^[A-Z_][A-Z0-9_]*=' # the names it held, and no value
API_TOKEN=
In what you build
A repository scanner never sees a build, so a value that left the source can still ship. One study found secrets in 8.5 percent of 337,171 container images, and Flare found 10,456 images with exposed keys among those uploaded to Docker Hub in November 2025, about 42 percent of them with five or more. Flare says that a .env file often comes along by accident when a project folder is copied in. In 2024, JFrog found a PyPI admin’s GitHub token, gone from source and still alive, in a compiled .pyc file inside a public Docker image.
Each build step writes a layer, and the image keeps them all, so build arguments and environment variables persist, and a file that a later layer deletes is still in the earlier one. A front end build is a publication, since NEXT_PUBLIC_, VITE_ and REACT_APP_ values all reach the browser, and a source map may carry the original files, comments included. As of November 2026, Vite and Next.js leave production source maps off by default, so a .map file in their output suggests that someone turned them on. Mobile apps reach further, and Symantec found 1,859 public apps with hard coded AWS credentials.
# Shell, sample output shortened. Your own terminal only, since the TruffleHog lines print values
# Image: build arguments and ENV values stay in its history, so print the names, not values
$ docker history --no-trunc --format '{{.CreatedBy}}' example/billing-web:1.4.2 \
| grep -oE '[A-Z_]*(TOKEN|KEY|SECRET|PASS)[A-Z_]*=|^COPY [^ ]+'
NPM_TOKEN=
COPY .npmrc
API_KEY=
# The RUN line with NPM_TOKEN also ran rm /root/.npmrc, but the COPY layer keeps the file
$ docker save example/billing-web:1.4.2 -o image.tar
$ trufflehog docker --image file://image.tar --results=verified,unknown --no-update
# Front end: the bundler swaps a name for its value, so search for the shape of the value
$ grep -rlE 'sk_live_|AKIA[A-Z2-7]{16}|BEGIN [A-Z ]*PRIVATE KEY' dist/
# Mobile: an APK is a ZIP archive, and an iOS .app bundle is already a directory
$ unzip -q app.apk -d apk/ && trufflehog filesystem apk/ --results=verified,unknown --no-update
In our example, the rm did nothing, since the token sits in that same history line, and the file sits in the COPY layer below. Rebuilding changes nothing for anyone who has pulled the tag, so revoke first, then rebuild with a build secret, as Docker build secrets shows.
On the machines around the code
Laptops and build servers hold credentials in files that tools create on their own, in home folders that no repository scanner reads. These files get committed by accident, and malware looks for them, since Trend Micro’s May 2026 report on QLNX names seven of the eight below as targets. In 2018, an attacker took over an ESLint maintainer’s npm account and published eslint-scope 3.7.2, which sent each user’s .npmrc to the attacker, and npm revoked every token issued before that day’s cutoff. A 600 file stops other accounts, but a package that you install runs as you and reads whatever you can read.
| File | What it holds | How it sits on disk |
|---|---|---|
.npmrc |
Registry tokens in _authToken or _auth |
Plain text, unless it uses ${NPM_TOKEN} |
.pypirc |
PyPI upload tokens | Plain text, and the specification says so |
~/.aws/credentials |
Access key pairs, one set per profile | Plain text, and AWS says so |
~/.kube/config |
token, password, client-key-data |
Tokens in plain text, client keys as Base64 |
~/.docker/config.json |
Registry logins under auths |
Base64, unless you set a credential store |
terraform.tfstate |
Resource attributes, secrets included | Plain text, even if marked sensitive. See Terraform and OpenTofu |
~/.git-credentials |
Lines of https:// |
Plain text, and Git warns about it |
~/.zsh_history |
Whatever was typed as an argument | Plain text, and listed by MITRE as T1552.003 |
In Sysdig’s 2023 report on SCARLETEEL, a plain text IAM key for a second AWS account sat in a Terraform state file in S3. The attacker got no further, because “the IAM user was very well scoped, so all of the requests failed due to a lack of permissions.” You may not always stop a file from being read, but you can decide in advance what its secrets are able to do. For safer forms, a project can commit an .npmrc line that reads ${NPM_TOKEN}, which npm fills in from the environment, although an undefined variable stays as literal text, and you can use a Docker credential helper for ~/.docker/config.json.
Shell history, which you feed yourself
Both bash and zsh save each command line as plain text, so a token passed as an argument stays on disk after the terminal closes. A leading space keeps a line out only if HISTCONTROL includes ignorespace or ignoreboth in bash, or setopt HIST_IGNORE_SPACE is set in zsh, and in our test on bash 3.2 and zsh 5.9 on macOS, neither was on in a plain shell, so a habit that works on one machine may fail silently on the next. Terminal there also keeps per session copies under ~/.zsh_sessions, so search those too, and let read or pg_dump prompt for a value instead.
# Shell, tested in bash and zsh, sample output
# Before: both values are now in your history file
$ export STRIPE_KEY=sk_live_EXAMPLE000000000000000000
$ pg_dump "postgres://app:EXAMPLE-Password@db.example.com/billing" > dump.sql
# After: read asks with no echo, and pg_dump prompts for the password itself
$ read -rs STRIPE_KEY && export STRIPE_KEY
$ pg_dump -h db.example.com -U app billing > dump.sql
# Find: lines that hold a token shape, counted per file, so that no value prints
$ grep -cE 'sk_live_|ghp_|AKIA[A-Z2-7]{16}' ~/.zsh_history ~/.bash_history 2>/dev/null
/Users/dev/.zsh_history:3
# Clean, zsh: from a new terminal, after closing the one that holds the line. LC_ALL=C keeps
# sed on a Mac from stopping at bytes that zsh writes for some characters. Then rerun Find
$ LC_ALL=C sed -i.bak -E '/sk_live_|ghp_/d' ~/.zsh_history && rm ~/.zsh_history.bak
# Clean, bash: in the same session, write the list out, clean it, then empty and reload it
$ history -w && LC_ALL=C sed -i.bak -E '/sk_live_|ghp_/d' ~/.bash_history
$ rm ~/.bash_history.bak && history -c && history -r
To clean up, revoke first, since a value that reached this file may also be in your backups. Then close the terminal that typed the value and clean up from a new one, since in our test a zsh session wrote a line back when it closed, even after we had removed it. In bash, empty the list in memory too, since bash writes all of it back on exit. The patterns above are token shapes, so Find misses the password in the pg_dump URL and Clean leaves it in place, along with any AKIA line that Find counted. For each kind of value that you typed, add a shape to both, such as postgres: for that URL and never the value itself, then rerun Find until every count is 0. In a shell set to write its history as you go, Find also counts its own lines, since they name the shapes but hold no value.
Next to the code
GitGuardian’s 2026 report says that about 28 percent of incidents originate entirely outside repositories, in places like Slack, Jira and Confluence. A value usually leaves the code for a good reason, such as debugging a job, and where it lands generally has no scanner, no owner and no expiry. CloudSEK found over 30,000 public Postman workspaces disclosing tokens and API keys in December 2024, so check which of yours are public and scan them with TruffleHog’s postman source. In 2023, an attacker searched Cloudflare’s wiki for “secret” and “token”, among other words, and viewed 202 pages, so run the same searches and replace each value with a pointer to the vault. For chat, which MITRE tracks as T1552.008, search for your token prefixes and treat each hit as leaked.
Notebooks keep what they printed
A Jupyter notebook stores each code cell’s source and outputs as separate fields, so the text that a cell printed stays in the saved file until someone clears it or runs the cell again. A cell that now reads from the environment can still hold a key in its old output. Wiz found Langsmith API keys in .ipynb files in its 2025 review of 50 AI companies, although we know of no study of output cells in particular.
// notebooks/reconcile.ipynb, one saved cell: clean source, and the old output
{ "cell_type": "code", "execution_count": 7,
"source": ["conn = os.environ[\"BILLING_CONN\"]\n", "client = from_connection_string(conn)"],
"outputs": [{ "output_type": "stream", "name": "stdout",
"text": ["AccountName=stexample;AccountKey=EXAMPLE00000000000000000000==;\n"] }] }
# Shell, find and CI lines run on a sample notebook
# Find: saved outputs per notebook that match SHAPES from A look without a scanner, in Finding secrets, as a count
$ git ls-files '*.ipynb' | while read -r f; do
printf '%s %s\n' "$(jq -c '.cells[].outputs[]?' "$f" | grep -cE "$SHAPES")" "$f"
done | grep -v '^0 '
1 notebooks/reconcile.ipynb
# Clear the file on disk, or strip outputs from every later commit with a Git filter
$ jupyter nbconvert --clear-output notebooks/reconcile.ipynb
$ nbstripout --install && git add --renormalize notebooks/
# In CI: fail the build if a committed notebook holds an output
$ git ls-files '*.ipynb' | xargs -I{} jq -e '[.cells[].outputs | length] | add // 0 | . == 0' {}
Scanners read the file as text, so a shape in an output will usually surface, although an image output is Base64 under image/png. Clearing an output with nbconvert fixes the file but leaves the leak in place, since the old commit still holds the value, so revoke first. A filter such as nbstripout lives in .git/config, and Git treats a missing filter as a silent pass. In our test, a fresh clone without the setup committed outputs with no warning, so a step that depends on each laptop needs a check on the server behind it.
MCP configuration is meant to be committed
An “MCP server”, from the Model Context Protocol, connects an AI coding assistant to a tool or a data source. Claude Code reads project servers from a .mcp.json file, and its documentation says to check that file into version control, so a token typed into it is a committed token. GitGuardian counted 24,008 unique secrets in MCP configuration files on public GitHub, 2,117 of them valid. The safe form holds only ${VAR} references, which Claude Code expands from the environment. We checked this for Claude Code as of November 2026, and other assistants keep their own files and rules.
// Unsafe: .mcp.json, committed with the values in it
{ "mcpServers": {
"db": { "command": "mcp-pg", "args": ["postgres://app:EXAMPLE@db.example.com/billing"] },
"api": { "type": "http", "url": "https://api.example.com/mcp",
"headers": { "Authorization": "Bearer EXAMPLE-Api-Token" } } } }
// Safe: .mcp.json, safe to commit, since each machine sets the values
{ "mcpServers": {
"db": { "command": "mcp-pg", "args": ["${BILLING_DB_URL}"] },
"api": { "type": "http", "url": "${API_BASE_URL:-https://api.example.com}/mcp",
"headers": { "Authorization": "Bearer ${API_KEY}" } } } }
The likely failure is an unset variable: the config still loads, claude mcp list may warn, and the server gets the literal ${API_KEY}, or in some remote headers an empty value. Removing a server does not revoke its tokens. As of November 2026, Claude Code keeps session transcripts in plain text under ~/.claude/projects/ for 30 days by default, so a pasted value is one more copy on disk.
A HAR file is a recording of your session
A “HAR file”, or HTTP Archive, is a JSON recording of what a browser sent and received, and support teams often ask for one. Each entry keeps the headers, the cookies, the query string and any posted data, so a capture taken while you are logged in holds the very values that keep you logged in. At Okta in 2023, HAR files in its support system held session tokens, files for 134 customers were accessed, and the attacker hijacked sessions at 5 of them.
As of November 2026, Chrome DevTools exports a “sanitized” HAR by default, which leaves out the Cookie, Set-Cookie and Authorization headers. In our test of those three rules on a sample capture, though, a token in a URL, a custom X-Api-Key header and a response body all survived. Cloudflare’s open source HAR Sanitizer strips session cookies and JSON Web Tokens, and it runs in your browser. If a capture with a live session has gone out, end that session at the issuer, since deleting the file does not take the session back from anyone who has already read it. The listing below sanitizes a capture and proves it with a count, and if you add one of your own tokens to that count, search on its name or shape, never on the value.
# Shell, output from a sample capture
# Sanitize: the three headers, any custom ones of yours, then query strings and bodies
$ jq '(.log.entries[].request, .log.entries[].response) |= (.cookies = [] | .headers |=
map(select(.name | test("^(authorization|cookie|set-cookie|x-api-key)$"; "i") | not)))
| .log.entries[].request |= (.url |= sub("\\?.*$"; "") | .queryString = [] | del(.postData))
| .log.entries[].response.content |= del(.text)' session.har > session.clean.har
$ grep -cE 'Bearer |session=' session.clean.har
0
Finding secrets
No single scanner sees everything, so a working setup stacks several of them at the points where a secret can enter the code, as Stacking the layers shows.
How a scanner decides
A scanner judges a string on four kinds of evidence. A “provider pattern” asks whether the string has the shape of one issuer’s credential, “entropy” asks how random it looks, context asks what sits around it, and “verification” asks the issuer whether it works today. Each provider pattern is a regular expression for one issuer’s format, often anchored on a prefix such as ghp_. A pattern decides on shape alone, so AKIA[0-9A-Z]{16} fires on the sample key in the AWS documentation just as it would on a production key. A match starts triage, then, without ending it. Entropy knows nothing of meaning: EXAMPLE2026 scores 3.10 bits per character and falls under the 3.5 threshold of the Gitleaks generic-api-key rule.
Where a format carries a checksum, a scanner can reject a fake offline, and in GitHub’s words, “A checksum virtually eliminates false positives for secret scanning offline.” It proves a well formed string, though, and never a live token. Entropy misleads both ways, since the public Git commit 83d9cd68... scores 3.69 bits per character by our count, over the same threshold. The author of Gitleaks says himself that entropy alone is not a very good heuristic, so treat a high score as a reason to look, and do not clear a string because its score is low.
Context is what sits near the string, such as a keyword, a variable name or the other half of a pair. Context cuts noise and creates misses, and GitHub says that “Pairs pushed to different files, or not pushed to the same repository, will not result in alerts.” The README of detect-secrets admits as much, since it will not catch login = "hunter2". Verification is the only signal that leaves the repository, because the tool calls the issuer. A yes is strong evidence, while a no may mean a failed call, a key limited to certain addresses or an issuer with no check. Treat a no, then, as something that you still have to explain. Beneath all four, every scanner reads text, so a secret that is broken up, wrapped, shaped like ordinary text or compiled into a binary may pass in silence.
In our example, GitHub’s azure_storage_account_key pattern fires on alert 17’s connection string, which settles the shape and nothing else, since a sample config would fire too. Random 88 character base64 values that we generated scored 5.29 to 5.54 bits, past every threshold, as a hash would. TruffleHog wants an account name near the key, and here it has one, but only a person knows that stexamplebilling holds production invoices. The fourth signal is where triage takes over, under Five questions.
What a scanner cannot see
Split, encoded, generic and binary values all turn up on their own as a codebase ages, and none of them shows in a clean report. A regular expression needs one contiguous match, so the pattern below finds no match in AWS’s public sample key once it is split in two, and we found no claim from GitHub, TruffleHog or Gitleaks that they rejoin strings. A value under stripe-key decodes to the shape of a Stripe key, and printing only its prefix keeps a live one off your screen.
# Split and encoded values, which a pattern reads straight past
echo '"AKIAIOSFODNN7" + "EXAMPLE"' | grep -cE 'AKIA[0-9A-Z]{16}'
# prints 0
printf 'c2tfbGl2ZV9FWEFNUExF' | base64 -d | cut -c1-8
# prints sk_live_ and no more
As of November 7, 2026, GitHub decodes Base64 only per pattern, on 21 of its 531 provider rows, which cover Azure Storage but not Stripe, npm or Slack, and never for generic patterns. TruffleHog decodes by default, and so does Gitleaks in its v8.30.1 source, although its README says otherwise. A generic password such as EXAMPLE2026 has no prefix and no issuer to call, and GitHub’s AI-detected secrets “may miss instances of credentials” and skip test code, such as tests/test_db.py. TruffleHog reads binaries by default, Gitleaks skips them and opens archives only when given --max-archive-depth, and GitHub’s docs do not say, which matters for a compiled file such as the .pyc in In what you build.
Since November 5, 2026, GitHub may suppress an AWS match with no paired secret, and the default allowlist in Gitleaks drops values shaped like ${VAR}, along with lock files, Go and Ruby vendor folders, Office files and PDFs. Force-pushed commits are a gap of their own, covered in In history, not in the tree. Generally, a rule list says what a tool was written to find, and nothing about the rest.
A clean result says nothing about what the tool skipped, such as forks and dangling commits, and an inactive or unverified result does not mean dead. Of the five sample lines below, TruffleHog would flag one, Gitleaks and GitHub three each, and all three miss line 4.
# Five sample lines. Read each as a real value of its shape
1 STRIPE_TEST=sk_test_EXAMPLE00000000000000000000
2 STORAGE_CONN=AccountName=stexamplebilling;AccountKey=EXAMPLE...88 characters...==
3 AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE # AWS's public sample, secret in another file
4 DB_PASSWORD=EXAMPLE2026
5 AZDO_PAT=EXAMPLE...84 characters... # the current Azure DevOps format
| Sample line | What fires | What stays silent |
|---|---|---|
| 1. Stripe test key | Gitleaks: Fires. stripe-access-token allows test. GitHub: Alerts, as stripe_test_secret_key. |
TruffleHog: Silent. Its regex is [rs]k_live_ alone. |
| 2. Storage connection string | Gitleaks: No Azure rule. generic-api-key should fire on a real key. TruffleHog: Fires. AzureStorage pairs name and key, then calls Azure. GitHub: Alerts, as azure_storage_account_key. |
None of the three |
| 3. AWS key ID, secret in another file | Gitleaks: A real ID fires. One ending in EXAMPLE is allowlisted. |
TruffleHog: Silent. It needs a secret in the same chunk of text. GitHub: Silent. Pairs in different files “will not result in alerts”. |
| 4. Database password, no prefix | None of the three | Gitleaks: Silent. It scores 3.10, and the generic rule needs over 3.5. TruffleHog: Silent. No built in detector for a bare password. GitHub: Silent on patterns. AI password detection is a separate setting. |
| 5. Azure DevOps PAT, current format | GitHub: Alerts, as azure_devops_personal_access_token. |
Gitleaks: Silent. No rule of its own, and PAT is no keyword, as TOKEN is. TruffleHog: Silent. Its regex is the old 52 character form. |
Note. Read against the Gitleaks v8.30.1 default configuration, TruffleHog’s Stripe, Azure Storage, AWS and Azure DevOps detectors, and GitHub’s supported patterns, scanning scope and AI features for secret scanning.
Only GitHub alerts on line 5, the current 84 character Azure DevOps PAT, since the TruffleHog v3.99.8 regex is the old 52 character form and Gitleaks v8.30.1 has no rule of its own. Record what each clean result covered: the tool, its version, its rules and the sources it read. Stack tools as well, since several skip PDFs, images and lock files, and when you need every lead, keep every state.
# Your own terminal only, since plain output prints the values
trufflehog git file://. --results=verified,unknown,unverified --no-update
GitHub secret scanning
The earliest layer that cannot be skipped by accident is push protection, which stops known token formats on the server before they land, but it checks only new pushes, which is why it never saw the commit behind alert 17. The names have moved: GitHub Advanced Security was one bundle until April 1, 2025, when GitHub split it into GitHub Secret Protection and GitHub Code Security. The old name survives as the family name. An “active committer” pushed to a covered repository in the last 90 days, and GitHub App bots do not count. As of November 2026, Secret Protection lists at $19 per active committer per month, on a Team or Enterprise plan. With 12 of its 31 people pushing in the last 90 days, billing-api costs $228 a month. Scanning and push protection are free on public repositories, and custom patterns, AI-detected secrets and delegated bypass are not.
In our example, Secret Protection and push protection went on for billing-api on Monday evening, when commit 7f3c2ab was already 10 days old, and the first scan of history raised alert 17 at 09:14 on Tuesday. Push protection went on after the commit was pushed, so only a scan of history could find it. The scan reads your “entire Git history on all branches”, along with issues, pull requests, Discussions, wikis and secret gists, although, as of November 2026, Actions logs are covered only in an Enterprise Cloud preview. It rescans when GitHub adds a secret type, so a commit that was clean last year may alert today.
A setting applied one repository at a time drifts, so an organization should use a “security configuration”, a named bundle that you attach to a scope and set as the default for new repositories. Attaching starts the billing, although the Assessments banner shows an estimated cost first, and if licenses run short, “only free features will be enabled” on the remainder. So we attach to a pilot with selected, using its ID from the repositories API, read the numbers, widen on purpose and check what each repository received through the configurations API, and leave validity checks for later.
# Secret Protection for the organization, run as an organization owner
# One repository: Settings, Advanced Security, Secret Protection, then Push protection.
# An organization: create the bundle once, pilot one repository, widen, cover what comes next.
# Validity checks are a later decision, since they send each key to its issuer.
O=orgs/example-systems/code-security/configurations
ID=$(gh api --method POST $O --jq .id -f name=secrets-baseline -f secret_protection=enabled \
-f secret_scanning=enabled -f secret_scanning_push_protection=enabled \
-f secret_scanning_non_provider_patterns=enabled)
RID=$(gh api repos/example-systems/billing-api --jq .id)
echo "{\"scope\":\"selected\",\"selected_repository_ids\":[$RID]}" \
| gh api --method POST $O/$ID/attach --input -
gh api --method POST $O/$ID/attach -f scope=all_without_configurations
gh api --method PUT $O/$ID/defaults -f default_for_new_repos=all
gh api "$O/$ID/repositories" --jq '.[] | [.repository.name, .status] | @tsv' # any failed?
# Later: gh api --method PATCH $O/$ID -f secret_scanning_validity_checks=enabled
For public repositories and public npm packages, a partner alert goes to the provider and never to you, and the provider “decides whether they should revoke the secret”, so the key may or may not be dead already, and your first sign of a leak may be an outage. A public repository made private without a license loses all of this coverage at that moment. Blocked pushes and bypasses come next, in When push protection blocks a push.
A look without a scanner
You do not need a scanner to learn whether you have a problem. These commands send no secret to any provider, and steps 2 to 4 print only locations, so that your terminal does not become another copy. Run them in a fresh clone first, then in your own. The fresh clone shows what the server holds, apart from the pull request refs and dangling commits in In history, not in the tree, and your own may add what steps 4 and 5 find only there.
# ten-minute-look.sh, which needs Git, grep and the GitHub CLI
SHAPES='A[KS]IA[A-Z2-7]{16}|gh[pour]_[A-Za-z0-9]{36}|ghs_[A-Za-z0-9._-]{36,}'
SHAPES="$SHAPES|github_pat_[A-Za-z0-9_]{20}|glpat-|xox[abeprs]-|[rs]k_live_|npm_[A-Za-z0-9]{36}"
SHAPES="$SHAPES|AIza[0-9A-Za-z_-]{35}|AccountKey=[^;\"]{20}|[78]Q~[0-9A-Za-z_~.-]{31,34}"
SHAPES="$SHAPES|[?&]sig=[0-9A-Za-z%+/=]{20}|BEGIN [A-Z ]*PRIVATE KEY|://[^/:@ ]+:[^/@ ]+@"
NAMES='(^|/)(\.env(\..+)?|.*\.(pem|pfx|p12|key|tfstate.*|tfvars)|id_(rsa|ed25519)|\.npmrc)$'
# 1. Files that should not be tracked, anywhere in history, deleted ones included
git log --all -m --format= --name-only | sort -u | grep -E "$NAMES"
# 2. Token shapes in the tree today: file and line only
git grep -nIE "$SHAPES" | cut -d: -f1,2
# 3. Generic names in the tree today: a noisy count per file, highest first
git grep -cIiE "(password|pwd|secret|api_?key|token)[\"']? *[:=]" | sort -t: -k2 -rn | head
# 4. Token shapes in every commit on every ref, oldest first, merges included
git log --all --reflog -m --reverse --date=short --format='%h %ad %s' -G"$SHAPES" | uniq
# 5. What only this clone holds: stashes and orphaned commits
git stash list; git fsck --unreachable --no-reflogs 2>/dev/null | grep ' commit '
# 6. Which stored secrets the workflows use. gh prints names and dates, never values
grep -rn 'secrets\.' .github/workflows/; gh secret list --repo example-systems/billing-api
# Sample output, a first run on a test repository, with step labels added
.env # step 1: a file that should not be tracked
infra/providers.tf:2 # step 2: token shapes in the tree today
tests/StorageFixture.cs:1
config/dev.yml:2 # step 3: a noisy count, so open the file
7f1f550 2026-03-09 Add .env for local runs # step 4: the oldest line opens the window
6e90c2c 2026-03-09 Remove .env
bb84c27 2026-03-16 Add storage tests and provider
8bd51ce 2026-03-18 Spike S3 export # also unreachable in step 5
stash@{0}: On main: try the real password # step 5: git stash show --name-only
unreachable commit 8bd51cea4f379c324408d38879c0d16555e77a3a
In the sample, the token in .env opened its window on March 9. Commit 8bd51ce appears in step 4 and again as unreachable in step 5, which marks a deleted branch. A clean run means less than a hit, because these shapes miss a password with no prefix, a bare AWS secret key and anything encoded. Treat every hit as a lead for Real or not, and install a dedicated scanner either way. At Example Systems, steps 2 and 4 found the key on main, from commit 7f3c2ab, 11 days old, and merge 5e91d0c, 9 days old, so the window opens 11 days back. Step 6, through gh secret list, led to the Actions secret STORAGE_CONN, which an App Service app setting duplicates, so two consumers must keep working through the fix.
TruffleHog
TruffleHog, an open source scanner, is at v3.99.8 as of November 2026, after eight releases in four weeks, so pin a version and expect to move the pin. We reach for it first, since it reads far more than Git and can tell you which findings still work. Results come in three states: verified, which the provider accepted from where you ran the scan; unknown, where the call failed and the detector recorded why, which counts as live; and unverified, which is still a lead to work. Verification sends a live request, which may appear in the issuer’s logs. A plain run prints the value on its Raw result line, so a job in CI should never use plain output, while the official action’s output leaves it out.
# Your own terminal only, since plain output prints the value
# One branch against main. --since-commit alone still means every branch
# In v3.99.8 this filter drops a live Stripe key whose check fails, since its detector records no error
trufflehog git file://. --since-commit main --branch pdf-export \
--results=verified,unknown --fail --no-update
A release binary also updates itself. In the v3.99.8 source, unless given --no-update, it reports its version and platform to oss.trufflehog.org and may swap in a newer build, with no signature check, so a pinned version can replace itself at run time. We read this in the code and did not run it. The official action passes the flag for you and a plain binary does not, so treat the flag as a security control, in every script, hook and job. On a laptop, brew install trufflehog is fine, and for anything shared, check the release’s signed checksums with cosign first.
# Check a TruffleHog release before anything shared runs it
S=trufflehog_3.99.8_checksums.txt # with its .pem and .sig, from the release page
ID='https://github\.com/trufflesecurity/trufflehog/\.github/workflows/.+'
cosign verify-blob "$S" --certificate "$S.pem" --signature "$S.sig" \
--certificate-identity-regexp "$ID" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
sha256sum --ignore-missing -c "$S" # cosign prints Verified OK, and this prints OK
Each subcommand reads a different source, so the mode decides what a clean result means, and since flags move, check them against trufflehog --help on your pinned version. Organization mode reads every repository that GITHUB_TOKEN can see, without forks by default, and filesystem mode reads files with no history, such as a local .env. Docker mode reads every layer, so a file deleted in a later layer still shows, and a saved tarball needs no Docker socket. With no Actions log source, a finished run goes in through stdin.
# Your own terminal only, since plain output prints the value
trufflehog github --org=example-systems --results=verified,unknown \
--no-update
trufflehog filesystem .env bin/ --results=verified,unknown --no-update
trufflehog docker --image file://image.tar --results=verified,unknown \
--no-update
gh run view RUN_ID --log \
| trufflehog stdin --results=verified,unknown --no-update
| State | How it prints | What it means, and what you do |
|---|---|---|
verified |
Found verified result |
The provider accepted it, from where you ran the scan. Real and live. Score it. |
unknown |
Found unverified result, then Verification issue |
The call failed, perhaps on a network error, and the detector recorded why. Treat as live, and retry. A verified only filter drops it. |
unverified |
Found unverified result, alone |
Detected and not confirmed. A detector that records no error, such as Stripe, lands here when its call fails. A lead and not a false positive. |
In JSON, Verified is true for a verified result, and VerificationError has text for an unknown one. The live value sits in Raw, RawV2 and SecretParts, so a results file is itself a secret, and the filter below makes a copy for the ticket. Hostile content can also steer the verification calls, as CVE-2024-43379 showed before v3.81.9, so use --no-verification on code that you do not own.
# TruffleHog JSON results, with the three value fields taken out
jq -c 'del(.Raw, .RawV2, .SecretParts)' findings.json > findings.safe.jsonl # for the ticket
| Detector | What it checks | How a result reads |
|---|---|---|
| AWS | Calls GetCallerIdentity. |
The account field in ExtraData is decoded from the key ID before any call, so even an unverified result names the account, and success adds arn, which says whose key it is. A 403 reads as unverified, and other errors as unknown. Raw holds only the key ID, and the secret is in RawV2. |
| AzureStorage | Signs a request to list containers. | A 200 reads as verified, a 403 as unverified and any other status as unknown. An account that limits network access may also answer 403, so a live key can read as unverified. |
| Stripe | Calls /v1/charges. |
Both 200 and 403 read as verified, so a restricted key without that permission still counts. A failed call records no error, so an outage reads as unverified, and --results= drops the key. |
| Postgres | Connects to the host in the string. | A rejected password reads as unverified and a failed connection as unknown, so a private database, scanned from outside, is unknown at best. |
Note. Read in the v3.99.8 source of the AWS, AzureStorage, Stripe and Postgres detectors.
An unverified result covers a dead key, a key that your scanner could not reach and a key with no check at all, so we ask three things, in order. First, did a provider answer at all? Second, could it have answered from here, given private hosts and address limits? Third, does the issuer’s own record still show the credential as alive? If that last answer is yes, or the record cannot settle it, score it as unknown, which counts as live, and if the record shows it dead, it goes behind a baseline with an owner and a date.
Your own token formats need a TruffleHog custom detector, an alpha feature as of November 2026, built from a keyword and an RE2 expression. Without a verify block, every match reports as unverified, and no server sees the candidates. The run below keeps unverified results as a report, without their values.
# A report and not a gate. --no-ignore-tag also shows lines that someone has suppressed.
trufflehog git file://. --results=unverified --filter-entropy=3.0 --no-ignore-tag \
--no-update --json | jq -c 'del(.Raw, .RawV2, .SecretParts)' > unverified.safe.jsonl
Gitleaks, Betterleaks and detect-secrets
Gitleaks is in security patch mode as of November 2026. Its README says “Future releases will be security patches only”, the latest release is v8.30.1, from March 21, 2026, and its author has written, “Betterleaks is the successor to Gitleaks.” Keep Gitleaks where it works, and move new work to Betterleaks once v2 is stable, with a pin. Mind that -v means verbose in Gitleaks and --validate in Betterleaks v2, that Gitleaks deprecated detect and protect in v8.19.0 in favor of git, dir and stdin, and that Betterleaks is “drop-in” only for v1, since v2 changes the JSON and config lookup. Although a Gitleaks pre-commit hook helps, it cannot be enforced, because git commit --no-verify and SKIP=gitleaks both step around it. Hooks do not come with a clone either, so the enforced gate has to run on the server, in CI.
Gitleaks and detect-secrets earn their place as fast, local hooks, although some detect-secrets plugins call providers unless given --no-verify. The file below runs both, beside a local TruffleHog hook. The Gitleaks hook reads only what you staged, redacts what it prints and makes no provider calls, the detect-secrets hook checks staged files against its baseline, and TruffleHog calls providers from each laptop. Its hook also passes --trust-local-git-config, which skips the temporary clone that guards against a malicious Git config, CVE-2025-41390, so use that flag only on your own repositories.
# .pre-commit-config.yaml, a file to copy, with pins as of November 2026. Your own terminal only, since the TruffleHog hook's plain output prints the value
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: 83d9cd684c87d95d656c1458ef04895a7f1cbd8e # frozen: v8.30.1
hooks:
- id: gitleaks
- repo: https://github.com/Yelp/detect-secrets
rev: 01886c8a910c64595c47f186ca1ffc0b77fa5458 # frozen: v1.5.0
hooks:
- id: detect-secrets
args: [--baseline, .secrets.baseline]
- repo: local
hooks:
- id: trufflehog
name: TruffleHog
entry: >-
bash -c 'trufflehog git file://. --since-commit HEAD --results=verified,unknown
--fail --trust-local-git-config --no-update'
language: system
pass_filenames: false
stages: ["pre-commit"]
Gates, alerts and a first sweep
Finding secrets showed how each tool decides, and here we run those tools as gates and work through what they report.
When push protection blocks a push
Push protection runs on the server, so it cannot be skipped by accident, but on a private repository, repository push protection stays off until an administrator turns it on. Push protection for users, a separate default, covers public repositories only, and a bypass of it raises no alert unless the repository has push protection on as well. Once it is on, a match rejects the push like this.
# A rejected push, trimmed and illustrative. The wording varies over time
remote: error: GH013: Repository rule violations found for refs/heads/pdf-export.
remote: -- GitHub Personal Access Token ------------------
remote: - commit: 8728dbe67
remote: path: scripts/release.sh:12
remote: (?) To push, remove secret from commit(s) or follow this URL to allow the secret.
! [remote rejected] pdf-export -> pdf-export (push declined due to repository rule violations)
If the value is a working credential, take it out of every commit that holds it, and rotate it as well, since it has left the laptop. A new commit that deletes the line does not clear the block, because the old commit is still in the push. Amend the latest commit, or rebase from an earlier one’s parent with edit in place of pick, then search by the token’s shape, which keeps the value out of your shell history. Generally, this is the cheapest moment that you will ever have to deal with a leak.
# The repair, then the proof. We ran this flow on a scratch repository, and no output from the last line means no unpushed commit touches it
git commit --amend --all # latest commit
git rebase -i 8728dbe67~1 # earlier commit
git add scripts/release.sh && git commit --amend
git rebase --continue
git log --oneline -G 'ghp_' origin/main..HEAD
The block has gaps. Only 465 of 531 provider rows block, as of November 7, 2026, and a timed out push goes through, as does a secret that already has an alert. By default, anyone with write access may follow the URL, choose a reason and push again within three hours, and each bypass leaves an alert, an audit log entry and an email. A ruleset rule, “Require secret scanning alerts are resolved”, can also block a merge, although as of November 2026 it is in public preview and does not cover AI-detected secrets.
| Reason, in short | What GitHub records | REST value |
|---|---|---|
| Used in tests | A closed alert, resolved as used in tests | used_in_tests |
| A false positive | A closed alert, resolved as false positive | false_positive |
| Fix it later | An open alert | will_fix_later |
Two of these close the alert on the developer’s word, so a live key marked as a test value never reaches your open queue. “Delegated bypass” puts a named team in place of the click. On a repository, set “Who can bypass push protection for secret scanning” to Specific roles or teams under Settings, Advanced Security, and for an organization, set Bypass privileges to Specific actors in a security configuration. Requests then wait under Security and quality, Requests, Push protection bypass. Since an approval covers “any future commits that contain the same secret”, approve only a value shown to be harmless.
# Delegated bypass for the security team
TEAM=$(gh api orgs/example-systems/teams/security --jq .id) # O and ID as set for the security configuration in GitHub secret scanning
gh api --method PATCH $O/$ID --input - <<JSON
{"secret_scanning_delegated_bypass": "enabled",
"secret_scanning_delegated_bypass_options": {"reviewers": [
{"reviewer_id": $TEAM, "reviewer_type": "TEAM", "mode": "ALWAYS"}]}}
JSON
# ALWAYS may bypass and review. EXEMPT skips push protection, so keep it for bots, never people.
# Requests expire after 7 days, and owners and security managers can always bypass.
Validity checks, custom patterns and exclusions
GitHub’s “validity checks” send the secret to its issuer and mark the alert active, unknown or inactive, and Verify secret on the alert repeats it on demand. Treat inactive as a reason to hunt, since in GitHub’s words, “You should make sure no unauthorized access has already occurred.” On November 7, 2026, 165 of 531 provider rows had a check, Azure Storage among them, while Entra application secrets and PyPI tokens had none, and generic patterns and passwords never do.
“Custom patterns” cover the formats that only you issue, in a subset of Hyperscan regex with no option modifiers, up to 100 per repository and 500 per organization or enterprise. In our example, Example Systems issues tokens of exs_live_ and 32 letters and digits, and its docs use a placeholder of the same shape. Choose New pattern under Settings, Advanced Security, or for an organization under Advanced Security, Global settings, and fill the form as below, with More options open.
| Field | Value | What it does |
|---|---|---|
| Secret format | exs_live_[A-Za-z0-9]{32} |
The shape of the secret itself |
| Before, After secret | Left at the defaults | Refuse a match inside a longer word |
| Must not match | EXAMPLE |
Throws out the placeholder, and nothing else |
| Test string | token = "exs_live_x0x0..., at full length |
Confirms the form before a dry run |
A dry run shows what the pattern would find without raising one alert, across the repository, or all or up to 10 selected repositories in an organization, with a sample of up to 1,000 results. Only Publish pattern scans history and raises alerts, and only then can push protection be enabled, which protects developers from a first draft of a regular expression. In our example, 46 results over 10 repositories fall to 5 internal tokens once Must not match drops the 41 placeholders.
# Two dry runs, sample lines, where the third body is 34 characters long
# We ran the same logic in Python, which is not Hyperscan, so the dry run stays the real test
fires "exs_live_x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0"
dropped "exs_live_EXAMPLE0000000000000000000000000" Must not match
no match exs_live_x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0 After secret
no match myexs_live_x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0 Before secret
A file at .github/ excludes paths by name, in the filter syntax of Actions, for a case such as a directory of dummy keys. Since Record the decision asks every suppression for a reason and a reviewer, keep each entry narrow and put the reason on the line.
# .github/secret_scanning.yml
paths-ignore:
- "tests/fixtures/dummy-keys/**" # FP-231. Generated values, never issued. Reviewed by tkim.
An exclusion silences alerts in those paths, which close as “ignored by configuration”, and it silences push protection there, with no webhook. On our reading, anyone who can merge to the default branch can add one, so put the file under CODEOWNERS, as in Guardrails that stay on. The secret stays in Git and keeps working. Other scanners, on our reading of their docs, still read those paths, and GitHub scopes the file to alerts for users, so do not assume that it stops a partner alert. GitHub reads only the first 1,000 entries and ignores a file over 1 MB entirely. To test an exclusion, commit the custom pattern’s test string, once the pattern is published, under an excluded path and again under a scanned one, and confirm that only the second raises an alert or a block.
“Generic patterns”, once non-provider patterns, catch private keys, auth headers and connection strings, and “AI-detected secrets”, once Copilot secret scanning, look for passwords. Both fill a separate list, capped at 5,000 alerts per repository, and GitHub warns of “a higher rate of false positives”. As of November 2026, push protection for AI-detected secrets is a metered preview, and each scanned push can consume credits, even with no finding.
Working alerts from the command line
The alert page is fine for ten alerts and slow for four hundred, so learn the API early. Its hide_secret parameter defaults to false, so a plain list prints every live credential into your terminal and your logs unless you pass hide_secret=true. Every call below takes an administrator of the repository or organization, and a caller may resolve an alert as false_positive, wont_fix, revoked or used_in_tests. Among the flags, public leak means that GitHub has seen the value in public, and multi-repo means that it sits elsewhere too, so dedupe it as in One secret, forty alerts.
# What is live, active and unknown, then where alert 17 sits. Work the active ones first. Sample output
R=repos/example-systems/billing-api V="X-GitHub-Api-Version: 2026-03-10"
Q="state=open&validity=active,unknown&hide_secret=true&per_page=100"
gh api -H "$V" --paginate "$R/secret-scanning/alerts?$Q" \
--jq '.[] | [.number, .secret_type, .validity, .first_location_detected.path] | @tsv'
# 17 azure_storage_account_key unknown src/Billing.Api/appsettings.Development.json
gh api -H "$V" --paginate "$R/secret-scanning/alerts/17/locations" \
--jq '.[] | [.type, .details.commit_sha[0:7], .details.path, .details.start_line] | @tsv'
# commit 7f3c2ab src/Billing.Api/appsettings.Development.json 14
Then give each alert an owner and a ticket, here -f assignee=dreyes and SEC-1187, and close it only on the proof in The closure record. In our example, the weekly bypass list below, which, unlike the one in Guardrails that stay on, prints how each alert closed, shows a Slack token in scripts/release.sh closed as used_in_tests. A release script is not a test, and the status reads unknown, which counts as live, so the reviewer reopens it for triage.
# Each week, the bypasses and how each one closed. Sample output
gh api -H "$V" --paginate \
"orgs/example-systems/secret-scanning/alerts?hide_secret=true&per_page=100" \
--jq '.[] | select(.push_protection_bypassed)
| [.repository.name, .number, .resolution, .push_protection_bypassed_by.login] | @tsv'
# billing-api 21 used_in_tests dreyes
# Alert 21, reopened for triage
gh api -H "$V" --method PATCH "$R/secret-scanning/alerts/21" \
-f state=open -f resolution_comment='Reviewed: not a test value.'
A secret scan in CI
In CI, the line that most setups miss is the TruffleHog action’s version input. A SHA pin holds the action’s code, but the action runs a container tagged by that input, which defaults to latest, so set it to 3.99.8, with no v. As of November 2026, GitHub’s advice has moved toward a lockfile, .github/, which GitHub calls “the strongest protection available”. On our reading, it locks the commits of actions and says nothing of an image that an action pulls by tag at run time. The action adds --fail and --no-update. Before 3.98.0, a crafted filename could forge annotations, per GHSA-fg8w-m62h-jvvw, so trust the job’s result over its annotations. Keep untrusted text, such as a pull request title or a branch name, out of extra_args too, since the image passes its last argument through a shell.
# .github/workflows/secret-scan.yml, with references read on November 7, 2026
# Pull requests scan base to head, the schedule scans all of history. Make it a required check.
on:
pull_request:
schedule: [{ cron: "23 3 * * 1" }]
permissions: { contents: read }
jobs:
trufflehog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: { fetch-depth: 0, persist-credentials: false }
- uses: trufflesecurity/trufflehog@ea68ca03c865c5f324e86e849ced20ed3cf9dbbb # v3.99.8
with:
version: "3.99.8" # the image tag, with no v
extra_args: --results=verified,unknown --fail-on-scan-errors
Make it a required check, base to head on every pull request and all of history each week. A history scan needs a full clone, fetch-depth: 0 here and fetchDepth: 0 in Azure Pipelines, where some organizations default new pipelines to a depth of 1. Set persist-credentials: false too, since actions/checkout otherwise leaves its token where any later step can read it, in a clear text file under $RUNNER_TEMP, from v6.0.0 on, until the job’s cleanup. With --fail, TruffleHog exits 183 on a finding, so set -o pipefail wherever its output goes through jq, and Federation from CI shows a deploy that runs only after this scan.
For Gitleaks, the gate is the commented last line below, which scans only the range of the pull request and redacts its output. Exit 1 means leaks or an error, so set --exit-code to a value of your own, or a crash looks like a leak. As of November 2026, the official Gitleaks Action needs a license key for organizations and runs v8.24.3 unless told otherwise, so run the MIT binary in a plain step.
# .gitleaks.toml, with a custom rule, a reviewed allowlist and the CI gate at the end
[extend]
useDefault = true
[[rules]] # the internal token, unknown to the default rules
id = "exs-live-token"
regex = '''\bexs_live_[A-Za-z0-9]{32}\b'''
keywords = ["exs_live_"]
[[rules.allowlists]] # the Must not match row
regexes = ['''EXAMPLE''']
[[allowlists]]
description = "FP-233. Generated fixtures, never issued. Reviewed by tkim."
paths = ['''tests/fixtures/.*''']
# CI, with BASE and HEAD set to the base and head commits of the pull request:
# gitleaks git --redact --exit-code 42 --log-opts="$BASE..$HEAD" --report-path findings.json
What a reviewer checks
A reviewer sees a change after it has been pushed, so a catch in review is a late catch, and GitHub’s rule applies, since a committed secret should be considered compromised and amending the commit is not enough. Even so, review can see what a scanner cannot, such as a password with no prefix or a step with no reason to hold a key. Point to the file and line in a review comment, and never quote the value, since its edit history keeps a copy.
| In the diff | What to ask, and why |
|---|---|
appsettings.*.json, .env.example, *.tfvars, values.yaml |
Is this a name that points elsewhere, or the value itself? A file name proves nothing. |
gitleaks:allow, trufflehog:ignore, pragma: allowlist secret |
Who agreed, and why? These tags trust the author, and Record the decision asks for a reason. |
A new workflow step, or env: with secrets. |
Does this step need that secret at all? Whatever it runs can read it, as From the pipeline shows. |
Dockerfile, VITE_ and NEXT_PUBLIC_ names, source map settings |
What ends up in a layer or a bundle? Both are publications, as in In what you build. |
*.ipynb, .mcp.json and any *.har |
Is there a saved output, a literal after Bearer, or a capture at all? |
# pre-merge-look.sh, run on a pull request branch
# A first pass with SHAPES and NAMES from A look without a scanner, in Finding secrets. A zero means no known shape, not safe
BASE=origin/main # the branch that the pull request targets
git diff --name-only --diff-filter=ACMR "$BASE"...HEAD | grep -E "$NAMES" # risky names
git diff -U0 "$BASE"...HEAD | grep -E '^\+' | grep -cE "$SHAPES" # added shapes
git diff -U0 "$BASE"...HEAD | grep -E '^\+' | grep -cE 'gitleaks:allow|trufflehog:ignore'
Baselines for old repositories
An old repository needs a “baseline” before it gets a gate, since a first scan may return hundreds of findings, and a gate that is red from day one gets switched off. A baseline records them once, so that the gate reports only what is new. Treat it as a debt register, since every entry still has to be rotated or labeled. Gitleaks can baseline the history itself, while detect-secrets reads the current tree.
# Gitleaks: record the debt, then report only what is new. Use --redact on both, or on neither.
gitleaks git --redact --report-path .gitleaks-baseline.json
gitleaks git --redact --baseline-path .gitleaks-baseline.json --report-path findings.json
# detect-secrets v1.5.0: create it, label each entry, then gate every tracked file
detect-secrets scan > .secrets.baseline && detect-secrets audit .secrets.baseline
git ls-files -z | xargs -0 detect-secrets-hook --baseline .secrets.baseline
detect-secrets audit --report --only-real .secrets.baseline \
| jq -c '.results[] | {filename, types, lines: (.lines | keys)}' # what to rotate, no values
A Gitleaks baseline holds raw secrets unless it is made with --redact, which later runs then need too, or old findings return as new, as, on our reading, do rewritten commits and moved lines. A clean commit can fail the detect-secrets hook with exit 3, since the hook rewrote the baseline to keep line numbers current, and git add .secrets. fixes it. A rescan drops “any secrets no longer in your codebase”, even if never rotated, and the hashes are unsalted SHA-1, so anyone with the file can test a guess such as EXAMPLE2026. The audit report itself prints each raw value and its line, which is why the listing keeps only file names, types and line numbers.
In our example, Gitleaks finds 63 secrets in the old legacy-admin repository, and with a redacted baseline committed, the next pull request fails only on what it adds, while a ticket holds the 63 until 41 close as false positives and 22 are rotated. A green check over a baseline means only that the debt has not grown.
Azure DevOps and GitLab
Azure Repos has GitHub Secret Protection for Azure DevOps, at $19 per committer per month as of November 2026, on Azure DevOps Services only. Begin billing, under Project settings, Repos, Repositories, turns on push protection for new pushes, a scan of history and validity checks, whose alerts read Active or Unknown, and Microsoft advises treating Unknown as possibly active. Defender for Cloud displays findings without scanning for them, so it needs such a product underneath.
A bypass is only a commit message, skip-secret-scanning:true, although an alert is still raised, and for Azure Storage keys the product “only supports recently created tokens, not tokens that match the legacy patterns”, so an older key may pass in silence. Without a license, the job below is a gate at no cost, whose filter matters as much as its exit code, since plain output would print the value into the log.
# azure-pipelines.yml, a TruffleHog gate with no license
pool: { vmImage: ubuntu-latest }
steps:
- checkout: self
fetchDepth: 0 # some organizations default new pipelines to a depth of 1
- bash: |
set -o pipefail # keeps the scanner's exit 183 alive through the pipe
docker run --rm -v "$(Build.SourcesDirectory):/tmp" -w /tmp \
ghcr.io/trufflesecurity/trufflehog:3.99.8 git file:///tmp/ --json \
--results=verified,unknown --fail --no-update \
| jq -c 'del(.Raw, .RawV2, .SecretParts)'
On GitLab, as of November 2026, pipeline secret detection is free, and “Only the current state of the repository and future commits are scanned for secrets”, so history needs one run with SECRET_DETECTION_HISTORIC_SCAN=. Secret push protection, on Ultimate, reads diffs only and skips binaries, files over 1 MiB and “the initial push that created the repository”, while client-side detection, on every tier, checks issue and merge request text in the browser. When the analyzer cannot scan what was asked, it falls back without failing the job, and after a force push it scans only the current state, so a green job does not prove what it covered.
On the free tier, a finding stops nothing, since the template sets allow_failure: true and merge request results are Ultimate features. The report also “contains detected secrets”, with a field that its schema describes as “an unsanitized excerpt of the affected source code”, so we add a job that reads it, prints no value and fails the pipeline.
# .gitlab-ci.yml. We ran both jq lines on a sample report: exit 1 with a finding, 0 with none
include:
- template: Jobs/Secret-Detection.gitlab-ci.yml
secret_gate:
stage: .post
image: alpine:3.22
needs: [{ job: secret_detection, artifacts: true }]
script:
- apk add --no-cache jq
- jq -r '.vulnerabilities[] | [.name, .location.file, .location.start_line] | @tsv'
gl-secret-detection-report.json
- jq -e '.vulnerabilities | length == 0' gl-secret-detection-report.json > /dev/null
In our example, a storage key in the legacy format passes Azure DevOps push protection, the TruffleHog job exits 183 with a line in its log that holds no value, and on GitLab Free, secret_gate prints the rule, file and line and exits 1. Either way the key is on the server, so it still gets rotated.
Stacking the layers
Judge each layer by what happens when it is the only one that fires. A secret travels from the editor to a commit, a push, a pull request, the default branch, build output, and runtime and logs. Along that road, pre-commit hooks, push protection and the CI gate can stop it, secret scanning alerts and image and log scans find it afterward, and a scheduled full history sweep runs underneath.
| Layer | What it misses |
|---|---|
| Pre-commit | --no-verify, a fresh clone before install, web edits, and text pasted into an issue |
| Push protection | Patterns that do not block, old formats, generic passwords, timed out pushes, any bypass |
| CI range gate | Everything outside the range, and passwords. By then the secret is on the server |
| Scheduled full scan | refs/pull/*, deleted branches, dangling commits, and issues without the org sweep |
| Image and log scans | Artifacts, caches, third party log sinks, logs past retention, and all that you do not feed it |
In our example, a contractor’s fresh clone has no hook, push protection blocks the key until the contractor marks it as used in tests, and the CI gate reads verified, exits 183 and stops the merge. The third layer held, and the key sits on the server all the same, so the owner rotates it before anyone tidies the branch.
| Tool | Verifies live, history, blocks a push | License or cost |
|---|---|---|
| GitHub Secret Protection | 165 of 531 rows. Yes, and issues. Yes, with a reason | Free on public, then $19 |
| TruffleHog | Yes, on by default. Yes. With your own hook | AGPL-3.0 |
| Gitleaks, Betterleaks | Betterleaks, optional. Yes. With your own hook | MIT, Gitleaks Action EULA |
| detect-secrets | Some plugins. No, current tree. With your own hook | Apache-2.0 |
| Azure DevOps product | Yes, automatic. Yes. Yes, with a bypass | $19 a committer |
| GitLab | Ultimate. One off run. Ultimate, diffs only | Pipeline job is free |
Rule lists age, too. In the source on November 7, 2026, Gitleaks had no rule for Azure Storage, the key type behind alert 17, and TruffleHog matched only live Stripe keys, while GitHub App tokens grew to about 520 characters in 2026, and on our reading neither open source rule captures them whole. Generally, assume that a rule misses a format until your own test shows otherwise. A finding past the first layer means rotate first, and cleaning history is optional and comes last, after the hunt, as History comes last explains.
Your first sweep
A first scan stalls programs, because the output is a wall and all of it looks urgent, so this order keeps the count falling and spends the first week on the few findings that can hurt you today. Turn on push protection first, so that the count can only fall, and run the free secret risk assessment, which, as of November 2026, GitHub allows once every 90 days on Team and Enterprise, under Security and quality, Assessments. Then sweep from a trusted host, with a read only token in GITHUB_TOKEN, keeping every state so that nothing is thrown away unseen, and treat the output as a secret.
# The first sweep, from a trusted host. Treat findings.json as a secret
trufflehog github --org=example-systems --issue-comments --pr-comments --gist-comments \
--include-wikis --results=verified,unknown,unverified --no-update --json > findings.json
Group the results by secret, as below, where the repository count reads SourceMetadata., a field in TruffleHog’s source metadata definitions. Then work verified before unknown, and fix by issuer, as One secret, forty alerts does for alerts. Gate the rest behind a baseline with a review date, and put a safe copy in the ticket, made with the filter under TruffleHog.
# The sweep grouped by secret. We ran this jq filter on sample data, with the real field names
# One row per distinct secret, verified first. AWS keeps its secret in RawV2, so key on both.
jq -cs 'group_by(.RawV2 + .Raw) | map({detector: .[0].DetectorName,
state: (if any(.[]; .Verified) then "verified" elif any(.[]; .VerificationError)
then "unknown" else "unverified" end),
places: length, repos: (map(.SourceMetadata.Data.Github.repository) | unique | length)})
| sort_by(.state != "verified", -.places) | .[]' findings.json
# {"detector":"SendGrid","state":"verified","places":40,"repos":12} sample output
# {"detector":"AzureStorage","state":"verified","places":2,"repos":1}
In our example, a sweep of 120 repositories returns 2,400 findings, which group to 310 distinct secrets, and the 9 that verify and 14 that are unknown are the first week’s work. The 9 go to their issuers in groups, so one person kills the shared SendGrid key behind 40 of the rows, while the 14 count as live and are retried from a host that can reach them. The other 287 are unverified, so each gets the three questions from TruffleHog, behind a baseline with an owner and a date, because working verified results first only sets the order, and the unverified ones still need a decision.
A rotated key returns on every full scan, so the record says what closed each finding, and its old window still gets a hunt. Only a schedule keeps the debt from growing back, so the weekly scan then runs on every repository, and each new verified or unknown result gets an owner and a ticket of its own, as alert 17 did, while Metrics that prove the program works names the number to watch.
Real or not
Five questions
A scanner reports strings, and you act on credentials, so triage is the work of getting from one to the other. Treat a finding as real until you can show that it is not, because the two mistakes do not cost the same. In a 2023 benchmark of nine scanners, the most precise tool, GitHub’s own, was right 75 percent of the time, so a false alarm is common and may cost an hour. A working key closed by mistake costs far more, because it stays live with a record that tells the team to stop looking. The benchmark, by Basak and colleagues at ESEM 2023, put the next best tool at 46 percent precision and the best recall at 88 percent, and since it predates current versions, read it for the general pattern and expect the exact figures to have moved. The burden of proof sits on the false positive, which is a conclusion and never a starting point, and no step of triage may make the leak worse.
Every finding gets the same five questions, in the same order, because each answer decides whether the next one is worth asking. Is it a credential at all, is it public by design or a documented sample, and is it live? Whose is it and what can it do? How far did it travel? Only then do you score it. For alert 17, the value fills an AccountKey= slot for stexamplebilling, not the Azurite sample, its validity reads unknown, which counts as live, and it is key1, with full data plane access to the invoices. It reads unknown because validity checks were left off when Secret Protection went on, and unknown may also mean that GitHub has no check for that type.
The route can be quick, and in our example, a ghp_ token in a deploy script that shows active, belongs to a bot with repo scope and sat private for six days scores High after ten minutes of work. Only the first two questions can end a finding, as a false positive with its reason recorded, or as a key that is public by design once its restrictions are checked. A key confirmed dead at the issuer does not close, but rejoins the route at how far it traveled, since its old window needs a hunt.
Is it a credential at all
A string is a credential only if some system accepts it as proof of identity, and a scanner cannot know that. Its signals run from a prefix with a checksum, such as ghp_, through a fixed prefix and context, such as AKIA and AccountKey=, down to entropy alone, which is only a lead. Generally, the stronger the signal, the more evidence a false positive needs.
A false positive must pass three tests: name exactly what the string is, point to a source other than its author, and say why nothing of yours would accept it. Each verdict should also say what would flip it, and end with the record for the alert. The dangerous mistake runs the other way, when a working key looks harmless. A reassuring name, comment or folder comes from whoever wrote the line, and the issuer accepts the key whatever the comment beside it says.
Some strings are never a credential. A lockfile’s integrity value is a hash of the package, which proves no identity, so check only the resolved URL beside it, where a count above 0 in the first check below opens a finding of its own. A UUID proves nothing by its shape, so find the code that reads it, since npm tokens before 2021 were UUID shaped. A documented sample must match whole, because Gitleaks v8.30.1 skips any AWS key ending in EXAMPLE, a test of the suffix alone, and a production key looks just like the well known key that Microsoft prints for devstoreaccount1 in Azurite, its local storage emulator. So compare the whole value with Microsoft’s page, reading both values with read -rs so that neither is ever printed, and one different character, or an endpoint that is not a local Azurite, flips the verdict. Each of these closes as false_positive, with a comment that says why.
# Strings that are never a credential, and documented samples matched whole. Sample output
grep -cE '"resolved": *"[^"]*(://[^/"]*@|[Tt]oken=)' package-lock.json
# 0
grep -rn 'ExportJobId' src/ --include='*.cs'
# src/Billing.Api/Jobs/ExportJob.cs:22: Guid.Parse(cfg["ExportJobId"]);
F=docs/quickstart.md; grep -cF 'AKIAIOSFODNN7EXAMPLE' $F # 1
grep -cF 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' $F # 1
read -rs FOUND; read -rs PUBLISHED # Paste each. Nothing echoes.
[ "$FOUND" = "$PUBLISHED" ] && echo same || echo DIFFERENT # same
unset FOUND PUBLISHED
A key that is public by design is only as safe as the control behind it, since “public on purpose” describes a role with its restrictions in place, and never a shape. Wiz said of a Supabase key in client code that “without RLS policies, this key grants full database access to anyone who has it.” Stripe documents a pk_ key as safe to embed, so its check, the first line below, counts sk_ and rk_ keys under the web source, and the verdict flips if the prefix is sk_ or rk_, whatever the variable is called. A PEM label is only text, and one file can hold both halves, so record a file marked PUBLIC KEY as a false positive only if its PRIVATE KEY count below prints 0.
# Two keys that are public by design, each checked for a secret beside it. Sample output
grep -rnE '(sk|rk)_(live|test)_' src/web/ | wc -l # 0
grep -c 'PRIVATE KEY' jwt-verify.pem # 0
openssl pkey -pubin -in jwt-verify.pem -noout -text | head -1
# Public-Key: (2048 bit)
| Finding | Verdict | What settled it |
|---|---|---|
AKIAIOSFODNN7EXAMPLE and its secret key, in docs/quickstart.md |
Documented sample | AWS prints the pair in its IAM guide, and both values match it, which we check because a real secret can sit beside a sample ID |
pk_live_EXAMPLE0000000000000000 in checkout.tsx |
Public by design | pk_ names the role, and no sk_ or rk_ key sits beside it |
adminPassword: changeme-EXAMPLE in values-prod.yaml |
Real | Its digest matches what is deployed, and a weak value suggests a bad password, not a fake one |
sk_test_EXAMPLE0000000000000000 in PaymentTests.cs |
Real, scored Low | It authenticates as your account, and used_in_tests says where a value sits, not what it can do |
Harmless labels on working credentials
Treat every innocent label as a claim to test, and leave the finding open if the check cannot rule it out. For the placeholder in the table, we compare SHA-256 digests of the found and deployed values, as below, so that the check never logs in with the value, and keep the digests out of the ticket, since the hash of a weak password can be guessed. Only digests that differ, with an owner who shows that nothing reads the file, would flip it, and otherwise the record is a changed password, then revoked.
# A placeholder compared by digest with what is deployed, so that nobody logs in. Sample output
read -rs V; printf %s "$V" | openssl sha256; unset V
kubectl -n billing get secret billing-admin \
-o jsonpath='{.data.adminPassword}' | base64 --decode | openssl sha256
# SHA2-256(stdin)= 1eb5f9e8... The same on both lines.
Encoding adds no protection, because an encoded value has exactly the protection of the file that holds it. The same goes for registry credentials in Docker’s config.json, whose auth value the Kubernetes docs call “obscured but not secret”, for _auth in .npmrc and for every Authorization: Basic header, and hex and URL encoding fare no better. When Aqua searched GitHub in 2023, it found 438 registry credentials of this kind, and 203 were valid. Decode what looks encoded, locally and never on a website, with commands that print a length or a user name instead of the value, and score any Base64 credential as plain text, since a decoded password is a password to change and then close as revoked. What a scanner cannot see covers what GitHub decodes for you.
# Decodes that print a length or a user name, never the value. Sample output
$ grep -A1 '^data:' k8s/billing-db.yaml | sed 's/: .*/: .../'
data:
password: ...
$ awk '/password:/ {print $2}' k8s/billing-db.yaml | base64 -d | wc -c
27
$ jq -r '.auths[].auth' ~/.docker/config.json | base64 -d | cut -d: -f1
ci-bot
A connection string in a Kubernetes Secret manifest can be decoded locally too, never on a website, with its password and key fields masked on the way to the screen, so that the server and login show and the secret does not. Its verdict flips only if it decodes to no credential at all.
# A Base64 connection string decoded with its password and key fields masked. Sample output
read -rs B64 # Paste, Enter. Nothing echoes.
printf %s "$B64" | base64 --decode | sed -E \
-e 's/(password|pwd|accountkey)=[^;"]*/\1=<hidden>/gI' \
-e 's/(sharedaccess(key|signature))=[^;"]*/\1=<hidden>/gI'
echo; unset B64
# Server=tcp:sql-example-prod...;User Id=app_login;Password=<hidden>;
A test key may be exactly what its name says, and a comment may be accurate, but only the issuer can settle either, and you ask it only about keys that your organization owns. In TruffleHog v3.99.8, the Stripe detector skips the sk_test_ shape. That key goes only to Stripe, its issuer, whose livemode tells test from live, and since nothing flips its verdict while Stripe accepts it, the record is a rotation, then revoked. A Slack token that a comment calls dead goes to auth.test, which identifies a token and changes nothing. A reply of invalid_auth means dead at the issuer, recorded as revoked with the reply and its date, since it was once a working key, and the old window still gets a hunt. A reply of "ok":true leaves a live key to score.
# A test key and a key that a comment calls dead, each asked of its issuer. Sample output
read -rs KEY; printf 'Authorization: Bearer %s\n' "$KEY" \
| curl -sS -H @- https://api.stripe.com/v1/balance | jq .livemode
unset KEY # false
read -rs TOKEN; printf 'Authorization: Bearer %s\n' "$TOKEN" \
| curl -sS -X POST -H @- https://slack.com/api/auth.test \
| jq -c '{ok, error}'; unset TOKEN
# {"ok":false,"error":"invalid_auth"}
When a verdict rests on a fact outside the file, date the evidence, since that fact can change after you close the finding. A private key in a test folder is compared offline with each deployed certificate, by the SHA-256 of its public half, and it closes as used_in_tests only if it matches nothing that you can list, while a match means revoking that certificate and recording revoked. A Google Maps key that your organization owns is read for its restrictions with gcloud from your own login, by the key’s ID and never the key string, then tested for Gemini with the value in a header and not in the URL. Unless the restrictions are empty or the call returns 200, either of which flips the verdict, it is public by design, for now, and the record is false_positive, with the date and the restrictions. In February 2026, Truffle Security reported 2,863 live keys that could reach Gemini, since enabling its API let a project’s existing keys call it, and although Google has since tightened this, the key string stayed the same while what it could reach grew.
# Two verdicts that rest on a fact outside the file, so date what they print. Sample output
F=tests/fixtures/client.pem
openssl pkey -in $F -pubout -outform DER | openssl sha256
openssl x509 -in deployed.crt -pubkey -noout \
| openssl pkey -pubin -outform DER | openssl sha256 # They differ.
gcloud services api-keys describe KEY_ID --location=global \
--format='yaml(restrictions)' # maps-backend.googleapis.com only
read -rs KEY; printf 'x-goog-api-key: %s\n' "$KEY" | curl -s -H @- \
-o /dev/null -w '%{http_code}\n' \
https://generativelanguage.googleapis.com/v1beta/files; unset KEY
# 403
Test credentials have opened production systems. In January 2024, Microsoft reported that Midnight Blizzard went from a test tenant account to “a legacy test OAuth application that had elevated access to the Microsoft corporate environment”. In 2025, the username and password 123456 opened an admin login on a test restaurant in the live McHire system. CERT/CC’s PKfail note says that “many OEM devices shipped with hard-coded, untrusted keys in their production UEFI firmware”, test keys marked “DO NOT TRUST”. Tools share the blind spot, since GitHub says that AI detection “may not detect secrets in test code”, and the bypass reason used_in_tests closes an alert on the developer’s own word.
In our example, a client secret in tests/ belongs to the Entra app billing-integration-test, whose role assignment, read from your own login as below, is Contributor on rg-billing-prod. The name says test and the role says production, so we score it on the role. Going forward, ask what issued a key, what it can reach today and who owns it, and keep test identities in a separate tenant, account or subscription, so that a boundary backs the name.
# What an app with a test name can reach, read from your own login. Sample output
$ az role assignment list --assignee APP_ID --all \
--query "[].{Role:roleDefinitionName, Scope:scope}" -o table
Role Scope
----------- ----------------------------------------------------
Contributor /subscriptions/SUB_ID/resourceGroups/rg-billing-prod
Is it live
Liveness is the one question whose answer may mean touching a live system. If the key is in a public commit, skip the test and revoke it now. Otherwise, validate only what belongs to your organization, with the smallest call that answers. Read what you already have first, such as GitHub’s validity field, and when you do test, stop at yes or no. Keep the value off the command line, out of URLs and out of your shell history, since the only host that should ever see it is its own issuer. Never paste it into an online decoder.
A credential that is not yours is not yours to test, even to be helpful, so report it to its owner. A GitHub personal, OAuth or user token can go to POST /credentials/revoke with no authentication, although an installation token cannot, as GitHub and GitLab tokens explains. For your own keys, the listing keeps the value out of your history, and out of the process list, since curl reads the header from stdin. The AWS test is the subshell listing in The negative test, which keeps every other credential out of play and lets the values die with the subshell. An ASIA key also needs its AWS_SESSION_TOKEN, and you run the test only after the account check below has named an account that is yours.
# Liveness with the value off the command line, for keys your organization owns. Sample output
read -rs TOKEN # Paste, Enter. Nothing echoes, and nothing lands in your history.
# printf is a builtin, so the value stays out of the process list. curl reads it from stdin.
printf 'Authorization: Bearer %s\n' "$TOKEN" | curl -sS -H @- https://api.github.com/user \
| jq .login; unset TOKEN # "svc-release"
AWS needs its own caution. The call aws sts get-caller-identity needs no permissions, and attackers use it to test a key. It made up 25 percent of the calls on the honeypot key that Orca leaked to GitHub in 2023, and TruffleHog reports that it sets off AWS “canary tokens”, decoys that raise an alarm. Start with the call below, from your own credentials, and if the account is not yours, stop. Either way, note who ran each test, when and from where, since your hunt will find it.
# From your own credentials. It names the account from the key ID, not its state
aws sts get-access-key-info --access-key-id AKIAIOSFODNN7EXAMPLE
We could confirm no call that names the owner of a PyPI or Docker Hub token without using it, as aws sts get-access-key-info does for an AWS key ID, so never test those with an upload. For the others, the table gives the least invasive call, and a yes from any of them ends the question, since only a working credential can answer.
| Credential | Least invasive call | What it tells you |
|---|---|---|
| AWS key pair | aws sts get-caller-identity |
Account, ARN and user ID |
| GitHub PAT, OAuth | GET https:// |
Login, and scopes on classic and OAuth |
| GitLab PAT | GET /api/ |
Token name, scopes and expiry |
| Slack token | POST https:// |
Team and user. A webhook test would post |
Stripe sk_, rk_ |
GET https:// |
livemode tells test from live |
| Entra client secret | POST .../, client credentials |
Mints a real token, logged as a sign-in. Print only expires_in |
| Azure Storage key | az storage container list --auth-mode key, with the key in AZURE_STORAGE_KEY behind the empty check in The negative test, since an empty variable makes az fetch the live key |
It works. The key list match below never uses the key |
| Azure SAS URL | None. Read se, sp, sr and skoid offline |
Expiry, rights, resource, user delegation |
Once the account is yours, a yes from aws sts get-caller-identity ends the question, and a no is weaker, because a check can fail for reasons that have nothing to do with the key. GitHub labels unknown as “Possibly active secret”, and an AWS key set to Inactive can be switched back on. GitHub’s unknown means no check for the type or a failed check, and its inactive still calls for a hunt, as Validity checks, custom patterns and exclusions explains. TruffleHog’s unverified “may be invalid, expired, or verification disabled”, or IP restricted, and the AWS quarantine policy and a deleted stored access policy leave a no weaker than it looks, as AWS access keys and Azure Storage keys and SAS explain. A finding therefore stays open until the issuer’s own record agrees.
Blast radius and how far it traveled
“Blast radius” is everything a credential can reach, directly or in one more step, including whether it can reach or create other credentials. Its four parts are the identity, its resources, what it can do to them and that last question, which decides whether revoking the leaked value ends the access at all. The issuer holds every answer, so a commit author or a variable name is only a lead. We read it from our own identity, never with the leaked key. Microsoft says that a storage key gives “full access to the storage account data and the ability to generate SAS tokens”. For the leaked storage key, a match against the account’s own key list proves ownership and liveness without using the key.
# Shell, from your own login. Sample output
read -rs LEAKED # Prints the name of the matching key, and never a key.
az storage account keys list -g rg-billing-prod -n stexamplebilling -o json \
| LEAKED="$LEAKED" jq -r '.[] | select(.value == env.LEAKED) | .keyName' # key1
unset LEAKED
Q='{sharedKey:allowSharedKeyAccess, network:networkRuleSet.defaultAction}'
az storage account show -g rg-billing-prod -n stexamplebilling --query "$Q"
# {"network":"Allow","sharedKey":null}
az storage container-rm list --storage-account stexamplebilling -g rg-billing-prod \
--query '[].name' -o tsv # invoices
In that output, a null sharedKey means allowed, as does any value but false, so with network at Allow, the key works from any network. No output from the match would mean that the value matches neither key, and the match needs listkeys, which Owner and Contributor hold. The key reaches every blob in the account, not only the invoices, since listKeys reports its permissions as full, and az storage container-rm list from your own login shows what is there. The key itself works on the data plane, so it cannot change roles, network rules or keys. It can still sign account and service SAS, each a new bearer URL that Azure does not track, and anything signed with key1 works until key1 is renewed, as Azure Storage keys and SAS explains. For an Entra secret, start with az role assignment list --assignee APP_ID --all.
An AWS key takes more reading. The IAM policy simulator counts group policies and runs nothing, and as of November 2026 it returns one aggregate result per action, so ask about one resource at a time. Since it skips resource policies, and AWS warns that its results can differ from your live environment, read the bucket policy too.
# Shell, from your own identity. Sample output
U=ci-deploy; ARN=arn:aws:iam::123456789012:user/$U # From the identity call in Is it live.
aws iam list-attached-user-policies --user-name $U --query 'AttachedPolicies[].PolicyName' # []
aws iam list-groups-for-user --user-name $U --query 'Groups[].GroupName' # []
aws iam list-user-policies --user-name $U --query PolicyNames # ["deploy-exports"]
aws iam get-user-policy --user-name $U --policy-name deploy-exports \
--query PolicyDocument.Statement
# [{"Effect":"Allow","Action":["s3:ListBucket","s3:GetObject","s3:PutObject","s3:DeleteObject"],
# "Resource":["arn:aws:s3:::example-billing-exports","arn:aws:s3:::example-billing-exports/*"]},
# {"Effect":"Allow","Action":"iam:CreateAccessKey",
# "Resource":"arn:aws:iam::123456789012:user/ci-deploy"}]
aws iam simulate-principal-policy --policy-source-arn $ARN --resource-arns $ARN \
--action-names iam:CreateAccessKey iam:AttachUserPolicy \
--query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text
# iam:CreateAccessKey allowed
# iam:AttachUserPolicy implicitDeny
In our example, the ci-deploy key can list, read, replace and delete every object in example-billing-exports, so it reaches any customer data in those exports, and it cannot raise its own rights. Its iam:CreateAccessKey on its own user lets a holder mint a second key, though, so deleting the leaked one may not end the access: list the keys again at revocation, and take that line out of the policy. For SaaS tokens and databases, trufflehog analyze does this reading, and it reports, in the project’s words, “Who created it? What resources can it access? What permissions does it have on those resources?” As of November 2026 it has 43 analyzers, including GitHub, Slack, Stripe and Postgres, and none for AWS, Azure or Google Cloud. It is interactive, so it needs a terminal, and where a liveness check sends one request, it “can send many requests” that may land in the issuer’s logs. Keep it to keys that your organization owns, and note the time in the ticket, or your analysis may look like abuse in the hunt.
How far it traveled sets the audience. It also sets the row of the score and the window that the hunt must search. Search history for the setting name, never the value, as git log -S'AccountKey=' does, and round the window outward, because a commit date is whatever the author’s machine said. A secret can also reach images, logs and laptops, where Git cannot show it, and if the leak is public, treat the key as used, since AWS keys on public GitHub have been used within minutes, as Revoke now or rotate with overlap shows. A fork of a private repository stays private and follows its parent’s team permissions, so by itself the contractor’s fork adds a copy to chase without widening the audience, while the parent stays private. A fork network shares Git data, though, and Truffle Security showed in 2024 that commits pushed to a private fork before its upstream went public could be read from the public one. GitHub’s docs do not say whether the forks call shows a private fork that you cannot read, so ask as well.
The calls below trace alert 17, reading the alert with hide_secret=true so that the value stays out of your terminal, and publicly_leaked says whether GitHub has seen it in public. The collaborators call needs write access, and on an organization repository it also counts teams and base permissions.
# How far it traveled, for alert 17. Sample output
# When did it arrive? Search for the setting name, never for the value.
git log --all -m --reverse --format='%h %cr %an %s' -S'AccountKey=' \
-- src/Billing.Api/appsettings.Development.json | uniq
# 7f3c2ab 11 days ago dreyes Add blob settings for local testing
# 5e91d0c 9 days ago dreyes Merge pull request #212 from example-systems/blob-settings
# What has GitHub seen? hide_secret keeps the value out of your terminal.
R=repos/example-systems/billing-api
gh api "$R/secret-scanning/alerts/17?hide_secret=true" --jq '{validity, publicly_leaked}'
# {"publicly_leaked":false,"validity":"unknown"}
gh api "$R/secret-scanning/alerts/17/locations" --paginate --jq '.[] | [.type, .details.path]'
# ["commit","src/Billing.Api/appsettings.Development.json"]
# Who holds a copy, and who could read it?
gh api "$R/forks" --paginate --jq '.[] | {full_name, private}'
# {"full_name":"example-contractor/billing-api","private":true}
gh api "$R/collaborators" --paginate --jq '.[].login' | wc -l # 31
Read those answers as evidence for one row of the score. In our example, the first commit opens the window 11 days back, publicly_leaked is false, and the fork and the 31 logins give a complete list of who had access. None of that proves that no other copy exists, so any copy that you cannot account for moves the finding to a wider row. Score a private leak on privilege and audience, never on the word private, and rotate it as you would any other leak.
Score it
Although CVSS is the obvious tool, it rates vulnerabilities and does not measure one credential exposed in one place. We found no published standard for this either, so the table is our own model and its hours are our recommendation. Private means you can list everyone with access, and shared widely means you cannot. Those are the rows, from public through shared widely to private. The columns run from the first, production write or admin, through production read or customer data to non production or narrow scope, so a cell is one row against one column. Unknown validity counts as live, the clock starts when the alert fires and stops at revocation, and between two cells, you take the higher. A lower score should need evidence, and a higher one needs only doubt. Alert 17 is private with production write access, which gives High, so key1 must be dead by 09:14 on Wednesday.
A score should take minutes, since it asks only what the questions above have answered, and when two people disagree about one, an answer is usually still a guess, so go back and get it. A printable list may run to a few dozen names, while org wide read, a vendor’s system or a fork outside your access rules is shared widely, and each column is the worst single thing the credential can do, from the issuer’s record and never from its name. A scanner’s inactive keeps the score until the issuer’s own record agrees, and confirmed dead means deleted, revoked or expired for good, as seen at the issuer.
| Score | Who could read it, and what it reaches | Revoke within |
|---|---|---|
| Critical | Public, with production write, admin, read or customer data; shared widely, with production write or admin | 1 hour. Page the owner at any hour |
| High | Public, non production or narrow scope; shared widely, production read or customer data; private, production write or admin | 24 hours. Pull the owner in today |
| Medium | Shared widely, non production or narrow scope; private, production read or customer data | 7 days. Schedule it this week |
| Low | Private, non production or narrow scope, or a key confirmed dead at the issuer, whose old window still gets a hunt | 30 days. Backlog, with an owner and a date |
Note. OWASP says only “immediate revocation”, so these hours are ours.
In our example, the ci-deploy key turns up in a public gist at 14:05, untested, so unknown counts as live, and the public row and the production write column give Critical, dead by 15:05 with no overlap. A live Slack bot token in deploy-notes, which every member of the organization can read, is shared widely, since base permissions admit anyone who joins tomorrow, but the analyzer shows only chat:write, which can post but cannot read, so it scores Medium, where channels:history would make it High. The sk_test_ key, in a private repository with 9 people and no forks, scores Low, with an owner and a date, since an unassigned credential tends to stay leaked.
When you cannot tell
Some findings will not answer, because the owner has left or no safe call exists. Uncertainty should move a finding toward the next action and never toward closed, so each gap has a default that errs high.
| You cannot tell | Assume | Do this |
|---|---|---|
| If it is a credential | It is one | Find the code that reads it, and ask the last committer. With neither, keep it open with an owner and a date. |
| If it is live | It is live | Rotate it, which answers the question and often costs less than proving it. |
| If it is yours | It is not | Do not test it. Ask the vendor, report a stranger’s key to its owner, and treat a customer’s as an incident, as in When a leak becomes an incident. |
| What it can do | The first column | Read the issuer’s record, and move right only on what it shows. |
| Who could read it | The wider row | If you cannot print the list, it is Shared widely. If a public copy is possible, it is Public. |
| Who depends on it | Something does | Use a reversible switch, such as an AWS key set to Inactive, in a planned slot, and delete the key after the hunt. |
Record the decision
A decision that lives in a head gets made again at the next scan, so write each one down. We recommend a record of the finding ID, decision, reason, evidence, who decided and when, and for accepted risk, a review date, with the reason code and a short comment on the alert and the rest in the ticket. Every suppression also needs a reason and a reviewer, because a file in the repository can silence a finding before any reviewer sees it. The first allowlist below hides every secret under src/, while the second names one file and one value.
# Bad, in .gitleaks.toml
[[allowlists]] # Every secret under src/ is invisible, with no reason and no owner.
paths = ['''src/.*''']
# Good, in .gitleaks.toml
[[allowlists]] # One file, one exact value. A new secret still fires.
description = "FP-212. AWS documented sample. Checked by dreyes, reviewed by tkim."
condition = "AND"
paths = ['''^docs/quickstart\.md$''']
regexes = ['''^AKIAIOSFODNN7EXAMPLE$''']
In .gitleaks.toml, [[allowlists]] replaced [allowlist] in v8.25.0, and mixing the two is an error. A comment in the code, such as gitleaks:allow, or a path in .github/ can hide a finding too, and since inline tags trust the author, audit them with --ignore-gitleaks-allow in Gitleaks, --no-ignore-tag in TruffleHog and detect-secrets scan --only-allowlisted.
| Tool | Close with | Suppress with |
|---|---|---|
| GitHub | false_positive, wont_fix, revoked, used_in_tests |
paths-ignore in secret_scanning.yml |
| GitLab | Acceptable risk, False positive, Mitigating control, Used in tests, Not applicable | Exclusions, an Ultimate experiment, for push protection only |
| Gitleaks | No alert state | gitleaks:allow, .gitleaksignore, [[allowlists]], --baseline-path |
| TruffleHog | No alert state | trufflehog:ignore, after the secret |
| detect-secrets | Labels set with detect-secrets audit |
# pragma: allowlist secret, baseline |
On GitHub, each reason code is a claim with a precondition: false_positive only after the three tests, used_in_tests only when the value reaches no live system, revoked only after the issuer has killed the credential, and wont_fix only with an owner and a review date. Through the API, a 403 means that delegated alert dismissal is on and you are not a reviewer, a 422 is usually a state that does not fit its resolution, and a 400 is a bad comment or no change.
# Closing alerts, each with a reason code and a comment that says why
R=repos/example-systems/billing-api
resolve() { # alert number, reason code, comment
gh api -X PATCH "$R/secret-scanning/alerts/$1" -f state=resolved -f resolution="$2" \
-f resolution_comment="$3" --jq '{number, state, resolution}'
}
resolve 31 false_positive 'AWS documented sample. Whole pair matches the IAM guide. tkim.'
resolve 33 used_in_tests 'Self signed fixture. Digest matches no deployed certificate. tkim.'
resolve 36 wont_fix 'Vendor offers no rotation yet. Owner tkim, review next quarter.'
One secret, forty alerts
Scanners count places, and you revoke credentials. In our example, a shared CI template carried one SendGrid key into 12 repositories, and 40 alerts would send 12 teams after a problem that one person can fix at the issuer. Dedupe by secret, then group by issuer, so that each issuer gets one owner and one revocation, and close the alerts together once the issuer has killed the key. Work verified findings first, but do not drop the rest, since GitGuardian, a vendor, reports that ranking by validity alone misses 46 percent of critical secrets.
# Open alerts across the organization, grouped by secret. Sample output
# One row per distinct secret. The value stays in the pipe, so hide_secret is left out here.
gh api --paginate "orgs/example-systems/secret-scanning/alerts?state=open&per_page=100" \
| jq -cs 'add | group_by(.secret) | map({type: .[0].secret_type, alerts: length,
repos: (map(.repository.full_name) | unique | length)}) | sort_by(-.alerts) | .[]'
# {"type":"sendgrid_api_key","alerts":40,"repos":12}
Contain before you clean
Most of the harm in a secrets response comes from sensible steps taken in the wrong order. Under pressure, the first instinct is to edit the file that holds the value, which is the least urgent task of all. Deleting the line feels like progress, yet the key still works in every clone, while revoking takes the value out of every copy at once, which is why GitHub, OWASP and AWS all contain first. The full order runs record, issue, deploy, revoke, hunt, clean and prove: note the facts, create the replacement, move every consumer, kill the old value, search the window, rewrite history if it helps and show that the old value fails. A credential that is public or under attack goes straight to revoke. The advice to preserve every log before you change anything goes further than the sources do. NIST warns that delaying containment to gather evidence “can be dangerous because an attacker could escalate unauthorized access or compromise other systems.” That warning comes from SP 800-61r3, of April 2025, which describes outcomes without prescribing procedures, so it sets no fixed order beyond the warning. You therefore record what you know and start your exports in parallel, and neither is ever a reason to wait.
Record and export, in parallel
Give the record 15 minutes at the most, ideally kept by a second person while the first contains the key. Write down what leaked by identifier and never by value, such as a key ID or a key name like key1, with the repository, commit SHA, first commit time and who could read it. An Entra keyId or the SHA-256 of a token serves as well, and every action goes in one log, with a time and a name. Then export first the logs that expire soonest. Revoking a key does not erase its logs, but retention can: as of November 2026, GitHub Git events, which need Enterprise Cloud, and Entra Free sign-ins last 7 days, and Entra P1 and P2 keep sign-ins for 30, although upgrading is not retroactive. CloudTrail Event history and the Azure Activity log last 90 days, and GitHub web events reach back 180. In our example, a key found after 11 days has already lost 4 days of Git events, and each day loses one more.
# Record and export, in parallel with containment
mkdir -p ~/ir/billing-key && cd ~/ir/billing-key # outside the repository, and never a secret
# The commit. Search on a part of the line that is not secret.
TZ=UTC git -C ~/src/billing-api log --all -m --date=format-local:'%Y-%m-%dT%H:%M:%SZ' \
--format='%H %cd' -S'stexamplebilling' | uniq > commits.txt
START=$(tail -1 commits.txt | cut -d' ' -f2) # the first commit, in UTC
# GitHub Enterprise Cloud, as an organization owner: Git events last 7 days. The token needs
# read:audit_log, so run gh auth refresh -s read:audit_log first, or hand this line to an owner.
gh api --method GET /orgs/example-systems/audit-log --paginate -f include=git > git-events.json
The CloudTrail export can therefore wait until the 7 day logs are safe. The GitHub line calls the organization audit log API, and gh auth refresh adds the scope that it needs. Entra sign-ins since the first commit come from the Microsoft Graph beta endpoint one page at a time, so the loop below follows each next link.
# Entra: needs this beta filter, AuditLog.Read.All and a role such as Security Reader.
F="createdDateTime ge $START and signInEventTypes/any(t: t eq 'servicePrincipal')"
URL="https://graph.microsoft.com/beta/auditLogs/signIns?\$filter=$F"
while [ -n "$URL" ]; do # az rest reads one page, so follow @odata.nextLink
az rest --url "$URL" > page.json || break
jq -c '.value[]' page.json >> signins.jsonl
URL=$(jq -r '."@odata.nextLink" // empty' page.json)
done
# Azure: the defaults are 6 hours and 50 events as of November 2026, so widen both.
az monitor activity-log list -g rg-billing-prod --offset 30d --max-events 5000 > activity.json
Revoke now or rotate with overlap
The one test asks whether the credential is public, or whether someone you cannot name is using it, and a yes to either means revoke now. For a storage key, two queries make the first look at the logs that answers the second question: the StorageBlobLogs query in Azure, for who signed with a key or a SAS, and the AzureActivity query after the table below, for who listed or regenerated keys. Every address they return should be one that you can name, such as the app or a deploy runner, and an address that you cannot name, or an empty result from an account that was not logging, sends you to revoke now. For an AWS key, the first look is the CloudTrail lookup in AWS CloudTrail instead, since its export can wait but the one test cannot. Overlap is fair, for minutes to hours, when the leak is private, every caller has a name and revoking would stop customers, but the leaked value keeps working throughout. In three vendor experiments from 2023 to 2024, AWS keys on public GitHub were first used in about 2 to 7 minutes.
Those figures come from Orca, Unit 42 and Clutch, for one kind of secret in one place, and other places were slower, from hours to months. Either way, GitHub’s tutorial allows a new secret first when downtime is the worry, so overlap is fair if you know its cost. The table below sets the move from what you know, and adds two more, for a test key and for a credential with no second slot.
| What you know | Move | Example |
|---|---|---|
| Public, or logs show use that you cannot explain | Revoke now | An AWS key for ci-deploy lands in a public gist. You set it Inactive at once and take the failed export. |
| Production, private, no sign of use in the logs, and revoking would stop customers | Overlap, minutes to hours | A Stripe key sits in a private repository with 9 people. You rotate with a short expiry and deploy, so checkout never stops. |
| Test, development or inactive, in a private repository | Planned slot | A sandbox key in a test fixture. You agree a time with its owner, inside the response clock. |
| No second slot exists | Short planned outage | A GitHub webhook secret has no documented overlap, so you change both ends together. |
// The rest of the first look for the one test. The StorageBlobLogs half is in Azure, under Hunt for abuse.
// Activity: who listed or regenerated keys. in~ is deliberate, since case varies in this table.
// A listKeys from a caller that you cannot name means that both keys may be held, so renew both.
AzureActivity | where TimeGenerated > ago(90d)
| where OperationNameValue in~ ("Microsoft.Storage/storageAccounts/listKeys/action",
"Microsoft.Storage/storageAccounts/regenerateKey/action")
| project TimeGenerated, Caller, CallerIpAddress, OperationNameValue, ActivityStatusValue
At Example Systems, the key was not public, and the first look showed only the app and the deploy runners, so the team chose overlap, since renewing at once would stop invoices, key2 was never exposed and no stranger had called listKeys. The team moved every consumer to key2 without the value touching a command line, waited out the app’s 15 minute invoice links and confirmed in StorageBlobLogs that nothing still signed with key1. Along the way, a deploy proved that key2 worked before anyone touched key1. Only then did the team renew key1, which ended the leaked value and every account and service SAS signed with it. For a SAS, renewal is the only fix, because no API lists one to search for, and Microsoft says that “The SAS token is not tracked by Azure Storage in any way.”
# Move every consumer to key2. Code that signs a SAS is a consumer too
CONN=$(az storage account show-connection-string -g rg-billing-prod -n stexamplebilling \
--key key2 --query connectionString -o tsv)
printf %s "$CONN" | gh secret set STORAGE_CONN -R example-systems/billing-api
( umask 077; printf '{"STORAGE_CONN":"%s"}' "$CONN" > conn.json ) # off the command line
az webapp config appsettings set -g rg-billing-prod -n app-billing-prod \
--settings @conn.json -o none; rm -f conn.json; unset CONN
# Once the longest SAS that the app signs has expired, renew key1. The command returns
# the new keys, so send its output nowhere
az storage account keys renew -g rg-billing-prod -n stexamplebilling --key key1 -o none
# Then run the negative test in Prove the fix, before shared key access goes off
What survives a rotation
For every credential, ask what outlives the key. Although revoking ends the credential, sessions, tokens and open connections may carry on. A leaked signing key is the worst case. It can “mint” credentials for anyone, and they appear in no list. Whatever the old key already minted, and whatever an attacker built with it, keeps working until you end it by name, and Trivy’s 2026 advisory shows the cost: “credential rotation was performed but was not atomic”, which “could have allowed the attacker to retain access”.
| You rotated | What survives | Do this instead |
|---|---|---|
| AWS access key | STS sessions, which AWS says are “valid until they expire”, up to 36 hours, so we assume they outlive the key, and role sessions it opened | Deny them by aws:TokenIssueTime on the user, and on each role it could assume |
| Azure Storage key | A SAS signed with the other key, until it expires, and any user delegation SAS | az storage account revoke-delegation-keys, which also ends your own good links, perhaps after a delay, and renew the other key if a caller you cannot name may hold it |
| Entra client secret | Access tokens already issued, which last 60 to 90 minutes, or up to 24 hours for Microsoft Graph under continuous access evaluation, or CAE, and on our reading run on after the deletion | Expect the lifetime, since disabling the principal ends only the CAE tokens |
| Database password | Open connections, not reauthenticated, and on Azure SQL Database reauthorized with the original password | Change it first, then pg_terminate_backend or KILL, and DBCC FLUSHAUTHCACHE on Azure SQL Database |
| JWT signing key | Forged tokens, until verifiers drop the key | Pull it from the JWKS and every verifier now, flush caches, review sessions |
Note. Rotation touches nothing that an attacker may have added, such as a key, a user or a webhook, so the hunt removes those.
The deny on aws:TokenIssueTime touches only temporary credentials, so long term keys are unaffected, although on a shared role it also stops legitimate sessions until they assume the role again. It cannot change root sessions, and a resource policy that allows the principal needs its own deny. The steps for each credential are in Rotate and revoke, credential by credential.
Hunt across the whole window
The hunt reads the issuer’s logs across the whole window. It opens when someone unintended could first read the credential, at the first commit or earlier and never at the alert, and it closes when the last thing that the credential spawned has died. Where logs are missing, record that part of the window as unknown. On Azure, tables such as StorageBlobLogs exist only if diagnostic settings were sending them to Log Analytics before the incident, and the queries are ours, so run each one on a known event before you trust an empty result. For Example Systems, StorageBlobLogs showed only known addresses, but Git events reach back only 7 of the 11 days, so the record reports no evidence of abuse, not proof of none. The queries for each platform, from CloudTrail to the GitHub audit log, are in Hunt for abuse.
When a leak becomes an incident
No primary source says when a leak becomes an incident, so we recommend six triggers: use that you cannot account for, access to personal, payment, health or customer data, reach into production or a signing key, missing or expired logs, a public secret with unknown exposure, or a key owned by a customer or partner. Bring people in at the first one, without waiting for proof of abuse: the secret owner and repository admins during or after the revocation, then your security lead and whoever owns legal and privacy. Leave any question of notice or law to counsel, including whether a legal hold should pause any cleanup. Regimes that often come up include SEC Form 8-K Item 1.05, GDPR Article 33 and 45 CFR 164.404, although naming one does not mean that it applies. NCSL keeps a list of state breach notification laws, and we stop at the questions below.
| No. | Question |
|---|---|
| 01 | Is this an incident under our contracts? |
| 02 | Does access alone count, or only acquisition? |
| 03 | Which regimes and insurers apply? |
| 04 | When did each clock start? |
| 05 | Do contracts require notice, and how fast? |
| 06 | Must the insurer hear about it before we bring in outside help? |
| 07 | How do we treat a window with no logs? |
| 08 | Should a legal hold pause any cleanup? |
| 09 | Who speaks externally, and is this privileged? |
| 10 | Does law enforcement change the timing? |
History comes last
A history rewrite comes last, if at all, because a rewrite does not undo a leak. It reaches the branches and tags that you control, while pull request refs, forks, cached commit views, clones, CI caches and event feeds keep what they have. A force push may even draw attention, since public event data records it, and an old commit can stay reachable by its SHA. With a private repository, a known audience and a dead key, Example Systems waited for the hunt, chose a limited rewrite of the one string and asked the contractor to delete the fork. Then the whole team cloned again, for the reason given in Afterward, everyone clones again. The rewrite itself, and when to skip it, is in Clean up Git history.
Rotate and revoke, credential by credential
Each recipe below runs the same five steps in the same order, find it, replace it, revoke it, what survives and confirm, so that you can find your place on a bad day. What survives is where most responses stop too early, and every test touches only what your organization owns, using the calls in Is it live and the dead signs in What a Dead Credential Looks Like.
Two slots and an overlap
Almost every credential that rotates without an outage works the same way: the issuer holds two valid values for a while, which we call “slots”. You issue into the free slot, deploy everywhere, confirm that nothing presents the old value, as The first hour requires, and only then retire it. The old value works throughout, so in a leak the one test there comes first, and if it says to revoke now, you run the recipe’s revoke step before its replace step and take the failed calls. A second key from the same leaked config file is not a clean slot. The pattern tends to fail in three places: a consumer missing from the list, a process that read the value at startup and still presents it, and a retire step that never runs, which leaves two live values.
| Credential | The two slots and the longest overlap | How you see the old slot is idle, and how you retire it |
|---|---|---|
| AWS access key | Two keys per user. Yours to set | get-access-key-last-used, then CloudTrail by key ID. update-access-key, then delete |
| Azure Storage key | key1 and key2. Yours to set |
AuthenticationHash in StorageBlobLogs. az storage account keys renew |
| Entra client secret | Several at once. Until it expires | The credential key ID in the sign-in logs. az ad app credential delete |
| Stripe API key | Rotate key, with an expiry. Up to 7 days | The request logs for that key, in Workbench. The expiry, or rotate with Now |
In ticket SEC-1204, which is not the public gist case in Score it, an alert named the ci-deploy key in deploy.yml, in a private repository, and CloudTrail showed only the runner range. So Example Systems overlapped: a second key went in, a deploy proved it, and at 14:02 UTC the old key went Inactive, with a deny on older sessions. At 14:31, an hourly export.yml in billing-reports failed with a 403. It held its own copy of the key, and CloudTrail could not separate it from the deploy, since both ran in the same runner range. The failed run posted to the team channel, the job moved to the new key by 14:40 and the negative test passed at 14:47. Because the key was retired in working hours, with people watching, the cost was one late file.
AWS access keys
Work from an identity other than the leaked one, since the quarantine policy that AWS may attach denies that user the calls that manage access keys. That policy is not containment, because AWSCompromisedKeyQuarantineV3 leaves the keys valid and does not deny sts:AssumeRole, Secrets Manager, SSM, DynamoDB or RDS. AWS also records only the first use in any 15 minute span, so the last use shown can trail the latest actual use by up to 15 minutes. To list every consumer of the key by address, read its timeline in AWS CloudTrail. In the listings, AWS’s public sample key ID stands in for the leaked one.
# Find it: the keys on ci-deploy, and when the leaked one last worked
aws iam list-access-keys --user-name ci-deploy
aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE
A user can hold two keys at most, so ci-deploy may have to lose a spare first. The new secret appears only in the create response, so it goes to a file and on to each consumer, never to the screen. Better still, make it the last static key, with Federation from GitHub to AWS.
# Replace it: the new secret goes to a file, then to each consumer
( umask 077; aws iam create-access-key --user-name ci-deploy --output text \
--query 'AccessKey.[AccessKeyId,SecretAccessKey]' > new-key.txt ) # never on screen
REPO=example-systems/billing-api # and every other consumer
printf %s "$(cut -f1 new-key.txt)" | gh secret set AWS_ACCESS_KEY_ID -R "$REPO"
printf %s "$(cut -f2 new-key.txt)" | gh secret set AWS_SECRET_ACCESS_KEY -R "$REPO"
rm -f new-key.txt
Set the old key Inactive, which is reversible and keeps the evidence, and delete it after the hunt, since anyone with iam:UpdateAccessKey can switch it back on. Sessions made with the key can outlive it. Suppose an attacker opens one at 14:20 UTC and you rotate at 15:00: by default that session runs 12 hours, until 02:20. So deny tokens issued before now, on the user and on each role that the key could assume, and test a denied session with a call other than get-caller-identity, which an explicit deny does not stop. The deny’s limits are set out in What survives a rotation, and the old key’s negative test is in The negative test.
# Revoke it, then deny every session issued before now
aws iam update-access-key --user-name ci-deploy \
--access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ); cat > deny.json <<JSON
{"Version":"2012-10-17","Statement":{"Effect":"Deny","Action":"*","Resource":"*",
"Condition":{"DateLessThan":{"aws:TokenIssueTime":"$NOW"}}}}
JSON
aws iam put-user-policy --user-name ci-deploy --policy-name DenyOlderSessions \
--policy-document file://deny.json # put-role-policy, with --role-name, for each role
Azure Storage keys and SAS
Beyond the running case, each of the two 512 bit keys grants full data plane access. Since Owner and Contributor can both read them, we treat both as suspect if anyone could have called listKeys, although that is our inference, and if a caller that you cannot name did, renew both and take the outage. Depending on what signed it, a SAS ends when its key is renewed, within about 30 seconds of its stored access policy being deleted, or once az storage account revoke-delegation-keys -n stexamplebilling -g rg-billing-prod revokes the delegation keys, perhaps after a delay. Its URL then returns 403, but re-creating the same policy name revives every old SAS tied to it. A renewal leaves a user delegation SAS untouched, since it hangs off a delegation key that lasts up to seven days, and revoking the delegation keys ends all of them, your own good links included. No API lists a SAS, so you cannot count them or end one by ID, and after a renewal, keyCreationTime shows the new date.
Entra application secrets
An Entra client secret is a password for an application, and an app may hold several at once, which makes overlap simple and a careless reset dangerous. Its exposure is whatever its tokens reach, generally including the app’s Key Vault secrets. Listing the app’s credentials returns metadata only, with --cert for certificates, and hint holds the first three characters of each secret, enough to match the leak to its keyId. Always pass --append to az ad app credential reset, because without it the command clears every password and key on the app, an outage that also wipes the evidence, and the new secret goes through a file to Key Vault.
# Entra: find by hint, replace with --append, then revoke. The last line is only for hostile use, and in that case it runs first
az ad app credential list --id APP_ID --query "[].{keyId:keyId, hint:hint}" -o table
( umask 077; printf %s "$(az ad app credential reset --id APP_ID --append --years 1 \
--display-name rotated-2026-11 --query password -o tsv)" > ./new.txt )
[ -s ./new.txt ] && az keyvault secret set --vault-name kv-example-prod \
-n billing-client-secret --file ./new.txt -o none && rm ./new.txt
# Deploy the new secret to every consumer, and delete the old one once its keyId is idle in the sign-in logs
az ad app credential delete --id APP_ID --key-id KEY_ID
az ad sp update --id APP_ID --set accountEnabled=false # a tenant-local stop
If the use looks hostile, Microsoft’s playbook disables sign-in first, then adds a new certificate and removes every old credential, not only the leaked one, which the listing deletes by keyId. Microsoft says that an access token cannot be revoked, and removing a secret is not a CAE revocation event, while disabling the service principal is. So start a clock at the deletion, run it for the token lifetime in What survives a rotation, and record its end as the close of the window. Whatever the app could read rotates too, in Microsoft’s order: Key Vault secrets it fetched with GetSecret, then the rest of those vaults, then the rest of the subscriptions. Read the audit log for new credentials and app role assignments as well, since either may outlive this secret. The old secret’s negative test is in The negative test.
GitHub and GitLab tokens
Check the prefix first, since ghp_ and github_pat_ belong to a person, ghs_ to an App installation, and a deploy key is an SSH key on one repository. A revoked token cannot be restored, so generally you hold the replacement first, although GitHub revokes a valid token pushed to a public repository or a public gist, so it may be dead already. GET /user, with the header on standard input, names the account, and gh api /repos/ lists deploy keys. Issue a fine-grained token for the fewest repositories that work, and replace an immutable deploy key by adding a new one and deleting the old. A GitHub App can hold up to 25 private keys, so add its new one first.
Delete your own token in Settings, Developer settings. One that is not yours goes to the revoke endpoint, which works by value and answers 403 to an auth header, so send none. It takes up to 1,000 tokens, tells the owner and does not cover ghs_ tokens.
# Revoke a token by value, with no auth header, then a deploy key
read -rs TOKEN; printf '{"credentials":["%s"]}' "$TOKEN" | curl -sS -d @- \
-H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/credentials/revoke; unset TOKEN
gh api --method DELETE /repos/example-systems/billing-api/keys/KEY_ID # a deploy key
Clones survive with whatever the token pulled and anything its holder added, so compare the repository’s /hooks, /keys and /collaborators with what you expect, then the SSH keys on the token owner’s account. A classic token with write:public_key can add one, and GitHub does not say whether it goes with the token. An installation token lasts one hour, and GitHub does not say whether deleting an App key ends a live one, so plan on the hour, and deleting a token that created a deploy key deletes that key too. On GitLab, GET /api/, with the header on standard input, shows a token’s name, scopes and expiry, and you revoke by ID with DELETE /personal_access_tokens/:id, since /revoke is not in the API as of November 2026. Rotating through /:id/rotate revokes at once with no overlap, revoking by value exists only as an admin Experiment on Self-Managed, and a dead token gets a 401 from the same GET.
Database passwords
A password change does nothing to an open connection, and AWS says so about rotation in general: “open database connections are not dropped.” So change the password first and end the sessions second, since the reverse lets the holder reconnect, and begin with who is connected, because those addresses are your list of consumers. On SQL Server, read the address from the connection, since the host name on a session is whatever the client chose to send. We checked these steps against the PostgreSQL 18, MySQL 8.4 and SQL Server 2025 documentation.
-- Find it: who is connected, in PostgreSQL, then SQL Server and Azure SQL, then MySQL
SELECT pid, client_addr, backend_start FROM pg_stat_activity WHERE usename = 'app_user';
SELECT s.session_id, c.client_net_address FROM sys.dm_exec_sessions AS s
JOIN sys.dm_exec_connections AS c ON c.session_id = s.session_id
WHERE s.login_name = 'app_login';
SHOW PROCESSLIST;
Two logins with equal grants allow a planned flip, but in a leak you change the leaked one now. MySQL’s RETAIN CURRENT PASSWORD keeps the leaked password valid, so save it for planned work, and add DISCARD OLD PASSWORD, since a plain change leaves a secondary password valid. In psql, \password encrypts on the client, whereas a literal ALTER ROLE can reach the server log, and a contained SQL user needs ALTER USER. Every open connection survives, disabling a SQL Server login does not touch them, and Azure SQL Database even reauthorizes a connection at least every 10 hours with the original password. So end them all, which also makes every consumer log in again while you watch.
-- Revoke it, then end every session, in that order
\password app_user -- PostgreSQL, in psql
ALTER LOGIN app_login WITH PASSWORD = '<new>'; -- SQL Server, or master on Azure SQL
ALTER USER 'app_user'@'%' IDENTIFIED BY '<new>'; -- MySQL
ALTER USER 'app_user'@'%' DISCARD OLD PASSWORD;
SELECT pg_terminate_backend(pid, 5000) FROM pg_stat_activity
WHERE usename = 'app_user' AND pid <> pg_backend_pid(); -- PostgreSQL
KILL 53; -- SQL Server and Azure SQL, with the session_id from above
DBCC FLUSHAUTHCACHE; -- Azure SQL Database, in the user database
KILL CONNECTION 42; -- MySQL, with the Id from SHOW PROCESSLIST
A team that ran \password at 16:10 and stopped kept 40 pooled connections that never had to log in again, and a worker held the old password until the nightly restart at 03:05, when it failed and invoices stopped. A queue depth alarm paged the on call engineer at 03:20, and the log held password authentication failed. Terminating the sessions at 16:12 would have shown that failure at the desk, and would have ended any stranger’s session 11 hours sooner. To confirm, log in with the old password, where -W forces the prompt so that a saved password cannot answer, then rerun the find query, which should show only known addresses, and read the error against What a Dead Credential Looks Like.
# Confirm with the old password, typed at the prompt
psql -W "host=db.example.com user=app_user dbname=billing" -c 'SELECT 1'
# Expected, illustrative
# FATAL: password authentication failed for user "app_user"
Signing keys and session secrets
With HS256, an HMAC algorithm, the one secret both signs and verifies, so every verifier is a mint, and the only cure is to stop trusting the key. Name every verifier, gateways and partners included, and match its kid, since one that you miss keeps accepting forged tokens, and decode a token offline to read its alg, iss and exp, never in an online decoder. Generate a new random key and kid, since RFC 8725 rules out a memorable password as a MAC key. Then take the old key out of the JWKS, every verifier and any fallback list now, without the grace period that a planned rotation would allow, and accept that everything it signed fails. On Auth0, rotate first, since only a previously used key can be revoked.
A verifier accepts forged tokens until its JWKS cache refreshes, and Microsoft calls a check every 24 hours reasonable, so force a refresh on each one. Since a forged token needs no login, review sessions and look for what one may have bought that outlives the key, such as an API key or a changed password. To confirm, sign with the old key through your own test suite’s signing helper in a test environment, and show that every verifier answers 401 while the new key passes on the same run. Session secrets follow the same rule, as in this Django 6.1 setting.
# settings.py, Django 6.1
SECRET_KEY = env("SECRET_KEY") # the new value
SECRET_KEY_FALLBACKS = [env("OLD_SECRET")] # planned rotation: the old value still verifies
SECRET_KEY_FALLBACKS = [] # a leak: sessions end, passwords stay
Flask 3.1 has the same setting, but extensions may not honor it. The express-session package verifies with every value in secret, so a leaked value must leave the array, and Rails 8.1 expires sessions when secret_key_base changes. For webhook secrets, GitHub documents no overlap, so plan a short mismatch, while Stripe’s Roll secret allows up to 24 hours.
Any other API key
Most keys have no recipe here, so read the issuer’s page for four answers: can two keys live at once, does revocation take effect at once, can this key create other credentials, and is there a usage log. Where the page is silent, we assume the worse answer and write that down. Match the key by name or prefix in the dashboard without using it, or with one logged cheap read from Is it live, then learn whether it is a live or a test key, who administers the account and what it reaches. To list consumers, search your organization for the variable name, never the value.
Create the replacement with the narrowest scope on offer and an expiry if one exists, deployed from your vault, and if only one key can exist, plan the short outage in Revoke now or rotate with overlap. Read what the revoke button does, since “rotate” may mean a delayed expiry and “disable” may be reversible by anyone with access, and with no way to revoke, open a support case and record the time. Then ask what the key created, such as further keys, webhooks, users or scheduled exports, each of which keeps working without it, and what it changed that nothing puts back, such as a payout account or an allow list. Ask what it read, since data that left stays out and may raise the questions in When a leak becomes an incident, and what tokens or sessions it spawned, and how long they last. Read the audit or request log across the whole window, and where none exists, the record says so.
To confirm, repeat the cheap read once and read the body as well as the status, since a 403 can mean a live key with too little scope. Never test with a call that does something, such as a send, a charge or a publish.
# Confirm: the header goes in on standard input, and only the status comes back
read -rs TOKEN; printf 'Authorization: Bearer %s\n' "$TOKEN" \
| curl -sS -H @- -o /dev/null -w '%{http_code}\n' https://api.example.com/v1/me
unset TOKEN
# Expected, illustrative
# 401
Stripe, Slack and Twilio
The two tables run Stripe, Slack and Twilio through the same five steps, and the What survives column holds the surprises.
| Vendor | Find it and replace it | Revoke it |
|---|---|---|
| Stripe | sk_live_ or rk_live_ reaches real money. GET /v1/balance shows livemode, which tells test from live. API keys, the overflow menu, Rotate key, then the Expiration. Both keys work for up to 7 days, so take the shortest span, and prefer a restricted rk_ key. |
A compromised key gets Now, which deletes it at once. After an overlap, expire it once its request logs fall to zero. |
| Slack | auth.test names the workspace, user and bot. A webhook URL is a secret too, and Slack actively revokes leaked ones. A new token comes through the OAuth flow, or, for a single workspace app, the app management console. For a webhook, add a new URL in the app’s settings. |
auth.revoke, with the token in the header, as in the listing above. For a webhook, remove the URL in the app’s settings. |
| Twilio | An Account SID starts with AC and a key SID with SK, while an Auth Token has no prefix. Twilio says that a compromised Auth Token means a compromised account. Create a secondary Auth Token, deploy it and test it. Request signatures use the primary until promotion, so validators keep the old token until then. |
Promote the secondary, which deletes the old primary, and flip your validators then. DELETE an API key on its Keys resource, with the Auth Token or a Main key. |
| Vendor | What survives | Confirm |
|---|---|---|
| Stripe | What the key did stays done, so open View request logs for that key, in Workbench, and read the window for refunds, payouts and new webhook endpoints. Webhook signing secrets are separate from API keys, so a rotation leaves them alone, and each one needs its own Roll secret if it leaked. | The same call answers 401, and a 403 means a live key that lacks the permission. |
| Slack | Revoking a bot token does not uninstall the app, but it deactivates the bot user and removes its channel memberships, so the bot must be invited back, and what the token already read stays read. Slack does not say what happens to incoming webhooks, so remove a leaked webhook URL in the app’s settings on its own, and never assume that the token took it. | auth.test answers ok false, with invalid_auth or token_revoked. |
| Twilio | Deleting an API key invalidates the Access Tokens made from it, but a promotion leaves every API key alone, so a key that a holder created stays valid. For a taken over account, Twilio says to delete every API key and rotate affected subaccount tokens, which breaks your apps until they are updated. | Fetch /Accounts/{Sid}.json under /2010-04-01 on api.twilio.com with the old token, and expect error 20003. Pass the pair to curl -K - on standard input. |
Note. Never test a webhook, since a POST that works posts a message to the channel.
Package registry tokens
A registry token protects other people’s machines, so the hunt asks what it published as well as who used it, since a version pushed by a stranger runs wherever it is installed. Revoking the token does nothing to a release that it already published, and where a registry offers trusted publishing, we would make that the replacement.
On npm, every classic token was revoked on December 9, 2025, but an old one still gets a hunt for its window. Trusted publishing needs CLI 11.5.1 and Node 22.14.0 or later and, as of November 2026, no self hosted runners, and once it works, set the package to require two-factor authentication and disallow tokens, under Settings, Publishing access. In npm 11, npm token revoke takes the ID from the list or the whole token, and it will not accept the truncated value that the list prints, although an older page says npm token delete. Revocation can take up to an hour, during which the window stays open, and an owner added with the token stays an owner.
A version published with a stolen token stays published, so match every version in the window against your release log. You may unpublish one that is not yours in the first 72 hours if no public package depends on it, and later only with no dependents, under 300 downloads in the last week and a single owner. Otherwise you deprecate it, which warns users but leaves it installable, and publish a clean version under a new number, since a version number can never be used twice. After the hour, whoami should stop returning a username, and since npm documents only the success case, we count anything without one as a failure.
# npm: find it, deprecate a version that is not yours, then confirm after the hour
npm token list
npm view @example-systems/billing-client time --json # every publish, with its time
npm owner ls @example-systems/billing-client # who can publish
npm deprecate @example-systems/billing-client@2.4.1 "Not published by us. Upgrade to 2.4.2."
read -rs TOKEN; printf 'Authorization: Bearer %s\n' "$TOKEN" | curl -sS -H @- \
https://registry.npmjs.org/-/whoami | jq -r '.username // "no username"'; unset TOKEN
On PyPI, a token starts with pypi-, and GitHub and deps.dev report exposed tokens to PyPI, which revokes them, so it may be dead already. Prefer trusted publishing, whose tokens last 15 minutes, or a token for the one project, and remove the old one in your account settings, on PyPI and on TestPyPI if you use both. A release uploaded with the token stays on the index. To yank it, open your project’s releases page, choose Options, then Yank, and give a reason. Yanking applies to a whole release, never to a single file, and installers still pick a yanked release when it is the only match for an == or === pin. If anything unauthorized happened, PyPI says to contact admin@pypi.org, and since we found no documented check that a token is dead, read your security history and keep the record as What closes a finding describes.
On Docker Hub, the token list shows each token’s last use, and Docker Desktop creates up to five tokens on its own, so read those too. Give the replacement the narrowest repository permission and an expiry, since as of November 2026 the default is None, and an expiry cannot be edited later. Then Deactivate the old token, which is reversible, and Delete it after the hunt. Images pushed with the token stay, and a tag may now point at something that you did not build. Compare each tag’s digest, tag_last_pushed and last_updater_username from the Hub API with your build log, and rebuild anything that you cannot match from a clean source. On the route below, Docker’s API reference lists 401 as a failed sign-in, a 200 means that the token is alive, and docker login with the old token should fail with Incorrect authentication credentials.
# Docker Hub: confirm. The short lived token that comes back goes nowhere
read -rs TOKEN; printf '{"identifier":"examplebot","secret":"%s"}' "$TOKEN" | curl -sS \
-H 'Content-Type: application/json' -d @- -o /dev/null -w '%{http_code}\n' \
https://hub.docker.com/v2/auth/token; unset TOKEN
SSH and TLS private keys
For SSH, search authorized_keys and authorized_keys2 on each host, and for TLS, a hash of the public key finds every certificate for it at crt.sh. Replace an SSH key with ssh-keygen -t ed25519, testing a login with it before you remove the old one. To revoke, clear the key line from both files or point RevokedKeys at a list, but sshd refuses all public key logins if it cannot read that file. Pass -u to ssh-keygen -k whenever the list exists, or it writes a new one and every earlier revocation works again. Leave -u out only for the first key, while the file does not exist yet, since -k -u fails on a missing file. Revoke the certificate first, with the reason keycompromise, and delete it, or certbot will try to renew it. Then issue the new one with certbot --new-key, since --reuse-key keeps the old private key.
We would not assume that removing a key closes an open session, so check with who, and compare the whole key file with your known list, since a holder may have planted another line. TLS revocation only protects a client that checks, so a stolen key may impersonate you until the certificate expires. Since Let’s Encrypt ended OCSP on August 6, 2025, and publishes revocation only in CRLs, read its CRL and treat the certificate’s lifetime as the effective limit. To confirm, use the old SSH key alone with -o IdentitiesOnly=yes, or an agent may offer one that works, expect Permission denied (publickey), and check that the live serial matches the new certificate.
# Find, replace and revoke the SSH key, revoke and replace the certificate, then read the serial
openssl pkey -outform DER -in privkey.pem -pubout | openssl sha256
ssh-keygen -t ed25519 -C "deploy billing-api 2026-11" -f deploy_2026_11
# Install deploy_2026_11.pub on each host, and test a login with it before you revoke the old key
sudo ssh-keygen -k -u -f /etc/ssh/revoked_keys leaked.pub # -u keeps earlier revocations
sudo certbot revoke --cert-name example.com --reason keycompromise --delete-after-revoke
sudo certbot certonly --cert-name example.com -d example.com --new-key --force-renewal
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
| openssl x509 -noout -serial
Hunt for abuse
Revoking stops the next misuse and says nothing about the last one, which is why the hunt is a step of its own. Hunt across the whole window sets its rules, so here we read the window in AWS, Azure and GitHub, and separate your own checks from a stranger’s.
The window
The window opens earlier than the first commit if the value came from somewhere else, and it can close well after the revocation. Picture an AWS key set Inactive at 10:00 on a Monday, which may have made a session at 09:59 that runs for 36 hours. We assume that a session outlives its key, so without a deny on older sessions, the window stays open until about 22:00 on the Tuesday. For Example Systems, it opens at commit 7f3c2ab, 11 days before the alert, and closes when key1 is renewed. The logs that cover a window expire on their own clocks, from GitHub’s audit log to Entra sign-ins, and Record and export, in parallel gives each clock.
AWS CloudTrail
A lookup covers one Region and one attribute, so you loop, and it finds only calls signed by that key. Each session has its own ASIA key ID, so read those from the session events and look them up too, and calls to the global STS endpoint land in us-east-1. Run it as your own responder, since the quarantine policy denies cloudtrail:LookupEvents to the key’s user. No loop fixes one blind spot, though: S3 GetObject is a data event, and it is not in Event history, so a key that only read example-billing-exports leaves nothing there.
# Every Region, one lookup each, by the key ID. Run it as your own responder
KEY=AKIAIOSFODNN7EXAMPLE
for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
aws cloudtrail lookup-events --region "$r" --start-time 2026-10-05T00:00:00Z \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue="$KEY" > "ct-$r.json"
done
# When, where, what, from which address, with which client, and whether it worked
jq -r '.Events[].CloudTrailEvent | fromjson | [.eventTime, .awsRegion, .eventName,
.sourceIPAddress, .userAgent, (.errorCode // "ok")] | @tsv' ct-*.json | sort > timeline.tsv
# What normal looks like for this key: count the calls by address and name
cut -f3,4 timeline.tsv | sort | uniq -c | sort -rn | head -20
# The session key IDs that this key created. Look up each of these as well.
jq -r '.Events[].CloudTrailEvent | fromjson
| select(.eventName | test("GetSessionToken|AssumeRole|GetFederationToken"))
| .responseElements.credentials.accessKeyId' ct-*.json | sort -u
In SEC-1209, a different leak from SEC-1204, the leaked AWS key normally appears only from the runner range during a deploy, and the invented rows below come from elsewhere at 02:14.
# Invented rows from SEC-1209, sample output, abridged to time, call, address, client and result
02:14:51Z GetCallerIdentity 203.0.113.77 aws-cli ok
02:15:03Z ListAttachedUserPolicies 203.0.113.77 aws-cli AccessDenied
02:15:09Z ListBuckets 203.0.113.77 aws-cli ok
02:16:20Z CreateUser 203.0.113.77 aws-cli ok
The sequence of calls is what marks these rows as abuse: identity, permissions, inventory, then a way back in, since GetCallerIdentity alone is also your own first call. Delete the user that requestParameters names, along with its access keys, and leave the log record as it is, then look up every ASIA ID.
Azure
The tables behind the queries below exist only if diagnostic settings were sending them to Log Analytics before the incident, as Hunt across the whole window warns, and the queries are ours, so run each one on a known event before you trust an empty result. Without those tables, use the Entra admin center and az monitor activity-log list, and write down what you could not see. Either way, each answer means something only against addresses that you can name, so list those first: for an App Service, the possibleOutboundIpAddresses property of az webapp show gives every outbound address, although on Premium V4 it comes back empty as of November 2026. Add your deploy runners, and everything else is a question.
The queries below read AADServicePrincipalSignInLogs for one credential and StorageBlobLogs for key and SAS traffic. AuthenticationHash names the key that signed, in a format documented for the resource log, so test the key1 or key2 extract before you trust it. The AzureActivity query for who listed or regenerated keys is in Revoke now or rotate with overlap.
// Entra: which addresses used this credential
AADServicePrincipalSignInLogs
| where TimeGenerated > ago(30d) and AppId == "APP_ID"
| where ServicePrincipalCredentialKeyId == "KEY_ID"
| summarize Events = count(), Last = max(TimeGenerated)
by IPAddress, ResourceDisplayName, ResultType
// Storage: who signed with a key or a SAS, by address. Also the first look in Revoke now or rotate with overlap.
// 30 days covers the 11 since the first commit, so widen it if START is older.
// CallerIpAddress carries the port, so split it.
StorageBlobLogs
| where TimeGenerated > ago(30d) and AccountName == "stexamplebilling"
| where AuthenticationType in ("AccountKey", "SAS")
| extend Ip = tostring(split(CallerIpAddress, ":")[0]),
KeyUsed = extract(@"^(key[12])\(", 1, AuthenticationHash)
| summarize Ops = count(), Bytes = sum(ResponseBodySize), Last = max(TimeGenerated)
by Ip, KeyUsed, AuthenticationType, OperationName
The rows below are invented to show a finding, and Example Systems found nothing like them.
# Invented rows, sample output, abridged to address, key, operation, count and bytes. StorageBlobLogs
Ip KeyUsed OperationName Ops Bytes
198.51.100.24 key1 GetBlob 9120 2147483648
203.0.113.77 key1 ListBlobs 3 18234
203.0.113.77 key1 GetBlob 1846 5368709120
# AzureActivity
TimeGenerated Caller CallerIpAddress OperationNameValue
2026-10-09T02:20:11Z 0000... 203.0.113.77 .../listKeys/action
# AADServicePrincipalSignInLogs
IPAddress Resource ResultType Events Last
203.0.113.77 Graph 0 14 2026-10-09T03:02:11Z
203.0.113.77 Graph 7000215 6 2026-11-03T16:41:09Z
The first storage row is the app and the other two are not, and a listing followed by many reads from one address is someone walking a container. A listKeys from your deploy identity is routine, while one from an address that you cannot name means that the caller may hold both keys, so key2 stops being a clean slot, and Microsoft documents Caller as a GUID to resolve in Entra. In the sign-ins, bad secret failures after the secret was deleted mean that someone still holds the old value, but Microsoft describes ResultType only as success or failure, so read one known failure first.
On findings like these, renew both keys and take the outage, end the user delegation keys as well, and treat Bytes as an estimate of data that left, which brings in the questions for counsel in When a leak becomes an incident.
GitHub audit log
On Enterprise Cloud, as of November 2026, an organization owner can search by token hash, which keeps the raw token out of the query, and the token that runs the search needs read:audit_log. Git events do not come back from a search, so you pull them and filter them yourself. A fine-grained token has an empty hash in the credentials export, so join on token_id. On other plans, owners have the web audit log, with no Git events, and GET /repos/ counts clones for 14 days.
# Enterprise Cloud, as an organization owner. The hash keeps the token out of the query
read -rs TOKEN; printf %s "$TOKEN" | openssl dgst -sha256 -binary | base64; unset TOKEN
gh api --method GET /orgs/example-systems/audit-log --paginate \
-f phrase='hashed_token:"HASH_VALUE"' -f include=web -F per_page=100
# Git events, 7 days at most. An address appears only if the enterprise turned on IP disclosure.
gh api --method GET /orgs/example-systems/audit-log --paginate -f include=git -F per_page=100 \
--jq '.[] | select(.repo == "example-systems/billing-api")
| [(.created_at / 1000 | todate), .action, .actor, .actor_ip] | @tsv'
In the invented rows below, svc-reporting normally fetches from one host, and it has never cloned this repository. The git.clone row comes from the Git events pull, while hook.create is a web event, which that pull never returns, so it comes from the hash search.
# Invented rows, sample output
2026-11-01T22:41:07Z git.clone svc-reporting 203.0.113.77
2026-11-01T22:43:30Z hook.create svc-reporting 203.0.113.77
A clone means that a copy has left, and the webhook is a way back in, so read its URL, delete it, and rotate anything that the clone held. Your own runners write the same git.clone rows, so judge each row by its actor and address, since the action looks the same either way.
What abuse looks like
Each log shows abuse in its own terms, so the table gives the signal to look for in each, and what it means.
| Signal | Where | What it means |
|---|---|---|
GetCallerIdentity from an unknown address, then CreateAccessKey, CreateUser or compute that you did not launch |
CloudTrail | Orca saw it as the top call on a leaked key, at 25 percent, and Unit 42 counted 474 miners in one campaign. |
| A new secret on an app, or odd location, failures or timing | Entra audit, sign-ins | The checks in Microsoft’s playbook. |
listKeys from a stranger, or large reads from one address |
AzureActivity, StorageBlobLogs |
Both keys may be held, and data may be leaving. |
git.clone by an unknown actor, or public_key.create, hook.create, repo.add_member |
GitHub audit log | A copy left, or a way back in was added. Remove these by hand. |
Your checks or theirs
Every check that you run against a live key lands in the same logs as an attacker’s first move, and so does a TruffleHog scan or a GitHub validity check. So write each check in the incident log before you run it, with the name, the UTC time, the address and the call, and run them all from one known host, which turns the note in Is it live into a rule. A row that nothing explains counts as someone else’s, as the 02:14 row in the table below shows.
AWS says that it logs authenticated requests, so a test with a dead key may leave no row at all, and we would not count on either result. The listing below marks your own calls with an application ID, then prints every row of the timeline from AWS CloudTrail that no check of yours explains.
# Mark your own calls. AWS CLI v2 adds this ID to its user agent, which should reach userAgent
# in CloudTrail, so try it once on a harmless call first.
export AWS_SDK_UA_APP_ID=ir-sec-1209
# ours.tsv holds one line per check: the address, a tab, and the UTC minute, 2026-11-03T13:51.
# Print every row in the timeline that no check of yours explains.
awk -F'\t' 'NR==FNR {ours[$1 FS $2]; next} !(($4 FS substr($1,1,16)) in ours)' \
ours.tsv timeline.tsv
| Incident log | Log row | Verdict |
|---|---|---|
| 13:51, R. Okafor, 198.51.100.7 | 13:51:22Z, 198.51.100.7, GetCallerIdentity |
Ours. Minute, address and call all match. |
| No entry | 02:14:51Z, 203.0.113.77, GetCallerIdentity |
Not ours. The same call proves nothing on its own. |
At Example Systems, triage matched the value without using it, so the team’s own row is the negative test, logged in SEC-1187 with its time and address. Generally, the fewer calls you make with a live key, the less of the hunt you spend explaining yourself.
Clean up Git history
Why a rewrite is mostly hygiene
Once a commit is pushed, you should assume that it was copied. GitHub’s own page on removing sensitive data says that revoking “may be sufficient to solve your problem”, while Truffle Security says that “the only way to securely remediate a leaked key on a public GitHub repository is through key rotation.” Generally, then, a rewrite is hygiene, which earns its cost only where you can reach every holder of that history.
Beyond the places that History comes last lists, CI artifacts and mirrors keep a copy too, and Copies a rewrite cannot reach says who can clear each one. As for the attention that a force push may draw, a guest post on Truffle Security’s blog in mid 2025 scanned GH Archive, a public archive of GitHub’s event feed, for force pushes that carried zero commits, back to 2020, and reported thousands of active secrets and about $25,000 in bounties. An old clone compared with your new history also points straight at what you removed, and GitHub says that “Git data from any repository in a network may be accessed from any repository in the same network.”
When to rewrite and when not
The cost of a rewrite lands all at once, since hashes change and the whole team clones again, so we ask first who it protects. Under a legal hold, the questions in When a leak becomes an incident come before any push.
| Situation | Rewrite? | Why |
|---|---|---|
| Revoked, in a public repository | Usually no | Treat it as published, and close the alert as revoked. |
| Revoked, private, with a known audience | A limited rewrite | Replace the one string, without a Support ticket, since you can reach every clone. |
| Data that cannot be rotated, such as personal data | Yes, with Support | GitHub Support helps only where rotation cannot mitigate the risk. |
| A legal hold or an open investigation | Ask first | The push is irreversible, so take it to counsel. |
A rewrite with git filter-repo
If you have never used git filter-repo, three facts will help. It rewrites a local clone and touches nothing on the server until you push. It changes every commit from the first one that held the value onward, so all of those hashes change. The push is the one step that cannot be undone, so every check comes before it. Run from an old clone, the tool stops with “this does not look like a fresh clone”, and the answer is a new clone, not --force. BFG Repo-Cleaner, if you know it instead, is legacy as of November 2026, since its last release was January 2025, and GitHub’s page no longer mentions it.
First, get ready. Revoke, finish the hunt, and record the SHAs that you need, such as the old SHA for the CI gate. Send the first of the two messages in Afterward, everyone clones again, and merge or close open pull requests. Install version 2.47 or later, which --sensitive-data-removal needs, and since --version prints a hash and not a number, test that --help lists the option. Then make a fresh clone, since the option fetches all refs and may discard local work.
# First, get ready: install, check the version, then make a fresh clone
brew install git-filter-repo
git-filter-repo --help | grep -c -- --sensitive-data-removal # 1 or more
git clone https://github.com/example-systems/billing-api && cd billing-api
Second, list the values, each on its own line in ../passwords.txt, outside the repository. Every line is a literal search string, so leave out comments, list the value and not the whole setting, and expect each match to become ***REMOVED***. Leave out the ==> form that sets another replacement, since the checks in the fourth and sixth steps search for each whole line, and such a line never matches the value. Since a rewrite cannot remove a string that it is not given, this procedure searches on the value itself, which In history, not in the tree otherwise rules out. Even so, the key is already revoked, the file keeps every value out of your shell history, each check prints a count and never a matching line, and the sixth step ends by deleting the file.
Third, run it, and keep the First Changed Commit that the output names, since a Support request needs it. To drop a whole file instead, use --invert-paths --path with its Git path, and name every old path if it moved. Text replacement skips binary files, and --replace-message covers commit and tag messages.
# Third, run it in the fresh clone
git-filter-repo --sensitive-data-removal --replace-text ../passwords.txt \
--replace-message ../passwords.txt
# Expected, abridged and illustrative
# NOTE: First Changed Commit(s) is/are:
# 7f3c2ab...
Fourth, check before you push. The search must print 0, and keep -m, because without it git log shows no diff for a merge or a stash, so a value that lives only there is missed. If the count of changed pull request refs looks wrong, discard this clone, since the next step is irreversible.
# Fourth, check before you push
git log --all -m -p | grep -c -F -f ../passwords.txt # expect 0
grep -c '^refs/pull/.*/head$' .git/filter-repo/changed-refs
Fifth, lift branch protection, push, and restore it. Failures on refs/pull/ are expected, since GitHub marks them read-only, and any other failure means that protection is still on.
# Fifth, the one step that cannot be undone
git push --force --mirror origin
Sixth, verify from outside, since your clone proves nothing about the server. Clone again and search there, and expect git cat-file on the old SHA to fail, although the old SHA may still open on GitHub. The scan adds unverified, since a revoked key no longer verifies, and jq prints only the detector, whether it verified and the file, since a plain run prints any value that it finds. Then delete ../passwords.txt and send the second message.
# Sixth, verify from outside, in a new clone
cd .. && git clone https://github.com/example-systems/billing-api check && cd check
git log --all -m -p | grep -c -F -f ../passwords.txt # expect 0
git cat-file -t OLD_SHA # expect a fatal error
trufflehog git file://. --results=verified,unknown,unverified --no-update --json \
| jq -c '{DetectorName, Verified, file: .SourceMetadata.Data.Git.file}' # expect no lines
Copies a rewrite cannot reach
Each place below keeps its copy until someone acts on it, whether that is you, GitHub Support, a fork owner or another host.
| Where | What stays | What to do |
|---|---|---|
| Pull request refs, cached views | Read-only refs, and commits stay reachable, in GitHub’s words, “Directly via their SHA-1 hashes in cached views”. | Only GitHub Support can clear them, and only where rotation cannot mitigate the risk, so for a key that you can revoke, it may decline. Send the count and the First Changed Commit. |
| Forks | Each keeps the data, and GitHub cannot give you owner contacts. | Ask each owner to delete the fork. |
| Comments and logs | Anyone with read access can see edit history. | Use Delete revision from history, and delete the run’s logs by API. |
| Azure Repos | No documented purge, and the object still comes down on clone. | Rotate. Full removal means a new repository. |
| GitLab | GitLab says that git filter-repo purge is not suitable for secrets. |
Use Remove blobs and Redact text, from 17.9, then housekeeping and Prune unreachable objects. |
A fork keeps its own copy of history, so ask its owner to delete it once the key is dead, and the command that deletes a run’s logs is in Masking hides log text and nothing more. A value in a file kept in Git LFS needs one more step, since GitHub says that once you remove such a file, “the Git LFS objects still exist on the remote storage”, so on GitHub you delete and re-create the repository or ask GitHub Support. Generally, the push cleans the branches and tags, and every other copy needs a step of its own.
Afterward, everyone clones again
Every old clone still holds the old history, so one pull and push from it brings the secret back, and one merge can restore what the rewrite removed, which is why the message to the team matters as much as the command. Hashes change and links to an old commit break, signatures vanish, and closed pull requests may lose their diffs, so we send two messages, one before the rewrite and one after it.
# Two messages to everyone with a clone
Before: SEC-1187. We will rewrite billing-api history at 17:00 today to remove a revoked key.
By 16:30, merge or close your pull requests. From 16:30 until the all clear, push nothing.
After: SEC-1187. The rewrite is done. Do not pull, fetch, merge or push from an old clone.
Save unpushed work as below, delete the clone, clone again and rebase. Never merge an old
branch. Anything pinned to an old hash needs the new one, and signatures are gone.
A colleague with unpushed work saves it as patches before fetching anything, deletes the clone, clones again and applies the patches with git am. A CI gate, run on a full clone, then fails if the old history is back in the branch.
# Each colleague with unpushed work, before fetching anything
git format-patch origin/main..my-branch -o ../saved
cd .. && rm -rf billing-api && git clone https://github.com/example-systems/billing-api
cd billing-api && git switch -c my-branch origin/main && git am ../saved/*.patch
# A CI gate, on a full clone: fail if the old history is back in this branch
if git merge-base --is-ancestor OLD_SHA HEAD 2>/dev/null; then exit 1; fi
At Example Systems, the 31 people with access got both messages once the hunt was done, and the CI gate went in. GitHub cannot give a fork owner’s contact, but the contractor is known, so the request went through the contract, and the record notes when it was sent. From a fresh clone, the search in the sixth step printed 0 and the scan printed no lines, and both outputs went into SEC-1187 with the time of the push. Generally, a rewrite is finished only when CI would catch the old history coming back.
Replace it with something better
The ladder
Revoking a leaked credential ends the incident but leaves the cause in place, because a static secret must be stored, delivered and remembered, and each of those steps makes another copy. A rotated key can leak the same way as the one it replaced, so the lasting fix is to change the kind of credential that a service uses, and where possible to remove the stored secret altogether. To get there, we climb a ladder and stand on the highest rung that the other side of the connection will accept. Ask in order whether the other side accepts an identity, whether it can issue something short lived, and whether the value can live in a vault that you audit.
| Rung | Kind | Examples |
|---|---|---|
| 1 | No secret at all | Managed identity, OIDC federation |
| 2 | Short lived, issued on demand | Vault dynamic credentials, STS, user delegation SAS |
| 3 | Vault held, read at run time, rotated | Key Vault, Secrets Manager |
| 4 | Encrypted in the repository | SOPS |
| 5 | Platform store, narrowly scoped | CI secret, app setting |
Note. Below rung 5 sits a secret hard coded in source, which is never acceptable.
In our example, that test puts app-billing-prod and the deploy job on rung 1, since Azure Storage, Azure SQL and Azure itself all accept an identity. The table below takes a credential and a platform and names the rung, and rung 5 appears only where nothing better exists.
| Credential and platform | Rung | Move it to |
|---|---|---|
| Azure Storage key, Azure SQL password. App Service, Functions, Container Apps | 1 | Managed identity with a data role |
| Azure client secret. GitHub Actions, Azure Pipelines | 1 | Workload identity federation |
AWS access key such as ci-deploy. GitHub Actions |
1 | IAM role with an OIDC trust |
| Production value on a laptop. Developer machine | 1 | Your own identity, development vault |
| Link signed with an account key. Azure Storage | 2 | User delegation SAS |
| RDS password. RDS for MariaDB, MySQL, PostgreSQL | 2 | IAM database authentication |
| Password with no identity option. Anything that can reach Vault | 2 | Vault or OpenBao dynamic credentials |
| Password that must stay static. AWS, or a Pod on AKS | 3 | Secrets Manager, or Key Vault by CSI |
| Vendor API key, such as Stripe. Any server, or a browser bundle | 3 | Key Vault, behind a backend for frontend |
| Secret set by infrastructure code. Terraform, OpenTofu | 3 | Vault, with a write only argument |
| GitOps values. Git | 4 | SOPS |
| Registry token in a build. Docker on CI | 5 | Narrow CI secret, as a build secret |
Key Vault, set up properly
Azure Key Vault is rung 3, for the credentials that cannot climb any higher. Something still has to sign in to a vault, which is the “secret zero” problem, and on Azure a managed identity answers it. The vault’s own defaults are weaker than many teams assume. As of November 2026, a new vault has its firewall and purge protection off, and it records nothing about who read a secret until a diagnostic setting exists. We follow Microsoft’s advice of one vault per app, per environment and per region, with roles at vault scope.
# One vault for billing-api in production, audited before its first secret
RG=rg-billing-prod
# RBAC is the default as of November 2026, and the flag says so. Purge protection is final.
# The new default comes with Azure CLI 2.90.0, and existing vaults keep their model.
az keyvault create -n kv-example-prod -g $RG --enable-rbac-authorization true \
--enable-purge-protection true --retention-days 90
VAULT_ID=$(az keyvault show -n kv-example-prod --query id -o tsv)
# Audit first, or a read leaves no record. WORKSPACE_ID is your Log Analytics workspace.
az monitor diagnostic-settings create --name kv-audit --resource "$VAULT_ID" \
--workspace "$WORKSPACE_ID" --export-to-resource-specific true \
--logs '[{"category":"AuditEvent","enabled":true}]'
New vaults default to Azure RBAC from control plane API 2026-02-01, which needs Azure CLI 2.90.0, and older API versions retire on February 27, 2027. A client secret in an app setting, used to sign in to the vault, only moves the leak one step back, so our setup gives the app its own system assigned identity and a role at vault scope by ID, never one at subscription scope like the first listing below.
# The pattern to avoid, a role as broad as a shared env file
# One role at subscription scope: every app reads every vault, every secret
az role assignment create --role "Key Vault Secrets User" --assignee APP_ID \
--scope /subscriptions/SUB_ID
# The app's identity and role, then the first secret. Needs RG and VAULT_ID from the az keyvault create listing above
# The app gets Key Vault Secrets User, by role ID, as Microsoft recommends in scripts.
APP_OID=$(az webapp identity assign -g $RG -n app-billing-prod --query principalId -o tsv)
az role assignment create --role 4633458b-17de-408a-b874-0445c86b69e6 --scope "$VAULT_ID" \
--assignee-object-id "$APP_OID" --assignee-principal-type ServicePrincipal
# You need a data role too, which may take minutes. Key Vault Contributor cannot read secrets.
az role assignment create --role "Key Vault Secrets Officer" --assignee you@example.com \
--scope "$VAULT_ID"
# --file keeps the value out of history, printf adds no newline, -o none keeps it off screen.
( umask 077; read -rs V; printf %s "$V" > ./stripe.txt ) # paste the key, then Enter
az keyvault secret set --vault-name kv-example-prod -n stripe-api-key --file ./stripe.txt \
--expires 2027-02-01T00:00:00Z -o none && rm ./stripe.txt
# Once the set has worked, drop the role. No person keeps a data role on the production vault.
az role assignment delete --role "Key Vault Secrets Officer" --assignee you@example.com \
--scope "$VAULT_ID"
For an existing vault, start by listing who can read today. Under the older access policy model, a Contributor “can grant themselves data plane access”, and moving a vault to RBAC voids every access policy at once, so assign roles first.
# Who can read the vault today, inherited roles included. Needs VAULT_ID
az role assignment list --scope "$VAULT_ID" --include-inherited -o table
Key Vault does not rotate a secret. Its keys have rotation policies and its certificates renew, but only your code changes a secret, and deleting one leaves the credential live at the issuer. The --expires date stops nothing, because a get still returns an expired secret. The date still works as a reminder, since the vault raises SecretNearExpiry 30 days before it. The control that does stop reads is the enabled attribute, set to false on a version that you know is bad. If a broad role can read everything, the vault protects little more than a shared env file would.
Microsoft calls expiry “for informational purposes only”, so set exp and watch for that event, and the built in policy “Key Vault secrets should have an expiration date” can require one on new secrets, though Deny leaves existing ones alone. A previous version cannot be deleted alone, so disable a bad one with az keyvault secret set-attributes --enabled false. Under soft delete, a deleted name stays taken, which breaks delete and recreate scripts, and recovering a deleted vault restores neither its role assignments nor its Event Grid subscriptions. Once read, the value is plain text in your process, and a vault moves the question from where a secret sits to who can read it and how they prove it.
Close the vault’s network
Close the network last, in the order below, because with public access off your own az keyvault secret set fails from outside, and Microsoft says to finish the private endpoint first. The app then joins the virtual network, and on a Linux plan it must send all of its traffic through it. A 403 that names a client address as not authorized is the firewall, not a missing role. Trusted Microsoft services still get in, though Azure DevOps is not one of them, and the vault name still resolves in public DNS, by design.
# Last, close the network. Needs RG and VAULT_ID from the listings above
az network private-dns zone create -g $RG -n privatelink.vaultcore.azure.net
az network private-dns link vnet create -g $RG -n kv-dns --registration-enabled false \
--zone-name privatelink.vaultcore.azure.net --virtual-network vnet-billing-prod
az network private-endpoint create -g $RG -n pe-kv --vnet-name vnet-billing-prod \
--subnet snet-endpoints --connection-name kv-link --group-ids vault \
--private-connection-resource-id "$VAULT_ID"
az network private-endpoint dns-zone-group create -g $RG -n default --endpoint-name pe-kv \
--zone-name vault --private-dns-zone privatelink.vaultcore.azure.net
az webapp vnet-integration add -g $RG -n app-billing-prod --vnet vnet-billing-prod \
--subnet snet-app
az webapp config set -g $RG -n app-billing-prod \
--generic-configurations '{"vnetRouteAllEnabled": true}'
az keyvault update -g $RG -n kv-example-prod --public-network-access Disabled
Reading from Key Vault
On App Service and Azure Functions, a “Key Vault reference” is an app setting that points at a secret, which the platform resolves with the app’s managed identity, so your code sees ordinary configuration. Mark these as slot settings, since each environment has its own vault, and on Container Apps, a keyvaultref: secret with a versionless URI picks up the latest version within 30 minutes.
# Needs RG from Key Vault, set up properly. Leave the version out, so that rotation reaches the app.
az webapp config appsettings set -g $RG -n app-billing-prod --slot-settings \
"Stripe__ApiKey=@Microsoft.KeyVault(VaultName=kv-example-prod;SecretName=stripe-api-key)"
# After a rotation, force a refetch rather than wait out the 24 hour cycle
SITE_ID=$(az webapp show -g $RG -n app-billing-prod --query id -o tsv)
REFRESH="$SITE_ID/config/configreferences/appsettings/refresh"
az rest --method post --url "https://management.azure.com$REFRESH?api-version=2022-03-01"
App Service refetches a reference every 24 hours, so we can infer that revoking the old value too soon may break the app for up to a day, unless you force the refresh above. A reference that fails passes its literal @Microsoft.KeyVault(...) text to your code as if it were the key, so test Stripe:ApiKey for that prefix at startup and refuse to run. One that works is still an environment variable, as Environment variables at run time explains.
Elsewhere, read through SecretClient with ManagedIdentityCredential, and avoid the DefaultAzureCredential that Managed identity warns about, since a deterministic credential retries, up to five times by default, then fails out loud. From Azure Identity 1.14.0 in .NET, 1.23.0 in Python and 4.10.0 in JavaScript, AZURE_TOKEN_CREDENTIALS restricts the chain.
# Python, azure-keyvault-secrets 4.12.0
from azure.identity import ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient
client = SecretClient("https://kv-example-prod.vault.azure.net", ManagedIdentityCredential())
stripe_key = client.get_secret("stripe-api-key").value
// JavaScript, @azure/keyvault-secrets 4.12.0
import { ManagedIdentityCredential } from "@azure/identity";
import { SecretClient } from "@azure/keyvault-secrets";
const client = new SecretClient("https://kv-example-prod.vault.azure.net",
new ManagedIdentityCredential());
const { value: stripeKey } = await client.getSecret("stripe-api-key");
A vault allows 4,000 secret reads in 10 seconds, per vault and per region, then returns HTTP 429, and Microsoft says that its limits “are not designed to support such high throughput scenarios”. So you cache in memory and re-read “only when the cached copy stops working”. The SDK already retries a 429, so we set it to Microsoft’s steps of 1, 2, 4, 8 and 16 seconds and do not write a second loop. Azure.Core doubles each Delay with up to 20 percent jitter, and a longer Retry-After wins. Microsoft’s sample starts at 2 seconds, and its written steps at 1. The class below adds what the SDK cannot know: when a value has stopped working, and how often to ask again.
// SecretCache.cs and its wiring
// Needs Azure.Security.KeyVault.Secrets 4.12.0, Azure.Identity 1.21.0 and .NET 8 or later.
public sealed class SecretCache(SecretClient client, TimeSpan rereadFloor) // one per process
{
private sealed record Entry(string Value, DateTimeOffset ReadAt);
private readonly ConcurrentDictionary<string, Entry> _entries = new();
private readonly SemaphoreSlim _gate = new(1, 1);
public Task<string> GetAsync(string name, CancellationToken ct = default) =>
_entries.TryGetValue(name, out var hit)
? Task.FromResult(hit.Value) : ReadAsync(name, null, ct); // memory first, no lock
// Call when the other side refuses the value with 401 or 403. Pass the value that failed.
public Task<string> RefreshAsync(string name, string failed, CancellationToken ct = default)
=> ReadAsync(name, failed, ct);
private async Task<string> ReadAsync(string name, string? failed, CancellationToken ct)
{
await _gate.WaitAsync(ct); // a cold start costs one read, not one per request
try
{
if (_entries.TryGetValue(name, out var hit) && (hit.Value != failed
|| DateTimeOffset.UtcNow - hit.ReadAt < rereadFloor)) return hit.Value; // floor
KeyVaultSecret secret = await client.GetSecretAsync(name, null, ct);
_entries[name] = new Entry(secret.Value, DateTimeOffset.UtcNow);
return secret.Value;
}
finally { _gate.Release(); }
}
}
// Program.cs. The SDK does the backoff, about 31 seconds in all. The floor is 30 seconds.
var client = new SecretClient(new Uri("https://kv-example-prod.vault.azure.net"),
new ManagedIdentityCredential(ManagedIdentityId.SystemAssigned),
new SecretClientOptions { Retry = { Mode = RetryMode.Exponential, MaxRetries = 5,
Delay = TimeSpan.FromSeconds(1), MaxDelay = TimeSpan.FromSeconds(16) } });
builder.Services.AddSingleton(new SecretCache(client, TimeSpan.FromSeconds(30)));
// At the call site, try once more with a fresh value when the vendor refuses the cached one.
string key = await secrets.GetAsync("stripe-api-key", ct);
var response = await SendAsync(key, ct);
if (response.StatusCode is HttpStatusCode.Unauthorized or HttpStatusCode.Forbidden)
response = await SendAsync(await secrets.RefreshAsync("stripe-api-key", key, ct), ct);
In our example, billing-api takes 900 requests per second across 6 instances, so a read per request would be 9,000 reads in 10 seconds against a limit of 4,000. We ran a model of a cache like this one for 60 seconds against a fake vault that enforces the limit, and nothing touched Azure. Line D is an upper bound, since the model leaves out the lock and the SDK backoff, which hold each instance to one read at a time. In lines D and E the vendor refuses a key that the vault still holds, so requests fail either way. Without a “floor”, a minimum wait between re-reads, each failure may re-read, and every app that shares the vault may start to see 429, so a single dead key may throttle the vault for every app that uses it.
# Model output, 60 seconds against a fake vault. Nothing touched Azure
ok failed reads in each 10 s window
A read per request 24000 30000 4000 and 5000 x 429, every window
B cache, no change 54000 0 6 0 0 0 0 0
C rotated at 30 s 54000 0 6 0 0 6 0 0
D dead, no floor 27000 27000 6 0 0 then 4000 and 5000 x 429
E dead, 30 s floor 27000 27000 6 0 0 6 0 0
Rotation with Event Grid, and audit logs
Event Grid tells your own function when to change a secret: the vault publishes SecretNearExpiry 30 days before exp, for the current version only, so we can infer that a secret with no expiry gives it nothing to fire on. Microsoft’s tutorials open by preferring a managed identity, so treat rotation as the fallback, and port the single credential sample first, since it targets the .NET in-process model, whose support ends on November 10, 2026.
# Needs RG and VAULT_ID from Key Vault, set up properly. The event carries no value.
FUNC_ID=$(az functionapp show -g $RG -n func-rotate-prod --query id -o tsv)
az eventgrid event-subscription create -n rotate-near-expiry --source-resource-id "$VAULT_ID" \
--endpoint "$FUNC_ID/functions/RotateSecret" --endpoint-type azurefunction \
--included-event-types Microsoft.KeyVault.SecretNearExpiry
The choice that matters is how many credentials stay alive at once. With one credential, such as a SQL login, the function writes a new version and then updates the database, and Microsoft states the gap plainly: during that window, the secret in the vault cannot authenticate. With two, such as storage keys, it regenerates the key that is not in the latest version, so that readers keep a working key for a full cycle, which we would take wherever two slots exist.
Event Grid retries a failed step for two hours, and after that a broken rotation is silent, so send SecretExpired to a person. With the diagnostic setting from Key Vault, set up properly, each read reaches AZKVAuditLogs as a SecretGet, usually within 3 to 10 minutes, so an empty result for the last few minutes proves nothing. Behind a locked vault, a 403 from the app’s public IP followed by a success from its private IP is by design. In our example, the app is the only expected reader, so keep the second query below as an alert.
// Every SecretGet in 30 days, then an alert on any reader that is not the app
AZKVAuditLogs
| where OperationName == "SecretGet" and TimeGenerated > ago(30d)
| project TimeGenerated, Id, Identity, CallerIpAddress, ResultSignature, HttpStatusCode
// Alert: a reader that is not the app. Paste the app's principal ID, APP_OID from Key Vault, set up properly.
AZKVAuditLogs
| where OperationName == "SecretGet" and TimeGenerated > ago(1h)
| where tostring(Identity) !contains "<app-principal-id>"
| summarize Reads = count() by Caller = tostring(Identity), CallerIpAddress
// Without the resource specific flag, rows land in AzureDiagnostics, under ResourceProvider
// MICROSOFT.KEYVAULT, and its claim column names vary, so an empty result proves nothing.
Environment variables at run time
Environment variables are better than source code, and the case for them goes no further than that. The Twelve Factor App recommends them to keep credentials out of code, and that advice still stands, but it says nothing of what happens next: every child process inherits them, and they turn up in crash reports, debug endpoints and /proc. On Linux, /proc/PID/environ shows the environment from exec time to any process that passes a ptrace check, and unsetting a variable later does not change that file.
Google warns about “debug endpoints” and dependencies that log the process environment, and Sentry’s Python SDK sends “the names and values of local variables” with an unhandled error, so a secret read into a local goes too. Set include_local_variables= in sentry_sdk.init(). OWASP’s verdict: “Using environment variables is therefore not recommended unless the other methods are not possible.” A Node.js diagnostic report keeps the environment unless you pass --report-exclude-env, from v22.13.0 and v23.3.0.
# A Node.js diagnostic report keeps the environment by default, with sample output
$ DB_PASSWORD=EXAMPLE-not-real node --report-uncaught-exception app.js
$ grep -c DB_PASSWORD report.*.json
1
# The same crash, with the flag from v22.13.0 and v23.3.0
$ DB_PASSWORD=EXAMPLE-not-real node --report-uncaught-exception --report-exclude-env app.js
$ grep -c DB_PASSWORD report.*.json
0
Generally, fetching through identity at run time comes first, a tmpfs file with mode 0400 comes second, and a variable comes last. Lifetime matters more than mechanism: AWS’s own sample puts a database token in PGPASSWORD, which is tolerable because it expires in 15 minutes.
Managed identity
The app reaches storage with a “managed identity”, which gives an Azure resource its own identity in Microsoft Entra, so that no password or key exists to leak. It does not limit the app itself, so the scope of each role is what limits the damage. In our example, app-billing-prod gets Storage Blob Data Contributor on the invoices container alone, plus Storage Blob Delegator at account scope, since a user delegation SAS is signed with a key requested there. In code, use ManagedIdentityCredential. Microsoft warns that with DefaultAzureCredential, a failed managed identity on a host where someone ran az login falls through, silently, to that person’s account.
A user assigned identity is its own resource, useful when a permission must exist first, but sharing one widens the blast radius. Storage takes a data role, because Owner or Contributor gives no blob access through Microsoft Entra ID, although either can list the account keys and reach the blobs with them while shared key access is on. A new role may take up to 10 minutes by Microsoft’s blob pages, or up to 30 in one note, so a 403 straight after may only mean that it has not arrived.
# Needs RG and APP_OID from Key Vault, set up properly.
ACCOUNT_ID=$(az storage account show -g $RG -n stexamplebilling --query id -o tsv)
az role assignment create --role "Storage Blob Data Contributor" \
--assignee-object-id "$APP_OID" --assignee-principal-type ServicePrincipal \
--scope "$ACCOUNT_ID/blobServices/default/containers/invoices"
az role assignment create --role "Storage Blob Delegator" --scope "$ACCOUNT_ID" \
--assignee-object-id "$APP_OID" --assignee-principal-type ServicePrincipal
// Before: a storage client and a 15 minute service SAS, both signed with the account key
var client = new BlobServiceClient(config["STORAGE_CONN"]); // the string holds the key
var blob = client.GetBlobContainerClient("invoices").GetBlobClient(name);
Uri link = blob.GenerateSasUri(BlobSasPermissions.Read, DateTimeOffset.UtcNow.AddMinutes(15));
// After: the same two jobs with no key, as a user delegation SAS. Azure.Storage.Blobs 12.29.2
var client = new BlobServiceClient(new Uri("https://stexamplebilling.blob.core.windows.net"),
new ManagedIdentityCredential(ManagedIdentityId.SystemAssigned));
var blob = client.GetBlobContainerClient("invoices").GetBlobClient(name);
UserDelegationKey key = await client.GetUserDelegationKeyAsync( // lasts up to seven days
null, DateTimeOffset.UtcNow.AddHours(1), default); // null starts it now, so reuse it
var sas = new BlobSasBuilder { BlobContainerName = "invoices", BlobName = name, Resource = "b",
ExpiresOn = DateTimeOffset.UtcNow.AddMinutes(15) };
sas.SetPermissions(BlobSasPermissions.Read);
Uri link = new BlobUriBuilder(blob.Uri)
{ Sas = sas.ToSasQueryParameters(key, client.AccountName) }.ToUri();
Turning off shared key access then closes more than the key. Setting it to false also denies every service SAS and account SAS, so the invoice links move to user delegation SAS first. Measure with StorageBlobLogs before the switch, because a forgotten client becomes an outage. The first look query in Revoke now or rotate with overlap does that for blob traffic, and the portal is one more client to plan for, since it uses Shared Key for file shares by default. In our example, key2 stayed in use only until the managed identity replaced it, so with no key traffic left, turning off shared key access retires both keys. The setting does not delete a leaked key, so if you may ever turn it back on, rotate the key as well.
# Needs RG. Run the negative test first, since after this every key returns the same 403.
# Once every client has moved, the account refuses keys
az storage account update -n stexamplebilling -g $RG --allow-shared-key-access false
az webapp config appsettings delete -g $RG -n app-billing-prod --setting-names STORAGE_CONN
Azure SQL without a password
Azure SQL moves in three steps, and the last one ends every SQL password at once. A Microsoft Entra admin creates a database user for the identity, the connection string drops its password so that the driver fetches a token, and only then do you turn on Entra only mode.
-- In the billing database, as a Microsoft Entra admin. Add db_ddladmin only for migrations.
CREATE USER [app-billing-prod] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [app-billing-prod];
ALTER ROLE db_datawriter ADD MEMBER [app-billing-prod];
# Before: a SQL connection string, shown on two lines
Server=tcp:sql-example-prod.database.windows.net,1433;Database=billing;User ID=billing_app;
Password=EXAMPLE-not-a-real-password;Encrypt=True;TrustServerCertificate=False;
# After: Microsoft.Data.SqlClient. A user assigned identity adds User ID=<client-id>;
Server=tcp:sql-example-prod.database.windows.net,1433;Database=billing;
Authentication=Active Directory Managed Identity;Encrypt=True;TrustServerCertificate=False;
# Needs RG. Once every client has moved, no SQL password works
az sql server ad-only-auth enable -g $RG -n sql-example-prod
Federation from CI
Workload identity federation takes the credential out of the pipeline. The job trades a signed token from GitHub for a short lived Azure token, and the only thing stored is a rule that names one issuer and one subject, so if that rule leaks, an attacker has learned a string. Use azure/login v3 as of November 2026, although Microsoft’s page prints @v2. Federation still leaves you to control what runs, so we prefer an environment subject. It brings its protection rules with it, so a reviewer can stand between a merge and the token, and a branch subject offers no such gate. Create prod with its reviewer before the first run, because an environment that a run creates on its own has no rules. As of November 2026, though, required reviewers on GitHub Free, Pro and Team need a public repository, as From the pipeline notes, so for the private billing-api that gate needs GitHub Enterprise. The listing below uses gh secret set to store the three IDs that the workflow reads.
# Needs RG. An identity for deployment, a role on one app, and no client secret.
SITE_ID=$(az webapp show -g $RG -n app-billing-prod --query id -o tsv)
APP_ID=$(az ad app create --display-name billing-api-deploy --query appId -o tsv)
SP_OID=$(az ad sp create --id "$APP_ID" --query id -o tsv)
az role assignment create --role "Website Contributor" --scope "$SITE_ID" \
--assignee-object-id "$SP_OID" --assignee-principal-type ServicePrincipal
# The workflow reads the three IDs as secrets, so store them. Then, under Settings and
# Environments, create prod with a required reviewer, before any run can create it bare.
gh secret set AZURE_CLIENT_ID -R example-systems/billing-api --body "$APP_ID"
gh secret set AZURE_TENANT_ID -R example-systems/billing-api \
--body "$(az account show --query tenantId -o tsv)"
gh secret set AZURE_SUBSCRIPTION_ID -R example-systems/billing-api \
--body "$(az account show --query id -o tsv)"
# Run the workflow below once. Login fails, and the log prints the subject to trust, such as:
# repo:example-systems/billing-api:environment:prod
# repo:example-systems@123456/billing-api@456789:ref:refs/heads/main
# Matching is case sensitive, and a wrong subject saves with no error.
cat > credential.json <<'JSON'
{ "name": "github-prod", "issuer": "https://token.actions.githubusercontent.com",
"subject": "repo:example-systems/billing-api:environment:prod",
"audiences": ["api://AzureADTokenExchange"] }
JSON
az ad app federated-credential create --id "$APP_ID" --parameters credential.json
# Run it again, then delete what the job no longer needs. The last line should print [].
gh secret delete STORAGE_CONN --repo example-systems/billing-api
az ad app credential list --id "$APP_ID"
# Excerpt of .github/workflows/deploy.yml. The three IDs are identifiers, not credentials
permissions: { id-token: write, contents: read }
jobs:
deploy:
runs-on: ubuntu-latest
environment: prod
steps:
- uses: azure/login@v3 # pin to a full commit SHA in real use
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
Once a scan job shares the file, id-token: write belongs in the deploy job alone, as in the full workflow below. At run time, the signed token carries repository, ref and environment claims, Entra checks its issuer, subject and audience before it returns an access token, and the job calls the resource with that token. Nothing is stored, and the token expires on its own.
Copy the subject from a run in your own repository, because an article, this one included, may show an old format. Since July 15, 2026, new, renamed and transferred repositories carry immutable IDs in the subject by default, while billing-api is older, has not opted in and keeps the plain names. The IDs stop a recycled name from minting a match, and as of November 2026, the azure/login README says that numeric IDs never match, which is out of date, while the portal may suggest them. A failed login prints the subject that GitHub sent, so one ending in ref:refs/heads/main means that the job left out environment: prod. The fix is to add it. A second credential for the branch subject would also make the error go away, but it would let any job on the branch skip the protection rules.
# Read the subject from a failed run, then set it beside what Entra holds
# RUN_ID is the number after /runs/ in the failed run's URL. The token itself is never printed.
gh run view "$RUN_ID" -R example-systems/billing-api --log-failed | grep 'subject claim'
# Expect one line that ends in the subject to trust, as in the sample output below.
az ad app federated-credential list --id "$APP_ID" \
--query "[].{issuer:issuer,subject:subject,audiences:audiences}"
# Sample output, a new workflow file that fails at login
Federated token details:
issuer - https://token.actions.githubusercontent.com
subject claim - repo:example-systems/billing-api:ref:refs/heads/main
audience - api://AzureADTokenExchange
job_workflow_ref -
example-systems/billing-api/.github/workflows/release.yml@refs/heads/main
Error: AADSTS700213: No matching federated identity record found for
presented assertion subject
'repo:example-systems/billing-api:ref:refs/heads/main'.
In that sample, the issuer and audience agree, and the subject ends in ref:refs/heads/main because GitHub issues an environment subject only to a job that declares one. Of the other messages, AADSTS7002138 means that the strings differ only in case, and Failed to fetch federated token from GitHub means that the job lacks id-token: write.
The full workflow adds the scan gate from A secret scan in CI: it scans every pull request, every push to main and, on Mondays, the full history, and deploy runs only on a push, after a clean scan, with no stored cloud secret. As of November 2026, actions/checkout v7.0.1, the TruffleHog action v3.99.8 and azure/login v3.1.0 are pinned by SHA, the scan keeps verified and unknown results through extra_args, and the last steps publish, zip and deploy the app. We checked each line against vendor pages and parsed the YAML, but we did not run it, so try it on a test repository first.
# .github/workflows/deploy.yml
name: deploy
on:
pull_request:
push:
branches: [main]
schedule:
- cron: "23 3 * * 1" # Mondays 03:23 UTC, the full history
workflow_dispatch:
permissions:
contents: read
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- uses: trufflesecurity/trufflehog@ea68ca03c865c5f324e86e849ced20ed3cf9dbbb # v3.99.8
with:
version: "3.99.8" # image tag, no v. Without it the scanner floats on latest.
extra_args: --results=verified,unknown
deploy:
needs: scan
if: github.event_name == 'push'
runs-on: ubuntu-latest
environment: prod # the federated subject, with its protection rules
permissions:
contents: read
id-token: write # the token exchange. No client secret exists.
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: azure/login@a641126d1b8aa4d1fa005f4f92df94a3a4c4c906 # v3.1.0
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
- run: dotnet publish src/Billing.Api -c Release -o out
- run: cd out && zip -qr ../app.zip .
- run: az webapp deploy -g rg-billing-prod -n app-billing-prod --src-path app.zip
Federation from GitHub to AWS
From GitHub Actions to AWS, a trust policy on a role takes the place of the federated credential, and the role replaces the ci-deploy user from What counts as a secret. The classic mistake is a wildcard in the subject, which AWS warns can admit workflows “outside of your control”. As of November 2026, IAM rejects a missing or bare wildcard sub, but repo:example-systems/* still trusts your whole organization, while a plain Entra credential takes no wildcards at all. The old wildcard repo:my-org/* does not match immutable subjects, and the configure-aws-credentials action, at v6 as of November 2026, needs runner v2.327.1 or later.
# Needs the role billing-deploy, with the deploy permissions. The provider is once per account.
aws iam create-open-id-connect-provider --client-id-list sts.amazonaws.com \
--url https://token.actions.githubusercontent.com
cat > trust.json <<'JSON'
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow",
"Principal": { "Federated":
"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": { "StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub":
"repo:example-systems/billing-api:environment:prod" } } }] }
JSON
aws iam update-assume-role-policy --role-name billing-deploy --policy-document file://trust.json
# The workflow step, with id-token: write, in place of the two stored keys. Pin to a full SHA.
# - uses: aws-actions/configure-aws-credentials@v6.3.0
# with: { role-to-assume: arn:aws:iam::123456789012:role/billing-deploy,
# aws-region: us-east-1 }
Azure DevOps service connections
In Azure DevOps, workload identity federation is the recommended way to authenticate a service connection. The Azure DevOps issuer retires on July 1, 2027, for single tenant apps and managed identities in the public cloud, and as of November 2026, new connections use the Microsoft Entra issuer. Both the issuer and the subject change, so a credential that you keep by hand needs both new values, while multitenant apps and non public clouds keep the old issuer.
| Field | Azure DevOps issuer, retiring | Microsoft Entra issuer, new |
|---|---|---|
| Issuer | https:// |
https:// |
| Subject | sc:// |
<microsoft entra prefix>/ |
The conversion tool works only if Azure DevOps created the connection and only one project uses it, because a connection made by hand gives Azure DevOps no rights over its own credentials. For one made by hand, you change the federated credential yourself, by Microsoft’s manual guide, and the tool cannot revert it. A conversion from a secret can be reverted, with its secret, for seven days, but Microsoft documents no revert for the issuer update, so update one connection first and watch a few deployments before you move the rest. Leave “Grant access permission to all pipelines” clear, as From the pipeline says. A Microsoft-hosted agent cannot reach a vault whose public access is off.
When a static secret is still the answer
A static secret is still the right answer where the vendor offers only a key, and a vault does not make that key short lived. As of November 2026, Stripe documents its own Workload Identity Federation in private preview, so stripe-api-key stays on rung 3 for now, restricted, dated so that SecretNearExpiry fires, rotated with an overlap window, watched by an alert on other SecretGet callers, and fetched by a managed identity. Check each vendor for such an option at the time of the fix. Generally, a static secret can stay where it is narrow, watched and quick to replace, and it should climb a rung as soon as the vendor offers one.
Replace it everywhere else
The ladder from Replace it with something better reaches past the app server, and each part below names its rung.
AWS Secrets Manager and IAM database authentication
Where Key Vault only signals, AWS Secrets Manager rotates as a service feature, through managed rotation or a Lambda function, on rung 3. It protects against age, since a copy that leaks from a log stops working once its user is rotated. It does not protect during a value’s life, so on a 30 day schedule a leak may stay useful for up to 30 days with a single user, or up to 60 with alternating users, because a password set at one rotation stays valid until the rotation after next. With a single user, the database may briefly deny the rotated credentials, so the app needs a retry, while alternating users keep a valid pair readable, at the price of a superuser secret and a clone whose permissions you must keep in step. As of November 2026, the local caching daemon is AWS Workload Credentials Provider, formerly the Secrets Manager Agent, refreshing every 300 seconds by default.
# Secrets Manager rotation for billing/reporting-db, then its labels
# Needs ROTATE_FN_ARN, a rotation function. Omit --rotation-lambda-arn for managed rotation.
aws secretsmanager rotate-secret --secret-id billing/reporting-db \
--rotation-lambda-arn "$ROTATE_FN_ARN" --rotation-rules '{"AutomaticallyAfterDays": 30}'
# Reads the labels, not the value. Expect AWSCURRENT and AWSPREVIOUS after a clean rotation.
aws secretsmanager describe-secret --secret-id billing/reporting-db --query VersionIdsToStages
On RDS for MariaDB, MySQL and PostgreSQL, IAM database authentication climbs to rung 2, swapping the stored password for a 15 minute token. It does not end a session, since the token matters only at login, or limit an administrator, who can connect with no explicit IAM policy. CloudWatch and CloudTrail do not log it either, so your record of who connected comes from the database.
# IAM database authentication on billing-reporting, then a token for 15 minutes
aws rds modify-db-instance --db-instance-identifier billing-reporting \
--enable-iam-database-authentication --apply-immediately
# In PostgreSQL: CREATE USER billing_app; GRANT rds_iam TO billing_app; and rds-db:connect
RDSHOST=$(aws rds describe-db-instances --db-instance-identifier billing-reporting \
--query 'DBInstances[0].Endpoint.Address' --output text)
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname "$RDSHOST" --port 5432 \
--region us-east-1 --username billing_app)" # an env var, tolerable for 15 minutes
A pool that fetches one token at startup fails on its next new connection after 15 minutes, so it needs a token callback. On PostgreSQL, granting rds_iam to any user, the master user included, makes IAM authentication take precedence, so that user can no longer log in with a password.
Vault and OpenBao dynamic credentials
As of November 2026, HashiCorp Vault v2.2.1 and its fork OpenBao v2.7.2 put a database that accepts no identity on rung 2. Each creates a database user per caller under a lease, so a leaked credential has a name, one caller and an end. Vault is under BUSL 1.1, and its license says that internal use “is not considered a competitive offering”, but any license question belongs with your counsel. OpenBao is MPL 2.0, run under the Linux Foundation’s OpenSSF, with the same database engine and the CLI bao.
# Vault dynamic credentials for the reporting database, and a role for billing-app
# Needs a Vault server, a login, and a dedicated vault_admin database user that is not root.
vault secrets enable database
read -rs BOOTSTRAP_PASSWORD # vault_admin's current password, typed once. Nothing echoes.
printf %s "$BOOTSTRAP_PASSWORD" | vault write database/config/reporting password=- \
plugin_name="postgresql-database-plugin" allowed_roles="billing-app" username="vault_admin" \
connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/reporting"
unset BOOTSTRAP_PASSWORD # stdin kept it off the command line
vault write -force database/rotate-root/reporting # from here on, only Vault knows it
vault write database/roles/billing-app db_name="reporting" default_ttl="1h" max_ttl="24h" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
# A new user and lease on every call. The JSON holds the password too, so jq drops it.
vault read -format=json database/creds/billing-app \
| jq -c '{lease_id, ttl: .lease_duration, user: .data.username}'
# Sample: {"lease_id":"database/creds/billing-app/EXAMPLE","ttl":3600,"user":"v-EXAMPLE"}
Vault does not remove secret zero, since your app must still sign in to it, so use a platform identity such as the azure auth method. It does not narrow the grant, since every caller gets what creation_statements says, and it adds a service that must stay up. A pool built once at startup fails when its lease ends and Vault revokes its user. Unless the app renews the lease, that happens at its ttl, an hour in the sample above, so the app must renew it or read a new credential and rebuild before then, and rebuild before max_ttl at the latest. If a job leaks its connection string before you know which other jobs did the same, one lease revoke ends every credential under the role, and each job gets a new user on its next read.
# Ends every credential under the billing-app role at once
vault lease revoke -prefix database/creds/billing-app
Sessions may survive, so test what your database does with an open one, as under Database passwords. Avoid -force at first, since it deletes the lease even when the database refused, which can leave a live user that nothing tracks.
Kubernetes
Limit list on Secrets as tightly as get, and remember that anyone who can create a Pod in a namespace can read its Secrets, even with no right to read Secrets directly. Three approaches answer three questions: who can read the store, where the value comes from, and whether a Secret needs to exist. Generally, you want etcd encryption on every cluster, plus External Secrets Operator or the Secrets Store CSI Driver, reading a vault on rung 3, for each credential that matters.
Upstream, etcd keeps Secrets unencrypted by default, so turn encryption on where needed, although EKS on 1.28 or later encrypts all API data. On AKS, as of November 2026, Microsoft calls --enable-azure-keyvault-kms alone the legacy setup and recommends KMS data encryption, still in preview, for new clusters on 1.33 or later. The newer setup adds --kms-infrastructure-encryption and keeps the older flag only for customer managed keys, so check your cluster. Encryption protects a stolen backup or disk, but not against anyone who asks the API server, which decrypts for whoever it authorizes. Existing Secrets stay as they were until something rewrites them, and since the first provider encrypts writes, identity goes last.
# EncryptionConfiguration for the API server, with KMS v2 first and identity last
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- kms: { apiVersion: v2, name: myKmsPlugin, endpoint: "unix:///tmp/socketfile.sock" }
- identity: {} # last. The first provider encrypts writes, so first means no encryption
# Pass it with --encryption-provider-config, restart the API server, then rewrite what exists:
# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
# Proof on a test Secret, with your etcd flags. The stored bytes start with k8s:enc:kms:v2:
# ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 | hexdump -C
External Secrets Operator v2.14.0 reads a vault with its own workload identity and writes a normal Secret, so Git holds a reference and never a value, which protects your repository but not the cluster. A one hour refreshInterval may keep an old value for an hour after a rotation, and a Pod that took it as an environment variable keeps it until it restarts, so rotate with overlap. As of November 2026, the project supports only its current minor version, so pin it and keep up.
# ExternalSecret for billing-db
# After a SecretStore named azure-store, with authType WorkloadIdentity and the vault's URL
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: billing-db }
spec: { refreshInterval: 1h0m0s, secretStoreRef: { kind: SecretStore, name: azure-store },
target: { name: billing-db, creationPolicy: Owner },
data: [{ secretKey: password, remoteRef: { key: secret/billing-db-password } }] }
The Secrets Store CSI Driver reads the vault when a Pod starts and mounts the value as a tmpfs file with no Secret by default, so no copy reaches etcd, although the driver does not protect the file from anything that runs in the Pod. As of November 2026, rotation stays off until you pass --enable-secret-rotation, and then only the file changes, while a new firewall rule may surface later as a Pod that will not start. Adding secretObjects for an environment variable brings a Secret back, and a Pod must mount the volume before that Secret exists.
# SecretProviderClass billing-kv, for the Secrets Store CSI Driver
# On AKS, the azure-keyvault-secrets-provider add-on with Microsoft Entra Workload ID
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: billing-kv, namespace: billing }
spec:
provider: azure
parameters:
clientID: "<uami-client-id>"
tenantId: "TENANT_ID"
keyvaultName: kv-example-prod
objects: |
array:
- |
objectName: billing-db-password
objectType: secret
# The Pod carries the label azure.workload.identity/use: "true", a service account annotated
# with azure.workload.identity/client-id, and a csi volume that names billing-kv.
Either way, keep Secret manifests out of Git, and list any that are there by name only.
# Secret manifests in Git, by name only
git grep -lE '^kind: *Secret *$' # names only: Secret manifests in Git
Docker build secrets
Docker says that build arguments and environment variables “are inappropriate for passing secrets to your build, because they persist in the final image”, as In what you build explains. A registry token in a build belongs on rung 5, as a narrow CI secret, passed through a secret mount that hands the value to one RUN step and writes it to no layer.
# Before. .npmrc holds a reference: _authToken=${NPM_TOKEN}
FROM node:24-alpine
COPY package*.json .npmrc ./
ARG NPM_TOKEN
RUN npm ci
# Build: docker build --build-arg NPM_TOKEN -t billing-web .
# Sample, docker history --no-trunc: RUN |1 NPM_TOKEN=EXAMPLE-not-real /bin/sh -c npm ci
# syntax=docker/dockerfile:1
# After. env= needs Dockerfile v1.10.0 or later, so keep the syntax line first
FROM node:24-alpine
COPY package*.json .npmrc ./
RUN --mount=type=secret,id=npm_token,env=NPM_TOKEN,required=true npm ci
# Build: docker build --secret id=npm_token,env=NPM_TOKEN -t billing-web .
# Check: docker history --no-trunc billing-web no layer holds the value
The mount keeps the token out of the image, but the build step itself can still expose it, since npm ci runs install scripts with the token in their environment, and anything the step writes to a file is baked in. Without required=true, a missing secret does not stop the build, so a build can pass with no secret at all. At run time, a Compose secret is a plain file on the host, and docker inspect shows a container’s environment to anyone with the Docker socket. BuildKit’s SecretsUsedInArgOrEnv rule flags secrets in ARG and ENV, a cheap gate to keep.
Terraform and OpenTofu
The sensitive flag hides a value in CLI output and the HCP Terraform UI and nothing more, since Terraform “stores your state in a plaintext file, which includes any secret values you defined in your configuration”, in HashiCorp’s words. Even the azurerm_key_vault_secret docs say, “All arguments including the secret value will be stored in the raw state as plain-text.” Backend encryption protects the stored file, but anyone who can read state still sees the value, so terraform state pull still prints it, and the SCARLETEEL key under On the machines around the code was found in such a file. “Ephemeral” values, from Terraform 1.10, and “write only” arguments, from Terraform and OpenTofu 1.11, are used during a run and stored in neither state nor plan. A secret set by infrastructure code can then stay on rung 3 with no copy in state.
# Before: a sensitive variable written to value
variable "stripe_api_key" { sensitive = true } # redacted on screen, kept in state
resource "azurerm_key_vault_secret" "stripe" {
name = "stripe-api-key"
key_vault_id = data.azurerm_key_vault.main.id
value = var.stripe_api_key
}
# After: an ephemeral variable written to value_wo
variable "stripe_api_key" { ephemeral = true } # Terraform 1.11, never in state or plan
resource "azurerm_key_vault_secret" "stripe" {
name = "stripe-api-key"
key_vault_id = data.azurerm_key_vault.main.id
value_wo = var.stripe_api_key
value_wo_version = 2 # raise this number to push a new value
}
# True means that the value sits in state. Prints no value. Before: [true] After: [false]
terraform state pull | jq '[.resources[] | select(.type == "azurerm_key_vault_secret")
| .instances[].attributes.value | (. != null and . != "")]'
Terraform cannot read a write only value back, so a changed variable with the same value_wo_version changes nothing. Moving to value_wo clears current state, but not older copies that a backend or a laptop may keep, so treat existing state as a secret, restrict the backend and rotate what it holds. None of this protects the run, since the value still passes through the machine that applies the plan. OpenTofu adds client side state encryption from 1.7, although losing that key loses the state. On Azure, an ephemeral azurerm_key_vault_secret block reads a secret for one run, and use_azuread_auth and use_oidc keep Terraform itself off stored keys.
SOPS for config that lives in Git
SOPS, at getsops/sops rather than mozilla/sops, encrypts the values in a file and leaves the keys readable, so configuration can live in Git with working diffs and a meaningful review, and a MAC over the values shows tampering. It sits on rung 4 because its decryption key is itself a long lived secret, so an age key in a CI variable is one static secret guarding the rest. The project recommends age over PGP, and since names stay readable, stripe_api_key still tells a reader what the file holds.
# secrets.prod.yaml, committed as plain text
stripe_api_key: EXAMPLE-not-a-real-key
# The same file after sops encrypt --in-place. Illustrative shape, sample values
stripe_api_key: ENC[AES256_GCM,data:EXAMPLE=,iv:EXAMPLE=,tag:EXAMPLE=,type:str]
sops:
age:
- recipient: age1EXAMPLEplatform
enc: <the data key, encrypted to this recipient>
- recipient: age1EXAMPLEcontractor
enc: <the same data key, encrypted to this recipient>
When a contractor leaves, deleting that key from .sops.yaml and stopping there changes nothing, because the encrypted file keeps its own list. Follow the order below from the SOPS documentation, which warns that if you rotate the underlying credentials before step 2, the removed key could still read them. Every earlier commit stays decryptable with the old key, so those values stay exposed for good, and step 3 is not optional.
# Removing a recipient, in the order that the SOPS documentation gives
# 1. Take the key out of .sops.yaml, then apply that to the file
sops updatekeys secrets.prod.yaml
# 2. A new data key, with every value encrypted again
sops rotate --in-place secrets.prod.yaml
# 3. Only now rotate each real credential at its issuer, and store it
sops edit secrets.prod.yaml
With age, decryption is local, so no log records who read a value. Since the first creation_rules entry to match wins, a broad \.yaml$ rule above the production rule sends production values to development recipients.
Nothing secret in the browser
Everything in a bundle is public, so a prefix such as NEXT_PUBLIC_ or VITE_ is a publish instruction, as In what you build shows. The fix is a “backend for frontend”, or BFF, the first pattern in RFC 10017, where a server of yours holds the credential on rung 3, in Key Vault, and the browser holds only a session. The RFC describes it for OAuth tokens, and the same shape fits a vendor key, although a Stripe publishable key is public by design, as Is it a credential at all rules.
// src/Forecast.tsx. The build inlines the key, so every visitor can read it in the bundle.
const res = await fetch("https://api.example.com/v1/forecast", {
headers: { Authorization: `Bearer ${import.meta.env.VITE_FORECAST_KEY}` },
});
// src/Forecast.tsx. The browser calls your own origin, with its session cookie and no key.
const res = await fetch("/api/forecast", { headers: { "X-Requested-With": "fetch" } });
// Program.cs, .NET 8 or later. Needs SecretCache from Reading from Key Vault, as a singleton, and a cookie
// session as RFC 10017 asks: HttpOnly, Secure, SameSite=Strict, a __Host-Http- name.
using System.Threading.RateLimiting; // RateLimitPartition, FixedWindowRateLimiterOptions
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClient("forecast", c => c.BaseAddress = new("https://api.example.com"));
builder.Services.AddRateLimiter(o =>
{
o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
o.AddPolicy("per-user", ctx => RateLimitPartition.GetFixedWindowLimiter(
ctx.User.Identity?.Name ?? "anonymous", _ => new FixedWindowRateLimiterOptions
{ PermitLimit = 30, Window = TimeSpan.FromMinutes(1) }));
});
var app = builder.Build();
app.UseAuthentication(); app.UseAuthorization(); app.UseRateLimiter();
app.MapGet("/api/forecast", async (HttpRequest req, IHttpClientFactory http,
SecretCache secrets, CancellationToken ct) =>
{
if (req.Headers["X-Requested-With"] != "fetch") return Results.BadRequest(); // CSRF
// The host and path are fixed here. A value from the browser in either makes an open proxy.
var call = new HttpRequestMessage(HttpMethod.Get, "/v1/forecast");
call.Headers.Authorization = new("Bearer", await secrets.GetAsync("forecast-api-key", ct));
using var res = await http.CreateClient("forecast").SendAsync(call, ct);
return res.IsSuccessStatusCode // never echo the vendor's error
? Results.Content(await res.Content.ReadAsStringAsync(ct), "application/json")
: Results.StatusCode(StatusCodes.Status502BadGateway);
}).RequireAuthorization().RequireRateLimiting("per-user");
app.Run();
In the RFC’s words, “there are simply no tokens to be stolen”, although an attacker in your own origin still “is able to send requests to the BFF from within the user’s browser”. The usual failure is an open proxy, a handler that forwards whatever the browser names or skips the session check, so your paid API answers the internet on your key. On Next.js, a route handler with no NEXT_PUBLIC_ prefix follows the same rules, and after each build, grep -rlE 'api\. should find no files, since a match means the browser still calls the vendor. ASP.NET Core counts the rate limit per instance, so 30 a minute on 6 instances allows up to 180, and a shared limit belongs in a gateway.
Local development without production values
Alert 17 began here, with a connection string in appsettings., which is how a production key often comes to sit in a development file: someone needed the app to start. The aim is that a developer never needs a production value at all, which puts your laptop on rung 1, with your own identity and a development vault. This setup does not protect a laptop that is itself compromised, since your session reaches whatever your roles allow, which is why those roles stop at the development vault.
First, sign in as yourself and leave AZURE_CLIENT_SECRET out of your shell, since EnvironmentCredential comes first in the default chain. Second, the owner of kv-example-dev grants you Key Vault Secrets User once, and no developer needs a role on kv-example-prod. Third, choose the credential by environment so that production cannot fall through to you, or, if you keep DefaultAzureCredential, export AZURE_TOKEN_CREDENTIALS=, which keeps the chain to developer tools. Fourth, keep vendor test keys out of the tree with a reference file for op run, which holds no value and is safe to commit, while .NET’s Secret Manager keeps values outside the project but does not encrypt them, as Microsoft warns. Fifth, prove the boundary against kv-example-prod, where a 403 is the result that you want, and --query id keeps a value off your screen if you turn out to have a role. Read the message, though. With public access off on kv-example-prod, a 403 that names your client address comes from the firewall and says nothing about your roles, so confirm those with the role listing in Key Vault, set up properly.
# Local development, steps 1 to 5. Step 3 is C#, and the rest is shell
# 1. Sign in as yourself
az login && az account show --query user.name -o tsv # expect your own name
# 2. The owner of kv-example-dev runs this once
DEV_VAULT_ID=$(az keyvault show -n kv-example-dev --query id -o tsv)
az role assignment create --role "Key Vault Secrets User" --assignee you@example.com \
--scope "$DEV_VAULT_ID"
// 3. Choose the credential by environment
TokenCredential credential = builder.Environment.IsDevelopment()
? new AzureCliCredential()
: new ManagedIdentityCredential(ManagedIdentityId.SystemAssigned);
# 4. Keep vendor test keys out of the tree
# .env.op: STRIPE_KEY="op://Dev/stripe-test/credential"
op run --env-file=.env.op -- dotnet run
# 5. Prove the boundary. A 403 is the result that you want
az keyvault secret show --vault-name kv-example-prod -n stripe-api-key --query id
The failure that teams hit is the shared .env, passed around in chat because one person needed something to work on a Friday. A client side hook cannot be enforced, as Gitleaks, Betterleaks and detect-secrets explains, so the control stays on the server, with push protection.
Prove the fix
A closed alert tells you that someone pressed a button, and nothing about whether the credential still works. GitHub does not close an alert when the token leaves the repository, and closing an alert revokes nothing. Generally, we treat a finding as open until a second person could repeat the proof from the record alone.
What closes a finding
Most findings close on a sentence such as “rotated the key”, typed by whoever did the work at the end of a long day. It may well be true, but a reviewer has no way to check it, and the resolved_at time on a GitHub alert records only when someone closed the alert.
# Two ticket comments on SEC-1204, the ci-deploy key. The first is an assertion
SEC-1204 closed. Rotated the ci-deploy key, removed it from the repo, all good.
# The second is evidence
SEC-1204 closed. Key AKIAIOSFODNN7EXAMPLE, user ci-deploy, account 123456789012.
Inactive since 14:02 UTC. A deny policy ended every session issued before 14:02.
Negative test 14:47 UTC from 198.51.100.7, R. Okafor: 403 InvalidClientTokenId, output attached.
CloudTrail, all Regions, 90 days: only the runner range. Older days are a stated gap.
TruffleHog 3.99.8, fresh clone, verified and unknown: clean. History keeps the dead key.
Cause: key in deploy.yml, now an OIDC role. export.yml in billing-reports is on the new key.
Repeated Wed 09:30 by J. Rivera, same result.
The second gives an identifier, a time, an address, a name and an exact error, which anyone can check at the issuer without asking the author. Of the eight lines that we expect behind every closure, the last matters most, because whoever ran the fix wants it to have worked, and may miss a test that ran against the wrong account.
| No. | The record shows |
|---|---|
| 01 | The old credential fails at the issuer, shown by a negative test with the output saved. |
| 02 | What it created has ended: sessions, SAS tokens, access tokens and open connections. |
| 03 | Every consumer runs on the replacement, and every other copy is gone or named. |
| 04 | The window was hunted, and the record says which logs existed and which did not. |
| 05 | A fresh scan is clean, and the record says what it covered and what it cannot see. |
| 06 | The alert is closed with the right reason and a comment, in every tool that raised it. |
| 07 | The cause has a guardrail, so that the same path cannot leak the next key. |
| 08 | A second person has repeated the negative test, or watched it run, and read the record. |
Not every line will come back clean, because logs expire and fork owners go quiet. Where one cannot, the record names the gap and who accepted it, since a stated gap tells a reviewer what was not checked, and a blank field does not. Not every credential can be watched failing either, so the closing rule at the top of this section has exceptions. We found no call that names a PyPI token’s owner, and an upload is not a test, so its record shows the token gone from your list and says that no negative test exists. A Slack webhook, never tested because a POST that works posts a message to the channel, closes the same way.
The negative test
The proof starts with a “negative test”, which tries the old credential the way an attacker would and expects it to fail, and it is the one step that shows the issuer has stopped honoring the value. Send the value to its own issuer and nowhere else, with no other credential in play, and save the output. Generally, a test counts only when the request reached the issuer’s authentication step and was refused there, and a permission error is worse than no answer. A permission error means that the key authenticated and was merely told no. A failing test also needs a control, or you have only shown that a command can fail.
# The old storage key, sent only to its own issuer, then a control
read -rs OLD_KEY # paste the old key. If it is empty, az fetches the live one.
[ -n "$OLD_KEY" ] && AZURE_STORAGE_ACCOUNT=stexamplebilling AZURE_STORAGE_KEY="$OLD_KEY" \
az storage container list --auth-mode key -o table 2>&1 | tee negative-test.txt; unset OLD_KEY
# Expected in the output, after "Server failed to authenticate the request."
# ErrorCode:AuthenticationFailed
# Control, your own identity with a data role, same machine. This one must list containers.
az storage container list --account-name stexamplebilling --auth-mode login -o table
If the first command lists containers, the old key is alive, and the second, an identity that should work, keeps a bad network from passing for proof. The test is still a validation call, so run it only on credentials your organization owns and log it as in Your checks or theirs. The rules in Is it live that keep a value out of shell history and process lists hold until the test says that the credential is dead.
| Credential and test | Dead looks like | Proves nothing |
|---|---|---|
AWS access key, aws sts get-caller-identity |
403 InvalidClientTokenId, or UnrecognizedClientException on newer AWS pages. AWS documents no code for an Inactive key, so save the exact text it returns |
AccessDenied on any other call is a live key. SignatureDoesNotMatch is a bad clock or a damaged secret |
Azure Storage key, az storage container list --auth-mode key |
403 AuthenticationFailed |
403 AuthorizationFailure is the firewall, and 403 KeyBasedAuthenticationNotPermitted means shared key is off, whatever the key |
| Entra client secret, a client credentials request | AADSTS7000215. An expired secret gives AADSTS7000222 |
Tokens already issued may work on for 60 to 90 minutes, up to 24 hours for Microsoft Graph under CAE |
GitHub token, GET /user on api.github.com |
401 | A 404 on a private repository can be a live token. Do not loop, since failures bring a temporary 403 |
Database password, psql, sqlcmd or mysql |
password authentication failed for user, error 18456 with state 8 in the log, or ERROR 1045 (28000) |
A timeout never reached the server. Open connections survive until you end them |
Note. A negative test speaks only for the old value, so the record also answers the What Survives a Rotation row for each credential.
A 403 alone proves little. Tried from an office laptop that the firewall refuses, the old key draws AuthorizationFailure, as the live one would, so only a rerun from an allowed host, with the control listing containers first, lets AuthenticationFailed count.
On AWS, GetCallerIdentity needs no permissions, so a deny policy cannot make a live key look dead, while the Entra test takes the tenant’s OpenID configuration as its control. The test for an old SSH key is in SSH and TLS private keys. Repeat any test before you trust it, and retry an IAM success once, since AWS warns that changes “take time to become visible from all possible endpoints” and npm revocation can take up to an hour.
# The old AWS pair and the old Entra secret, each sent only to its issuer, then a control. Sample output
# AWS. No profile or session token in play, and the values die with the subshell.
read -rs AK; read -rs SK
( unset AWS_PROFILE AWS_SESSION_TOKEN
export AWS_ACCESS_KEY_ID="$AK" AWS_SECRET_ACCESS_KEY="$SK"; aws sts get-caller-identity
) 2>&1 | tee negative-aws.txt; unset AK SK
# An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation:
# The security token included in the request is invalid.
aws sts get-caller-identity --query Arn --output text # control: your own identity, same host
# Entra. curl encodes the value from stdin, and jq keeps two fields, so no token can print.
read -rs CS
printf %s "$CS" | curl -sS "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token" \
-d client_id=APP_ID -d grant_type=client_credentials \
-d scope=https%3A%2F%2Fgraph.microsoft.com%2F.default --data-urlencode client_secret@- \
| jq -c '{error, error_codes}' | tee negative-entra.txt; unset CS
# {"error":"invalid_client","error_codes":[7000215]}
# Entra control, same host and tenant. It must print 200.
curl -sS -o /dev/null -w '%{http_code}\n' \
"https://login.microsoftonline.com/TENANT_ID/v2.0/.well-known/openid-configuration"
Order matters on Azure. Once shared key access is off, every key on the account returns the same 403, renewed or not, so test the old value right after renewal, while shared key access is still on. With --results=, a fresh TruffleHog scan goes quiet once a key is revoked, so a claim that a string has left history needs unverified as well. Close the alert last, because the comment is permanent and its reason should be true when it is written.
The record also needs a second line for what each credential created. For an AWS key, the deny on aws:TokenIssueTime is in place and its session key IDs show no later calls. For an Entra secret, the token lifetime has passed with no sign-in on that keyId, and for a database password, no session is older than the change.
The closure record
GitHub’s secret scanning REST API says that a resolution_comment “Cannot be updated or deleted”, so write it once, for an auditor, with no value in it.
# Only after the negative test. The comment is permanent, so it names the proof, never a value
R=repos/example-systems/billing-api V="X-GitHub-Api-Version: 2026-03-10"
gh api -H "$V" --method PATCH "$R/secret-scanning/alerts/17" -f state=resolved \
-f resolution=revoked \
-f resolution_comment='key1 renewed, old value returns 403 AuthenticationFailed. See SEC-1187.'
The closure record is one page in the ticket, and it lets someone who was not in the room answer four questions: what leaked, what it could reach, whether anyone used it, and how we know that it is dead. It may name a key but never contains one, and it names each gap and who accepted it. It reports what the logs show and takes no view on whether anything was a breach, which is a question for counsel.
# SEC-1187, the closure record, in the ticket by Thursday 16:00
Finding High. Real and live, full data plane on the invoice PDFs.
billing-api, GitHub alert 17, 09:14 Tuesday. Secret Protection
went on Monday evening, so push protection never saw it.
Leaked key1 of stexamplebilling, matched against the account's key
list without using it, in appsettings.Development.json, commit
7f3c2ab by dreyes, merged in 5e91d0c. Also the Actions secret
STORAGE_CONN and an App Service app setting.
Exposure 11 days, 9 on main, 31 people and one fork,
example-contractor/billing-api. No sign of public exposure.
Contained Consumers moved to key2, never exposed. After 15 minutes, so
that links signed with key1 had expired, key1 was renewed
Tuesday, ahead of the 09:14 Wednesday clock. Any SAS signed
with it ended there. Security and legal heard that morning.
Hunt StorageBlobLogs, already on: known addresses only, the App
Service and the deploy runners.
Gap Git events reach back 7 of the 11 days, so no evidence of
abuse, not proof of none. Accepted by T. Kim, security lead.
History Limited rewrite of one string. The old commit may stay
reachable by SHA, and the key is dead. Contractor asked to
delete the fork.
Replaced Managed identity on app-billing-prod, with Storage Blob Data
Contributor on invoices and Storage Blob Delegator at account
scope for user delegation SAS. Actions federation.
STORAGE_CONN and the app setting deleted. Shared key access
off, which retired key1 and key2.
Proof Old key from a test client, with a control: 403
AuthenticationFailed, then, with shared key off, 403
KeyBasedAuthenticationNotPermitted. Fresh TruffleHog scan of
branches and tags, unverified included, clean. A second
engineer repeated it. Alert 17 closed as revoked.
Start every record from the blank one below. Fill in identifiers and never a value, write “none” on any line that would otherwise be blank, and let a line that cannot be filled become a stated gap. Its Cause and Closed lines go further than SEC-1187.
# The closure record template, one per finding. Identifiers only, never a value
SEC-____ Repository, alert number and tool:
Finding Score and clock: Real, live, what it reaches:
Leaked Identifier, never the value: Where, commit, author:
Other copies:
Exposure First commit: Days: Who could read it: Forks:
Contained Revoked at: Overlap? Sessions and SAS ended: Told:
Hunt Logs read, with dates: Result:
Gap What cannot be known: What we assume: Accepted by:
History Rewritten or not, and why: What stays reachable:
Replaced New mechanism: Old copies deleted:
Proof Negative test, with time, address, person and exact error:
Control: Second person:
Scan, version and scope: What it cannot see:
Closed Reason and comment, in each tool:
Cause What let it leak: Guardrail added:
A record that ends with a gap
Some findings cannot reach a clean record, because the evidence stopped existing before anyone looked. The ci-deploy key of SEC-1204 sat in deploy.yml for 140 days, CloudTrail Event history reaches back 90, and no trail wrote to S3, so there is no record of what the dead key did in its first 50 days. AWS also says that trails and event data stores “do not log data events” by default, so S3 reads may leave no record at any age. Writing “no abuse found” would claim more than the evidence holds, and an open ticket changes nothing.
What you can still do is read state in place of events, since state does not expire, and aws iam list-access-keys shows every key on the user today, so a second key minted in the blind days would still appear. Then bound the worst case, and act as if it happened. Expired logs are a trigger in When a leak becomes an incident, so security and legal hear about it, and counsel decides how to treat the window. The record then says what cannot be known, what we assume in its place, who accepted that and what changed, so that a later reader does not mistake silence for a clean result.
# SEC-1204, the same key, closed with a stated gap
Finding High. Live, with production write on the bucket
example-billing-exports.
Leaked AKIAIOSFODNN7EXAMPLE, user ci-deploy, in the private file
.github/workflows/deploy.yml, for 140 days.
Contained Inactive at 14:02 UTC. A deny on aws:TokenIssueTime ended
every session issued before 14:02. export.yml in
billing-reports moved to the replacement key by 14:40.
Hunt Event history, all Regions, last 90 days: the runner range
and our own test only, no CreateAccessKey.
list-access-keys shows two keys, the leaked one, Inactive,
and the replacement. No other exists.
Gap The first 50 days have no management events, since no trail
was writing to S3. All 140 days have no object events, since
S3 data events were never on, so a read leaves no trace.
Assumed Every object in the bucket was read, and any could have been
replaced. The owner listed what the exports hold, for legal.
Accepted T. Kim, security lead, Wednesday. Counsel holds the question
of the unlogged days, and this record takes no view on it.
Fixed A trail writes to S3 with data events on the bucket.
An OIDC role replaced the key in deploy.yml.
Proof 403 InvalidClientTokenId at 14:47 UTC, output attached,
repeated by J. Rivera Wed 09:30. TruffleHog 3.99.8, fresh
clone, verified and unknown: clean. History keeps the key.
Its fix, a trail that writes to S3 with data events on the bucket, is in Guardrails that stay on. Generally, a window that you cannot see goes in the record as a limit on the finding.
Going forward, every guardrail needs a check that someone reads, such as a weekly list of active repositories without push protection and of every bypass, as in Guardrails that stay on, and every credential needs a named owner, as in An inventory with owners. An unrehearsed rotation will be slow at 09:14 on a Tuesday, which is why we recommend a rotation drill once a quarter.
Keep it fixed
The replacement in Replace it with something better closes one path, and guardrails keep the next developer from opening another. Settings drift, so each one needs a check and a person who reads the result.
Guardrails that stay on
| Guardrail | What it keeps out | How you check that it is still on |
|---|---|---|
| Push protection | Known token formats, at the server | The listing below, weekly, with is_bypassed=true alerts |
| CI scan, weekly sweep | Old leaks, and new detectors | A required check, version set, full run under 7 days old |
| Suppression audit | Findings hidden by a comment | Monthly, --ignore-gitleaks-allow, --no-ignore-tag, and changes to secret_scanning.yml |
| Azure Policy | Shared key, secrets with no expiry | Two built in policies, read with az policy state list |
| Log retention | A hunt with nothing to read | Diagnostic settings, retention, a trail, and a Git event stream on Enterprise Cloud |
On GitHub, the setting that matters most reaches repositories that do not exist yet. We make the secrets-baseline configuration from GitHub secret scanning the default for new repositories and enforce it, so owners cannot change it, though a repository transferred in needs it by hand. The same configuration turns on delegated bypass and delegated alert dismissal, with reviewers named in its bypass options. As of November 2026, the ruleset rule “Require secret scanning alerts are resolved” is in preview, and it also blocks a merge.
# GitHub guardrails, as an organization owner or security manager
O=orgs/example-systems/code-security/configurations # reviewers: ..._bypass_options
C=$O/$(gh api $O --jq '.[] | select(.name == "secrets-baseline") | .id') # from GitHub secret scanning
gh api --method PATCH $C -f enforcement=enforced -f secret_scanning_delegated_bypass=enabled \
-f secret_scanning_delegated_alert_dismissal=enabled
gh api --method PUT $C/defaults -f default_for_new_repos=all
# Weekly. Active repositories without push protection. A clean run prints nothing.
gh api --paginate orgs/example-systems/repos --jq '.[] | select(.archived == false
and .security_and_analysis.secret_scanning_push_protection.status != "enabled") | .full_name'
# Weekly. Every bypass and who made it. Read the reason on each, and chase any left open.
A="orgs/example-systems/secret-scanning/alerts?is_bypassed=true&hide_secret=true&per_page=100"
gh api --paginate "$A" --jq '.[] | [.html_url, .push_protection_bypassed_by.login] | @tsv'
The first weekly check lists active repositories without push protection, and a clean run prints nothing. The second lists every bypass and who made it, with hide_secret=true for the reason in Working alerts from the command line. Someone reads the reason on each and chases any left open, since two of the three bypass reasons close an alert before anyone looks.
On Azure, two built in policies do the same job for shared key and for secrets with no expiry. Assign each one in Audit, let the first evaluation finish, read what it reports, and move to Deny only once rows appear and each reads Compliant. Deny refuses any new or changed account that permits shared key, which may stop a template that you forgot. A policy change may take up to 30 minutes, and up to an hour for Key Vault Deny, which leaves existing secrets alone, so keep reading the report.
Logs need the same care, because a diagnostic setting only starts the flow, and Log Analytics tables keep 30 days by default as of November 2026, which a 140 day leak outlives. Blob logs need their own setting, the listing raises the workspace to 365 days, and on AWS a multi-region trail outlasts the 90 days of Event history. Its advanced selectors replace the default ones, so the first keeps management events and the second adds object events for the exports bucket, which cost extra and close the gap in A record that ends with a gap, while an existing bucket needs CloudTrail’s bucket policy.
# Azure Policy in Audit, then Deny, then blob logs, retention and the AWS trail
S=/subscriptions/SUB_ID
az policy assignment create -n no-shared-key --scope "$S" -p '{"effect":{"value":"Audit"}}' \
--policy 8c6a50c6-9ffd-4ae7-986f-5fa6111f9a54 # prevent shared key access
az policy assignment create -n secret-expiry --scope "$S" -p '{"effect":{"value":"Audit"}}' \
--policy 98728c90-32c7-4049-8429-847dc0f4fe37 # secrets should have an expiration date
az policy state trigger-scan # waits for a fresh evaluation
az policy state list -a no-shared-key --query '[].[complianceState, resourceId]' -o tsv
# Deny only once rows appear and each reads Compliant. No rows means no evaluation yet.
az policy assignment update -n no-shared-key --scope "$S" -p '{"effect":{"value":"Deny"}}'
# Blob logs, kept for a year, into WORKSPACE_ID, your Log Analytics workspace. Storage needs its own setting.
STORAGE_ID=$(az storage account show -g rg-billing-prod -n stexamplebilling --query id -o tsv)
az monitor diagnostic-settings create --name blob-audit --workspace "$WORKSPACE_ID" \
--resource "$STORAGE_ID/blobServices/default" --logs '[{"category":"StorageRead",
"enabled":true},{"category":"StorageWrite","enabled":true},
{"category":"StorageDelete","enabled":true}]'
az monitor log-analytics workspace update --ids "$WORKSPACE_ID" --retention-time 365
# AWS. A trail outlasts the 90 days of Event history. Advanced selectors replace the trail's
# default ones, so the first keeps management events and the second adds object events, which
# cost extra and close the gap in A record that ends with a gap. An existing bucket needs CloudTrail's policy.
aws cloudtrail create-trail --name audit --s3-bucket-name example-logs --is-multi-region-trail
aws cloudtrail start-logging --name audit
aws cloudtrail put-event-selectors --trail-name audit --advanced-event-selectors '[
{"Name":"management","FieldSelectors":[{"Field":"eventCategory","Equals":["Management"]}]},
{"Name":"exports","FieldSelectors":[{"Field":"eventCategory","Equals":["Data"]},
{"Field":"resources.type","Equals":["AWS::S3::Object"]},
{"Field":"resources.ARN","StartsWith":["arn:aws:s3:::example-billing-exports/"]}]}]'
Keep the repository private, trim its access list on a schedule, and leave private forking off, which is the default for new organizations as of November 2026. The block below puts the files that can silence a scanner in front of the people who own security. It takes effect only with “Require review from Code Owners” on the branch rule, the team needs write access, and a pull request reads the file from its base branch.
# .github/CODEOWNERS
/.github/ @example-systems/security # workflows, scanner config, this file
/.gitleaks.toml @example-systems/security
/.gitleaksignore @example-systems/security
/.secrets.baseline @example-systems/security
/.pre-commit-config.yaml @example-systems/security
An inventory with owners
A credential that cannot be found, or that has no owner, does not get rotated, so the inventory exists to fix both. It holds one row per credential, not per copy, and records names and metadata, never a value. In our example, the row below lets the person on call replace the Stripe key without guessing who reads it. Its owner is a named person as well as a team, and its expiry is a reminder in the vault that revokes nothing.
# inventory/credentials.yaml, a sample row. Names and metadata only, never a value.
- id: stripe-api-key # Stripe restricted key, live mode
lives_in: kv-example-prod, read by app-billing-prod through its managed identity
owner: j.rivera@example.com # a person, not only a team
can_reach: charges and refunds on the live Stripe account
expires: 2027-02-01 # a reminder in the vault, it revokes nothing
rotation: quarterly, overlap up to 7 days, runbook docs/rotate-stripe.md
last_proved: 2026-11-03, old key 401 on GET /v1/balance
Seed the list from the issuers, since memory misses the keys that have gone untouched for years. Compare both ways: a credential with no row has no owner, and a row with no credential is stale. The listing reads names and dates only, from storage accounts, Entra apps, the vault, GitHub and the IAM credential report, which it generates and then reads.
# Names and dates, never a value. Anything but false in column 2 means shared key is allowed.
az storage account list -o tsv --query \
"[].[name, allowSharedKeyAccess, keyCreationTime.key1, keyCreationTime.key2]"
az ad app list --all --query '[].appId' -o tsv | while read -r id; do
az ad app credential list --id "$id" --query "[].[displayName, keyId, endDateTime]" -o tsv
done
az keyvault secret list --vault-name kv-example-prod -o tsv \
--query "[].[name, attributes.expires]"
gh secret list --org example-systems --json name,updatedAt,visibility
until s=$(aws iam generate-credential-report --query State --output text) && [ "$s" = COMPLETE ]
do [ -n "$s" ] || break; sleep 5; done # columns: user, then active and last rotated
aws iam get-credential-report --query Content --output text | base64 --decode \
| cut -d, -f1,9,10,14,15
Metrics that prove the program works
We found no published benchmark for most of these numbers, so the five below are our recommendation. The one measured figure close to this work, the 94 day median in the Verizon 2025 report, comes from one contributor with a sample of 141 and counts days to remediate, so we use it as a point of reference and never as a target.
| Metric | How to measure it | What has been published |
|---|---|---|
| Median and worst time to revoke | Alert created_at to the revocation in the record, not resolved_at, which is only closure |
No benchmark. Nearest is the 94 days above |
| Share of old secrets still valid | Revalidate on a schedule, or read validity |
GitGuardian: more than 64 percent of those valid in 2022 were still valid in January 2026. Vendor data |
| Share of credentials with an owner | Inventory rows with a named person | None found |
| Age of the oldest static credential | AWS access_key_1_last_rotated, Azure keyCreationTime |
A ceiling only: 90 days in Security Hub IAM.3 |
| Share of closures with a reason | Closed alerts with a resolution and a comment | None found |
Measure at the issuer, since closing alerts without revoking improves nothing else, and note that the IAM credential report omits service specific credentials, such as Bedrock long term API keys, so it cannot clear an account. In our example, ten findings close in a quarter, nine revoked in 4 hours and one in 41 days, so the mean says about 4 days, the median says 4 hours, and only the worst case shows the key that sat for six weeks, which is why we report the median and the worst together.
# Median and worst hours to revoke. One JSON line per finding from the closure records,
# each with id, created_at and revoked_at
# {"id":"SEC-1100","created_at":"2026-07-06T09:00:00Z","revoked_at":"2026-07-06T13:00:00Z"}
jq -cs 'map(((.revoked_at | fromdate) - (.created_at | fromdate)) / 3600) | sort | {findings:
length, median_hours: ((.[(length - 1) / 2 | floor] + .[length / 2 | floor]) / 2),
worst_hours: .[-1]}' revocations.json
# Sample output, for the ten findings above
# {"findings":10,"median_hours":4,"worst_hours":984}
The age of the oldest static credential comes from the seeding listing, since the last rotated columns of the IAM credential report and keyCreationTime on each storage account are recorded dates. Generally, we sort both and publish the oldest five by name, because a single age hides which key it belongs to, while a named key tends to find an owner and a date. Over time, watch the count of new findings each week more closely than the total.
Practice: a rotation drill
Once a quarter, we recommend rotating one credential on purpose, with the person on call at the keyboard, a second person keeping time and the runbook followed as written. In our example, app-billing-staging reads a client secret through a Key Vault reference to kv-example-staging. Beforehand, confirm that AuditEvent has reached the workspace for 30 days, and that the reference names no version, since a pinned one does not rotate.
Start the clock and read which callers use the secret, since a caller missing from the inventory row is the first finding. Note the old version and keyId, then add the new secret beside the old one with --append, for the reason in Entra application secrets, and store it as a new version. The command substitution drops the newline from -o tsv, and printf %s adds none, so --file stores the value alone. Refresh the reference, or App Service may wait 24 hours. Microsoft does not say that a running process sees the change, so prove that every reader moved while a miss still costs nothing, allowing up to 90 minutes for an app that holds a token, or restarting it with az webapp restart if you cannot wait. Last, delete the old credential, run the Entra client secret test from The negative test and expect AADSTS7000215, then disable the old version, which cannot be deleted alone, and stop the clock.
# The drill, steps 1 to 5. The KQL runs in Log Analytics, the rest in a shell
// Step 1. Start the clock, and read who really uses the secret
AZKVAuditLogs
| where OperationName == "SecretGet" and TimeGenerated > ago(30d)
| where Id contains "/secrets/billing-client-secret"
| summarize Reads = count(), Last = max(TimeGenerated)
by Caller = tostring(Identity), CallerIpAddress, HttpStatusCode
# Step 2. Note the old version and keyId, then add the new secret as a new version
OLD_VERSION=$(az keyvault secret show --vault-name kv-example-staging \
-n billing-client-secret --query id -o tsv); OLD_VERSION=${OLD_VERSION##*/}
az ad app credential list --id APP_ID --query "[].[keyId, displayName, endDateTime]" -o tsv
OLD_KEY_ID=KEY_ID # the keyId above that is in use today
umask 077
printf %s "$(az ad app credential reset --id APP_ID --append --years 1 \
--display-name drill-2026-q4 --query password -o tsv)" > ./new.txt
az keyvault secret set --vault-name kv-example-staging -n billing-client-secret \
--file ./new.txt --expires 2027-11-01T00:00:00Z -o none && rm ./new.txt
# Step 3. Refresh the reference
SITE_ID=$(az webapp show -g rg-billing-staging -n app-billing-staging --query id -o tsv)
REFRESH="$SITE_ID/config/configreferences/appsettings/refresh"
az rest --method post --url "https://management.azure.com$REFRESH?api-version=2022-03-01"
// Step 4. Prove that every reader moved
AADServicePrincipalSignInLogs
| where TimeGenerated > ago(2h) and AppId == "APP_ID"
| summarize Last = max(TimeGenerated) by ServicePrincipalCredentialKeyId, IPAddress, ResultType
# Step 5. Delete the old credential, take the old value for the negative test, disable its version
az ad app credential delete --id APP_ID --key-id "$OLD_KEY_ID"
CS=$(az keyvault secret show --vault-name kv-example-staging -n billing-client-secret \
--version "$OLD_VERSION" --query value -o tsv) # then the Entra test, without the read
az keyvault secret set-attributes --vault-name kv-example-staging -n billing-client-secret \
--version "$OLD_VERSION" --enabled false
| Check | The drill passes when |
|---|---|
| Old value | AADSTS7000215 comes back, saved, and a second person repeats it |
| Readers | Each signs in with the new keyId before the delete, and the callers match the row |
| The app | Health checks stay green, and no setting shows a literal @Microsoft. |
| Clock, record | Under 1 hour, the Critical clock, with record, runbook and inventory done that day |
The sample record below passed in 41 minutes and still found two problems: SecretGet showed three readers against the row’s two, and the runbook had no restart step.
# SEC-1290, a sample drill record. Passed
00:00 Clock starts. The row lists two readers.
00:06 SecretGet shows three: the app, func-export-staging and a laptop.
00:13 New secret added with --append, stored as a new version.
00:15 Reference refreshed. No sign-in yet, since the app holds a token.
00:21 App and func-export-staging restarted, both on the new keyId.
00:38 Old credential deleted. Old value read from its version.
00:41 Negative test: AADSTS7000215. Old version disabled.
Clock stops. Health stayed green.
Found A reader missing from the row, and no restart in the runbook.
A minimum secret policy
The ten lines below are our recommendation for a minimum policy, which a team can adopt as SECURITY.md. It is one of six files to copy in this article, with .gitignore, .pre-commit-config.yaml, CODEOWNERS, deploy.yml and the closure record template, and their versions and pins are as of November 2026, so check them first. Line 10 sets a minimum, so the suppression audit in Guardrails that stay on still runs monthly.
# SECURITY.md, our recommendation
01 No secret in source, config, tickets, chat or logs. If one lands there, it has leaked.
02 Prefer no secret: identity and federation first, a vault second, a CI store last.
03 Every credential has a named owner, an inventory row and a way to rotate it.
04 Every static secret has an expiry date and the narrowest scope that the issuer allows.
05 Push protection and a CI scan stay on. A bypass needs a reason and a reviewer.
06 Every finding is real until someone shows that it is not, and records why.
07 Revoke at the issuer: Critical 1 hour, High 24 hours, Medium 7 days, Low 30 days.
08 Unknown validity counts as live. Revoke first, and clean history later, if at all.
09 A finding closes on evidence: the old credential fails, and a second person checks.
10 Suppressions, this policy and the metrics are reviewed every quarter.
Token formats
A scanner can read four things in a token: its issuer and type, the random body, a checksum where the format has one, and the context around it. A checksum proves a well formed string, never a live token, as How a scanner decides explains. A shape marked “by scanner rules” comes from scanner rules, such as the Gitleaks default rules, rather than from the vendor’s documentation, and “Key holder only” means that only whoever holds the signing key can check the value, since its signature is a MAC. The shapes in A look without a scanner are a subset of these.
| Credential | Shape | Offline check |
|---|---|---|
| GitHub PAT and OAuth | ghp_, gho_, ghu_, ghr_ + 36 Base62, 40 total |
CRC32, last 6 |
| GitHub fine-grained PAT | github_pat_ + 82 word characters, 93 total, by scanner rules |
Not documented |
| GitHub App installation | New: ghs_APPID_JWT, about 520 characters. Legacy: ghs_ + 36 |
Legacy: CRC32 |
| GitLab PAT | glpat-, payload, ., version, 7 character CRC32. Legacy: glpat- + 20 |
Routable only |
| AWS access key | AKIA long term or ASIA temporary + 16. Secret: 40 Base64, by scanner rules |
None |
| Azure Storage key | 88 Base64 characters, by AccountKey=. Newer form: +ASt at offset 76 |
Newer form |
| Azure SAS | Query string with sv, sr, sp, se, sig |
Key holder only |
| Entra client secret | 3 characters, then 7Q~ or 8Q~ at index 3, 37 or 40 total |
8Q~ only |
| Azure DevOps PAT | 84 characters, AZDO at positions 76 to 80. Legacy: 52 characters |
Yes, Marvin32 |
| Slack | xoxb- bot, xoxp- user, xapp- app, and hooks. URLs |
Not documented |
| Stripe | sk_live_, rk_live_, sk_test_, rk_test_, and sk_org_ for an organization. Public: pk_live_, pk_test_ |
Not documented |
| Google API key | AIza + 35, 39 total, by scanner rules |
None |
| npm, PyPI | npm_ + 36, 40 total. pypi- + Base64 macaroon, at least 85 characters |
npm_, CRC32 |
| OpenAI | sk-proj-, sk-svcacct-, sk-admin-, marker T3BlbkFJ, by scanner rules |
None |
| Anthropic | sk-ant-api03-, sk-ant-admin01-, by scanner rules |
Not documented |
| SendGrid, Twilio | SG. + 22 + . + 43, by scanner rules. Twilio SIDs AC or SK + 32 hex, and a 32 character secret |
None |
| HashiCorp Vault | hvs. service, hvb. batch, hvr. recovery. Older: s., b., r. |
None |
| JWT | Three Base64url parts, usually starting eyJ |
Public key, or key holder if MAC |
| Private keys, URLs | -----BEGIN PRIVATE KEY-----, postgres://, https:// |
None |
Note. Every shape and check here is as of November 2026, and “Routable only” means that only the newer, routable GitLab form, the one that ends in a CRC32, has an offline check.
Standards map
For the security leads who map this work to their own controls, the first table below gives identifiers and titles only, as of November 2026. SSDF v1.1 never mentions hard coded secrets, so its entries support the practice without requiring it, and PCI DSS 8.6.2 covers passwords on accounts usable for interactive login, not every API key. The PCI DSS and ISO/IEC 27002 entries rest on secondary sources, since both documents are gated.
| Source | Identifiers and titles |
|---|---|
| CWE 4.21 | CWE-798 Use of Hard-coded Credentials. CWE-259 Use of Hard-coded Password. CWE-321 Use of Hard-coded Cryptographic Key. CWE-312 Cleartext Storage of Sensitive Information. CWE-522 Insufficiently Protected Credentials. CWE-532 Insertion of Sensitive Information into Log File. CWE-540 Inclusion of Sensitive Information in Source Code. CWE-615 Inclusion of Sensitive Information in Source Code Comments. |
| OWASP Top 10:2025 | A01 Broken Access Control. A02 Security Misconfiguration. A04 Cryptographic Failures. A06 Insecure Design. A07 Authentication Failures. A09 Security Logging and Alerting Failures. |
| OWASP ASVS 5.0.0 | V13.3 Secret Management, in V13 Configuration, verifications 13.3.1 to 13.3.4. |
| OWASP NHI Top 10 | NHI2:2025 Secret Leakage. NHI5:2025 Overprivileged NHI. NHI7:2025 Long-Lived Secrets. |
| OWASP CI/CD Top 10 | CICD-SEC-4 Poisoned Pipeline Execution. CICD-SEC-6 Insufficient Credential Hygiene. |
| OWASP Secrets Management | 3.2.1 As part of your CI/CD tooling, 9 Incident Response. |
NIST SP 800-53 Rev. 5, Release 5.2.0 |
IA-5 Authenticator Management. IA-5(7) No Embedded Unencrypted Static Authenticators. SC-12 Cryptographic Key Establishment and Management. SC-28 Protection of Information at Rest. |
| NIST, other | SP 800-218 SSDF Version 1.1, tasks PO.5.1, PO.5.2, PS.1.1, PW.1.2, PW.5.1, PW.9.1. SP 800-204D, Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines, COMMIT-REQ-1, COMMIT-REQ-2, DEPLOY-REQ-1. SP 800-57 Part 1 Rev. 5, Recommendation for Key Management. SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile. |
| MITRE ATT&CK v20.1 | T1552 Unsecured Credentials: .001 Credentials In Files, .003 Shell History, .004 Private Keys, .005 Cloud Instance Metadata API, .007 Container API, .008 Chat Messages. T1528 Steal Application Access Token. T1078.004 Valid Accounts: Cloud Accounts. |
| PCI DSS v4.0.1 | Requirements 8.6.1, 8.6.2 and 8.6.3, on application and system accounts and their passwords. |
| ISO/IEC 27002:2022 | 5.17 Authentication information. 8.4 Access to source code. 8.24 Use of cryptography. |
Each false assumption also carries its own references to the same taxonomies, listed below by its number in Twelve false assumptions, and two of them, CWE-526 and NHI8:2025, sit outside the map above.
| No. | References |
|---|---|
| 01 | CWE-522 / CICD-SEC-4, CICD-SEC-6 |
| 02 | CWE-532 / CICD-SEC-6 |
| 03 | CWE-798 / CWE-540 |
| 04 | CWE-540 / NHI2:2025 |
| 05 | NHI7:2025 / T1528 |
| 06 | CWE-540 / T1552.001 |
| 07 | CWE-312 / CWE-522 |
| 08 | CWE-526 |
| 09 | CWE-798 / CICD-SEC-6 |
| 10 | CWE-798 / NHI8:2025 |
| 11 | CWE-522 / NHI5:2025 |
| 12 | CWE-312 / T1552.001 |
Where to start
From a standing start, stop new leaks, clear the backlog, then replace what remains and prove it. In the first 30 days, that means push protection across the organization, a CI gate and hooks, and a first sweep, verified and unknown findings first, with an owner on each of them. Those findings are revoked inside the response clock as the sweep finds them. By day 60, the unverified ones are worked through too, an inventory covers every production credential, and the log exports run. By day 90, at least the ten riskiest static secrets have moved up the ladder, one drill has run, and the first metrics, such as the share of credentials with an owner, have gone to a named reader.
Each horizon has its evidence. By day 30, the weekly push protection listing in Guardrails that stay on prints nothing and every row of the sweep has a name, and by day 60, no open alert reads validity=active or validity=unknown, and a log query returns yesterday’s rows. By day 90, ten closure records, one drill record and one metrics report are on file. Although we found no published benchmark for most of these metrics, report the median time to revoke beside the worst, as Metrics that prove the program works explains.
A program is working when the next leak is blocked or found quickly, lands on a named owner, and shows up in a number that someone reads. Teams that reach that point tend to remove secrets where the other side allows it, close findings only on evidence, and give every check a reader. Leaks will still happen, but such a team finds each one sooner, ends it at the issuer and can show that it is dead.
References and further reading
These are the defining primary sources for each topic in this article, with every link checked on November 7, 2026.
- Kinds and scale of secrets: RFC 6750, GitGuardian’s 2026 report and Verizon’s 2026 DBIR.
- CI/CD secret stores: GitHub, GitLab, Azure Pipelines, Jenkins, TeamCity, CircleCI, Bitbucket, CodeBuild, Bamboo, Kubernetes, EKS and Argo CD, with OWASP CICD-SEC-6.
- CI incidents: Codecov, Travis CI, reviewdog, tj-actions with CISA’s alert, the npm campaign with GitHub on the next one, GhostAction, Trivy, Axios, TanStack and Nx Console.
- Checking your own pipeline: GitHub workflows and rules, CodeBuild BatchGetProjects, Kubernetes encryption at rest, Jenkins controller isolation and the GitLab variables API.
- CI platform details: GitHub’s Actions secrets API, reusable workflows, branches API and environments API, GitLab’s secrets.yml.example, members API and glab api, Azure Pipelines on secure files, the secure file task and the agent source, Jenkins on backing up, using credentials, CredentialsScope and hudson.model.Item, TeamCity on typed parameters, SSH keys and its data directory, Bitbucket on OIDC, Bamboo on variables, Specs encryption and ephemeral agents, and Kubernetes on managing Secrets with kubectl.
- Private repositories, forks and token revocation: GitHub on token revocation, forks and visibility, the forks API and the organization forking policy, GitHub’s October 2026 secret scanning changelog, and Truffle Security on cross fork object references.
- Deleted commits and force pushes: GitHub on removing sensitive data, Truffle Security on oops commits, Unit 42 on EleKtra-Leak and Clutch Security on the rotation illusion.
- What survives a rotation: AWS on temporary credentials and role sessions, Microsoft on user delegation SAS and CAE, ALTER LOGIN, PostgreSQL admin functions and OpenID Connect Core.
- Where secrets hide: docker image history, the BuildKit Dockerfile frontend, ECMA-426, Android app size and Apple’s Bundle.
- Scanners and GitHub secret scanning: GitHub secret scanning and its supported patterns, the TruffleHog and Gitleaks READMEs, Betterleaks and Pro Git on Git hooks.
- Triage and validation: Basak and colleagues, GitHub on evaluating alerts and AWS GetAccessKeyInfo.
- Containment and incident response: NIST SP 800-61r3, GitHub on remediating a leaked secret and Entra data retention.
- Rotation by credential: AWS on updating access keys, GitHub’s revocation API, Slack’s auth.revoke and RFC 8725.
- The hunt: CloudTrail LookupEvents, StorageBlobLogs and AADServicePrincipalSignInLogs.
- History cleanup: the git-filter-repo manual.
- Replacement, from Key Vault to federation: Key Vault RBAC, managed identities, GitHub’s OIDC reference and Entra federated credentials.
- Kubernetes, Docker, Terraform, SOPS and front ends: Kubernetes, Docker, Terraform, SOPS and Vite.
- Proof and keeping it fixed: Azure Storage and Entra error codes, GitHub security configurations and Verizon’s 2025 DBIR.
- Standards: OWASP on secrets management, the OWASP Non-Human Identities Top 10, ASVS 5.0, NIST SP 800-53r5 and MITRE ATT&CK T1552.