Detecting secrets in the wild is only the first step. A scanner that flags every string matching a credential pattern is useful, but it leaves a critical question unanswered: does this secret actually work? And a related one: is this really a secret at all, or just a placeholder left in the docs? With this release, SecretsBuster answers both questions automatically, for every secret it finds.
The problem with detection alone
Pattern-based secret detection generates a list of matches — but a match is not a finding. Two questions immediately follow every detection: is this credential still active, and is it even a real credential to begin with? Scanners have historically left both questions for humans to answer manually.
The false positive problem is particularly corrosive. Strings that look like secrets but aren't are
everywhere: example keys from documentation, template placeholders like
<YOUR_API_KEY_HERE>, dummy values hardcoded in test fixtures, or canonical examples
that providers publish in their own tutorials. When these land in a report alongside real exposures,
the signal-to-noise ratio drops. Teams start skimming findings rather than investigating them, and real
credentials get dismissed alongside the noise.
The two features shipped in this release address both questions directly: live validation answers whether a detected secret is still active, and confidence scoring answers whether it was likely a real secret in the first place.
Live validation: is this secret still active?
For supported secret types, SecretsBuster now actively tests each detected credential against its provider's API immediately after a scan completes. The validation result is stored alongside the secret and displayed in the report dashboard.
There are four possible outcomes:
- Valid — the secret was accepted by the provider's API with active privileges. This is the highest-severity outcome: the credential is live and exploitable right now.
- Invalid — the secret was rejected. It may have been rotated, expired, or simply never been a real credential. Lower urgency, but worth reviewing in case rotation was recent.
- Validation error — something went wrong during the check (network issue, provider rate limiting, unexpected response). The secret may still be active; manual verification is recommended.
- No validator — SecretsBuster does not yet have an automated validator for this secret type. The detection still stands; automated validation is not available for every provider.
Validation is performed asynchronously after the scan, so it never slows down the initial detection phase. Results appear in the report as soon as they are ready.
Prioritising a remediation queue becomes straightforward: start with Valid findings, rotate those credentials immediately, then work through the rest. Teams no longer have to treat every detected secret as equally urgent.
Confidence scoring: is this really a secret?
The second feature addresses the false positive problem with a purpose-built classifier that runs on every detected secret, regardless of whether a live validator exists for its type.
The classifier produces a true positive confidence score between 0 and 100%, indicating how likely the detected value is a genuine credential rather than a placeholder, example, or test string. The score is derived from a set of features extracted directly from the secret value:
- Entropy — real credentials tend to have high Shannon entropy; human-readable placeholders do not.
- Character composition — ratios of digits, uppercase letters, lowercase letters, special characters, and non-ASCII bytes all carry signal about whether a string was machine-generated or hand-typed.
- Template markers — patterns like
<YOUR_KEY>,${VAR},,%ENV_VAR%, and__PLACEHOLDER__are strong indicators of a false positive. - Placeholder vocabulary — the classifier tokenises the value and checks it against a vocabulary of words commonly found in fake credentials: test, example, dummy, changeme, admin, null, and equivalents across several languages.
- Provider example patterns — known canonical examples published by providers
themselves are matched explicitly: AWS's
AKIAIOSFODNN7EXAMPLE, OpenAI'ssk-xxxxstubs, Stripe's4eC39HqLyjWDarjtT7zdGnFm, and others. - Sequential and repeated runs — strings like
abcdeforaaaaaaaascore very differently from cryptographically random output. - Language trigrams — a human-readable string leaves a trigram fingerprint that diverges sharply from what a random token generator produces.
The score maps to three labels shown in the dashboard:
- High confidence (≥ 50%) — the value strongly resembles a real credential.
- Medium confidence (25–49%) — ambiguous; worth a closer look.
- Low confidence (below 25%) — likely a placeholder or example value.
One deliberate design choice: when a secret passes live validation, its confidence score is shown as 100%. There is no ambiguity when a credential has been confirmed active against a real API. The classifier score is a heuristic; a successful validation is ground truth.
What this looks like in practice
Open any scan report in the SecretsBuster dashboard and each detected secret now shows a row of badges beneath its value. A green Valid chip alongside a High confidence chip identifies the findings that need immediate action. A red Invalid chip next to a Low confidence chip is the clearest signal that something is likely noise. Every badge carries a tooltip explaining exactly what it means and how to interpret it, so the context is always one hover away.
Secrets that appear in multiple scan reports also surface a duplicates indicator with a dropdown listing the other affected targets. A credential that is hard-coded and consistently exposed across scans is a different kind of problem than one that appeared once in a single file.
Why this matters
Secret detection is a solved problem in broad strokes: pattern matching against known formats works well, and the space of credential shapes is well-documented. What has historically been harder is everything that comes after the match: Is it real? Is it active? Should I wake someone up at 2 a.m. to rotate it?
Validation and confidence scoring turn a list of pattern matches into an actionable risk queue. The findings that matter most are immediately obvious, and the ones that don't are flagged rather than silently mixed in. Over time, that distinction is what keeps security teams engaged with scanner output rather than tuning it out.
Both features are live for all new SecretsBuster scans.