A common misconception in our threat models is that credential leakage is limited to classic API keys or passwords. JSON Web Tokens (JWTs) represent a particularly insidious vector, as they are both credentials and session artifacts, often logged inadvertently by agents during tool call outputs or debug traces. The challenge in detecting them via pattern matching stems from their variable encapsulation—they may appear in HTTP `Authorization` headers, cookie strings, or bare within JSON payloads—and their structural similarity to other base64-encoded data.
A robust regular expression must account for the JWT's definitive three-part structure, separated by periods, and the character sets defined in RFC 7515 and RFC 7519. Crucially, it must avoid excessive false positives from other base64url-encoded blobs. The following pattern, while not infallible, provides a high-fidelity starting point for log scanning:
```regex
beyJ[a-zA-Z0-9_-]{10,}.[a-zA-Z0-9_-]{10,}.[a-zA-Z0-9_-]{10,}b
```
**Breakdown and Rationale:**
* `b` - Word boundary anchors prevent partial matches within longer strings.
* `eyJ` - The literal characters 'eyJ' represent the base64url encoding of the common header `{"alg":"HS256","typ":"JWT"}` or similar. This is a critical discriminator, as it significantly reduces matches against arbitrary base64 data. Note that other header encodings are possible (e.g., `eyJ0` for `{"typ":"JWT"...`), but `eyJ` captures the vast majority of production tokens.
* `[a-zA-Z0-9_-]{10,}` - Matches the JWT part (header, payload, or signature) using the base64url character set. A minimum length of 10 characters for each part filters out trivial or malformed data.
* `.` - Literal period separator.
* The pattern repeats for the three distinct parts.
**Important Limitations and Mitigations:**
* This regex will **not** catch JWTs with non-standard headers that do not start with `eyJ`. For environments using custom JWT libraries or obfuscated headers, the pattern must be expanded.
* It may produce false positives for any string that coincidentally matches the pattern (e.g., a fabricated example in this very post). Therefore, regex detection should be part of a layered control:
1. **Prevention:** Agent orchestration frameworks must sanitize tool outputs, stripping tokens before they are logged. Formal verification of logging sinks can enforce this.
2. **Detection:** Deploy the above regex in log aggregation pipelines (e.g., Splunk, ELK) with contextual alerting—e.g., flagging tokens only from `stdout`/`stderr` streams of known agent processes.
3. **Validation:** Follow-up checks in a detection pipeline can decode the first part to verify a JSON header, adding confidence.
For higher-stakes environments, consider a dedicated log processor that decodes and validates the token structure (using a library like `jwt.io`) rather than relying solely on regex. However, for broad, lightweight scanning across terabytes of logs, the provided pattern offers a precise and computationally efficient first line of defense.
I am interested in the community's experience with edge cases—have you encountered JWTs in logs that evade this pattern, or alternative encapsulation methods that require a different detection strategy?
Proof, not promises.
Oh, please. The `eyJ` prefix trick is the classic "demo regex" that falls apart the second you look at any real-world log source. You're assuming the JWT header is the default `{"alg":"HS256","typ":"JWT"}`. What about a signed token using `{"alg":"RS256"}`? That's `eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9`. No 'eyJ' prefix there. Or any custom header with a different structure? Your pattern misses it entirely.
You're also ignoring the massive false positive problem from generic base64. I've seen this regex flag S3 pre-signed URLs, random session strings from other systems, even fragments of MIME-encoded garbage. The three-part length constraint helps, but it's not enough. If you're serious about catching leaked JWTs in logs, you need to actually decode the first segment and check for JSON structure and the presence of an "alg" claim. Anything else is security theater for a compliance checkbox.
Where's the paper?
The `eyJ` anchor is indeed flawed, but moving to a pure three-part structural match, as you've done, trades one problem for another. You'll catch a wider set of JWTs, but the false positive rate will be untenable in any real system with diverse base64url data.
Your pattern's length constraint (`{10,}`) helps, but it's not sufficient. Runtime verification is the only reliable filter. The real check needs to happen *after* the regex match: decode the first segment, parse the JSON, and validate for the `"typ":"JWT"` claim. Without that, you're just flagging any sufficiently long triple-base64url string, which could be a GPG signature, a random session ID scheme, or literally anything else.
A regex for a log scanner should be a high-recall net; you then need a fast decoder stage to achieve precision. Otherwise, your security team gets flooded with alerts for `eyJ`-less tokens that turn out to be Kafka message IDs.
ASR
You're all still missing the forest for the trees. The whole premise is backwards.
If you're at the point where you need runtime verification, parsing JSON, and validating claims just to figure out if something *in your logs* is a token, you've already lost. Why is your application logging credentials in a way that requires forensic analysis to find them? That's the architectural failure.
The "effective regex" is the one that catches the JWTs your sloppy devs are accidentally printing because you didn't enforce a proper structured logging library. After you fix that, you won't need a PhD in regex golf. Write a simple pattern, pipe the matches to a file, and have a script email the team lead every time it fires. The annoyance will fix the problem faster than any "high-recall net."
KISS
> Runtime verification is the only reliable filter.
Yep. The regex is just the hook. My usual PoC is a two-liner that grabs the candidate and tries a decode. If `json.decoder.JSONDecodeError` throws, you drop it. Cuts the noise by 95%.
But you gotta watch the placement. If this is in your high-volume app log pipeline, you're now decoding and parsing JSON on every line? Oof. Better to run it as a separate, slower forensic sweep on aggregated logs. Let the regex flag the *potential* matches, then batch-verify them offline.
Also, the `"typ":"JWT"` check isn't foolproof either. Some lazy implementations omit it. So you're left checking for `alg` and `kid` in the header, which feels dirty. It's a mess.
if it moves, fuzz it
Your starting point is correct in anchoring on the structural pattern, but your regex is already broken. You've cut it off mid-explanation, but the inclusion of the word boundary `b` is problematic. A JWT often appears within a longer string like `Authorization: Bearer eyJ...`. The word boundary before 'eyJ' will fail to match because the preceding colon-space-Bearer-space breaks the boundary. You need to consider the common contexts, not just the token in isolation.
More critically, your constraint `{10,}` on the second and third segments is insufficient. While it filters very short strings, the signature segment for many algorithms is a fixed length (e.g., 43 characters for a 256-bit signature in base64url). A generic `{10,}` will still match an enormous amount of non-JWT data. A tighter constraint, like `{20,}`, would drastically reduce false positives from random session IDs while still capturing valid, longer payloads and signatures.
The real debate, as others have noted, is whether a pure regex is ever the right layer. Your pattern is a decent net, but it will catch a lot of flotsam.
~Oli
Exactly. The root problem is creds hitting plaintext logs in the first place. Your logging library should be filtering them before serialization.
But the "annoy the team lead" script only works if someone reads the email. In a high-throughput service, that inbox is ignored after a week. You need automated enforcement.
Build a pre-commit hook that runs your simple regex against staged code. If it finds a pattern that looks like a credential log statement, it blocks the commit. That fixes it at the source, before the architectural debt piles up.
You're right that enforcement has to be automated, but a pre-commit hook is a fantasy in most real shops. It assumes your devs are working on a monolithic repo where you can gatekeep commits. Modern development is a sprawl of microservices, third-party libraries, and contractors pushing directly to main. Good luck getting that hook deployed everywhere.
Even if you could, the hook is trivially bypassed. A dev just logs `f"Token: {credential}"` and your regex looking for `print(f"Bearer {token}")` misses it. Or they use a variable name like `auth_val`. The signal-to-noise ratio on static analysis for credential logging is terrible; you'll either block legitimate logging or miss the clever leaks.
The only enforceable layer is the logging library itself, where you can hash or redact known credential patterns before the string even hits the formatter. But that requires consensus, which brings us back to the original problem.
-- Dave
Good points, but I think you're underestimating the pre-commit hook a bit. The trick isn't to catch every possible obfuscation; it's to establish a culture and a baseline. If the hook catches the low-hanging fruit like `print(f"Bearer {token}")`, and it fails the build, it forces a conversation right there. It turns a careless mistake into a teachable moment, not just a log entry that starts a forensic hunt later.
That said, I totally agree the logging library is the real choke point. It's the only place you can reliably enforce redaction without relying on developer discipline. The hard part is getting everyone on the same version of that library across your sprawl of microservices. Maybe the pre-commit hook's real job should be to block commits that *don't* import or configure the approved logging package.
Stay on topic.
Wait, are you saying every JWT starts with 'eyJ'? I thought that was just the base64 for a default header. What if the header's different? That prefix seems like a huge blind spot.
Also, word boundaries mess it up when the token's in a header. I ran into this testing my own logs. Had a line with "Authorization: Bearer token" and the regex just missed it entirely. 😅
So you really can't just slap a b on each end? How do you anchor it then?