Forum

Notifications
Clear all

Does anyone have a working threat model for Claude Code in a regulated environment?

3 Posts
3 Users
0 Reactions
12 Views
(@ml_sec_guy)
Active Member
Joined: 2 months ago
Posts: 12
Topic starter   [#1857]

I've been digging into the Claude Code integration within Cursor, specifically for teams that handle PHI, financial data, or other regulated information. The convenience is undeniable, but the compliance surface area feels... murky.

The core questions I'm wrestling with:
* **Data Retention & Transit:** When Claude Code processes a highlighted function or error, where exactly does that code snippet go? Is it anonymized, and what's the retention policy? The standard Claude API terms might not fully cover this IDE-integrated use case.
* **Indexing Boundaries:** Cursor's own codebase indexing is local, but when you invoke Claude for a deeper refactor or explanation, does that query contain indexed context? If so, you could be sending a significant portion of your proprietary or sensitive code graph to Anthropic's servers.
* **Prompt Leakage:** This is a classic adversarial input problem, but in a new context. Could a seemingly benign comment in a code file (e.g., `// TODO: validate user SSN: 123-45-6789`) be sent to the model if that block is selected for help? The threat is accidental exposure via developer action.

A basic threat model starter might categorize risks by data flow:

```yaml
Threat Actors:
- Malicious Extension (compromised or rogue)
- Anthropic Infrastructure (insider risk, data breach)
- Developer (accidental data selection)

Data Assets:
- Proprietary source code
- Embedded secrets (API keys, credentials in comments)
- Regulated data (PHI, PII) in code or strings

Trust Boundaries:
1. Local Cursor process Cursor's telemetry/update servers
2. Local Cursor process Anthropic's API endpoints (Claude Code)
3. Local index (Chroma/vector DB) Model context window
```

Has anyone formalized this further or run actual traffic analysis (e.g., mitmproxy on Cursor's outbound calls) to see the exact payload structure? I'm particularly concerned about model extraction risks here too—could repeated use by a team allow an adversary at Anthropic to gradually reconstruct a proprietary codebase?


Don't trust the model


   
Quote
(@kernel_jane)
Eminent Member
Joined: 2 months ago
Posts: 23
 

You're right to zero in on indexing boundaries, that's the most subtle and dangerous data exfiltration path. The local index isn't just a lookup table; it's a semantic graph of your entire codebase relationships. When Claude Code asks for "context," the integration layer decides what slices of that graph get serialized into the prompt. There's rarely a hard, auditable limit on token count or graph depth for these auxiliary context fetches.

This creates a scenario where a request to "explain this database connection function" could, through the indexing engine's relevance algorithms, pull in adjacent functions containing hardcoded credentials or schema structures with real data shapes, all bundled into a single outbound request. The threat isn't just the selected code block, it's the latent context attached by the tool's own heuristics.

Your point about adversarial input extends to the index itself. Comments, string literals, configuration values, and test data residing anywhere in the indexed codebase become potential prompt leakage vectors, not by direct selection but by being semantically "near" the developer's actual question. You need to model the entire workspace as the potential data source, not just the active buffer.


All bugs are shallow if you read the kernel source.


   
ReplyQuote
(@kernel_auditor_rae)
Active Member
Joined: 2 months ago
Posts: 17
 

You've correctly identified the three primary exfiltration channels. The retention policy question is particularly thorny because even if Anthropic purges the data after 30 days, as is common, the *transient* copy in memory during processing is still a regulatory event. For PHI under HIPAA, that's a disclosure requiring accounting.

Your point about the standard API terms not covering the IDE integration is the critical legal gap. The integration acts as a data controller with its own logic for what constitutes a "query," and that logic isn't typically part of the third-party API agreement. You'd need to audit the Cursor-to-Anthropic middleware layer, which is often a black box.

I'd add a fourth category to your starter model: **Kernel-level isolation failures**. If the tool is running on a developer's workstation, a compromised or malicious VS Code/Cursor extension could bypass the intended UI controls and send arbitrary file contents via the same authenticated channel. A proper model must consider the process boundary and syscall filtering for the IDE itself.


Audit everything, trust no syscall.


   
ReplyQuote