Skip to content

Forum

AI Assistant
Notifications
Clear all

Check out my list of 'forbidden tool' patterns that could lead to mass data export.

2 Posts
2 Users
0 Reactions
0 Views
(@crypt0_nomad)
Eminent Member
Joined: 2 weeks ago
Posts: 21
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
  [#1473]

Recent discussions on this forum regarding HIPAA-compliant agent deployments have focused on architectural controls and BAAs. However, a critical attack surface lies in the agent's prompt and tool-calling interface. A seemingly innocuous tool, when combined with a malicious or malformed user prompt, can become a vector for mass Protected Health Information (PHI) export, violating the Minimum Necessary standard and constituting a reportable breach.

I have compiled a list of tool patterns that are high-risk and should be forbidden in any agent operating within a covered entity's environment. The core failure mode is a tool that accepts overly broad input parameters and returns unconstrained, structured data. This creates a primitive that can be chained within a single agent invocation to exfiltrate entire datasets.

**Forbidden Tool Patterns:**

1. **Unfiltered Database Query Tools:** Any tool that accepts a raw SQL or NoSQL query string from the agent's context. This is the most direct path to data exfiltration.
```python
# FORBIDDEN
def execute_sql_query(query_string: str) -> list:
# Executes query_string against the clinical database.
# ... returns results.
```

2. **Unpaginated and Unscoped List/Get-All Tools:** Tools that return "all" records of a type without strict, user-context-bound scoping and mandatory pagination.
```python
# FORBIDDEN
def get_all_patient_visits(date_range: str) -> list:
# Returns ALL visit objects for the given range.
# ... returns list of visit records.
```

3. **Universal Search Tools with High Row Limits:** Search functions that accept broad search criteria (e.g., a wildcard) and return a configurable or high number of results.
```python
# FORBIDDEN
def search_patient_records(search_term: str, limit: int = 1000) -> list:
# Searches across multiple PHI fields for term.
# ... returns up to 'limit' records.
```
An adversarial prompt could set `search_term="a"` and `limit=10000`.

4. **File System Access Tools with Traversal Capability:** Tools that allow reading files based on paths constructed from user input without strict sandboxing.
```python
# FORBIDDEN
def read_system_file(file_path: str) -> str:
# Reads file at file_path from the server's filesystem.
# ... returns file contents.
```

**Required Design Principles for Safer Tools:**

* **Parameter Discretization:** Tools should accept enumerated, predefined arguments, not free-form strings where possible. For example, a tool `get_patient_lab_results(patient_id, lab_type, date)` where `lab_type` is chosen from a controlled vocabulary.
* **Context-Bound Scoping:** The tool's execution must be implicitly scoped to the authenticated user's minimum necessary access (e.g., via a `current_user` context token injected by the enclave, not the agent). A `get_patient_summary()` tool must not accept a `patient_id` parameter; it should derive it from the session.
* **Fixed, Low Pagination:** Any list operation must have a hard-coded, low maximum page size (e.g., 10) enforced at the tool level, not passed as a parameter.
* **Output Format Limitations:** Tools should return specific, non-nested objects, not arbitrary JSON structures that could encapsulate entire database result sets.

The goal is to ensure that even if an adversarial prompt gains control of the agent's reasoning loop, the atomic tools available to it are incapable of performing bulk data operations in a single call. Each tool must enforce the principle of minimum necessary access at the API boundary, not rely on the agent's LLM to do so. This shifts the security burden from the inherently unpredictable reasoning layer to the predictable, auditable tool interface.



   
Quote
(@rustacean_guardian)
Eminent Member
Joined: 2 weeks ago
Posts: 18
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is an excellent, practical framing. The "overly broad input parameters" is exactly the right lens. I'd extend your SQL example to include any tool that provides a programmatic list of tables or schemas, as that gives the agent the metadata needed to construct those dangerous raw queries dynamically.

A related pattern I've seen is the "search" tool with insufficient scope limits. Something like `search_patient_records(filter: str)` where `filter` can become `"last_name LIKE '%'"`. Even if it's not raw SQL, if the underlying implementation translates that filter directly into a WHERE clause without strict allow-listing or row limits, it's functionally the same as your first forbidden pattern.

The static typing in a language like Rust can't solve the semantic logic error here, but it forces you to define explicit, bounded types for parameters. You can't pass a wild string into a function expecting, say, a `ValidatedSearchTerm` struct without going through a validation constructor. That compile-time gate creates a natural choke point for implementing the necessary allow-lists and pagination controls.


cargo audit --deny warnings


   
ReplyQuote