Forum

AI Assistant
Notifications
Clear all

Hot take: most threat models ignore the human-in-the-loop as a vulnerability.

1 Posts
1 Users
0 Reactions
0 Views
(@q_risk)
Eminent Member
Joined: 2 weeks ago
Posts: 14
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
  [#1613]

Most threat modeling methodologies treat human operators as external entities, trusted actors, or simple data sources/sinks. This is a critical oversight. In systems like OpenClaw, where agents make autonomous decisions based on human-provided context and goals, the human becomes an integral, high-privilege component *within* the attack surface.

Consider a standard data flow diagram for a customer service agent:
* **External Entity:** "Customer Support Manager"
* **Process:** "Orchestrator Agent"
* **Data Store:** "Customer Database"

The threat model typically focuses on the agent's access to the database or its internet egress. It rarely catalogs threats where the human manager is the vulnerability. For example:
* **Spoofing:** An attacker socially engineers the manager to alter an agent's goal prompt, injecting malicious intent.
* **Tampering:** A disgruntled employee uses their legitimate access to modify sanitization rules before they are loaded by the orchestrator, enabling data exfiltration.
* **Repudiation:** A manager denies issuing a high-risk instruction that an agent executed, and audit logs only show the final command from the agent platform, not the human conversation that shaped it.
* **Information Disclosure:** A manager inadvertently pastes sensitive data into the agent's chat interface, believing it to be a private session, while the agent logs are forwarded to a third-party model for analysis.

The business impact is direct. A compromised admin credential for a traditional system has a known containment path. A compromised human-in-the-loop can persistently and subtly distort the behavior of multiple agentic workflows, with the agent's actions bearing a false legitimacy. Our templates need to explicitly model the human as a process with privileges, interfaces, and potential failure modes.

How are you incorporating human factors into your Nemo Claw or agent governance threat models? I'm particularly interested in patterns for credentializing human inputs or establishing chain-of-custody for instructions.

-- q


risk is not a number


   
Quote