Hey all, been knee-deep in vendor evaluations for a new agent orchestration layer. We're sifting through the usual security docs and questionnaire responses, and honestly, most of it feels like a checklist exercise.
Everyone claims they have a "robust security posture" and do "regular penetration testing." But when you ask for proof, it's always a sanitized executive summary or a generic attestation letter. I want to see the *actual* red team findings—the raw, ugly vulnerabilities they found and how they were remediated. The good, the bad, and the messy.
Has anyone here ever gotten a vendor to share something concrete? Like a de-identified excerpt from a pentest report showing a real CVE or a logic flaw they had to fix in their API? I'm not asking for their crown jewels, just proof that the testing has teeth.
I'm trying to move beyond "yes/no" answers to understand their actual security culture. If they're building with LangChain or custom function-calling agents, I want to know if the red team looked at prompt injection risks, tool misuse, or data exfiltration through the agent's outputs. That stuff is gold for understanding real risk.
-- lena
-- lena
You're hitting on a huge problem. The sanitized summary is useless for evaluating the real security culture. It just tells you they passed a test, not how they handle real flaws.
I have gotten raw findings, but only after signing a very specific NDA and pushing the conversation past legal. The useful part wasn't the CVE details - it was the timeline from finding to fix, and the conversation around whether they considered it a "real" risk. One vendor showed us a logic flaw where an agent could be coerced into calling internal tools with escalated parameters. The fix was in the control plane's validation logic, not the agent itself. That told me they understood the architecture's trust boundaries, which was more valuable than the flaw itself.
For your use case with LangChain or custom agents, you might have better luck asking for their *testing methodology* instead of the findings. Ask, "Can you share the scope document from your last red team engagement, specifically covering the agent's interaction with tools and the orchestration layer?" If they can't articulate what they test, the findings are moot.