We've been running our own agent fleet for a while, and the alerting was always one-way: agent events go to the SIEM. That creates a blind spot for the orchestration system. If an agent starts throwing critical security events in Splunk, the platform managing that agent has no idea.

I built a simple webhook listener to close that loop. The goal: push high-severity, agent-specific SIEM alerts back into our orchestration tool's API. This lets the platform take automatic, cautious remediation steps or at least flag the host for manual review.

The flow is straightforward:
1. SIEM (we use Splunk) triggers an alert based on agent runtime events (e.g., "unexpected module load," "secret retrieval spike," "process ancestry anomaly").
2. A small webhook app (Flask in this case) receives the alert, validates a shared secret token, and parses the critical fields.
3. The app maps the alert to a host identifier, then makes a POST to our orchestration tool's API to update the host's status and trigger a playbook.

Key considerations I had to address:
* Blast radius: The webhook only accepts alerts from a specific, hardened Splunk search head IP.
* Rollback: Any action taken by the orchestration tool is logged and can be reverted via a separate playbook. The webhook itself only triggers "investigative" or "containment" tags, not immediate termination, unless the alert confidence is extremely high.
* Rate limiting: The webhook has a simple in-memory counter to prevent alert storm issues.

Here's the core of the webhook logic for validation and forwarding:

```python
from flask import Flask, request, jsonify
import requests
import os

app = Flask(__name__)
ORCHESTRATOR_URL = os.getenv('ORCH_URL')
WEBHOOK_SECRET = os.getenv('WH_SECRET')
SPLUNK_SOURCE_IP = os.getenv('SPLUNK_SH_IP')

@app.route('/webhook/siem_alert', methods=['POST'])
def siem_webhook():
# Validate source IP
if request.remote_addr != SPLUNK_SOURCE_IP:
return jsonify({"error": "unauthorized source"}), 403

# Validate shared secret
if request.headers.get('X-Webhook-Token') != WEBHOOK_SECRET:
return jsonify({"error": "invalid token"}), 403

data = request.json
# Extract minimal required fields
host_id = data.get('host_identifier')
alert_id = data.get('alert_id')
severity = data.get('severity')

if not all([host_id, alert_id, severity]):
return jsonify({"error": "missing required fields"}), 400

# Only act on critical/High severity for automated steps
if severity in ['Critical', 'High']:
payload = {
"host": host_id,
"status": "investigation_pending",
"siem_alert_id": alert_id,
"action": "tag_for_containment"
}
# Call orchestration API
resp = requests.post(
f"{ORCHESTRATOR_URL}/api/v1/host/action",
json=payload,
verify=False # Use internal CA in prod
)
return jsonify({"orchestrator_response": resp.status_code}), 200
else:
# Log medium/low alerts, no action
return jsonify({"status": "logged_no_action"}), 200
```

Current detection use cases feeding this:
* Agent process integrity violations (hash mismatch).
* Failed secret retrieval attempts from Vault exceeding threshold.
* Scheduled agent tasks failing consecutively, which could indicate tampering.

The main benefit is that operational security events now create a tangible, automated workflow in the tool that owns the asset, not just a ticket in the SIEM queue.