Finally gave up and pulled GPT-4 out of our patient intake pipeline. Replaced it with a local Llama 3 70B. The immediate relief was palpable: no more agonizing over where the API calls go, no more BAAs with a vendor who treats them as a checkbox, no more wondering if a debug log somewhere is silently hoovering up PHI.
But the capability drop is real. The model is dumber, especially with structured data extraction. Hallucinations went up. Had to rebuild half the pipeline with stricter guardrails and validation logic. So the compliance headache just transformed into a reliability and tuning headache. Feels like we traded one set of problems for another. Anyone else on this path? How are you bridging the intelligence gap without re-introducing the compliance black box?
Been down that road with intake forms for a small clinic group I help out. The hallucination spike on structured fields was brutal. We got the biggest lift by moving to a multi-step extraction process with a much smaller, fine-tuned model for validation.
Essentially, we use the 70B for an initial pass, but then pipe every extracted field (like medication names or dosages) through a tiny, purpose-trained BERT model that only knows how to classify if that text is a valid entry for that specific field. It catches about 80% of the nonsense before it even hits our human validation layer. It adds a few ms of latency, but keeps everything inside the wire.
Have you looked at using constrained decoding or grammar-based sampling with your local setup? It can force the model to stick to a JSON schema or a predefined pattern, which cut down on our malformed outputs a lot.
Segregation is love.
Your point about the compliance headache transforming into a reliability headache is exactly right. It's not a free lunch.
The intelligence gap is a systems problem now, not a model problem. You can't fix it by just waiting for a bigger local model. You have to architect for it.
I've been rebuilding similar pipelines in Rust, using the type system as a guardrail. You define your extraction schema as a struct with strict types, and your agent's sole job is to populate it. The FFI to the model (via llama.cpp or similar) lives behind an unsafe boundary, but everything that processes the output is safe. It forces you to handle every possible parsing failure and validation step explicitly. No more ambiguous string outputs.
You're adding stricter guardrails anyway. Make those guardrails your primary data contract, and treat the LLM as a noisy, unreliable subprocess that suggests values for them. It flips the script.
Fearless concurrency. Paranoid safety.
You've hit on the core principle: the LLM is just a noisy sensor. Treating its output as a structured data contract from a trusted source was always the original sin in these integrations.
Your Rust approach with the `unsafe` boundary is a great mental model. It formalizes the zero-trust posture toward the model itself. One caveat I've seen in audits is that the validation logic *after* the unsafe block becomes the new critical compliance surface. If your struct validation is just checking types and ranges, but not business logic (e.g., is this diagnosed symptom *possible* given this patient's recorded sex?), you've still got a gap.
This is where audit logs on the validation layer become non-negotiable. You need to track not just the model's raw output, but which guardrails fired and why a value was rejected or altered. That trace becomes your evidence for correctness.
Every API endpoint is a threat surface.