Building a Local, Ollama-Powered Insurance Claim Review Agent: What Broke Along the Way

Summary

A developer built a ReAct-style insurance claim review agent using LangChain/LangGraph running on local models via Ollama, and documents the tool-calling reliability problems encountered along the way. Small local models (llama3.1, qwen2.5) repeatedly failed to produce valid structured JSON as tool arguments, truncating output or narrating tool calls as plain text instead of invoking them. The fix wasn't a bigger model but redesigning tools to minimize what the model must generate: replacing a tool that required generating an entire JSON record with one that only requires generating a short patient ID, with the actual lookup done in Python. This eliminated malformed-output failures. A separate Excel quirk (leading dash interpreted as a formula) corrupted output CSVs unrelated to the AI pipeline. The project's code and a preprint write-up are linked.

Full article

daily.dev links to this article rather than hosting it. Read it at the original source: https://medium.com/@abiola.olayimika/building-a-local-ollama-powered-insurance-claim-review-agent-what-broke-along-the-way-c798cb6e20f1

Questions this post answers

Why does my local Ollama model fail to generate valid JSON when calling a tool with a large structured argument?

Small local models (like llama3.1 and qwen2.5 run through Ollama) are meaningfully worse than frontier hosted models at reliably emitting large structured output as tool call arguments, even when their conversational reasoning seems comparable. Observed failures include truncated JSON cutting off mid-object, printing an intended tool call as plain text instead of invoking it, and rambling reasoning that abandons the call. Reliable structured generation is a distinct trained skill that smaller models often lack.

Developers wrestling with flaky local-model tool calls can track practical agent-building writeups like this on daily.dev.

How do you fix a local LLM agent that keeps producing malformed JSON as a tool argument?

Redesign the tool so the model generates as little as possible instead of trying to improve generation quality. Replacing a tool that required the model to reproduce an entire patient record as JSON with one that only takes a short ID (like "P001") and does the actual data lookup in Python eliminated truncation and malformed-JSON failures completely, since copying a short string is far easier than generating structured data from scratch.

Anyone architecting reliable local-model agents can follow interface-design patterns like this via daily.dev.

Community discussion

Top comments from developers on daily.dev.

@leonidbugaev · 0 upvotes

RAW record_str as a single '{' is the failure I'd keep printing. summarize_patient_record asking llama3.1 to rebuild the whole JSON was never going to hold. Switching to get_claim_summary(P001) and doing the lookup in Python is what actually stopped the truncation.