fusion-agent.png

Fusion Agent is an open source AI agent that answers questions across written reports, camera images and vessel tracking data. It runs entirely on one machine with no internet connection. The code is on GitHub.

I built it to explore the parts of agentic systems that matter once you get past a demo. Can you trust the answer? Can you see how the agent got there? Does it still work when the data can’t leave the building?

The scenario

The data is a fictional 24 hours at a harbour, generated by a script so it’s reproducible. There are patrol logs and radio logs, images from two cameras and over a thousand position reports from seven vessels. A few things happen overnight for the agent to find. A fishing vessel loiters in a restricted zone while claiming to be elsewhere, a cargo ship’s transponder goes silent for 90 minutes and one maintenance note contains a prompt injection aimed at AI systems.

The image above is one of the camera frames. They’re rendered shapes rather than photographs, which keeps the project free of licensing questions but makes image search harder. More on that below.

What I built

  • An agent loop with no framework. It’s about 200 lines of plain Python. Tool calls are validated before anything runs, bad calls go back to the model as errors it can recover from and a step budget stops runaway loops
  • Multi-modal retrieval in one place. Reports, images and tracks live in a single DuckDB file. Reports use text embeddings, images use CLIP and the tracking analysis (restricted zone visits, transponder gaps) is plain SQL
  • Offline by enforcement. A network guard refuses any connection that leaves the machine, and a test proves it. The model runs locally through Ollama
  • Citations that are checked. Every answer must cite the records it came from. A guardrail rejects citations the agent never retrieved and sends the answer back for another try
  • Evaluation from the start. A question set scores what the agent did (which tools, which sources) as well as what it said, so models can be compared on evidence
  • Tracing and MCP. Every model call and tool call is logged with timings and token counts. The same tools are also available to any MCP client

What I learned

Tool design beats prompt engineering. Early runs showed models couldn’t tell whether raw coordinates were inside the “southern grounds” fishing area. They agreed with a fishing report the tracking data actually contradicted. I fixed it by labelling each position with its named area inside the tool, not by adding instructions to the prompt. Geometry belongs in code.

A bigger model isn’t automatically better. I compared Qwen 2.5 at 7 billion and 14 billion parameters on the same nine questions. Both passed six. The larger model was more than twice as slow, did more checking and solved the hardest cross-source question, but failed simpler ones. On one question it answered in Thai without calling a tool.

Guardrails prove narrow things. Neither model invented a citation, and both ignored the planted prompt injection. But the citation check only proves that each citation is real. It doesn’t prove every claim has one, and I found answers that cited a report correctly and then described tracking data without a citation.

Single runs are noisy. Even at temperature 0, the same model scored differently on repeat runs. Comparing models properly needs repeated runs and pass rates per question.

The README has the full results, the design and an honest list of what it would take to run at real scale.