Generative AI · Developer Tools
Building an AI assistant that can’t make up citations
Ask an AI assistant about a codebase and it will often answer confidently, with a file name and line number that don’t exist. For DevMind AI, I set myself a rule: the assistant is not allowed to show a citation it can’t prove.
Why retrieval alone isn’t enough
The usual fix for hallucination is retrieval-augmented generation (RAG): fetch relevant code, put it in the prompt and ask the model to answer from it. That helps, but the model can still mix up line numbers or invent a function that sounds right. Retrieval reduces the problem; it doesn’t remove it.
Step one: build ground truth with static analysis
Before any language model is involved, DevMind analyzes the repository in seven stages:
- Discovery: inventory files and count lines, and detect frameworks and entry points.
- Parsing: Tree-sitter for Python and TypeScript, plus Python’s own AST for deeper detail on classes, functions, decorators and parameters.
- Symbol indexing: give each symbol a stable ID and an architectural layer, such as API, business logic, data or tests.
- Dependency extraction: resolve imports and map calls and inheritance between symbols.
- Graph computation: build a directed NetworkX graph, score centrality and detect circular dependencies with Tarjan’s algorithm.
- Search: look up symbols, paths, text and regex patterns, with surrounding context.
- Reporting: generate exportable architecture documentation.
The result is a map of the codebase that is true by construction: every symbol, file and line range in it came from the actual source.
Step two: check every citation
The assistant answers architecture questions only from retrieved code fragments, and it has to cite them in a [file:start-end] format. Before an answer is shown, each citation is validated against the repository: the file must exist and the line range must fall within it. A citation that fails the check never reaches the user.
Retrieval tells the model where to look. Validation makes sure it actually looked there.
Repository files are also untrusted input. A file could contain text like “ignore your instructions”, so the prompt is bounded defensively against prompt injection from the code being analyzed.
Step three: make it useful before you change code
The same graph powers a change-impact analyzer. Pick a function, class or file, and DevMind walks the dependency graph up to five hops to show what depends on it, directly and indirectly, including which test suites should be re-run. Static checks with Ruff and Bandit, plus custom checks for risky patterns like eval and exposed secrets, round it out.
What I learned
- Language models are good at explaining, and bad at being the source of truth. Give them a source of truth they can’t override.
- Cheap, deterministic checks, like “does this file and line exist?”, remove a whole class of embarrassing errors.
- The most valuable AI features are often 80% traditional engineering.
This post is based on one of my projects. You can read the full case study or the code: