I recently ran an experiment around a simple idea: what if LLM agents had access to documentation for the exact dependency versions used in a project?
My motivation was simple: reduce agent mistakes when working with dependencies, hence reduce token usage. I mean sometimes they assume newest version APIs, or just use deprecated behaviour. It was my hypothesis that if the agnet had searchable, version-specific dependency documentation it would reduce those mistakes and in turn reduce token usage and time to task completion.
To test this, I built three tools:
Kaşif: a dependency discovery tool. It finds dependencies used in a project, locates their source repository, and maps the requested version to a specific commit.
Ocak: a language-agnostic static documentation generator. It takes a codebase and produces extensive markdown documentation from it. I built this tool since normal documentation generation tools sometimes require compilation/ dependency retrieval. This method was also language agnostic since I used tree-sitter.
Pir: an MCP server that enables search for version specific documentation.
To test it, I initially ran just some simple prompts using opencode with & without the MCP available. In these initial tests I had some mixed results, until I realized that the tests were flawed: they were wayyy to broad in scope and allowed for runs with&without the toolset to edit different files. After this was fixed, and tests were validated against each other to ensure they edited the same files, I got the following results:
+256% average token usage, +60% average time to task completion, meaning the toolset made the runs more expensive and slower.
Please note that these runs were validated & cross-checked against each other, and were repeated 5 times each to reduce variability.
The model used was Qwen Coder Next 80B with oQ6e quantization with Unsloth recommended settings for coding with oMLX engine build 2193.
The toolset failed its role, so I'm open sourcing the tools for anyone to use under MIT license. Also, I plan to do further research of this on a smaller model. I want to ensure Pir is up to my standards before I release that as open source, but that is coming as well.
AI Disclosure: I used AI to create these two tools. The results and tests I verified myself. I also kept track of tool outputs and traces.