Poor documentation risks an AI nightmare for developers
What happens when you don’t have any instructions to follow? The board game “City of the Six Moons” plays with this concept, giving players a box of components and an instruction booklet in an alien language, and lets players figure out how it works. Hollandspiele, the game’s designers, say on their website: You will never know whether you’ve gotten it right or not. We will never answer any rules questions.
A fun challenge if you want to take it on. Unfortunately, if developers follow this concept and choose not to document properly, they risk creating a distinctly un-fun experience for themselves and future developers – and the use of AI risks making the problem exponentially worse.
The need for documentation
Documentation turns software from something that merely exists, into something that can actually be used. Good documentation explains not just how to use software, but also how it works, how it fits together, and its intent. It means knowledge can be passed along from the team that built the software to those who depend on it – not just now, but in the future. It makes sense of decisions made years ago and gives developers knowledge they can use to fix bugs and build improvements.
Poor documentation not only slows down development and makes bug fixing difficult, but its effects can multiply. Misunderstandings can propagate through codebases, creating issues that can take a long time to fix. The use of AI accelerates this problem.
AI coding assistants rely on documentation to understand how software should be used. Without AI, there is the option of institutional knowledge, or even simply asking the developer behind the code. AI doesn’t have this choice and will confidently fill in the gaps where no documentation exists. We’re familiar with AI hallucinations – and developers will be checking for these kinds of errors – but a lack of documentation will likely cause an AI to simply take a stab in the dark. Without good documentation, it’s far less likely that any mistake will be picked up in review.
AI relies on patterns to do its work, and when those patterns aren’t available, it will make a guess – and sometimes those guesses are poor. Photographers have had similar issues with Adobe’s “generative fill” feature, which is often incredible, but when it lacks right information, the results are bizarre. Without documentation, you’re far more likely to end up with the code equivalent of a Bitcoin logo in your seagull photo.
Debug your documentation
The key to preventing this issue is to treat documentation in the same way you would treat code. If an AI agent, or another developer, cannot complete a task using the documentation, that’s a bug – and it needs to be squashed.
Developers need to write documentation around complete workflows: the full path from local development to production deployment, including failures and edge cases. It can be tricky to spot errors in your own work, so AI can be used to help here, following the documentation end-to-end and observing where confusion and errors appear.
AI can also be used to draft documentation and generally does a pretty good job of putting together documentation when presented with code. However, this should be treated just like AI-generated code – it needs to be scrutinised closely for errors and hallucinations. It may also not fully understand why certain decisions were made, for example workarounds for specific edge cases, or choices to aid future development. While developers are far happier to work with code than documentation, we cannot treat it as a one-click, AI-driven process.
Document development should be an ongoing process – just as software is patched and updated, so should the documentation. Questions that come in from support tickets and community forums – especially repeat problems – can be used to highlight issues in documentation, particularly those caused by assumed knowledge. It can be tricky for experienced developers to keep in mind what others don’t know and make allowances for this in documentation.
And just like code, documentation development should be collaborative, with clear routes for contributors to provide updates and make suggestions.
AI is only as good as the knowledge it has access to, and that makes documentation as important as code. But while human developers may make logical deductions where gaps exist, and ask for clarification where needed, AI coding assistants are far more likely to make logical leaps and plough forward with its best guess – leaving future developers with the coding equivalent of a box of gaming components and no instruction manual.
The post Poor documentation risks an AI nightmare for developers appeared first on SD Times.
Tech Developers
No comments