I was reading Neil Kakkar’s latest newsletter this weekend. He used to work at Bloomberg until a period of burn out saw him leave and take time out to recalibrate what he wanted from his work. His newsletters prior to this were always fascinating, mixing software design and development with a broader systemic outlook on the world of work. I read a lot of his output during my MSc.
And then he went quiet. I didn’t know why until he randomly popped up in my inbox a few weeks ago. He told his story in that first new post, so that’s how I know what’s been happening in his life since he went quiet. It didn’t sound fun.
However, thankfully, Neil is back and writing again. This weekend’s newsletter was about agentic debt.
Before I deal with that, a quick word about the word ‘debt’. Traditionally, we think of debt as financial. An extension of credit. Without debt, though, there can be no monetary system. Our entire financial system relies on debt. All systems, perhaps, rely on debt, but this is the most obvious to the majority of people. Debt accrues in many ways, through transactions, credit, use of resources, etc. Entropy means there will always be some sort of debt, (although the Laws of Thermodynamics might allow me to question that to some degree but not now and not here).
What’s important to understand is we can build up debt in all sorts of ways. Blindspots in our knowledge cause debt to build in a multitude of ways, for example. Recently, the idea of ‘cognitive debt’ has sprung up in conversations I’ve seen on LinkedIn. The more we don’t understand how we think about a problematic situation and lean heavily on ai to do our thinking for us, the more we cannot be certain of how we came to an the answer. We cannot explain our process or show our working. This debt compounds, too.
Of course, the ai grifters say that skepticism like this makes those expressing these ideas Luddites. Although I fear they don’t really know what the Luddites stood for and use the term in the prejorative, they might also be missing a point. Not understanding is one thing, but the compound nature of how these debts increase, much as they do when you spend on a credit card and don’t pay off more than the minimum amount each month, so that understanding becomes more and more opaque over time as you add more ‘thinking’ steps to your thinking processes.
I hadn’t really thought about how that type of cognitive debt might also be present within systems that are almost wholly ai generated, such as coding. Neil sets out how ai has no concept of a mental model of what it’s working on, so it does not comprehend the bigger system in which it is coding, as it’s working solely on a single problem. This causes issues over time, which compound in ways that can, and do, cause huge issues for software development. This optimisation for the task at hand is where the issue lies.
I like how Neil refers to ‘tending the garden’ when it comes to managing agentic debt. I think this metaphor is useful here. It may not work for every kind of debt, but it helps in the sense that most people could draw a map of their own garden – or childhood garden – from memory. You can hold a mental model of a specific garden and recreate it at a moment’s notice. Better yet, you can use this mental model as a sandbox for trying things out mentally before you execute any code. Or change any settings. Or ship something that might break that model. It sets boundaries, and in systems thinking boundary judgement is one of the most important steps in any systemic thinking. Without understanding the system you cannot change the system, interact with it or anything within it. Not without unintended consequences and, quite possibly, without harm being done.
The question that Neil asks that I think had the most profound impact on me as I read through his email was: “does this feature fit into the system in a way that won’t confuse the next agent, or the next human?” As he notes in his footnote, taste – and therefore choice – becomes more important than simply making changes.
I’m drawn back to John V, Willshire’s Zenko Mapping which asks, what’s the next right thing? While there are shortcomings to the tool – as there are with any and every tool or framework or model – the question being posed is a good one. More so when we apply it to ai.
