Where are we headed?

I’ve been thinking a lot about thinking lately. I’ve been thinking about how we build models of our world. I’ve been thinking about how we engage with the world through our bodies and through our minds, and how tools extend our minds into the world.

From the perspective of a software engineer, AI is the latest in developments to the abstractions that allow us unlock different levels of thinking. By applying these abstractions to our problems we employ new strategies for solutions. When Grace Hopper proposed we use human-readable languages to write code instead of machine-language, people pushed back. On the abstraction level, we’re seeing the same thing play out with AI. As seen throughout history, this has its benefits. These contributions to technology have created opportunity, and unlocked possibility dreamt of by science fiction writers. In my career, I get to work on a product that fits this progression, that I as an engineer find useful. Thinking less about searching through logs and more about operations as a whole is a step in the right direction when it comes to abstraction. In an ideal world, us humans leave these computational tasks to computers and have more time to think about the real problems. I’m lucky enough to be surrounded by smart people who are building useful tools because they care about them and the people who use them. In this environment I’ve been able to experiment, learn, and build. In building, using, and sharing these tools as a team, we came together and talked about them, we talked about the tools like coding agents. The protocols and patterns around it, how to develop and share our own workflows. Early on, we recognized this as a human problem. Factors like interaction, collaboration, and emotional response shape how people adopt tools, how they share them. In thinking about these factors, I’ve tried to build a framework around my own interaction and what I see being built by others. This helps inform what I want to build.

Whatever your opinion, there has been a fundamental shift in how we do work, and more importantly, how we think. Not just how we think but what it’s possible to think about. People can spin up ideas faster than ever before. What once required weeks of developer work to prototype can now be done in a weekend (how many of those are good ideas remains to be seen). This is powerful stuff, and objectively, LLMs and agents are extremely powerful & innovative tools. I’ve also experienced a lot of the downsides of using these tools. More code is being produced than ever before, making it more and more difficult to review that code. Many people who generate that code cannot point to specific implementation details, and worse, lack a fundamental understanding of the code being generated. Someone can fix a bug with AI and lack the understanding required to explain to someone how that bug was fixed. This could be chalked up to laziness & “human nature” (lazy things to completely chalk it up to), but I think this represents a deeper problem with these tools, how we are using them, and how they’re designed to be used. How many times have you read something written by AI and felt sad? Disconnected from humanity, a little bit. As funny as it sounds, part of what’s driven me to write this is annoyance with the amount of AI generated Instagram comments and posts there are. I’m not sure if these comments & posts are bots, using AI because of “efficiency” or perhaps because people have lost the confidence (and maybe ability?) to write and share their own voice. Probably a bit of all three. Through exposure to these problems and feelings, I’ve developed a slightly negative sentiment. Not to say that these tools aren’t useful, I am trying to walk with both hands full, as Kris Krüg put it at a recent BC+AI community meetup. I’ve felt (and I’ll get back to “felt”) more productive, and been able to tackle breadth not previously possible. AI gives me the ability to parallelize a lot of computational work. However I, like many hold the opinion that these tools are not designed to protect your thinking, nor are they generally being used in a way that protects your thinking. In “Your Brain On ChatGPT”, a study where participants wrote essays with and without Google and/or ChatGPT, researchers found participants of the AI-assisted group could not recall essays they themselves wrote minutes after writing them. This points to a fundamental problem in the ownership model between our own thinking and the work we produce using AI. These researchers also found that people who used AI experienced both less and weaker connections across part of their brains. And not to say using different parts of your brain is a bad thing, but Sam Altman recently spoke at the BlackRock Infrastructure summit. He envisioned a future “where intelligence is a utility like electricity or water and people buy it from us on a meter and use it for whatever they want to use it for”. Comments like these get me thinking about thinking. Protecting and expanding your thinking. Buying intelligence on a meter is not the dystopian future I want to live in, is that where we’re headed?

“Where does the mind stop and the world begin?” Andy Clark and David Chalmers asked in pursuit of their Extended Mind Thesis. In this paper, they argue that the boundary of the mind and the world is not the skull, and tools or objects which exhibit a set of criteria become part of the “extended mind”. That is, they exhibit parts of the cognitive process itself. The tools that fit these criteria and those which are considered the best tools display some kind of “active externalism”. Active externalism is to have the human organism tightly coupled to the externalized cognitive process, the actions being part of thought itself. Removal of the object leaves only the exact cognitive process which was extended. In their paper they present a thought experiment, Otto & Inga, a man with Alzheimer’s and a woman without. Otto has trouble remembering, but Otto has access to a notebook that contains his memories. Otto & Inga both want to go to the museum. Otto writes down directions to the museum, while Inga remembers how to get there. Although Otto has Alzheimer’s, through using this notebook Otto is able to navigate to the museum. Clark & Chalmers argue that the only difference is how Inga and Otto got to the museum is that Inga’s memory is being internally processed, and Otto’s memory is served by the notebook. Otto and Inga both display the same cognitive process, only Otto has extended his mind into the world through this notebook. This thought experiment is one example. People have a general tendency to rely on enviornmental supports to support their cognition. For example, a person rearranging scrabble letters to think through word combinations. An even more basic example is using a pen & paper to record and apply structure to your thoughts. These examples of tools, extensions of your cognition, fulfill the set of criteria defined in the thesis. That is, they are readily available, they are trusted, and they are automatically & consiously endorsed. Using Otto’s notebook as an example, Otto’s notes are easily accessible. Otto rightly believes he is the one who wrote the directions and that they are correct, and Otto at once point endorsed the writing of it, also endorsing the use of it while he follows the directions. I believe that AI, in its current form, fails the extended mind thesis.

Think about how you learn and how you interface with the world and things around you. Your mind is constantly discovering or encountering, its applying patterns to make sense of the world, its using that sense you’ve just made to create, to act. Think of a baby’s eyes as it creates understanding of the world. This is meaning making happening, in real time. That meaning making is the most fundamental part of the learning process, it is what makes you and your beliefs your own. This is epistemic action. You alter the world so as to aid your cognitive process. We’ve established that some tools, like a notebook or a set of scrabble letters, can enhance and extend that meaning making, your thinking. Now, how much of this pattern, or loop of meaning-making, happens when you’re interacting with a chatbot? The current interaction model is the human will attempt to put their thought into words, passing it into an LLM. The model will interpret the language (which may or may not be the right language on the side of the human), and using prediction generate some response. This is pragmatic action, the world is altered so as to achieve some result. The response, as you may know, is generally very plausible sounding, but requires a lot of scrutiny (it may even be not at all what you meant). The plausibility also creates an illusion of confidence which when broken, breaks the automatically endorsed criterion of the extended mind. While experimenting I try to build models and create frameworks around my interaction with chatbots, especially coding agents, and I’ve come to realize that what is happening in this interaction model is much of your thinking process is delegated into this forward pass of a prompt and response. This is “cognitive offloading” in real time. What looks like thinking is simply a response in an attempt to achieve some goal. If you removed this tool, would the act of prompt to response be the thinking that it was designed to replace? Tools that enable epistemic action are those which act through you, not those which act for you.

Think of playing Tetris. You rotate and place a block, immediately receiving feedback. One enters a flow state. This feedback and flow state allow you to grow and ultimately be great at Tetris. Rachel Thomas recently wrote about breaking the spell of vibe coding, where she compared what is happening with agents and AI in engineering to multi-line slot machines. Multi-line slot machines employ a technique of Loss-Disguised-As-Win where you are paid out small amounts, but ultimately your result is loss. She referenced the concept of “dark flow”, coined by researchers in the context of gambling. Csikszentmihalyi, author of and he who coined Flow, described “junk flow”, which is when “a person becomes addicted to an experience which may be flow, but becomes something addicting rather than something that makes you grow”. The forward pass cognitive offloading happening with LLMs and agents has created this same negative feedback loop, where small incremental pragmatic progress towards a goal creates the illusion of progress, sticking people in this dark flow state. In an METR study, people who were given codebases felt like they were producing 20% faster, in reality they found they produced 19% slower. This almost 40 point gap in perceived reality is dark flow in action. In practice, with coding agents, the only feedback I receive (unless I explicitly ask, a tactic I myself have to employ), is “does it run”, do the (poorly written) test cases pass? And in this most of these interactions there is a complete loss of feedback related to me building understanding, asking questions, and making meaning, unless I explicitly direct the interaction. This interaction model breaks the loop I described. These agents force you into a state of acting, of executing for some pragmatic purpose. In the context of a workplace, this might result down the line in a suite of highly efficient workers, but it will likely result in reduced understanding, and less diverse ways of thinking. Advait Sarkar, a researcher at Tools for Thought, said in his TedAI talk “we are becoming middle managers for our thoughts”.

Tools must be designed to preserve epistemic action and the thinking loops that support our meaning making and cognition. AI tools can (and I personally try to employ tactics to use them this way) be developed to enable coupling in a way that preserves this loop. Some have called this the “age of outsourced reason”. In acting as these thought middle managers, which I believe is partially the fault of these chatbots, we offload the parts of thinking required to grow. Growth, however, is necessary for society & human flourishing. What happens to the talent pipeline when it becomes harder to train, and junior engineers are no longer? What happens to the future when there is less diversity in thought? We are seeing understanding degrade in real time. Perhaps this is a transition period, as so many say, and we’ll come out on the other side more powerful & capable humans. This remains to be seen. In my experience a lot of the time people do not have the confidence to reply without putting their words into a language model first, and people produce more and more AI generated content. The more AI generated content there is, the worse it becomes. AI trained on itself will statistically experience autophagy. This to me feels like humans may experience autophagy as well. Less diverse thought, writing styles, and ways of working are happening in front of our eyes in real time. Mental fatigue, or “brain fry”, from managing so many agents is apparent. We must treat computers as computers, and maintain the parts of ourselves that make us so valuable.

In trying to replace so much of our work and ourselves we are optimizing for the wrong thing. The past few years of the anthropocene have forced us to confront our most essential qualities like understanding. What’s left when computers become better computers is more room for that which are the most human qualities. Thinking, judgement, connection, love. At their core, AI and LLMs are summary and synthesis tools. These tools have been harnessed into automating away much of our thinking, described as cognitive offloading. Instead, these tools can be harnessed into creating & enhancing meaning-making & creative loops that help us to grow into more capable and knowledgeable individuals, so if Anthropic or OpenAI shut down we would not be left without a meter to purchase knowledge from. I was lucky enough to recently listen to a Q&A with Karen Hao at the Chan Centre at UBC. Corporations like OpenAI which she calls in her book Empires of AI have created tools that try to fulfill pragmatic action and automate “mundanity”, but will only degrade our understanding and our ability to think. Many see the possibilities and opportunities of humanity as never being greater than this epoch. The capacity to think about them, however, is shrinking. We must write, we must connect, we must focus on what matters. I’m reminded of this quote from John Adams.

Are we building a world where the next generation can think about what matters, or have we already started to forget about what makes us human? Only years ago it seems as though we were promised cures to disease, famine, and war, but all of that was overshadowed by a pursuit of hyper-scaling towards automated reasoning. I’ve almost been forced through interacting with these models to think more about thinking, and I am very grateful for that. If one thing is to come out of this war on thought, I hope it is our ability to see through, as Harry Frankfurt puts it in his essay, bullshit.