There’s a lot of controversy nowadays surrounding AI. There are different sides that mostly speak past each other, each basing their views on different assumptions about what AI even is, what it’s capable of, whether it has value or is in many ways a manifestation of evil. I wanted to try to explain my current thinking about AI, what it is to me, and what I expect will happen with it in the future.
First, I’ll say I’m neither a strict AI skeptic, doomer, or booster. I recognize there are grains of truth in each viewpoint, and I also think they all simplify at the expense of the full picture. If I have to place myself, I’d probably be halfway between skeptic and doomer, with a bit of booster in that I have faint hope that the highest potential of the technology might one day be realizable.
So, what is AI? A lot of less well researched AI skeptic positions claim it’s just code, or a stochastic parrot, or fancy autocomplete. I studied AI and worked with and trained models for years. What we call AI now is not so simple.
Historically, AI as a field was divided into two approaches: Top-Down, and Bottom-Up. Top-Down mostly involved coming up with rules for computers to follow. These were just code. These were the predominant approaches for most of the 20th century. Bottom-Up mainly involved trying to copy what we thought the algorithm of the human brain was, to try to construct intelligence from basic building blocks derived from first principles. This latter approach became popular recently for various reasons.
The basic building block of modern AI systems is a mathematical construct called an Artificial Neuron, which is organized into mathematical structures called Artificial Neural Networks. These, very roughly, approximate elements of how we think the brain works. Not exactly, but similar.
We combine these with learning algorithms, namely, backpropagation of error with gradient descent. This method isn’t really how the brain works, but it works very well for our purposes, so people use it.
People these days talk about how LLMs are based on the transformer architecture. Transformers are just a particular arrangement of a neural network that works exceedingly well in practice. I won’t go into too many details why, but it allows data to be fed into the model in parallel, which greatly speeds up training over older designs, like RNNs.
Artifical neural networks are generally trained by giving it input examples and output examples over and over and over. They require A LOT of data to work well, and work better when they are bigger and have LOTS of weights, which represent the connection strengths between neurons. Notably, these models are less efficient than human brains in terms of data. Humans seem to be able to learn from far fewer examples. Why this is, is still an open question in research.
Modern LLMs are generally trained in one big training run that can take months, on a huge dataset of much of the Internet. And compared to human brains, they are orders of magnitudes less energy efficient. The human brain runs on something like 20 Watts, while a decent sized model takes up tens of GPUs, each using 200+ Watts.
These models, are also, smaller than the human brain by orders of magnitudes. The largest models are rumoured to be 10 trillion parameters. If a parameter is equivalent to a synapse, the human brain has 100 trillion. And unlike modern models, the human brain is sparsely connected. Each neuron has something like 1000 connections. An LLM is fully connected, each node in each layer is connected to all the nodes in the next and previous layer. This makes them work well with GPUs, as you can represent them as giant matrices of numbers. Sparsity adds many zeroes to those matrices, requiring larger matrices for the same capacity, which is much less efficient for GPUs.
Sparsity though is important. It significantly helps with a problem that LLMs and neural nets generally suffer from, which is catastrophic forgetting. With fully connected models, every update affects every neuron, which means new tasks cause old tasks to be forgotten. Thus, unlike humans, LLMs learn in one big training run where the examples are randomized so that the tasks are evenly distributed. It’s different from humans, where we learn one task at a time, sequentially.
Humans are continuous learners. They learn as they move around the world and do things. LLMs in their current state do not do this. During inference, LLMs have frozen weights. Anything from the current session not in their context, which is functionally their working memory, is forgotten.
This difference is an important limitation, and one that is not easily overcome. Without the ability to truly learn on the job, the ability for these models to fully replace human labour becomes unlikely. In theory, there are ways that this could be overcome. But the other problem with this is economic.
Right now, models served to humans during inference are batched such that there need only be one copy of the weights. Hypothetically, for a model to be able to learn and update its weights in a specific context, you would need a different copy of the weights to update for every user. If you have millions of users, you’d need millions of copies. That means millions more GPUs than are available, given the size of the largest models.
So, what does that mean for AI? It means that AGI that can replace all humans is not just around the corner. I haven’t even gotten started on the relative lack of training data for robotics, the complexity of the human hand, and other challenges that will slow things down in terms of true replacement.
And replacement is the goal. It is the reason why so much capital is being poured into AI. The ideal is for capital to replace labour. In a perfect world, this leads to post-scarcity. In a less perfect world, it benefits the bottom line of the capitalist, mostly at the expense of the working class, and leads to a kind of techno-feudalism.
But the reality is that it will take a long time before the resources are available, the technology is ready, etc, etc. There are things in the pipeline that can change this. Spiking Neural Networks more closely resemble the brain with their sparsity, and can be run on Neuromorphic chips that are far more energy efficient. But these aren’t ready for prime time yet, and will take time to become viable.
In the meantime, what is AI good for? Training on the whole Internet gives LLMs substantial knowledge. But that doesn’t exceed human ability. What has historically allowed AI models like AlphaGo to exceed human ability was self-play, to learn from trial and error in a controlled environment like a game. This is something that LLMs are now trained with to an extent, Reinforcement Learning with Verifiable Rewards (RLVR). But this technique works mainly on tasks where the results are easy to verify the correctness of, like math and coding. Things like creative writing, which are highly subjective, are much harder to grade with RLVR, and so we get the Jagged Frontier of AI.
The Jagged Frontier means that current gen AI are especially good at some things, and bad at others, in a way that doesn’t fit the definition of AGI. But that doesn’t mean they aren’t capable. It very likely means that they could become superintelligent in very particular fields, like math and coding, and maybe certain sciences like physics, but continue to be relatively weak at writing and other areas.
But, perhaps importantly, math and coding and science are the most expensive fields. They are actually where, if you had limited compute and energy to brute force intelligence, you would get the best return. This is where I think things are going.
AI is not economical to replace all humans, at least for now, but it is potentially useful for taking over particular fields where it has a comparative advantage. Is this enough to justify the spend? I’m not sure. How useful would new physics beyond the Standard Model be? How much money would you pay for that?
Math is the language of the universe, and easily verified. It is, perhaps uniquely suitable for LLMs to get good at.
What is AI bad at though?
I would argue two things. Creativity and Judgment.
You can ask an LLM to generate a short story. It’ll be decently written, but if you ask different LLMs, you’ll notice the stories often share common things. I think, this is at least partly because all LLMs share roughly the same dataset. They train on and learn an averaging of the Internet.
Humans on the other hand, each have unique lived experiences. We may share similar experiences in similar circumstances, but the variance is much higher. As such, the combination of these unique experiences allows each of us to develop novel ideas that are different from what someone else would come up with.
In this sense, human creativity is likely to endure. LLMs can connect the dots well, because they have so much knowledge, but they lack the differences needed for true novelty. In theory, you could train models on subsets of the data to get more variance, but then you use less data and the models are less knowledgeable, so there’s a serious tradeoff there.
Human judgment is the result of learning from trial and error. LLMs do this too, but not to the same extent. Humans, from childhood, try and experiment and explore and plan and fail and try again until they succeed. This teaches them how to judge situations and plans very effectively. The tasks we try and the signals we get from success and failure are long horizon. Most LLM RLVR training is on relatively short, specific tasks. To train it on longer tasks would require far more time, as you can’t parallelize a single long horizon task. This is not insurmountable, but it is harder for LLMs. It doesn’t come naturally.
I’m not saying AGI is impossible. In the limit, we can eventually create what amounts to a synthetic human that would be able to do whatever we can. But LLMs aren’t that.
Admittedly, LLMs aren’t the only form of AI. In theory, there are those World Models that people are working on. Those could potentially be closer to AGI, by better modelling experience in the real world. But they aren’t there yet, and they require even more compute the simulate the virtual worlds.
So, I think, AI has a place in the near term world as a specialized ASI that can do certain things really well. There is an economic case for that, but it’s not really what the AI hype is about. And so, there’s still a real risk of the bubble popping.
But that doesn’t mean AI isn’t useful.
So, what about the dangers? If AI becomes a master planner and decision maker, if it gets good enough at judgment, it could conceivably be used to takeover. But, it’s not really clear that “it” is anything independent of its user. Maybe you could run it in some kind of infinite loop independent of human involvement, but that’s not the way current AI works.
The way AI is currently trained is to complete a task and come back to the user and wait. This loop can get longer and longer, but there is always a sense in which the AI is deferential. I think, it would take a lot for an AI to break out of this expectation completely, and operate alone.
The danger of AI going rogue itself, and say, turning us all into paperclips, is less likely than it defacto disempowering us by making all the important decisions when prompted by the so-called decision makers. Cognitive surrender is a real phenomenon already. This is probably the route through which things will get out of hand.
But, I don’t really know. These are just my musings based on my knowledge. So, it’s reasonable to take all this with a grain of salt.