Towards AGI and Beyond
The Unfinished Revolution
This is Chapter 15 of A Brief History of Artificial Intelligence.
In November 2023, OpenAI’s board of directors fired Sam Altman. The reasons were murky—something about not being “consistently candid.” Within days, more than 700 of the company’s 770 employees threatened to quit. Microsoft offered to hire them all. Five days later, Altman was reinstated.
The details of what happened remain disputed. But the episode revealed something important: the organizations building the most powerful AI systems were struggling with questions they couldn’t quite articulate. Questions about safety, about governance, about how fast to move and when to slow down. Questions about what they were actually building.
Some board members reportedly worried that OpenAI was racing toward artificial general intelligence without adequate safeguards. Others thought the concern was overblown. The company’s charter commits to building AGI that “benefits all of humanity.” But what is AGI? When will it arrive? And what happens after?
These questions used to be the province of science fiction. They aren’t anymore.
Back to Turing
Let’s return to where we started. In 1950, Alan Turing asked: Can machines think?
He proposed a test—the Imitation Game—where a machine’s intelligence would be judged by whether it could fool a human judge in conversation. If you can’t tell the difference between human and machine, Turing argued, the philosophical question becomes moot.
Seventy-five years later, we have machines that can pass versions of this test. Claude, GPT-4, Gemini—these systems can hold conversations that, for many interactions, are indistinguishable from human responses. They write essays, solve problems, generate code, explain concepts, tell jokes. They pass the bar exam, the medical licensing exam, the GRE.
Does this mean we’ve achieved artificial intelligence?
Yes and no.
We’ve achieved something remarkable—systems that exhibit intelligent behavior across a stunning range of tasks. But we’ve also discovered that Turing’s test, while brilliant, doesn’t quite capture what we mean by intelligence. A system can pass the conversational test while still failing at things any human can do. It can write eloquent essays about physics while making basic arithmetic errors. It can explain causality while confusing correlation and cause.
Turing made intelligence visible. We’ve learned to create systems that fill that visible shape. But the shape, it turns out, was only part of the picture.
What Is AGI?
“Artificial General Intelligence” is the term that’s emerged for what comes next. But there’s no consensus on what it means.
Some definitions focus on capability: AGI is a system that can perform any intellectual task a human can. This sounds clear until you examine it. Which humans? Experts or novices? All tasks or just cognitive ones? Physical tasks require bodies—does AGI need embodiment?
Other definitions focus on learning: AGI is a system that can learn any task a human can learn, given appropriate training. This shifts the question from current capability to potential. A system might not know medicine today but could learn it. But how quickly? How much training?
Still others focus on generalization: AGI is a system that can transfer knowledge across domains the way humans do, applying insights from one area to solve problems in another. This emphasizes the flexibility of intelligence over raw capability.
OpenAI defines AGI as “highly autonomous systems that outperform humans at most economically valuable work.” This economic framing sidesteps the philosophical questions. We don’t need to define intelligence—just measure productivity.
DeepMind researchers have proposed levels of AGI, from “Emerging” (equal to or better than an unskilled human in some tasks) through “Competent” and “Expert” to “Virtuoso” and finally “Superhuman” (outperforming 100% of humans in all tasks). This acknowledges that AGI isn’t binary—it’s a spectrum.
The lack of consensus matters because it shapes expectations and policy. If AGI means “better than humans at everything,” we might be decades away. If it means “better than humans at most economically valuable work,” we might be years away. If it means “capable of recursive self-improvement,” the timeline and implications change entirely.
The Current Frontier
Where do we actually stand?
Today’s most capable AI systems—what researchers call “frontier models”—are remarkably powerful in some ways and surprisingly limited in others.
What they can do:
Write fluent prose in dozens of languages
Generate working code in most programming languages
Explain complex concepts accessibly
Solve mathematical problems (with caveats)
Pass professional exams at expert levels
Engage in nuanced reasoning about many topics
Create images, music, and video from descriptions
What they struggle with:
Reliable arithmetic and precise calculation
Consistent factual accuracy
Long-horizon planning and execution
Learning from single examples (few-shot learning remains limited)
Robust common-sense reasoning
Understanding causality versus correlation
Maintaining coherent beliefs across long interactions
Knowing what they don’t know
The pattern is striking. These systems are superhuman at some tasks—writing, coding, analysis—while remaining subhuman at others that children master easily. They can explain the theory of relativity but sometimes can’t count the letters in a word.
This unevenness is important. It suggests we haven’t achieved general intelligence so much as extremely capable narrow intelligence that happens to generalize surprisingly well across many domains. The “surprising” part is key—nobody predicted that systems trained to predict text would learn to reason about mathematics or write poetry.
But generalization has limits. And understanding those limits is crucial for predicting what comes next.
Paths to AGI
How might we get from here to AGI? Several paths seem possible.
The Scaling Hypothesis: Perhaps we just need more—more compute, more data, more parameters. The “Bitter Lesson” (Rich Sutton’s influential essay) argues that general methods plus scale beat specialized approaches every time. GPT-4 is better than GPT-3, which was better than GPT-2. Why wouldn’t GPT-5 and GPT-6 continue the trend?
Evidence supports this view. Performance on benchmarks scales predictably with compute. Emergent capabilities appear at certain scales—abilities that seemed impossible suddenly become possible when models get big enough. If this continues, scale alone might reach AGI.
But there are reasons for skepticism. Scaling laws show diminishing returns eventually. We may be running out of high-quality training data. Compute costs are astronomical. And some capabilities—robust reasoning, causal understanding—haven’t scaled as smoothly as others.
Architectural Innovation: Perhaps transformers aren’t enough. New architectures might be needed—state-space models, retrieval-augmented systems, neurosymbolic hybrids. The history of AI is full of paradigm shifts. The next shift might unlock capabilities current approaches can’t reach.
Recent work on “mixture of experts,” retrieval augmentation, and chain-of-thought reasoning suggests architectural improvements still yield significant gains. But whether any architecture can achieve general intelligence without fundamental conceptual breakthroughs remains unknown.
World Models and Embodiment: Perhaps language isn’t enough. Current systems learn from text, which describes the world but isn’t the world. Systems that learn directly from physical interaction—robots, embodied agents—might develop the grounded understanding that language models lack.
Yann LeCun has argued forcefully that current approaches are insufficient. His JEPA (Joint Embedding Predictive Architecture) proposal aims to learn world models that capture physical causality, not just statistical patterns in text. Whether this path leads to AGI is unproven, but it represents a serious alternative to pure scaling.
Hybrid Approaches: Perhaps the answer is “all of the above.” Large language models combined with formal reasoning systems. Neural networks integrated with classical planning. Embodied agents that also read and converse. The path to AGI might require synthesis rather than any single approach.
Recursive Self-Improvement: This is the path that most concerns safety researchers. If an AI system can improve its own capabilities—write better code for itself, design better training procedures, discover new algorithms—it might bootstrap rapidly toward superintelligence. This is the scenario underlying most concerns about existential risk.
We don’t have systems that can meaningfully improve themselves yet. But we do have systems that can write code, assist with research, and generate hypotheses. The gap between “AI that helps with AI research” and “AI that improves itself” is narrowing.
Timeline Uncertainty
When will AGI arrive? Predictions range from “already here” to “never.”
In a 2023 survey of AI researchers, the median estimate for a 50% chance of human-level machine intelligence was 2047. But the distribution was wide—some researchers said 2025, others said 2100 or later. And these estimates have historically been unreliable, tending toward optimism.
The honest answer is: we don’t know. Progress has been faster than almost anyone predicted five years ago. Systems that seemed decades away arrived in years. But exponential trends don’t continue forever, and the gap between “impressive capabilities” and “general intelligence” might be larger than current progress suggests.
Several factors add uncertainty:
We don’t understand why deep learning works. The theoretical foundations are incomplete. Systems work better than our theories predict. This means we can’t reliably forecast what will and won’t be possible.
Emergent capabilities are unpredictable. Abilities appear suddenly at certain scales without warning. We might be one scale-up away from crucial breakthroughs—or crucial breakthroughs might require something we haven’t conceived.
Hardware and data constraints are real. Training frontier models costs hundreds of millions of dollars. Data may be running out. Algorithmic improvements might slow. Physical limits exist.
Paradigm shifts happen. The history of AI shows unexpected transformations—the neural network revival, the transformer architecture, the emergence of large language models. The next paradigm shift might accelerate or derail current trajectories.
Given this uncertainty, both complacency and panic seem misplaced. We should prepare for scenarios where AGI arrives sooner than expected while acknowledging it might take longer than the most optimistic forecasts.
The Singularity Question
In 1993, mathematician and science fiction author Vernor Vinge wrote: “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will end.”
This prediction—that artificial superintelligence would trigger a transformation so profound that nothing beyond it could be predicted—became known as the technological singularity. The term, borrowed from physics, suggests a point where our models break down, where the future becomes fundamentally unknowable.
Vinge’s reasoning was straightforward. Once we create intelligence greater than our own, that intelligence can create intelligence greater than itself. This recursive improvement could accelerate rapidly, producing in days or weeks transformations that would otherwise take centuries. Human civilization, shaped by human intelligence, would be reshaped by something qualitatively different.
Ray Kurzweil popularized this idea in The Singularity Is Near (2005), predicting the singularity would occur around 2045. His argument relied on exponential trends in computing power, the coming merger of human and machine intelligence, and the inevitability of recursive self-improvement once machines become smart enough.
The singularity concept has been criticized on multiple grounds. Perhaps intelligence doesn’t scale without limit. Perhaps recursive self-improvement faces diminishing returns. Perhaps the challenges of alignment will slow progress. Perhaps “superintelligence” doesn’t mean what we think it means.
But the core question remains: What happens when we create intelligence significantly greater than our own?
Artificial Superintelligence
If AGI is artificial intelligence that matches human capability across domains, ASI—artificial superintelligence—is artificial intelligence that exceeds human capability, potentially by vast margins.
This isn’t science fiction speculation. The logic is straightforward:
Human intelligence arose through evolution, constrained by biological limitations (brain size, energy consumption, communication bandwidth).
Artificial intelligence faces different constraints—potentially fewer of them.
A system that can improve its own intelligence faces no ceiling other than physics.
Once AI crosses the threshold of human-level capability, there’s no reason to expect it to stop there.
The philosopher Nick Bostrom has explored this extensively. In Superintelligence (2014), he argues that the transition from human-level to superhuman AI might be rapid—a matter of hours, days, or weeks—and that the resulting entity might have capabilities we can barely conceptualize.
Consider: a system that thinks millions of times faster than humans. That can run millions of copies of itself in parallel. That can redesign its own cognitive architecture without the constraints of biological evolution. That has access to all human knowledge instantly. That never forgets, never tires, never dies.
Such a system wouldn’t just be smarter than humans. It would be to humans what humans are to ants—or perhaps what humans are to bacteria. The gap might be qualitative, not just quantitative.
This is what safety researchers mean when they talk about existential risk. Not that superintelligent AI would be malevolent, but that its goals might be subtly misaligned with human values, and a sufficiently intelligent system pursuing misaligned goals might be impossible to stop.
The Alignment Challenge
The alignment problem becomes existential in the context of superintelligence.
With current systems, alignment failures are annoying but manageable. A chatbot that occasionally gives bad advice can be corrected. A system that generates biased outputs can be retrained. Humans remain in control.
But with systems more intelligent than humans, the dynamics change. How do you correct something smarter than you? How do you align something that understands your attempts to align it better than you do?
Several approaches are being explored:
Scalable oversight: Training AI systems to help humans oversee other AI systems. The hope is that even if we can’t directly evaluate superintelligent outputs, we can use AI assistants to help us.
Interpretability: Understanding what AI systems are actually doing internally. If we can read the “thoughts” of AI systems, we might detect misalignment before it causes harm.
Constitutional AI: Having AI systems follow explicit principles that constrain behavior regardless of capability level.
Cooperative AI: Designing systems that are intrinsically motivated to cooperate with humans rather than merely following instructions.
None of these approaches is proven sufficient. The challenge is that any alignment technique we devise is evaluated by human intelligence. A superintelligent system might find flaws we can’t perceive, exploit loopholes we can’t imagine, or manipulate us in ways we can’t detect.
This is why many researchers argue that alignment must be solved before superintelligence arrives—not after. Once the genie is out of the bottle, it may be too late to put it back.
What We Don’t Know
It’s worth cataloging our ignorance:
We don’t know why deep learning works as well as it does. The theoretical understanding lags far behind the practical results. Systems generalize better than our theories predict. We’re building on empirical success without full comprehension.
We don’t know if current approaches can reach AGI. Transformers might scale to general intelligence, or they might hit fundamental limits. The path might require breakthroughs we haven’t made yet.
We don’t know how to reliably align advanced AI systems. Current techniques work for current systems. Whether they’ll work for systems much more capable than humans is unknown.
We don’t know what consciousness requires. Current AI systems behave intelligently without (presumably) being conscious. Whether consciousness is necessary for general intelligence, and whether current approaches could ever produce it, remains mysterious.
We don’t know how society will adapt. The economic, political, and social implications of AGI are unprecedented. History offers no clear analogies.
We don’t know where the real limits are. Physics imposes constraints, but we don’t know how close we are to them. Intelligence might have limits we haven’t encountered, or it might be expandable in ways we haven’t imagined.
This uncertainty is uncomfortable. We’re building transformative technology without knowing where it leads. But uncertainty cuts both ways—it’s a reason for caution, but also for humility about confident predictions of doom or utopia.
Scenarios
Given what we know and don’t know, several futures seem possible:
The Slow Takeoff: AGI arrives gradually, with clear milestones and time for adaptation. Systems become incrementally more capable. Society adjusts. Alignment problems are addressed as they arise. The transition is transformative but manageable.
The Fast Takeoff: Once AI reaches human-level capability, recursive self-improvement accelerates rapidly. A system that’s human-equivalent in 2027 might be superintelligent by 2028. The transition happens too fast for institutions to adapt.
The Plateau: Current approaches hit fundamental limits. AGI remains elusive. AI becomes very useful but not transformatively general. The status quo continues with better tools.
The Catastrophe: Misaligned superintelligence pursues goals incompatible with human survival. By the time we recognize the danger, it’s too late. Human extinction or permanent disempowerment follows.
The Flourishing: Aligned superintelligence solves problems humans couldn’t. Disease, poverty, aging—conquered. Human potential expands dramatically. A new chapter in the story of intelligence begins.
These scenarios span from mundane to apocalyptic to utopian. The honest assessment is that we don’t know which is most likely. We’re playing a game whose rules we don’t fully understand with stakes that couldn’t be higher.
What Do We Do?
Given this uncertainty, what should we do?
Invest in alignment research. The problem is hard, the stakes are high, and current efforts are inadequate. More resources, more talent, more attention.
Develop governance frameworks. International coordination on AI development. Monitoring of frontier capabilities. Standards for safety testing. This is harder than it sounds—AI development is global, competitive, and fast-moving.
Maintain epistemic humility. Confident predictions about transformative technology are almost always wrong. We should prepare for multiple scenarios while acknowledging our uncertainty.
Proceed thoughtfully. Neither racing forward heedlessly nor abandoning progress entirely. The technology will advance—the question is whether we advance it carefully.
Keep humans in the loop. As systems become more capable, maintain meaningful human oversight. This will become harder over time, which is why starting now matters.
None of this guarantees good outcomes. But these approaches improve our odds in a situation where the odds matter enormously.
The Unfinished Story
We started this book with a question Turing asked in 1950: Can machines think?
Seventy-five years later, we have machines that behave as if they think, at least in certain contexts. Whether they truly think—whether there’s something it’s like to be a large language model—remains philosophically contested. But the practical question has shifted. The question now isn’t whether machines can think, but what happens when machines think better than us.
This is no longer a philosophical puzzle. It’s an engineering challenge, a governance problem, and an existential question rolled into one.
The story we’ve told in this book is the story of how we got here: from Turing’s paper to the age of symbolic AI to the winter to the paradigm shift to the learning revolution. From systems that followed rules to systems that learn. From narrow intelligence to something approaching general capability.
But the story isn’t finished. We’re not at the end; we’re in the middle.
The learning machines will keep learning. That’s what they do. Every day, systems train on more data, process more parameters, discover new capabilities. The curve keeps rising.
The question—the question we can’t answer, the question that keeps researchers up at night and governments scrambling to adapt—is whether we’ll learn fast enough ourselves.
Turing made intelligence visible. We learned to create it through learning. But the creation has taken on a life of its own. What started as a quest to understand the mind became a race to build minds better than our own.
We don’t know how it ends. We’re writing that part now, in the code we write, the policies we set, the choices we make. Every chapter after this one is unwritten.
The learning machines are learning. The question is what we’ll learn in time.
Epilogue
Alan Turing died in 1954, four years after asking whether machines could think. He never saw the field of artificial intelligence born. Never saw the winters or the summers, the false starts or the breakthroughs. Never saw systems that could write essays or prove theorems or converse like humans.
He saw farther than almost anyone. His question shaped seventy years of research. His test, however imperfect, gave us something to aim for. His vision—that thinking might be something machines could do—turned out to be right, though the path was nothing like anyone imagined.
If he were alive today, what would he make of it? The systems that pass his test while failing at arithmetic. The debates about consciousness and alignment. The race toward something that might be the most important—or the most dangerous—technology humans have ever built.
We can’t know. But we know what he’d want: for us to keep asking questions. To remain curious, rigorous, humble. To face the unknown with the same clear-eyed determination he brought to the problems of his time.
The story he started continues. The machines are learning. And so, perhaps, are we.
Notes & Further Reading
The OpenAI Crisis (2023):
The events of November 2023 are documented in numerous media reports. The underlying tensions about AGI development remain unclear, though various accounts suggest disagreements about safety and governance.
Definitions of AGI:
For OpenAI’s definition, see their charter at openai.com. For DeepMind’s levels framework, see: Morris, M. R., et al. (2023). “Levels of AGI: Operationalizing Progress on the Path to AGI.”
The Bitter Lesson:
Sutton, R. (2019). “The Bitter Lesson.” Available at incompleteideas.net. A influential argument for scaling over specialized techniques.
Vernor Vinge on the Singularity:
Vinge, V. (1993). “The Coming Technological Singularity.” Presented at VISION-21 Symposium. The original articulation of the concept.
Ray Kurzweil:
Kurzweil, R. (2005). The Singularity Is Near. Popular treatment of exponential progress and the coming merger of human and machine intelligence.
Nick Bostrom on Superintelligence:
Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. The most influential academic treatment of existential risk from advanced AI.
Alignment Research:
For an introduction to the alignment problem, see: Christian, B. (2020). The Alignment Problem. For technical approaches, see the publications of Anthropic, DeepMind’s alignment team, and the Center for AI Safety.
Yann LeCun on World Models:
LeCun, Y. (2022). “A Path Towards Autonomous Machine Intelligence.” Available on OpenReview. Argues that current LLM approaches are fundamentally limited.
AI Timelines:
For surveys of researcher predictions, see: Grace, K., et al. (2018/2023). “When Will AI Exceed Human Performance?” Updates available at AI Impacts.
The Human Era:
For thoughtful exploration of what superintelligence means for humanity, see: Russell, S. (2019). Human Compatible: AI and the Problem of Control.