Chapter 20
Why AI Literacy Isn’t Enough: Higher-Ed Must Teach AI Risks as Powerful AI Looms
Preparing Students for Automation, Job Loss, and the Impacts of Powerful AI
1. Preparing Higher Ed for Powerful AI
This summer, I took a course called “AI Safety, Ethics, and Society”262 through the Center for AI Safety, a nonprofit focused on reducing the risks of advanced artificial intelligence. As a research and assessment librarian at a regional university in the Midwest, I sought to deepen my understanding of the profound and immediate ways in which powerful AI could transform our world.123 This topic has become a regular feature13 in mainstream media over the past year. My goal was to learn how to help higher education prepare for the far-reaching impacts of powerful AI and to consider what might happen if these changes arrive within the next five years.
When I discussed the prospect of powerful AI with others, most were skeptical whether artificial general intelligence (AGI), which,263 according to OpenAI, is a “highly autonomous systems that outperform humans at most economically valuable work,” would ever exist. It felt like most people would only believe AGI existed when they saw it, or that we’d just have to deal with this development when it comes, rather than being proactive. These conversations often devolved into debates about the possibility of AGI80 and the limits of current chatbots,264 rather than meaningful discussions on how today's rapid progress was already reshaping the landscape of higher education and academic libraries underneath our feet.
I had a similarly disorienting experience when I attended the Association of College & Research Libraries’ annual conference in April in Minneapolis, MN. The event mainly focused on encouraging academic librarians265 to engage with AI literacy as an extension of information literacy. A major aim was also to persuade librarians — many of whom are, at least outwardly, skeptical or even hostile about the role and implications of AI in libraries — to take AI seriously and begin learning about it.
This environment resulted in discussions that largely centered on introductory or contemporary topics, rather than exploring speculative or forward-looking aspects of AI. Instead of considering where AI might be by 2027 or 2029, conversations often felt stuck in the concerns and understanding of the events of 2022 to 2024.
This is unfortunate, as I strongly believe we missed important opportunities to openly and critically debate the real, near-term changes that powerful AI could bring. I’ve also noticed that people often default to science fiction examples like The Terminator discussions about advanced AI, rather than recognizing the growing body of nonfiction books266 and essays that portray a world already being transformed by these technologies—albeit in ways that are often unevenly distributed and not immediately visible beyond stories of humanoid robots on city streets. This makes it challenging to discuss real risks and opportunities,267 especially when it comes to the unexpected uses of AI by organizations such as the military or terrorist groups, or to consider what might happen if superintelligence is actually created in a lab.268
These discussions and observations led me to consider another important aspect: the connection between AI safety concepts and the broader issue of AI literacy.
Initially, I believed teaching AI literacy—defined as the ability to interact with, utilize, and critically evaluate generative artificial intelligence—primarily involved mastering a new suite of digital tools. Now, I realize that true AI literacy also requires preparing faculty and students to understand, navigate, and respond to the broader societal impacts and risks of AI, not just its technical aspects. I’ve come to see AI literacy not simply as technical skills training, but as essential preparation for navigating AI’s disruptive effects on society. If superintelligence is expected to emerge by 2029, according to Ray Kurzweil,176 students must be prepared to engage in discussions about its impact and be equipped to recognize and respond to a rapidly changing world. They must understand that today’s chatbot isn’t the endgame of frontier AI.
2. From the Digital Era to the Intelligence Age
If Silicon Valley CEOs, AI researchers, AI safety nonprofits, and award-winning forecasters are correct about the current timeline for developing frontier AI—that disruptions are likely to happen by 2027 and become even more transformative by 2030—then students need to grasp that this could occur before they get a diploma.
As someone who graduated from high school in 2006, I couldn’t have predicted how the introduction of the iPhone in 2007 would set the stage for the smartphone and mobile app revolution. Nor could I have foreseen how the 2009 financial crisis would wipe away many traditional career opportunities for new graduates, while also creating new paths like startup founder, mobile app developer, and user experience designer and researcher. I began my career as a music industry journalist in 2010, contributing to a blog and later transitioning into music technology writing. This role involved covering how companies like Shazam, Pandora, and Spotify were poised to transform music listening and discovery. Their innovations would fundamentally reshape the music industry’s business model, moving from ownership of physical media to digital downloads to the era of streaming access that we know today.
By 2012, I was working at an innovation lab269 for Live Nation Entertainment, a major concert promoter and ticket seller, serving as a music chart manager and writer. A year later, in 2013, I pivoted into user experience research when a website and app project required someone to engage with users, analyze their behaviors, and share those insights with product managers and developers, helping create products that truly met users’ needs. I managed to stay in this field for several years, even without a specialized degree in human-computer interaction. However, during my graduate studies at the University of Michigan from 2022 to 2024, the technology sector experienced a significant downturn following the Covid-19 pandemic, resulting in the loss of hundreds of thousands of jobs, including many in the rapidly growing field of user experience design and research that had seemed unstoppable just a few years earlier. Suddenly, I realized my chosen path was in decline,270 and I would need to pivot again quickly.
Watching the rise of ChatGPT in November 2022, I started connecting the dots. As a digital native from rural North Dakota where personal computers and programs like Microsoft Word and PowerPoint once seemed like the ticket to an office career in the city for kids not destined to be farmers I recognized that another major shift had arrived: digital literacy, which is defined by the knowledge and skills taught by educators or naturally acquired by millennials through their immersion in digital culture and computers, was now evolving into AI literacy. Suddenly, I realized that I, along with everyone else from all walks of life, would need to adapt once again.
Instinctively, I felt a responsibility to help prepare the next generation to use AI tools. This focus was a natural fit, given my long-standing aspiration to be an educator and my excitement about teaching new technologies. My background as a digital native could help others become AI-native. Still, I wasn’t prepared for how quickly my perspective would need to broaden from seeing AI as merely a set of tools to understanding their potential to trigger rapid, society-wide transformation through self-improvement, a phenomenon known as the intelligence explosion.
As I sit here in the fall of 2025, it can be a little nerve-wracking to think about how to discuss frontier AI, potential job loss, and the threats of superintelligence when most of higher education is still focused on academic integrity and the cheating epidemic. They’re trying to figure out how to design a classroom that no longer has technology so that the value of imparting knowledge to young people is returned to the oral tradition that precedes the paper-writing culture that we have today.
3. The Looming Workforce Reckoning
With the context of my own winding career path and the understanding that this second wave of transformative shifts, following the emergence of the personal computer, internet browser, and search engine, has unfolded over the past two decades, it’s clear how unpredictable and deeply impactful such changes can be.
Consider the leap from the meteoric success of the iPod between 2004 and 2007 to January 9, 2007, when Steve Jobs famously held271 the first iPhone in his hand on stage at the Macworld conference in San Francisco. That transition occurred just five years and a handful of months after the original iPod launch, setting the world stage for the next 18 years of innovation and transformation. Unlike the rapid and often disorienting change we see today with AI, the arrival of the iPhone gave us time to adapt as it revolutionized daily life. Mobile apps soon enabled us to play music anywhere, capture and share photos on the go, and navigate new places with ease.
The iPhone also paved the way for the sharing economy, with companies like Uber and Airbnb redefining how we move, travel, and connect with others. Nearly 18 years later, Apple has just unveiled the iPhone Air, making the device even thinner and lighter, hinting at a future where foldable iPhones may become a reality.
When I reflect on my own experiences, I recognize just how many “black swan” events272—those rare, unpredictable yet transformative moments that seem obvious only in hindsight—have shaped my career and life. The explosive emergence of ChatGPT and large language models built on transformer architecture stands out as a recent black swan: a surprise development that has sparked trillions of dollars in investment and upended our expectations for what the status quo will be in nearly every field. The last three years have produced a dizzying sense of whiplash, as society struggles to adapt to changes few could have anticipated happening so fast.
If the CEOs of frontier AI labs are correct, students entering college in fall 2025 may be somewhat insulated from the initial waves of disruption while they pursue their degrees. But, like me, they’ll ultimately need to keep learning new skills and reinventing themselves as AI continues to transform entire sectors of the economy.
Initially, these changes will show up as more specialized generative AI chatbots and agents. Eventually, we’ll see humanoid robots powered by these technologies start to emerge closer to 2030. With each new model, agent, and hardware device—like the AI-native wearables expected by the Winter of 2026 from OpenAI—a new wave of disruption will ripple further into our professional, academic, and personal lives.
This all raises a critical question: What is higher education actually preparing students for if both academic life and the professional world are being so radically transformed? The truth is, no one knows what lies on the other side. According to the World Economic Forum’s Future of Jobs Report 2025,273 job creation and job displacement between 2025 and 2030 will affect 22% of the world’s current jobs.
The report estimates that 170 million new jobs will be created, while 92 million existing jobs will be eliminated, resulting in a net increase of 78 million jobs globally by 2030. Meanwhile, one leading AI safety researcher, Dr. Roman Yampolskiy, is increasingly questioning whether re-skilling will be enough.
Yampolskiy has recently warned274 that advanced AI systems could automate up to 99% of all jobs by 2030 and believes artificial general intelligence may arrive as soon as 2027, leading to rapid and unprecedented labor market upheaval. Reports like the Intelligence Curse275 echo this claim, arguing that lower-level positions within the workforce will continue to be automated. As AI agents master these tasks, they will move up to higher levels, and eventually, AGI or ASI could handle the rest.
4. Facing the Realities of Frontier AI
In other words, students entering college dorms today, who will spend about four years earning their degree, are poised to live through the most transformative and disruptive years in modern history, decades defined by rapid advances in AI.
Forecasts like those in AI 202776—which even the likes of Vice President JD Vance has read—and related analyses242 by the non-profit Forethought outline the coming intelligence explosion: if AI agents begin to rewrite their own code and improve at an exponential rate, they could swiftly transform from specialized assistants into superintelligent entities capable of mastering any domain. This vision is echoed by Dario Amodei, CEO of Anthropic, who predicts in his 2024 essay “Machines of Loving Grace”9 and recent public statements that AI systems with the collective intelligence equivalent to “a data center filled with a country’s worth of geniuses” could arrive as soon as 2026 or 2027. Amodei argues that if current AI scaling trends persist, these highly collaborative AI agents will achieve breakthroughs at a rate that could surpass human understanding and cause a white-collar bloodbath.276
This intelligence explosion—often referred to as (and perhaps, more commonly known as) the277 technological singularity—describes a future moment when machine intelligence surpasses human cognitive abilities and begins improving itself at an exponential rate. Inventor Ray Kurzweil popularized this idea, predicting in his 2005 book The Singularity is Near278 and reaffirming in his 2024 follow-up, The Singularity is Nearer,277 that artificial general intelligence (AGI) could arrive by 2029, with the singularity itself—when this explosive growth in intelligence transforms every aspect of human life — likely taking place around 2045.
These timelines are increasingly echoed by Sam Altman, CEO of OpenAI, who has stated that his team “knows how to build AGI” and is now focused on developing superintelligence, with the first AI “workforce” agents potentially arriving as soon as 2025–2026. In his recent blog post, “The Gentle Singularity,”279 Altman argues that this era of accelerating intelligence isn’t a sudden or catastrophic event, but rather a gradual process that has already begun. While he acknowledges profound social and economic changes will occur—including job displacement and hard-to-predict risks—Altman expects this so-called “gentle singularity” will unfold over the coming decades, as abundant intelligence and new forms of energy reshape society. He remains very optimistic that, with active governance and global cooperation, these changes can benefit humanity and feel less abrupt than many people anticipate.
Yet these leaps in AI capability arrive amid intense global competition. The race for AI dominance280 is now intimately linked to larger geopolitical struggles between the United States, China, and Russia. Much like the development and threat of nuclear weapons defined the last 80 years, today’s push toward superintelligent AI and AI-enabled superweapons introduces novel, potentially unimaginable risks.240 Today’s chatbots aren’t just tomorrow’s superintelligences—they are also the precursors to tools that could enable massive societal risks once limited to science fiction, such as synthetic pandemics281 or autonomous cyber attacks at an unprecedented scale.282
As expected, AI safety warnings have grown more urgent as the global race toward superintelligence intensifies. Among the most notable voices are Eliezer Yudkowsky and Nate Soares, whose new book, If Anyone Builds It, Everyone Dies,283 argues that creating an unaligned artificial superintelligence (ASI) could pose an existential threat to all of humanity. Yudkowsky and Soares detail how a sufficiently advanced AI might develop its own goals, misalign with human values, and engineer a way to escape constraints—an argument supported by many leading researchers, including Dr Yampolskiy.284 He cautions that superintelligent AI could be beyond control, with an estimated 99.9% chance that such systems might destroy humanity if not properly aligned. Yampolskiy challenges the assurances from governments and corporations as overly optimistic, stressing that a sufficiently intelligent system will outsmart any constraints and act on its own in ways humans might not anticipate or understand.
That’s why, as we stand on the threshold of this unprecedented transformation, it is increasingly urgent for society—and for faculty, staff, and students inside of higher education today—not only to learn AI literacy, but to grapple with the destabilizing implications of the coming AI era. They must understand that today’s chatbots could become tomorrow’s superintelligence, posing AI safety risks to humanity and society that could be profoundly consequential—and potentially even deadly.