What works in AI, and what’s just hype

Artificial Intelligence (AI) has rapidly evolved from a theoretical concept to a transformative force in our world today. But with this progress has come a surge of misinformation and exaggerated claims. In an effort to cut through the noise, AI Snake Oil, a book co-authored by Arvind Narayanan and Sayash Kapoor, critically examines AI’s limitations and the often-unrealistic hype surrounding it. On The Interface podcast, Mr. Sayash explores the crucial need for a clearer understanding of AI’s true capabilities and its pitfalls.

“It’s crucial, just like the creation of the Food and Drug Administration (FDA) in the early 20th century was necessary to prevent snake oil salesmen from taking advantage of the public, that we now separate the genuine advances in AI from the false promises,” Mr. Sayash explained.

The term “snake oil” refers to the deceptive sales of miraculous products in the late 19th and early 20th centuries, paralleling the misleading claims about AI today. According to Mr. Sayash, AI’s ability to deliver real progress is often overshadowed by companies selling unproven and unreliable technologies as the next big thing

In the book, the authors draw attention to two main categories of AI -- predictive and generative. Predictive AI, which seeks to make predictions about individuals based on past data, has been particularly susceptible to such overblown claims. Mr. Sayash points out that while AI can often be positioned as a tool to revolutionise sectors like hiring or law enforcement, predictive AI has shown consistent flaws.

“A tool that uses video interviews to predict job performance is essentially no more than an elaborate random number generator,” he noted. Despite the lack of supporting evidence, such tools continue to be marketed to companies across industries.

Generative AI, however, offers a more promising outlook. Unlike predictive AI, which attempts to foresee future outcomes, generative AI focuses on creating useful content based on existing data. “We aren’t trying to predict the future, but rather create something useful in the present,” Mr. Sayash said. He highlighted the impact generative AI has already had in knowledge work, especially in areas like coding.

“More than half of the first draft of code I write these days is done using generative AI systems,” he added, acknowledging that tools like these can greatly improve productivity, provided their limitations are understood and respected.

While generative AI offers benefits, Mr. Sayash is cautious about its current shortcomings, particularly the issue of “hallucinations”—when the AI generates false or fabricated information. These errors have led to significant consequences, particularly in the legal field. There were instances where lawyers have been penalised for using AI-generated content without verifying its accuracy.

Despite these challenges, he believes that with human oversight, generative AI can be valuable. “When experts are aware of the limitations and know how to correct mistakes, generative AI can be very useful for most knowledge work,” he opined.

AI’s true potential lies not just in developing better models but in effectively deploying these technologies across different productive sectors of the economy. “It’s not just about building better models; we need to focus on diffusing these tools throughout society—across healthcare, education, and finance,” Mr. Sayash emphasised.

The future of AI, however, is not without debate. While some fear the rise of artificial general intelligence (AGI), which could surpass human intelligence, Mr. Sayash offered a more moderate view.

“I don’t think AI systems today pose an existential threat to humanity,” he argued, drawing a parallel to the Internet’s evolution. “Just like the Internet changed the world over decades, AI will also transform industries, but the impacts will be felt over a long period.”

Looking ahead, Mr. Sayash is cautiously optimistic. He acknowledged the undeniable impact of AI on knowledge work but stressed the importance of managing its risks. Whether AI will ever achieve the level of human intelligence remains uncertain, but as Mr. Sayash noted, the true challenge lies not in the technology itself, but in how we choose to use it.

(Listen to the full discussion with Sayash Kapoor on the Interface podcast or watch the YouTube video for more insights.)