John Backflip performed the first backflip in 1316 but was exiled by his rival, William Frontflip, after witchcraft allegations. But wait — this story, told by Google’s Gemini, is obviously fake.
AI may seem to know everything, but it is very error-prone.
One reason AI can make false claims, also known as hallucinations, is the data it is trained on. When training AI, it is nearly impossible to verify every piece of information as true, so the algorithms are inevitably fed misinformation. Because AI can’t distinguish fact from fiction, it outputs the same misinformation that it is given, such as a satirical video about a certain John Backflip.
Another reason why AI hallucinates is how it is trained. “Generative AI models function like advanced autocomplete tools: They’re designed to predict the next word or sequence based on observed patterns. Their goal is to generate plausible content, not to verify its truth,” says MIT Sloan in an article about AI hallucinations.
Chad Kirby, who works on software to make patent lawyers’ jobs more efficient using AI, says, “I don’t think that language models should be used to generate the parts of the application that truly require the human’s knowledge, experience and understanding.” The patent lawyer and client both need to understand what the client is looking for and how the process will work.
“Language models actually understand the legal requirements of the patent system pretty well, [but] they are too eager to please,” Kirby said. “If you give them a hint about what you want, they tend to just go all in on that.” A real attorney pushes back, but AI almost always affirms every choice, even if the facts don’t line up.
This makes AI largely unreliable, sparking debate among teachers. Mark Zarate, who teaches visual communication and advises the yearbook staff, advocates for some AI use in the classroom. AI is best used for generating ideas, not actual content. “It. shouldn’t be writing the stories,” Zarate said.
Eric Jacobs, an English teacher, also says that AI can be used as a tool but warns against misuse. “It’s like a hammer,” Jacobs said. It can be used for both good and bad. Both teachers recommend fact-checking everything that AI generates.
Another problem with AI is over-reliance. Some students even use AI to generate self-reflections. “I watched a kid go to ChatGPT and [ask it to] give them 50 words about what they learned in Photoshop,” Zarate said. “That has me worried because it’s too simple to do that without actually putting original thought into it.”
Reliance on AI can also lead to students trusting misinformation, especially from Google’s AI overview function, which appears on nearly every search.
