Getting ahead of ourselves with chatbots - Fine Point Communications

Nearly two years after generative artificial intelligence’s public debut, the excitement over chatbots and their potential shows no sign of peaking. Big businesses are anxious to find generative AI features to incorporate, while start-ups look for their own ways to ride and promote the wave.

Even the National Association of Independent Schools has bought the hype.

Each summer, NAIS produces a Trendbook that counsels independent school leaders and trustees on various timely topics, an aid for their decision-making.

NAIS first discussed generative AI in the 2023-2024 Trendbook, months after ChatGPT went live to the public. It advised then that educators needed to subject the technology to more study before bringing it into classrooms.

A year later, that air of caution has vanished.

The same authors write in the newly published 2024-2025 Trendbook that generative AI’s place in education is suddenly secure and “inevitable.” “Schools cannot say no to artificial intelligence,” a subhead trumpets.

Thanks to generative AI, the flagship association for independent education now advises, “shifts in the educator experience will be inevitable.” (There’s that word again.) Schools should anticipate artificial intelligence as their new faculty, NAIS says, and start retraining human teachers “to become facilitators of each student’s personal journey with AI.”

The AI chapter is shockingly credulous, not least because of the writers’ choice to ignore a huge and perhaps unsolvable flaw at the heart of generative AI: The robot is wrong a lot of the time.

Binary code, unitary expectations

To be sure, techno-optimism is hardly limited to innovation educators and others whose careers rely on the embrace of new paradigms. According to a 2016 academic journal article, we all have a strong tendency to be overconfident about emergent technology, especially its economic benefits, to the point where decision-makers often can envision only best-case scenarios.

When technology fails us, meanwhile, such shortcomings are quickly discounted or forgotten — seen as a fluke rather than a failure — even though they occur all the time.

Take, for example, Google. While it’s helping drive the generative AI boom through its Gemini product, the company remains best known for what made Google a verb: its search engine.

There, its technology is mature, stable and a routine failure at its job: Delivering the most accurate information in response to a query. Instead, Google settles for providing “close enough” information.

In practice, close enough often equates to “good enough,” which is how Google came to dominate search. Dominance removed any pressure on the company to keep improving the product.

Yet any user of Google knows how often the search engine still does a poor job, answering a query with incorrect, slanted or outdated information.

We have simply decided to rationalize that failure as a part of using the product. Google is not reliable, but it may be the best we can get.

Technology failure like that is less dramatic than a cellular network crash or a faulty software update or a database corruption or a ransomware attack. It draws few headlines.

But it occurs far more frequently, happening around us many times a day — a digital white noise, ignored by a society dead-set on celebrating technology.

Shaped by our computers

Winston Churchill once said, “We shape our buildings; thereafter, they shape us.”

Sixty years later, Nicholas Carr echoed the point in his 2008 book The Big Switch: “The most revolutionary consequence of [the Internet] may be not that computers will start to think like us but that we will come to think like computers … as our minds are trained, link by link.”

An example of such training is search-engine optimization — the multi-billion-dollar industry of manipulating Google search results through keyword-stuffing, link-building and sentence structure designed for machines. None of this human effort makes a web page more useful, which is what people want. Instead, SEO practices aim only to meet the needs of cyberspace “web crawlers.”

The same shaping drives many social media norms (i.e., “doing it for the ’Gram”) — human activity done to satisfy computer algorithms.

As we rationalize and conform to computing’s failures, we also lower our natural skepticism and sense of right and wrong when technology is involved.

If a piece of online information seems erroneous, the awareness that “it came from the Internet” amounts to a powerful counterweight. How likely is anyone to confirm their gut feeling? Fact-check a computer? If that feels ridiculous, the Internet wins the benefit of the doubt.

Independent schools have responded to this particular dynamic by teaching “information literacy.” Still, schools and universities continue to wrestle with the common, if obviously flawed, student perception that everything worth knowing is online — and therefore searchable.

Even as cynical an observer as veteran journalist Ken Auletta made the mistake, in his book Googled, of conflating what search engines do. He described the job not as providing information but as providing knowledge.

‘No way to guarantee the factuality’

Which brings us back to generative AI’s Achilles heel, especially in regards to education: its tendency to make up stuff.

These errors go by the terms “hallucination” and “confabulation.” (The cognoscenti are having an Oxford-comma-level argument about which is proper.) No matter what it’s called, this habit is baked into the very operating system of chatbots, computing researchers say.

Essentially the problem is that ChatGPT, Gemini and their peers don’t want to admit their inability to answer a question. Providing any answer, even a confabulated one, is better than giving none at all.

“Today’s LLMs [large-language models, the technology underpinning chatbots] were never designed to be purely accurate,” explains Scientific American magazine, which quotes an Arizona State professor as saying, “The reality is: there’s no way to guarantee the factuality of what is generated.”

How often are chatbots wrong? Between 5 and 25 percent of the time, according to a recent study by Canadian computer scientists.

Such information would have been a valuable reality check, alongside the optimistic futurism, for NAIS to share with administrators and trustees in its summary of “the new reality of AI.”

After all, it’s hard to see independent school students and their parents rationalizing this degree of failure in classrooms as simply a part of embracing the future — not at $32,000 a year, the average day school’s tuition and fees.

Beyond hallucinations, generative AI consistently shows the same tendency as people to trip on two highly common sources of error:

Jumping to assumptions and faulty conclusions
Relying on secondhand sources (someone wrote that someone else said …)

All things considered, those who foresee generative AI, based on the technology we have today, as an expert or mentor and delivering true knowledge to students are fooling themselves.

The opposite of hype

Generative AI has the potential to help in limited situations. But those situations should include plenty of safeguards and extreme skepticism; anyone who copies out of a chatbot and then pastes and publishes deserves the trouble they get.

We advise clients to treat ChatGPT like it’s an intern — an eager, well-meaning but very inexperienced intern: Trust in nothing it provides you. Everything must be fact-checked with a bias toward “How do we know that?” (And that might mean first getting better at sniffing out faulty facts.)

If that counsel amounts to the opposite of hype, we plead guilty. We remain unconvinced that today’s chatbots are worth any serious investment of precious time, training and attention by overtaxed school communications offices. The reliability is simply not “good enough” for schools — not yet.

As computing companies battle for generative AI market share, they have a vested interest in making their products better and more useful. Perhaps one will eventually solve the factuality problem.

Until then, though, if you should hear anyone else recommend the retraining of teachers into IT specialists, kindly suggest: Computer scientists are cautious about the ability and potential of chatbots and LLMs, so perhaps independent schools — and their associations — should be, too.