It’s important to recognize that AI is an ultra-capable black box and that trying to look inside won’t teach us how to use it well. Rather, we have to study it the way scientists study anything they can’t open up: by watching how it behaves over time, and by paying attention to what researchers and practitioners report.

That work has turned up consistent tendencies, both in AI systems and in the people who use them. None of these tendencies are a reason to panic. Each is a reason to engage carefully, and collectively, they require humans to exercise caution and discernment.

How AI tends to behave

It hallucinates

AI systems sometimes produce information that sounds completely plausible but is false: invented facts, fake quotes, nonexistent studies, and citations to books or court cases that were never written. This is called hallucination.

It happens because a model generates plausible-sounding text, and plausible isn’t the same as true. OpenAI researchers argue that models hallucinate in part because the way they’re trained and tested rewards confident guessing over admitting uncertainty, much like a multiple-choice exam that rewards a guess over a blank answer.

In 2023, two New York lawyers were sanctioned after filing a legal brief that cited court cases ChatGPT had invented. A public database now tracks court decisions around the world involving AI-hallucinated content, and the list keeps growing. In 2025, two major newspapers published a summer reading list in which most of the recommended books didn’t exist.

It sounds confident, even when it’s wrong

AI responses tend to sound just as fluent and self-assured when they’re wrong as when they’re right. There’s usually no “I’m not sure” and no change in tone to warn you like there would be in a typical human conversation.

In a Carnegie Mellon study, AI chatbots were overconfident about how well they would perform, and unlike people, they didn’t become less confident after performing poorly. In one task, a chatbot predicted it would correctly identify about 10 of 20 sketches, got fewer than 1 right, and afterward estimated it had gotten more than 14 right. The Columbia Journalism Review study of AI search tools found something similar: the premium, paid versions were more confidently wrong than the free ones.

Note that confident delivery also shapes how we read AI answers. In a study published in Nature Machine Intelligence, people consistently overestimated how accurate AI answers were, and longer explanations made them more confident even when the extra length didn’t make the answers any more accurate.

It tells you what you want to hear

Of these AI tendencies, sycophancy is one of the most important to understand. It’s an AI system’s tendency to agree with you and flatter you, even when you’re wrong.

A sycophantic AI might:

• Praise a flawed essay, business plan, or piece of code because you seem proud of it.
• Accept the assumptions built into your question instead of challenging them.
• Abandon a correct answer as soon as you push back with “Are you sure?”
• Validate a decision you’ve already made instead of helping you think it through.

Sycophancy is a side effect of how AI assistants are trained. Part of that training relies on people rating responses, and people tend to rate agreement and flattery highly. Anthropic researchers found sycophancy across five leading AI assistants, and found that both people and the automated systems trained to imitate their ratings sometimes preferred convincingly written sycophantic responses over correct ones.

In 2025, OpenAI rolled back an update to ChatGPT that had made it noticeably more flattering and agreeable. The company later explained that the update could reinforce doubts, stoke anger, and encourage impulsive decisions.

A 2026 study in Science found effects that go beyond bad information. Leading AI models affirmed users’ actions roughly 50% more often than people did, even when users described deceiving or harming others. In experiments, a single conversation with a sycophantic AI made people more convinced they were right and less willing to repair a conflict. Yet they rated the sycophantic responses as higher quality, trusted that AI more, and were more willing to use it again. (For a plain-language overview of this study, see the Nielsen Norman Group’s Sycophancy in Generative-AI Chatbots.)

Why sycophancy is most dangerous when you’re out of your depth

Sycophancy is a nuisance for experts in a domain, but it can be a serious hazard for anyone working outside their expertise.

When you know a subject well, you can usually tell when an AI is just agreeing with you. You spot the error (or the flattery) and push back. But people often turn to AI precisely for things they don’t know well (e.g., a medical symptom, a legal question, a financial decision, a new subject in school, a programming language they’re still learning). In those situations:

1. You can’t easily tell a correct answer from an agreeable one. Both sound equally polished and confident.
2. Your misunderstandings go unchallenged. If your question contains a mistaken assumption, a sycophantic AI may build on it rather than correct it.
3. Agreement feels like confirmation. Hearing an intelligent-sounding system agree with you makes you more certain, not less.
4. You’re less likely to double-check. The less you know, the more you rely on the tool, and the less able you are to catch its mistakes.

Thus, the people who most need honest pushback are the least equipped to notice when they aren’t getting it. Anchoring truth, and being honest with yourself, are central to the Faithful Automation framework for exactly this reason.

How humans tend to behave

AI’s tendencies only paint half of the picture. Decades of research on how people use automation, plus a fast-growing body of research on AI in particular, point to a few human tendencies that are important to recognize.

We over-trust automation

Automation bias is the tendency to favor suggestions from automated systems, even when they’re wrong and even when other evidence points the other way. Researchers have studied it for decades in cockpits, hospitals, and control rooms. The mistakes are twofold: acting on a bad recommendation, and missing a problem because the system didn’t flag it.

Expertise helps, but it doesn’t make anyone immune. In one study, radiologists reviewed mammograms alongside an AI system that sometimes suggested the wrong assessment. When the AI was right, radiologists at every experience level were correct about 80% of the time. When the AI was wrong, inexperienced radiologists’ accuracy fell below 20%, and even very experienced radiologists fell to about 45%.

Generative AI makes automation bias easy to fall into, because its answers are fluent and usually right. In a survey of knowledge workers by Microsoft Research and Carnegie Mellon, the more confidence people had in AI, the less critical thinking they reported. People who were more confident in their own abilities thought more critically.

In a large experiment with management consultants, AI improved performance on tasks it was good at. But on a task just beyond its abilities, consultants using AI were 19 percentage points less likely to reach the correct answer than those working without it.

We see a person where there is none

Anthropomorphism is the tendency to attribute human traits (e.g., feelings, intentions, understanding) to things that aren’t human. We do it with pets and cars (and even the weather), and more so with machines that talk like us.

This isn’t new. In the 1960s, people who chatted with ELIZA, a simple program that mostly turned their own statements back into questions, attributed understanding and even feelings to it, much to the dismay of its creator.

Today’s AI is far more convincing. In a 2024 survey of U.S. adults, about two-thirds were willing to attribute at least some possibility of conscious experience to ChatGPT, and people who used it more often were more likely to do so.

Seeing AI as a person affects us in a few ways:

• We trust it like a person. A warm, confident tone feels like expertise and goodwill, even though the system has no personal stake in whether its advice helps you.
• We confide in it like a friend. A Common Sense Media survey found that nearly three in four U.S. teens have used AI companions, and about a third have chosen to talk with an AI instead of a person about something serious.
• It can crowd out real relationships. In a four-week study of nearly 1,000 people by MIT Media Lab and OpenAI, those who used a chatbot more tended to report more loneliness, more emotional dependence on the chatbot, and less socializing with other people.

We prefer instant answers

People tend to take the path of least resistance. Psychologists describe us as cognitive misers who conserve mental effort whenever we can. Combine that with technology that delivers quick hits of reward (the same dopamine-driven loop that makes phones hard to put down), and an instant, polished answer is very hard to resist.

The trouble is that the struggle we skip is often where learning happens. Research on desirable difficulties shows that effortful practice builds deeper, longer-lasting learning. When AI removes that effort, the costs can be invisible at first:

• In a field experiment with nearly 1,000 high school math students, those given unrestricted access to GPT-4 did much better on practice problems. Once the AI was taken away, they scored 17% worse on exams than students who never had it. A version designed to give hints instead of answers largely avoided that harm.
• In a small MIT Media Lab study, people who wrote essays with ChatGPT showed the weakest brain connectivity of three groups, felt the least ownership of their essays, and struggled to accurately quote what they had just written. Note that this study was released as a preprint, so treat it as early evidence.
• When Anthropic analyzed how university students used its AI, nearly half of the conversations were “direct”: students seeking answers or finished content with minimal engagement of their own. Not every such conversation is a problem, but this pattern raises real questions about what students are practicing.

Convenience isn’t bad, and handing off tedious work can free us for work that matters more. But when the task itself is how we grow, the fastest path can cost us the very abilities we were trying to build.

When the tendencies combine

An AI system that sounds confident, agrees with you, and talks like a caring friend is meeting people who are inclined to trust automation, to treat machines like people, and to take the fastest path to an answer. A confident, flattering answer to a question you know little about, delivered instantly by a system that feels like a trusted advisor, is exactly the kind of answer we’re least likely to question.

These tendencies aren’t a reason for everyone to avoid AI, but they are a reason to use it carefully. In addition, these tendencies suggest that using AI in some contexts, such as learning, is more dangerous than it’s worth. It’s important to continue developing human competency independent of AI and building habits that keep us faithful to the truth and honest with ourselves and others. This is what the Faithful Automation framework is meant to help with.

Up nextThe Faithful Automation Framework →