You don’t need to be a computer scientist to use AI wisely, but it helps to understand what happens when you type a question into a chatbot. Today’s most capable AI systems are, in effect, black boxes, and the sections below explain why.

“AI” is a big umbrella

“Artificial intelligence” is a broad term for computer systems that do tasks we normally associate with human intelligence. It has described many different technologies over the years:

• IBM’s Deep Blue chess engine beat world champion Garry Kasparov in 1997. Today, free programs like Stockfish play far better than any human.
• Google Maps uses machine learning to predict traffic and estimate when you’ll arrive.
• Phone dictation and automatic video captions rely on speech recognition.
• Spam filters, streaming suggestions, and social media feeds use AI to decide what you see.

Note that each of these systems does one narrow job. A chess engine can’t give you directions, and a map app can’t write an essay. (For a broader overview, see IBM’s What is artificial intelligence?)

What most people mean by “AI” today

When people talk about AI today, they usually mean assistants like ChatGPT, Claude, Gemini, and Copilot. These run on foundation models, which are very large AI models trained on enormous amounts of data and adaptable to a wide range of tasks. Stanford researchers coined the term in 2021 to describe this new kind of general-purpose AI.

The foundation models behind today’s assistants are large language models (LLMs), trained on vast amounts of text to process and generate language. Many can now work with images and audio, too. The assistants built on them also connect the models to other tools (e.g., web search, document readers, tools for running code).

Unlike a chess engine or a map app, these systems do something new. They take on a wide range of cognitive labor that has traditionally required a person: drafting and editing, summarizing, analyzing, writing code, tutoring, answering questions, and giving advice.

If you’d like to go deeper, Timothy B. Lee and Sean Trott’s Large language models, explained with a minimum of math and jargon is a good, readable introduction. 3Blue1Brown’s Large Language Models explained briefly covers the basics in a short video.

Built from neural networks no one fully understands

LLMs are built from neural networks, a type of software loosely inspired by the brain. A neural network is made of many layers of simple calculations, connected by numbers called parameters (or weights). During training, those numbers get nudged a tiny amount at a time, across an enormous number of examples, until the network is good at its task.

GPT-3, released in 2020, had 175 billion parameters. Counting them at one per second, without stopping, would take more than 5,500 years. Today’s leading models are likely larger, but most companies no longer say.

No one writes an LLM’s knowledge or skills into it line by line. Engineers design the network’s structure and the training process, and the abilities emerge from training. What comes out works remarkably well, but it’s billions of numbers that can’t be read like a rulebook.

Even the scientists who build these systems can’t fully explain why they work so well. As MIT Technology Review put it, large language models can do jaw-dropping things, but nobody knows exactly why.

A research field called interpretability is trying to change that. Anthropic researchers, for example, have identified millions of concepts represented inside one of their models and have begun tracing the steps a model takes to reach an answer. By their own account, these methods still capture only a fraction of what happens inside the model, even for short, simple prompts. Anthropic’s CEO has written about the urgency of interpretability because we still don’t understand how these systems work.

One token at a time, with a roll of the dice

When an LLM answers you, it’s repeating one simple step: predicting the next token.

A token is a word or a piece of a word. A common word like “the” is usually a single token, while a longer word might be split into several pieces. (You can see how text gets divided with OpenAI’s tokenizer tool.)

Given all the text so far (your question, plus anything the model has already written), the model calculates how likely each possible next token is. Then it picks one at random, weighted by those probabilities. The chosen token gets added to the text, and the process repeats until the response is complete.

The demo below shows a single step. A model has read the start of a sentence and assigned a probability to each possible next word. (The numbers are made up for illustration. A real model weighs many thousands of possible tokens at every step.)

The sailboat drifted slowly across the ______

• bay38%
• water27%
• lake16%
• harbor11%
• horizon8%

This is why you can ask the same question twice and get two different answers. The randomness is real, but it isn’t arbitrary. It’s more like rolling heavily weighted dice: likely words come up often, and unlikely ones come up occasionally. The companies and developers who build AI tools decide how much randomness to allow, balancing predictability against variety.

It’s surprising how much apparent intelligence comes out of this one repeated step. It also means every response is a chain of weighted random choices, each shaped by billions of numbers no one can interpret. For more, see Stephen Wolfram’s What Is ChatGPT Doing … and Why Does It Work?, or try Georgia Tech’s interactive Transformer Explainer, which shows next-token probabilities as a real model generates text.

Trained on data we can’t see

Before an LLM can predict anything, it has to be trained. Developers feed it an enormous collection of text (e.g., websites, books, articles, code), and the model adjusts its parameters to get better at predicting what comes next. Meta, for example, reported that its Llama 3 models were pretrained on more than 15 trillion tokens, far more text than a person could read in thousands of lifetimes.

Developers filter, curate, and supplement that data. After the initial training, they shape the model further, often with human feedback, to teach it how to respond (e.g., what tone to take, what to refuse, how to handle sensitive topics). Each of those choices affects what a model “knows” and how it answers.

The concern is that the companies behind the leading AI assistants, including ChatGPT, Claude, and Gemini, don’t disclose their models’ training data or parameters. OpenAI’s GPT-4 technical report, for instance, declined to share details about its training data, training methods, or even the model’s size, citing competition and safety. Stanford’s Foundation Model Transparency Index finds that major developers are least transparent about their training data, and that average transparency scores dropped sharply in 2025.

Some companies release “open-weight” models whose parameters anyone can download, but they rarely publish the training data. Fully open models, such as Ai2’s OLMo, are the exception.

Say you ask an AI assistant:

Explain whether it is ever right to lie to protect someone’s feelings.

You’ll get a thoughtful-sounding answer. That answer draws on the model’s foundational knowledge, which comes from training data you can’t inspect, curated by people you’ll never meet, according to priorities they haven’t published. We don’t know where that knowledge came from, let alone why the model’s weighted dice produced this particular sequence of words.

Looking things up: retrieval-augmented generation

A model’s training data stops at a cutoff date, and no model memorizes everything. To answer questions about recent events, or about specific documents, AI systems often use retrieval-augmented generation, or RAG.

RAG works in three steps:

1. Search. The system looks for relevant information (e.g., web pages, an organization’s documents, files you’ve uploaded).
2. Retrieve. It selects the passages that seem most relevant to your question.
3. Generate. It places those passages into the model’s working memory (its context) alongside your question, and the model writes a response.

RAG can make answers more current and more accurate, and it lets AI tools cite sources. But it doesn’t change what the model does. The model is still generating one weighted-random token at a time, so it can still misread a source, blend sources together, or invent details that appear in none of them.

Stanford researchers found that even specialized legal research tools built this way hallucinated on at least 1 in 6 benchmark questions. A Columbia Journalism Review study found that AI search tools answered more than 60% of queries incorrectly when asked to identify the source of a news excerpt.

Note that retrieval adds another layer you can’t see. You rarely know everything the system searched for, what it found and left out, or why it favored one source over another.

“Thinking” before answering: reasoning models

Many of today’s most capable systems are reasoning models. Before giving a final answer, they generate a long stream of intermediate text, working through the problem step by step, trying different approaches, checking their work, and backing up when something doesn’t add up.

This still happens one token at a time. Each new token is predicted from everything that came before it, so when a model writes out intermediate steps, those steps become part of the text it builds on, like a scratchpad. Rather than jumping straight from question to answer, the model can take many smaller steps, each informed by the last.

A few years ago, researchers found that simply prompting a model to work through a problem step by step improved its answers. Today’s reasoning models are trained specifically to reason this way, and they may generate thousands of tokens of “thinking” before responding.

The results can be dramatically better, especially in math, coding, and complex analysis. But the reasoning is largely hidden from you:

• Companies don’t show you the raw reasoning. OpenAI chose not to show users its reasoning models’ raw “chain of thought,” offering a model-written summary instead. Anthropic’s documentation says no setting returns Claude’s raw chain of thought; developers get a summary or nothing at all.
• Visible reasoning may not tell the whole story. Anthropic researchers found that reasoning models often leave out important factors that influenced their answers, so a tidy explanation isn’t proof of how an answer was reached.
• The reasoning is hard to interpret. It can be long and difficult to follow, and it comes from the same network no one can fully read. Researchers from several leading AI companies have described the ability to monitor AI reasoning as a new and fragile opportunity.

An ultra-capable black box

The foundation models behind ChatGPT, Claude, and Gemini are extraordinarily capable. On many tasks, their writing and analysis rival the work of skilled professionals. But they aren’t transparent. In practice, they’re ultra-capable black boxes.

None of this makes AI useless, and it isn’t a reason to avoid it. Rather, it’s a reason to treat AI the way a good sailor treats the sea, with respect and a clear sense of where you’re headed. Since no one can look inside the box, we have to watch how it behaves, and how we behave when we use it.

Up nextWhat do we know about AI and how humans use it? →