How a model actually works (without a PhD)

A large language model does not retrieve a stored answer. It is a giant statistical machine that, given a sequence of tokens (pieces of text), predicts a probability distribution over the next token, samples from it, appends that token, and repeats. That loop is the entire 'conversation.' There is no second system that checks the answer against a database of truth unless you built one. The fluency is the product of the loop, not proof that a lookup happened. If you remember one sentence from today, remember that. Fluency is cheap. A lookup is a system you design. Do not let a fluent paragraph talk you into believing a lookup occurred.

Training squeezed a huge amount of human text into the weights — the numbers inside the model. Those weights are frozen when you use a hosted chat product or an API (unless you fine-tune, which you almost never should in month one). The model is not browsing your hard drive. It is completing the prompt you sent. If your policy PDF was not in the training mix, and you did not put it in the prompt, it is not 'in there somewhere.' Completion will still produce a policy-shaped paragraph. That is the trap.

This is why style, format, and 'sounds right' come for free, and why facts about your Q3 policy do not. Completion is not knowledge lookup. A model that has seen a million status reports will write a status report in the house style of the internet. It will not know that workstream B is red because the vendor missed the gate on Thursday unless that fact is in the context you sent, or in a tool result you sent back in.

Sampling is where 'creativity' actually lives. After the model produces a probability distribution, a token is chosen. If you always pick the single most likely token, the text is dry and repetitive. If you allow more randomness, the text is livelier and more willing to invent. Temperature is the common knob: low for extraction and JSON, higher for brainstorming. Delivery implication: a RAID miner should not be on a creative setting. A naming workshop can be. This is a spec choice, not a personality trait of 'the AI.'

Stop saying 'the AI thinks' in working conversations. Say 'the model predicted' or 'the model completed.' It sounds pedantic for one week and then it saves you. Thinking implies a belief state and a memory. Completing implies a prompt and a distribution. When a sponsor says 'it thought we were green,' you can answer 'it completed a green-shaped paragraph because the source was thin and nobody required evidence.' That sentence points at a control. 'It thought' points at a ghost.

Meeting language you can steal: 'The model is extremely good at language. It is not a database of our company. If we need it to use our documents, we have to put the right documents in the prompt, or retrieve them. That is a delivery problem, not a model-brand problem.' Say it once in the first steering meeting. You will have to say it again. People relapse into magic because the output is fluent. Fluency is not the test. Grounding is.

Text is chopped into tokens — sub-word chunks. English is roughly 0.75 words per token (about 1.3 tokens per word); code and other languages differ. APIs charge per token in and per token out. A 4,000-word policy PDF might be ~5–6k tokens before you even ask a question. You do not need a tokenizer for a go / no-go. You need an order of magnitude. If someone says 'we'll just paste the contract,' multiply words by 1.3 and ask what else has to fit in the same window.

The context window is the working memory of one request: system instructions + retrieved docs + chat history + the user question + the model's answer all compete for the same budget. GPT-class models now offer windows from ~128k to 1M+ tokens, but stuffing the window is slow, expensive, and often less accurate than retrieving the two right pages. A bigger window is a bigger budget. It is not a better filing system. Attention gets noisy. The model over-weights the start and the end and skims the middle. Your important clause in page 47 is not safe just because it 'fit.'

Nothing in the context window is 'remembered' after the request unless you store it yourself (memory, a database, the chat log you resend). 'It learned our process last Tuesday' is almost always false — you resent the process, or you retrieved it. Consumer chat products simulate memory by resending history and, in some products, by writing notes to a memory store. That store is a product feature, not a property of the model. In an enterprise build, if you did not persist it, it is gone.

Budget like a RAID log, not like a hope. Allocate: system prompt (small, stable), retrieved evidence (the few pages that matter), user question, reserved output, a little history. If history is eating 40 percent of the window, you will retrieve less evidence and quality will drop. If the system prompt is a 2,000-token pep talk, you are paying for adjectives. Delivery leads can demand a budget table on the design. Builders respect a budget. They do not respect 'just put it all in.'

Why stuffing 200 pages fails, even when the window can hold them. Cost: you pay for every input token, every turn. Latency: reading 200 pages before writing a sentence is slow. Quality: the model is worse at finding the needle when you dump the haystack into the prompt. You wanted retrieval. You built a very expensive paste. The fix is not a bigger window. The fix is chunking, a search step, and putting the two right pages in the prompt. Day 6 and week 2 will make that mechanical. Today you only need to refuse the paste.

When a sponsor says 'but the new model has a million tokens,' answer with the budget, not a brand fight. 'We can use the long window as a safety net. We will still retrieve. We will measure whether stuffing beats retrieval on our eval set. We will look at p95 latency and cost at 10×. If stuffing wins on our tasks, we will use it. We will not assume the window is a knowledge base.' That is an eval-driven answer. It is also how you avoid buying a long-context model to paper over a missing search index.

The training objective rewards plausible next tokens, not true ones. When the model does not know, it still has to emit something that looks like the rest of the sentence. That is hallucination: confident, grammatical, wrong. It is not a bug you file with the vendor and wait for a patch. It is what you get when you ask a completion engine for a fact it was not given. You can reduce it. You cannot prompt it out of existence.

It gets worse when you ask for citations, IDs, legal clauses, or 'the email from March' that is not in the context. The model will often invent a document name that looks right. SOP-14b.pdf is a very plausible filename. So is a section 4.2.1. So is a case ID in the house format. Plausible form is cheap. Your control is: require a quote from the provided source, or say unknown. If the output contract allows a free-form citation, you will get free-form fiction.

Types you should name in a RAID log, because they have different controls. (1) Grounding miss: the source was not in context, so it completed from weights. Control: retrieve, or refuse. (2) Fusion: it mashed two real facts into a third that is false. Control: ask for quotes, eval on composition. (3) Fabricated citation: the claim might even be true, the source is invented. Control: verify the source exists, not just the vibe. (4) Sycophantic hallucination: the user wanted green, it wrote green. Control: rubric plus evidence, not 'confirm we are on track.' (5) Tool-confabulation: it describes a tool result it never got. Control: only allow claims that came back from the tool.

Delivery implication: you never ship a system whose only control is 'please don't make things up' in the prompt. You ground it (put sources in context), you constrain outputs (JSON schemas, allowed tools), you evaluate, and you put a human on the high-cost mistakes. Prompting helps. It does not replace those four. If a vendor demo 'never hallucinates,' they either constrained the task to something easy or they did not show you the hostile cases. Ask for the empty-source run.

High-cost vs low-cost mistakes. Inventing a punchy subject line for an internal draft is cheap. Inventing a contractual obligation, a medical dosage, a refund eligibility, or a safety procedure is expensive. Delivery people already triage defects by blast radius. Do the same here. Cheap mistakes can live in a copilot with a 'check this' warning. Expensive mistakes need retrieval, schema, eval, and a human queue. Do not let a demo on cheap mistakes set the control strategy for expensive ones.

Meeting language: 'Hallucination is the model doing its job — completing — without evidence. We will treat ungrounded answers as defects, not as personality. The fix is sources in the prompt, a schema that allows unknown, an eval set that includes empty and hostile inputs, and a human on anything that moves money, safety, or a customer commitment.' Then stop talking. If they want a promise that it will never happen, they want a database, not a model. Sometimes the right product is a database with a search box.

Three places knowledge can live. Weights: what the model absorbed in training (the public internet, books, code — stale, generic, not your SAP). Context: what you put in this request (the prompt, the files, the retrieved chunks). Tools: functions the model can call (search, ticket system, calculator, your API) whose results come back into the context. Almost every useful enterprise system is a model plus the right context plus a few tools plus a human. If you cannot say which of the three holds the fact you care about, you are not ready to scope.

Weights are a terrible place to put company policy. They are expensive to update, they mix your facts with the internet, and you cannot cite them. Fine-tuning the weights is the last lever, not the first. If a vendor leads with fine-tune, ask what they tried with retrieval and evals first. Fine-tune has a real use later: style, a narrow format, a domain dialect after the task is proven. It is not how you 'teach it our SharePoint.' That sentence should make you twitch.

Context is the default lever. You already know how to brief a contractor: give them the documents they need for this task, not the entire file share. Retrieval-augmented generation is a fancy name for 'search, then paste the hits into the brief.' The delivery problems are the search quality, the chunking, the permissions on the hits, and the eval. Those are week-2 and week-3 problems. Today's job is to stop people from skipping context and jumping to weights. When someone says 'we'll fine-tune it on our SharePoint,' they have skipped the default lever. Put the documents in the request first. Fine-tune is a last resort, not a knowledge strategy.

Tools are how the model sees live systems. A lookup_customer tool is not 'the model knows our customers.' It is your code running a query and stuffing the result into context for the next token loop. That is good. It is also how you get a write path: create_ticket, send_mail, post_journal. Read tools are a search problem. Write tools are a change-management and audit problem. Do not let a demo treat them as the same interface. Day 6 will press this. Today, just keep tools in the third column so nobody says 'we'll fine-tune it on the CRM.'

A worked placement. Fact: 'Our refund window is 30 days for SKU-A and 14 days for SKU-B, changed last Thursday.' Weights: no. Context: yes, if the policy page is retrieved. Tools: yes, if a policy API returns the current window. Human: yes, if the refund is above a threshold. If your design says 'the model knows our refund policy,' you have not placed the fact. Placement is a delivery artifact. Write the three columns on the charter. Empty cells are the build. Filled cells are the promise. A steering committee can live with an empty cell that has an owner and a date. They cannot live with a filled cell that was never true.

When someone says 'we need a bigger model,' ask which column is empty. A bigger model can be a little better at using the context you gave it. It cannot invent a policy that is not in weights, context, or tools. Upgrading the model is sometimes right (latency, tone, long-context behavior, tool use). It is the wrong first move when the policy is not in the request. Three questions before anyone buys a bigger model: Is the source in context? Did we eval on our tasks? What does 10× cost? If they cannot answer, it is not a model problem yet.

After the model scores the next token, software has to pick one. Greedy decoding always picks the top token. It is dull and stable — useful for classification and JSON. Temperature rescales the distribution before sampling: low (0–0.3) makes the top tokens even more dominant; high (0.8–1.2) flattens the distribution so unlikely tokens get a chance. Top-p (nucleus) sampling cuts the tail and samples from the smallest set of tokens whose probabilities add up to p. You do not need the formula. You need to know these are delivery settings with owners, not 'make it more creative' in a chat UI.

Worked defaults you can put in a spec. Extraction / schema / RAID miner: temperature 0 (or 0.1), top-p 1, stop sequences for the schema. Status digest: 0.2–0.4 so it can paraphrase without inventing workstreams. Brainstorming names: 0.8. Anything that may be pasted to a customer or a regulator: low temperature plus evidence rules. If a team cannot tell you the temperature of a production prompt, they do not have change control yet. Day 4 will make versioning explicit. Today, treat sampling as part of the contract.

Failure mode: a demo at temperature 0.7 that looks lively, then production at the same setting invents an owner because liveliness includes making up names. Failure mode: temperature 0 on a writing task that then repeats the same sentence structure until users hate it. Failure mode: different clients using different defaults so the eval set from last week does not match this week's behavior. Pin the settings next to the prompt version. When quality drifts, you want to know whether someone 'turned up creativity' in a playground.

Seed and determinism. Even at temperature 0, hosted APIs are not always bit-exact across calls — batching, load, and vendor-side changes exist. Do not promise a court that the same prompt will always emit the same bytes unless you have a vendor commitment and a pinned model version. Promise instead: same prompt, same retrieval, same schema, evaluated on a set, with a human on the high-cost path. That is an operational guarantee. Bit-exact replay is a research wish. If legal asks 'will it always say the same thing,' the honest answer is 'it will be evaluated the same way, and a human still signs the high-cost path.' That sentence survives a deposition. 'It is deterministic' often does not.

What to say when a sponsor asks for 'more creative' or 'more accurate' as if those were brands. Creative: raise temperature on a drafting prompt, keep the factual prompt at 0, do not mix them. Accurate: do not raise or lower temperature as your first move; put sources in context and require evidence. Sampling is a weak lever for truth. It is a strong lever for variety. Mixing those two requests into one prompt is how you get lively fiction. Split the prompts.

Put sampling on the RAID log if the task is high-cost. Risk: playground settings leak into production. Mitigation: settings live in the version header of the prompt, reviewed like a config change. Risk: a 'creative' brand workshop prompt gets reused on a policy copilot. Mitigation: do-not-use-for line, which you will write on day 4 and day 7. You already know config drift. This is config drift with better PR. If nobody owns the setting, the playground will become production on a Friday afternoon. Name the owner in the header. Unowned knobs turn.

Short answer: no, not by default. A hosted model does not 'know' your SharePoint, your SAP, or last Tuesday's decision log. Those live in your systems. The model knows (in the weights) a lot of public language, including generic versions of processes that sound like yours. That is why a first demo is so convincing and so dangerous. It can write a refund policy. It cannot write your refund policy unless you put your refund policy in the request. Say that once in the first steering meeting. You will have to say it again. Fluency makes people forget. Your job is to keep putting the fact back in the right column.

The sentence to say out loud: 'It does not know our data. It can use our data if we retrieve the right pieces into the prompt, or if we give it a tool that queries the system of record. Until we build that path, it is guessing in our house style.' Then draw the three columns. Put 'Q3 policy' under context or tools, never under weights. If they point at a consumer chat product that 'remembers' them, explain that the product is storing and resending notes. That is an app feature. We can build a feature. We should not confuse it with knowledge in the model.

What they often mean, decoded. Sometimes they mean 'will employees paste confidential data into a consumer tool' — that is a governance problem (week 3). Sometimes they mean 'can it answer questions about our policies' — that is retrieval plus permissions. Sometimes they mean 'will the vendor train on our prompts' — that is a contract problem (day 3). Sometimes they mean 'can it see the CRM' — that is a tool with authz. Unpack the question before you answer. 'Does it know our data' is four questions wearing a trench coat.

A bad answer: 'Yes, we'll upload everything.' That is stuffing, permissions soup, and a future incident. A bad answer: 'No, so we must fine-tune.' That is the wrong column. A bad answer: 'The enterprise version automatically knows our tenant.' Some products index your drive. Many do not, or they index poorly, or they ignore permissions. Ask what is actually indexed, how permissions flow, and how you eval. Do not take 'connected to Microsoft 365' as 'knows our data.' Connected is a pipe. Knowledge is a retrieval eval.

Worked exchange. VP: 'So once we buy the seats, it knows our policies?' You: 'Seats give us a model and a window. Policies live in Confluence with messy titles and old versions. We will pick one policy set, retrieve it, cite it, and measure whether answers match the page. If that works, we expand. We will not upload the whole tenant on Friday.' Then you put a date on the first eval. VPs can live with no if you give them a path. They cannot live with yes that becomes an invented clause in month two.

Write this as a standing slide. Title: Where facts live. Three columns. One row for each fact class: policy, customer state, prices, HR, live tickets. For each, mark weights / context / tools / human. The empty cells are the build. The marked cells are the promise. When the VP asks the question again in week six, open the slide. Do not reopen the philosophy. You already ran this meeting. Today you are installing the picture so that meeting is short.

You now have a mechanical picture: next-token, a context budget, a sampling knob, and three places a fact can live. The delivery question is what you put in front of that machine so a fluent wrong paragraph does not become a program. Four controls, in order: grounding (the right sources in this request), constraints (schema, allowed tools, unknown is a legal answer), evals (empty, thin, hostile — scored), and a human on the high-cost mistakes. Prompting helps those four. Prompting is not a fifth control that makes the four optional. If a design has a long system prompt and none of the four, it is a demo.

Grounding is the control you own first. Put the two right pages in the prompt, or put a tool result in the prompt, or refuse. 'We'll tell it to be accurate' is not grounding. Neither is a million-token window stuffed with the haystack. Grounding has an eval: on a labeled set, did the answer's claims appear in the provided source, and did the citations point at real quotes. If that score is bad, you have a retrieve problem or a refuse problem. You do not yet have a model-brand problem. Fix the source path before you buy a smarter completion engine.

Constraints are how you stop plausible form from becoming a business record. JSON/schema, allowed enums, owner null if unnamed, a fallback token when nothing qualifies, tools the task is not allowed to call. Constraints do not make the model truthful. They make untruthful output harder to ingest. A valid object with an invented owner is still a fail — schema is not grounding — but an invalid essay cannot load into Jira at 2 a.m., which is progress. Pair the schema with the missing-data rule. Pair both with an eval that includes empty source. Constraints without that eval are a style guide.

Evals are how you know the first three worked on purpose. Twenty to fifty rows, including empty and paint-us-green, scored as format / evidence / usefulness, dated, with a prompt version. A steering meeting that has never seen a failing row is a steering meeting that has only seen a demo. Bring one failing row to the first steer. It teaches the room faster than a lecture on next-token. Week 3 will make evals grown-up. Today's bar is: you can name the four controls and you will not ship a system whose only control is a polite sentence in the prompt.

The human is a control, not a lack of faith. Cheap mistakes (a punchy subject line) can live with a warning. Expensive mistakes (refund, safety, a customer commitment, a contractual clause) need a queue and a named reviewer. HITL without a queue is a slogan: the output will go straight through the first Thursday you are late. Write the queue, the SLA, and the fallback ('if the queue is more than a day deep, we turn the path off'). You already staff hypercare. Staff this the same way, smaller. Unstaffed HITL is how a demo becomes a production incident with a fluent paragraph at the top.

Meeting language that ties the day together: 'It completes. It does not know our data. Hallucination is the objective running without evidence. We will ground, constrain, eval, and put a human on the expensive mistakes. Sampling is a spec setting, not a personality. A bigger model does not replace an empty column in weights / context / tools.' Then draw the loop, the budget, and the three columns. Then stop. If they still want a promise that it will never invent, they want a database. Sometimes the right product is a database with a search box. Saying that is a delivery skill, not a lack of ambition.

Worked failure that uses all four. Policy copilot invents a clause. Grounding miss: the page was not retrieved, so it completed from weights. Constraint miss: the spec allowed a free-form citation instead of a quote-or-unknown. Eval miss: nobody ran the empty-source case. Human miss: the paragraph went to a manager without a queue. The RAID row is not 'the AI hallucinated.' The RAID row is four missing controls. Your job in the war room is to name them in that order and pick the first fix (retrieve the page, require a quote, add the empty-source test, put a human on clause-level answers). That order is the day, applied.