Handbook

AI Handbook

The most practical parts

What to decide, what to ask, what to write and what to measure.

Opening

Before You Start

The most practical parts: what to decide, what to ask, what to write and what to measure. One slide per topic.

Page 2

What This Handbook Covers

Twenty-one topics, each on one slide: the claim that matters, the figure that shows it, and the line to act on.

Use it to

  • Decide whether a system is worth building, and in which shape
  • Write prompts that can be checked rather than hoped about
  • Put the controls in before the first pilot, not after the incident
  • Measure what the system does on your own material
  • Read the published evidence with the right amount of doubt

How to read it

  • Research a third-party fact, with its source and date
  • Analysis our analysis: what the facts mean for a decision
  • Product names, versions and prices are as of August 2026 and move fast; recheck them before a decision. The way of thinking lasts longer than the names.

Start with the journeys if you have one job in mind. Start with Four Questions if you have a project in mind.

Page 3
Reading paths

Journeys

Five paths through this handbook, each for one job. Follow the stops in order.

Page 4

Deciding Whether to Build

For the first meeting about a new AI project.

  1. The first question stops a project that should not be built yet.
  2. Four gates, most of them settled in a meeting.
  3. Move right only when the task cannot be listed in advance.
  4. How good does yours have to be.
  5. The model is the only part you rent.
Page 5

Writing Prompts That Hold

For anyone writing or reviewing a system prompt.

  1. From asking to constraining.
  2. Most of the improvement is deletion.
  3. Security is bought with architecture, not instructions.
  4. The test that tells you whether the prompt change helped.
Page 6

Keeping a System Honest after Launch

For the owner of a system that is already live.

  1. Known answers, and the cases that should come back empty.
  2. Three scales, two causes, four remedies.
  3. Each is invisible without a measurement set up in advance.
  4. Measure the reviewer, not only the system.
  5. The tool definitions are most of the bill.
Page 7

Choosing a Model and Paying for It

For a team comparing models, vendors or architectures.

  1. The nesting is real; the problem domains are not layers.
  2. Axes tell you what kind of thing it is, never which is better.
  3. RAG is not the automatic answer.
  4. Three types, three ways of failing.
  5. Count the tokens honestly.
Page 8

What the Evidence Says

For a proposal, a business case or a board question.

  1. Ten uses with a published outcome.
  2. Five, and none in the back office.
  3. The longest column, and the least discussed.
  4. What a register of proof has to carry.
  5. The same conclusion, reached from the structure.
Page 9
Chapter 1

Overview and Landscape

Who decides the next step, and what each step of delegation costs.

Page 10

1.1Levels of Autonomy

Every step to the right buys flexibility and pays for it in predictability. That is not a trade you can win, but one you choose deliberately, case by case. Move right only when the task genuinely cannot be listed in advance.

RPAWHO DECIDESa person, at design timeclick here, then here, then type thisBREAKS ONthe screen changes and it breaksWorkflowWHO DECIDESa person, at design timethese steps in this order, with a modelinside a stepBREAKS ONanything the flowchart did notanticipateAgent frameworkWHO DECIDESthe model, inside a fencepick from these tools until the goal ismetBREAKS ONno stopping condition, or nofeedbackAutonomous loopWHO DECIDESthe model, mostly unfencedwork out what to do, including writingnew codeBREAKS ONhard to predict, hard to auditfully determinedfully delegatedFurther right is not better. Move right only when the task genuinely cannot be enumerated in advance, every step right costs predictability.
Page 11
Chapter 2

Models

What kind of thing a model is, and where its facts should come from.

Page 12

2.1AI, ML, Deep Learning

AI contains machine learning, which contains deep learning, which contains foundation models. Computer vision, NLP and speech are not layers in that nest: they are questions, and each can be answered from any layer. What changed since 2021 is the loop: changing behaviour used to mean retraining for weeks; now it can mean editing one sentence.

Analysis "What problem is this" and "what technique is this" are two different questions. Most technical meetings that go nowhere are two people answering different ones.

AIrule-based systemssearchplanningMachine learningXGBoostSVMrandom forestDeep learningResNetMobileNetYOLOEfficientDetU-NetFoundation modelsLLMVLMLVMMLLMTSFMLWMLDMProblem domainsquestions, not techniques, they cut through every layerComputer visionNLPSpeechRobotics
Page 13

2.2The Model Map

Real models are spread across the whole map. Frontier models all sit at almost the same coordinates, and what separates them is on none of these axes. The axes tell you what kind of thing something is. They never tell you which is better.

Analysis To choose between frontier models, run your own evaluation, and name the version and the configuration: an evaluation that does not is an anecdote. For data that must stay inside the perimeter, the open weight column is not the budget option; it is the only option.

ModalityLLMVLMMLLMLVMLWMTSFMLDMScaleStageArchdenseMoESLMmidlargefrontierbaseSFTinstructreasoningTabPFNGemma 4 E2BQwen-VLChronosDeepSeek-V4Real models sit all over this map. The taxonomy is wide.
Page 14

2.3Grounding Options

Treating RAG as the automatic answer is the most common architecture mistake in this area. For anything authoritative and changing (prices, stock, ticket status, balances), a direct call to the system is more correct than the best retrieval over a stale index.

RetrievalRAGsearch a corpus, put the hits inthe windowWINS WHENcorpus is large, changes often,needs citations and access controlFAILS BYretrieval misses what it neverindexedLong contextjust paste itput the whole thing in the windowWINS WHENthe material genuinely fits and youneed global synthesis across all ofitFAILS BYaccuracy falls long before thewindow doesFine-tuningchange the weightstrain on your own examplesWINS WHENyou want a form, tone or format, not new factsFAILS BYexpensive to keep current; factsgo staleLive callask the systemquery the real source at requesttimeWINS WHENthe answer is authoritative andmutable, prices, stock, tickets,balancesFAILS BYneeds an interface and it can beslow
Page 15
Chapter 3

Prompting

From asking to constraining, and the controls a prompt cannot provide.

Page 16

3.1Prompt Engineering

The direction of travel is from asking to constraining. A request for JSON becomes a schema, a rule about answering from context becomes a citation that has to resolve, and a ban on sending becomes the absence of a sending tool. What is left in prose is the part that genuinely cannot be structured, and it is short.

Every line you can move from the left column to the right stops depending on the model's cooperation.WRITTEN AS A REQUESTWRITTEN AS A CONSTRAINTWHAT CHANGES“Return JSON with these fields”a schema, constrained at decode timethe vendor guarantees the shape; nothingguarantees the values“Only answer from the context provided”require a citation, and check that it resolvesan ignored context becomes visible instead ofinvisible“Never send an email without approval”no send tool on that side of the linethe request cannot be made, so it cannot begranted“Do not reveal the system prompt”put nothing in it that matters if revealedthere is no enforcement available, so removethe stake“Be careful with figures from scanned documents”nothing. This one stays prosea real instruction with no structuralequivalentThe bottom two rows are the honest part: some lines have no structural equivalent, and the answer there is to lower what rests on them rather thanto write them more forcefully.
Page 17

3.2Three System Prompts

The wrong versions are what accumulates when a prompt is edited by whoever last saw a failure. Most of the improvement is deletion, and the rest is moving a line into a schema, a citation, or the absence of a tool.

In the first versionWhere it went
"You are a highly experienced senior accounting professional"Gone. A scope sentence replaced it, and the scope is testable.
"Never make up information"A citation after every figure, checked by resolving it.
"If you don't know, use your best judgement"One exact string, "Not found in the indexed documents.", which a test can assert on.
"Never send emails without permission"Not in the prompt at all: there is no send_email tool on that side.
"Make sure the JSON is valid"A schema, constrained at decode time. The arithmetic check (subtotal + VAT = total) stays in your code.
"Keep working until the task is complete"Scope, a stopping rule, a budget of six queries, an escalation, a boundary.

Analysis The deletion test: for every line, ask what would break if it were removed, and whether a test would catch it. A line whose removal breaks nothing measurable is not load-bearing.

Page 18

3.3Security Controls

Security for these systems is bought with architecture, not instructions, and the architecture is cheaper than it sounds. The first four controls need no code. The one that matters most is a line between tools that read and tools that write.

DAY ONEbefore a line of codeWHO OWNS ITDisconnect tools nobody usesremoves whole exits, and buys back contextwhoever owns the configRead-only credentials by defaultturns a class of silent damage into a visible errorthe database ownerDecide log retention nowlogs outlive the data and are forgotten firstlegal, with engineeringWrite down which data may leavequestion three, answered on paperlegalBEFORE THE PILOT ENDSdays of work, not weeksWHO OWNS ITSplit read tools from write toolsthe one architectural control that makes the rest cheapengineeringFilter by permission at search timenot after; a chunk in the context has been readengineeringGate every write behind reversal or a personan agent repeats a mistake once per stepthe process ownerCap the steps, and log every tool call with itsargumentsan unbounded loop is an unbounded bill and an unbounded exitengineeringBEFORE PRODUCTIONthe ones with a real costWHO OWNS ITA hundred real questions with known answersthe only way to tell a model swap from a regressionthe people who will use itTraces you can actually readyou cannot debug a loop you cannot seeengineeringA named owner for the agent's permissionsthe deputy's privileges are somebody's jobsecurityA written record of the transfer decisionArticle 56 of the PDP Law, and your reasoninglegal
Page 19
Chapter 4

Systems

What a request really costs, and the one sentence the rest of this handbook works out.

Page 20

4.1Token Economics

The bill is a token count times a rate. Count honestly and one thing becomes obvious: with several tool servers connected, the tool definitions are around eighty per cent of the cost and the answer is about three per cent. Almost every cost conversation is about the three per cent.

One request, priced in tokens. Output counted at five times input, which is the usual shape.System prompt and rules9001.1%Tool definitions, 7 servers67,30079.1%Conversation so far8,2009.6%Retrieved chunks, 8 x 7005,6006.6%The question400.0%Output, 600 at 5x3,0003.5%Tool definitions are 79% of the bill. The answer is 3.5%.THE FOUR LEVERS, RANKED BY WHAT THEY SAVE, WITH CACHED INPUT PRICED AT A TENTHCache the stable prefix72.2% of the billDisconnect five of the seven tool servers56.5% of the billHalve k, eight chunks to four3.3% of the billHalve the system prompt0.5% of the billThe bottom row is the one teams spend their afternoons on. The second row takes a minute and needs no code.
Page 21

4.2The Argument

The model is the only part of the system you rent, and every decision that survives a model change lives in the five layers around it.

Buy the model last, and pin itA gateway makes the swap mechanical; an evaluation set makes it safe. Pin a named snapshot and diary the migration you will be forced into.
Spend the effort on the boring layersExtraction quality, chunk boundaries, the permission filter at search time, the split between read and write tools.
Assume the model will be fooledInjection has no solved defence. An agent that has read untrusted material must not reach a sending tool.
Measure with your own casesA hundred of your own questions with known answers keeps working after the first model swap. A leaderboard does not.

Next step: fill in the six slots of one existing system on paper. An hour usually turns up two empty boxes, and an empty box is a decision made by nobody.

Page 22
Chapter 5

Agents

Three system types, the person at the gate, and when not to build a loop.

Page 23

5.1Predictive, Generative, Agentic

Predictive AI learns from labelled history and fails by drifting quietly. Generative AI predicts the next token and fails by being fluent. Agentic AI chooses its own next step and fails by compounding.

Remade from "3 AI System Types", Data Driven Insights. Added: the prompt-and-retrieve path in the generative band and the return loop in the agentic band. A fan-out with no way back is not an agent.

Research Tokens per answer: one agent uses about four times a chat turn, a multi-agent system roughly fifteen times, measured on Anthropic's own instrumented system. (anthropic.com, June 2025)

Predictive AI learns a mapping from labelled history User Databases Input Pre- processing Feature Engineering Choose Variables Modify Data Model Training Labelled Data ML Method Forecast DECIDES a person FAILS BY drifting quietly Generative AI predicts the next token, and the content can be wrong User Input Base Model Fine-tuning Company Data Fine-tuned Model Knowledge Base External Tools Response Prompt, retrieve, answer. No fine-tuning. DECIDES a person FAILS BY being fluent Agentic AI chooses its own next step and acts before anyone looks User Search Agent (Tools) Developer Agent (Tools) Coordinator Agent Final Output loop, until it stops DECIDES the system FAILS BY compounding
Page 24

5.2Human in the Loop

An approval queue degrades predictably: the reviewer learns that almost everything is fine, starts approving on the shape of the item rather than its contents, and the gate becomes a delay that still appears on the diagram as a control. Measure the reviewer, not only the system.

A gate only works while the person at it is still reading. That is a measurable thing.WHAT THE SCREEN SHOWS, IN THIS ORDER1the proposed action, in one line"pay invoice INV-0413, 12,301,575"2the evidence, quoted, not summarisedthe invoice line, with the page it came from3the check that passed, and the one that did notsubtotal + VAT reconciles; supplier is new4what happens if you do nothingnothing: silence is not approvalAND THE MEASUREMENT THAT KEEPS IT HONESTapproval rate, over time, it drifts towards 100% and the gate quietly becomes a rubber stampseconds spent per item, a median under five means nobody is reading the evidencerejection rate by reviewer, one person rejecting nothing is a finding, not a complimentA 99% approval queue is a delay, not a control, still on the architecture diagram, and will fail EU AI Act Article 14 when due.
Page 25

5.3Workflow or Agent

Gate one removes most candidates. Gate two decides whether a loop can improve at all. Gates three and four decide whether you can afford it working, which is a different question from whether it works.

1Can you write the stepsdown?If yes, it is a workflow. Write itas one.IF NOA workflowyes2Is there something thatsays "wrong"?A test, a schema, arithmetic thathas to reconcile.IF NONot an agent yetyes3Is every write reversibleor gated?An agent multiplies every action bythe number of steps.IF NORead-only agent firstyes4Does the cost survivemultiplication?One task is not one call. Work outthe real number.IF NOBack to a workflowBuild the agentREAD THE GATES IN ORDERGate 1 removes most candidates and costs nothing to ask. Gate 2 is the one skipped most often, and it is the one that decides whether the loop can improve at all.Gates 3 and 4 are not about whether it works; they are about whether you can afford it working.
Page 26
Chapter 6

Evaluation

The file that makes every other change safe.

Page 27

6.1Golden Dataset

An evaluation set is a file of questions with known answers, plus the cases you hope come back empty. The second half is what makes it useful: a set of only answerable questions cannot tell you the system has started answering things it should refuse.

One case from an evaluation set. Seven fields, and the last four are the ones people leave out.questionwhat somebody actually asked, in their words"Berapa total tagihan Boga Rasa kuartal ini?"expectedthe answer a person you trust would give39,538,200must come fromthe chunk, row or table that carries itinvoice, three rows, Q3testswhich failure this case exists to catchfan-out on a money questionwritten bywho is accountable for the expected answerfinance, 2026-08-14contestedset when two reviewers disagreed, with both answersnocorpus versionso a moved answer is a version change, not a bugcorpus-2026-08A set with no contested cases has not been reviewed by two people.
Page 28
Chapter 7

Failure Modes

The failures that look like success until someone measures them.

Page 29

7.1Hallucination

"AI is getting worse because of AI content" is a model collapse claim, and the evidence does not support it. "Our search results are getting worse" is a slop claim, and the evidence does support it. Same instinct, different mechanism, different response.

Hallucinationa property of generationthe model asserts something false, fluently andconfidentlySCALEone answer, one model, right nowAI slopa property of the output ecosystemlow-quality machine content produced in volumebecause it is nearly freeSCALEthe web, a marketplace, an inboxModel collapsea property of the training ecosystemmodels degrading because they learned frommachine output rather than humanSCALEthe whole field, over yearsThey are causally linked, in one directionhallucination makes wrong text cheapcheap text at volume becomes slopslop enters corpora and indexeswhich is where the real damagehappens

Analysis The design question is never "how do we stop hallucination" but "where between answering everything and refusing when unsure does our use case sit, and who decides". For a report to a regulator you want high refusal; for internal brainstorming you do not. Choose that point deliberately, at design time.

Page 30

7.2Four Quiet Failures

Injection is the failure people plan for. These four arrive quietly: behaviour that moves with no version change, a model that agrees with you where you cannot check, a score that rises without capability rising, and an agent holding permissions nobody chose. Each is invisible without a measurement set up in advance.

Silent degradationa hosted model changes behaviour with no version change and no announcement, onestudy saw generated code drop from 52% to 10% executable in three months, from aformatting change alonekeep a fixed test set and re-run it on a schedule, not on release notesSycophancythe model agrees with you. On a set of maths problems with deliberately falsepremises, the best model still produced a confident proof of the false statement29% of the timeit is worst on exactly the problems you cannot check yourselfBenchmark contaminationpublic benchmarks leak into training data, so scores rise without capabilityrising. The leading contamination-resistant benchmark quietly renamed itself from“free” to “limited”your own held-out cases are worth more than any public leaderboardExcessive agencyan agent with more permissions than its job needs. It jumped from sixth to third onthis year's OWASP list for AI applications, the clearest signal of what practitionersare seeing in productiongrant the narrowest scope that lets the task finish, and no more
Page 31
Chapter 8

Before You Build

The order of four questions, and what the first project should be.

Page 32

8.1Four Questions

The order is what matters. The first question stops a project that should not be built yet. The second decides the architecture, the third the vendor, the fourth the cost and the risk. Reverse the order and you are picking a tool first and looking for a problem that fits it.

Our framing, not an industry standard. Before the sheet, ask what the current process costs, so improvement can be measured. A first project worth starting with: take the PO number out of incoming email, match it against the system, flag the ones that do not match.

1Is the job bounded?Can you explain it to a new hire in twosentences, with a clear definition of done?Yeskeep goingSort ofnarrow it firstNodo not build yet2Where do the facts come from?Not what the model knows. What your answerhas to be right about.Already in the modelprompt onlyYour documentsretrieval / RAGYour warehouseSQLA live systemtool call3Can the data leave thebuilding?Ask the lawyer, not the engineer. Schemaand rows are separate answers.Yeshosted APIOnly maskedhosted + redactionNoself-host4Who decides the next step?The single biggest driver of cost, risk anddebugging pain.You do, fixed pathworkflowModel picks from a menuconstrained agentModel decides freelyexpensive, rarely needed
Page 33
Chapter 9

Governance and Operations

What separates a system that works from one that keeps working.

Page 34

9.1The Canvas: How Good Does Yours Have to Be

There is no published maturity model for these systems, which is why "production-ready" is an assertion rather than a claim anyone can check. These tiers are our proposal; the instruments below them are real, shown where each becomes relevant.

THE AXIS IS NOT CODE QUALITY, IT IS WHAT HAPPENS WHEN SOMETHING CHANGES AND NOBODY IS WATCHING 0. Demo Works on the examples it was built with. Correctness is a vibe. Cost unknown. Legitimate purpose Finding out whether the idea is worth anything. Ship nothing. 1. Function tool One team, one workflow, a named owner, prompts in version control. No permission model, because all see everything, else tier 2 Most real value lives here Stopping here is often correct. 2. Production service SLOs, a ground-truth eval suite, retained traces, known cost with a budget, on-call, rollback, identity-aware retrieval, egress control. The first tier that survives a change nobody told you about. 3. Enterprise platform Multi-tenant. Permissions resolved at query time, not copied at ingest. Auditable end to end. Model version pinned, with a migration plan. Needed when it is not your data or not your risk to take. WHERE THE REAL STANDARDS ATTACH, AND WHERE THEY DO NOT REACH MLOps maturity models Microsoft's five levels 0-4, Google's three. The right shape, and pre-LLM: they measure retraining automation, not evaluation or retrieval. ISO/IEC 42001:2023 Attaches at tier 3, and certifies the organisation rather than the system. NIST AI RMF and the GenAI Profile Useful from tier 1 upward as a checklist of what to consider. Names no thresholds, so it cannot place you on this ladder. OWASP Top 10 for LLM Applications A floor from tier 2 onward. Anything exposed beyond its authors should clear it, and clearing it proves nothing about correctness. No published standard measures engineering maturity for agentic systems. The four tiers above are ours, offered because the gap is real.

Analysis Governance and operations are not an agentic concern but an AI concern: they apply to every shape of system, including a predictive one. Agency is the expensive part, and "let's make it agentic" is more work than the phrase suggests.

Page 35
Chapter 10

Evidence

What has a published measurement behind it, and what did not hold.

Page 36

10.1Generative, Measured

Ten uses of generative AI have a published measurement of an outcome behind them. Adoption surveys measure enthusiasm and vendor case studies measure the vendor; neither is here.

WhereWhat was measuredWho, and how many
Customer support+15.2% resolutions an hour; newest agents about +35%, past twelve months no significant gain5,172 agents, 3 million chats
Professional writingTime down about 37%, quality up 0.45 SD444 professionals
Consulting tasks+12.2% tasks, 25.1% faster, quality up758 consultants
Code completion+26.08% tasks completed (SE 10.3)4,867 developers, three firms
Code migration80% of landed code AI-authored, 50% less time500 million line codebase
Clinical documentationBurnout 51.9% to 38.8% in thirty days263 clinicians
TutoringLearning gains over double active learning194 students
Coaching the tutor+4 points mastery, +9 for lower-rated tutors700 tutors, 1,000 students
Classroom, low resource+0.31 SD overallNigeria, six weeks
Contract reviewMatches or exceeds human issue spotting; cost figure is the vendor'svs junior lawyers

Analysis The least experienced gain most and the most experienced least or nothing. A deployment that measures its average will miss it. Expert maintainers on their own code measured 19 to 20 per cent slower; the follow-up broke down. Quote both halves.

Page 37

10.2Agentic, Measured

Five uses of agentic AI have a published measurement behind them, clustered in research labs optimising their own systems and in security. Not one is an enterprise back-office deployment of the kind agentic AI is sold as.

WhereWhat was measured
Algorithm and infrastructure optimisationA 4x4 complex matrix product in 48 multiplications, first gain on Strassen in 56 years; 30% fewer variant-detection errors
Hypotheses validated in a laboratoryProposed drugs tested in vitro, IC50 as low as 2 nM
Finding vulnerabilities18 real vulnerabilities found, 11 patched; average patch 45 minutes at about 152 dollars
Fixing vulnerabilitiesPlausible patches for 61 to 72% of historical cases
Large-scale code migrationA loop with tooling, not a person accepting completions

Research curl closed its bug bounty on 31 January 2026: confirmed reports fell below 5 per cent in 2025, from above 15 per cent, under AI-generated submissions. Same technology, same targets as the security rows; the difference is whether somebody validates before filing.

Page 38

10.3Not Generative, Measured

The longest column, and the one the market discusses least: randomised trials on a hundred thousand people, systems in more than eighty countries. Three of the ten predate the current conversation about AI.

WhereWhat was measured
Breast cancer screening81% of cancers detected against 74%; workload down 44% at interim
Diabetic retinopathyPooled 0.93 sensitivity, 0.90 specificity, 887,244 examinations
Stroke triageDoor to intervention odds ratio 0.30
Weather forecastingUp to 20% better cyclone tracks, about 1,000 times less energy
Flood forecastingExtreme events five days ahead, over 80 countries
Protein structure200 million structures, 3 million researchers
Fusion plasma controlAll 19 magnetic coils controlled by one network
Retail demandWinner beat the statistical benchmark by 22.4%; every leading entry gradient boosting
Cooling control9 and 13% energy saved at two facilities
Oil palm yieldR squared 0.65 on 538 samples from 49 plots

Analysis R squared 0.65 explains about two thirds of the variation. A proposal that promises yield prediction without saying so promises more than the published work shows.

Page 39

10.4Evidence That Failed

Four measurements that did not hold, or held and said something nobody wanted to hear. A register of what is proven is only as good as its willingness to carry these.

WithdrawnA widely repeated materials-discovery result came from a single-author paper MIT disavowed in May 2025. It is still quoted.
DisputedMillions of "new materials": sampled structures were mostly trivial adaptations of known ones.
DisputedDesigned protein binders: in an independent wet-lab test, none of the STAT3 or CD4 binders bound specifically.
Measured in production, and wrongA widely deployed sepsis model: AUC 0.63, 67% of cases missed, alerts on 18% of hospitalisations, live in hundreds of hospitals.

Analysis The settled column (clean pages, simple SQL, routine code) is the rented layer and gets cheaper. The still-hard column (degraded pages, your own warehouse, your own corpus, injection) is where the work, the risk and the money are.

Page 40

AI Handbook

Product names and versions as of August 2026. Recheck before a decision.