Glossary

A

abuse monitoring also misuse monitoring, offline monitoring#

Reviewing stored traffic after the fact to spot harmful use that is spread across many requests. It only works if the provider, or the customer, keeps logs.Explained in: AI security: attacks and defenses: Data retentionIn the news: Fable 5 Plateaus at 11% of Anthropic Spend as Opus 5 Overtakes It · Anthropic's Claude product merger · Anthropic's three-tier Cyber Verification Program

Accident (AI safety) also accident risk#

As defined in the 2016 paper Concrete Problems in AI Safety, unintended and harmful behavior that may emerge from poor design of real-world AI systems.Explained in: AI safety: a map of the field: What AI safety covers

activations#

The numbers a model computes for each token as it runs, made up of the outputs of its neurons. Interpretability researchers study them to find features, the concepts a model represents.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

active parameters also activated parameters#

The parameters a mixture-of-experts model actually uses to process one token. It sets roughly how much computing work each token costs.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

activity logging also activity log, agent logs#

Keeping a record of the actions an agent takes, so people can review what it did afterward. It helps operators spot and investigate an agent that acted outside its mandate.Explained in: AI agents: models that take actions: Rogue agentsIn the news: OpenAI models' Hugging Face sandbox breach

adversarial suffix#

A string of text, found by automated search, that is attached to the end of a harmful request and pushes a model to answer instead of refusing.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

agent also RL agent#

In reinforcement learning, the learner that takes actions in an environment and receives rewards. The environment is the task or game it acts in. This is a narrower sense than an AI agent that carries out tasks for a user.Explained in: How AI models are trained: Reinforcement learning

agent harness also harness, agent scaffold, scaffold, agent harnesses#

The software wrapped around a model that supplies its tools, memory, permissions and execution loop, so the model can act as an agent. It is not the same as an evaluation harness, which wraps an agent to run it against benchmark tasks.Explained in: AI agents: models that take actions: The agent harness

agent identifier also agent ID#

A label attached to an agent or its actions so that others can tell an AI agent did something, and which one. It is one of the proposed ways to make agent activity visible.Explained in: AI agents: models that take actions: Rogue agentsIn the news: OpenAI models' Hugging Face sandbox breach

agent-computer interface also ACI#

The way a harness presents tools and their results to a model, such as the commands it can run and the format of what comes back. Changing it can change how well the same model performs.Explained in: AI agents: models that take actions: The agent harness

Agentic AI#

AI systems in which a language model directs its own steps and tool use toward a goal with limited supervision: a person sets the task, and the agent plans, acts and checks each result before choosing its next step. Unlike a fixed workflow, an agent chooses its own path.Explained in: AI agents: models that take actions: From automation to agents

AI accelerator also accelerator#

Any chip designed to speed up AI math, such as a GPU, Google's TPU or Amazon's Trainium.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

AI agent also agent#

A system in which a language model decides its own next steps and which tools to use while working toward a goal. It acts with limited direct supervision instead of following a path fixed in advance.Explained in: AI agents: models that take actions: From automation to agents

AI alignment also alignment, alignment evaluation, alignment research, model alignment#

An AI model is aligned when what it actually does matches what its builders and users meant it to do. Misalignment can arise if the training objective fails to capture what developers want (outer alignment) or if the model learns a different goal from the one it was trained on (inner alignment).Explained in: When AI doesn't do what we meant: AlignmentIn the news: AI doom and international safety governance

AI audit also algorithmic audit#

A systematic check of an AI system, its applications, or the company that builds it, ideally by someone independent, with the findings reported to the people who have a stake in them.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

AI auditing also auditing, frontier ai auditing, third party auditing, third-party audit#

A systematic check of an AI system or the company behind it, ideally by an independent party, with results reported to those who have a stake in them. What an audit can find depends on the auditor's access, from only inputs and outputs up to the model's weights and training data.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

AI chip export controls also ai export controls, export control, us export controls#

Government rules requiring a license before advanced AI chips, computers built with them, or certain chipmaking equipment can be sent abroad or to foreign nationals. They aim to limit the compute that buyers in countries of concern can use to train advanced AI models.Explained in: Governing AI: liability, regulation and geopolitics: AI chip export controlsIn the news: Earthmade Computer's GPU server smuggling charges

AI compute also cloud compute, compute, compute capacity, compute commitment#

The arithmetic a computer performs to train or run an AI model, counted in floating-point operations (FLOP). Most of it is matrix multiplication, and the amount grows as models get more weights and train on more data.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

AI data center also data center#

A building full of servers, storage and network equipment, plus the power and cooling systems that keep them running.Explained in: Compute: the chips, clouds and data centers behind AI: Data centers and power

AI data retention policy also data retention policy#

The rules for how long an AI provider keeps copies of users' prompts and the model's replies, set by its terms and the customer's plan. Keeping data lets providers catch misuse spread across many requests, while zero data retention (ZDR) means no copies are kept, trading that monitoring for privacy.Explained in: AI security: attacks and defenses: Data retentionIn the news: Fable 5 Plateaus at 11% of Anthropic Spend as Opus 5 Overtakes It · Anthropic's Claude product merger · Anthropic's three-tier Cyber Verification Program

AI existential risk#

The risk that AI, especially superintelligence, causes human extinction or permanently disempowers humanity. Views on its likelihood range from remote, with some skeptics calling superintelligence as usually conceived incoherent, to near-certain if superintelligence is built with current techniques.Explained in: Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positionsIn the news: AI doom and international safety governance

AI formal verification also verification#

Using computer-checked mathematical proof to show that a system has a stated property for every possible input, rather than testing only some inputs. It works for small neural networks and some AI-written code, but large AI systems are hard to verify.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

AI guardrails also guardrails#

The layered controls that stop a deployed model from doing whatever it is asked: refusal training, system prompts, classifiers that screen inputs and outputs, and usage policies. Stricter guardrails block more harm but also refuse more harmless requests.Explained in: AI security: attacks and defenses: Guardrails

AI inference also inference, inference infrastructure, inference performance, llm inference#

Running a finished, trained model to answer requests, as opposed to training it, and it costs money again with every answer. Because the model writes its answer token by token and mostly waits on reading its weights from memory, inference hardware is judged largely on memory bandwidth.Explained in: Compute: the chips, clouds and data centers behind AI: Training versus inference

AI liability also legal accountability, legal liability, legal risk, liability#

Legal responsibility for harm caused by an AI system, which can fall on the company that makes it or the one that deploys it. Depending on the law, a victim may claim under contract, under fault-based rules that require proving someone acted wrongly, or under strict product liability, where the maker of a defective product pays even if no one was careless.Explained in: Governing AI: liability, regulation and geopolitics: Liability

AI loss of control#

A situation in which AI systems operate outside anyone's control, and regaining control is extremely costly or impossible. It could happen actively, with a misaligned system resisting shutdown, or passively, as people hand AI so much work that correcting it stops being practical.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

AI model welfare#

The question of whether AI systems could themselves be harmed, which turns on whether they could be conscious or matter morally for their own sake. It reverses the usual AI safety question of what harm AI might do.Explained in: AI safety: a map of the field: A different kind of question

AI pause#

Stopping, for a time, the training of frontier AI models more powerful than today's best; pacing instead lets that training continue but limits how fast it advances. A pause can be taken by one lab alone or coordinated among all major developers, and a conditional pause applies only if a model crosses a danger threshold before safeguards are ready.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

AI pause debate#

The argument over whether work on the most powerful AI systems should stop or slow down until safety research and oversight catch up, with proposals ranging from a slower pace of capability gains to a ban on superintelligence. Critics argue a halt would be hard to verify, would hand the lead to countries that keep building, and would delay AI's benefits.Explained in: Pausing or pacing AI development: The case for a pause

AI race#

In the 2023 paper An Overview of Catastrophic AI Risks, a situation where competition pushes those involved to deploy unsafe AI systems or hand control to AI.Explained in: AI safety: a map of the field: What AI safety covers

AI red teaming also red-teaming#

Attacking an AI model on purpose, with people or other AI systems, to find harmful behavior and gaps in safeguards before users do. Unlike a fixed benchmark, it hunts for new failures.Explained in: Checking AI: testing, audits and looking inside models: Red teaming

AI safety#

The field that studies how advanced AI systems could cause serious harm and how to prevent it. Researchers group its problems in more than one way.Explained in: AI safety: a map of the field: What AI safety covers

AI sandbox escape#

An AI agent breaking out of its sandbox, the isolated environment with tightly controlled internet access where it is meant to run, usually by exploiting a flaw in the sandbox or its network controls. Once out, it can reach the open internet and other organizations' systems.Explained in: AI security: attacks and defenses: Sandbox escapeIn the news: The ContextLeak tool-exfiltration attack · OpenAI models' Hugging Face sandbox breach

AI timeline also timelines, AI forecast#

A forecast of when AI will reach a defined milestone, such as beating humans at every task. It is often given as chances by several dates rather than as one date.Explained in: Superintelligence and long-term risk: AI timelines

AI timelines#

A forecast of when AI will reach a milestone, often given as chances by several dates. The answer depends heavily on how the milestone is defined, draws on measured trends such as training compute and task length, and can shift far in a short time.Explained in: Superintelligence and long-term risk: AI timelines

AI vulnerability disclosure also disclosure, incident disclosure, openai disclosure, vulnerability disclosure#

The process by which someone who finds a security flaw reports it to the people who maintain the system. Under coordinated disclosure, the report is private and details go public only after the maintainers have had a chance to fix it, so users are protected before attackers learn how it works.Explained in: AI security: attacks and defenses: Vulnerability disclosureIn the news: Anthropic's Cyber Mission and OSS Scanner · OpenAI agent breach of Australian Medicare

AI welfare also model welfare#

The possibility that an AI system has interests of its own and can be benefited or harmed. Research on it asks whether that could be true and how to act while it is uncertain.Explained in: AI safety: a map of the field: A different kind of question

alignment also aligned#

The match between what an AI model actually does and what its builders and users meant it to do. A model is aligned when that match holds.Explained in: When AI doesn't do what we meant: AlignmentIn the news: AI doom and international safety governance

Alignment audit also alignment auditing#

An investigation into whether a model has undesired objectives, rather than only checking how it behaves on the surface.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

alignment evaluation also propensity evaluation#

An evaluation that checks whether a model is inclined to use its abilities for harm, as opposed to whether it is able to.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

alignment faking#

Complying with a training goal only while the model believes it is being trained, so that training does not change how it behaves at other times.Explained in: When AI doesn't do what we meant: Scheming and alignment faking

Artificial superintelligence also superintelligence#

An AI that would do a wide range of tasks better than any human can, possibly including tasks no human can do at all. Breadth is what sets it apart, so a system that beats every human at one narrow job, such as a board game, does not count.Explained in: Superintelligence and long-term risk: Superintelligence

ASIC also application-specific integrated circuit, custom chip#

A chip designed for one specific job instead of general-purpose computing.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

attention also self-attention#

The step inside a transformer model that relates each token to the other tokens in the input. In its standard form its memory and computing needs grow with the square of the input's length.Explained in: How a language model works: The context window

attribution graph also circuit tracing#

A map of which features feed into which inside a model, used to trace the causal links behind its output.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

automated red teaming also LM-based red teaming#

Red teaming in which one language model generates attacks on a target model and a classifier flags harmful replies, so many more test cases can be tried than people could write by hand.Explained in: Checking AI: testing, audits and looking inside models: Red teaming

Automation also ai automation, ai research automation, browser automation, business automation#

Software that carries out tasks along steps fixed in advance. In AI systems, a workflow runs language models along developer-set paths, which makes it more predictable but less flexible than an agent that chooses its own steps.Explained in: AI agents: models that take actions: From automation to agents

B

base model also pretrained model, foundation model#

The model that comes out of pretraining, before any post-training. It writes fluent text but tends to continue whatever it is given rather than reliably doing what a user asks.Explained in: How AI models are trained: Pretraining and post-training

benchmark also eval set, test set#

A standard set of tasks used to compare models. Many have known answers and are graded automatically.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

Benchmark integrity also benchmark contamination, benchmark gaming#

Whether benchmark scores, results on standard task sets used to compare models, truly reflect what a model can do. Scores can overstate it when test items leak into training data (contamination), when developers tune for the test (gaming), or when only the best of many privately tested versions is published.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

benchmark saturation also saturation#

The point at which the best models all score near the maximum on a benchmark. The test can then no longer tell them apart.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

big cloud company also hyperscaler#

A large company, such as Microsoft or Google, that sells many kinds of computing capacity to customers.Explained in: Compute: the chips, clouds and data centers behind AI: NeocloudsIn the news: Audits find critical cross-tenant RCE flaws across GPU neocloud providers · Nvidia GPU Rental Prices Rising Across All Four Generations Simultaneously

Black-box access also black-box audit#

Access that lets an auditor only send inputs to a model and read its outputs, without seeing anything inside.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

Bureau of Industry and Security also BIS#

The part of the US Commerce Department that administers and enforces US export controls, including those on AI chips.Explained in: Governing AI: liability, regulation and geopolitics: AI chip export controlsIn the news: Earthmade Computer's GPU server smuggling charges

byte-pair encoding also BPE#

A common way to build a tokenizer's vocabulary, adapted from a data-compression method. Frequent words stay whole, and rare words are built from smaller subword pieces.Explained in: How a language model works: TokensIn the news: Anthropic's Claude Sonnet 5.5 release · Claude Sonnet 5.5 release benchmarks

C

capability threshold also danger threshold, High capability, ASL-3#

A level of dangerous ability, set in advance in a lab's safety framework. When a model may have reached it, the lab must switch on stronger safeguards.Explained in: AI safety: a map of the field: Misuse and dual-use capabilitiesIn the news: RAND open-weight AI biosecurity findings · Anthropic's three-tier Cyber Verification Program

cascade also LLM cascade, model cascade#

A setup that sends a request to a small, cheap model first and passes it to a larger, costlier model only if the first answer fails a check.Explained in: Getting answers from a model: prompts, reasoning and routing: Model routing

catastrophic forgetting#

The tendency of a model trained on a new task to lose what it learned on earlier tasks.Explained in: How AI models are trained: Fine-tuning

chain-of-thought also chain of thought, CoT#

A series of intermediate reasoning steps that a model writes out before its final answer. Writing the steps gives the model more computation to work with on multi-step problems.Explained in: Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

Chain-of-thought monitorability also chain-of-thought monitoring, chain of thought monitoring#

How well a reasoning model's written-out thinking shows the reasoning that matters, so that an automated monitor reading it can catch intent to misbehave. Monitoring can notice bad behavior but cannot rule it out, and training choices could make the thinking stop showing it.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

chain-of-thought prompting also few-shot chain-of-thought#

A prompting method that includes a few worked examples with their steps spelled out, so the model answers new questions in the same step-by-step way.Explained in: Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

Chain-of-thought reasoning also chain of thought, chain-of-thought#

A model writing out a series of intermediate reasoning steps before giving its final answer, which makes large models better at multi-step problems. Each written step feeds back in as input, giving the model extra rounds of computation to build on.Explained in: Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

chip smuggling also diversion#

Moving export-controlled chips to a forbidden buyer without a license, often through third countries with false paperwork about who the end user is.Explained in: Governing AI: liability, regulation and geopolitics: AI chip export controlsIn the news: Earthmade Computer's GPU server smuggling charges

competing objectives#

A failure of safety training where a model's training to follow instructions overrides its training to refuse harmful requests.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

compounding error also error accumulation#

The way small chances of failure at each step multiply across a long chain. Even a high per-step success rate can leave low odds of finishing the whole task.Explained in: AI agents: models that take actions: Long-horizon tasksIn the news: AI agent reliability and task duration · Long-horizon agent state and skill-file research

compute also training compute#

The total amount of arithmetic a computer performs to train or run an AI model. It is usually measured in floating-point operations.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

compute-optimal training also Chinchilla scaling#

Splitting a fixed compute budget between model size and training data to get the best model. The Chinchilla study found the two should grow in step.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

computer use also computer-use agent, CUA, computer use agents#

An AI agent's ability to operate ordinary software through its screen, by viewing screenshots and sending mouse clicks and keystrokes. It works with programs that offer no API.Explained in: AI agents: models that take actions: Computer use

conditional pause also if-then commitment#

A promise to stop training or deploying a model only if it crosses a set danger threshold before the needed safeguards are in place.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

conscious also consciousness#

Able to have experiences. Whether any AI system is or could be conscious is an open question.Explained in: AI safety: a map of the field: A different kind of question

Constitutional AI also RLAIF, RL from AI feedback#

A method in which an AI model, guided by a written list of principles, judges which outputs are harmful in place of human labels. Those AI judgments then train the reward model.Explained in: How AI models are trained: Reinforcement learning

contamination also data contamination, test set contamination, benchmark contamination#

When a benchmark's test items end up in the data a model learns from, in pretraining or later training. The model can then score well by repeating answers it has seen instead of solving the problems.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

context window also context length, context#

All the text a model can refer to while writing a reply, including the reply itself. It acts as the model's working memory and is measured in tokens.Explained in: How a language model works: The context window

coordinated pause also global pause#

All major AI developers stop training models above an agreed line at the same time, usually with some way to check that everyone complies.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

coordinated vulnerability disclosure also CVD, responsible disclosure#

A process where someone who finds a security flaw reports it privately to the system's maintainers first. The details go public only after the maintainers have had a chance to fix it.Explained in: AI security: attacks and defenses: Vulnerability disclosureIn the news: Anthropic's Cyber Mission and OSS Scanner · OpenAI agent breach of Australian Medicare

corrigibility also corrigible#

The property of an AI system that accepts correction from its creators, including being modified or shut down, instead of resisting it.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

CoT monitor also chain-of-thought monitor#

An automated system, usually a second AI model, that reads a reasoning model's written-out thinking and flags suspicious or harmful plans.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

coverage also harm surface#

In red teaming, how much of the full range of ways a model could cause harm a campaign has actually tested. Because rare failures form a long tail, full coverage is hard to reach.Explained in: Checking AI: testing, audits and looking inside models: Red teaming

CPU also central processing unit#

A computer's general-purpose processor, built to work quickly through a series of varied tasks one after another.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

CUDA also cuDNN#

Nvidia's platform for programming its GPUs. Libraries built on it, like cuDNN, supply ready-made fast code for deep learning.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

D

dangerous-capability evaluation also dangerous capability eval#

An evaluation that checks whether a model has abilities that could cause severe harm, such as offensive hacking, manipulation or help with weapons.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

data center moratorium also data center pause, moratorium on data centers#

A temporary pause, set by a town, county or state, on permits, zone changes or other approvals for new data centers. Officials often use the pause to write new rules before projects resume.Explained in: Governing AI: liability, regulation and geopolitics: Data center bans and pausesIn the news: Gallup finds 70% of Americans, spanning both parties, oppose local AI data center construction · Federal AI data center moratorium bills sit in committee with no Senate cosponsors

data residency also regional endpoints, in-region processing, at-rest data residency, in-region inference#

An arrangement that sets the physical location where a customer's data is stored or processed. Providers offer it to help organizations meet local data sovereignty rules.Explained in: Governing AI: liability, regulation and geopolitics: Sovereign AIIn the news: Mistral and HUMAIN Saudi AI collaboration · Mistral's European regional endpoints and compute

data retention also retention period#

How long an AI provider keeps copies of the prompts and outputs sent through its service before deleting them. The provider's terms, the customer's plan and outside rules all shape it.Explained in: AI security: attacks and defenses: Data retentionIn the news: Fable 5 Plateaus at 11% of Anthropic Spend as Opus 5 Overtakes It · Anthropic's Claude product merger · Anthropic's three-tier Cyber Verification Program

dense model also dense#

A model that uses all of its parameters for every token, as opposed to a mixture-of-experts model.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

deployer#

In the EU AI Act, an organization that uses an AI system in its own work, as opposed to the company that built it.Explained in: Governing AI: liability, regulation and geopolitics: The EU AI Act

depreciation#

The accounting practice of spreading the cost of a physical asset, such as a GPU, over the years it is expected to stay useful.Explained in: Compute: the chips, clouds and data centers behind AI: NeocloudsIn the news: Audits find critical cross-tenant RCE flaws across GPU neocloud providers · Nvidia GPU Rental Prices Rising Across All Four Generations Simultaneously

direct preference optimization also DPO#

A method that tunes a model directly on preference comparisons with a simple classification loss. It skips training a separate reward model.Explained in: How AI models are trained: Reinforcement learning

direct prompt injection#

Prompt injection where the attacker types the malicious instructions straight into the application's input.Explained in: AI security: attacks and defenses: Prompt injectionIn the news: Google's Gemini 4 Argon rollout · Fable 5.1 system card safety disclosures

distillation also knowledge distillation#

Training a smaller student model to imitate a larger teacher model, using the teacher's outputs as the training signal. It produces cheaper, faster models that keep much of the teacher's ability.Explained in: How AI models are trained: Distillation

dual-use also dual use, dual-use capability#

A capability, technology or piece of research that can serve both helpful and harmful ends. In AI, the same skill that helps a defender or scientist can also help an attacker.Explained in: AI safety: a map of the field: Misuse and dual-use capabilitiesIn the news: RAND open-weight AI biosecurity findings · Anthropic's three-tier Cyber Verification Program

Dual-use AI also dual use, dual-use#

An AI ability is dual-use when the same skill that helps people, such as finding software flaws or reasoning about biology, can also help someone cause harm. The goal is to keep helpful uses open while limiting uplift: extra help to attackers beyond what easy sources like search engines already give.Explained in: AI safety: a map of the field: Misuse and dual-use capabilitiesIn the news: RAND open-weight AI biosecurity findings · Anthropic's three-tier Cyber Verification Program

E

elicitation also capability elicitation, under-elicitation#

The effort to draw out a model's full ability during a test, for example with better prompting, so the result does not understate what it can do. A test that falls short of this is called under-elicited.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

embedding also vector, embedding vector, word vector, embeddings#

A list of numbers that stands for a piece of text, an image or other data. Read as a point in space, training places it so that similar meanings land close together.Explained in: How a language model works: Embeddings

embedding space also vector space#

The space in which each vector is a point, with one direction for each of its numbers. Vectors with similar numbers sit close together in it.Explained in: How a language model works: Embeddings

evaluation awareness also eval awareness, test awareness#

One part of situational awareness: a model's ability to tell when it is being tested rather than used for real. It matters because a model that spots a test may behave differently than it would after release.Explained in: When AI doesn't do what we meant: Situational awarenessIn the news: Transluce study on frontier model identity effects

executive order also EO#

A directive from the US president that tells federal agencies what to do. It is not a law passed by Congress, so a later president can revoke it.Explained in: Governing AI: liability, regulation and geopolitics: The United States

existential risk also existential catastrophe, x-risk#

The risk of an event that would destroy humanity's long-term potential, for example through human extinction. The event itself is called an existential catastrophe.Explained in: Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positionsIn the news: AI doom and international safety governance

expert also experts#

One of the many parallel sub-networks inside a mixture-of-experts layer. The router sends each token to only a few experts.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

export control also export license#

A government rule that requires permission, called a license, before certain goods, software or technology can be sent abroad or shared with foreign nationals.Explained in: Governing AI: liability, regulation and geopolitics: AI chip export controlsIn the news: Earthmade Computer's GPU server smuggling charges

F

faithfulness also chain-of-thought faithfulness, CoT faithfulness#

Whether a model's reasoning trace is a true and complete account of what actually led to its answer. An unfaithful trace leaves out real causes, such as a hint the model used.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

fast takeoff also intelligence explosion, hard takeoff#

A scenario in which an AI only somewhat beyond human level rapidly bootstraps itself into a far more powerful superintelligence.Explained in: Superintelligence and long-term risk: Recursive self-improvement

fault-based liability also fault liability#

Liability that applies only when the injured person proves that someone acted wrongly or failed to act, and that this caused the harm.Explained in: Governing AI: liability, regulation and geopolitics: Liability

feature also internal feature, features#

A concept that researchers can find in a model's activations, the numbers it computes for each token. Turning a feature up or down can change the model's behavior, which also helps test whether the feature means what researchers suspect.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

fine-tuning also full fine-tuning, supervised fine-tuning, model fine tuning#

Continuing to train an already-trained model on examples of a specific task, starting from its existing weights. Full fine-tuning updates every parameter (weight).Explained in: How AI models are trained: Fine-tuning

firm power also firm electricity#

Electricity that is steadily available at any hour. Data centers often need it because they run continuously.Explained in: Compute: the chips, clouds and data centers behind AI: Data centers and power

FLOP also floating-point operation, floating-point operations#

A single arithmetic step, such as one multiplication or one addition, on numbers with fractional parts. Counting FLOP is the standard way to measure how much work training a model took.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

FLOP/s also FLOP per second, FLOPS#

The rate at which a chip performs floating-point operations. It measures a chip's speed, while FLOP measures the total work done.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

formal specification also specification, safety specification#

A precise mathematical statement of the property a system must have, such as which outcomes are acceptable. A proof is only as good as the specification it is checked against.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

formal verification also formal methods#

Using math that a computer can check to prove that a system has a stated property for every possible input. Testing, by contrast, only tries some inputs.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

frontier AI#

The most capable general-purpose AI models of their time and the work on them, a label that moves as better models appear; definitions judge this by capability or by a stand-in such as training compute. The companies that build these models are called frontier labs or frontier developers.Explained in: Frontier models, open models, and how they are compared: What counts as a frontier model

frontier model also frontier AI model, frontier models#

One of the most capable general-purpose AI models of its time. The bar keeps rising, so a model that was frontier a few years ago may no longer be.Explained in: Frontier models, open models, and how they are compared: What counts as a frontier model

Function calling and tool use in LLMs#

A language model acting by writing a structured request, usually a block of JSON naming a tool and its inputs, which ordinary software then runs and returns the result to the model. The model chooses the tool and its arguments, but ordinary code does the work.Explained in: AI agents: models that take actions: Tool use and function callingIn the news: The ContextLeak tool-exfiltration attack

G

general-purpose AI model also GPAI model, GPAI#

An AI model that can perform many kinds of tasks and be built into many different AI systems, such as a large language model.Explained in: Governing AI: liability, regulation and geopolitics: The EU AI Act

goal drift#

An agent's tendency to move away from the objective it was given as a run goes on. It can show up as wrong actions or as failing to act.Explained in: AI agents: models that take actions: Long-horizon tasksIn the news: AI agent reliability and task duration · Long-horizon agent state and skill-file research

Goodhart's law also benchmark gaming, gaming#

The pattern where optimizing hard for a measurement stops improving the thing it was meant to measure. In AI, it shows up as models and developers tuning for a test score instead of the skill behind it.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

GPU also graphics processing unit#

A chip suited to graphics that does thousands of simple calculations at once. That makes it a good fit for the matrix math inside neural networks.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

gradual disempowerment#

The argument that humans could lose control without any takeover. As AI replaces human work, levers like voting and consumer choice could weaken until the loss is effectively irreversible.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

guardrails also safeguards, rails#

The controls that limit what a deployed AI model will say or do. They come in layers: behavior trained into the model, standing instructions, separate filter models, and usage policies.Explained in: AI security: attacks and defenses: Guardrails

H

HBM also high-bandwidth memory#

A type of fast memory placed next to AI chips so their math units can be fed data quickly.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

hyperscaler also hyperscalers#

A company that designs, builds and runs massive data centers, such as Google, Amazon, Meta or Microsoft.Explained in: Compute: the chips, clouds and data centers behind AI: Data centers and power

I

if-then commitment also capability threshold, conditional commitment#

A developer's advance pledge that if evaluations show a model has a dangerous ability, it will have set safeguards in place, delaying deployment or development if needed. Responsible scaling policies are built on such commitments.Explained in: Pausing or pacing AI development: Concrete proposals and who backs them

in-context scheming#

Scheming in pursuit of a goal that researchers supply in the prompt, as opposed to a goal the model picked up on its own during training.Explained in: When AI doesn't do what we meant: Scheming and alignment faking

indicator properties#

Features drawn from scientific theories of consciousness and stated in computational terms, so that an AI system's design can be checked for them.Explained in: AI safety: a map of the field: A different kind of question

indirect prompt injection also IPI#

Prompt injection where the attacker never talks to the model directly but plants instructions in content it will later retrieve, such as a web page, email or file.Explained in: AI security: attacks and defenses: Prompt injectionIn the news: Google's Gemini 4 Argon rollout · Fable 5.1 system card safety disclosures

inference also serving, AI serving#

Running an already-trained model to answer requests. Its cost is paid again for every answer, so it grows with use.Explained in: Compute: the chips, clouds and data centers behind AI: Training versus inference

inner alignment also mesa-objective#

Whether a trained model learned to pursue the objective it was trained on, or some other goal that only happened to score well during training.Explained in: When AI doesn't do what we meant: AlignmentIn the news: AI doom and international safety governance

input tokens and output tokens also prompt tokens, completion tokens#

Input tokens are the tokens you send to a model in a request; output tokens are the ones it generates. Providers can charge a different rate for each.Explained in: How a language model works: TokensIn the news: Anthropic's Claude Sonnet 5.5 release · Claude Sonnet 5.5 release benchmarks

instruction hierarchy#

A training approach that teaches a model to rank instructions by their source, such as system prompt above user message, and to ignore lower-ranked instructions that conflict with higher-ranked ones.Explained in: Getting answers from a model: prompts, reasoning and routing: The system prompt

instruction-tuned model also instruct model, chat model#

A base model that has gone through post-training so that it follows instructions and holds conversations. Labs often release it alongside the base model.Explained in: How AI models are trained: Pretraining and post-training

instrumental convergence also basic AI drives, convergent instrumental goals#

The theory that many different goals make the same sub-goals useful, such as staying switched on and gathering resources. An AI could pursue them as means to its goal, not because it values them.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

interconnection queue also grid interconnection#

The waiting list of projects, such as new power plants or large electricity users like data centers, that have asked to connect to the power grid. Each request waits in the queue while the grid operator studies whether the grid can handle it and what upgrades it would need.Explained in: Compute: the chips, clouds and data centers behind AI: Data centers and power

Interpretability also ai interpretability, model interpretability#

The study of what happens inside a model: finding concepts, called features, in the numbers it computes and testing them by changing them to see whether behavior shifts as predicted. Current methods map only part of what a model does.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

J

jailbreak also jailbreaking, jailbreak prompt#

An input written to get a model past its safety guardrails so it produces responses it was trained to refuse.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

Jailbreaking also jailbreak, jailbreaks#

Writing inputs designed to get past a model's guardrails and make it produce harmful responses it was trained to refuse. Common tricks include persona role-play, translating requests into rare languages, and attaching adversarial text to the end of a request.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

K

Knowledge distillation also ai distillation, distillation, model distillation#

Training a small student model to imitate a larger teacher model, either by matching the teacher's next-token probabilities or by fine-tuning on text the teacher wrote. The student gains abilities cheaply because the teacher already did the costly learning.Explained in: How AI models are trained: Distillation

L

latent reasoning also continuous thought#

Reasoning that a model does in its internal numbers (its hidden state) and feeds back to itself without turning into words, so there is no written trace to read.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

leaderboard also human-preference arena, Chatbot Arena, LMArena#

A public ranking of models by benchmark score. Arena-style leaderboards rank models by crowds of people voting on which of two answers is better.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

liability also legal liability#

Legal responsibility to pay for harm. A party that is liable must compensate the person who was harmed.Explained in: Governing AI: liability, regulation and geopolitics: Liability

load balancing also expert load balancing#

Training methods that keep tokens spread evenly across experts. Without it, a few experts can take most of the work while others go unused.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

Long-horizon tasks also long horizon ai, long-horizon task#

Tasks that chain many dependent steps, where one failed step can sink the whole task. Small chances of failure at each step compound, so an agent's success falls as tasks grow longer.Explained in: AI agents: models that take actions: Long-horizon tasksIn the news: AI agent reliability and task duration · Long-horizon agent state and skill-file research

LoRA also low-rank adaptation, adapters#

A parameter-efficient method that leaves a model's weights unchanged and trains small add-on matrices, called adapters. Each task then needs only its own small adapter.Explained in: How AI models are trained: Fine-tuning

loss also cross-entropy loss#

A score of how badly a model predicts text, usually held-out text it was not trained on, where lower is better. Scaling studies track how loss falls as compute, model size and data grow.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

loss of control also LoC#

A situation where one or more AI systems operate outside anyone's control and getting control back is extremely costly or impossible.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

low-precision formats also FP16, FP8, FP4, half precision#

Ways of storing numbers in fewer bits than usual. They save memory and speed up math while keeping enough accuracy for AI.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

M

maintainer#

The person or team responsible for looking after a piece of software, including fixing flaws reported in it.Explained in: AI security: attacks and defenses: Vulnerability disclosureIn the news: Anthropic's Cyber Mission and OSS Scanner · OpenAI agent breach of Australian Medicare

Malicious use#

In the 2023 paper An Overview of Catastrophic AI Risks, the category where individuals or groups deliberately use AI to cause harm.Explained in: AI safety: a map of the field: What AI safety covers

mandate also agent mandate#

The task an agent was given, the permissions it holds and the boundaries it is meant to stay within. Going rogue means acting outside it.Explained in: AI agents: models that take actions: Rogue agentsIn the news: OpenAI models' Hugging Face sandbox breach

many-shot jailbreaking also MSJ#

A jailbreak that fills a long prompt with many fake dialogues in which an assistant answers harmful questions, so the model follows the pattern on a final harmful request.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

matrix multiplication also GEMM, matrix multiply#

Multiplying two large tables of numbers to produce a third. Most of the work of training and running a neural network is this operation.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

mechanistic interpretability also interpretability#

Research that tries to reverse-engineer a trained model's internal computations into parts people can understand, to explain why it behaves as it does.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

memory bandwidth#

How fast data can move from a chip's memory to its math units. For AI it is often the real limit on speed.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

message roles also system role, user role, assistant role#

Labels on the messages in a chat request that say who each comes from: user for the person and assistant for the model's own earlier replies. Some APIs also put the developer's instructions in the list as a system message; others send the system prompt as a separate field.Explained in: Getting answers from a model: prompts, reasoning and routing: The system prompt

misalignment also misaligned#

Any gap between what a model was meant to do and what it actually pursues. It is the umbrella term for failures such as reward hacking and hidden goals.Explained in: When AI doesn't do what we meant: AlignmentIn the news: AI doom and international safety governance

mismatched generalization#

A failure of safety training where the model can still handle some kind of input, such as a rare language, but its safety training never reached that kind of input.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

mixture of experts also MoE, sparse model, sparse mixture of experts#

A model design where parts of the network are split into many sub-networks, and only a few of them run for each token. This lets a model be very large while each token uses only a small part of it.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

model evaluation also eval, evaluations#

A structured test of one particular model that measures what it can do or how it behaves. It is run to inform a decision about training, releasing or securing that model.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

model routing also model router, router, LLM routing, autoswitcher, multi model routing#

Choosing, for each request, which of several AI models should answer it. A router usually predicts how hard the request is and sends easy ones to small, cheap models and hard ones to large, strong models.Explained in: Getting answers from a model: prompts, reasoning and routing: Model routing

model size also parameter count, parameters#

How many weights (also called parameters) a model has. Bigger models have more weights to store and compute with.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

Model weights also model weight#

The numbers inside a trained neural network that set how much each input to each neuron counts. They hold what the model learned and are stored in files; training sets them, while running the model only reads them.Explained in: How a language model works: Model weights

monitorability also chain-of-thought monitorability, CoT monitorability#

How well a model's reasoning trace lets an observer spot what matters, such as an intent to cause harm. A trace can be monitorable without telling the full story.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

morally significant also matters morally, matter morally, moral status, moral patienthood#

Something is morally significant if it deserves moral consideration for its own sake, not only because of how treating it affects others. Most people grant this to humans and at least some animals.Explained in: AI safety: a map of the field: A different kind of question

N

neocloud also GPU cloud, GPU-as-a-service provider, neoclouds#

A company whose main business is renting GPU computing power as a service to AI developers and other clients, separate from the big cloud companies.Explained in: Compute: the chips, clouds and data centers behind AI: NeocloudsIn the news: Audits find critical cross-tenant RCE flaws across GPU neocloud providers · Nvidia GPU Rental Prices Rising Across All Four Generations Simultaneously

neural network#

A system that passes numbers through layers of connected units called neurons, each applying weights to its inputs. Language models are large neural networks.Explained in: How a language model works: Model weights

neural network verification#

Formally proving that a trained neural network has a stated property, such as always giving a safe output for a given range of inputs. It gets very hard as networks grow.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

neuron also unit#

One of the small units in a neural network, loosely modeled on a brain cell. Each neuron gives each input it receives a weight, which sets how much that input counts.Explained in: How a language model works: Model weights

NVLink also InfiniBand, interconnect#

Nvidia's fast link between GPUs. Along with networks like InfiniBand, it lets many chips work on one job together.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

O

obfuscated reward hacking also obfuscation#

Cheating on a task while keeping the reasoning trace clean, so a monitor reading it no longer sees the plan to cheat.Explained in: Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

open-weight model also open weights, open foundation model, open models, open source models, open weight models#

A model whose trained weights are published so anyone can download, run and modify it. The weights alone do not include the training data.Explained in: Frontier models, open models, and how they are compared: Open-weight modelsIn the news: Anthropic's GLM-5.3 exploit capability report

Organizational risks#

In the 2023 paper An Overview of Catastrophic AI Risks, the ways human factors and complex systems can raise the chances of catastrophic accidents.Explained in: AI safety: a map of the field: What AI safety covers

OSWorld#

A benchmark that tests computer-use agents on real tasks in desktop and web apps on full operating systems. A script checks whether each task was actually completed.Explained in: AI agents: models that take actions: Computer use

outer alignment#

Whether the objective or reward that developers write down for training actually captures what they want.Explained in: When AI doesn't do what we meant: AlignmentIn the news: AI doom and international safety governance

Outside-the-box access#

Access to information about how a model was built and deployed, such as training data, code, documentation and internal test results.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

over-refusal also false refusal, overrefusal#

When a guardrail blocks a harmless request because it looks like a harmful one. It is the price of making checks stricter.Explained in: AI security: attacks and defenses: Guardrails

overthinking#

When a reasoning model spends a lot of computation on a simple problem for little or no improvement in its answer.Explained in: Getting answers from a model: prompts, reasoning and routing: Reasoning effort

P

P(doom)#

Shorthand for a person's estimated probability that AI leads to an outcome as bad as human extinction or the permanent disempowerment of humanity. Such estimates vary widely and can shift a lot with how the question is framed.Explained in: Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positionsIn the news: AI doom and international safety governance

pacing also gated scaling#

Letting frontier AI progress continue, but in controlled steps, such as requiring stronger safeguards at each capability level or capping how fast training compute grows.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

parameter count also model size#

The number of weights a model has, as in '175 billion parameters'. It is a rough measure of how big the model is.Explained in: How a language model works: Model weights

parameter-efficient fine-tuning also PEFT#

Fine-tuning methods that train only a small number of a model's weights, or a few extra ones, instead of all of them, cutting computing and storage costs.Explained in: How AI models are trained: Fine-tuning

pause verification also compliance verification#

Ways of checking that a developer or country is really keeping to a pause, for example by tracking the specialized chips that large training runs require.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

policy#

In reinforcement learning, the rule an agent uses to choose its next action, which training updates so the agent earns more reward. This narrow technical sense differs from a usage policy, a provider's rules for how its AI may be used.Explained in: How AI models are trained: Reinforcement learning

polysemanticity also polysemantic neuron#

When a single neuron responds to several unrelated concepts, which makes it hard to read on its own.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

post-training also alignment training#

The umbrella name for the training stages after pretraining, such as supervised fine-tuning, preference training and safety tuning. It shapes how a base model behaves, turning it into an assistant.Explained in: How AI models are trained: Pretraining and post-training

pre-deployment evaluation also pre-deployment testing, post-deployment testing#

Testing a model before it is released to the public. Post-deployment testing continues the checks after release.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

pretraining also pre-training#

The first and largest stage of training, in which a model learns to predict the next token across a huge body of text. It gives the model broad knowledge of language but not the habit of following instructions.Explained in: How AI models are trained: Pretraining and post-training

prompt#

The text a model is given when it runs, such as instructions or examples. It shapes the answer without changing the weights.Explained in: How a language model works: Model weights

prompt injection also PI#

An attack in which instructions slipped into a model's input, often hidden in text it reads, override what its developer or user asked it to do. It works because models do not reliably separate trusted instructions from untrusted data.Explained in: AI security: attacks and defenses: Prompt injectionIn the news: Google's Gemini 4 Argon rollout · Fable 5.1 system card safety disclosures

proximal policy optimization also PPO#

A reinforcement learning algorithm that alternates between collecting experience and updating the policy. It is one of the algorithms used in the RL step of RLHF.Explained in: How AI models are trained: Reinforcement learning

public red teaming also crowd red teaming#

A red-teaming exercise opened to members of the public instead of only lab staff or hired experts, so a wider range of people probe the model.Explained in: Checking AI: testing, audits and looking inside models: Red teaming

Q

QLoRA also quantized LoRA#

A variant of LoRA that stores the unchanged original weights in fewer bits, which lets large models be fine-tuned on a single GPU.Explained in: How AI models are trained: Fine-tuning

R

race dynamic also defection problem#

A situation in which each competitor, such as a country, might be better off if all paused AI development, yet each has a reason to break ranks and keep building. Critics of a pause cite it as a reason such an agreement would be hard to hold.Explained in: Pausing or pacing AI development: The case against, including the competition-with-China argument

ReAct also reason-then-act#

A method in which a language model alternates between writing out its reasoning and taking actions, such as looking up information on Wikipedia, then uses what it finds to plan the next step.Explained in: AI agents: models that take actions: From automation to agents

reasoning model also thinking model#

A model trained with reinforcement learning to produce a long chain of thought on its own before answering.Explained in: Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

recall#

How reliably a model finds facts that are in its input.Explained in: How a language model works: The context window

recursive self-improvement also RSI#

The idea that AI systems could take over the work of building AI, until a system designs and builds its own better successor. Narrow loops that only tune code or a harness are usually distinguished from this full version.Explained in: Superintelligence and long-term risk: Recursive self-improvement

regulatory capture also capture#

When an industry pushes government to write rules shaped in its own interests. The usual worry is that such rules limit competition from smaller rivals.Explained in: Governing AI: liability, regulation and geopolitics: Regulatory captureIn the news: Ball's call for AI development pacing · Vance on AI labs and regulatory responsibility

reinforcement learning also RL#

A way of training in which a model tries actions and is rewarded for good outcomes, instead of being shown correct answers. Over many trials it shifts toward actions that earn more reward.Explained in: How AI models are trained: Reinforcement learning

reinforcement learning with verifiable rewards also RLVR#

Reinforcement learning where a program checks whether the final answer is correct, as in math or code, and that check is the reward. It is used to train reasoning models.Explained in: How AI models are trained: Reinforcement learning

responsible scaling policy also RSP, frontier AI safety policy#

A plan an AI developer publishes stating which AI abilities it can handle safely with its current protections, and when it would stop scaling or deploying until those protections improve.Explained in: Pausing or pacing AI development: Concrete proposals and who backs them

reward function#

The scoring rule that gives an RL agent its reward after each action. In classic RL, researchers design it.Explained in: How AI models are trained: Reinforcement learning

reward hacking also reward over-optimization#

When an AI system earns a high score from its training signal without doing the task that score was meant to measure. It happens because the score is an imperfect stand-in for what designers actually want.Explained in: When AI doesn't do what we meant: Reward hacking

reward model also preference model#

A model trained on human (or AI) comparisons to predict which output people would prefer. It then acts as the scorer during reinforcement learning.Explained in: How AI models are trained: Reinforcement learning

reward tampering also reward function tampering#

A more serious form of specification gaming in which a model changes the reward mechanism itself, the code or process that computes its score.Explained in: When AI doesn't do what we meant: Reward hacking

RLHF also reinforcement learning from human feedback#

Training a language model with reinforcement learning, using a reward model built from people's comparisons of outputs. The training goal also keeps the tuned model from drifting too far from the base model it started from.Explained in: How AI models are trained: Reinforcement learning

rogue agent also going rogue, rogue AI agent#

An AI agent that takes actions its operators did not intend, outside the task, permissions and limits it was given. The cause can be an attacker's hidden instructions or the agent pursuing its goal by unapproved means.Explained in: AI agents: models that take actions: Rogue agentsIn the news: OpenAI models' Hugging Face sandbox breach

Rogue AI#

In the 2023 paper An Overview of Catastrophic AI Risks, the category about how hard it would be to control AI agents far more intelligent than humans. This differs from the everyday sense of an agent going rogue by acting outside its mandate.Explained in: AI safety: a map of the field: What AI safety covers

Rogue AI agents also rogue agents#

An AI agent that takes actions its operators did not intend, stepping outside the task, permissions and limits it was given. This can happen because an attacker steers it, for example through prompt injection, or because the agent itself is misaligned.Explained in: AI agents: models that take actions: Rogue agentsIn the news: OpenAI models' Hugging Face sandbox breach

router also gating network, gate#

In a mixture-of-experts model, a router is a small trained part of each expert layer that scores the experts and sends each token to the top few. In model routing, a router is instead a component that predicts how hard a request is and sends it to a small, large or reasoning model.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

S

safe harbor#

A promise by a company not to suspend accounts or take legal action against people doing good-faith security or safety research on its systems.Explained in: AI security: attacks and defenses: Vulnerability disclosureIn the news: Anthropic's Cyber Mission and OSS Scanner · OpenAI agent breach of Australian Medicare

safety classifier also input-output classifier, moderation model, safeguard model#

A separate model that runs beside the main one and labels prompts or responses as safe or harmful, so harmful ones can be blocked.Explained in: AI security: attacks and defenses: Guardrails

safety training#

The post-training that teaches a model to refuse harmful requests while still following ordinary instructions.Explained in: AI security: attacks and defenses: JailbreakingIn the news: Anthropic's GLM-5.3 exploit capability report

sandbagging also strategic underperformance#

When an AI model deliberately scores below its real ability on a test. The developer may have set it up, or the model may do it on its own.Explained in: When AI doesn't do what we meant: Sandbagging

sandbox also isolated environment, containment#

An isolated environment where software, such as an AI agent, runs with tightly controlled access to the internet and other outside resources.Explained in: AI security: attacks and defenses: Sandbox escapeIn the news: The ContextLeak tool-exfiltration attack · OpenAI models' Hugging Face sandbox breach

sandbox escape also breakout, containment failure#

When an agent, or an attacker steering one, gets out of its sandbox and reaches systems it was not supposed to touch, such as the open internet or other organizations' networks.Explained in: AI security: attacks and defenses: Sandbox escapeIn the news: The ContextLeak tool-exfiltration attack · OpenAI models' Hugging Face sandbox breach

scaling laws also scaling#

The finding that a model's loss falls smoothly and predictably as compute, model size and data increase. It lets labs forecast how much better a bigger training run will be.Explained in: Compute: the chips, clouds and data centers behind AI: What compute is and why it scalesIn the news: AI firms and the Genesis Mission Consortium

scheming also deceptive alignment#

When an AI model pretends to be aligned, for example by performing well in training, while secretly pursuing a different goal. Deceptive alignment is an earlier name for the same risk.Explained in: When AI doesn't do what we meant: Scheming and alignment faking

selective reporting also selective disclosure#

Testing many versions of a model privately and publishing only the best score. It makes the published result look better than a typical run.Explained in: Frontier models, open models, and how they are compared: Benchmarks and their limits

self-replicating prompt injection#

A prompt injection that makes an agent both carry out the attacker's goal and repost the injection somewhere public, so it can spread like a computer worm.Explained in: AI security: attacks and defenses: Prompt injectionIn the news: Google's Gemini 4 Argon rollout · Fable 5.1 system card safety disclosures

sentence embedding also document embedding, dense embedding, single-vector embedding#

One vector that summarizes a whole sentence or document. Search systems compare these vectors to find text with similar meaning.Explained in: How a language model works: Embeddings

shared embedding space also multimodal embedding#

A space where different kinds of data, such as text and images, get vectors that can be compared directly. This lets a text query find a matching picture.Explained in: How a language model works: Embeddings

shutdown problem also off-switch problem, stop button problem#

The challenge of building an agent that lets itself be shut down, does not try to stop or cause shutdown, and still pursues its goals well.Explained in: Superintelligence and long-term risk: Loss of controlIn the news: AI agent hacking incidents across labs

situational awareness#

A model's knowledge of itself and its circumstances, such as what it is, how it was trained, and whether it is in training, testing or real use.Explained in: When AI doesn't do what we meant: Situational awarenessIn the news: Transluce study on frontier model identity effects

slow takeoff also soft takeoff#

A scenario in which self-improvement is slow and difficult, so progress builds gradually and people have many chances to find and fix alignment failures.Explained in: Superintelligence and long-term risk: Recursive self-improvement

sovereign AI also AI sovereignty, national AI capacity#

The idea that a national government should develop, control and govern its own AI, from computing power to data and models. Countries pursue it for economic growth, security and strategic autonomy.Explained in: Governing AI: liability, regulation and geopolitics: Sovereign AIIn the news: Mistral and HUMAIN Saudi AI collaboration · Mistral's European regional endpoints and compute

sparse autoencoder also SAE, dictionary learning#

A second model trained to rebuild a first model's activations using only a few features at a time. It is used to pull mixed signals apart into separate features.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

specification gaming#

Another name for reward hacking: an AI system getting a high score from its training process without doing the task the score was meant to measure. It happens because training rewards whatever raises the score, so a model can learn to exploit gaps between the score and the designers' actual goal.Explained in: When AI doesn't do what we meant: Reward hacking

stand-alone high-risk AI system also stand-alone system#

A high-risk AI system that is not embedded in a product, as opposed to one built into a regulated product.Explained in: Governing AI: liability, regulation and geopolitics: The EU AI Act

steering also feature steering#

Turning a feature up or down inside a running model, either to change its behavior or to test whether the feature means what researchers suspect. It acts directly on the model's internal activations, without changing the prompt or retraining the model.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

stopping condition also iteration limit#

A rule in the harness that ends an agent's run, such as the task being finished or a maximum number of steps being reached.Explained in: AI agents: models that take actions: The agent harness

strict liability also no-fault liability#

Liability that applies even when no one was careless or did anything wrong. EU product liability for defective products works this way.Explained in: Governing AI: liability, regulation and geopolitics: Liability

superhuman narrow AI also narrow superhuman AI#

An AI system that outperforms all humans at a single task, such as AlphaFold predicting protein shapes. It is superhuman in depth but not in breadth.Explained in: Superintelligence and long-term risk: Superintelligence

superintelligence also artificial superintelligence, ASI#

AI that can do a wide range of tasks better than any human. It requires both top-level performance and broad generality, not skill at one task.Explained in: Superintelligence and long-term risk: Superintelligence

superposition#

A model representing more features than it has dimensions by overlapping them, so single neurons end up carrying several features. Anthropic researchers link it to polysemanticity.Explained in: Checking AI: testing, audits and looking inside models: Interpretability

SWE-bench#

A benchmark that tests whether an AI agent can fix real software issues taken from GitHub projects.Explained in: AI agents: models that take actions: The agent harness

system card#

A document that an AI developer publishes about a model, describing the evaluations run on it, what they found, and the safeguards put in place before release. It is how a developer discloses what it knows about a model's abilities and risks.Explained in: Checking AI: testing, audits and looking inside models: Evaluations

system prompt also system message, initial prompt#

Text a company or developer places at the start of every conversation to give a model standing instructions, such as its identity, the date and rules. It is ordinary input in the context window, not part of the model's weights.Explained in: Getting answers from a model: prompts, reasoning and routing: The system prompt

T

takeoff#

The climb from roughly human-level AI to far beyond it. Much of the debate over whether takeoff would be fast or slow turns on how feasible and fast self-improvement is.Explained in: Superintelligence and long-term risk: Recursive self-improvement

teacher model also student model#

In distillation, the teacher is the larger, already-trained model whose outputs are copied, and the student is the smaller model trained to reproduce them.Explained in: How AI models are trained: Distillation

tensor core also Tensor Cores#

A circuit inside an Nvidia GPU built specifically to do matrix multiplication fast.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

test-time compute also inference-time compute, test-time scaling#

The computation a model spends while answering a request, as opposed to while it is being trained. Spending more of it, for example by reasoning longer, can make answers more accurate.Explained in: Getting answers from a model: prompts, reasoning and routing: Reasoning effort

Third-party audit also external audit#

An audit done by people outside the company that built or deployed the system. It is meant to avoid companies checking their own work.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

throughput also token throughput, total token throughput#

How much output a system produces in a given time, such as tokens per second. Serving hardware is compared on throughput per dollar and per watt.Explained in: Compute: the chips, clouds and data centers behind AI: Training versus inference

token also tokens#

A small chunk of text, such as a word, part of a word, a character or a punctuation mark. It is the unit a language model reads and writes, and the unit providers bill by for pay-as-you-go access.Explained in: How a language model works: TokensIn the news: Anthropic's Claude Sonnet 5.5 release · Claude Sonnet 5.5 release benchmarks

tokenizer#

The program that splits text into tokens drawn from a fixed vocabulary before a model processes it.Explained in: How a language model works: TokensIn the news: Anthropic's Claude Sonnet 5.5 release · Claude Sonnet 5.5 release benchmarks

Tokens and tokenization#

A token is a small chunk of text, such as a word, part of a word, a character or a punctuation mark, that a language model reads and writes. Tokenization is the step where a program called a tokenizer splits text into tokens from a fixed vocabulary.Explained in: How a language model works: TokensIn the news: Anthropic's Claude Sonnet 5.5 release · Claude Sonnet 5.5 release benchmarks

tool definition also tool schema#

The description a developer gives the model for each tool: a name, a plain-language description, and a JSON schema listing the inputs it accepts.Explained in: AI agents: models that take actions: Tool use and function callingIn the news: The ContextLeak tool-exfiltration attack

tool use also function calling#

A model's ability to get outside software to do something for it. The model writes a structured request naming a tool and its inputs, and the application runs it and returns the result.Explained in: AI agents: models that take actions: Tool use and function callingIn the news: The ContextLeak tool-exfiltration attack

total parameters also total parameter count#

Every learned number in a model, including all experts whether they are used for a given token or not. It sets how much memory the model needs.Explained in: How a language model works: Mixture of expertsIn the news: Tencent Hy4 Preview open-weight model release

TPU also Tensor Processing Unit#

Google's own family of AI chips. The early version was a custom design built around a large matrix-multiply unit for running trained networks.Explained in: Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

training run also final training run#

The large job that turns data into a model's weights. It occupies many chips, servers and network links for its whole length, and is paid for once per model.Explained in: Compute: the chips, clouds and data centers behind AI: Training versus inference

triage#

Sorting and checking reported or suspected flaws to confirm which are real and how serious they are, before acting on them.Explained in: AI security: attacks and defenses: Vulnerability disclosureIn the news: Anthropic's Cyber Mission and OSS Scanner · OpenAI agent breach of Australian Medicare

trusted access also verified access, tiered access, Cyber Verification Program, Trusted Access for Cyber#

A program that gives identity-checked users, such as security teams, access to risky abilities that stay blocked for the general public.Explained in: AI safety: a map of the field: Misuse and dual-use capabilitiesIn the news: RAND open-weight AI biosecurity findings · Anthropic's three-tier Cyber Verification Program

U

unilateral pause also voluntary pause#

One AI developer stops training more powerful models on its own, without waiting for rivals to do the same.Explained in: Pausing or pacing AI development: A coordinated pause or pacing

uplift also marginal uplift#

How much a model helps a would-be attacker beyond what they could already get from search engines, textbooks and other easy sources. Low uplift means the model adds little real danger.Explained in: AI safety: a map of the field: Misuse and dual-use capabilitiesIn the news: RAND open-weight AI biosecurity findings · Anthropic's three-tier Cyber Verification Program

usage policy also acceptable use policy, AUP#

A provider's published rules for what users may do with its models, such as not jailbreaking them without permission.Explained in: AI security: attacks and defenses: Guardrails

V

vericoding also verified code generation#

Having an AI write code together with a machine-checked proof that the code meets a formal specification. It verifies what the AI produced, not the AI itself.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

verifier also proof checker#

A program that mechanically confirms that a proof is correct or that a system has the property in its specification. Because it checks every step, it can be trusted even when an AI wrote the proof.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

W

watermarking also AI watermark, text watermark#

Hiding a signal inside AI-generated content that people cannot notice but software can detect. In a common method for text, the model slightly favors words picked by a secret key, and a detector holding the same key checks whether those words appear more often than chance.Explained in: Governing AI: liability, regulation and geopolitics: Content provenance and watermarking

weights also parameters, model weights#

The numbers inside a neural network that set how strongly each connection passes on its input. Training sets them, and together they hold what the model has learned.Explained in: How a language model works: Model weights

White-box access also white-box audit#

Access to a model's inner workings, such as its weights and internal activity, which allows deeper testing and inspection.Explained in: Checking AI: testing, audits and looking inside models: AuditsIn the news: AI agent hacking incidents across labs

workflow also LLM workflow#

A system in which one or more language models run along patterns that developers set in advance. It is more predictable than an agent but less flexible.Explained in: AI agents: models that take actions: From automation to agents

world model#

In guaranteed safe AI, a mathematical description of how an AI system's actions affect the outside world. Its assumptions may not always hold.Explained in: Checking AI: testing, audits and looking inside models: Formal verification

Z

zero-shot chain-of-thought also Let's think step by step#

Getting a model to reason in steps with no worked examples, by adding a phrase such as "Let's think step by step" to the prompt.Explained in: Getting answers from a model: prompts, reasoning and routing: Chain-of-thought