The ContextQA glossary of AI testing terms

Acceptance testing verifies that a system meets business requirements and is ready for release, usually performed by end users or product owners. It answers 'did we build the right thing?' rather than 'did we build it right?'

Accessibility testing checks that an application is usable by people with disabilities, validating against standards such as WCAG for keyboard navigation, screen-reader support, color contrast, and ARIA roles.

An action step is a test case step that performs an interaction with the UI, such as clicking a button, typing in a field, selecting a dropdown, scrolling, hovering, or navigating to a URL. It contrasts with a verification step, which checks state rather than acting.

Agentic AI is software built around autonomous AI agents that plan and act toward a goal rather than following fixed scripts. In testing, an agentic system decides how to reach an outcome and adapts when the application changes.

The AI agent is the autonomous execution engine at the heart of ContextQA. It interprets natural-language test steps, locates UI elements with visual AI and DOM analysis, performs browser actions via Playwright, and self-heals broken locators. Multiple specialized agents collaborate across a nine-stage pipeline.

AI agent testing validates AI agents and agentic workflows for correctness, hallucinations, prompt regressions, and non-deterministic outputs, where the same input can produce different results across runs.

The AI Assistant is ContextQA's in-app copilot that helps you navigate the platform, generate test cases from natural-language descriptions, and get contextual help during test creation and execution.

The AI Data Analyst is a ContextQA capability that interprets test results and execution data to surface trends, risks, and coverage insights in plain English, turning raw run data into decisions.

AI evaluation (an 'eval') is the practice of scoring an AI system's outputs against expected behavior using datasets and metrics, so teams can catch regressions in quality, accuracy, and safety before shipping.

AI reasoning is the per-step log of the AI agent's decision-making during a test execution, showing why each action was chosen. It is accessible from the AI Insights tab of an execution report.

AI test generation automatically creates executable test cases from Jira tickets, Figma designs, Swagger or OpenAPI specs, videos, code diffs, or a plain-English description, so authoring no longer means writing scripts.

AI verification is a step type that uses visual AI to validate a condition described in plain English, analyzing the page screenshot instead of checking a specific DOM attribute. It is useful for dynamic content and visual states that are hard to express as DOM queries.

The analytics dashboard presents execution trends, pass rates, coverage, flaky-test signals, and AI insights so QA teams, developers, and managers can understand test health at a glance.

API chaining connects multiple API calls in one test by extracting fields from a response into variables and passing them to later requests, enabling realistic flows such as authenticate, then call a protected endpoint.

API testing validates application programming interfaces directly at the protocol level, checking status codes, response payloads, schemas, and headers without driving the UI. ContextQA supports REST request configuration, response validation, and chaining, including hybrid API-plus-UI flows.

An assertion is a check that a condition is true at a point in a test; if it fails, the test fails. In ContextQA, assertions are written as plain-English checks and resolved by AI verification.

Auto-healing is another name for self-healing: the automatic repair of a broken test step when the original locator no longer matches an element.

Autonomous testing uses AI agents to plan, create, run, and maintain tests with minimal human scripting, adapting to application changes on its own instead of executing fixed, pre-written steps.

A test automation framework is the set of conventions, libraries, and tools that structure how automated tests are written, run, and maintained. AI-native platforms replace much of this scaffolding with natural-language steps and self-healing.

Behavior-driven development (BDD) is a collaborative approach that describes desired behavior in plain, structured language (often Given-When-Then) shared by business and engineering, which then guides tests. Natural-language testing extends the same readability to execution.

The browser automation layer is the technology through which the AI agent controls browser instances. ContextQA uses Playwright internally to drive Chromium, Firefox, and WebKit; users interact through natural language and never write Playwright code unless they export a test.

A bug, or defect, is a flaw in software that causes it to behave incorrectly or differ from its requirements. Testing exists to find bugs before users do; ContextQA can file them automatically with reproduction evidence.

CI/CD integration connects ContextQA test-plan executions to a continuous integration and delivery pipeline, so tests run automatically on pull requests, merges, or any pipeline event. GitHub Actions, GitLab CI, Jenkins, and CircleCI are supported.

Code coverage measures the percentage of source code exercised by a test suite, such as lines, branches, or functions. It is distinct from test coverage, which measures how much of the requirements or functionality is tested.

Code export converts a ContextQA test into runnable Playwright code, so teams can inspect the underlying automation, run it in their own framework, or migrate between systems without rewriting from scratch.

Codeless (no-code) testing lets people create automated tests without writing programming code, using recorders, visual editors, or natural-language steps. It widens authoring beyond engineers to the whole QA team.

A confidence score is a number from 0.0 to 1.0 the AI produces when judging how well a candidate UI element matches the intended target. Self-healing auto-applies matches at or above 0.90, flags 0.70-0.89, and leaves anything lower for manual review.

The context graph is the accumulated, reusable application knowledge ContextQA builds over time from UI elements, knowledge bases, custom agents, and past runs, so tests get smarter and more stable with every execution.

Continuous integration and continuous delivery (CI/CD) is the practice of merging code frequently and shipping it through an automated pipeline of builds, tests, and deployments. Automated testing is the quality gate that makes fast delivery safe.

Continuous testing runs automated tests throughout the delivery pipeline, not just at the end, giving teams constant feedback on release readiness. It is essential to shipping quickly without sacrificing quality.

Cross-browser testing verifies that an application works consistently across browsers and rendering engines such as Chromium, Firefox, and WebKit. ContextQA runs the same test across browsers without separate scripts.

A custom agent is a user-defined AI persona with a custom system prompt that specializes the agent's behavior for a specific application or scenario, such as a checkout agent that always uses a test card, or a Salesforce or accessibility agent.

A custom code step runs custom JavaScript as part of a test, for complex logic, calculations, or DOM interactions that go beyond natural-language steps.

Data-driven testing runs one test case many times, each time using a different row from a test data profile, so a single test covers many input combinations. Each row produces its own pass/fail result and evidence.

A database step runs a SQL query during a test, stores the result in a variable, and asserts expected values, letting tests verify data the way users never see it directly.

Database testing validates the data layer, checking that reads, writes, integrity constraints, and migrations behave correctly. It is often combined with UI and API checks for end-to-end confidence.

A defect ticket is a bug report created in an external tracker such as Jira or Azure DevOps directly from a ContextQA failure, populated automatically with the failing step, error, screenshot, and a link to the execution report.

DevOps is a culture and set of practices that unite software development and IT operations to deliver changes faster and more reliably. Continuous, automated testing is what lets DevOps teams move fast without breaking production.

Dynamic application security testing (DAST) probes a running application by simulating attack patterns, rather than analyzing source code at rest. ContextQA integrates DAST scanning with passive, standard, and full scan profiles.

A real device farm is a pool of physical phones and tablets that tests run on, rather than emulators. ContextQA executes mobile tests on real iOS and Android devices, so results reflect real-world behavior.

The AI execution pipeline is ContextQA's nine-stage process that turns a natural-language test into actions: each stage, from intent parsing to verification and self-healing, is handled by a specialized sub-agent.

An element is a UI component on a web or mobile page that the AI agent can interact with or assert against, such as a button, input, dropdown, link, or table row. ContextQA identifies elements using both DOM analysis and visual AI.

The element repository is the workspace-level catalog of UI elements discovered during executions, reused across tests so the agent recognizes components it has seen before and builds a map of the application over time.

End-to-end (E2E) testing validates a complete user journey through the real application, from the first click to the final outcome, exercising the UI, APIs, and data together the way a user would.

An environment is a named configuration with a base URL and key-value parameters for a deployment target such as staging or production. You pick which environment to run against, and steps reference its parameters as ${ENV.KEY}.

An evidence package is the complete set of artifacts produced by a single execution: per-step screenshots, full session video, network HAR log, console log, Playwright trace, and AI reasoning. Every run produces one, whether it passed or failed.

An execution is one complete run of a test case, suite, or plan, identified by an execution ID and producing a status (passed, failed, error, or running) with its own evidence package.

Exploratory testing is unscripted, simultaneous learning, test design, and execution, where a tester investigates the application to find issues that scripted tests miss. AI automation frees testers' time to do more of it.

Fix and apply is a workflow that, given a failed execution and a repository, diagnoses the failure, proposes a code-level fix, and applies it to the test or application code, used when an AI agent both detects and resolves failures.

A flaky test produces inconsistent results, passing on some runs and failing on others, without any change to the application, often due to race conditions, timing, or network latency. ContextQA's AI separates true flakiness from real bugs.

Functional testing checks that features behave according to requirements, validating what the system does. It contrasts with non-functional testing, which validates qualities such as speed, security, and accessibility.

A generation source is the artifact AI test generation builds tests from. ContextQA supports ten: natural language, Jira or ADO tickets, Figma designs, Excel or CSV, Swagger or OpenAPI specs, video recordings, requirements documents, code diffs, n8n workflows, and edge-case specs.

Generative AI is a class of models that create new content, such as text, code, or images, from learned patterns. In QA it powers test generation and analysis, and it is also a system under test in its own right.

A hallucination is when an AI model produces output that is fluent and confident but factually wrong or unsupported. Detecting hallucinations is a core part of testing AI agents and generative features.

A HAR (HTTP Archive) is a JSON log of every HTTP request and response the browser made during an execution, capturing URLs, methods, status codes, headers, bodies, and timing. ContextQA captures it automatically for debugging.

A headless browser is a real browser that runs without a visible window, used to execute tests faster in CI environments. The same engine renders pages identically to its visible counterpart.

Integration testing verifies that separately developed modules or services work correctly together, catching defects at the seams that unit tests miss because they check components in isolation.

The intent parser is stage three of ContextQA's AI execution pipeline. It converts each natural-language step into a structured action specification: action type, target element, data value, and confidence score, then passes it to the action executor.

A knowledge base is a set of plain-English instructions the AI agent reads before each execution, teaching it app-specific patterns such as dismissing consent banners, which test accounts to use, or how to navigate a multi-step wizard.

A large language model (LLM) is an AI model trained on vast text to understand and generate natural language. LLMs power ContextQA's interpretation of plain-English test steps, and are themselves a class of system that must be tested.

Load testing measures how a system behaves under expected and peak concurrent usage, revealing slowdowns and breaking points before users hit them. It is a form of performance testing.

A locator (or selector) is the strategy used to identify a specific UI element in the DOM or a screenshot. ContextQA manages locators automatically with AI, and self-healing finds a new match when one breaks, so users never maintain selectors by hand.

LockDataGuard is a feature-access gate that restricts certain ContextQA features by subscription plan, such as execution history, knowledge bases, or custom agents. A lock icon means the limit is at the plan level.

Manual testing is checking software by hand, without automation, by following steps and observing results. It remains valuable for exploratory and usability work, while repetitive checks are best automated.

The Model Context Protocol (MCP) is an open standard that lets AI models call external tools in a structured, typed way, defining how tool manifests are declared and how calls are made. ContextQA implements MCP so AI clients can drive the platform.

The ContextQA MCP server implements the Model Context Protocol and exposes the platform's tools to MCP-compatible AI clients such as Claude, Cursor, and VS Code, so AI agents can create, run, and analyze tests through natural conversation.

Mean time to detect (MTTD) is the average time between a defect being introduced and being discovered. Continuous, automated testing lowers MTTD by catching regressions within minutes of a change.

The migration platform is the set of MCP tools that analyze an existing test repository and move suites between ContextQA and other frameworks, following an analyze, review, then migrate flow.

Mobile testing validates iOS and Android apps, including gestures, permissions, and device-specific behavior. ContextQA runs mobile tests on a real device farm rather than emulators.

Mock API testing captures real API responses during a run and replays them as mocks on later runs, making tests faster and more stable and letting you test against services that are unavailable or expensive to call.

An n8n workflow is an automation built in the n8n low-code platform. ContextQA can generate test cases from an n8n workflow's JSON, mapping each node to corresponding verification steps.

An NLP step is a test step written in plain language, such as 'Click the Submit button' or 'Verify the success message is visible', that the AI interprets and executes. It needs no CSS selectors, XPath, or automation code.

Non-deterministic testing addresses systems, especially AI agents, that can return different valid outputs for the same input. Instead of exact-match assertions, it evaluates whether outputs meet criteria across many runs.

Non-functional testing validates how well a system works rather than what it does, covering performance, security, accessibility, reliability, and usability, often mapped to quality models such as ISO/IEC 25010.

The page object model (POM) is a design pattern that wraps each page's elements and actions in a reusable class, reducing duplication in code-based automation. AI-native testing achieves the same reuse through step groups and an element repository.

Parallel execution runs multiple test cases simultaneously to cut total plan duration and surface failures faster. ContextQA schedules runs across browsers and devices concurrently.

Performance testing measures responsiveness, throughput, and stability under a given workload, including load and stress conditions, to ensure an application is fast and reliable for real users.

A Playwright trace is a binary archive with DOM snapshots before and after each action, network requests, console output, and synchronized screenshots. ContextQA includes one in every evidence package for deep debugging, viewable at trace.playwright.dev.

A pre-requisite is a dependency between test cases: when test B lists test A as a pre-requisite, B runs only if A passed in the same run. It models flows like 'only run checkout tests if login passed'.

A profile variable references a column in a test data profile, written as ${VAR.COLUMN}. At run time it is replaced with the value from the current data row, supplying different data on each data-driven run.

Prompt engineering is the practice of crafting the instructions given to an AI model to get reliable, accurate outputs. In testing, prompts drive both how tests are generated and how AI agents are evaluated.

A prompt regression is a drop in an AI system's output quality caused by a change to a prompt, model, or context. Evaluations and AI agent testing catch these before they reach production.

Quality assurance (QA) is the discipline of ensuring software meets quality standards through process, prevention, and testing across the development lifecycle, not just checking the finished product.

Quality engineering (QE) embeds quality into how software is built, with engineers owning testing, automation, and reliability rather than treating QA as a separate, final phase. AI tooling makes this scale.

Regression testing re-runs existing tests after a change to confirm that previously working features still work. It is the highest-value use of automation, since suites grow and must run on every release.

A coverage gap is a requirement, flow, or edge case that has no linked test. ContextQA's AI identifies gaps with similarity scoring and suggests tests to close them, so nothing important ships untested.

Requirements management in ContextQA lets you upload requirement documents and have AI generate test cases, analyze coverage gaps, and track change history from a single requirements library.

A result ID uniquely identifies the result object produced by an execution, distinct from the execution ID that identifies the run. It is the key used to fetch telemetry such as step details, network logs, and AI reasoning.

Retrieval-augmented generation (RAG) is a technique where an AI model retrieves relevant documents and uses them to ground its answers, reducing hallucinations. RAG-based features need testing for retrieval quality and answer accuracy.

Risk-based testing prioritizes test effort by the likelihood and impact of failure, focusing on the areas where a defect would hurt most. AI coverage analysis helps target the riskiest changes first.

Root cause analysis is an AI-generated explanation of why an execution failed, built from the evidence package. It returns a plain-English summary, the affected step, the evidence used, a suggested fix, and a classification (test bug, app bug, flaky, or environment).

Sanity testing is a quick, narrow check that a specific function or fix works after a change, before committing to fuller testing. It is a focused subset of regression testing.

Scheduling runs test plans automatically on a recurring basis using cron expressions or frequency presets, with results delivered to Slack or email, so regression suites run unattended overnight or on every release.

An SDET (Software Development Engineer in Test) is an engineer who builds the tools, frameworks, and automation that make testing scalable, blending software development with quality engineering.

Security testing finds vulnerabilities that could be exploited, covering authentication, authorization, injection, and data exposure. ContextQA includes DAST scanning to probe running applications safely.

Self-healing is the automatic repair of a broken test step when the target element cannot be found with its original locator. ContextQA's AI searches the live DOM and screenshot for a semantically equivalent element and, when confident, updates the step and continues, removing most test maintenance.

A service account is a dedicated ContextQA login created for CI/CD automation and MCP connections, separate from any individual, so access can be revoked independently and audit logs stay clean.

Shift-left testing moves quality activities earlier in development, so defects are caught when they are cheapest to fix. Generating tests from tickets and designs is shift-left in practice.

Smoke testing is a fast set of checks on the most critical paths to confirm a build is stable enough for deeper testing. If smoke tests fail, the build is rejected before more time is spent.

A step group is a reusable collection of steps inserted into many test cases, like a function. Updating the group updates every test that uses it, so common sequences such as 'log in as admin' live in one place.

Test automation is the use of software to run tests, compare results, and report outcomes without manual effort, so checks repeat reliably at speed. AI-native automation also writes and maintains the tests, not just runs them.

A test case is the atomic unit of testing in ContextQA: a name, a starting URL, and an ordered list of natural-language action and verification steps that validate one scenario, producing one result per run.

Test coverage measures how much of the application's requirements or functionality is exercised by tests. It differs from code coverage, which measures lines of source code run, and guides where to add tests.

A test data profile is a parameterized data table for data-driven testing: each column defines a variable and each row one set of inputs. Linking it to a test case creates one execution per row.

A test plan is an execution configuration specifying which suites to run, on which browsers or devices, in which environment, how many in parallel, and on what schedule. It is the primary artifact used for CI/CD.

The test pyramid is a model recommending many fast unit tests at the base, fewer integration tests in the middle, and few end-to-end tests at the top, balancing speed, cost, and confidence.

A test suite is a logical grouping of related test cases, organized by feature (such as Checkout) or purpose (such as Smoke or Regression). Suites are the unit added to a test plan, and a case can belong to many.

Test-driven development (TDD) is a practice where developers write a failing test first, then write just enough code to pass it, then refactor. It drives design and guarantees coverage at the unit level.

A token is the unit of text an AI model reads and generates, roughly a word fragment. Token counts drive model cost and context limits, which matter when testing and budgeting AI features.

A trace file is the saved Playwright trace included in every evidence package, used to replay exactly what happened during a run.

Unit testing checks the smallest pieces of code, such as functions or methods, in isolation. It is fast and forms the base of the test pyramid, complementing the UI and end-to-end tests ContextQA automates.

A variable is a named placeholder used in steps to decouple test data from test logic. ContextQA supports local, global, environment (${ENV.KEY}), and profile (${VAR.COLUMN}) variable scopes.

A vector database stores embeddings, numeric representations of meaning, and retrieves them by similarity. It underpins retrieval-augmented generation and semantic search in AI features that must be tested for relevance.

A verification step asserts a condition rather than performing an action, such as 'Verify the URL contains /dashboard'. In ContextQA these are resolved by the verification agent using visual AI.

A version ID is the workspace version identifier that appears in most ContextQA portal URL paths. It is embedded automatically as you navigate, so you rarely need to handle it directly.

Visual regression testing compares screenshots of the UI across versions to catch unintended visual changes, such as broken layouts or shifted elements, that functional checks would pass over.

Web testing validates web applications in real browsers, covering user flows, forms, and dynamic content. ContextQA creates web tests from plain English, recordings, or imports and runs them across browsers.

A webhook sends test-plan results to any HTTP endpoint on completion, enabling integrations with PagerDuty, custom dashboards, and internal tools.

A workspace is the top-level isolated project in ContextQA, holding its own tests, data, elements, integrations, users, and settings. Workspaces are fully isolated, so teams separate access by product, team, or client.

XPath is a query language for selecting nodes in an XML or HTML tree. ContextQA uses it only as a locator of last resort, preferring more stable strategies such as data attributes, ARIA labels, and text, so users never write XPath by hand.