Teaching Your AI to Test: How to Build Custom Skills for QA Agents

There's a version of AI-powered QA that looks impressive in a demo and falls apart in production. The agent catches obvious regressions, misses the weird edge cases your senior QA engineer would have flagged instantly, and generates reports that are technically accurate but practically useless. Sound familiar?

The problem usually isn't the AI itself — it's that the team handed it a generic testing framework and expected domain expertise in return. Building a QA agent that actually works like a skilled tester means teaching it the way you'd onboard a new team member: with context, judgment, and deliberate practice. That process starts with understanding what these agents are, and what they need to do their job well.

This post covers what QA agents actually are (beyond the buzzword), what "skills" mean in this context, and, most practically, how to build those skills so your agent tests the way a seasoned QA engineer would, not just the way a script runner does.

What is a QA agent?

A QA agent is an AI system that autonomously plans, executes, and evaluates software tests. That definition sounds simple, but the word autonomously is doing real work here. 

It’s important not to confuse an AI agent with a QA agent or use them interchangeably. Not every AI agent. An AI agent is a general category. Specifically, a system that uses an LLM to reason, plan, and act on tools to accomplish a goal, whether that goal is booking travel, managing a calendar, or writing code. A QA agent, on the other hand, is a specific application of that architecture, purpose-built to plan, execute, and evaluate software tests. The reasoning loop is the same, however, what makes it a QA agent is the domain it's aimed at and the judgment it's expected to exercise about software quality. 

Traditional test automation is deterministic. You write a Selenium script, Cypress spec, or Playwright test, and the agent follows it step by step, checks a condition, passes or fails. If the UI changes, the script breaks. If an edge case wasn't anticipated, it goes untested. The human does the thinking, while the tool does the running.

A QA agent inverts that relationship. It uses a large language model (LLM) as a reasoning engine to decide what to test, interpret what it observes, and adapt when something unexpected happens. Give it a feature description and it can generate test cases. Point it at a failing API and it can triage why. Ask it to regression-test a checkout flow and it will probe beyond the happy path without being told to.

The tools are the agent's hands. A browser controller, API client, database inspector, or log reader — these let the agent act on the system under test. But the LLM is the mind. It connects observations to decisions in the same way a human tester would, just faster and without lunch breaks.

What separates a useful QA agent from an impressive demo is how well it's been taught. Out of the box, an LLM is a general-purpose reasoner. It knows a lot about software, APIs, and testing patterns, but it knows nothing about your application, your failure history, your tolerance for false positives, or the subtle business rules that make a "successful" response actually incorrect. That knowledge gap is what skills are designed to close.

Understanding custom skills for QA agents

If the LLM is the brain of the operation, the custom skills are the hands. An agent might have a "Database Verification Skill" that allows it to execute read-only SQL queries to ensure a frontend action actually wrote to the backend. Or it might have a "Visual Regression Skill" that takes a DOM snapshot and compares it against a baseline.

A custom skill is a packaged, reusable unit of expertise you teach an agent: a clear description of when to use it, step-by-step instructions for how to execute it, the tools it's allowed to call, and examples of good and bad output. Think of it as the difference between telling a new hire "go test stuff" versus handing them a runbook for exactly how your team triages a failed payment flow. The skill is the runbook, made executable.

The appeal is real, but so are the trade-offs. Skills reduce prompt drift, make agent behavior auditable, and let you version-control testing judgment the same way you version-control code. They also require upfront investment, ongoing maintenance as your product changes, and discipline to avoid overlapping or contradictory skills. Here's how the two dominant approaches compare in practice:

You must decouple the thinking from the doing. The LLM decides which skill to use based on the context, but the execution of that skill is hardcoded in Python, TypeScript, or Go. The anatomy of a well-architected AI QA skill follows a strict four-phase pipeline: Trigger, Context retrieval, Tool execution, and Verification.

• Trigger. The agent hits a state (new schema, unfamiliar field, CI webhook) and recognizes it needs a tool, not a guess.
• Context retrieval. Before executing a test, the agent must understand the rules of engagement. It pulls the OpenAPI spec, the current DOM snapshot, or queries a vector store containing prior defect patterns and business requirements. Testing without context is just fuzzing, and blind fuzzing is an inefficient use of expensive compute.
• Tool execution. The deterministic core. This is real code: an HTTP client firing a crafted payload, a Playwright module handling a drag-and-drop, a schema validator checking a response body. The LLM decides which tool and which parameters. It never executes the test logic itself.
• Verification. The tool returns a raw result — a stack trace, a status code, a database row — back to the LLM. The agent checks that result against explicit assertions, not vibes, and only then reports back in natural language what happened and why it matters.

Skills vs scripts: What is the difference?

Skills are different from test scripts in a few important ways:

• Scripts encode steps. Skills encode judgment. A script says "click button, assert text equals 'Success'." A skill says "for checkout flows, validate that payment confirmation includes an order ID, the correct total with tax, and a timestamp — and if any of these are missing, that's a critical failure, not a warning."
• Scripts break when things change. Skills adapt. Because a skill defines intent rather than exact selectors or sequences, an agent using a skill can navigate a redesigned UI or a refactored API endpoint without the whole test suite collapsing.
• Scripts are single-use. Skills are composable. You write a skill once for "testing authentication flows" and the agent applies it every time it touches a login, a password reset, or a session expiry — across every environment.

In practice, skills live as structured documents, system prompts, or configuration files that you load into the agent's context. They're the difference between an agent that runs your tests and one that understands what your tests are actually for.

How to teach your AI to test: A practical guide

Building effective QA skills isn't a one-time setup task. It's an iterative process of encoding knowledge, observing behavior, and refining. Here's how to approach it.

1. Start with your best QA engineer's mental model

Before writing a single skill, sit down with your most experienced tester and ask: What do you look for that the scripts don't catch?

Their answers are your raw material. They'll tell you things like:

• "Whenever we deploy a pricing change, I always check that the cart total updates in real time, not just on refresh."
• "API error messages here tend to leak internal stack traces. I always verify the 400 and 500 responses are sanitized."
• "The export function silently truncates CSVs over 10,000 rows. Nobody knows this but me."

That institutional knowledge is exactly what belongs in a skill. An agent can't intuit it from the codebase, you have to transfer it explicitly.

2. Structure skills around testing intent, not implementation

Write skills that describe what good looks like, not how to click through the app. A well-structured skill has four components:

• Scope: What feature or flow does this skill apply to?
• Risk areas: What are the most failure-prone behaviors in this area?
• Evaluation criteria: What constitutes a pass, a warning, and a critical failure?
• Edge cases: What non-obvious scenarios should always be checked?

Avoid encoding UI selectors, specific endpoint paths, or hardcoded values unless they're truly stable. Those belong in configuration, not in the skill itself.

3. Give the agent the right context window

An agent is only as good as the context it has access to when making decisions. When loading skills, also provide:

• System architecture overview: What does this service do? What does it depend on?
• Recent change history: What shipped in the last sprint? What's known to be flaky?
• Environment details: Is this staging, production, a feature branch? The expected behavior differs.
• Failure taxonomy: How does your team classify bugs? What's P0 vs. P3 in your world?

An agent that knows "this endpoint calls a third-party payment processor that rate-limits at 100 req/min" will test it very differently than one that treats it as a black box.

4. Run the agent on known failures first

Before trusting your agent on live testing, run it against a set of bugs you've already found and documented. This is calibration. Does the agent catch what your team would catch? Does it flag the right severity? Does it explain the failure clearly enough that a developer can act on it?

If it misses things, that's a signal that you need either a new skill or a refinement of an existing one. If it over-reports noise, your evaluation criteria are too broad.

5. Treat skill development as a living practice

Your product changes. Your skills need to change with it. Build a lightweight process for reviewing and updating skills after major releases, when new failure modes appear, or when your team's testing priorities shift. The teams that get the most out of QA agents treat skill maintenance the same way they treat updating documentation — as part of shipping.

6.  Find the right balance between determinism and creativity

The biggest trap in AI-augmented testing is demanding determinism from a creative engine. LLMs are built to please you. Ask one "did the checkout process work?" and it will often glance at a 200 status code and eagerly say yes, ignoring that the response body was missing the transaction ID entirely. We ran an internal experiment feeding the same regression prompt to an LLM three days running. Bug counts varied by nearly 40%. Some flagged issues were real. Others were confident hallucinations invented from a stray phrase in a nearby log line.

One enterprise deployment I reviewed made the same point even more starkly. The same prompt ran against the same, completely unchanged application fifteen times. The reported bug count swung from four to thirteen. Nothing in the application had changed between runs. Only the model's mood had. Present that kind of spread in a release-readiness meeting and you will watch engineering leadership's confidence evaporate in real time.

We need the LLM's creativity for exploration: traversing complex user journeys, reading documentation, adapting to a UI that shifted overnight. We need determinism for assertion: deciding, unambiguously, whether a test actually passed. Blur that line and you get an agent that either misses obvious regressions or invents phantom ones, both of which erode trust in the pipeline fast. By pushing the assertion logic down into custom skills, the agent explores creatively but verifies deterministically. That is the whole game.

This table is the argument in miniature. Naive prompting looks impressive in a demo. It falls apart in a CI pipeline running four hundred times a week.

7. Measure agentic QA ROI with the right metrics and KPIs

Engineering teams implementing AI-driven testing inevitably report the wrong metrics to leadership, such as "time saved in test creation," or "number of automated test cases generated." These are vanity metrics. A million auto-generated tests that find no bugs are worse than useless — they are a maintenance liability clogging your pipeline. I push every team I work with toward four harder KPIs instead:

Defect leakage rate to production 

What percentage of production incidents correspond to conditions the agent's skill library should have caught? If this number is not dropping quarter over quarter, your skill coverage has gaps, full stop. This is the metric that tells you whether the agent is functioning as a real gatekeeper or just theater before the merge button.

Escaped edge cases 

When a bug does reach production, ask whether the agent lacked the necessary skill to find it. If a SQL injection vulnerability slips through, the fix is not a longer prompt. The fix is a dedicated database-fuzzing Skill added to the toolbox. The agent's capability should grow iteratively, skill by skill, alongside the application it tests.

Agent token efficiency 

Tokens consumed per verified defect found. Agentic loops can spiral, endlessly calling tools and burning API budget on nothing. A well-designed Skill returns concise, parsable data, not a wall of unstructured noise for the model to re-interpret. If a basic regression run is costing fifty dollars in tokens, the problem is almost never the model — it is a Skill returning too much for the verification phase to digest cleanly. Track the ratio, not the raw total. A platform that spends more tokens overall but finds proportionally more verified defects is healthier than one that looks cheap and finds nothing.

False positive ratio 

Of all "FAIL" verdicts the agent reports, what fraction do humans confirm as real defects on review? Anything above single digits means your verification layer is too loose, or the LLM is annotating results with unearned confidence.

Track these four KPIs and you get a far sharper picture of ROI than any "hours saved" slide will ever give you.

Practical example: Creating a skill for API error handling

Let's make this concrete. Say your team has a REST API and you've noticed that error responses are inconsistent. Some return structured JSON, some return plain strings, some accidentally expose internal details. Here's how a skill creation for this looks like.

Create a skill folder in your .claude. Call it api-error-response-validation. Inside it, create a file called SKILL.md. And at the top of that file, in a small block of YAML, you write a name and a short description:

name: api-error-response-validation
description: >
  Validates that HTTP error responses (4xx/5xx) follow a consistent, safe
  format across an API. Use when a user asks to audit, validate, check, or
  test error handling, error responses, or exception behavior for one or
  more API endpoints.
license: Internal use only
---

That’s the metadata the agent uses to decide whether the skill is relevant for the task in front of it (the triple dashes represent the area of the text dedicated to YAML).

Below it, you write the actual instructions for your api-error-response-validation skill:

# API Error Response Validation

## Scope
All HTTP endpoints returning 4xx and 5xx status codes.

## Risk Areas
- Inconsistent error response format (JSON vs. plain text)
- Missing or incorrect `error_code` field
- Stack traces or internal service/hostnames exposed in `message` or `details`
- HTTP status code mismatches (e.g. a 200 with an error body, or a 500 for a client-side mistake)

## Evaluation Criteria
| Verdict | Condition |
|---|---|
| PASS | Response is JSON, includes `error_code` and a user-safe `message`, and the status code matches the error type |
| WARNING | Response is JSON but `error_code` is missing or non-standard |
| CRITICAL FAILURE | Response exposes a stack trace, internal hostname, database schema, or returns an incorrect status code |

## Edge Cases to Always Test
- Malformed request body (missing required fields, wrong types)
- Requests with expired or revoked authentication tokens
- Requests that exceed rate limits
- Concurrent requests that trigger a race condition

## Execution
Route every check through `scripts/validate_error_response.py`. Do not
hand-write requests or judge the response body by reading it — call the
script and report its verdict.

## Example Invocation
"For each endpoint in the /api/v2/orders namespace, submit a request with
a missing required field. Validate the response against the API Error
Response Validation criteria. Document any endpoints that fail or warn,
with the actual response body."

With this skill loaded, the agent doesn't just check status codes, it reasons about what the response means and flags failures your integration tests would have let through. 

The Skill.md file alone is not the skill, it's the trigger and the spec. The actual determinism lives in the script it points to:

# scripts/validate_error_response.py
import re

STACK_TRACE_SIGNALS = ("Traceback", "at java.", "  File \"", "Exception in thread")
INTERNAL_HOST_PATTERN = re.compile(r"\b(?:10\.\d+\.\d+\.\d+|internal|staging-|\.local)\b", re.I)

def validate_error_response(status_code: int, body: dict | str, expected_status: int) -> dict:
    """Deterministic verdict for one captured error response. No LLM judgment involved."""
    if not isinstance(body, dict):
        return {"verdict": "CRITICAL FAILURE", "reason": "Response is not JSON"}

    raw = str(body)
    if any(s in raw for s in STACK_TRACE_SIGNALS) or INTERNAL_HOST_PATTERN.search(raw):
        return {"verdict": "CRITICAL FAILURE", "reason": "Internal details exposed"}

    if status_code != expected_status:
        return {"verdict": "CRITICAL FAILURE", "reason": f"Status {status_code}, expected {expected_status}"}

    if "error_code" not in body:
        return {"verdict": "WARNING", "reason": "Missing error_code field"}

    if not body.get("message") or len(str(body["message"])) > 300:
        return {"verdict": "WARNING", "reason": "Message missing or unusually long"}

    return {"verdict": "PASS", "reason": "Conforms to error response contract"}

Notice the shape: SKILL.md tells the agent when to reach for this and what good looks like. The script tells it how to check, deterministically, every time. That split is the whole discipline this article has been arguing for, in its smallest possible form.

Here is the structured authoring prompt that would produce this exact skill, following the same template introduced later in this article:

ROLE: A staff QA engineer specializing in API contract and error-handling
consistency.
OBJECTIVE: Produce a skill that validates HTTP error responses (4xx/5xx)
for consistent, safe formatting across a given API surface.
INPUT CONTRACT: a target endpoint or namespace, plus the set of conditions
to provoke an error response (malformed body, expired/revoked token, rate
limit exceeded, concurrent request race).
OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter (name, a
what-plus-when description, license) and body sections for Scope, Risk
Areas, Evaluation Criteria (PASS / WARNING / CRITICAL FAILURE), Edge Cases
to always test, and an Execution section naming the backing script.
DETERMINISM CONSTRAINTS: all PASS/WARNING/CRITICAL FAILURE logic must live
in scripts/validate_error_response.py, never in the SKILL.md prose or in
the model's own reading of the response body. The LLM's only job is
selecting which endpoints and conditions to target, then calling the
script.
VERIFICATION CRITERIA: the same endpoint and condition must return the
same verdict on repeated runs; a deliberately clean response must always
PASS; a response containing a stack trace or internal hostname must
always return CRITICAL FAILURE.

More examples of structured prompts for creating custom QA skills 

1. Test case generation & planning skill

ROLE: A staff QA engineer specializing in test planning and coverage design.

OBJECTIVE: Produce a skill that, given a feature specification, user
story, API schema, or PR diff, generates a structured, traceable set of
test cases covering functional, boundary, negative, and regression
scenarios — without inventing requirements the source material doesn't
support.

INPUT CONTRACT: the finished skill accepts source_type (enum:
user_story | api_spec | ui_mockup | pr_diff), source_content (the spec
text, OpenAPI schema, or diff itself), existing_test_inventory (optional
list of already-covered case IDs, to avoid duplicate generation), and
risk_tier (enum: low | medium | high — determines how exhaustive
coverage must be).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: test-case-generation-and-planning
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks to generate test cases, plan test coverage, or
    identify untested scenarios for a spec, schema, or diff. Do NOT use
    for writing the test automation code itself.")
  license: <org's internal license>
— plus body sections for Scope, Input Types Supported, Case Categories
and Their Trigger Rules (the field-type → required-case-type lookup
table — e.g. every numeric field always gets a boundary + negative case,
every enum field gets one case per invalid value, every required field
gets a "missing field" case), Traceability Rules, Priority Assignment
Criteria (P0/P1/P2), and an Execution section naming
scripts/generate_test_cases.py as the only place field parsing,
category-lookup, deduplication, and priority assignment happen.

DETERMINISM CONSTRAINTS: the field-type → case-type lookup table is
hardcoded in scripts/generate_test_cases.py, never left to model
judgment — the LLM's only job is parsing source_content to identify the
fields, states, and transitions present, not deciding which categories
of case apply to them. Deduplication against existing_test_inventory
runs as a string/hash comparison in code, never as an LLM "does this
look similar" judgment. Every generated case must carry a traceable_to
value pointing to something explicitly present in source_content — the
script rejects any case missing that field before it's returned.

VERIFICATION CRITERIA: re-running against the same source_content and
same existing_test_inventory must produce the same case set, same IDs,
and same priorities every time. Every generated case's traceable_to
field must resolve to a locatable line, field, or requirement in
source_content — checked programmatically, not asserted by the model.
For a given risk_tier, the generated set must include at least the
minimum required category mix (e.g. high risk_tier requires at least
one security-category case per externally-facing field), verified
against the output rather than trusted from the model's own summary.

2. Test data generation & management skill 

ROLE: A staff QA engineer specializing in test data engineering and
environment hygiene.

OBJECTIVE: Produce a skill that, given a target schema or fixture
definition, generates realistic, constraint-valid synthetic test data,
provisions it into the correct environment, and tears it down
afterward — without ever touching production data or leaking real PII
into a test environment.

INPUT CONTRACT: the finished skill accepts schema_source (JSON Schema,
DB DDL, or ORM model reference), record_count (integer, how many
rows/objects to generate), environment (enum: local | staging |
ephemeral_ci), data_profile (enum: happy_path | edge_case | volume_load
| referential_integrity), and retention (enum: session_only |
persist_until_teardown).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: test-data-generation-and-management
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks to generate test data, seed a test environment, or
    provision/tear down fixtures for testing. Do NOT use for querying
    or modifying data in a production or customer-facing environment.")
  license: <org's internal license>
— plus body sections for Scope, Supported Environments (the explicit
allowlist — staging, local, ephemeral_ci, never production), Data
Profiles and What Each Generates, PII Safety Rules, Referential
Integrity Rules, Teardown Requirements, and an Execution section naming
scripts/generate_test_data.py and scripts/teardown_test_data.py as the
only code paths permitted to touch a database.

DETERMINISM CONSTRAINTS: data generation runs through a seeded generator
(e.g. Faker with a fixed seed per batch_id) inside
scripts/generate_test_data.py — the LLM never invents field values
free-form. Referential integrity (foreign keys, unique constraints,
required relationships) is enforced by code that reads schema_source,
never by the LLM "remembering" which tables relate to which. Environment
targeting is a hardcoded allowlist check inside the script: the skill
refuses to run against any environment not explicitly present in an
environment-config file, regardless of what the LLM is told the
environment is — the LLM selects an environment name, it never
constructs a connection string. The PII scan is a pattern-matching pass
(emails, SSNs, phone numbers, real-looking names checked against a
blocklist) run in code against the generated output before it's
confirmed written, never a judgment call by the model on whether the
data "looks fake enough." Teardown is generated as an executable script
(matching DELETE/rollback statements for exactly what was inserted) at
creation time, never reconstructed from memory later.

VERIFICATION CRITERIA: re-running with the same seed, schema_source, and
record_count must produce byte-identical data. Every foreign-key
reference in generated data must resolve to a record either generated in
the same batch or already verified to exist — zero orphaned references,
checked programmatically. Running the teardown script must leave the
target environment in exactly its pre-batch state, confirmed by a
row-count diff, not assumed to have worked. The skill must refuse
execution outright if environment resolves to anything matching a
production naming pattern (prod, live, master), with no override path
available to the LLM.

3. Self-healing / Test maintenance skill

ROLE: A staff QA automation engineer specializing in locator resilience
and test suite maintenance.

OBJECTIVE: Produce a skill that, when a UI or API test fails due to a
changed selector, moved element, or renamed field — not a genuine
behavioral regression — locates a valid replacement using a fixed
strategy hierarchy, applies the fix with full audit history, and
explicitly refuses to "heal" anything that looks like a real defect
instead of a cosmetic change.

INPUT CONTRACT: the finished skill accepts failed_test_id (string),
original_locator (string, e.g. a CSS selector, XPath, or field name),
locator_type (enum: id | css | xpath | api_field |
accessibility_label), failure_context (raw error, screenshot, or DOM
snapshot at time of failure), and last_known_good_snapshot (DOM/schema
state from the last passing run).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: self-healing-test-maintenance
  description: states what it does and when to trigger it (e.g. "Use
    when a test fails due to a changed selector, moved element, or
    renamed field. Do NOT use for triaging assertion failures on
    unchanged UI, or for any failure where the underlying behavior —
    not just the locator — has changed.")
  license: <org's internal license>
— plus body sections for Scope, Strategy Hierarchy (in priority order:
stable ID > data-testid attribute > accessibility label > visible text
> structural DOM-path similarity), Confidence Threshold and What
Happens Below It, Escalation vs. Rejection Criteria, Human Review
Requirements, and an Execution section naming
scripts/locate_replacement.py as the only place similarity scoring and
locator matching happen.

DETERMINISM CONSTRAINTS: candidate replacement locators are found by
comparing the current DOM or schema against last_known_good_snapshot
using the fixed strategy hierarchy above, evaluated in that order by
scripts/locate_replacement.py — the LLM does not invent a
plausible-looking selector from general knowledge of the page.
confidence_score is computed deterministically from a similarity
metric (e.g. attribute overlap percentage, DOM-path edit distance)
inside the script, never asserted by the model as a feeling of
certainty. The script enforces a hardcoded rule to separate drift from
regression: if no candidate clears a minimum similarity floor, or if
the surrounding assertion — not just the locator — also fails, the
verdict is REJECTED, not HEALED, because that pattern indicates a real
regression. The LLM's only role is selecting which failure_context to
investigate and deciding whether to invoke this skill at all; it never
edits the test file directly. All locator replacement writes go through
a diff-and-commit step gated by human or automated approval above a
configured risk tier — the script never applies a fix silently.

VERIFICATION CRITERIA: every HEALED verdict must include a diff_summary
a human can read and approve or reject in under thirty seconds, or the
skill must reject its own output before returning it. Re-running the
skill against the same failure_context and last_known_good_snapshot
must produce the same verdict and the same confidence_score. As a
control, the "healed" locator must also be replayed against
last_known_good_snapshot itself — if it fails to resolve there, the
heal is invalid by construction and the script must flag it, not the
model. Escalation rate is tracked over time; a skill escalating fewer
cases than its confidence threshold would predict signals the threshold
itself has drifted too permissive, not that the skill is improving.

4. Impact analysis & smart test selection skill

ROLE: A staff test infrastructure engineer specializing in dependency
analysis and CI test selection.

OBJECTIVE: Produce a skill that, given a code diff or pull request,
identifies the minimal set of tests that must run to validate the
change — based on actual coverage and dependency data, not a guess
about which tests "seem related" by file name or folder proximity.

INPUT CONTRACT: the finished skill accepts diff_source (git diff, PR
reference, or list of changed file paths), coverage_map (a precomputed
test-to-code coverage index), dependency_graph_source (import/call-graph
data, if coverage_map is incomplete), risk_tier (enum: low | medium |
high — determines how aggressively the skill is allowed to narrow the
test set), and test_inventory (the full list of available test IDs).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: impact-analysis-smart-test-selection
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks which tests to run for a given change, wants to
    reduce CI runtime, or asks for impacted-test selection based on a
    diff or PR. Do NOT use this skill to judge whether a change is safe
    to merge — it selects tests, it does not assess risk.")
  license: <org's internal license>
— plus body sections for Scope, Selection Strategy (in priority order:
direct coverage-map lookup > dependency/import-graph traversal >
module-level fallback > full-suite fallback), Fallback Rules by
risk_tier, Coverage Confidence Reporting, Stale-Coverage Handling, and
an Execution section naming scripts/select_impacted_tests.py as the
only place the diff-to-test mapping is computed.

DETERMINISM CONSTRAINTS: the mapping from changed files or lines to
affected tests is computed inside scripts/select_impacted_tests.py by
querying coverage_map or traversing dependency_graph_source — never by
the LLM inferring relevance from file names, folder structure, or
general familiarity with the codebase. If coverage_map has no entry for
a changed file, or its last-updated timestamp is older than a
configured freshness threshold, the script must automatically widen to
the next fallback tier (module-level, then full suite) rather than let
the LLM decide the risk is acceptable. The LLM's only job is supplying
diff_source and risk_tier and reporting the result; it never edits the
selected test list by hand and never overrides a fallback the script
triggered.

VERIFICATION CRITERIA: re-running against the same diff_source,
coverage_map, and dependency_graph_source must produce the same selected
test set every time. The selected set must be provably a superset of
every test with direct coverage over the changed lines — checked by
cross-referencing coverage_map programmatically, not asserted by the
model's summary. Every escaped defect traced back through this skill
must be classified by the script as either a selection failure (a
relevant test existed and wasn't chosen) or a coverage gap (no relevant
test existed at all) — that distinction is computed from coverage data,
never decided by the model after the fact.

5. Exploratory testing & visual/UI validation skill

ROLE: A staff QA engineer specializing in exploratory testing and
visual/UI regression detection.

OBJECTIVE: Produce a skill that performs bounded, systematic traversal
of an application's UI states, flags visual regressions against a
baseline, and surfaces broken interactions or console/network errors
encountered along the way — without wandering unboundedly or judging
correctness of computed business values (that belongs to a separate
functional skill).

INPUT CONTRACT: the finished skill accepts target_url_or_component
(string), baseline_snapshot_set (reference screenshots or DOM
fingerprints per state), exploration_scope (enum: single_page | flow |
full_app), max_actions (integer, hard bound on traversal depth per
run), and ignore_regions (list of coordinates or selectors for known
dynamic content — timestamps, ads, live counters — excluded from visual
diffing).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: exploratory-testing-visual-ui-validation
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks to explore an app for unexpected UI states, hunt
    for visual regressions, or validate that a page or flow renders
    correctly. Do NOT use to assert the correctness of computed
    business values — pair with a functional or API validation skill
    for that.")
  license: <org's internal license>
— plus body sections for Scope, Exploration Strategy (state-graph
traversal order: unvisited clickable elements > form boundary states >
error-triggering paths, bounded by max_actions), Visual Diff Rules
(threshold and ignore_regions handling), Interaction Health Checks
(console errors, 4xx/5xx network calls, dead links or unresponsive
buttons encountered mid-traversal), Finding Classification
(VISUAL_REGRESSION | BROKEN_INTERACTION | CONSOLE_ERROR |
EXPECTED_DYNAMIC_CHANGE), and an Execution section naming
scripts/explore_and_validate.py as the only place DOM traversal,
screenshotting, and diffing happen.

DETERMINISM CONSTRAINTS: state-graph traversal — which element gets
clicked next — runs on a deterministic algorithm (e.g. breadth-first
over unvisited interactive elements, capped at max_actions) inside
scripts/explore_and_validate.py, never on the LLM wandering by
impression. Visual differences are computed via a pixel-comparison
library (e.g. pixelmatch/Resemble.js) against baseline_snapshot_set,
never by the model's vision judging "this looks different." Console and
network errors are captured through browser instrumentation hooks
during traversal, not reconstructed afterward from a log the LLM reads
and interprets. ignore_regions is a fixed list supplied as input, never
inferred on the fly by the model deciding a region "looks dynamic." The
LLM's only role is choosing the target and setting exploration_scope
and max_actions, then reading back the script's classified findings —
it does not reclassify a finding in real time.

VERIFICATION CRITERIA: re-running against the same target,
baseline_snapshot_set, and max_actions — with the app unchanged — must
produce the same traversal path and the same finding set every time.
Every VISUAL_REGRESSION finding must report an identical diff
percentage across repeat runs. A replay against a known-clean baseline
must return zero findings, used as a negative control. A deliberately
injected visual change or console error must always be caught,
confirming the skill hasn't silently degraded.

6. Requirements traceability & gap analysis skill

ROLE: A staff QA engineer specializing in requirements traceability and
test coverage auditing.

OBJECTIVE: Produce a skill that, given a set of requirements or
acceptance criteria and an existing test inventory, builds an explicit
traceability map between the two and surfaces every requirement with
no corresponding test, every test with no traceable requirement, and
every requirement covered only partially — without inferring coverage
from naming similarity or file proximity.

INPUT CONTRACT: the finished skill accepts requirements_source
(user stories, acceptance criteria, a PRD excerpt, or a ticket-tracker
export), test_inventory (the full list of existing test cases with
their stated traceable_to fields, if any), matching_mode (enum:
explicit_id | semantic_assist — explicit_id trusts only stated
requirement IDs, semantic_assist allows the LLM to propose candidate
matches for human confirmation when no explicit ID exists), and
risk_tier (enum: low | medium | high — determines what counts as an
acceptable gap).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: requirements-traceability-gap-analysis
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks for a traceability matrix, wants to know which
    requirements lack test coverage, or needs a gap analysis before a
    release or audit. Do NOT use this skill to generate the missing
    tests themselves — pair with a test case generation skill for
    that.")
  license: <org's internal license>
— plus body sections for Scope, Matching Rules by matching_mode,
Coverage Classification (FULL | PARTIAL | NONE | ORPHANED_TEST),
Minimum Coverage Requirements by risk_tier, Confidence Reporting for
semantic_assist matches, and an Execution section naming
scripts/build_traceability_map.py as the only place matching and
classification happen.

DETERMINISM CONSTRAINTS: in explicit_id mode, a requirement is only
considered covered if a test's traceable_to field contains an exact,
matching identifier — string equality checked in code, never inferred
by the LLM from similar-sounding titles. In semantic_assist mode, the
LLM may propose a candidate match, but every proposed match is returned
with requires_human_review set to true and a confidence_score computed
by a fixed similarity metric in scripts/build_traceability_map.py — it
is never auto-promoted to FULL coverage without that review flag.
Coverage classification (FULL/PARTIAL/NONE/ORPHANED_TEST) is assigned
by the script based on the count and category of matched tests per
requirement, never asserted by the model's own summary of the map.

VERIFICATION CRITERIA: re-running against the same requirements_source
and test_inventory in explicit_id mode must produce an identical
traceability map every time. Every requirement marked FULL must have at
least one matched test whose traceable_to field is independently
verifiable by a human against requirements_source — checked
programmatically, not asserted. For a given risk_tier, any requirement
classified NONE must be surfaced as a blocking finding, not a footnote,
and the skill must refuse to summarize a release as "fully traced" if
any risk_tier-relevant requirement remains NONE.

7. Bug reproduction & debugging assistant skill

ROLE: A staff QA engineer specializing in bug reproduction and
diagnostic evidence gathering.

OBJECTIVE: Produce a skill that, given a bug report, deterministically
attempts to reproduce it by replaying the reported steps against a
target environment, captures diagnostic evidence, and narrows the
result to a minimal reproduction — without asserting a root cause the
evidence doesn't actually support.

INPUT CONTRACT: the finished skill accepts bug_report (description,
reported steps, environment, expected vs. actual behavior),
attached_evidence (logs, stack traces, screenshots, or request/response
payloads if available), target_environment (enum: staging | local |
ephemeral_repro), reproduction_attempts_allowed (integer cap, e.g. 10),
and known_flaky_signature (optional reference to check against an
existing registry of known-flaky failures).

OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: bug-reproduction-debugging-assistant
  description: states what it does and when to trigger it (e.g. "Use
    when a user reports a bug and asks to reproduce it, gather
    diagnostics, or determine exact repro steps. Do NOT use this skill
    to fix the bug or assert a root cause — it gathers evidence, it
    does not diagnose.")
  license: <org's internal license>
— plus body sections for Scope, Reproduction Strategy (exact
request/response replay first, if captured; guided UI replay second;
manual step list as last resort), Evidence Capture Rules (logs,
screenshots, network traces always collected, never summarized before
capture), Minimization Rule (bisecting steps to find the smallest
reliable repro), Flaky-Failure Handling (checking against
known_flaky_signature and repeating runs to test consistency), Verdict
Classification (REPRODUCED_DETERMINISTIC | REPRODUCED_INTERMITTENT |
NOT_REPRODUCED | ENVIRONMENT_MISMATCH), and an Execution section naming
scripts/reproduce_bug.py as the only place replay and evidence capture
happen.

DETERMINISM CONSTRAINTS: replay of recorded steps (HTTP requests, UI
actions) executes through scripts/reproduce_bug.py exactly as
captured — the LLM does not paraphrase, "clean up," or improve the
steps before replay. Evidence capture (logs, screenshots, network HAR)
is collected by fixed instrumentation in the script, never
summarized or interpreted by the model on the fly. Flakiness
classification is based on a fixed repeat-count threshold computed by
the script (e.g. must reproduce in at least 8 of 10 attempts to count
as REPRODUCED_DETERMINISTIC), never asserted by the LLM after a single
run. The LLM's only job is parsing bug_report to extract the initial
step list and target environment, and deciding when to invoke the
skill — it never fabricates a stack trace or claims a root cause the
script's evidence doesn't show, and the output schema has no field for
one.

VERIFICATION CRITERIA: re-running against the same bug_report and
target_environment must produce the same verdict classification and
the same reproduction rate every time. Every REPRODUCED verdict must
include a minimal step list a human can independently execute and get
the identical result. A failure already present in
known_flaky_signature must be flagged as such, never re-classified as
a new deterministic bug just because it happened to reproduce once.
The skill must structurally refuse to emit a root-cause claim — that
field doesn't exist in the output contract, so there's nothing for the
model to fill in even if it tried.

8. Boundary value & edge case injector skill

ROLE: A staff QA engineer specializing in API input validation.
OBJECTIVE: Produce a skill that generates and executes a fixed catalog
of boundary and malformed values for a schema field, classifying each
response PASS or FAIL.
INPUT CONTRACT: the finished skill accepts endpoint, field_name,
field_type, and constraints.
OUTPUT CONTRACT: a SKILL.md file with YAML frontmatter —
  name: boundary-value-edge-case-injector
  description: states what it does and when to trigger it (e.g. "Use
    when a user asks to fuzz, edge-case-test, or boundary-test a schema
    field.")
  license: <org's internal license>
— plus body sections for Scope, Boundary Case Catalog by Field Type,
Evaluation Criteria, and an Execution section naming
scripts/run_boundary_value_injection.py as the only place payloads are
generated and requests are fired.
DETERMINISM CONSTRAINTS: the case catalog is hardcoded per field_type
inside the script, never in SKILL.md prose; the LLM selects field_type
and constraints, never payload values.
VERIFICATION CRITERIA: identical inputs must produce identical verdicts
across runs.

Common mistakes teams make when teaching AI to test

Even teams with good intentions run into the same pitfalls. Here's what to watch for.

• Treating the agent like a script runner. If you're just using an AI agent to execute the same deterministic test scripts you had before, you're not using it wrong, but you're leaving most of its value on the table. The leverage is in giving it judgment-heavy tasks that don't fit neatly into a script.
• Giving it too much scope without enough context. "Test the whole app" is not a skill. An agent with no constraints will produce broad, shallow coverage. Scoped skills with clear evaluation criteria produce focused, actionable results.
• Skipping the calibration step. Teams often deploy agents directly into their CI pipeline without first testing the agent's judgment against known failures. This leads to false confidence when the agent misses things, and alert fatigue when it over-reports.
• Letting skills go stale. A skill written for your app six months ago may be describing behavior that no longer exists, or missing failure modes that have emerged since. Skills without maintenance dates and owners tend to drift out of sync with the product.
• Over-relying on the agent for test case generation. AI agents are good at executing tests and evaluating results, however, they're less reliable at designing comprehensive test suites from scratch. Use them to scale and sharpen testing work that humans have scoped and prioritized, not to replace that scoping entirely.
• Ignoring the output format. An agent that finds real bugs but reports them in a format that doesn't integrate with your issue tracker, Slack workflow, or sprint process won't get used. Teach the agent not just what to find, but how to communicate it.
• Writing a vague trigger description and blaming the model when it misfires. A description like "helps with testing" will under-trigger. The agent won't reach for the skill when it should, and someone ends up manually invoking it every time. Overcorrect with something too broad and it over-triggers on unrelated requests instead. The fix in both directions is the description, not the model: add specific trigger phrases to fix under-triggering, add explicit negative examples ("do NOT use for general data exploration") to fix over-triggering.

The bottom line

A QA agent is only as effective as the knowledge you've given it to work with, which is why human input remains key. The agents that catch the bugs that matter, the ones that would have made it to production, aren't running generic test suites. They're carrying the accumulated judgment of the engineers who've been burned before, encoded into skills that can be applied consistently, adapted over time, and scaled across every release.

The investment in building those skills is an investment in making your agent actually useful. That's what separates "AI-assisted QA" from "AI that checks a box."