AI Pentesting Checklist for Web Apps: 15 Questions to Ask Before Choosing a Vendor
Exploitation of web‑application flaws is now happening before patches exist. This guide explains why AI‑driven pentesting matters and provides a 15‑question, 36‑point checklist to help you evaluate AI pentesting vendors safely and effectively.
Why the pace of exploitation matters
Web‑facing applications are a favourite hunting ground for attackers. In the first half of 2026, 23% of the vulnerabilities that entered VulnCheck’s exploited list were hit on or before the day they were disclosed. That means a flaw can be weaponised before any organisation has a chance to patch it. The speed of exploitation creates a clear business risk: services can be taken offline, data can be stolen, and remediation costs can swell before the problem is even known.
What AI‑pentesting agents can do
Traditional static analysis and fuzzing tools struggle to keep up with this pace. Google’s Threat Intelligence Group recently reported a threat actor using a zero‑day exploit that was believed to be built with AI to bypass two‑factor authentication via a logic flaw that conventional testing missed. At the same time, a Nasdaq‑cited survey found that 87% of US security leaders are planning, piloting, or already using AI‑powered pentesting agents. These agents can automatically map an application, probe trust relationships, and simulate attacks faster than a manual team.
Real‑world findings that matter
- An AI agent uncovered an unauthenticated remote code execution vulnerability in Apache Airflow (CVE‑2026‑68982). Instead of scanning source code, the agent examined how deployment components trusted each other and identified a path to execute code without credentials.
- During an authorised assessment of a major insurer’s production website, an AI agent demonstrated a one‑time passcode (OTP) bypass. By sending login codes to any email address, the agent logged in with a code sent to its own inbox. Guardrails that were in place stopped the test before further exploitation. These cases show that AI can discover flaws that manual testing or classic automation often miss, but they also highlight the need for strong safety controls.
How to compare AI‑pentesting vendors
Vendors differ in how they build and maintain site maps. Some start from a blank URL on every run, rebuilding the model each time. Others keep an evolving map of the application that is updated incrementally. This choice affects scan speed, coverage, and the ability to detect subtle, persistent logic errors.
The checklist below focuses on five pillars that matter most when selecting a vendor: 1. Context, Does the vendor understand your architecture, business logic, and risk profile? 2. Reach, How comprehensively does the agent explore your attack surface, including hidden endpoints and inter‑service trust? 3. Validated findings, Are vulnerabilities confirmed through safe exploitation or realistic simulation? 4. Guardrails, What mechanisms prevent the agent from causing disruption during testing? 5. Retests, Can the vendor reliably verify that fixes work after remediation?
The 15‑question checklist (summary)
| # | Question | Why it matters |
|---|---|---|
| 1 | Does the vendor require a clean‑room environment or can it test directly on production copies? | Limits impact on live services. |
| 2 | How does the agent build the site map, from scratch each run or from an evolving model? | Affects consistency and coverage. |
| 3 | Can the tool discover unauthenticated remote code execution paths like CVE‑2026‑68982? | Demonstrates depth of analysis. |
| 4 | Does the vendor provide an OTP‑bypass guardrail that stops the test after a successful proof? | Prevents unintended credential abuse. |
| 5 | Are findings ranked by severity and likelihood of exploitation? | Helps prioritise remediation. |
| 6 | Does the solution integrate with existing ticketing or SIEM systems? | Streamlines workflow. |
| 7 | What is the false‑positive rate based on independent benchmarks? | Reduces wasted effort. |
| 8 | Are there limits on request rates to avoid denial‑of‑service impact? | Protects availability. |
| 9 | How does the vendor handle data privacy for any captured payloads? | Meets compliance requirements. |
| 10 | Is there a clear process for retesting after patches are applied? | Confirms remediation success. |
| 11 | Does the vendor support multi‑tenant testing for SaaS environments? | Aligns with modern cloud deployments. |
| 12 | Are the AI models explainable to non‑technical stakeholders? | Increases trust in findings. |
| 13 | What SLA guarantees are offered for scan completion and reporting? | Sets expectations for delivery. |
| 14 | Does the service include continuous monitoring or only point‑in‑time scans? | Determines long‑term coverage. |
| 15 | Are there configurable safety thresholds that can be tuned to your risk appetite? | Balances thoroughness with safety. |
Each affirmative answer scores points on a 36‑point scorecard. A vendor that reaches 30 points or more can be considered mature enough for production use, provided the guardrails meet your organisation’s risk tolerance.

Photo by Matheus Bertelli on Pexels
Running a vendor bake‑off
- Deploy a test instance of a critical web application that mirrors production configurations.
- Create limited‑privilege test accounts that reflect real user roles.
- Run each vendor’s AI agent against the test instance, observing how it builds the site map and whether it discovers the known RCE in Apache Airflow.
- Verify that the OTP‑bypass guardrail triggers as expected on the insurer‑style flow.
- Record the findings, false‑positive rate, and any impact on performance. Score each vendor using the 36‑point sheet.
Key takeaways
- Exploitation is outpacing patching; rapid detection is now a business imperative.
- AI‑driven pentesting agents can reveal logic‑flaw exploits that static tools miss, but they must be evaluated for safety.
- Use the 15‑question checklist and 36‑point scorecard to compare vendors objectively.
- Conduct a hands‑on bake‑off on your own applications before committing to a vendor.
- Guardrails that halt testing after a successful exploit are essential to avoid accidental damage.
By applying a disciplined evaluation process, security teams can harness the speed of AI while keeping control over risk.

Photo by AS Photography on Pexels
Terms used in this article
- AI, Artificial intelligence, computer systems that can learn patterns and make decisions.
- Pentesting, Penetration testing, a simulated attack that checks for security weaknesses.
- RCE, Remote code execution, a flaw that allows an attacker to run code on a server.
- OTP, One‑time passcode, a short‑lived numeric code used for two‑factor authentication.
- CVE, Common Vulnerabilities and Exposures, a public identifier for a security flaw.
- Guardrails, Built‑in safety controls that stop a test if it threatens production stability.
- Site map, A model of all the pages, endpoints and services that make up a web application.
Read the full AI Pentesting Checklist and vendor scorecard here