AI E2E Test Runner

Write end-to-end tests in plain English. No Playwright code required. This Actor turns natural-language steps into browser actions, runs them against a public website, and returns pass/fail results with screenshots, session video, console logs, and network errors.

Paste a URL, list the steps a user would take, and start the run. Indie hackers, small product teams, and non-technical PMs can smoke-test signup, search, checkout, and dashboard flows without standing up a CI pipeline.

What does AI E2E Test Runner do?

Give it a website URL and a list of instructions such as “Click Sign Up”, “Fill email field with test@example.com”, or “Verify that Welcome appears”. The Actor:

• Parses each instruction into a structured action (navigate, click, fill, verify, screenshot, and more)
• Opens Chromium (Firefox/WebKit when available) with smart waits and self-healing locators
• Retries a quick-failing step up to three times (retries stop once the step timeout is used up)
• Saves screenshots and an optional WebM recording
• Writes a single JSON result with per-step status, duration, and artifacts. A failing test is a result, not a crashed run: the run still succeeds and the dataset item has status: "failed" and an error message

Unrecognized phrasing is handled with a rule-based parser first. When an OpenRouter key is configured on the Actor, leftover instructions can be interpreted by a small language model. The runner never executes generated JavaScript — only structured actions.

Why run E2E tests in plain English?

• Ship faster — product people can describe a flow instead of waiting on a test engineer
• Catch broken CTAs — verify that buttons, forms, and welcome copy still appear after a deploy
• Keep evidence — screenshots and video make failures easy to share in Slack or a ticket
• Automate on Apify — schedule nightly smoke tests, call the Actor from CI, or fan out URLs via API
• No test repo required — useful when you do not yet have Playwright in the application codebase

Who is it for?

• Indie hackers smoke-testing marketing sites and SaaS onboarding
• Small engineering teams that want a scheduled Playwright check without maintaining fixtures
• PMs and QA generalists who can write steps but not selectors
• Agencies running the same “happy path” against many client staging URLs

How to run a plain-English Playwright test

1. Open the Actor in Apify Console
2. Set Website URL to a publicly reachable page
3. Add Test instructions in order (navigate, click, fill, verify, screenshot)
4. Keep Maximum steps small for a first run
5. Click Start
6. Open the dataset for pass/fail, then the key-value store for PNG screenshots and WebM video

What instructions are supported?

Locators try accessible names, labels, placeholders, and visible text. If one strategy misses, the next is tried automatically.

Input

{
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "instructions": [
    "Navigate to the homepage",
    "Fill 'What needs to be done?' field with 'Buy groceries'",
    "Press Enter",
    "Verify that 'Buy groceries' appears",
    "Take a screenshot of the todo list"
  ],
  "maxItems": 5,
  "browser": "chromium",
  "viewportWidth": 1920,
  "viewportHeight": 1080,
  "timeout": 30000,
  "screenshots": true,
  "video": true,
  "continueOnFailure": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

This Actor only drives public pages. Do not use it to log into other people’s accounts or to bypass paywalls. For your own staging site, prefer a dedicated test user that you control.

Output

One dataset item is written per run (also stored as OUTPUT in the key-value store):

{
  "status": "passed",
  "duration": 4523,
  "websiteUrl": "https://demo.playwright.dev/todomvc",
  "finalUrl": "https://demo.playwright.dev/todomvc/#/",
  "imageUrl": "https://api.apify.com/v2/key-value-stores/.../records/screenshot-todo_list?fpr=uuazcu",
  "videoUrl": "https://api.apify.com/v2/key-value-stores/.../records/video-test-recording?fpr=uuazcu",
  "steps": [
    {
      "action": "navigate",
      "instruction": "Navigate to the homepage",
      "status": "passed",
      "duration": 1200,
      "url": "https://demo.playwright.dev/todomvc"
    },
    {
      "action": "click",
      "selector": "button|link|text='Sign Up'",
      "status": "passed",
      "duration": 340
    }
  ],
  "artifacts": {
    "screenshots": ["screenshot-todo_list"],
    "video": "video-test-recording",
    "consoleLogs": ["info: App initialized"]
  },
  "error": null
}

When a step fails, status is failed, error holds the first failure, and later steps are skipped (unless Continue on failure is on). If the run gets close to its Apify timeout, the remaining steps are marked skipped so the result and artifacts are still saved. If the browser cannot record video, the test continues without it and videoUrl is null.

Download the dataset as JSON, CSV, or Excel. Screenshot files use image/png; recordings use video/webm (Playwright’s native format).

How much does a test run cost?

Pricing is pay per test run. You are charged one test-run-completed event after the Actor saves the result and artifacts — not per instruction. A test that fails is still a completed test run. If the browser itself cannot start, the error is saved to the dataset and no event is charged.

A typical smoke test (a handful of steps, screenshots on, video on, 1 GB memory) finishes in well under a minute of compute. Enable Apify Proxy only when the target site requires it; proxy traffic is billed separately by Apify.

Use cases

• Nightly smoke test of marketing homepage CTAs
• Staging signup form still submits and shows a welcome message
• Todo, search, or filter UI still accepts typed input
• Pricing page still shows the advertised plan names
• Docs site search still returns a known article title
• Agency client launch checklist before go-live
• Post-deploy check from GitHub Actions via the Apify API
• Capture a short video of a flaky flow for a bug report
• Verify cookie-banner copy without writing selectors
• Regression check after a CMS content change
• Multi-viewport screenshot of a landing page (set width/height)
• Onboarding checklist for a non-technical founder

Tips

• Start with 3–5 instructions and a known-good public demo before pointing at production
• Quote visible labels exactly: Fill 'What needs to be done?' field with 'Buy groceries'
• Prefer Verify that ‘…’ appears over vague “the page looks good” steps
• Turn Continue on failure on when you want later screenshots even if an early click misses
• The default Chromium image is enough for most sites; Firefox/WebKit may fall back to Chromium
• Failed steps store a screenshot-failed-step-N record (PNG) so you can see what the browser showed

FAQ

Is this a replacement for a full Playwright suite?
No. It is a fast, instruction-driven smoke runner. Keep unit tests and committed Playwright specs for complex application logic.

Can it log into Gmail, a bank, or a customer’s private account?
No. Only test pages that are publicly reachable or that you already made available without harvesting third-party credentials.

Why is the video WebM instead of MP4?
Playwright records WebM. Store players and most browsers play it; convert downstream if you need MP4.

What if a selector is missing?
The step is retried up to three times with alternate locators, as long as it is still within the step timeout. The error message names the instruction, and a failure screenshot is saved when screenshots are enabled.

Do I need an OpenRouter key?
No. Common English patterns run without an LLM. A configured OPENROUTER_API_KEY only helps with unusual phrasing.

Other rainminer Actors

If you need diagnostics instead of a scripted user flow, try URL to Console Logs to capture JavaScript errors from a page load.

Learn more

Read Stop writing Playwright for a five-step smoke test on rainminer.com for the problem this Actor solves, who it's for, and where it falls short.

Image Credit

Image credit: Playwright