AI-powered natural language end-to-end testing framework.

• Natural language E2E testing framework
• AI-powered test execution using Anthropic Claude API
• Built on Playwright
• GitHub integration with 2FA support
• Email validation with Mailosaur

If helpful, here's a short video!

npm install -D @antiwork/shortest
# or
pnpm add -D @antiwork/shortest
# or
yarn add -D @antiwork/shortest

Add .shortest/ to your .gitignore (where Shortest stores screenshots of each test run):

echo ".shortest/" >> .gitignore

Run tests:

npx shortest
# or
pnpm shortest
# or
yarn shortest

1. Determine your test entry and add your Anthropic API key in config file: shortest.config.ts

import type { ShortestConfig } from '@antiwork/shortest';

export default {
  headless: false,
  baseUrl: 'http://localhost:3000',
  testDir: 'app/__tests__',
  anthropicKey: process.env.ANTHROPIC_API_KEY
} satisfies ShortestConfig;

2. Write your test in your test directory: app/__tests__/login.test.ts

import { shortest } from '@antiwork/shortest'

shortest('Login to the app using email and password', { username: process.env.GITHUB_USERNAME, password: process.env.GITHUB_PASSWORD })

You can also use callback functions to add additional assertions and other logic. AI will execute the callback function after the test execution in browser is completed.

import { shortest } from '@antiwork/shortest';
import { db } from '@/lib/db/drizzle';
import { users } from '@/lib/db/schema';
import { eq } from 'drizzle-orm';

shortest('Login to the app using username and password', {
  username: process.env.USERNAME,
  password: process.env.PASSWORD
}).after(async ({ page }) => {
  // Get current user's clerk ID from the page
  const clerkId = await page.evaluate(() => {
    return window.localStorage.getItem('clerk-user');
  });

  if (!clerkId) {
    throw new Error('User not found in database');
  }

  // Query the database
  const [user] = await db
    .select()
    .from(users)
    .where(eq(users.clerkId, clerkId))
    .limit(1);

  expect(user).toBeDefined();
});

You can use lifecycle hooks to run code before and after the test.

import { shortest } from '@antiwork/shortest';

shortest.beforeAll(async ({ page }) => {
  await clerkSetup({
    frontendApiUrl: process.env.PLAYWRIGHT_TEST_BASE_URL ?? "http://localhost:3000",
  });
});

shortest.beforeEach(async ({ page }) => {
  await clerk.signIn({
    page,
    signInParams: {
      strategy: "email_code",
      identifier: "iffy+clerk_test@example.com"
    },
  });
});

shortest.afterEach(async ({ page }) => {
  await page.close();
});

shortest.afterAll(async ({ page }) => {
  await clerk.signOut({ page });
});

Shortest supports flexible test chaining patterns:

// Sequential test chain
shortest([
  "user can login with email and password",
  "user can modify their account-level refund policy"
])

// Reusable test flows
const loginAsLawyer = "login as lawyer with valid credentials"
const loginAsContractor = "login as contractor with valid credentials"
const allAppActions = [
  "send invoice to company",
  "view invoices"
]

// Combine flows with spread operator
shortest([loginAsLawyer, ...allAppActions])
shortest([loginAsContractor, ...allAppActions])

pnpm shortest                   # Run all tests
pnpm shortest login.test.ts     # Run specific test
pnpm shortest --headless        # Run in headless mode using cli

You can find example tests in the examples directory.

Shortest currently supports login using Github 2FA. For GitHub authentication tests:

1. Go to your repository settings
2. Navigate to "Password and Authentication"
3. Click on "Authenticator App"
4. Select "Use your authenticator app"
5. Click "Setup key" to obtain the OTP secret
6. Add the OTP secret to your .env.local file or use the Shortest CLI to add it
7. Enter the 2FA code displayed in your terminal into Github's Authenticator setup page to complete the process

shortest --github-code --secret=<OTP_SECRET>

Required in .env.local:

ANTHROPIC_API_KEY=your_api_key
GITHUB_TOTP_SECRET=your_secret  # Only for GitHub auth tests

You can run Shortest in your CI/CD pipeline by running tests in headless mode. Make sure to add your Anthropic API key to your CI/CD pipeline secrets.

This guide will help you set up the Shortest web app for local development.

• React >=19.0.0 (if using with Next.js 14+ or Server Actions)
• Next.js >=14.0.0 (if using Server Components/Actions)

1. Clone the repository:

   git clone https://github.com/anti-work/shortest.git
   cd shortest

2. Install dependencies:

   npm install -g pnpm
   pnpm install

Pull Vercel env vars:

pnpm i -g vercel
vercel link
vercel env pull

1. Run pnpm run setup to configure the environment variables.
2. The setup wizard will ask you for information. Refer to "Services Configuration" section below for more details.

pnpm drizzle-kit generate
pnpm db:migrate
pnpm db:seed # creates stripe products, currently unused

You'll need to set up the following services for local development. If you're not a Anti-Work Vercel team member, you'll need to either run the setup wizard pnpm run setup or manually configure each of these services and add the corresponding environment variables to your .env.local file:

Run the development server:

pnpm dev

Open http://localhost:3000 in your browser to see the app in action.

1. Make changes to the package source code in packages/shortest/

2. Test changes instantly during development (no build needed):

pnpm shortest:dev -h

3. To test the actual built package:

pnpm build:pkg
pnpm shortest --help

4. To test in another project:

# In Shortest package directory
cd packages/shortest
pnpm pack

# In your test project
npm install /path/to/antiwork-shortest-{version}.tgz.tgz
npx shortest -h