E2E Testing
Norrix runs end-to-end tests against your simulator and emulator builds on Norrix cloud runners. This guide takes a NativeScript app from no tests to an e2e suite that runs on your machine, on Norrix, and in CI. It ends with the Maestro equivalent and fixes for common failures.
Every flag used here is listed in the norrix test reference.
E2E tests are available to Enterprise organizations.
Choose a Runner
Norrix supports two runners. Both use the same build, the same norrix test command, the same dashboard, and the same JUnit output.
| Maestro | e2e | |
|---|---|---|
| Tests are written as | YAML flows | TypeScript tests |
| Steps | Exact taps and assertions | Plain-English agent steps (agent.act, agent.assert) mixed with exact locator checks |
| Model key | Not needed | Needed for agent steps, from your own model provider |
| Default directory | .maestro/ | e2e/ (or the current directory) |
| Selected by | Default, or --runner maestro | --runner e2e, e2e-only options, or a detected e2e.config.ts |
Maestro stays the default. Existing Maestro flows and norrix test commands keep working unchanged. e2e is an additional option: pick it when you want tests that describe intent (“sign in”, “open the first order”) instead of exact coordinates and element names, and that keep working when the UI shifts. A project can hold both; see Choosing a Runner for how norrix test picks one.
Prerequisites
-
An Enterprise organization, and the Norrix CLI signed in (
norrix sign-in) or aNORRIX_API_KEYin CI. -
A successful build that installs on a virtual device. A device or store build will not work:
# iOS Simulator .app norrix build ios debug --simulator # Android emulator APK norrix build android debug --emulator -
To run tests locally: macOS with Xcode and an iOS simulator, and Node.js 22.22.3 or later on Node 22, or 24.8.0 or later, which the
e2epackage requires. -
An API key for the model that runs agent steps. This guide uses Anthropic (
ANTHROPIC_API_KEY); the e2e Models page lists the other providers.
Set Up an e2e Suite
The steps below build this layout in your app’s directory:
my-app/
├── app/ # your NativeScript app
├── nativescript.config.ts
└── e2e/
├── package.json # e2e, @e2e-dev/mobile, ai, @ai-sdk/anthropic
├── package-lock.json
├── e2e.config.ts
└── tests/
└── smoke.e2e.ts1. Create the Suite Directory
Give the suite its own package.json. Norrix installs only this directory’s dependencies on the runner, not your app’s:
mkdir e2e
cd e2e
npm init -y
npm install -D e2e @e2e-dev/mobile ai @ai-sdk/anthropicCommit the lockfile. Norrix installs with the package manager your packageManager field or lockfile names (npm, pnpm, Yarn, or Bun). Yarn Plug’n’Play is not supported; use nodeLinker: node-modules.
2. Write the Config
import type { E2EConfig } from 'e2e';
import { mobile } from '@e2e-dev/mobile';
import { anthropic } from '@ai-sdk/anthropic';
export default {
agents: {
default: { model: anthropic('claude-sonnet-5-5') },
},
targets: [
{
name: 'ios',
engine: mobile({ platform: 'ios' }),
app: { bundleId: 'org.example.myapp' },
},
],
workers: 1,
} satisfies E2EConfig;- Set
bundleIdto theidin yournativescript.config.ts. - Leave
deviceout ofmobile(). Locally the engine uses the simulator you booted; on Norrix it uses the simulator Norrix booted. - To run the same tests against an Android emulator build, add a target with
mobile({ platform: 'android' }), as in the config example in the CLI reference.
3. Write a First Test
Give the element your test checks a testID. On iOS, NativeScript sets it as the view’s accessibility identifier, which getByTestId reads:
<Label testID="item-title" text="{{ name }}" />Then write the test:
import { test } from '@e2e-dev/mobile';
import { expect } from 'e2e';
test('opens an item', { tags: ['smoke'] }, async ({ agent, app, screen }) => {
await app.open();
await agent.act('open the first item in the list');
await agent.assert('a details screen for the selected item is showing');
await expect(screen.getByTestId('item-title')).toBeVisible();
});app.open()launches the app fresh. Without it, a test starts wherever the previous test left the app.agent.actgives the agent one goal. It works out the taps and typing from what is on screen.agent.assertasks the model to judge the current screen.expect(screen.getByTestId(...))is an exact check with no model call.screen.getByText('...')andscreen.getByRole('button', '...')work the same way.- The
smoketag lets you run this test alone with--tag smoke.
The e2e Core concepts page covers how to word goals and assertions.
4. Run It Locally
Install your app on a booted simulator, then run the suite from e2e/:
# From your app directory: build, install, and launch on a simulator.
# Stop it once the app has launched; the app stays installed.
ns run ios
cd e2e
export ANTHROPIC_API_KEY=sk-ant-...
npx e2e run --target iosThe mobile engine installs nothing on its own, so the app must already be on the simulator. If the run cannot reach the simulator, npx agent-device doctor checks your Xcode and simulator setup.
The run prints each test’s result and writes its report to e2e/.e2e/. Keep node_modules/ and .e2e/ out of git; Norrix never uploads them, except recordings in .e2e/cache/ (see Replay Cache).
5. Store the Model Key in Norrix
Agent steps on Norrix use your key, stored as a Norrix environment variable:
norrix env set ANTHROPIC_API_KEY "$ANTHROPIC_API_KEY"Add --project my-app to scope the key to one project. Every Norrix environment variable reaches the suite’s processes. Model Credentials and Secrets covers the details.
6. Run on Norrix
From your app directory:
norrix build ios debug --simulator
norrix test ios --runner e2e --waitOnce the build succeeds, norrix test lists your testable builds to pick from. It finds the suite in ./e2e or the current directory; pass --suite-dir <path> when it lives elsewhere, for example in an Nx workspace. --runner e2e is optional when the project has no .maestro/ directory.
norrix test uploads the suite directory without node_modules, .git, platforms/, .env files, build artifacts, or run output (the replay cache in .e2e/cache/ is kept). The Norrix runner then:
- Installs the suite’s dependencies.
- Boots an iPhone simulator and installs your build on it. Pick the simulator with
--device-modeland--device-os; see Device Pinning. - Runs
e2e runfor every target on the build’s platform, or the one you pass with--target.
7. Read the Results
With --wait, the CLI polls until the run finishes, then prints the passed, failed, and flaky counts, model tokens, estimated cost, and replay cache hits. It exits non-zero when the run fails, is cancelled, or times out (60 minutes by default; change it with --timeout-minutes).
- Dashboard: open the run under
/testand choose View Report for the report and a video of every failed attempt. - CLI:
norrix test-status <e2eTestId>shows a run’s status and results later. - JUnit:
--junit-output ./norrix-e2e-junitdownloads the JUnit XML into that directory for your CI to publish. It implies--wait.
When a run fails, its message names the failed tests. When it never reaches its tests, the message says why.
Sign In with a Test Account
Declare the account in the config, with the password read from the environment:
export default {
// agents, targets, and workers as above
credentials: {
tester: {
username: 'tester@example.com',
password: process.env.TESTER_PASSWORD ?? '',
},
},
} satisfies E2EConfig;Pass the credential to an agent step. The model sees only the password’s name, and the runner types the value:
import { test } from '@e2e-dev/mobile';
import { credentials } from 'e2e';
test('signs in with the test account', async ({ agent, app }) => {
const tester = credentials.user('tester');
await app.open();
await agent.act('Sign in', {
params: { username: tester.username, password: tester.password },
});
await agent.assert('the home screen is showing and the user is signed in');
});Store the password in Norrix the same way as the model key:
norrix env set TESTER_PASSWORD "$TESTER_PASSWORD"- The config fails to load while
TESTER_PASSWORDis unset or shorter than 6 characters, so set it locally too before any run. - An
E2E_USER_TESTER_PASSWORDvariable overrides the config value, andE2E_SECRET_<NAME>fills asecretsentry. See e2e’s Signing in . - Saved sign-in sessions (
test.setup) are not available on devices. Sign in within each test that needs it.
Select Tests
| Flag | Runs | Example |
|---|---|---|
--target <name> | One target from e2e.config.ts. Default: every target on the build’s platform | --target ios |
--tests <files> | Comma-delimited test files, relative to the suite directory or your working directory | --tests tests/login.e2e.ts |
--grep <pattern> | Tests whose title matches the pattern | --grep "signs in" |
--tag <tags> | Comma-delimited tags; tests carrying any of them run | --tag smoke,checkout |
The flags combine. Each file in --tests must exist in the suite directory before upload. A --target must exist in the config and match the build’s platform. These flags apply to --runner e2e only and cannot be combined with --flows-dir or --flows.
norrix test ios --suite-dir e2e --tag smoke --waitRun in CI
This GitHub Actions workflow builds a simulator app, waits for the build, runs the smoke tests, and publishes JUnit. Create the API key and the NORRIX_API_KEY secret as described in CI/CD Integration. The CI job does not need the suite’s dependencies; Norrix installs them on its runner.
name: E2E
on:
pull_request:
workflow_dispatch:
jobs:
e2e-ios:
runs-on: ubuntu-latest
env:
NORRIX_API_KEY: ${{ secrets.NORRIX_API_KEY }}
NORRIX_ORG_ID: ${{ vars.NORRIX_ORG_ID }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Norrix CLI
run: npm install -g @norrix/cli
- name: Build simulator app
id: build
run: |
norrix build ios debug --simulator --org "$NORRIX_ORG_ID" -n | tee build.log
BUILD_ID=$(sed -n 's/.*Build ID: //p' build.log | head -n 1)
test -n "$BUILD_ID"
echo "build_id=$BUILD_ID" >> "$GITHUB_OUTPUT"
- name: Wait for the build
env:
BUILD_ID: ${{ steps.build.outputs.build_id }}
run: |
for attempt in $(seq 1 90); do
STATUS=$(norrix build-status "$BUILD_ID" --org "$NORRIX_ORG_ID" | sed -n 's/^Status: //p' | head -n 1)
echo "Build $BUILD_ID: $STATUS"
case "$STATUS" in
success) exit 0 ;;
failed|cancelled) exit 1 ;;
esac
sleep 30
done
exit 1
- name: Run e2e tests
run: |
norrix test ios \
--build "${{ steps.build.outputs.build_id }}" \
--runner e2e \
--suite-dir e2e \
--tag smoke \
--wait \
--timeout-minutes 50 \
--junit-output ./norrix-e2e-junit \
--org "$NORRIX_ORG_ID" \
-n
- name: Upload JUnit results
if: always()
uses: actions/upload-artifact@v4
with:
name: norrix-e2e-junit
path: norrix-e2e-junit/*.xml
if-no-files-found: ignorenorrix buildreturns once the build is queued, so the job pollsnorrix build-statusuntil it succeeds.norrix testrefuses a build that has not finished.-ndisables prompts.norrix test -nrequires--build.- With
--junit-output, the CLI still downloads JUnit XML for a failed run when the run uploaded it, so the failing tests show up in CI.
Maestro
Maestro flows live in .maestro/ by default and need no model key. A minimal flow:
appId: org.example.myapp
---
- launchApp
- assertVisible: 'Items'Run every flow, or a selection from another directory:
norrix test ios --wait
norrix test ios \
--flows-dir apps/my-app-e2e/src/flows \
--flows 01-launch.yaml,02-login.yaml \
--waitNorrix environment variables whose names start with MAESTRO_ are passed to flows, so ${MAESTRO_USERNAME} works in a flow after norrix env set MAESTRO_USERNAME .... The CLI reference covers test-time environment variables, device pinning, and a CI-gated Maestro run.
If a project has both .maestro/ and an e2e suite, norrix test asks which to run. With -n it stops instead, so pass --runner maestro or --runner e2e in CI.
Troubleshooting
Agent steps fail with a model error
The run log shows No model credential found in the org's Norrix env vars. Set your provider’s key with norrix env set, using the variable name the provider reads (for example ANTHROPIC_API_KEY). The notice checks a fixed list of common names; a variable your provider reads under another name still reaches the suite. Tests without agent steps run without a key.
Unknown target
The run stops before any test with Target '<name>' is not in e2e.config.ts, Target '<name>' runs on android, but build is ios, or e2e.config.ts has no ios target. Pass a --target from your config that matches the build’s platform, or add a target with mobile({ platform: 'ios' }) (or 'android').
The suite’s dependencies do not include the e2e package
Add e2e and @e2e-dev/mobile to the suite’s own package.json. If the suite uses Yarn Plug’n’Play, switch to nodeLinker: node-modules in .yarnrc.yml; Plug’n’Play installs are not supported.
The packaged e2e suite is over 100 MB
norrix test refuses suite archives over 100 MB. This usually means the suite directory holds app sources, builds, or large fixtures. Move the suite into its own directory and pass --suite-dir.
No available iPhone simulator matched the requested model/runtime
--device-model or --device-os named a simulator the runner does not have, and Norrix fails rather than using a different one. Change or drop those flags. A device set in mobile() must also exist on the runner; leave it out to use the simulator Norrix booted.
The config fails to load on Norrix but works locally
.env files are never uploaded, and a variable you only export in your shell does not exist on the runner. Store every variable the config reads with norrix env set. If the config calls process.loadEnvFile('.env'), guard it with existsSync from node:fs.
Other setup errors
| Message | Fix |
|---|---|
No e2e.config.ts or e2e.config.mts in <dir> | Pass --suite-dir pointing at the directory that holds the config |
<dir> has no package.json | Create the suite’s own package.json (step 1) |
Found both a .maestro directory and an e2e suite | Pass --runner maestro or --runner e2e |
Build with ID <id> is not ready for E2E testing | Wait until norrix build-status reports success |
No testable builds found | Build with norrix build ios --simulator or norrix build android --emulator |