Skip to Content
Platform GuidesE2E Testing

E2E Testing

Norrix runs end-to-end tests against your simulator and emulator builds on Norrix cloud runners. This guide takes a NativeScript app from no tests to an e2e suite that runs on your machine, on Norrix, and in CI. It ends with the Maestro equivalent and fixes for common failures.

Every flag used here is listed in the norrix test reference.

E2E tests are available to Enterprise organizations.

Choose a Runner

Norrix supports two runners. Both use the same build, the same norrix test command, the same dashboard, and the same JUnit output.

Maestroe2e
Tests are written asYAML flowsTypeScript tests
StepsExact taps and assertionsPlain-English agent steps (agent.act, agent.assert) mixed with exact locator checks
Model keyNot neededNeeded for agent steps, from your own model provider
Default directory.maestro/e2e/ (or the current directory)
Selected byDefault, or --runner maestro--runner e2e, e2e-only options, or a detected e2e.config.ts

Maestro stays the default. Existing Maestro flows and norrix test commands keep working unchanged. e2e  is an additional option: pick it when you want tests that describe intent (“sign in”, “open the first order”) instead of exact coordinates and element names, and that keep working when the UI shifts. A project can hold both; see Choosing a Runner for how norrix test picks one.

Prerequisites

  • An Enterprise organization, and the Norrix CLI signed in (norrix sign-in) or a NORRIX_API_KEY in CI.

  • A successful build that installs on a virtual device. A device or store build will not work:

    # iOS Simulator .app norrix build ios debug --simulator # Android emulator APK norrix build android debug --emulator
  • To run tests locally: macOS with Xcode and an iOS simulator, and Node.js 22.22.3 or later on Node 22, or 24.8.0 or later, which the e2e package requires.

  • An API key for the model that runs agent steps. This guide uses Anthropic (ANTHROPIC_API_KEY); the e2e Models  page lists the other providers.

Set Up an e2e Suite

The steps below build this layout in your app’s directory:

my-app/ ├── app/ # your NativeScript app ├── nativescript.config.ts └── e2e/ ├── package.json # e2e, @e2e-dev/mobile, ai, @ai-sdk/anthropic ├── package-lock.json ├── e2e.config.ts └── tests/ └── smoke.e2e.ts

1. Create the Suite Directory

Give the suite its own package.json. Norrix installs only this directory’s dependencies on the runner, not your app’s:

mkdir e2e cd e2e npm init -y npm install -D e2e @e2e-dev/mobile ai @ai-sdk/anthropic

Commit the lockfile. Norrix installs with the package manager your packageManager field or lockfile names (npm, pnpm, Yarn, or Bun). Yarn Plug’n’Play is not supported; use nodeLinker: node-modules.

2. Write the Config

e2e/e2e.config.ts
import type { E2EConfig } from 'e2e'; import { mobile } from '@e2e-dev/mobile'; import { anthropic } from '@ai-sdk/anthropic'; export default { agents: { default: { model: anthropic('claude-sonnet-5-5') }, }, targets: [ { name: 'ios', engine: mobile({ platform: 'ios' }), app: { bundleId: 'org.example.myapp' }, }, ], workers: 1, } satisfies E2EConfig;
  • Set bundleId to the id in your nativescript.config.ts.
  • Leave device out of mobile(). Locally the engine uses the simulator you booted; on Norrix it uses the simulator Norrix booted.
  • To run the same tests against an Android emulator build, add a target with mobile({ platform: 'android' }), as in the config example in the CLI reference.

3. Write a First Test

Give the element your test checks a testID. On iOS, NativeScript sets it as the view’s accessibility identifier, which getByTestId reads:

app/item-detail-page.xml
<Label testID="item-title" text="{{ name }}" />

Then write the test:

e2e/tests/smoke.e2e.ts
import { test } from '@e2e-dev/mobile'; import { expect } from 'e2e'; test('opens an item', { tags: ['smoke'] }, async ({ agent, app, screen }) => { await app.open(); await agent.act('open the first item in the list'); await agent.assert('a details screen for the selected item is showing'); await expect(screen.getByTestId('item-title')).toBeVisible(); });
  • app.open() launches the app fresh. Without it, a test starts wherever the previous test left the app.
  • agent.act gives the agent one goal. It works out the taps and typing from what is on screen.
  • agent.assert asks the model to judge the current screen.
  • expect(screen.getByTestId(...)) is an exact check with no model call. screen.getByText('...') and screen.getByRole('button', '...') work the same way.
  • The smoke tag lets you run this test alone with --tag smoke.

The e2e Core concepts  page covers how to word goals and assertions.

4. Run It Locally

Install your app on a booted simulator, then run the suite from e2e/:

# From your app directory: build, install, and launch on a simulator. # Stop it once the app has launched; the app stays installed. ns run ios cd e2e export ANTHROPIC_API_KEY=sk-ant-... npx e2e run --target ios

The mobile engine installs nothing on its own, so the app must already be on the simulator. If the run cannot reach the simulator, npx agent-device doctor checks your Xcode and simulator setup.

The run prints each test’s result and writes its report to e2e/.e2e/. Keep node_modules/ and .e2e/ out of git; Norrix never uploads them, except recordings in .e2e/cache/ (see Replay Cache).

5. Store the Model Key in Norrix

Agent steps on Norrix use your key, stored as a Norrix environment variable:

norrix env set ANTHROPIC_API_KEY "$ANTHROPIC_API_KEY"

Add --project my-app to scope the key to one project. Every Norrix environment variable reaches the suite’s processes. Model Credentials and Secrets covers the details.

6. Run on Norrix

From your app directory:

norrix build ios debug --simulator norrix test ios --runner e2e --wait

Once the build succeeds, norrix test lists your testable builds to pick from. It finds the suite in ./e2e or the current directory; pass --suite-dir <path> when it lives elsewhere, for example in an Nx workspace. --runner e2e is optional when the project has no .maestro/ directory.

norrix test uploads the suite directory without node_modules, .git, platforms/, .env files, build artifacts, or run output (the replay cache in .e2e/cache/ is kept). The Norrix runner then:

  1. Installs the suite’s dependencies.
  2. Boots an iPhone simulator and installs your build on it. Pick the simulator with --device-model and --device-os; see Device Pinning.
  3. Runs e2e run for every target on the build’s platform, or the one you pass with --target.

7. Read the Results

With --wait, the CLI polls until the run finishes, then prints the passed, failed, and flaky counts, model tokens, estimated cost, and replay cache hits. It exits non-zero when the run fails, is cancelled, or times out (60 minutes by default; change it with --timeout-minutes).

  • Dashboard: open the run under /test and choose View Report for the report and a video of every failed attempt.
  • CLI: norrix test-status <e2eTestId> shows a run’s status and results later.
  • JUnit: --junit-output ./norrix-e2e-junit downloads the JUnit XML into that directory for your CI to publish. It implies --wait.

When a run fails, its message names the failed tests. When it never reaches its tests, the message says why.

Sign In with a Test Account

Declare the account in the config, with the password read from the environment:

e2e/e2e.config.ts
export default { // agents, targets, and workers as above credentials: { tester: { username: 'tester@example.com', password: process.env.TESTER_PASSWORD ?? '', }, }, } satisfies E2EConfig;

Pass the credential to an agent step. The model sees only the password’s name, and the runner types the value:

e2e/tests/login.e2e.ts
import { test } from '@e2e-dev/mobile'; import { credentials } from 'e2e'; test('signs in with the test account', async ({ agent, app }) => { const tester = credentials.user('tester'); await app.open(); await agent.act('Sign in', { params: { username: tester.username, password: tester.password }, }); await agent.assert('the home screen is showing and the user is signed in'); });

Store the password in Norrix the same way as the model key:

norrix env set TESTER_PASSWORD "$TESTER_PASSWORD"
  • The config fails to load while TESTER_PASSWORD is unset or shorter than 6 characters, so set it locally too before any run.
  • An E2E_USER_TESTER_PASSWORD variable overrides the config value, and E2E_SECRET_<NAME> fills a secrets entry. See e2e’s Signing in .
  • Saved sign-in sessions (test.setup) are not available on devices. Sign in within each test that needs it.

Select Tests

FlagRunsExample
--target <name>One target from e2e.config.ts. Default: every target on the build’s platform--target ios
--tests <files>Comma-delimited test files, relative to the suite directory or your working directory--tests tests/login.e2e.ts
--grep <pattern>Tests whose title matches the pattern--grep "signs in"
--tag <tags>Comma-delimited tags; tests carrying any of them run--tag smoke,checkout

The flags combine. Each file in --tests must exist in the suite directory before upload. A --target must exist in the config and match the build’s platform. These flags apply to --runner e2e only and cannot be combined with --flows-dir or --flows.

norrix test ios --suite-dir e2e --tag smoke --wait

Run in CI

This GitHub Actions workflow builds a simulator app, waits for the build, runs the smoke tests, and publishes JUnit. Create the API key and the NORRIX_API_KEY secret as described in CI/CD Integration. The CI job does not need the suite’s dependencies; Norrix installs them on its runner.

.github/workflows/e2e.yml
name: E2E on: pull_request: workflow_dispatch: jobs: e2e-ios: runs-on: ubuntu-latest env: NORRIX_API_KEY: ${{ secrets.NORRIX_API_KEY }} NORRIX_ORG_ID: ${{ vars.NORRIX_ORG_ID }} steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '22' cache: 'npm' - name: Install dependencies run: npm ci - name: Install Norrix CLI run: npm install -g @norrix/cli - name: Build simulator app id: build run: | norrix build ios debug --simulator --org "$NORRIX_ORG_ID" -n | tee build.log BUILD_ID=$(sed -n 's/.*Build ID: //p' build.log | head -n 1) test -n "$BUILD_ID" echo "build_id=$BUILD_ID" >> "$GITHUB_OUTPUT" - name: Wait for the build env: BUILD_ID: ${{ steps.build.outputs.build_id }} run: | for attempt in $(seq 1 90); do STATUS=$(norrix build-status "$BUILD_ID" --org "$NORRIX_ORG_ID" | sed -n 's/^Status: //p' | head -n 1) echo "Build $BUILD_ID: $STATUS" case "$STATUS" in success) exit 0 ;; failed|cancelled) exit 1 ;; esac sleep 30 done exit 1 - name: Run e2e tests run: | norrix test ios \ --build "${{ steps.build.outputs.build_id }}" \ --runner e2e \ --suite-dir e2e \ --tag smoke \ --wait \ --timeout-minutes 50 \ --junit-output ./norrix-e2e-junit \ --org "$NORRIX_ORG_ID" \ -n - name: Upload JUnit results if: always() uses: actions/upload-artifact@v4 with: name: norrix-e2e-junit path: norrix-e2e-junit/*.xml if-no-files-found: ignore
  • norrix build returns once the build is queued, so the job polls norrix build-status until it succeeds. norrix test refuses a build that has not finished.
  • -n disables prompts. norrix test -n requires --build.
  • With --junit-output, the CLI still downloads JUnit XML for a failed run when the run uploaded it, so the failing tests show up in CI.

Maestro

Maestro flows live in .maestro/ by default and need no model key. A minimal flow:

.maestro/01-launch.yaml
appId: org.example.myapp --- - launchApp - assertVisible: 'Items'

Run every flow, or a selection from another directory:

norrix test ios --wait norrix test ios \ --flows-dir apps/my-app-e2e/src/flows \ --flows 01-launch.yaml,02-login.yaml \ --wait

Norrix environment variables whose names start with MAESTRO_ are passed to flows, so ${MAESTRO_USERNAME} works in a flow after norrix env set MAESTRO_USERNAME .... The CLI reference covers test-time environment variables, device pinning, and a CI-gated Maestro run.

If a project has both .maestro/ and an e2e suite, norrix test asks which to run. With -n it stops instead, so pass --runner maestro or --runner e2e in CI.

Troubleshooting

Agent steps fail with a model error

The run log shows No model credential found in the org's Norrix env vars. Set your provider’s key with norrix env set, using the variable name the provider reads (for example ANTHROPIC_API_KEY). The notice checks a fixed list of common names; a variable your provider reads under another name still reaches the suite. Tests without agent steps run without a key.

Unknown target

The run stops before any test with Target '<name>' is not in e2e.config.ts, Target '<name>' runs on android, but build is ios, or e2e.config.ts has no ios target. Pass a --target from your config that matches the build’s platform, or add a target with mobile({ platform: 'ios' }) (or 'android').

The suite’s dependencies do not include the e2e package

Add e2e and @e2e-dev/mobile to the suite’s own package.json. If the suite uses Yarn Plug’n’Play, switch to nodeLinker: node-modules in .yarnrc.yml; Plug’n’Play installs are not supported.

The packaged e2e suite is over 100 MB

norrix test refuses suite archives over 100 MB. This usually means the suite directory holds app sources, builds, or large fixtures. Move the suite into its own directory and pass --suite-dir.

No available iPhone simulator matched the requested model/runtime

--device-model or --device-os named a simulator the runner does not have, and Norrix fails rather than using a different one. Change or drop those flags. A device set in mobile() must also exist on the runner; leave it out to use the simulator Norrix booted.

The config fails to load on Norrix but works locally

.env files are never uploaded, and a variable you only export in your shell does not exist on the runner. Store every variable the config reads with norrix env set. If the config calls process.loadEnvFile('.env'), guard it with existsSync from node:fs.

Other setup errors

MessageFix
No e2e.config.ts or e2e.config.mts in <dir>Pass --suite-dir pointing at the directory that holds the config
<dir> has no package.jsonCreate the suite’s own package.json (step 1)
Found both a .maestro directory and an e2e suitePass --runner maestro or --runner e2e
Build with ID <id> is not ready for E2E testingWait until norrix build-status reports success
No testable builds foundBuild with norrix build ios --simulator or norrix build android --emulator