OTA Crash Recovery

Norrix includes built-in crash recovery that automatically detects when an OTA update causes a launch crash and rolls back to the original store binary. This protects your users from being stuck in a crash loop with no way out.

The Problem

If a bad OTA update is installed that crashes the app on launch, the user is stuck:

1. App launches and loads the bad OTA bundle
2. App crashes immediately
3. User relaunches — same crash
4. The only escape is uninstalling and reinstalling from the app store

Norrix solves this automatically with a Launch Sentinel that detects the crash and reverts to the original binary on the next launch attempt.

How It Works

Launch Sentinel Pattern

The crash recovery system uses a simple counter (the “sentinel”) coordinated between the native layer and the JavaScript layer:

App Launch
    │
    ▼
┌──────────────────────────────┐
│  NATIVE LAYER (before JS)    │
│                              │
│  1. Read sentinel counter    │
│  2. If sentinel >= 1:        │
│     → Crash detected!        │
│     → Roll back to binary    │
│     → Return                 │
│  3. Increment sentinel to 1  │
│  4. Load OTA bundle          │
└──────────────────────────────┘
    │
    ▼
┌──────────────────────────────┐
│  JAVASCRIPT LAYER            │
│                              │
│  5. JS initializes           │
│  6. SDK constructor runs     │
│  7. Reset sentinel to 0      │
│     → Launch was successful  │
└──────────────────────────────┘

Normal OTA Launch

sentinel: 0 → 1 (native)  →  JS init succeeds  →  sentinel: 1 → 0 (JS)

The sentinel goes from 0 to 1 before loading the OTA, then back to 0 once JavaScript initializes successfully. Everything is fine.

OTA Launch Crash

sentinel: 0 → 1 (native)  →  JS CRASHES  →  sentinel stays at 1

The native layer sets the sentinel to 1, but the app crashes before JavaScript can reset it to 0.

Recovery on Next Launch

sentinel = 1  →  Native detects crash  →  Rolls back to binary  →  sentinel: 0

On the next launch, the native layer sees the sentinel is still 1, meaning the previous launch crashed. It skips the OTA bundle and boots from the original store binary instead.

What Triggers Recovery

Recovery is triggered only by launch crashes — crashes that occur before the SDK constructor finishes initializing. This is an important distinction:

This means:

• A syntax error in your main bundle → recovered automatically
• A missing module that prevents startup → recovered automatically
• A null reference that crashes 5 minutes into normal use → no rollback (correct behavior — this is a code bug, not an OTA issue)

Why This Distinction Matters

Runtime crashes during normal app use are developer bugs, not OTA delivery failures. Rolling back for every crash would make OTA updates unreliable and unpredictable. By only detecting launch crashes, the system targets the specific scenario where an OTA update makes the app completely unusable.

What Happens During Recovery

When the native layer detects a crash (sentinel >= 1), it performs these steps:

iOS

1. Logs: "Norrix: OTA crash loop detected, rolling back to binary bundle"
2. Saves the bad OTA’s version, build number, and fingerprint for telemetry
3. Clears all OTA state (fingerprint, version, build keys)
4. Deletes the bad OTA folder from disk
5. Resets the sentinel counter to 0
6. Boots from the original store binary

Android

1. Logs: "Norrix: OTA crash loop detected, rolling back to binary bundle"
2. Restores the /app_backup/ directory to /app/ (reverses the OTA file copy)
3. Saves the bad OTA’s version, build number, and fingerprint for telemetry
4. Clears all OTA state
5. Deletes the bad OTA folder from disk
6. Resets the sentinel counter to 0
7. Boots from the restored original binary

After recovery, the app launches normally with the original store binary. The SDK constructor detects the recovery flags and reports the incident to your OTA analytics.

OTA Analytics Reporting

When crash recovery occurs, the SDK automatically reports an ota_crash_recovery telemetry event that includes:

• OTA version that caused the crash
• OTA build number that caused the crash
• Fingerprint of the rolled-back OTA
• Platform (iOS or Android)
• Device information
• Timestamp

This data appears in your OTA Analytics dashboard.

Viewing Crash Recovery Data

1. Navigate to Dashboard → OTA Analytics (accessible from the Updates section)
2. The Rollbacks & Crash Recovery card shows the total count of crash recoveries
3. The Affected OTA Versions section shows which specific OTA versions caused crashes, broken down by:
   • OTA version and build number
   • Platform
   • Number of affected devices
   • When it last occurred

This information helps you quickly identify which OTA update is causing problems and how many users were affected.

Rollbacks vs Crash Recoveries

The analytics dashboard tracks two separate metrics under Rollbacks & Crash Recovery:

In short: Rollbacks are version-mismatch driven (the OTA became outdated), while Crash Recoveries are crash driven (the OTA was broken). Both result in the app returning to the store binary, but they represent different failure modes.

Recommended Course of Action

When you see crash recoveries in your OTA analytics, follow these steps:

1. Deactivate the Bad OTA

Immediately deactivate the problematic OTA update from the dashboard to prevent other users from receiving it:

1. Go to Dashboard → Updates
2. Find the OTA update matching the version/build shown in crash recovery analytics
3. Click Deactivate to stop it from being served to devices

Important: Users who already downloaded the bad OTA will recover automatically on their next launch. Deactivating prevents new devices from downloading it.

2. Investigate and Fix the Issue

Common causes of launch crashes in OTA updates:

• Syntax errors in bundled JavaScript
• Missing modules or broken imports
• Incompatible code that relies on native APIs not present in the binary
• Build configuration issues (e.g., wrong environment variables baked into the bundle)

To reproduce locally:

# Build the same bundle that was published
ns build ios --release  # or android
 
# Test the bundle on a device or simulator
ns run ios --release

3. Publish a Fixed OTA Update

Once you’ve identified and fixed the issue:

# Publish a corrected OTA with a bumped build number
norrix update ios -V 1.0.1 -B 44

The new OTA will be served to all devices, including those that recovered from the bad update. Since the crash recovery cleared the bad OTA’s state, devices will check for updates normally and receive the fixed version.

Summary Workflow

Crash detected in analytics
    │
    ▼
Deactivate bad OTA in dashboard
    │
    ▼
Fix the code issue locally
    │
    ▼
Test thoroughly
    │
    ▼
Publish new OTA with bumped build number
    │
    ▼
Verify analytics show no new crash recoveries

Edge Cases and Guarantees

False Positives

A false positive (rollback when the OTA was actually fine) can only occur if:

• The app is force-killed during the ~100-500ms window between the native sentinel increment and the JS sentinel reset

In practice, this is extremely unlikely. If it does happen, the only consequence is the OTA re-downloads on the next update check — no data is lost.

New Binary Installs

When a user installs a new version of your app from the App Store or Play Store, the crash recovery state is automatically cleared. The new binary always boots clean, regardless of any previous OTA state.

Multiple Consecutive Bad OTAs

If a second bad OTA is published after the first was rolled back, the same recovery process applies. Each OTA gets one chance — if it crashes on launch, it’s rolled back and the recovery is reported.

Related

• OTA Compatibility — What can and cannot be updated via OTA
• SDK Usage — Reset OTA Updates — Manually reset OTA state on a device
• Rolling Back an OTA Update — How to roll back all users to older code
• Troubleshooting — OTA Crash Recovery — Diagnosing crash recovery issues