RealityRogue | Testing ARKit Apps Without Waving a Phone Around: Replay Data, Fakes, and CI

Every AR project we join has the same untested layer. The team has unit tests on the networking client and the view models, and then there is a 2,000-line ARSessionManager that nobody touches because "you need a device and a room to test it." That is where the shipping bugs live: the anchor that drifts after relocalization, the model that spawns at the origin instead of on the table, the tracking-state handler that leaves the coaching overlay on screen forever.

You cannot unit test computer vision. You can test almost everything you wrote around it. This tutorial covers the four-layer strategy we put on client ARKit projects — a protocol seam around the session, pure functions for the math, deterministic replay for manual QA, and a realistic view of what belongs on CI.

Requirements: Xcode 16 or later, an ARKit-capable device for the replay and device-test sections, and a project using RealityKit (RealityView or ARView). Nothing here needs a third-party framework.

1. Put a seam between your app and ARKit

ARSession, ARFrame, and ARAnchor are hard to construct in a test — ARFrame in particular has no public initializer. So stop trying. Define the small surface your app actually consumes and let ARKit be one implementation of it.

import simd

/// What the rest of the app needs to know about the AR session.
struct ARWorldSnapshot {
    var cameraTransform: simd_float4x4
    var trackingState: TrackingState
    var horizontalPlanes: [PlaneInfo]
    var isRelocalizing: Bool
}

enum TrackingState: Equatable {
    case notAvailable
    case limited(Reason)
    case normal

    enum Reason: Equatable { case initializing, excessiveMotion, insufficientFeatures, relocalizing }
}

struct PlaneInfo: Equatable, Identifiable {
    let id: UUID
    var transform: simd_float4x4
    var extent: SIMD2<Float>
    var classification: String
}

protocol ARWorldSource: AnyObject {
    var snapshots: AsyncStream<ARWorldSnapshot> { get }
    func start()
    func pause()
    func raycastHorizontal(from screenPoint: CGPoint) -> simd_float4x4?
}

The ARKit implementation is a thin adapter: an ARSessionDelegate that maps ARFrame and ARPlaneAnchor into those structs and yields them into the stream. It contains no decisions, so it needs no tests beyond "does it compile and run on device."

The fake is trivial, and it is where every interesting test happens:

final class FakeWorldSource: ARWorldSource {
    private let continuation: AsyncStream<ARWorldSnapshot>.Continuation
    let snapshots: AsyncStream<ARWorldSnapshot>
    var raycastResult: simd_float4x4?
    private(set) var startCount = 0

    init() {
        var c: AsyncStream<ARWorldSnapshot>.Continuation!
        snapshots = AsyncStream { c = $0 }
        continuation = c
    }

    func start() { startCount += 1 }
    func pause() {}
    func raycastHorizontal(from screenPoint: CGPoint) -> simd_float4x4? { raycastResult }

    func emit(_ snapshot: ARWorldSnapshot) { continuation.yield(snapshot) }
}

Now your placement view model, your coaching-overlay logic, and your "can the user tap Place yet?" gate are all ordinary testable objects. Roughly 70% of the AR-specific code on a typical project moves above this line the first time you draw it.

2. Unit test the math, because the math is where the bugs are

Anchor and transform errors are the single most common category of AR defect, and they are pure arithmetic. Extract them from your renderer into free functions.

/// Places an entity flat on a detected plane, facing the camera, without tilting it.
func placementTransform(on plane: simd_float4x4,
                        at point: SIMD3<Float>,
                        facing camera: simd_float4x4) -> simd_float4x4 {
    let cameraPos = camera.columns.3.xyz
    var forward = cameraPos - point
    forward.y = 0
    guard length(forward) > 1e-4 else { return simd_float4x4(translation: point) }
    let yaw = atan2(forward.x, forward.z)
    return simd_float4x4(translation: point) * simd_float4x4(yaw: yaw)
}

And the test, with Swift Testing:

import Testing
import simd

@Test func placedEntityFacesCameraAndStaysLevel() {
    let camera = simd_float4x4(translation: SIMD3(0, 1.4, 1))   // 1 m in front, eye height
    let plane = simd_float4x4(translation: SIMD3(0, 0, 0))
    let m = placementTransform(on: plane, at: SIMD3(0, 0, 0), facing: camera)

    // Yaw only: the model's up axis must still be world up.
    let up = (m * SIMD4<Float>(0, 1, 0, 0)).xyz
    #expect(abs(up.y - 1) < 1e-5)

    // And it must face the camera, not away from it.
    let modelForward = (m * SIMD4<Float>(0, 0, 1, 0)).xyz
    #expect(dot(normalize(modelForward), SIMD3<Float>(0, 0, 1)) > 0.99)
}

Write these for the things that actually break in review builds:

Scale and units. A USDZ authored in centimetres placed in a metres world. Assert that the bounding box of a loaded asset is within an expected range.
Relative vs. world transforms. entity.position is relative to its parent; a test that reparents an entity and asserts position(relativeTo: nil) is unchanged catches a whole family of "model jumped across the room" bugs.
Anchor identity. When two plane anchors merge, ARKit removes one and grows the other. Assert your bookkeeping reparents content to the survivor rather than dropping it.
Distance clamping. Raycasts occasionally return a hit 40 m away through a window reflection. Test that your placement rejects results outside a sane range.

3. Test state machines against scripted tracking events

The second big defect category is lifecycle: session interruption, backgrounding, relocalization, thermal throttling, permission denial. Drive them through the fake.

@Test func coachingOverlayHidesOnceTrackingIsNormal() async {
    let source = FakeWorldSource()
    let model = PlacementViewModel(world: source)
    await model.begin()

    source.emit(.stub(tracking: .limited(.initializing)))
    await model.waitForUpdate()
    #expect(model.isCoachingVisible)

    source.emit(.stub(tracking: .normal))
    await model.waitForUpdate()
    #expect(!model.isCoachingVisible)
    #expect(model.canPlaceContent)
}

@Test func placementIsBlockedWhileRelocalizing() async {
    let source = FakeWorldSource()
    let model = PlacementViewModel(world: source)
    await model.begin()

    source.emit(.stub(tracking: .limited(.relocalizing), relocalizing: true))
    await model.waitForUpdate()
    #expect(!model.canPlaceContent)
}

These are the tests that stop a regression from shipping when someone "simplifies" the tracking-state switch six months later. They cost minutes to write and they run in the simulator in milliseconds.

4. ARKit replay data: deterministic manual QA

For the layer you genuinely cannot fake — ARKit's own tracking, plane detection, and scene reconstruction — Xcode gives you recorded sessions.

Install Apple's Reality Composer app on the test device and use its developer capture (Settings → Developer → Capture, or the capture flow inside the app) to record a .mov of the environment, including the sensor data ARKit needs.
AirDrop the recording to your Mac.
In Xcode: Product → Scheme → Edit Scheme → Run → Options → ARKit Replay data, pick the file, and run on device.

The app now receives the recorded camera and motion data instead of live sensors, so the same room, the same walk-around, and the same lighting play back on every run. That turns "the sofa drifts in Rita's kitchen" into a bug anyone on the team can reproduce.

Build a small library of recordings and treat it as test fixtures:

a well-lit room with clear horizontal planes — the happy path;
a low-texture white-wall corridor — forces .limited(.insufficientFeatures);
a fast pan — forces .limited(.excessiveMotion);
a session where the tester covers the camera and comes back — forces relocalization;
one recording per supported device class, because a non-LiDAR iPhone behaves differently from a Pro.

Caveats worth knowing before you build a process around this: replay still requires a physical device (it is not a simulator feature), recordings are large, they are tied to the sensor set of the device that captured them, and features such as RoomPlan and geo-anchored AR do not replay usefully. Treat replay as reproducible manual QA, not as automation.

5. What can actually run on CI

Be honest with your team about the boundary:

Runs on a simulator, so it runs on CI: everything above the ARWorldSource seam — view models, placement math, anchor bookkeeping, USDZ loading and bounding-box assertions, content-catalog validation, and RealityKit entity-graph tests. Recent Xcode releases run RealityKit in the simulator, so you can load a .usdz/.reality asset, build the entity hierarchy, and assert on components without a device. Verify this on your Xcode version before you depend on it — simulator support has moved around across releases.

Needs a device, so it belongs on a device lab or a manual checklist: anything that calls ARSession.run, camera permission flows, replay-data passes, thermal and frame-rate measurement (see ARKit performance), and LiDAR scene reconstruction.

Never automate: "does it look right." Occlusion quality, shadow plausibility, and drift are judgement calls. Script them as a human checklist with the replay recordings above, and run the checklist before every submission.

A pragmatic pipeline for a mid-sized ARKit app:

PR            → simulator unit tests + asset validation   (~2 min)
nightly       → build on device runner, launch smoke test
pre-release   → replay-data QA checklist across 3 device classes

6. Asset validation is the cheapest test you will ever write

Most "AR broke" tickets are asset regressions: a model re-exported at the wrong scale, a missing texture, a 60 MB USDZ that stalls the placement animation. Add one test that walks your bundled models:

@Test func bundledModelsAreSaneSizeAndScale() async throws {
    for url in Bundle.main.urls(forResourcesWithExtension: "usdz", subdirectory: nil) ?? [] {
        let entity = try await Entity(contentsOf: url)
        let bounds = entity.visualBounds(relativeTo: nil).extents
        #expect(bounds.max() < 5.0, "\(url.lastPathComponent) is larger than 5 m — check export units")
        #expect(bounds.max() > 0.01, "\(url.lastPathComponent) is smaller than 1 cm — check export units")

        let bytes = try FileManager.default.attributesOfItem(atPath: url.path)[.size] as? Int ?? 0
        #expect(bytes < 25_000_000, "\(url.lastPathComponent) is over 25 MB")
    }
}

That single test has caught more real defects on our client projects than any other AR test we write. Pair it with a disciplined export pipeline — see the USDZ asset pipeline.

Checklist

Draw a protocol seam between ARKit and your app on day one; retrofitting it later is a week of work.
Move every transform decision into a pure function and test it.
Script tracking-state and interruption sequences through a fake source.
Record a fixture library of ARKit replay sessions covering the failure modes, not just the happy path.
Keep simulator-safe tests on CI; keep device tests on a short, honest manual checklist.

Testing ARKit Apps Without Waving a Phone Around: Replay Data, Fakes, and CI