RealityRogue | One Reference Object, Two Platforms: Object Tracking on iOS 27 and visionOS 27

Object tracking — recognizing a specific physical thing and attaching content to it — is the feature enterprise AR buyers ask for most. A technician points a device at a pump and the app highlights the valve to turn; a warehouse worker looks at a crate and sees its contents. Until this year, delivering that on both iPhone and Apple Vision Pro meant two tracking pipelines: ARKit's ARReferenceObject on iOS and visionOS's ReferenceObject + ObjectTrackingProvider, each trained from its own capture.

At WWDC 2026 Apple made ARKit's reference objects cross-platform. With iOS 27 and visionOS 27, the same trained .referenceobject file tracks on both devices. This tutorial walks through training one object and using it, unmodified, on an iPhone and on a Vision Pro.

iOS 27 and visionOS 27 are in developer beta as of August 2026, with public release expected in September. Everything below was written against the beta SDKs; verify symbol names against the release documentation before shipping. The visionOS 2-era APIs (ObjectTrackingProvider, ReferenceObject) are stable and unchanged.

What you need

An iPhone with LiDAR (Pro models) running iOS 27 beta, or any recent iPhone for the iOS tracking half.
An Apple Vision Pro running visionOS 27 beta.
A Mac with Xcode 26 and Create ML.
A rigid, textured, opaque object roughly 10 cm to 1 m across. Shiny, transparent, or deformable objects do not train well on any platform.

1. Capture the object

Use Apple's Object Capture flow on the iPhone — the same photogrammetry pipeline covered in our Object Capture tutorial — to produce a USDZ of the object. A 3D reconstruction is the input Create ML wants for a spatial reference object. Shoot in even lighting, capture all three orbits, and flip the object if its underside matters for tracking.

Keep the resulting USDZ under a few hundred thousand triangles; Create ML trains on the geometry and texture, not on the photos themselves.

2. Train the reference object in Create ML

Open Create ML and choose the Spatial → Object Tracking template.
Drop in the USDZ.
Choose a viewing angle profile: All Angles for objects handled in the hand, Upright for objects that sit on a surface, Front for wall-mounted things.
Click Train. Training runs on the Mac and takes minutes to hours depending on the object.
Export the .referenceobject file.

Before iOS 27 this file was visionOS-only. With the WWDC26 SDKs the same export loads on iOS through ARReferenceObject, which is the whole point of this tutorial: one training, one file, two platforms.

Add the .referenceobject to your app target (or to an asset catalog AR Resource Group if you prefer to keep the iOS and visionOS loading code symmetrical).

3. Track it on Apple Vision Pro

On visionOS, object tracking is an ARKit data provider you run inside an immersive space:

import SwiftUI
import RealityKit
import ARKit

@MainActor
final class ObjectTrackingModel: ObservableObject {
    private let session = ARKitSession()
    private var provider: ObjectTrackingProvider?
    let root = Entity()
    private var overlays: [UUID: Entity] = [:]

    func start() async {
        guard let url = Bundle.main.url(forResource: "Pump", withExtension: "referenceobject"),
              let reference = try? await ReferenceObject(from: url) else {
            print("Missing or invalid reference object")
            return
        }

        let provider = ObjectTrackingProvider(referenceObjects: [reference])
        self.provider = provider

        do {
            try await session.run([provider])
        } catch {
            print("ARKit session failed: \(error)")
            return
        }

        for await update in provider.anchorUpdates {
            handle(update)
        }
    }

    private func handle(_ update: AnchorUpdate<ObjectAnchor>) {
        let anchor = update.anchor
        switch update.event {
        case .added:
            let overlay = makeOverlay(for: anchor)
            overlays[anchor.id] = overlay
            root.addChild(overlay)
            fallthrough
        case .updated:
            overlays[anchor.id]?.transform = Transform(matrix: anchor.originFromAnchorTransform)
            overlays[anchor.id]?.isEnabled = anchor.isTracked
        case .removed:
            overlays[anchor.id]?.removeFromParent()
            overlays[anchor.id] = nil
        }
    }

    private func makeOverlay(for anchor: ObjectAnchor) -> Entity {
        // A simple bounding-box wireframe; swap for your real content.
        let extent = anchor.boundingBox.extent
        let box = ModelEntity(mesh: .generateBox(size: extent),
                              materials: [UnlitMaterial(color: .cyan.withAlphaComponent(0.25))])
        box.position = anchor.boundingBox.center
        return box
    }
}

And the immersive space that hosts it:

struct TrackingSpace: View {
    @StateObject private var model = ObjectTrackingModel()

    var body: some View {
        RealityView { content in
            content.add(model.root)
        }
        .task { await model.start() }
    }
}

Remember that object tracking on visionOS requires an ImmersiveSpace; it is not available inside a window or a volume. ObjectTrackingProvider.isSupported is worth checking at launch, and the user grants world-sensing permission the first time the provider runs.

4. Track the same file on iPhone (iOS 27)

On iOS, ARKit's object detection has used ARReferenceObject and ARWorldTrackingConfiguration.detectionObjects since iOS 12. What changes in iOS 27 is the input: ARReferenceObject now loads the Create ML .referenceobject format, and detection is upgraded to continuous tracking of the object's pose rather than a one-time detection.

import ARKit
import RealityKit

final class PhoneTrackingCoordinator: NSObject, ARSessionDelegate {
    let arView = ARView(frame: .zero)
    private var overlays: [UUID: Entity] = [:]

    func start() {
        guard let url = Bundle.main.url(forResource: "Pump", withExtension: "referenceobject"),
              let reference = try? ARReferenceObject(archiveURL: url) else {
            print("Missing or invalid reference object")
            return
        }

        let config = ARWorldTrackingConfiguration()
        config.detectionObjects = [reference]
        arView.session.delegate = self
        arView.session.run(config)
    }

    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        for case let objectAnchor as ARObjectAnchor in anchors {
            let overlay = makeOverlay(for: objectAnchor)
            let anchorEntity = AnchorEntity(anchor: objectAnchor)
            anchorEntity.addChild(overlay)
            arView.scene.addAnchor(anchorEntity)
            overlays[objectAnchor.identifier] = anchorEntity
        }
    }

    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        // With iOS 27 continuous tracking, ARObjectAnchor updates arrive here
        // as the object moves; AnchorEntity(anchor:) follows them automatically.
    }

    private func makeOverlay(for anchor: ARObjectAnchor) -> Entity {
        let extent = anchor.referenceObject.extent
        let box = ModelEntity(mesh: .generateBox(size: extent),
                              materials: [UnlitMaterial(color: .cyan.withAlphaComponent(0.25))])
        box.position = anchor.referenceObject.center
        return box
    }
}

Notice the shape of the two listings: the same reference file, the same overlay construction, and an anchor that carries boundingBox/extent on both sides. On visionOS the anchor stream is an AsyncSequence; on iOS it is the session delegate. Everything above those two lines can be shared code.

5. Sharing the content layer

Put the overlay construction and any business logic in a Swift package that both targets import, keyed on the anchor's identity and transform:

public struct TrackedObjectPose: Sendable {
    public let id: UUID
    public let transform: simd_float4x4
    public let isTracked: Bool
}

public protocol ObjectOverlayBuilder {
    func overlay(for pose: TrackedObjectPose, extent: SIMD3<Float>) -> Entity
}

The platform targets reduce to adapters that turn ObjectAnchor or ARObjectAnchor into TrackedObjectPose. The RealityKit entities themselves are identical on both platforms, which is the real win: your designers build one overlay in Reality Composer Pro and it ships to the headset and the phone.

6. Enterprise use cases that justify the work

Guided maintenance. Vision Pro for hands-free procedures on the plant floor; iPhone for the technician who is up a ladder and cannot wear a headset.
Inspection and QA. Track a part, overlay the spec tolerances, log a photo with the pose embedded.
Training. Same reference object, same overlays, whether the trainee has a headset or a company iPad.
Retail and showroom. Track the product on the display stand; customers use their own iPhones, staff use Vision Pro.

Before iOS 27 each of these required double the tracking-model maintenance. Now the cost of supporting the second device is the adapter, not a second pipeline.

Gotchas

Train for the angles you will actually see. All Angles costs tracking robustness for objects that never leave the table.
Lighting differences between the capture and the field still matter; capture in lighting similar to deployment.
Large objects (vehicles, machinery) are better handled as several reference objects on distinctive sub-assemblies than as one.
Keep an eye on the release notes through September 2026; beta APIs change.

Want a working prototype on your own object, on iPhone and Vision Pro? RealityRogue's ARKit consultants can build one with you. Contact us.