Written against iOS 18 and later, and validated on the iOS 27 beta. Code is RealityKit-first; the ARKit session plumbing is the same if you are still on ARView.
Every few weeks a client asks for some version of the same thing: point the phone at a machine, a shelf, a parts bin, or a shipping label, and have the app know what it is looking at and pin a label to it in 3D. Recognise-and-annotate. It sounds like an object-tracking problem, and sometimes it is — but reference-object tracking needs a trained model per physical object, and it falls apart when you have four hundred SKUs or a valve that looks like every other valve.
The alternative is the one most teams should reach for first: run an ordinary 2D detector — Vision's built-in requests or your own Core ML model — on the ARKit camera feed, then project each detection into world space and drop an ARKit anchor there. You get the flexibility of a classifier or detector with the persistence of an AR anchor.
This post is the plumbing. It is the part that is not in the sample code, and it is where projects go wrong: threading, coordinate spaces, and stability.
The architecture, in one paragraph
ARKit owns the camera. You do not start an AVCaptureSession; you read ARFrame.capturedImage from the session you already have. A throttled worker pulls a frame, hands the pixel buffer to Vision, gets back normalised bounding boxes, converts each box centre to view coordinates, raycasts that point against ARKit's scene understanding to get a 3D position, and either creates a new anchor or merges the hit into an anchor you already have. The RealityKit layer only ever reads anchors — it never talks to Vision.
Four boundaries, in order: frames → Vision → world space → anchors. Keep them separate and this is a two-day feature. Blur them and you will spend two weeks chasing a stutter.
1. Get frames without starving the renderer
The single most common mistake is running inference on every frame. ARKit delivers 60 frames a second; your detector probably needs 30–80 ms. You will fall behind within a second, hold onto pixel buffers ARKit wants back, and the whole session will judder.
Run at most one inference at a time, and drop everything that arrives while one is in flight:
import ARKit
import Vision
final class FrameDetector: NSObject, ARSessionDelegate {
private let queue = DispatchQueue(label: "com.realityrogue.vision", qos: .userInitiated)
private var isBusy = false
private var lastRun = CFAbsoluteTimeGetCurrent()
private let minInterval: CFTimeInterval = 0.20 // 5 Hz is plenty for labelling
var onDetections: (([Detection], ARFrame) -> Void)?
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let now = CFAbsoluteTimeGetCurrent()
guard !isBusy, now - lastRun >= minInterval else { return }
guard case .normal = frame.camera.trackingState else { return }
isBusy = true
lastRun = now
// Copy what you need off the frame; do not retain the ARFrame on a worker queue.
let pixelBuffer = frame.capturedImage
let orientation = CGImagePropertyOrientation.right // portrait, rear camera
let camera = frame.camera
queue.async { [weak self] in
defer { self?.isBusy = false }
self?.run(pixelBuffer: pixelBuffer, orientation: orientation, camera: camera, frame: frame)
}
}
}
Two details that matter more than they look:
isBusyis a drop policy, not a queue. Never buffer frames. A stale detection projected into a moved camera pose lands in the wrong place, which looks worse than no detection at all.- Retaining
ARFrameblocks ARKit. ARKit recycles a small pool of frames. If you hold references on a background queue, the session throttles itself. Copy the pixel buffer reference, use it, and let go quickly. If you need the frame's camera transform later, captureframe.camera.transformas a value rather than keeping the frame alive.
At 5 Hz on an A17 or later, a small YOLO-class detector costs a few percent of frame budget and barely moves the thermal needle. See our ARKit performance post for how to prove that with Instruments rather than assuming it.
2. Run Vision on the ARKit pixel buffer
ARKit hands you a YCbCr CVPixelBuffer in camera sensor orientation — landscape-right, regardless of how the user is holding the phone. Vision needs to be told that, or every bounding box will be rotated 90° and you will lose an afternoon.
struct Detection {
let label: String
let confidence: Float
let boundingBox: CGRect // Vision-normalised, origin bottom-left
}
private func run(pixelBuffer: CVPixelBuffer,
orientation: CGImagePropertyOrientation,
camera: ARCamera,
frame: ARFrame) {
let request = VNCoreMLRequest(model: self.visionModel) { request, _ in
let results = (request.results as? [VNRecognizedObjectObservation]) ?? []
let detections = results
.filter { $0.confidence > 0.6 }
.compactMap { obs -> Detection? in
guard let top = obs.labels.first else { return nil }
return Detection(label: top.identifier,
confidence: top.confidence,
boundingBox: obs.boundingBox)
}
guard !detections.isEmpty else { return }
DispatchQueue.main.async { self.onDetections?(detections, frame) }
}
request.imageCropAndScaleOption = .scaleFill
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: orientation, options: [:])
try? handler.perform([request])
}
The model itself is loaded once, never per frame:
private lazy var visionModel: VNCoreMLModel = {
let config = MLModelConfiguration()
config.computeUnits = .all // let Core ML use the Neural Engine
let ml = try! EquipmentDetector(configuration: config).model
return try! VNCoreMLModel(for: ml)
}()
If you do not have a custom model yet, start with Vision's built-ins — VNRecognizeTextRequest for asset tags and labels, VNDetectBarcodesRequest for codes, VNClassifyImageRequest for coarse categories. A surprising number of "we need AI in AR" briefs are really read the serial number and pin it to the machine, which is a text request and no training data at all.
3. From a 2D box to a 3D position
This is where the coordinate spaces bite. Vision returns boxes normalised to the image, origin bottom-left. Your ARView is in UIKit points, origin top-left, and the camera image is cropped and scaled to fill the view. Do not hand-roll the maths — ARKit gives you the exact transform:
@MainActor
func worldPosition(for detection: Detection,
in view: ARView,
frame: ARFrame) -> SIMD3<Float>? {
let viewSize = view.bounds.size
let interfaceOrientation: UIInterfaceOrientation = .portrait
// ARKit's own image -> view transform. Invert Vision's bottom-left origin first.
let display = frame.displayTransform(for: interfaceOrientation, viewportSize: viewSize)
var box = detection.boundingBox
box.origin.y = 1 - box.origin.y - box.height // flip to top-left origin
let centreNormalised = CGPoint(x: box.midX, y: box.midY)
let centreInView = centreNormalised.applying(display)
let point = CGPoint(x: centreInView.x * viewSize.width,
y: centreInView.y * viewSize.height)
// Prefer the reconstructed mesh, fall back to planes, then to estimated geometry.
let queries: [(ARRaycastQuery.Target, ARRaycastQuery.TargetAlignment)] = [
(.estimatedPlane, .any),
(.existingPlaneGeometry, .any),
(.estimatedPlane, .vertical)
]
for (target, alignment) in queries {
if let query = view.makeRaycastQuery(from: point, allowing: target, alignment: alignment),
let hit = view.session.raycast(query).first {
return hit.worldTransform.translation
}
}
return nil
}
extension simd_float4x4 {
var translation: SIMD3<Float> { SIMD3(columns.3.x, columns.3.y, columns.3.z) }
}
On LiDAR devices with scene reconstruction enabled, the raycast lands on the actual mesh and the label sits on the object rather than on the wall behind it. That is a large enough quality difference that it is worth branching on device capability — the LiDAR scene reconstruction walkthrough covers how to turn it on and what to do on phones without it.
If nothing is hit — the detection was on a distant object, or on the sky — do not invent a position by pushing the ray out a fixed distance. Discard it. A label floating a metre from its object destroys user trust faster than a missing label.
4. Merge, do not multiply
Run the loop above naively and after ten seconds you will have forty anchors for one fire extinguisher. Every project needs a small tracker layer between detections and anchors. The rules that hold up in the field:
- Match on label plus distance. A new detection merges into an existing anchor if the label matches and the world position is within a radius that scales with distance from the camera — roughly 15 cm for something a metre away, more for something across the room.
- Require confirmation. Do not show anything until an anchor has been observed n times (three works well). One-frame false positives disappear entirely.
- Smooth the position. Exponential moving average with a low weight on new observations. Jitter reads as "broken" even when the label is correct.
- Expire on absence, not on time. If the anchor is in the camera frustum and the detector has not seen it for several consecutive runs, fade it out. If it is behind the user, leave it alone.
final class TrackedObject {
let label: String
private(set) var position: SIMD3<Float>
private(set) var hits = 1
private(set) var misses = 0
var isConfirmed: Bool { hits >= 3 }
init(label: String, position: SIMD3<Float>) {
self.label = label
self.position = position
}
func observe(at p: SIMD3<Float>) {
position = mix(position, p, t: 0.25) // smooth, do not snap
hits += 1
misses = 0
}
}
func mix(_ a: SIMD3<Float>, _ b: SIMD3<Float>, t: Float) -> SIMD3<Float> { a + (b - a) * t }
Only confirmed objects get an AnchorEntity. That single rule is the difference between a demo and something a technician will use.
5. Attach the label in RealityKit
Once you have a stable world position, the presentation layer is ordinary RealityKit — and it should be billboarded so the text always faces the user:
func addLabel(_ text: String, at position: SIMD3<Float>, in scene: RealityKit.Scene) {
let anchor = AnchorEntity(world: position)
let mesh = MeshResource.generateText(text,
extrusionDepth: 0.001,
font: .systemFont(ofSize: 0.04),
alignment: .center)
let entity = ModelEntity(mesh: mesh, materials: [UnlitMaterial(color: .white)])
entity.position.y = 0.05
entity.components.set(BillboardComponent())
anchor.addChild(entity)
scene.addAnchor(anchor)
}
BillboardComponent (iOS 18 and later) removes the hand-rolled look-at maths that used to be in every one of these codebases. Keep the label geometry cheap — generated text meshes are surprisingly heavy, so cache them per string rather than regenerating each frame.
6. Privacy, review, and the things clients forget
- Camera usage string.
NSCameraUsageDescriptionmust say what the app does with the images. "Used for augmented reality features" is fine; silence is a rejection. - Say where inference happens. Vision and Core ML run entirely on device. If your detector is on device, put that in the privacy nutrition label and the onboarding copy — for industrial and healthcare clients it is often the deciding procurement question.
- If you send frames to a server, everything changes. A cloud detector turns a camera feed into a data-transfer question, with consent, retention, and regional review. Nine times out of ten, a quantised on-device model at 5 Hz is both cheaper and easier to get approved.
- Do not log pixel buffers. Crash reporters and analytics SDKs that capture screenshots will happily exfiltrate a factory floor. Exclude the AR view.
7. When this approach is the wrong tool
Be honest about the boundary:
- You need pose, not position. If the label has to align to a specific face of an object — an overlay on a control panel, an arrow pointing at one port — a bounding box centre will not do it. That is reference-object tracking, and it is now a single trained asset across iOS 27 and visionOS 27.
- The target is flat and printed. Posters, packaging, and labels are image tracking — more accurate, far cheaper, and no model to train.
- You need room geometry. Walls, doors, and furniture are RoomPlan, not a detector.
- You have no training data. A custom detector needs hundreds to thousands of labelled images per class, gathered in the actual lighting of the actual site. Budget for the data collection trip, or start with text and barcode recognition, which need none.
A build order that works
- Ship the Vision layer first, with no AR at all: a plain camera preview and 2D boxes. Prove the model works on the real objects, in the real place.
- Add the raycast and print world positions to the console. Prove they are stable when you walk around.
- Add the tracker and confirmation rules.
- Add the RealityKit labels last. It is the part that takes an afternoon.
Teams that do it in the opposite order — pretty labels first — usually discover in week three that the model was never good enough on the shop floor, and by then the pretty labels are load-bearing.
Next steps
If you are scoping a recognise-and-annotate feature — inventory, field service, inspection, retail shelf audit — the decisive questions are what the labels have to be attached to, how precise the placement must be, and whether training data exists. We do fixed-fee technical discovery on exactly that, and can tell you in a week whether you need a detector, image tracking, reference objects, or none of the above.
Get in touch with your use case, or read our guide to choosing between native ARKit, AR Quick Look, and WebXR before you commit to an app at all.