Most of our ARKit tutorials are about the world in front of the camera: planes, rooms, scanned objects, printed images. Face tracking points the other way. It uses the front-facing TrueDepth camera to track the geometry and expression of the user's own face, and it is the basis of every virtual try-on experience — eyewear, earrings, watches on a wrist (no, that one is object tracking), lipstick, hair colour — plus avatars, filters, and accessibility experiments.
It is also the ARKit feature most likely to get a project stuck late, because the interesting problems are not the API. They are occlusion, scale, device coverage, and privacy review. This walkthrough covers the code first and then the four things that actually decide whether your try-on ships.
Targets iOS 18 or later, Xcode 16 or later, on a device with a TrueDepth camera.
1. Check for support before you build a UI around it
Face tracking is not available on every iPhone or iPad, and it is not available at all on Apple Vision Pro (there is no equivalent user-facing face-tracking API on visionOS). Always branch on the capability rather than on a device model list:
import ARKit
guard ARFaceTrackingConfiguration.isSupported else {
// Fall back to a 2D product view, a photo-upload flow, or a message.
return
}
On supported hardware you can also ask how many faces the device will track at once:
let maxFaces = ARFaceTrackingConfiguration.supportedNumberOfTrackedFaces
For a try-on you want exactly one. Tracking multiple faces is a party-filter feature and costs you frame rate.
2. A face-tracking session in a RealityKit app
RealityKit's RealityView on iOS drives a world-tracking session when you set content.camera = .spatialTracking. Face tracking needs a session you configure yourself, so the pragmatic pattern in a SwiftUI app is still an ARView wrapped in a UIViewRepresentable. That is not a step backwards: everything inside the view is modern RealityKit.
import SwiftUI
import RealityKit
import ARKit
struct TryOnView: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero, cameraMode: .ar, automaticallyConfigureSession: false)
let config = ARFaceTrackingConfiguration()
config.maximumNumberOfTrackedFaces = 1
config.isLightEstimationEnabled = true
arView.session.run(config, options: [.resetTracking, .removeExistingAnchors])
let faceAnchor = AnchorEntity(.face)
arView.scene.addAnchor(faceAnchor)
context.coordinator.faceAnchor = faceAnchor
Task { @MainActor in
if let glasses = try? await Entity(named: "glasses") {
glasses.name = "glasses"
faceAnchor.addChild(glasses)
}
}
return arView
}
func updateUIView(_ uiView: ARView, context: Context) {}
func makeCoordinator() -> Coordinator { Coordinator() }
final class Coordinator {
var faceAnchor: AnchorEntity?
}
}
AnchorEntity(.face) is doing the heavy lifting. RealityKit keeps its transform glued to the tracked face, so any child entity inherits head position and rotation for free. The anchor's origin sits roughly behind the nose, with +Y up and +Z toward the front of the face.
3. Getting the glasses to sit on the nose
This is where most first attempts look wrong, and it is almost never a tracking problem. Three rules:
Model in real-world metres. ARKit face geometry is life-size. A typical adult interpupillary distance is about 63 mm and a men's eyewear frame is about 140 mm wide. If your USDZ was authored in centimetres, the frames will be the size of a windscreen. Fix the scale in the asset, not with a magic number in code — the asset is what your merchandising team will version 400 times.
Position against the face anchor, not against the model's own pivot. Set the model pivot at the bridge of the nose in your DCC tool, then a small offset is all you need:
glasses.position = SIMD3<Float>(0, 0.015, 0.06) // 1.5 cm up, 6 cm forward
Use the face geometry for per-user fit if it matters. ARFaceAnchor exposes leftEyeTransform, rightEyeTransform, and a full mesh. Measuring the actual eye separation each session and scaling the frame slightly is the difference between "a picture of glasses" and "my glasses". Read it from the session delegate:
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let face = anchors.first(where: { $0 is ARFaceAnchor }) as? ARFaceAnchor else { return }
let left = face.leftEyeTransform.columns.3
let right = face.rightEyeTransform.columns.3
let ipd = simd_distance(SIMD3<Float>(left.x, left.y, left.z),
SIMD3<Float>(right.x, right.y, right.z))
// Compare against the frame's design IPD and nudge the scale.
}
Clamp any auto-fit to a narrow range (say ±8%). Unbounded auto-scaling turns a tracking wobble into a product that breathes.
4. Occlusion: the detail that sells it
Untried, the arms of a pair of glasses will render straight through the wearer's temples and the illusion dies instantly. The fix is to render the face mesh itself as an invisible depth-writing occluder.
ARKit gives you ARFaceAnchor.geometry, a live mesh of the user's face that deforms with expression. Build a RealityKit MeshResource from it and apply OcclusionMaterial():
func makeOccluder(from geometry: ARFaceGeometry) throws -> ModelEntity {
var descriptor = MeshDescriptor(name: "faceMesh")
descriptor.positions = MeshBuffer(geometry.vertices)
descriptor.primitives = .triangles(geometry.triangleIndices.map { UInt32($0) })
let mesh = try MeshResource.generate(from: [descriptor])
return ModelEntity(mesh: mesh, materials: [OcclusionMaterial()])
}
Regenerate (or update) the mesh on face-anchor updates so it follows expression. Two practical notes: the ARKit face mesh stops at the jaw and does not include hair or ears, so long earrings still need care; and generating a fresh MeshResource at 60 Hz is wasteful — throttle it, or use a static neutral mesh scaled to the anchor if your product does not need expression-accurate occlusion.
5. Blend shapes: expression without a rig
ARFaceAnchor.blendShapes is a dictionary of 52 named coefficients from 0 to 1 — eyeBlinkLeft, jawOpen, mouthSmileRight, browInnerUp, and so on. This is the whole avatar/filter toolkit, and it is useful in try-on too: detect a smile to trigger a capture, or a blink to cycle frames hands-free.
let smile = (face.blendShapes[.mouthSmileLeft]?.floatValue ?? 0 +
(face.blendShapes[.mouthSmileRight]?.floatValue ?? 0)) / 2
if smile > 0.6 { captureFrame() }
Drive an avatar by mapping the same coefficients onto your model's own blend-shape weights. Keep the mapping in data, not code; artists will retune it constantly.
6. Lighting so the product does not look pasted on
Set isLightEstimationEnabled = true and, for face tracking, ARKit returns an ARDirectionalLightEstimate with spherical-harmonics coefficients derived from the face itself. Feed that into an ImageBasedLightComponent or, at minimum, use the ambient intensity and colour temperature to modulate a directional light in your scene. Metal frames and glossy lipstick look wrong under a fixed studio light in a dim room; this is a low-effort, high-perceived-quality fix.
7. The four things that actually decide the project
Device coverage. TrueDepth-equipped iPhones and iPads only. Every commerce build needs a graceful non-AR fallback, and that fallback is a real design and engineering line item — not something to discover in QA week.
Asset pipeline. One pair of glasses is an afternoon. Four hundred SKUs with correct real-world dimensions, consistent pivots, PBR materials, and colourways is the project. Decide early who owns the conversion to USDZ, what the naming and units convention is, and how a new SKU reaches the app without a release. Reality Composer Pro is the right place to standardize materials.
Performance and heat. The front camera, face mesh, occlusion, and a PBR product model together will warm a phone quickly. Cap textures, keep the model under a sensible triangle budget, avoid regenerating meshes every frame, and test a ten-minute browsing session — not a ten-second demo.
Privacy and review. This is the one that surprises people. Apple's App Store rules are explicit that data derived from the TrueDepth camera — the face mesh and blend-shape data — may not be used for advertising or marketing, may not be shared with third parties, and may not be collected for purposes other than delivering the feature. Practically: keep face data on device, never upload the mesh, and be careful with "save your look" features that ship a photo to a server. Your NSCameraUsageDescription string should say plainly what the camera is for. Legal teams in regulated markets will also ask about biometric-data statutes; give them a written data-flow description showing nothing leaves the device.
Where face tracking is the wrong tool
- Watches and rings. Those are hand and wrist problems, not face problems, and are much harder on iPhone today.
- Anything needing Vision Pro parity. Face tracking is an iPhone/iPad feature; plan a different experience for the headset.
- Sizing decisions with medical or safety consequences. A try-on is a merchandising aid, not a measurement device. Say so in the UI.
Next steps
- Apple's ARFaceTrackingConfiguration documentation and the blend-shape location reference.
- Our Getting Started with ARKit in 2026 tutorial if you are new to RealityKit, and Do You Still Need a Native AR App? if you are weighing a native build against AR Quick Look — try-on is one of the cases where native is genuinely required.
- Building a virtual try-on and want the asset pipeline and privacy questions answered before you commit a budget? RealityRogue's senior ARKit consultants do short scoping engagements as well as full builds. Get in touch.