RealityRogue | ARKit Performance: Holding a Stable Frame Rate Without Cooking the iPhone

Most ARKit apps do not fail review because a feature is missing. They fail in the field because after eleven minutes the iPhone is hot, the frame rate has halved, the camera preview has gone dim, and the user has quit. AR is the most expensive thing a phone can do: the camera pipeline, the Neural Engine, the LiDAR sensor, the CPU running tracking, and the GPU rendering your content are all busy at the same time, and every one of them produces heat in the same aluminium body.

This tutorial is about the part of AR development nobody demos at WWDC: keeping a RealityKit app at a stable frame rate for a twenty-minute session on a mid-range iPhone. It assumes iOS 18 or later, RealityKit with RealityView, and Xcode 16 or later.

1. Measure before you optimize

Three tools, in order of usefulness.

Instruments → RealityKit Trace. The only profiler that shows you RealityKit's own frame breakdown: CPU time in systems and rendering, GPU time, and where a dropped frame actually went. Run it on a device, not the simulator, and run it for at least five minutes so thermal effects appear.

The Metal Performance HUD. The fastest signal for a quick loop. Set the environment variable MTL_HUD_ENABLED=1 in your scheme and you get a live overlay with frame time and GPU time.

Your own signposts. For app-level work — a photogrammetry job, a mesh classification pass, a network fetch of a USDZ — wrap it in signposts so it shows up in the same timeline:

import os

let arLog = OSLog(subsystem: "com.example.ar", category: .pointsOfInterest)

os_signpost(.begin, log: arLog, name: "LoadModel")
let entity = try await ModelEntity(named: "sofa")
os_signpost(.end, log: arLog, name: "LoadModel")

The budget you are profiling against: ARKit's world-tracking camera runs at 60 Hz on most formats, so a frame is 16.6 ms — and your rendering has to share it with tracking. If your GPU time is over about 8 ms you have no headroom for the thermal throttling that is coming.

2. The cheapest wins are in your session configuration

Every ARKit feature you enable costs power for the whole session, whether or not you use its output on a given frame. Audit the configuration first.

let config = ARWorldTrackingConfiguration()

// Turn plane detection OFF once the user has placed content.
config.planeDetection = hasPlacedContent ? [] : [.horizontal, .vertical]

// Scene reconstruction is expensive. Only if you need occlusion or physics.
if ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) && needsOcclusion {
    config.sceneReconstruction = .mesh          // .meshWithClassification costs more
}

// People occlusion is the single most expensive frame semantic.
if ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth)
    && needsPeopleOcclusion {
    config.frameSemantics.insert(.personSegmentationWithDepth)
}

config.environmentTexturing = .automatic       // .manual if you control the probes
session.run(config)

Rules of thumb from production apps we have profiled:

personSegmentationWithDepth runs a segmentation model on the Neural Engine every frame. It is gorgeous and it is the first thing to drop on older devices or when the phone heats up.
.meshWithClassification roughly doubles the reconstruction cost versus plain .mesh. Use classification only if you branch on wall-vs-floor-vs-table; see our LiDAR scene reconstruction tutorial for what each level buys you.
Plane detection after placement is pure waste in a "place one object" app, and it also causes anchors to drift as planes are re-merged. Reconfigure with planeDetection = [] and re-run without resetTracking.
Image tracking: maximumNumberOfTrackedImages defaults to 1 for a reason. Each simultaneously tracked image costs real CPU.
Video format: ARWorldTrackingConfiguration.supportedVideoFormats includes 4K options on recent devices. Unless you are recording, take a 1080p60 format. A 4K camera feed costs bandwidth, memory, and heat for pixels nobody sees.

if let format = ARWorldTrackingConfiguration.supportedVideoFormats
    .first(where: { $0.imageResolution.width <= 1920 && $0.framesPerSecond == 60 }) {
    config.videoFormat = format
}

3. Asset budgets: the number that actually predicts your frame rate

For handheld AR, the practical budgets we hand to 3D teams:

Item	Target for iPhone AR
Triangles on screen (all entities)	≤ 150k, ideally ≤ 100k
Draw calls	≤ 100
Unique materials per model	≤ 4
Texture resolution	2048² hero, 1024² everything else
USDZ file size (downloaded at runtime)	≤ 10 MB, ≤ 25 MB absolute cap

Draw calls matter more than triangles. A 30k-triangle model split into 60 mesh parts with 20 materials will beat up the CPU harder than a single 200k-triangle mesh with one material. Ask your artists to merge meshes and atlas textures; it is usually a half-day of work and the single biggest win available.

Reuse loaded resources. Loading the same USDZ five times gives you five copies of every texture in memory. Load once, clone:

let template = try await Entity(named: "chair")
for position in positions {
    let copy = template.clone(recursive: true)   // shares the underlying resources
    copy.position = position
    anchor.addChild(copy)
}

Compress textures. Reality Composer Pro and usdzconvert can emit compressed textures; a PNG normal map in a USDZ is a memory and bandwidth bug, not an asset.

4. RealityKit patterns that quietly cost frames

Do work in Systems, not in a per-frame closure that mutates SwiftUI state. A @State change per frame re-evaluates your view body 60 times a second. Put simulation in a RealityKit System and keep the SwiftUI layer for UI that changes when a human does something.
Attachments are UIKit/SwiftUI views rendered to textures. They are wonderful for labels and expensive in quantity. Ten attachments is fine; a hundred floating labels is a redesign.
Occlusion materials are not free. ModelComponent with OcclusionMaterial still rasterizes. Occlude with a simplified proxy mesh, not with the full reconstructed scene mesh, when you can.
Shadows. GroundingShadowComponent on one hero object reads as real. On thirty objects it is a shadow-map pass you did not budget for.
Physics. Give collision shapes as boxes and spheres — ShapeResource.generateConvex(from:) on a detailed mesh produces a hull that the solver will hate. Sleep bodies that are at rest.
Video textures. A 4K VideoMaterial on a poster is a decode plus an upload every frame; 1080p or lower is almost always indistinguishable at AR viewing distance. Our image tracking post covers the pinning; the resolution choice is a performance decision.

5. Handle thermal state explicitly, before iOS does it for you

When the system throttles, it does so bluntly: the CPU and GPU slow down and your app looks broken. If you degrade first, you control what the user loses. Watch ProcessInfo.thermalState and step down a quality ladder.

final class QualityGovernor {
    enum Level { case high, medium, low }
    private(set) var level: Level = .high

    init(session: ARSession) {
        NotificationCenter.default.addObserver(
            forName: ProcessInfo.thermalStateDidChangeNotification,
            object: nil, queue: .main
        ) { [weak self] _ in self?.apply(to: session) }
    }

    func apply(to session: ARSession) {
        switch ProcessInfo.processInfo.thermalState {
        case .nominal, .fair:  level = .high
        case .serious:         level = .medium
        case .critical:        level = .low
        @unknown default:      level = .medium
        }
        reconfigure(session)
    }

    private func reconfigure(_ session: ARSession) {
        let config = ARWorldTrackingConfiguration()
        switch level {
        case .high:
            config.frameSemantics = [.personSegmentationWithDepth]
            config.sceneReconstruction = .meshWithClassification
        case .medium:
            config.frameSemantics = []                 // drop people occlusion
            config.sceneReconstruction = .mesh
        case .low:
            config.frameSemantics = []
            config.sceneReconstruction = []            // rely on planes already found
        }
        config.planeDetection = []
        session.run(config)                            // no resetTracking: keeps anchors
    }
}

Pair the configuration ladder with a content ladder: at .medium disable grounding shadows and particle effects; at .low swap hero models for their LOD1 versions and cap the renderer with RealityViewCameraContent-level effects turned off. Users notice a soft shadow disappearing far less than they notice 25 fps.

Also handle ARSession interruptions properly — pause when your view disappears, and never leave a session running behind a full-screen modal:

.onDisappear { session.pause() }

6. Memory, downloads, and the long session

Unload entities the user has walked away from. AnchorEntity removal frees the meshes; keeping every model ever placed in the scene graph is the most common leak we find.
Stream USDZ assets on demand rather than shipping a 400 MB catalogue in the bundle; cache to disk, not to memory.
Watch ARFrame retention. Holding on to session.currentFrame outside the delegate callback stalls the pipeline and will eventually produce dropped frames and a memory spike. Copy what you need (the transform, the depth buffer) and let the frame go.

7. A test matrix you can actually run

Performance work only counts if it is verified on the devices your users have.

Pick three devices: the newest Pro, a three-year-old non-Pro, and the oldest model you support.
Run a 20-minute soak on each: place content, walk a realistic route, leave the app in the foreground. Record frame time at minutes 1, 5, 10, and 20.
Log thermal state transitions to your analytics with a session ID. Real-world thermal data beats any lab measurement.
Set a release gate: no shipped build may drop below 45 fps at minute 10 on the oldest supported device.
Re-run after every asset change. Art updates break performance far more often than code does.

ARKit Performance: Holding a Stable Frame Rate Without Cooking the iPhone