Targets iOS 18 or later with Xcode 16 or later. Everything here is shipping API — no betas required.
The single biggest reason a demo feels like a toy is that the virtual content ignores the room. A ball rolls through a wall. A chair renders in front of the user's own hand. A product model sits serenely on top of a sofa instead of behind it. Plane detection alone will never fix this, because a plane is an infinite flat guess about a lumpy world.
LiDAR scene reconstruction fixes it. ARKit builds a live triangle mesh of the room, and RealityKit can use that mesh for two things that matter enormously to perceived quality: occlusion (real geometry hides virtual geometry) and physics (virtual objects collide with the real world). This tutorial wires up both, adds people occlusion, and — the part most tutorials skip — handles the majority of iPhones that have no LiDAR sensor at all.
Our RoomPlan walkthrough covers the other half of the story: RoomPlan gives you a clean, parametric model of a room after a scan. Scene reconstruction gives you a raw mesh right now, every frame. They solve different problems and it is worth knowing which one you need.
1. Know what hardware you are on
Scene reconstruction requires a LiDAR scanner: iPhone 12 Pro and later Pro/Pro Max models, and iPad Pro from 2020 onward. Base-model iPhones do not have it. Check before you promise anything to the user:
import ARKit
let hasSceneReconstruction =
ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification)
let hasPeopleOcclusion =
ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth)
Two separate capabilities, two separate checks. People occlusion works on any A12 device or newer, LiDAR or not — it is driven by a machine-learning depth estimate from the camera, not the depth sensor. So a plain iPhone 15 gets people occlusion but not world mesh occlusion.
Never call supportsSceneReconstruction in the simulator and never assume it from a device model string. Ask the framework.
2. Turn it on with SpatialTrackingSession
If you are on the modern SwiftUI path (RealityView, which is where all new work should start — see Getting Started with ARKit in 2026), the entry point is SpatialTrackingSession. It is the RealityKit-level wrapper that configures the underlying ARKit session for you and works the same way on iOS and visionOS.
import SwiftUI
import RealityKit
import ARKit
struct SceneUnderstandingView: View {
var body: some View {
RealityView { content in
content.camera = .spatialTracking
let session = SpatialTrackingSession()
var configuration = SpatialTrackingSession.Configuration(
tracking: [.plane],
sceneUnderstanding: [.occlusion, .physics],
camera: .back
)
// Ask for what the device can actually do.
if !ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) {
configuration = SpatialTrackingSession.Configuration(
tracking: [.plane],
sceneUnderstanding: [],
camera: .back
)
}
let unavailable = await session.run(configuration)
if let unavailable, !unavailable.anchor.isEmpty {
print("Unavailable capabilities:", unavailable)
}
content.add(makeContent())
}
.ignoresSafeArea()
}
}
The return value of run(_:) is the honest part of the API: it tells you which requested capabilities the device or the user's permissions denied, so you can degrade the experience deliberately instead of shipping a silently broken feature.
The ARView route
Plenty of production apps still host RealityKit inside ARView. The equivalent there is a single options set, and it is worth knowing because the option names are more explicit:
arView.environment.sceneUnderstanding.options.insert(.occlusion)
arView.environment.sceneUnderstanding.options.insert(.physics)
arView.environment.sceneUnderstanding.options.insert(.collision)
// .receivesLighting casts virtual shadows onto the real mesh — expensive, use sparingly.
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal, .vertical]
if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
config.sceneReconstruction = .meshWithClassification
}
if ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) {
config.frameSemantics.insert(.personSegmentationWithDepth)
}
arView.session.run(config)
.occlusion and .physics are independent. Occlusion is a rendering trick — the mesh is drawn invisibly into the depth buffer. Physics generates real collision shapes from the mesh. You can have either without the other, and you often want occlusion only, because it is much cheaper.
3. Make a virtual object collide with the real room
Occlusion needs nothing from your entities. Physics does: the entity has to be a physics body with a collision shape.
func makeBall() -> ModelEntity {
let ball = ModelEntity(
mesh: .generateSphere(radius: 0.05),
materials: [SimpleMaterial(color: .systemOrange, isMetallic: false)]
)
ball.generateCollisionShapes(recursive: true)
ball.components.set(
PhysicsBodyComponent(
massProperties: .init(mass: 0.4),
material: .generate(friction: 0.4, restitution: 0.6),
mode: .dynamic
)
)
return ball
}
Drop that entity into an anchored scene with scene-understanding physics enabled and it will bounce off your real floor, roll under your real table, and stop at your real wall. No plane fitting, no manual colliders. This is the demo moment: hand the device to a stakeholder and let them throw a ball at their own furniture.
A caution about scale. The reconstruction mesh is coarse — expect a few centimeters of error and rounded-off edges on thin objects. Small, fast objects tunnel through it. If you need reliable contact for something the size of a coin, keep the object bigger than the mesh error or fall back to a plane collider for the specific surface you care about.
4. Raycasting against the mesh instead of a plane
Placement should hit the real world, not an inferred plane. RealityKit's raycast against scene-understanding geometry is a CollisionCastQueryType filter away:
// UIKit / ARView
if let result = arView.raycast(
from: tapPoint,
allowing: .estimatedPlane,
alignment: .any
).first {
// fallback: plane estimate
}
// Better on LiDAR devices: hit the reconstructed mesh directly.
let query = arView.makeRaycastQuery(
from: tapPoint,
allowing: .existingPlaneGeometry,
alignment: .any
)
And in a SwiftUI RealityView, use a spatial tap plus a scene raycast:
.gesture(
SpatialTapGesture()
.targetedToAnyEntity()
.onEnded { value in
let point = value.entity.position(relativeTo: nil)
// place content at `point`, or run scene.raycast(from:to:query:)
}
)
For mesh-accurate placement without RealityKit's helpers, iterate ARMeshAnchor geometry from the ARKit session delegate and raycast against the triangles yourself. That is rarely worth it in app code — but it is exactly what you do when you need to classify the surface.
5. Use mesh classification when the surface type matters
.meshWithClassification tags each face with a ARMeshClassification: .wall, .floor, .ceiling, .table, .seat, .window, .door, .none. That unlocks behaviour that feels smart:
- A wall-mounted TV mockup that refuses to snap onto a floor.
- A furniture app that offers a different catalogue when the user taps a table versus a floor.
- Field-service annotations that stick to equipment surfaces and not to windows.
func classification(of anchor: ARMeshAnchor, faceIndex: Int) -> ARMeshClassification {
guard let classifications = anchor.geometry.classification else { return .none }
let pointer = classifications.buffer.contents()
.advanced(by: faceIndex * classifications.stride)
return ARMeshClassification(rawValue: pointer.assumingMemoryBound(to: UInt8.self).pointee) ?? .none
}
Classification costs extra CPU. If you only need occlusion and physics, request .mesh rather than .meshWithClassification.
6. People occlusion: the cheap win
Of all the features here, people occlusion has the best ratio of perceived realism to effort, and it works on far more devices than LiDAR. One frame semantic and the user's hands and body correctly cut in front of virtual content:
if ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) {
config.frameSemantics.insert(.personSegmentationWithDepth)
}
Use .personSegmentationWithDepth when content can be nearer or farther than the person (almost always). Use plain .personSegmentation only when virtual content is always behind the person — a virtual background, for example — because it is cheaper and always puts people in front.
Known rough edges: fingers and hair fringe, motion blur on fast movement, and dark or low-contrast clothing. It still reads as convincing at arm's length, which is where most handheld AR happens.
7. Performance and thermals
Scene understanding is not free. On a real deployment we budget for it:
- Occlusion only, where possible. Enable
.physicsand.collisionwhen a physics interaction is actually on screen, and remove them from the options set afterwards. .receivesLightinglast. Virtual shadows on real geometry look great and are the most expensive option in the set. Test it on the oldest device you support before committing.- Watch thermals, not frame rate. A LiDAR AR session plus people occlusion will warm a phone. It typically holds 60 fps for the first few minutes and then throttles. Test a ten-minute session, not a thirty-second one — enterprise users hold a device up for a whole shift.
- Stop the session when the view is not visible.
session.pause()on background or navigation. This is the single most common battery bug we find in AR code reviews. - Do not render the debug mesh in production.
arView.debugOptions.insert(.showSceneUnderstanding)is invaluable while building and a frame-rate sink when shipped.
8. The fallback path is part of the feature
Because most iPhones in circulation have no LiDAR, design the non-LiDAR experience deliberately:
- Plane detection with both alignments. Horizontal and vertical planes get you most of the way for placement.
- People occlusion regardless. It works on A12 and later and covers the most noticeable occlusion failure — the user's own hand.
- A softer contract with the user. If physics against the room is unavailable, do not offer "throw the ball." Offer placement and inspection instead. Users forgive a smaller feature set; they do not forgive a feature that visibly misbehaves.
- Tell them what they gain. A single line — "Depth features are available on iPhone Pro models" — is better than a mysterious quality difference.
Where this pays off commercially
Every client project we have shipped with scene understanding has been in one of three buckets. Retail and furniture, where an object that clips through a real wall kills a sale. Field service and industrial, where annotations must sit on equipment with centimetre honesty and mesh classification keeps labels off windows and floors. Training and simulation, where physics against the real room is the point of the exercise.
If your AR content still floats in front of everything, that is the highest-leverage fix available to you, and it is a day of work, not a quarter.
Next steps
- Apple's scene reconstruction documentation and the SceneUnderstanding options reference.
- Pair this with Room-Scale AR with RoomPlan when you need a persistent, parametric room model rather than a live mesh.
- Still on SceneKit? Scene understanding is a RealityKit feature. Start with our 12-month migration plan.
Want senior ARKit help making AR content behave like it belongs in the room? RealityRogue provides ARKit consultants, architects, and contract developers on short or long engagements. Get in touch.