Body Tracking and Motion Capture with ARKit: Fitness, PT, and Fit Analysis That Actually Track a Person

Written against iOS 18–26 and re-checked on the iOS 27 betas. Swift and RealityKit-first — ARKit's body tracking API dates to iOS 13, but almost every tutorial you will find drives it through SceneKit, which is deprecated. Don't build there.

Face tracking gets all the attention because virtual try-on for eyewear and beauty is an easy sell. Body tracking is the quieter, harder sibling — and it is the one behind most of the AR briefs we have taken in the last year: form checking in fitness apps, range-of-motion measurement in physical therapy, apparel and PPE fit, sports technique review, and motion capture for indie animation pipelines.

It is also the ARKit feature most likely to burn a project, because teams assume it is a motion-capture system. It isn't. It's a single-person, monocular pose estimator with real and specific limits. This tutorial covers what it actually does, how to wire it up, when to reach for Vision instead, and how to scope a body-tracking feature so it survives contact with users.

What ARKit body tracking actually gives you

Turn on ARBodyTrackingConfiguration and ARKit hands you, per frame:

• One ARBodyAnchor for one person — the most prominent body in the rear camera feed. There is no multi-person 3D skeleton in ARKit. If your brief says "track the whole class," stop and read the Vision section below.
• An ARSkeleton3D on that anchor with roughly 90 joints, as transforms in model space (jointModelTransforms) or relative to each joint's parent (jointLocalTransforms).
• An estimatedScaleFactor — ARKit's guess at how big this person is relative to its default skeleton. This is an estimate, not a measurement.
• Optionally a 2D skeleton on ARFrame.detectedBody (the .bodyDetection frame semantic), in normalized image coordinates.

Requirements and limits worth knowing before you quote the work:

Constraint	Reality
Devices	A12 Bionic and later, rear camera. LiDAR helps depth and occlusion but is not required.
People tracked	One, in 3D. Switching subjects mid-session is jumpy.
Front camera	No. Body tracking is rear-camera only; the front camera does face tracking.
visionOS	Not available. Vision Pro gives you hand tracking, not third-person body tracking.
Occlusion of joints	Hidden limbs are inferred, not measured. Values keep arriving and look plausible while being wrong.
Accuracy	Good enough for animation, gesture, and coarse posture. Not a clinical goniometer. Say that out loud in the kickoff.

That last row is the one that matters commercially. We have been asked more than once for "degree-accurate" joint angles from a phone camera for a medical claim. ARKit will happily give you a number to two decimal places. Defending it to a regulator is a different project.

The minimum viable session

import ARKit
import RealityKit
import SwiftUI

struct BodyTrackingView: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let view = ARView(frame: .zero)

        guard ARBodyTrackingConfiguration.isSupported else {
            assertionFailure("Body tracking needs an A12 device or later")
            return view
        }

        let config = ARBodyTrackingConfiguration()
        config.automaticSkeletonScaleEstimationEnabled = true
        config.planeDetection = [.horizontal]     // so you can ground a character
        config.frameSemantics.insert(.personSegmentationWithDepth)

        view.session.delegate = context.coordinator
        view.session.run(config, options: [.resetTracking, .removeExistingAnchors])
        return view
    }

    func updateUIView(_ uiView: ARView, context: Context) {}
    func makeCoordinator() -> Coordinator { Coordinator() }
}

Note personSegmentationWithDepth: that is people occlusion, and it is a separate feature from body tracking. It lets real people pass in front of virtual content correctly. It is also the single biggest thermal cost you can add to an AR session — see ARKit performance and thermals before you leave it on for a twenty-minute workout.

Reading joints

The delegate is where the work happens. Joints are named, and you look them up through ARSkeleton.JointName or the raw joint names in ARSkeletonDefinition.defaultBody3D.

final class Coordinator: NSObject, ARSessionDelegate {

    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        for case let body as ARBodyAnchor in anchors {
            let skeleton = body.skeleton
            let root = body.transform          // hip/root in world space

            guard
                let hip = skeleton.modelTransform(for: .init(rawValue: "hips_joint")),
                let knee = skeleton.modelTransform(for: .init(rawValue: "left_leg_joint")),
                let ankle = skeleton.modelTransform(for: .init(rawValue: "left_foot_joint"))
            else { continue }

            let angle = jointAngle(
                a: position(hip, in: root),
                b: position(knee, in: root),
                c: position(ankle, in: root)
            )
            // Feed `angle` into a smoother, not straight into the UI.
        }
    }

    private func position(_ m: simd_float4x4, in root: simd_float4x4) -> SIMD3<Float> {
        let world = root * m
        return SIMD3(world.columns.3.x, world.columns.3.y, world.columns.3.z)
    }

    private func jointAngle(a: SIMD3<Float>, b: SIMD3<Float>, c: SIMD3<Float>) -> Float {
        let v1 = simd_normalize(a - b), v2 = simd_normalize(c - b)
        return acos(max(-1, min(1, simd_dot(v1, v2)))) * 180 / .pi
    }
}

Two things people get wrong here:

1. Model space is not world space. jointModelTransforms are relative to the body anchor. Multiply by bodyAnchor.transform before you compare anything to a real-world plane or another anchor.
2. Never show a raw per-frame angle. It jitters by several degrees at rest. Run a low-pass filter (a one-euro filter is the usual choice) and gate on ARFrame.camera.trackingState so you are not publishing numbers while tracking is .limited.

Driving a character: BodyTrackedEntity

If the goal is a puppet — an avatar mirroring the user — RealityKit does the retargeting for you. Load a rigged character and RealityKit returns a BodyTrackedEntity whose BodyTrackingComponent is driven by the session.

let character = try await Entity.loadBodyTracked(named: "robot")
character.scale = SIMD3(repeating: 1.0)

let anchor = AnchorEntity(.body)
anchor.addChild(character)
arView.scene.addAnchor(anchor)

The catch is the rig. The character's skeleton must match ARKit's biped joint hierarchy and naming, or you get a T-pose that ignores you, or a limb folded inside the torso. Budget rig work as its own line item: exporting a client's marketing mascot to a compliant USDZ is usually a day or two with a technical artist, not a drag-and-drop. Our USDZ asset pipeline guide covers the export side.

Also note BodyTrackedEntity has no collision or physics components. If you want the avatar to knock things over, drive invisible collider entities from the joint transforms yourself.

When to use Vision instead

Reach for Vision's human body pose requests, not ARKit, when:

• You need more than one person.
• You are analyzing recorded video rather than a live session.
• You only need 2D pose in image space — rep counting, "is the bar path straight," silhouette overlays.
• You need to run on devices or in contexts where an AR session is overkill.

Vision gives you multi-person 2D pose (and 3D pose from a single image, with its own caveats) with no ARKit session, no world tracking, and far less battery. A surprising share of "AR fitness" features are actually Vision features with an AR-looking overlay. That is a cheaper, more robust product — and it works on video the user already recorded. If you do combine them, our post on turning Vision and Core ML detections into stable anchors covers the handoff.

Rule of thumb: if the virtual content has to live in the room, use ARKit. If it only has to live on the screen, use Vision.

Privacy, consent, and App Review

Body tracking apps point a camera at a person's body, often in their home, often minimally dressed, sometimes for a health-adjacent purpose. Treat that seriously — reviewers and enterprise procurement teams both will.

• Ship a specific NSCameraUsageDescription. "This app uses the camera" is a rejection risk; "to analyze your movement for form feedback" is not.
• Keep it on device. Joint transforms are a few kilobytes per frame and it is tempting to stream them for server-side analysis. If you do, say so plainly, and never ship camera frames off device without explicit, separate consent.
• Do not persist video by default. Record only on an explicit user action, store in the app container, and give a one-tap delete.
• Anything health-adjacent needs disclaimer copy reviewed by someone who is not an engineer. "Not a medical device" belongs in the UI, not just the EULA.

Making it usable in a real room

The technical part is the easy half. What kills body-tracking apps in the field:

• Framing. A full-body skeleton needs the whole body in frame, which means the phone is 2.5–3 metres away, propped up, and the user cannot read the screen. Design for audio and haptic feedback — see spatial audio in RealityKit — and an onboarding flow that helps them position the phone.
• Lighting and clothing. Backlit windows and baggy clothes both degrade tracking. Detect it: if the skeleton's confidence is low or joints are flickering, coach the user instead of silently producing garbage.
• Session length. Fitness sessions run long. A phone recording video with people occlusion enabled at 60 fps will throttle. Drop to 30 fps, turn segmentation off when nothing needs occluding, and test on the oldest device you support with the phone in a case in a warm room.
• Recovery. When the person walks out of frame, ARKit keeps the anchor briefly, then removes it. Handle session(_:didRemove:) explicitly with a "step back into view" state rather than freezing the last pose.

Testing it without hiring a dancer

You do not want to re-perform a squat two hundred times to debug a rep counter.

• Record sessions with ARKit's session replay support and drive your analysis layer from the recording in CI. Our testing ARKit apps post covers the harness.
• Better still, keep the pose-analysis code pure: a function from a stream of joint transforms to angles, reps, and events. Then unit-test it against fixture arrays of transforms — synthetic ones for edge cases, recorded ones for realism — with no ARKit and no device.
• Fixture the failure modes deliberately: a dropped anchor mid-rep, a limb behind the torso, a second person walking through frame.

Scoping guidance

A realistic first slice, from our own project plans:

1. Week 1 — session, skeleton, on-screen debug overlay, device support gate, framing onboarding.
2. Week 2 — pure pose-analysis module with smoothing and confidence gating; fixture tests.
3. Week 3 — the actual product feature (rep counting, angle readout, avatar) plus audio/haptic feedback.
4. Week 4 — thermals, long-session testing, privacy copy, low-light and occlusion recovery.

Anything that says "clinical accuracy," "multiple people in 3D," or "works on any character model" needs a separate conversation before it goes in a statement of work.

---

Need a hand? RealityRogue's senior ARKit and RealityKit developers build body-tracking and motion-analysis features on client codebases — as a full app build, or as contract developers embedded in your iOS team. Get in touch with your concept, scope, or timeline and we will tell you honestly what ARKit can and cannot promise.