Home-made Arcade
(on-going)
//XR Prototype

Story

Plan

Current progress

Learning

My main learning resources are Apple sample code. At the same time, AI also helps a lot. I saved a lot time by chatting with AI tools, letting them explaining the logic and structure of the code. For the past three months I went through all the VisionOS sample code for my functionality, and three of them provides great insights for me.

- 'CreatingASpatialDrawingAppWithRealityKit' taught me how to use AnchorEntity and SpatialTrackingSession to do hand-tracking with RealityKit.
- 'SceneReconstruction' provided a great way to recognize surroundings and create mesh along the way.
- 'SimulatingPhysicsWithCollisionsInYourVisionOSApp' gave me a sense of entity-component system and using this system to apply physical simulation to virtual objects.

Apart from the code, math, especially linear algebra, is another chuck of knowledge that I need to acquire. Fortunately, this part of the knowledge has been covered when I took the Coursera course.

Building

To build the complete experience, I split the task into the following sub-tasks and plan to complete them in order:
✅ 1. Placing an virtual arcade basketball machine into the space
✅ 2. Setting different collision shapes for the machine
✅ 3. Simulating a basketball in VisionOS
⭕ 4. Tracking hands
⭕ 5. Ball handling with hand, shooting ball with hand
⭕ 6. Game logics, like score calculation, timers and sound effects.

Here I show the AppModel of this project, the part that control the data flow.

1@MainActor
2@Observable
3class AppModel {
4    let immersiveSpaceID = "ImmersiveSpace"
5    enum ImmersiveSpaceState {
6        case closed
7        case inTransition
8        case open
9    }
10    var immersiveSpaceState = ImmersiveSpaceState.closed
11
12    let session = ARKitSession()
13    // let handTracking = HandTrackingProvider()
14    let sceneReconstruction = SceneReconstructionProvider()
15
16    var contentEntity = Entity()
17
18    private var meshEntities = [UUID: ModelEntity]()
19
20    var errorState = false
21
22    /// Setup the content entity with a ball and a basketball machine
23    func setupContentEntity() async -> Entity {
24        let ball = Entity.ball()
25        let rim = await Entity.createRim()
26        if let scene = try? await Entity(named: "scene_basketball_machine", in: realityKitContentBundle) {
27            scene.position = SIMD3<Float>(x: 0, y: 0, z: -0)
28            
29            // Rim entity setup
30            rim.setPosition(SIMD3(x: 0, y: 1.87, z: -3.06), relativeTo: nil)
31            rim.orientation = simd_quatf(angle: .pi/2, axis: SIMD3<Float>(1, 0, 0))
32            scene.addChild(rim)
33            
34            contentEntity.addChild(ball)
35            contentEntity.addChild(scene)
36        }
37        return contentEntity
38    }
39
40    /// For the caller to validate if space tracking can be operated
41    var dataProvidersAreSupported: Bool {
42        SceneReconstructionProvider.isSupported
43    }
44    var isReadyToRun: Bool {
45        sceneReconstruction.state == .initialized
46    }
47
48    /// Update the scene reconstruction meshes as new data arrives from ARKit
49    func processReconstructionUpdates() async {
50        for await update in sceneReconstruction.anchorUpdates {
51            let meshAnchor = update.anchor
52
53            guard let shape = try? await ShapeResource.generateStaticMesh(from: meshAnchor) else { continue }
54
55            switch update.event {
56            case .added:
57                let entity = ModelEntity()
58                entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform)
59                entity.collision = CollisionComponent(shapes: [shape], isStatic: true)
60                entity.components.set(InputTargetComponent())
61
62                entity.physicsBody = PhysicsBodyComponent(mode: .static)
63
64                meshEntities[meshAnchor.id] = entity
65                contentEntity.addChild(entity)
66            case .updated:
67                guard let entity = meshEntities[meshAnchor.id] else { continue }
68                entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform)
69                entity.collision?.shapes = [shape]
70            case .removed:
71                meshEntities[meshAnchor.id]?.removeFromParent()
72                meshEntities.removeValue(forKey: meshAnchor.id)
73            }
74        }
75    }
76}

Testing mouse shooting on Vision Pro simulator

Testing on the Vision Pro hardware