Hi guys I'm currently developing a game for the Vision Pro and i'm trying to figure out how the hand tracking works so I can make a superpower appear when the user looks at their hand and widens it. But im really struggling to wrap my head around the whole concept and how to implement it in my code. Is there anything out there (other than apple doc) or anyone who could help me shed some light on the whole idea and how I could actually usefully implement it? would be much appreciated Thanks

I have been experimenting some experiences in which I would like to use SharePlay to allow the app to be used by multiple users. Currently I achieved sharing a volume containing a Reality Composer Pro scene inside of it, the scene contains some entities with an animation. So far I have been able to correctly share the volume and its content, with the animation playing without problems, but once I activate SharePlay different users see different moments of the animation if no animation at all. Is there a way to synchronize animations between all the users, no matter when someone entered the SharePlay session, aside from communicating the animation time once someone joins?

There is a flickering and slight dimming occurring specifically on skysphere, at initial load of the scene, when using Attachment. This is observed in the simulator and on the real device. Since we cannot upload a video illustrating the undesirable behaviour, I have to describe how to setup the project for you to observe it. To replicate the issue, follow these steps: Create a new visionOS app using Xcode template, see image. Configure the project to launch directly into an immersive space (set Preferred Default Scene Session Role to Immersive Space Application Session Role in Info.plist), see image. Replace all swift files with those you will find in the attached texts. Add the skysphere image asset Skydome_8k found at this Apple Sample App Presenting an artist’s scene. Launch the app in debug mode via Xcode and onto the AVP device or simulator Continuously open and dismiss the skysphere by pressing on buttons Open Skysphere and Close. Observe the skysphere flicker and dim upon display of the skysphere. The current workaround is commented in file ThreeSixtySkysphereRealityView at lines 65, 70, 71, and 72. Uncomment these lines, and the flickering and dimming do not occur. Are we using attachments wrongly? Is this behavior known and documented? Or, is there really a bug in visionOS? AppModel InitialImmersiveView MainImmersiveView TestSkysphereAttachmentFlickerApp ThreeSixtySkysphereRealityView

When I first install and run the app, it requests authorization for hand tracking data. But then if I go to the settings and disable hand tracking from the app, it no longer requests. The output of requestAuthorization(for:) method just says [handTracking : denied] Any idea why the push request only shows up once then never again?

We are using the ARKit image tracking feature on visionOS 2.0 with three pre-registered images. The image tracking works, but only one image is actively tracked at a time. When more than one target image is visible to the camera, it has difficulty detecting and tracking the other images. Is this the expected behavior in visionOS, or is there something we need to do to resolve this issue?

We are currently working with the Enterprise APIs for visionOS 2 and have successfully obtained the necessary entitlements for passthrough camera access. Our goal is to capture images of external real-world objects using the passthrough camera of the Vision Pro, not just take screenshots or screen captures. Our specific use case involves: 1. Accessing the raw passthrough camera feed. 2. Capturing high-resolution images of objects in the real world through the camera. 3. Processing and saving these images for further analysis within our custom enterprise app. We would greatly appreciate any guidance, tutorials, or sample code that could help us achieve this functionality. If there are specific APIs or best practices for handling real-world image capture via passthrough cameras with the Enterprise APIs, please let us know.

I am new to learning about concurrency and I am working on an app that uses the HandTrackingProvider class. In the Happy Beam sample code, there is a HearGestureModel which has a reference to the HandTrackingProvider() and this seems to write to a struct called HandUpdates inside the HeartGestureModel class through the publishHandTrackingUpdates() function. On another thread, there is a function called computeTransformofUserPerformedHeartGesture() which reads the values of the HandUpdates to determine whether the user is making the appropriate gesture. My question is, how is the code handling the constant read and write to the HandUpdates struct?

In WWDC24, visionOS hand tracking has a new function that can make an entity track the hand faster (but at the expense of a certain degree of accuracy), and the video only explains how to implement ARKit, so please ask how to implement the anchorEntiy in the reality view.

In visionOS, the virtual content is covered by the hand by default, so I want to know that in the hybrid space, if the distance of an entity is behind a real object, how can the object in the room be covered like the virtual content is covered by the hand?

There is a flickering occurring on 3D assets when switching immersive spaces, which is not the nicest user experience. The flickering does occur either when loading the scenes directly from the RealityKitContent package, or from memory (pre-loaded assets). Since we cannot upload a video illustrating the undesirable behaviour, I have to describe how to setup the project for you to observe it. To replicate the issue, follow these steps: Create a new visionOS app using Xcode template, see image. Configure the project to launch directly into an immersive space (set Preferred Default Scene Session Role to Immersive Space Application Session Role in Info.plist), see image. Replace all swift files with those you will find in the attached texts. In the RealityKitContent package, create a scene named YellowSpheres as illustrated below. In the RealityKitContent package, create a scene named RedSpheres as illustrated below. Launch the app in debug mode via Xcode and onto the AVP device or simulator Continuously switch immersive spaces by pressing on buttons Show RedSpheres and Show YellowSpheres. Observe the 3d assets flicker upon opening of the immersive spaces. AppModel RedSpheresImmersiveView YellowSpheresImmersiveView TestFlickeringBetweenImmersiveSpacesApp

Hello! I would like to do exactly this: https://youtu.be/Cun8K7ctKp0?si=TgWvtdw-VdlBVL0R I can't seem to find any documentation on getting a PS5 controller hooked up properly in Reality Composer Pro and Xcode, driving a character with animation (or, moving an object around freely) then over to the Vision Pro. Additionally, I would also like to learn how to use a controller to move a VR camera around a scene, so that we can navigate in custom built spaces - similar to Meta virtual environments, or Steam VR home environments. Typically, I would do this with Unreal Engine, but unfortunately, AVP support is still in its infancy there. So I figured, why not try to do it natively? Any help with concrete tutorials or documentation would be greatly appreciated. Thx!

While trying to control the following two scenes in 1 ImmersiveSpace, we found the following memory leak when we background the app while a stereoscopic video is playing. ImmersiveView's two scenes: Scene 1 has 1 toggle button Scene 2 has same toggle button with a 180 degree skysphere playing a stereoscopic video Attached are the files and images of the memory leak as captured in Xcode. To replicate this memory leak, follow these steps: Create a new visionOS app using Xcode template as illustrated below. Configure the project to launch directly into an immersive space (set Preferred Default Scene Session Role to Immersive Space Application Session Role in Info.plist. Replace all swift files with those you will find in the attached texts. In ImmersiveView, replace the stereoscopic video to play with a large 3d 180 degree video of your own bundled in your project. Launch the app in debug mode via Xcode and onto the AVP device or simulator Display the memory use by pressing on keys command+7 and selecting Memory in order to view the live memory graph Press on the first immersive space's button "Open ImmersiveView" Press on the second immersive space's button "Show Immersive Video" Background the app When the app tray appears, foreground the app by selecting it The first immersive space should appear Repeat steps 7, 8, 9, and 10 multiple times Observe the memory use going up, the graph should look similar to the below illustration. In ImmersiveView, upon backgrounding the app, I do: a reset method to clear the video's memory dismiss of the Immersive Space containing the video (even though upon execution, visionOS raises the purple warning "Unable to dismiss an Immersive Space since none is opened". It appears visionOS dismisses any ImmersiveSpace upon backgrounding, which makes sense..) Am I not releasing the memory correctly? Or, is there really a memory leak issue in either SwiftUI's ImmersiveSpace or in AVFoundation's AVPlayer upon background of an app? App file TestVideoLeakOneImmersiveView First ImmersiveSpace file InitialImmersiveView Second ImmersiveSpace File ImmersiveView Skysphere Model File Immersive180VideoViewModel File AppModel

I have two Apple Vision Pros so that I can make and test a multi-player immersive reality game. But I am one developer, so I need to be able to take one Apple Vision Pro off, and put the other one on to see what the other device is seeing, and to ensure my game information is correctlly being sent over the network with multipeer connectivity. But when I take one off, the Apple Vision Pro immediately goes to sleep. With Apple Vision Pro OS 1, I could put a piece of paper into the pro and they would stay on for hours, and I could take them on and off and debug my game. But now with VisionOS 2, even with the paper they soon go to sleep. Is there a setting I can change or override as a developer to stop this auto sleep or auto lock? I need to check things like: when two devices are on the network, can I see them both so that players can select each other from a menu? can i send object positions back and forth Thank you.

I'm porting over some code that uses ARKit to Swift 6 (with Complete Strict Concurrency Checking enabled). Some methods on ARSCNViewDelegate, namely Coordinator.renderer(_:didAdd:for:) among at least one other is causing a consistent crash. On Swift 5 this code works absolutely fine. The above method consistently crashes with _dispatch_assert_queue_fail. My assumption is that in Swift 6 a trap has been inserted by the compiler to validate that my downstream code is running on the main thread. In Implementing a Main Actor Protocol That’s Not @MainActor, Quinn “The Eskimo!” seems to address scenarios of this nature with 3 proposed workarounds yet none of them seem feasible here. For #1, marking ContentView.addPlane(renderer:node:anchor:) nonisolated and using @preconcurrency import ARKit compiles but still crashes :( For #2, applying @preconcurrency to the ARSCNViewDelegate conformance declaration site just yields this warning: @preconcurrency attribute on conformance to 'ARSCNViewDelegate' has no effect For #3, as Quinn recognizes, this is a non-starter as ARSCNViewDelegate is out of our control. The minimal reproducible set of code is below. Simply run the app, scan your camera back and forth across a well lit environment and the app should crash within a few seconds. Switch over to Swift Language Version 5 in build settings, retry and you'll see the current code works fine. import ARKit import SwiftUI struct ContentView: View { @State private var arViewProxy = ARSceneProxy() private let configuration: ARWorldTrackingConfiguration @State private var planeFound = false init() { configuration = ARWorldTrackingConfiguration() configuration.worldAlignment = .gravityAndHeading configuration.planeDetection = [.horizontal] } var body: some View { ARScene(proxy: arViewProxy) .onAddNode { renderer, node, anchor in addPlane(renderer: renderer, node: node, anchor: anchor) } .onAppear { arViewProxy.session.run(configuration) } .onDisappear { arViewProxy.session.pause() } .overlay(alignment: .top) { if !planeFound { Text("Slowly move device horizontally side to side to calibrate") } else { Text("Plane found!") .bold() .foregroundStyle(.green) } } } private func addPlane(renderer: SCNSceneRenderer, node: SCNNode, anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor, let device = renderer.device, let planeGeometry = ARSCNPlaneGeometry(device: device) else { return } planeFound = true planeGeometry.update(from: planeAnchor.geometry) let material = SCNMaterial() material.isDoubleSided = true material.diffuse.contents = UIColor.white.withAlphaComponent(0.65) planeGeometry.materials = [material] let planeNode = SCNNode(geometry: planeGeometry) node.addChildNode(planeNode) } } struct ARScene { private(set) var onAddNodeAction: ((SCNSceneRenderer, SCNNode, ARAnchor) -> Void)? private let proxy: ARSceneProxy init(proxy: ARSceneProxy) { self.proxy = proxy } func onAddNode( perform action: @escaping (SCNSceneRenderer, SCNNode, ARAnchor) -> Void ) -> Self { var view = self view.onAddNodeAction = action return view } } extension ARScene: UIViewRepresentable { func makeUIView(context: Context) -> ARSCNView { let arView = ARSCNView() arView.delegate = context.coordinator arView.session.delegate = context.coordinator proxy.arView = arView return arView } func updateUIView(_ uiView: ARSCNView, context: Context) { context.coordinator.onAddNodeAction = onAddNodeAction } func makeCoordinator() -> Coordinator { Coordinator() } } extension ARScene { class Coordinator: NSObject, ARSCNViewDelegate, ARSessionDelegate { var onAddNodeAction: ((SCNSceneRenderer, SCNNode, ARAnchor) -> Void)? func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { onAddNodeAction?(renderer, node, anchor) } } } @MainActor class ARSceneProxy: NSObject, @preconcurrency ARSessionProviding { fileprivate var arView: ARSCNView! @objc dynamic var session: ARSession { arView.session } } Any help is greatly appreciated!

When using the plane PlaneDetectionProvider in visionOS I seem to have hit a limitation which is that regardless of where the headset is in the space, planes will only be detected that are (as far as I can tell) less that 5m from the world origin. Mapping a room becomes very tricky as a result because you often find some walls are outside the radius, even if you're standing two feet away from a ten foot wall. It just won't see it. I've picked my way through the documentation but I cannot see any way to extend this distance. Am I missing something?

Does anyone have experience of creating their own EntityActions? Say for example I wanted one that faded up the opacity of an entity, then once it had completed set another property on one of the entity's components. I understand that I could use the FromToByAction to control the opacity (and have this working), but I am interested to learn how to create my own dedicated EntityAction, and finding the documentation hard to fathom. I got as far as creating a struct conforming to EntityAction protocol: var animatedValueType: (any AnimatableData.Type)? } Subscribing to update events on this: FadeUpAction.subscribe(to: .updated) { event in guard let animationState = event.animationState else { return } // My animation state is always nil, so I never get here! let newValue = \\\Some Calc... animationState.storeAnimatedValue(newValue) } And setting it up as an animation on an entity: let action = FadeUpAction() if let animation = try? AnimationResource.makeActionAnimation( for:action, duration: 2.0, bindTarget: .opacity ) { entity.playAnimation(animation) } ...but haven't been able to understand how to extract the current timeDelta or set the value in the event handler. Any pointers?

My name is Tom Shannon, a developer with Omnia (d.b.a Aequilibrium Inc.). We were recently approved for some of the Enterprise APIs for the Vision Pro. You can reference the history through our Case-ID: 9237594 We are contacting you for assistance as we have downloaded the entitlement license provided and added it to our target for an application under the bundle id: com.omnia.spatialbrowser Then under my project and with my developer account, which is under the Aequilibrium Inc. account (279PV9XKZ2), we tried to add the Barcode Scanner Enterprise API entitlement, but this does not show up as an option for us. I am on XCode 16.1 beta (16B5001e) for reference! Any help would be greatly appreciated. Best,

I have an app with a visionOS target, and I want to add an iOS target. Both are based on RealityKit. I want to use a SpatialTapGesture to get the tap coordinate local to the entity tapped. In visionOS this is easy: SpatialTapGesture(coordinateSpace: .local) .targetedToAnyEntity() .onEnded { tap in let entity = tap.entity let localPoint3D = tap.convert(tap.location3D, from: .local, to: entity) // … } However, according to the docs, the convert function seems to exist only in visionOS, not in iOS. So how can I do this conversion in iOS? PS: This was already posted on StackOverflow without success. There, I tried to find a workaround, but I failed.

I am working on a React Native app, specifically on the iOS native module with RealityKit. An apparently unexplainable error keeps happening at runtime, as you can see from the following image: Crash log from XCode When I try to retrieve the position of my AnchorEntity relative to the world space (so using relativeTo: nil), it triggers a runtime exception during some of its internal calls: CoreRE: re::BucketArray<unsigned short*, 32ul>::operator[](unsigned long) + 204 As you can see from the code, my AnchorEntity is not null as there is a guard check. I also tried to move that code into an objective c static function in order to use @try @catch and catch runtime exceptions, to later realise that RealityKit is not compatible with ObjectiveC. Do you have any idea/suggestion on how to fix it/prevent it?

Hi, I have a video player app that lost its audio spatialization since the VisionOS 2 update. I am using the VideoPlayerComponent (https://developer.apple.com/documentation/realitykit/videoplayercomponent), to implement my videos as entities, as I want a custom look and controls to my player. In VisionOS 1, there was automatic audio spatialization. Depending where my video entity is, the app automatically enables head tracking audio spatialization. Since VisionOS 2 however, I cannot get my video entities to play Spatial Audio. I've looked into DestinationVideo and even set up AVAudioSessionSpatialExperience but Spatial Audio is still not working. Appreciate any help. Thanks.