Connectome: A game of points and challenges to build Apple Vision Pro

Connectome: A Game of Points is an immersive art experience on MetaQuest and Apple Vision Pro, using hand tracking on both platforms and eye tracking on Vision Pro. Developer Grant Hinkson describes the pain points of getting his application across the finish line on Apple’s headset.

Hinkson and I met at AWE in June for a Connectome demo on Apple Vision Pro. UploadVR met Hinkson last year at AWE for a demo of the same game on Meta Quest shortly after its May 2025 release. At the time, Hinkson expressed interest in bringing the game to Apple’s Vision Pro.

The Connectome demo in AWE shows how connecting the dots can be spatial

Connectome on Quest is a meditation version of Connect the Dots.

As with Don Hopper’s experience last year, I was mesmerized by the elegant simplicity of the app, enhanced by AVP’s eye tracking to realize what I was doing. In practice, CONNECTOME is a straightforward series of connect-the-dots tasks with hand tracking to create different patterns and progress further through the world, in a soothing spatial environment set to a Zen musical score. It’s the kind of app that makes time disappear. It felt like I was only on the app for a few minutes when I was over twenty.

After my demo, I asked Hinkson a simple question: What were the challenges of creating the same app on Apple Vision Pro and MetaQuest?

Hinkson ran into a number of obstacles he hadn’t anticipated due to the lack of integration of some of the tools and libraries available to Vision Pro in Unity. I’ll do my best in this next section to avoid getting too deep into the weeds.

Connectome’s primary driver is a Unity library called Shapes, built by Freya Holmer and which provided Hinkson’s first AVP constraint. Unlike MetaQuest, Shapes is not compatible with polyspatial, Unity’s default toolkit builds for visionOS. Renders polyspatially with kit that does not accept custom shaders used by shapes. Hinkson found a solution to this by switching the Unity app mode to another rendering service, Metal. It solved one problem and created another.

Metal apps built into Unity don’t offer the eye-based hover effects that are a fundamental part of Apple Vision Pro’s user experience. In layman’s terms, this is what you see in an eye-tracked headset when the cursor moves to whatever icon you’re looking at. It’s not Apple’s fault that it introduced an API for Metal developers at its annual WWDC (Worldwide Developers Conference) in 2025. Instead, the gap here is with Unity, who doesn’t implement support for it. For its part, Unity has acknowledged this and has confirmed that it does not plan to add integration.

Faced with this new obstacle, Hinkson created a custom bridge from scratch that drilled down to Unity’s own composer level, recorded the necessary eye tracking information at each frame, and wrote the setting to visionOS to deliver the expected floating effect. Unfortunately, since Unity rebuilds its visionOS at every stage, this solution is destroyed every time a new build of the application is created. So a post-build patcher should be created that runs after the Unity export is complete.

Then, there were the colors. Each room in CONNECTOME has a hue as its anchor point, making it one of the best areas to move from room to room. For this, a second bridge had to be built because SwiftUI has APIs not exposed by Unity. Another post-build patcher was necessary to ensure that the Unity view maintained SwiftUI changes.

Hinkson’s experience illustrates the challenges of developing a new platform and provides some insight into why app development for the new XR devices is taking longer than consumers might expect. Obstacles like the ones described here aren’t known until developers get knee-deep in these projects and run full speed into the proverbial brick walls. Only then, after exploring alternative solutions including building from scratch, do these projects see the light of day.

As stated earlier, Apple was at least aware of these obstacles and provided tools to help developers. Meta has a long-standing partnership with Unity and has several software development tools for Unity and the Unreal Engine. Snap is doing the same with its upcoming glasses, making every effort to ensure teams have what they need, but it only goes so far. Without the full integrated support of Unity, Unreal, Godot and other engines, developers will continue to find themselves looking for workarounds due to the lack of said integrations.

As XR technology becomes more ubiquitous and the demand for more robust applications grows, these game engines will either have to force their hand or developers like Hinkson will make their custom tools and libraries publicly available, sometimes at a price, to bridge the gaps. Until adoption reaches that point, developers for new hardware platforms will continue to face these issues and spend significant time and resources reinventing the wheel to stay on the cutting edge of new technology.

CONNECTOME is now available on Apple Vision Pro and Meta Quest.

#Connectome #game #points #challenges #build #Apple #Vision #Pro

Leave a Comment