As the developer of an app called 3D Live Scanner that allows users to scan interiors, exteriors, objects, and faces with an Android phone, I’m very familiar with the Sketchfab platform. It is the easiest way for users of my app to show the captured data in augmented or virtual reality.
What I found to be missing on Sketchfab is the ability to assemble your own world from the available 3D models. With this idea, I participated in the Facebook XR Hackathon, where I was asked to develop a mixed reality prototype for the Oculus Quest headset in 10 days.
The requirement was to use a new Oculus Voice SDK and Hands SDK. After two days of prototyping, I discarded the idea of composing a world. It would be too complex to implement in such a short time. I decided instead to simplify the idea and use the 3D models as puppets.
How it works
The prototype uses Sketchfab’s database to access the world’s largest collection of 3D models. A simple user interface guides you through the steps and helps you register and log in to the system. Once logged in, you put your hand close to the headset and a magic ball appears. Tell it what 3D model you want to get, and within seconds you’ll have it in your hand.
Project realization
Development was done in Unity, the most developer-friendly tool for 3D games development. For this tool, there is an open-source Sketchfab integration, which made my job much easier. I reworked the Sketchfab integration to operate on the Android/Oculus Quest platform; this process was more complicated than I had expected. Since this might be a problem that many other developers have, I have provided my solution completely as open-source.
The conjuring itself is activated upon successful login to your Sketchfab account. It is implemented in a main loop that looks simple, but for a good user experience it needs to work completely asynchronously and solve all kinds of problems (no internet connection, model not found, etc.). Here is just a simplified version of the main loop:
The microphone is built into the headset, but it’s very natural to turn it on by putting anything close to your mouth. In this case, you are essentially talking into your hand, however, this approach limits silent voice recording, which would lead to a poor user experience.
For the voice recording itself, I created a magic ball. The magic ball helped me solve multiple user feedback problems. It clearly indicates if the microphone is on, and also creates a very nice effect when loading the model. The creation of the magic ball was inspired by the time travel effect in Terminator 2. My version of the effect doesn’t include the lightning outside the ball because that would be too dramatic.
The experimental Oculus Voice SDK was used for voice recognition. It works great even with my heavy accent. The problem was that the system very often recognized “Art Simpson” instead of “Bart Simpson”. The beginning of the voice recording was missing. This was solved with a small tweak to the Voice SDK configuration.
Recognized text is automatically entered into the Sketchfab model search. To prevent performance issues, models with too-complex geometry or too-large textures were filtered out. From all search results, I select the one with the shortest name. This turned out to be the best solution for obtaining the desired model. Of course there are exceptions, e.g., if you search for “table” then you get “holo-table”. In this case, you have to specify the type of table, e.g., “dinner table”.
Once the 3D model of the object is ready, it needs to be fit to the hand. The size of the object is calculated and rescaled to 20 cm. For some models, the center/pivot point is not set correctly, so I have to re-center the object. Additionally, I use the Hand SDK to get the position of the thumb and index finger. This is used to calculate the orientation of the model. Then the user gets the feeling that the model is actually on his/her hand.
Conclusion
This project is just a demo. It would be possible to make much more out of it, e.g., add shared experiences, let users do more with objects, and so on. Unfortunately, there are two major impediments. One is that the Sketchfab Download API has daily limits, which would be problematic for advanced use. The other impediment is that there is no easy way to monetize this project in order to continue development.
About the author
Luboš Vonásek
Not a startup, just making revolutionary software in my free time.
