The Cleveland Museum of Art and ArtLens Exhibition
The Cleveland Museum of Art (CMA), located in Cleveland, Ohio, USA, is renowned for the quality and breadth of its collection, which includes more than 61,000 objects and spans 6,000 years of achievement in the arts. As an international leader in innovative, iterative digital initiatives, CMA opened ArtLens Exhibition, in September 2017. ArtLens Exhibition represents one component of ARTLENS Gallery which also includes ArtLens Studio, ArtLens App, ArtLens Wall, and the Digital Beacon. Using gesture-sensing, eye-tracking, and facial-recognition technology, ArtLens Exhibition aims to provide visitors with the toolsets to understand how to look at art in any context.
The latest iteration of ArtLens Exhibition opened in June 2019 with 21 new masterworks of art from across the CMA’s permanent collection. An exciting new feature of this installation is the incorporation of a 3D model viewer into the existing projection interactives. For the first time, we used photogrammetry to create 15 3D models that visitors can manipulate and zoom using gesture-based Kinect technology. The CMA’s Digital Innovation team led this new initiative and the 3D models were created by CMA’s Chief Photographer, Howard Agriesti. Visitors can explore artworks in CMA’s collection like never before—turning them digitally around to see the tops, bottoms, and insides. As an inherently iterative space, ArtLens Exhibition seeks to push the boundaries of innovation, providing visitors with the tools to look closer at art. We pursued this feature because feedback from visitors and CMA staff indicated this was a desired experience of the interactives. After visitors engage with the 3D model, they can immediately play the artwork’s next assigned game to learn more and dive deeper into the object.
After researching several different approaches to incorporating our 3D models into the games, we decided to integrate the Sketchfab platform into our existing interactives. The decision was largely based on the sophistication of the interface and level of control for modifying and improving the models.
Artwork Selection Process and Photogrammetry
As the collaborative Digital Innovation team worked together with curators to select the artwork for the reinstallation, it was our goal that all 3D artworks would be photographed using photogrammetry. Photogrammetry is the process of using specialized cameras to capture a “point cloud” representation and texture map of a 3-dimensional object and recombine them as a 3D model. This concept was new to the institution and represents another way in which CMA continues to advance what is possible in the museum field. The previous iteration of ArtLens Exhibition used 360 photography for the “attract loops” in the projection interactives. The attract loop is the initial animation that plays with the intention of drawing the visitor in to interact with the game. Photogrammetry represents the next level of innovation. All 3D models were produced in our museum’s photo studio and we began the process over a year before ArtLens Exhibition reopened in June 2019.
Creating the Interactive
Our team partnered with Potion Design, who led the UX/UI development for incorporating the Sketchfab viewer into our existing gesture-sensing interactives. CMA spent a significant amount of time fine-tuning the visitor experience so that users could interact with the 3D models in a rewarding way. An important part of successfully interacting with the models includes advising visitors to use only one hand—all other existing gesture-sensing games use only one hand. To communicate this, we created an “onboarding” sequence that directs users to drag an icon from one point to another across the object. Users can then grab the object anywhere to rotate or use the zoom icon to enlarge it. When the timed 3D viewer experience is complete, the object’s next game begins seamlessly.
Within ArtLens Exhibition, there are six gesture-based projection interactives featuring an attract loop of digital projections of artworks from the gallery. After selecting an artwork from the attract loop, the Sketchfab viewer is loaded into a webview in the application, and the artwork appears to zoom in and rotate. This is actually achieved by zooming in the entire web view itself and then rotating the object within Sketchfab, providing the seamless zoom and rotate introduction. Once the object is scaled to size for interaction, some simple controls appear. These controls are simply Cinder UI elements that are rendered over the web view. The user can then manipulate and control by grabbing, dragging, and releasing gestures which are captured by the Kinect camera. The grab, drag, and release gestures are translated by our app into Sketchfab API calls to rotate, tilt, or zoom the object. The object can rotate a full 360°, but we decided to limit the tilting or vertical rotation of the object to 80°. In the context of the interaction, tilting the object upside-down and then rotating it was not useful and too disorienting.
The ArtLens Exhibition games are applications coded in C++ using the Open Source creative coding library Cinder. The web view uses the Chromium Embedded Framework (CEF) and loads a custom html page containing the Sketchfab viewer and API code. The app is configurable to load the Sketchfab models of the artwork all at once or one at a time, i.e., “on demand”. Through testing, we found that loading the models one at a time delivered better performance and frame rate.
Generation of Artwork 360s
In the attract loop for the application, sculptures and other 3D objects have a 360° animation. Previously, these objects were meticulously photographed and stitched together one by one. When introducing the Sketchfab models, we were able to use the API to generate these 360s. We created a javascript tool which allows us to load a model, select some parameters such as transparent or black background, or JPG or PNG, and export still images of the model. This tool uses the Sketchfab API to rotate the object and export screen captures so that we could create the required 750 still images for the 360° rotation the application required. From these still rotation images, we could either convert to KTX format to use in the ArtLens games, or use a tool like FFmpeg to create video files for use elsewhere. In order to ensure that the transitions between the 360s were seamless, we actually use stills from Sketchfab rather than actual artwork photography so that the stills and the 3D views match.
Looking Forward
Since ArtLens Exhibition reopened, we have witnessed visitors of all ages interacting with our new 3D models. We are very grateful to have partnered with Thomas Flynn and Dale Utt, who assisted with the lighting and editing of our 3D models in Sketchfab—their expertise and enthusiasm contributed significantly to the success of this project. The next iteration of ArtLens Exhibition will open in the next 18-24 months, but in the meantime, our team will continue to produce more 3D models from our collection as time permits. Finally, look for links to our Sketchfab models on our Collection Online later this summer.
CMA Website / CMA Facebook / CMA Instagram / CMA Twitter
About the author
Ethan Holda and Maddie Armitage
Ethan Holda is the Director of Technology at the Cleveland Museum of Art and oversaw the technical integration of the Sketchfab 3D models into ArtLens Exhibition. Maddie Armitage is the Cleveland Museum of Art’s ARTLENS Gallery Fellow and managed the photogrammetry process.
