3D Artists Kathrin Lipski and Christian Lipski stop by to share some hot tips on how to streamline your photogrammetry workflow. This article was originally published on Christian’s website.
For our latest personal project we experimented with combining traditional scale modeling and digital tools. We wanted to create a lighthouse on a floating island. We have been playing with this concept for quite a while and decided to go for a look that is reminiscent of scale modeling and miniature dioramas. The first idea was to build a scale model and add a few hand modeled 3D objects. After some failed attempts of putting real world bricks together, we decided to scan the individual building blocks and put them together digitally. We build some cube-shaped bricks from salt dough and painted them with some long forgotten techniques of ink washing and dry brushing we know from miniature painting. For the plants on the island we strove for a look that is somewhere between realistic and fantastic. We scanned mushrooms, ginger, an artichoke, and a leaf and used some 2D image manipulation to give them a greenish bioluminescent look.
We used photogrammetry to digitize the real world objects and were faced with the daunting task to scan 30 individual objects for the virtual diorama. Having never scanned so many objects for a single project, we decided to revisit some steps of our photogrammetry workflow and share the results with you. All 3D scans were performed on a simple turntable stage, so these tips are all geared towards a studio setup where you can control the position of the object, the lighting and the background (and by studio I mean occupying our kitchen table for several days). You’ll find both the individual PBR (physics-based rendering) assets and the final result of this project on Sketchfab, We release them under the very permissive CC-BY license and would be thrilled if they help you realize your own ambitious project.
We used both RealityCapture and MetaShape for the scans. While a few tricks are specific to these individual software tools, all results should be easy to reproduce with other photogrammetry software packages like Zephyr or Meshroom. We assume some basic familiarity with your tool of choice and won’t explain basic operation of your tool. If you are a photogrammetry rookie (and are still reading this article) we applaud your determination but would like to encourage you to also have a look at some more basic tutorials to close any knowledge gaps.
1. Foreground Separation
We think that having decent lighting, an appropriate background and a turntable (even if it’s not motorized) is just as important as having good camera and lens equipment. Since some light bulbs and white copy paper cost much less than a fancy DSLR, you should definitely invest in this area before getting a camera upgrade. First of all you should get some decent lights that do not cast strong shadows, we suggest getting two soft boxes that illuminate the objects from all sides. We highly recommend using a turntable for your scans, a manual one (i.e. one that isn’t motorized) will do just fine. Put marks in regular intervals on the turntable. We marked 80 possible positions ours so we can easily capture 80, 40 or 20 photos during a full 360 revolution. For ‘easy’ objects we tend to capture less photos. For more challenging ones we use all 80 available positions. We use a tripod for all studio shoots.
For the background, pick anything that does not have a strong texture and is either way brighter or darker than your scanned object (avoid green screens, they introduce color spill that you won’t enjoy having on your object). If foreground and background are distinguishable by brightness values, it will be much easier to process the scan (see the texture swapping section). We always recommend going for a white background if possible: A white background will allow for more light to be bounced back to your object and produce cleaner scenes.
2. Maximum Overlap
In order to get best results, make sure that every photo has sufficient ‘overlap’ with the rest of the data. By that we mean that there should be multiple other photos that see the same part of the object from roughly the same distance and angle.
When scanning the 16 salt dough bricks, we realized that manually moving the camera to get a different angle of the object is just too time consuming. It proved to be much faster to take a brick off the capture stage and rotate it so that the front facing side is now facing up (and vice versa). This way there is still enough visual overlap (both sides where visible in the previous photo) and we could continue without moving around and refocusing the DLSR camera.
Some objects with elongated or flat shapes are challenging to scan because they offer a lot of visible texture from one side but are hardly visible from another angle. Typical elongated objects are the pieces of wood we scanned. When photographed from the side, the object takes up a lot of space in the image and allows the software to pick up a lot of visual information. But when looking directly onto its ends, much less surface area can be observed and processed by your photogrammetry tool. Here it helps to shoot images at shorter intervals. In our setup we typically capture a photo after revolving the turntable by 9 degrees (40 photos for a full revolution). For those challenging parts we took a picture every 4.5 degrees.
An example of a flat object is a leaf: It shows a lot of visual information when taking picture from its front or back. When looking at it sideways, it covers only a few pixels in the image and your photogrammetry software will have a very hard time reconstructing it at all. You can also think about introducing other objects into the scene that can be removed later from the model (but help giving the photogrammetry software more context). This way there’s always at least one objects that provides enough visual information. We used this strategy when designing the capture stage in the Playroach project. When scanning the leaf, we captured the front and back separately, the leaf was just to wobbly to put it upright. The drawback was that we had to manually align the two scan in blender later on.
3. Texture Swapping
For this trick you need to familiarize yourself with the different workflow stages of photogrammetry software. For a typical scanning session the same image footage can be used for all stages: In some cases a deeper understanding of how data is processed can come in handy: There are three main stages.
First, the software is figuring out how the cameras are placed relative to the object (“Alignment Phase”). Since some surface points of the object are reconstructed as a by-product, this is sometimes called the “Sparse Reconstruction Phase”. For this phase the scanned objects has to show some visible texture on all captured photos. For best results in this phase make sure that all image content is part of the foreground object OR moves consistently with it. When we took images of the bricks, we would take the brick off the capture stage after a full 360 degree rotation, turn them around and put them back on the capture stage. As you might remember, this made it easy to ensure that there’s enough visual overlap between photos while making it possible to see the bricks from all sides. However, it is crucial to exclude as much of the background texture from the actual photographs as possible. Otherwise the photogrammetry software might be tricked into thinking that we want to reconstruct the white paper and that the gray brick is just a background object (instead of the other way around). We used the “convert” and “mogrify” command line tools from ImageMagick to modify whole batches of images with a single command. We increased the brightness so most part of the light background would be a uniform (255,255,255) white color. This way the photogrammetry software will not find any visible information in this area. You might also want to experiment with other ways to automatically produce foreground masks (e.g. with the threshold command), but we found the brightness modification to be the quickest solution.
The second phase is the “Dense Reconstruction” phase. Here the software reconstructs the actual surface geometry of your object. For this part it is crucial that each part of the surface is visible in high detail in at least two photographs (to enable stereo matching). Make sure that all foreground objects are well exposed (if you excessively boosted the brightness during the “Alignment” phase swap those images back for their original well-exposed versions). MetaShape explicitely allows swapping textures (right click an image and select “Change Path…”). In RealityCapture you have to close the applications, manually delete the temporary cache, swap out the image files on your hard drive and start the software again. If the photos still lack texture in certain areas, you can apply simple tricks: At the end of the capture session do some additional photographs and apply some (coffee) powder or other visible texture (make-up or removable paint) onto the object surface. Make sure not to use these ‘modified’ images for the “Texturing” phase.
The third phase is the “Texturing” of your object. The images from all different views contribute to a single texture of the object. Here it is crucial that you prevent any photos that show ‘modified’ surface areas from contributing to the result. Also make sure to exclude under-/overexposed images from this phase. Both MetaShape and RealityCapture let you exclude photos from the Texturing phase.
4. Destructive Scanning
If you are scanning items of low monetary and emotional value, you might consider modifying or completely destroying them during the process to make your life easier. One option is to hand paint or airbrush surface texture onto the model. Similar to the coffee powder trick you get additional texture information that is extremely helpful for the “Alignment” and “Dense Reconstruction”. The drawback is that you need to find a good alternative approach to get a good surface texture. For simple objects like this candle, manually defining the surface appearance might be a solution.
Another way to modify the object is to actually cut away pieces. We did this for the mushrooms since it was too much of a hassle to make them stand upright on their stem. We cut away their stems before taking photos of their caps. Do not use photos that show the “cut away” parts for Dense Reconstruction (more on that in the “Visibility Consistency Filtering” section) Pro tip: After cutting the mushroom you can actually fry and eat it, it’s yummy!
5. Feed Images in Right Order
The Alignment phase in photogrammetry produces very different results based on the order in which images are processed. There are different strategies for different tools: In RealityCapture, the alphabetical order of images is important. Try to always have enough visual overlap between two consecutive images. RealityCapture is smart enough to realize when two images do not see the same side of your object, but it doesn’t hurt to help. MetaShape solves that problem for you by matching all possible image pairs first. In a second pass, it selects the best pair to start with and incrementally adds other images with high visual overlap. This strategy makes sense but might fail if the initial pair is not the optimal one. A good initial pair should have crisp textures in all parts of the images, the object should be fully visible, and the camera should have moved around the object by approx 5-10 degrees (that number varies a lot for different object, experiment!). In MetaShape, you can force the software to add photos in a particular order. First, select all images, open the context menu and select “Reset image alignment”. Now you can pick an optimal image pair and click “Align Selected Cameras”. Now keep adding new images by selecting them individually and by clicking “Align Selected Cameras”. You can of course let MetaShape align multiple or all remaining images at once, but keep in mind that it then decides on its own in which order it adds them.
6. Visibility Consistency Filtering
RealityCapture enforces visibility consistency of the surfaces scanned with the software, and we would like to show how you can use this to your advantage. In plain English that means that spurious clutter during reconstruction is removed in favor of surfaces that are consistently observed from multiple angles. If you would like to read up on this algorithm, the Patch-Based Multi-View Stereo paper is a good (but heavy) read. The filtering is explained in Chapter 3.3, fig. 5. Knowing about this filtering step is very helpful for turntable scanning. Typically, the software will reconstruct some geometry during the ‘Alignment’ phase that is not part of the foreground object, for example the white paper that is beneath the mushroom.
When we turn the brick around to take pictures of its other sides, RealityCapture does not see any paper covering the brick that was previously observed at the bottom part of surface “A” and will filter out this geometry during the reconstruction phase. It’s very important to keep that in mind when doing destructive operations (as described above). In the case of the mushrooms we cut away the stem so it could easier stand on the flat surface. We had to make sure that the ‘cut away’ part of the mushroom does not become visible in any other photos, otherwise Reality Capture might have filtered out the stem of the mushroom.
MetaShape does not have such a filtering step (or at least it’s not as pronounced). In order to get great results here, we recommend putting objects on a small pedestal (can be a small wooden block, a piece of blu tack, a rubber band, etc.). It’s important that the region where your scanned object touches the pedestal is never visible in any of the photos. If you follow this rule, it’s very easy to remove any 3D clutter after Dense Reconstruction (using MetaShape’s lasso tool) or even after the Meshing phase (by removing any spurious blobs of geometry in your 3D software). You have to make sure to remove any excess geometry before the Texturing phase, though.
If your capture material shows the areas where the objects touches the background or pedestal, removing the faulty geometry becomes a very tedious process and you might want to recapture that material.
7. Consider a JPEG Workflow
As you might have realized, we are using .jpg images throughout the workflow. We realize most tutorials strongly suggest using raw image formats. In our experiments we of course realized that there is a measurable quality difference, but often do not find the ‘extra effort’ for using raw is justified. One drawback of raw is of course file size, but we also find them harder to work with since many tools do not support raw images. Similar thoughts apply to other aspects such as image resolution. In general, bigger is better, but for some workflows the highest settings might lead to unbearable processing times and will just cram your hard drive with unneeded pixels. Always try to strike a balance between resolution/quality and your (system’s) ability to actually handle the data. Especially when you’re in the experimenting phase of a new project, you don’t want to stare at progress bars for most of your day.
8. Blank Photo
This trick is almost too simple to mention it: When you photograph similar objects, always record a ‘blank’ photo (without any foreground elements) before starting with the next object. This way you can easily distinguish between data for different objects.
9. Don’t Follow Instructions
Yes, that’s right, you shouldn’t trust all advice you read here ?. There’s tons of other helpful but conflicting advice out there. Do your research and read all relevant tutorials you can find. But at the end of the day you have to experiment and see for yourself what works out best for your project.
Have fun creating!
-Kathrin and Christian