FoodSpotting 3D: MenuAR + ReCap API via Autodesk Cloud Accelerator

What if there was a way for anyone to share their food as a hologram – instantly?

Just take photos around the food you want to digitize. Tag it while uploading. Presto, 3D mesh as hologram!

The food@AReality3D team (yosun+fern) participated in the Autodesk Cloud Accelerator in March of 2015.

We used Recap API as a photogrammetry solution for a consumer-facing app in foodtech – the 3D augmented reality and virtual reality version of Foodspotting. In short, we made it possible for anyone with a current generation smartphone to cleanly create food holograms – Star Wars Obi-wan Kenobi-style.

To better process the photos to 3D geometry, Recap allowed for the entry of survey points, i.e., common points across several images. Using box detection, we were able to quickly identify the location of a Clipper Card’s coordinates in submitted images to provide four survey points across each of several photos.

To anticipate specularity resulting in holes or unwanted effects in the dish holding the food, we allowed the user to lathe their own plate from a selected image denoting the side profile.

To conserve bandwidth on consumer phones, we checked for the existence of content on the server before uploading. We created a “image hash” by checking select pixel data in each image to see if it already exists. There was a high false negative in this check, but it helped save resources when obvious duplicates exist already. The full image payload was only uploaded upon check positive.

Similar to the Youtube-upload model, meta-data entry happened as the upload progresses.

Further entertainment and preview options were provided to the user during the upload/processing process: Once the basic tags were selected, the user received a preview of other foods with similar tags – as well as ideas for other parts of the meal… that they could easily preview as holograms in front of them.

Recap in its current form does not provide clean meshes. We used Memento and MeshLab automation scripts in the backend before sending the mesh back up to the mobile user. We also added the ability for the mobile user to easily select common excess mesh geometry – a mesh-tolerance selection allows for the removal of the table or flat surface which the food is digitized on. And, in case the plate does not come out too cleanly, we also recreate the dish.

This is our first time participating in an accelerator, so we are both guinea pigs and students in a different kind of experience. We are also in the early stages of Recap being a commercially-usable API, so when asked by Amar Hanspal, Autodesk SVP of IPG to provide a wish list for Recap, we had to list quite a few items… 

Here’s a video of what we created and demonstrated at the culmination of the first ever Autodesk Cloud Accelerator: