Snap AR DSP Audio Filter
A custom Snapchat filter using multiple variable saw waves and a formant filter all triggered by facial expressions, paired with a generative displacement visual, to give this fun, kinda creepy, robotic, assimilation filter.
A custom Snapchat filter using multiple variable saw waves and a formant filter all triggered by facial expressions, paired with a generative displacement visual, to give this fun, kinda creepy, robotic, assimilation filter.
An artist, famous for adding a technological element, was interested in ways to utilize technology to give his art another dimension.
I worked with him to select a few pieces of his art, to come to life in a couple of ways.
Create 3D animations to explode out of his paintings.
Allow him to create 2D animated videos, that tell the story intended in his paintings.
I knew I want to create an augmented reality app, but how to accomplish that was yet a mystery. I did some research and settled on using Qualcomm’s Vuforia library, and the Unity game engine which I already had experience with.
From there, it was a fairly simple process, use a tool from Vuforia to convert pictures of the paintings into trackers. Import that to Unity. And then attach a video player object to each painting, and then trigger the video to play when the painting enters the camera viewport. Vuforia made the process very easy, all it took was some custom programming, and a lot of fine tuning to build the basic app.
Creating the 3D angel is what ended up being the most difficult. I took the picture of the painting, and removed the angel and used Adobe Photoshop’s clone stamp and Healing Brush to cover over the hole the angel used to be in. Then I took the angel and flipped its wings outwards as it would be fully outstretched, and then exported the angel and background as individual pieces. I took and used Adobe Illustrator to create a black on white outline of the angel, and exported it as an SVG object.
I then imported the SVG object into blender. Used the “Mesh > Faces > Beautiful Fill” option to convert it to a mesh. Then spent some time cleaning up the mesh by hand. The next step was applying the Angel we cut out earlier as a material (which proved to be a finicky process to map properly.)
By this point I had an angel with outstretched wings, and its background as two elements. Now it was time to animate it. I had created rigs before thanks to the Double Jump Intro, so that turned out to be an easy process.
From there, I watched a lot of videos of birds flying in slow motion, and did the animation in two parts, animate the flapping of the wings itself, which was just a simple up and down motion, with a little rotation. Then it was about animating the up and down motion of the angel itself. I noticed that as birds fly, they dip a little after each flap. This helped to give it a really natural feel.
I then took an additional digital painting given to me by the artist, and faded that in at end. Giving all the little specs and words depth as to help enhance the 3D effect.
And just like that I had an Augmented Reality app, and I had taken a 2D artwork and created a beautiful 3D flying angel. This wowed a lot of future clientele, and helped him to sell prints of his art as “AR Ready.” Increasing his revenues and reach.
A client came to me to discuss issues they were having, with explaining their new designs for a retail store with the client. There were areas the client was having difficulty visualizing, and worries about traffic control.
I decided to leverage (at the time) newly released Oculus Rift devkits, to create a Virtual Reality walkthrough of the designs of the store.
First I asked for the designs, which came in a Google Sketchup format. I then export those designs as a Collada file (a common 3D file) and imported that into the Unity Game engine. Immediately I noticed some serious performance issues. The model was so large that rendered in real-time, it would lag, shutter, and sometimes crash altogether.
I found that every element in the scene, was it’s own mesh, creating more overhead than necessary. To solve this, I started by taking all repeating elements (bottles, stacks of shirts, tables, etc) and began replacing them with an instance of one mesh. Then I took any and all repeating colors and textures, and replaced those with one.
That instantly sped up the model, but it still felt sluggish. I decided to employ a technique calling “Frustum Culling.” Instead of rendering all the models, all the time. This takes and hides all models outside of the viewport. Below is a gif of that process.
Doing that made a HUGE impact on the performance, making it render a consistent 60 frames per second, on our display computer.
I now needed to make create a body moving through the space. Simply making a floating camera would have sufficed, but the end user wouldn’t have gotten a sense for any choke points in the design. Basically they would have floated through sections where a human being, would have struggled to move. I created a simple, stretched sphere, at the same height and width of an average human. Then calculated the collision bounds for all the tables, floors, places where the user would be likely to run into. And then told the engine to collide the user, with those items.
With all this in place, it was time for the final touches. I played a documentary on the maker of the product on the TVs around the store. I build an inclined plane up the stairs, so that the user could go up and down stairs.
This was a quick and easy way to allow their client to experience the still unbuilt, and for the client to sell their designs and process. This led to beneficial conversations, and lasting changes.