For years, filmmakers have used 2.5D parallax to make static photos feel more dynamic, as in The Kid Stays In The Picture, the 2002 documentary about film producer Robert Evans that popularized the technique.
Traditionally, a video editor would use Photoshop to isolate the photo elements on separate layers, fill in the removed objects to complete the background, and animate the layers in a tool like After Effects. Here’s a typical tutorial, showing the time-consuming and tedious process.
Last September, a team at Adobe Research released a paper and video demonstrating a new technique for animating a still image with a virtual camera and zoom, adding parallax to create what they call “the 3D Ken Burns effect.”
This new technique uses deep learning and computer vision to turn a static photo into a 2.5D parallax animation in seconds, using a neural network to estimate depths, render a virtual camera through space, and fill in the missing areas.
On Monday night, researcher Simon Niklaus finally got permission to release the code, posting it to Github with a CC-NC license, allowing anyone to experiment with it for themselves.
Sample Animations
It’s incredibly fun to play with. I ran some famous images through it, and then put a call out to Twitter for more ideas. Here are the results. Click anywhere on the video to play it.
Surprisingly, it even works on illustrations and paintings.
Try It Yourself
Unlike the Spleeter library, the Ken Burns 3D library requires using PyTorch with an Nvidia GPU with CUDA drivers. Sadly, Apple phased out CUDA support in Mojave, but there’s an even easier way to play around with it.
I created a Google Colab notebook here, which will let you process images on Google’s GPUs entirely in your browser.
If you’re unfamiliar with Colab, it can be a bit intimidating. Try this, and let me know if you get stuck.
- Open the Google Colab notebook.
- Click File->Open In Playground Mode to run it yourself.
- Click “Connect” to connect to a hosted runtime, a temporary Google server where the commands will run.
- From the “Runtime” menu, click “Run All.” A warning will pop up, you can click “Run Anyway.”
- On the left-hand side of the window, click the tab to open the Files sidebar.
- The final command processes the “doublestrike.jpg” sample image, and generates a new file in the /images directory called “autozoom.mp4.”
- Upload your own images by right-clicking the “images” folder and clicking Upload. Change the input/output filenames, and click the play button to run the final command again.
Good luck!
Update: This Google Colab notebook by Manuel Romero is much faster and easier to use, with a handy widget to upload files, process images in bulk, and download all the finished animations.
Thank you for your article- and also for realisation as Colab notebook! The possibilities are so wonderful, even if sometimes a little bit buggy.
I’ll write a blog about it!
Can I run this on my computer? I have a RTX 2060 graphics card and 16GB of RAM. I don’t know anything about coding though.
You don’t need to know how to code, but Mac won’t run CUDA drivers and installing all the prerequisites in Windows 10 can be a bit tricky. There’s one tutorial here, which will set you up for the instructions in the Ken Burns 3D Github. If this all sounds overwhelming, you might want to just stick with the Google Colab notebook.
Thanks for the response. This is beyond me so I’ll stick with the Colab. How would I run the depth estimate function in Colab? The command is listed as [python depthestim.py –in ./images/doublestrike.jpg –out ./depthestim.npy]
Starting with Manuel’s Colab notebook, just click the “+Code” button to add a new cell and paste in the command preceded with an exclamation point. I just tried running a cell with “!python depthestim.py –in ./images/doublestrike.jpg –out ./depthestim.npy” and it generated an .npy and .png file in the 3d-ken-burns folder with the depth map and estimate.
In both notebooks, I get error: urllib.error.HTTPError: HTTP Error 403: Forbidden
Can you verify that these notebooks still work?
Seems to be working fine for me.
I’m getting this error
Traceback (most recent call last):
File “autozoom.py”, line 11, in
import gevent
Google changed something in Colab. You’ll need to add “!pip install gevent” to install the module.
Hi Andy – your instructions are very helpful. I’m with you up to “The final command processes the “doublestrike.jpg” sample image, and generates a new file in the /images directory called “autozoom.mp4.””
What is “the final command”? Thanks!
The final command in the Colab notebook is “!python autozoom.py –in ./images/doublestrike.jpg –out ./images/autozoom.mp4”. When that runs, it will generate the .MP4 file.
Gottit. And (sorry to be a complete ignoramous, I have zero code knowledge) in which fields do I change the input/output filenames?
#embarrassed
I’d recommend using Manuel Romero’s Colab notebook instead of mine. It’s easier to use, with a form field for uploading files.
Did I do something wrong. I didn’t get the mp4. There is also an error about a gevent module missing. What to do?
Traceback (most recent call last):
File “autozoom.py”, line 11, in
import gevent
ModuleNotFoundError: No module named ‘gevent’
I just modified the Colab notebook to install gevent, it should work now.
Its working and it looks very cool! Is the a way to modify the zoom length, direction and speed? I’d like to see if I can use it in a documentary I’m helping to make.
You can change settings in autozoom.py, but it’s not easy to do from Google Colab.
Not seeing Open Playground…could anybody help steer me?
Cheers!
It’s in the File menu now. I edited the post to reflect their UI change.
Not sure what I’m doing wrong but when I go to the Google Colab notebook.page, I don’t see the ‘Click File->Open In Playground Mode to run it yourself.’ option?
I’d love to try this out & appreciate your efforts in trying to make this easy for novices like me.
Hope to get some pointers soon from you to help me out …
… a few minutes after my previous comment, I realized that maybe I should copy the Google Colab notebook to my own Google account. Anyway, that’s what I did & managed to get as far as ‘Run All’.
But now it is just hanging on this cell ‘!pip install cupy moviepy gevent’
;-(
Try this modified version of Manu Romero’s notebook, which has been updated to install dependencies.
1. File->Save A Copy In Drive.
2. Make sure it’s running on a GPU runtime. Runtime->Change Runtime Type->GPU.
3. Runtime->Run All.
4. When it gets to step 3, select the images you want to transform.
Good luck!
Thanks so much for the reply Andy, I’ve now managed to follow your new steps successfully on the whole. However, for a few pictures the videos that get created are not immediately viewable. So, what I do for those is use a video converter & after that I can view them.
😉
Forgot to add that I am getting this error for some files & as a result they get converted to mp4:
Traceback (most recent call last):
File “autozoom.py”, line 66, in
intWidth = npyImage.shape[1]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
Any ideas on how to fix this?
Thanks …
Sorry that previous comment had a typo, so feel free to delete it.
It should have been:
Forgot to add that I am getting this error for some files & as a result they do not get converted to mp4:
Traceback (most recent call last):
File “autozoom.py”, line 66, in
intWidth = npyImage.shape[1]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
Any ideas on how to fix this?
Thanks …
Absolutely brilliant thank you so much to both you Andy and Manual Romero – it is generous of you to offer this.
The updated Manual Romero works, has the chainer dependency in it now
Ooops… again… no longer works… (was so excited to try!)
Traceback (most recent call last):
File “autozoom.py”, line 15, in
import moviepy.editor
File “/usr/local/lib/python3.7/dist-packages/moviepy/editor.py”, line 26, in
imageio.plugins.ffmpeg.download()
File “/usr/local/lib/python3.7/dist-packages/imageio/plugins/ffmpeg.py”, line 38, in download
“imageio.ffmpeg.download() has been deprecated. ”
RuntimeError: imageio.ffmpeg.download() has been deprecated. Use ‘pip install imageio-ffmpeg’ instead.’
As of 12/17/2022, following modules are needed to make this work:
!pip install moviepy
!pip install gevent
!pip install chainer
!pip install imageio==2.4.1
Click on Insert -> “Code Cell”
And paste the above commands and run the cell. That should resolve all the errors.
Yes, that addendum works awesome. Thank you for the update on getting this operational again.
No idea why there isn’t commercial desktop software module ready-to-buy yet.
A dozen “phone apps” can do these effects very well now, but it’s cumbersome to do it thru an app… one at a time…