Sometimes photos cannot truly capture a scene. How much more epic would that vacation photo of Niagara Falls be if the water were moving?

Researchers at the 91探花 have developed a deep learning method that can do just that: If given a single photo of a waterfall, the system creates a video showing that water cascading down. All that’s missing is the roar of the water and the feeling of the spray on your face.

The team’s method can animate any flowing material, including smoke and clouds. This technique produces a short video that loops seamlessly, giving the impression of endless movement. The researchers June 22 at the .

“A picture captures a moment frozen in time. But a lot of information is lost in a static image. What led to this moment, and how are things changing? Think about the last time you found yourself fixated on something really interesting 鈥� chances are, it wasn’t totally static,” said lead author , a doctoral student in the Paul G. Allen School of Computer Science & Engineering.

“What’s special about our method is that it doesn’t require any user input or extra information,” Ho艂y艅ski said. “All you need is a picture. And it produces as output a high-resolution, seamlessly looping video that quite often looks like a real video.”

Developing a method that turns a single photo into a believable video has been a challenge for the field.

“It effectively requires you to predict the future,” Ho艂y艅ski said. “And in the real world, there are nearly infinite possibilities of what might happen next.”

The team’s system consists of two parts: First, it predicts how things were moving when a photo was taken, and then uses that information to create the animation.

To estimate motion, the team trained a neural network with thousands of videos of waterfalls, rivers, oceans and other material with fluid motion. The training process consisted of asking the network to guess the motion of a video when only given the first frame. After comparing its prediction with the actual video, the network learned to identify clues 鈥� ripples in a stream, for example 鈥� to help it predict what happened next. Then the team’s system uses that information to determine if and how each pixel should move.

The researchers tried to use a technique called “splatting” to animate the photo. This method moves each pixel according to its predicted motion. But this created a problem.

“Think about a flowing waterfall,” Ho艂y艅ski said. “If you just move the pixels down the waterfall, after a few frames of the video, you’ll have no pixels at the top!”

So the team created “symmetric splatting.” Essentially, the method predicts both the future and the past for an image and then combines them into one animation.

“Looking back at the waterfall example, if we move into the past, the pixels will move up the waterfall. So we will start to see a hole near the bottom,” Ho艂y艅ski said. “We integrate information from both of these animations so there are never any glaringly large holes in our warped images.”

Finally, the researchers wanted their animation to loop seamlessly to create a look of continuous movement. The animation network follows a few tricks to keep things clean, including transitioning different parts of the frame at different times and deciding how quickly or slowly to blend each pixel depending on its surroundings.

The team’s method works best for objects with predictable fluid motion. Currently, the technology struggles to predict how reflections should move or how water distorts the appearance of objects beneath it.

“When we see a waterfall, we know how the water should behave. The same is true for fire or smoke. These types of motions obey the same set of physical laws, and there are usually cues in the image that tell us how things should be moving,” Ho艂y艅ski said. “We’d love to extend our work to operate on a wider range of objects, like animating a person’s hair blowing in the wind. I’m hoping that eventually the pictures that we share with our friends and family won’t be static images. Instead, they’ll all be dynamic animations like the ones our method produces.”

Co-authors are and , both professors in the Allen School, and , an affiliate professor in the Allen School. This research was funded by the 91探花Reality Lab, Facebook, Google, Futurewei and Amazon.

For more information, contact Ho艂y艅ski at holynski@cs.washington.edu.

Their algorithm, , can take a person from a 2D photo or a work of art and make them run, walk or jump out of the frame. The system also allows users to view the animation in three dimensions using augmented reality tools. The researchers will be presenting their results June 19 at the in Long Beach, California. This research first attracted media attention when it was in preprint form in December on ArXiv.

“This is a very hard fundamental problem in computer vision,” said co-author , an associate professor at the UW鈥檚 Paul G. Allen School of Computer Science & Engineering. “The big challenge here is that the input is only from a single camera position, so part of the person is invisible. Our work combines technical advancement on an open problem in the field with artistic creative visualization.”

Previously, researchers thought it would be impossible to animate a person running out of a single photo.

“There is some previous work that tries to create a 3D character using multiple viewpoints,” said co-author , a professor in the Allen School. “But you still couldn’t bring someone to life and have them run out of a scene, and you couldn’t bring AR into it. It was really surprising that we could get some compelling results with using just one photo.”

The applications of Photo Wake-Up are numerous, the team says. The researchers envision this could lead to a new way for gamers to create avatars that actually look like them, a method for visitors to interact with paintings in an art museum 鈥� say sitting down to have tea with Mona Lisa 鈥� or something that lets children to bring their drawings to life. Examples in the research paper include animating the Golden State Warriors’ Stephen Curry to run off the court, Paul McCartney to leap off the cover of the “Help!” album and Matisse’s 聽(1944) to leave his frame.

To make the magic a reality, Photo Wake-Up starts by identifying a person in an image and making a mask of the body’s outline. From there, it matches a 3D template to the subject’s body position. Then the algorithm does something surprising: In order to warp the template so that it actually looks like the person in the photo, it projects the 3D person back into 2D.

“It’s very hard to manipulate in 3D precisely,” said co-author , a doctoral student in the Allen School. “Maybe you can do it roughly, but any error will be obvious when you animate the character. So we have to find a way to handle things perfectly, and it’s easier to do this in 2D.”

Photo Wake-Up stores 3D information for each pixel: its distance from the camera or artist and how a person’s joints are connected together. Once the template has been warped to match the person’s shape, the algorithm pastes on the texture 鈥� the colors from the image. It also generates the back of the person by using information from the image and the 3D template. Then the tool stitches the two sides together to make a 3D person who will be able to turn around.

Once the 3D character is ready to run, the algorithm needs to set up the background so that the character doesn’t leave a blank space behind. Photo Wake-Up fills in the hole behind the person by borrowing information from other parts of the image.

Right now Photo Wake-Up works best with images of people facing forward, and can animate both artistic creations and photographs of real people. The algorithm can also handle some photos where people’s arms are blocking part of their bodies, but it is not yet capable of animating people who have their legs crossed or who are blocking large parts of themselves.

“Photo Wake-Up is a new way to interact with photos,” Weng said. “It can’t do everything yet, but this is just the beginning.”

This research was funded by the National Science Foundation, 91探花Animation Research, 91探花Reality Lab, Facebook, Huawei and Google.

###

For more information, contact Weng at chungyi@cs.washington.edu, Kemelmacher-Shlizerman at kemelmi@cs.washington.edu or Curless at curless@cs.washington.edu.

Grant number: #VEC1538618

The 91探花 is launching a new augmented and virtual reality research center 鈥� funded by Facebook, Google, and Huawei 鈥� to accelerate innovation in the field and educate the next generation of researchers and practitioners.

The $6 million , funded with equal contributions from the three initial sponsors, creates one of the world鈥檚 first academic centers dedicated to virtual and augmented reality. The new center in the Paul G. Allen School of Computer Science & Engineering and located in Seattle 鈥� a national hub of VR activity 鈥� will support research and education initiatives with potential to deliver game-changing breakthroughs in the field.

鈥淎llen School faculty have produced pioneering research in many of the areas that underpin AR and VR technologies, including computer vision, graphics, perception, and machine learning,鈥� said , Allen School director and Wissner-Slivka Chair in Computer Science and Engineering. 鈥淭hrough our partnership with Facebook, Google, and Huawei, the Allen School and 91探花will be at the forefront of the next great wave of AR and VR innovation 鈥� pursuing breakthrough research and educating the next generation of innovators in this exciting and rapidly expanding field.鈥�

To date, AR and VR applications are making their first steps, mostly focusing on entertainment and games. Yet everyone is interested to find the “killer app” for AR and VR. The goal of the 91探花Reality Lab is to develop technology to power the next generation of applications that will speak to a wider population. Those diverse ideas range from learning Spanish by seeing objects labeled in your field of view to achieving telepresence by conversing with a remote relative or co-worker as if you were in the same room.

鈥淲e’re seeing some really compelling and high quality AR and VR experiences being built today,鈥� said center co-lead and Allen School professor . 鈥淏ut, there are still many core research advances needed to move the industry forward 鈥� tools for easily creating content, infrastructure solutions for streaming 3D video, and privacy and security safeguards 鈥� that university researchers are uniquely positioned to tackle.鈥�

The 91探花Reality Lab will bring together an interdisciplinary team of 91探花faculty, graduate students and undergraduates working in 3D computer vision and perception, object recognition, graphics, game science and education, distributed computing, stream processing, databases, computer architecture, and privacy and security.

Another key function of the 91探花Reality Lab will be to educate tomorrow鈥檚 AR and VR researchers and workers. The funding will support new courses and access to state-of-the-art labs and infrastructure for 91探花students to develop new technologies and applications. That includes accessing emerging technologies from the center鈥檚 sponsors and allowing those companies to test new ideas in a focused setting with computer science students. An advisory board of luminaries from across the AR and VR community will help the center remain at the forefront of this burgeoning field.

The 91探花Reality Lab builds on the Allen School鈥檚 established leadership in cutting-edge AR/VR education and research. In one example, the school in 2016 introduced the world鈥檚 , in which students built AR applications using 40 HoloLens units loaned from Microsoft before they were commercially available.

鈥淪tudents had fantastic ideas and were able to create amazing AR and VR applications ranging from Holographic Chess to teaching one how to play the piano or cook. This opened our eyes to the potential of investing deeper in development of algorithms and applications for AR and VR,鈥� said center co-lead and Allen School assistant professor . 鈥淲e realized there were so many cool things we could do if only we had more resources, more time and more devices. Given those, we can help bring the world鈥檚 AR and VR dreams to life.鈥�

The 91探花Reality Lab鈥檚 location in Seattle 鈥� one of the world鈥檚 most active centers for VR and AR innovation 鈥� paves the way for unique industry and academic collaborations aimed at achieving new capabilities and offering users seamless experiences.

鈥淗aving an opportunity to be at the leading edge of this industry is really exciting,鈥� said co-lead and Allen School professor . 鈥淚t鈥檚 big, it鈥檚 happening now and there鈥檚 a lot of research to be done. We鈥檙e thrilled to take a leading role in making it all happen.鈥�

For more information, contact realitylab@cs.washington.edu or on Twitter.