Create generative AI video-to-video right from your phone with Runway’s iOS app
By: James Vincent
AI startup Runway has launched its first mobile app on iOS, letting people use the company’s video-to-video generative AI model — Gen-1 — directly from their phones. You can download the app here, with free users offered a limited number of credits.
Gen-1 allows you to transform an existing video based on a text, image, or video input. Functionally, it works a lot like a style transfer tool (though, unlike style transfer, it generates entirely new videos as an output rather than applying filters). You can upload a video of someone cycling in the park, for example, and apply an aesthetic or theme. You can give the video the look of a watercolor painting or charcoal sketch, and so on.
Of course, because this is a generative AI, the output is often… strange. If you add a claymation effect, for example, your resulting models won’t function like real claymation. The models will warp between each frame; limbs will grow and shrink; features will melt and smear. That’s all to be expected, though, and doesn’t take away from the fun.
Here, for example, are three different renderings of an iconic clip of Al Pacino in Heat (1995). Most notable to me is the clip in the bottom right, which uses a picture I’d taken of a cat as an intermediary. Without me having to specify, the model applied the cat’s face to Pacino’s and even gave his hands a bit of fur while leaving his suit more or less intact. The other two clips on the top row are preset filters.
Here’s another example: a video of St. Paul’s Cathedral in London with the “paper and ink” filter applied. It’s not a mind-blowing effect, but it was incredibly easy to make. And in the hands of a more experienced and creative individual, I’m sure it could be spectacular.
I’ve been testing Runway’s app for a few days now, and it certainly makes the whole process of creating this sort of video much more fluid. (Runway’s main software suite is available on the web, which makes the distance between capturing footage and generating it wider.) It’s not a seamless experience, of course. There are the usual inefficiencies and unexpected errors you’d expect to find in the first release of an app. But, as Runway CEO Cristóbal Valenzuela told The Verge, making these tools mobile is the important thing.
“That’s why the phone makes so much sense because you’re recording directly from your device, and then you tell Gen-1 how to transform that video,” said Valenzuela.
There are other limitations worth mentioning. You can’t work with footage longer than five seconds, and there are certain banned prompts. You can’t generate nudity, for example, and it seems copyright-protected work is off-limits, too. My prompt to create a video “in the style of a Studio Ghibli film” was rejected. Each video also takes around two to three minutes to create, which doesn’t sound like a lot but feels like an age in the era of instant mobile editing. The processing is done in the cloud and will likely speed up over time. The app only currently supports Runway’s Gen-1 model, but Valenzuela says the purely generative Gen-2 will be added soon.
The magic of Gen-1. Now on your phone.
Download the new Runway iOS app today: https://t.co/kYfzzHfOfp pic.twitter.com/1GySmhJkby
— Runway (@runwayml) April 24, 2023
What these notes don’t fully capture, though, is the huge sense of possibility of tools like this. The output of AI text-to-image models also started out as smeared and unrealistic. Now they’re being used to fool the public with swagged-out pictures of the pope.
Valenzuela has compared the current era of generative AI to the “optical toys” phase of the 19th century, when scientists and inventors were creating a whole range of devices that were trivial in their capabilities but also the ancestors of modern cameras. Runway’s mobile app feels like one of these toys. I can’t imagine it being used for professional production work, but I also can’t imagine how big an effect tools like this will have in the future.