Professional video editing powered by machine learning — all on the web.
Read the original article
Comments
By anigbrowl 2021-07-086:30 Former film pro here. I don't see anything 'impossible' here, most of this is stuff I could do on my desk 10 years ago with a modest setup - but only with a lot of patience and manual labor.
If these demos fairly represent the user experience then it's slick as hell and further blurs the line between editor, compositor, DI specialist etc. Much will depend on whether the ML marketplace can be competitive with many mature commercial offerings from software studios that have no intention of letting their lunch be eaten. Video production involves a lot of bleeding edge technology but the client base is also suspicious of new providers or do-it-all solutions at first, and very loyal to products and tech support offerings that have got them through difficult projects in the past, so there will be a big hill to climb between people acknowledging it's cool as hell and their willingness to sell it to a producer who is making a 5, 6, or 7 figure bet on what some editor/VFX geek is telling them.
The web browser/cloud storage is an issue. It could be a plus in many circumstances, but it's also a big barrier to any production that's working in a remote location without reliable internet, especially given the massive data volumes involved. That limits a lot of the use cases to post production, and many producers and directors are going to be wary of starting on one platform and finishing on something else; nobody likes workflow changes unless they can be shown something seamless, like having your Avid/Final Cut/Premiere/* project leave your machine and show up frame-perfect in Runway, along with a definite answer about export time like being able to take in 12 hours of video and have it online within 24h. There will also have to be a lot of questions about security, downtime, and being able to get your project back out of Runway if money runs out, the editor gets fired, creative differences rear their head etc.
Looks like an instant win for short-form projects like personal shorts, music videos, commercials, corporate, demo reels, spec pieces. Potentially very good for reality TV indie and low-mid budget films once the above questions are answered. Toughest nut to crack is large budget or episodic TV where there's very stiff competition and contractual or professional commitments already in place, but doable within 5 years.
I've just given this a go, with very little work I was able to produce a near seamless mask which was able to adapt to dynamic and changing pose - interesting stuff.
The main drawback I observed was computation speed, most edits required >20 seconds of loading/buffering (with a 1Gbps internet connection). I presume the computation occurs on their own servers, so with beefier hardware the performance could be increased, however my intuition is that the app would perform faster if running natively (rather than a shared resource quota on a remote server).
In this regard, the lack of performance could be a major productivity killer, my hypothesis is that it is the video processing/manipulation which is taking a long time (and not the model classification). Many companies have tackled this problem space with completely video transcoding chips (such as YouTube), however this generally still incurs long periods of waiting.
By eh9 2021-07-082:36 I would imagine it could be slightly easier to manage models deployed on their servers, but Adobe manages to package and ship its tools so they run natively.
My best guess is that the web can be a path of least resistance and that’s why this launched on the web, but as soon as they have available resources, they could package and ship a native app (hopefully one that can take advantage of modern chip ML tech)
I'm so glad to see this on the front page -- this website, in conjunction with a video from Two Minute Papers, is what convinced me to teach myself machine learning. I come from a filmmaking background, and have used green screen extensively. It's big, bulky, has to be set up correctly, lit correctly, and is expensive and just generally a pain to work with.
I'm currently working on a fully-automatic version of this based off some excellent research from the University of Washington. It's still in its infancy, but if you'd like to follow my progress, I post occasional updates over on https://nomoregreenscreen.com
This is just the beginning of what's possible in the intersection between DL and creative filmmaking, and I'm really excited to get to be in this field at a time when compute is cheaper than ever and all the information I could want is available for free on the internet!
Interesting, how long ago did you first see runwayml & decide to teach yourself machine learning?
By mkaic 2021-07-086:58 I started taking Coursera courses back in late fall/early winter 2020. I discovered RunwayML through my subscription to Corridor Crew on YouTube, where they demoed the tech back in January of this year.
