<img src="https://certify.alexametrics.com/atrk.gif?account=43vOv1Y1Mn20Io" style="display:none" height="1" width="1" alt="">

How AI transforms video production

5 minute read

David Shapton charts the course up the learning curve that is the use of AI in all aspects of video production.

Learning about AI developments from Twitter is like drinking from a firehose. Among the authoritative and often surprising revelations about the latest AI models, youll find items like 100,000 apps to transform your video editing” or improve your scriptwriting with these 50,000 brilliant Chat GPT  prompts."

These are all very well; if youre prepared to work through endless examples, youll probably learn something useful. But whats often missing is any context around these techniques. Thats not entirely surprising: AI is progressing so fast that there often is no context to it. When we see news about AI, its not like we can say, Oh, something similar happened in 1948." AI timescales feel like theyre quantised in milliseconds, and what happened yesterday isnt a reliable guide to what will happen today. 

So, rather than looking at individual and specific apps, its worth looking for areas that AI could usefully improve. Theres a lot of overlap, but broadly were talking about AI techniques that assist, create and process.

New frontiers

There are all kinds of areas where AI will, in the future, assist. For example, if you show a script to an AI assistant and tell it about your resources (equipment, personnel, budget, time, etc.), it could create a project management chart based on its knowledge of formal project planning methodologies. It could generate a list of equipment to order. Knowing what human resources are available, it could print out task lists for each individual. When the production is about to start, it could create call sheets. All of this assumes that it has accurate and relevant information and that inputting that data doesnt take longer than doing the whole thing the old-fashioned way. 

As a side note, its worth saying that all of this assumes that filmmakers want AI involved at this level. I feel it is natural to be wary of AI running the whole thing, but that over time it will be incorporated into productions because it will be not so good at some aspects but compellingly good at others. 

Its tempting to think around big-budget filmmaking when youre talking about AI, but its probably in other types of production that assistive AI will make the most difference. AI transcription services can already listen” to audio and display it as written text. Whats more, each word will link to timecode with a clickable link to the frames containing that word. This is a massive boon to editing news clips and documentaries, where several hours of interview footage can be quickly assessed for relevance and edited into the finished package.

AI will be able to sort your footage for you. It can dispose of obviously bad shots (like images of peoples feet and the catering van). However, this will always need a degree of caution - especially in a production where weird and unconventional shots are part of the plan. Through its ability to understand” content, AI will play significant roles in media management. It should replace almost all logging and metadata-tagging duties, although, again, this will require caution and restraint, until a time when either there are no errors or we have a much better understanding of areas where errors are likely, and what we can do to either work around or eliminate them. In future, editing is likely to be on a continuum from traditional” to hands-off” editing - the equivalent of autonomous driving. Tell your AI assistant editor your ideas, and it will scan the footage and make the cuts that reflect your intent. Quite obviously, this raises about a hundred more questions than it answers at this stage in the evolution of AI. 

AI as co-pilot

From a creative persons point of view - which includes the entirely reasonable aspiration that AI wont put them out of a job - the new technology can help most with mundane tasks. There are already applications that can apply the same colour look to an entire collection of clips, and go way beyond merely imposing the same LUT. AI can understand” a look in a deeper, more nuanced way than a LUT ever could because it is not restricted to a fixed number of dimensions”. These are not dimensions in the sense that we usually understand them; theyre more like different ways to recognise things or to characterise them. Its likely that AI sees” things entirely differently from our perception, and this is one of the reasons that AI models - or the conclusions they come to - are hard for us to reverse-engineer. But it does mean that AI can capture the essence” of a look and apply it to other clips, with a subtlety that would be hard to emulate without a lot of work.

In an ideal world, AI would do that basic matching, leaving the colour artist to work on their own and the directors artistic vision. Please understand this doesnt mean we have to rely on AI for inspiration: its possible to train the AI models with human expert knowledge, which would be stored in a nuanced, accurate and detailed way, ready to apply to your current project. Remember: this is much more powerful than LUTs because it can take the complete essence of a look and re-interpret it for radically different, as well as similar material.

RedShark readers might remember that for about a decade, Ive been predicting youll be able to feed a script into a computer, do a bit of rendering, and output a finished feature film. At first, it was 99% tongue-in-cheek, but still with a sense that this was the almost inevitable result of extrapolating filmmaking technology as far as you could go. If youd asked me then when I thought it would happen, Id have said maybe in twenty to fifty years, if ever. Fast-forward to today, and we know how long it will take - because we can do it now. The only drawback is that its not very good, although still remarkable. The problem is that its tough to generate realistic-looking video. 

Still images are pretty much solved: the latest outputs of Midjourney are photorealistic and can accept artistic” and technical instructions with great precision and nuance. Adobe Creative Clouds Firefly can generate amazing new content based on existing images and do it transparently without copyright worries. Want a backdrop thats autumnal instead of spring-like? Just ask. Want to extend a background? No problem. 

Video still a challenge

But video? Not so much. Someone will solve it soon, but for now, heres why its difficult. 

When you shoot virtually anything with a video camera, whatevers in front of the lens has to obey the laws of physics. Tables and chairs dont jump around randomly, nor do they look like Monet painted them in one frame and Hokusai or Van Gogh in the next. But in AI-generated video, the results literally do look like a different artist painted them in every frame. AI generative video models like Runway are rapidly getting better, producing videos that are at least watchable, but they still look as though theyre made by a team of animators who never look at each others work. Whats needed is consistency across time - lets call it temporal continuity”. But future models will be able to do this. Solving the problem of time is only a matter of time. 

It seems likely that AI will help with virtual production, and therefore become part of mainstream filmmaking. I hope that human artists still generate the main ideas, but AI can help to convert these ideas into fully-fledged backdrops for a virtual stage. Im assuming that VP professionals will welcome any ability to speed up the process of generating 3D assets from original ideas.

When generative AI arrived in the public consciousness last year, it was like a runaway train, and its still hard to see exactly where it might lead. But some of the dust is settling, and some things are getting clearer. Its important not to be distracted by those who say, We dont need creatives any more”. But the greatest works intended for humans will always be made by humans. If AI can help make those works faster and better, and allow us to create even more outstanding work, then thats a good result. There are no guarantees, but I think its OK to be optimistic about the future. 

Tags: AI