The quest to replace green screen accelerates

New technologies are in development that look to look to find faster and more accurate ways of replacing backgrounds in movies and TV.

Green screen has, of course, been around for a rather long time by now. The blue screen method to produce a travelling matte that predates it stretches back to the post war era, while green screens themselves became popular in the 1970s for the likes of weather forecasts produced by TV networks where the chroma key technique they used was less likely to be tripped up. 

Their use has, of course, become ubiquitous since, for everything from virtual studios to the latest Holywood special effects montage (sorry, movie), to the portable ones that people can pop up behind themselves on Zoom calls. But, as anyone with a halfway decent GPU in their computer processing a blurred background on said calls without the aid of one can tell you, the need for them is slowly diminishing.

At the high- (and increasingly middle-) end, LED volumes simply throw a pre-rendered background live behind the actors with the ultimate goal that this will enable everything to be shot in-camera with little or no post production required.

Meanwhile other teams are investigating the use of AI to further take chroma key technology on to the next stage.

Magenta? Devine…

According to New Scientist ($) Netflix researchers have developed a new technique called Magenta Green Screen (MGS). In this technique, the actors are filmed against a background of bright green LEDs, while simultaneously being lit from the front with red and blue LED lights. Together, these create a magenta glow on the actors, effectively enabling them to pop more effectively out of the background.

So, now take your RGB recording digital camera and film what’s in front of you. The net result is that you get a green channel that records only the background, and red and blue channels that record only the foreground. The green channel can be swapped with the desired background footage with “even potentially tricky areas, such as transparent bottles or the area around strands of hair, working without problems.”

The difficulty lies in that you are then left with annoyingly magenta-tinged actors, which is where the AI comes in. This is trained on a picture or scene of them shot under normal lighting, and uses this reference picture to recolour them. It’s fast but not real-time fast yet, though there is a way to average the red and blue channels to approximate green enough for monitoring on-set.

It sounds complex and probably at the moment is, but Netflix is keeping on pushing it with the hope that it can reduce some of the current restitutions working with green screen as well as the donkey work involved in tidying up a shoot in post.

What androids dream of…

Apologies for the blatant Philip K Dick reference, but it’s a way of introducing the VFX start-up Electric Sheep that is taking AI and using a different route to solving the green screen issue.

It says that the only way to deliver the detail needed for big screen releases is to manually cut around foregrounds frame-by-frame a process; the aforementioned donkey work which it says takes around 6 hours for 1 second of footage and costs the industry over $1.5 billion per year.

That is a lot of money and could explain why the nascent company has just held an oversubscribed pre-seed funding round of half a million dollars.

It’s developed a cloud-native solution, Spotlight, that it says is an impressive 360x faster, processing 1 second of footage in less than a minute. And the results look very accurate too.

It's not saying what the 'existing solutions' are, but the Electric Sheep detail around the hair is impressive

Steven Sullivan, a partner at one of the three venture capitalists funding the company, Spatial Capital, (and former Senior Technology Officer of LucasFilm to boot) said: “Electric Sheep's cutting-edge cloud and AI platform is poised to revolutionise several core VFX workflows. They're taking a smart and pragmatic approach to accelerate some of the most tedious post-production tasks.”

Exactly how it works the company is not yet saying, and it’s obviously not aiming it at live work, but it does reckon that flawless background removal negates the need for green screens on set entirely and it will be interesting to see what it comes up with as it ramps its technology up.