Tupac Martir interview: How Virtual Production is being taken to new levels

Replay: In this exclusive interview, we spoke with world-renowned visual artist and director Tupac Martir about the incredible new technologies that make virtual sets a reality.

Tupac Martir is the founder of Satore Studio. A visual and lighting pioneer and icon, he has worked with clients such as Alexander McQueen, Beyonce, Elton John, and Vivienne Westwood. Much of his success is down to using brand new technology, sometimes against the prevailing ‘wisdom’.

Nowhere is this more apparent than with a new promotion that he put together for Quite Brilliant in the middle of the pandemic to show how effective virtual sets can be, even on a low budget with a tight shooting schedule.

As you can see in the video below, the results speak volumes for the technology, with many shots simply being indistinguishable from an actual set.

I spoke with Tupac about the technology that went into it and the promotion itself.

Background

I began by asking him about the background to the technology and how the shoot came about.

“We’ve been in this particular realm of the world for about two years now. It was never a fight because it was something we were keen on working with. We did a few things at Siggraph as part of a demo, and we had already scheduled about three or four months worth of R&D this year for VP (Virtual Production).

We were originally going to do all this R&D through May, June, July, and August, and then the pandemic happened. So we moved it all forward to March and got cracking on it. One thing we always wanted to be aware of was the flexibility of what we could do. I think it’s very important to have three or four different workflows within your ‘Swiss knife’ without making that many changes to the architecture or equipment you have.

I think it’s a combination of a lot of work that we’ve done in virtual reality, from stitching with Autopano, but also because of the things we learned doing Cosmos Within Us. There’s a lot of techniques that come from VR that translate very, very well into virtual production, especially when you’re starting to think about frame rate, making sure you don’t drop frames and the quality and optimisation of assets and things like that. What are the best ways to create a world that feels believable without running the machine down?”

Those mountains aren't real. Image: Satore Studio.

Why has the technology taken so long?

The concept of displaying a live CGI set behind an actor is pretty straightforward, so I ask Tupac why he thinks it has taken this long to do it effectively.

"It's a combination of things. I was the first person in the world to use BlackTrax back in 2011. I'll never forget in early 2011, I was in a room full of people whose thinking I really appreciate, and they were telling me that I was three or four years (from using the technology for virtual production on a set). I was told that Disney is trying it, Marvel is trying it, everyone is trying it. What makes you think that you can crack it?

To their dismay and my happiness, I'm that little guy who managed to pull it off and make it happen. For the longest time I kept yelling about how tracking was the next frontier in live entertainment, broadcast, and film, and no one listened to me.

I think that once that crept into live entertainment, people started seeing the potential of it and started asking how we can break away from just being on the green screen, how can I create movement and make sure that I am keeping the tracking?"

It's a testament to the way virtual production is presented that it now looks like it should be easy, but this isn't the case yet. Tupac continued,

"I know we want to say that virtual production is an oven-ready thing and that people can just grab it and do it, but it's not. It does take a lot of alchemy.

It's possible to do this stuff now because of how trackers have gotten to the point they are right now, whereas if you think of where live trackers were three or four years ago, you couldn't even find one if you wanted one.

It's taken so much time because computers needed to get better, the way computation works, Unreal Engine all of a sudden stepping away from what people consider a games engine. When people saw Unreal Engine 5, they all wanted it! That's over a year away from happening, but it tells you there's a hunger and knowledge people want to have about this. And so the more brains you put into it, it becomes much faster to move in the right direction."

Despite the complexities of making virtual production happen, things took a very different turn this year with the pandemic. I wrote back when the pandemic began that we would find a lot of technology that had previously been developing slowly would now accelerate as people wanted to find urgent solutions to new problems. This is a theory that Tupac found applied firmly to virtual production,

"Because of the pandemic, virtual production became expedited. I think everybody thought that VP workflows were going to come into our lives in about two years, and all of a sudden, the pandemic happened. Then suddenly, it's not just people like me, but your everyday consumer who's going, 'Hey, I need this!'

That living room isn't real, either. Image: Satore Studio.

Bringing elements together

"One of my favourite things about VP is that it brings all the departments together."

Tupac recounts an encounter with a director of photography regarding filming with green screens, "I remember talking to a DOP and him saying that he hated green screens because they shoot something for six months, but they have no idea what it's going to look like in the end. So I think for commercial production to work, you need the art department, the lighting department, the cameraman, the director all to be in sync for what needs to be done."

One of the significant advantages of virtual production is there's a real reference point for them to react to, unlike with a green screen. Tupac compares it to a situation that occurred within a stage production where actors were struggling to visualise and respond to an imaginary dragon,

"Once a director had to tell his actors in a show 'Ok, here comes the dragon!' and the actors had no concept of how big the dragon is. They weren't getting it. So they got an artist at the National to draw one in Tilt, and then they gave every actor a headset. Then they got it!"

It cannot be underestimated how important that sort of real feedback to an actor is. Superficially at least, it would appear that virtual production could affect the use of green screens. I ask him if this new technology makes the green screen dead?

"I still think that green screen is needed, but in a different way. I'm currently writing a movie that takes place mainly on a green screen. However, sometimes it's quicker and easier to just do something on the green screen. There's enough space for both technologies to exist together, but it's just a case of why, how, and what you are using it for.

I think that's the interesting bit, that VP has just added an extra card, or various cards, to the deck of possibilities that you can use to make your piece."

A look behind the scenes at how the virtual sets work. Image: Satore Studio.

VP on a low budget

Concerning the recent promo, I asked him what kind of restrictions he was under and what sort of considerations had to be made when conceptualising the shots that were required,

"For starters we had to do eight scenes in two days. When you think about that, you have to make your location into multiple places, and the sets have to be dressed. So we were basically left with two hours to do every single one of those worlds."

It wasn't just the schedule that was extremely tight. Virtual production comes with a whole new set of planning considerations, and many of those plans are difficult, if not impossible, to change once physical production has begun. He continued,

"You have to plan your shot, and you have to plan where the camera is going to live, constantly. There was one moment where they wanted the camera just a bit further out, and we looked at it, and we found that we were going to need 50 minutes to do it because we need to extend or rebuild elements and patch it all up together so that you only see the bits that you need to see.

A lot of things need to be fixed in pre-production. So if, for example, I'm going to grab a shot of you and there's a Christmas tree in the background, I'll texturise the tree from one side, but I leave the other side untextured. So if you come in tomorrow and say that we're going to do the interview from the other side of the tree, we haven't got anything there.

So you have to think a lot about what it is that you are trying to do so that you light it properly and you're not using up all the power of the machine. You'reYou're budgeting all the time. Where do I spend my capabilities within that scene? Do I want it so that I have the light moving dynamically? Okay, if I'm going to do that, I need to make sure all the other elements are as low quality as possible so that as the light is moving, nothing is going to slow it down. The last thing you want if you are shooting at 25fps is for it to drop below 25."

A virtual forest. The power of photoreal realtime rendering. Image: Satore Studio.

The future of virtual production

Regarding the future, things are looking extremely positive for the technology, and it is only going to get better,

"We keep looking to see how we can get that much better. The first 80 percent has a very clear path, but that last 20 percent is harder than anything else. And so we want to chip away at it slowly, but surely, to get it to a better place."

The technology has also become much more reliable. Tracking systems like Ncam are now incredibly robust,

"We've now run continuous tracking for eight hours, and it never crashed. Do you know how hard that is? We never had to recalibrate it, and it stayed stable the entire time. I can't tell you how much of a relief that is for us to know that the system is that stable and works very well now. It's one less thing to have to worry about.

So with that in mind, we ask what other things can we do on top of that, and what other shots can we push even further?"

For all his massive enthusiasm for this new technology, Tupac is also pragmatic when it comes to how it should be used and is keen to emphasise the reasons why this technology exists in the first place,

"One thing I always talk about is that you always have to bring humanity to the technology. When the CGI just takes over completely or when the digital element just becomes the main element, your characters have lost. We have to remember that we do all of this for the story at the end of the day. If the story gets lost because you are too busy showcasing the technology, then you've lost."

Keep up with Tupac's work on the Satore Studio website.

RedShark would like to credit Ncam and Andrea Aiello from Liaison PR for the original story, arranging the interview, and Tupac Martir for his time to chat about these most important new technologies.