A Lightworks developer talks about the kind of issue you meet when coding for an NLE on Linux

Written by Rob Fearnside

Editshare/RedSharkLightworks on blue background

Here's a rare peek into the complexities you might encounter when writing NLE software for Linux. It's a constantly shifting platform, and even the most subtle changes can cause headaches for developers. But there's a good outcome: having solved the problem, the NLE code is more robust than ever. (This is also a good introduction into how a modern NLE copes with Long GOP footage). Lightworks Lead Developer Rob Fearnside shares his insights

In recent weeks, some of Lightworks' Linux users started to report reduced performance in both new and old versions of the software.  After some investigation, it appeared that the slow-down was only affecting systems with a Linux kernel newer than 3.13.  In tests, kernel performance in 3.14 did seem to be slightly worse than earlier versions, but nothing that could explain the symptoms being experienced by users.  So what could possibly explain the sudden change in behaviour?  First, a little background information:

Many of the media files that editors have to deal with these days are compressed using codecs that are designed to create small files.  Whilst this is beneficial in terms of storage requirements and transfer times, it presents challenges for editing software like Lightworks.   

Such codecs typically store frames in groups (or GOPs) which usually means that to obtain a single image, you need to decompress a whole group of them first.  This is time consuming, and also requires a lot of memory; once a GOP has been decompressed, it usually makes sense to keep all the resultant frames in memory for as long as possible (so that when another frame is required from the same GOP there's no need to process the whole group again).  When you have tens or hundreds of files open at once, it doesn't take long before you're running low on memory.  

Constantly monitoring memory

Lightworks handles this by constantly monitoring the memory that's in use and automatically limiting (and ultimately emptying) its various caches when it notices that the amount of free memory remaining is getting low.  This technique relies on being able to ask the operating system to report how much memory is in available in total, how much has been used and how much is available for use right now.  

Starting with the Linux 3.14 kernel, free memory is reported differently - differently enough to cause the Lightworks software to start believing that there was no free memory left.  As a result, the memory monitoring code responded as it was designed to, by ensuring that all the frame and file caches were completely empty - but all the time.  This, as you might imagine,  had a severely detrimental effect on performance - every time a frame or file was added to the cache, it was immediately purged, forcing it to be re-decompressed the next time it was required.

Tracking the problem down was quite an endeavour, mostly because it seemed likely that the culprit would be a piece of code which was running more slowly than before, rather than one which was running just fine, but unexpectedly.  Thankfully, rectifying the issue was not a time-consuming process, and afforded us the opportunity to rewrite the code in such a way as to make it immune to further possible kernel changes in the future.

Tags: Technology


Related Articles

27 April, 2020

Maxon Cinema 4D S22 first release from the new subscription model

Maxon has updated Cinema 4D, the company's acclaimed VFX, 3D modelling, animation and rendering software, to version S22. This is the first release...

Read Story

27 April, 2020

LaCie Rugged drives are the perfect storage you need when going gets really tough [Sponsored]

There can't be many more extreme environments to be in than the Arctic tundra during a full white out blizzard. This is the sort of environment that...

Read Story

16 April, 2020

Mac Pro Gains Radeon Pro W5700X Graphics Card Option

Apple has finally added a more affordable and powerful GPU option... the AMD Radeon Pro W5700X

Read Story