As I’ve mentioned in the previous post, I’ve finally tried rust-clippy to see what issues and suggestions it will have on my code. The results are not disappointing if you take the tool name seriously.
The output (at least in my case) can be divided into four categories: actual bugs spotted, useful suggestions, (mostly) useless suggestions and counterproductive suggestions.
Let’s start with actual bugs spotted. It spotted two cases when condition has identical code in both branches. So I had to remove such useless condition in one case and fix the data source taken in another. Also an error on casting pointer to several-byte buffer as pointer to integer. Depending on stack alignment and CPU architecture this may result in a buggy load so better fix it (but then this issue will be mentioned again when we get to the third category).
Useful suggestions were much more plentiful. I started developing NihAV in Rust a bit more than two years ago and the language has changed quite a bit since then so some parts of code were a bit outdated (e.g. nowadays you can simply write Struct { field1, field2 } if you initialise it from field1 and field2 variables and have the same fields; previously you had to write Struct { field1: field1, field2: field2 }). Plus of course there are features I did not know about either because they were introduced later or I simply did not think about it being possible. Here are some examples I encountered:
vec![elem; size] to create vector of fixed size filled with element (previously I invoked Vec::with_capacity() and then vec.resize() which is more error-prone);start..=end instead of start..end+1 (IIRC this was still an unstable feature when I learned about it so no wonder I forgot about it);println!() is equivalent to println!("") (and guess who uses it for debugging);ptr.offset(size as isize) you can simply write ptr.add(size);slice.swap(idx1, idx2) for the cases when you need to swap two values in an array;u32 to u32);Vec when passing the reference to slice would work as good (and even save some MiNicycles on access); same for borrowing Box.There were some stylistic suggestions that I found good (replacing some indexed loops with loops on iterators and copying slice with a call instead of loop) but mostly they belong to the next category, useless suggestions.
Yes, that’s where rust-clippy starts to live up to its name. Most of its suggestions were in the format “hey, you’re trying to use clear imperative programming to do the work that can be expressed with iterators in 10x text”. I understand that it’s more to the Rust style and it helps to prevent errors in some cases, but when it suggests to replace simple loop like for i in 3..7 { dst[i] = src[3 - i]; } with something that won’t fit onto single like I say it’s too much. Or how it suggested to change simple mem::transmute::<&mut $inttype, &mut [u8; $size]>(&mut buf) into &mut *(&mut buf as *mut $inttype as *mut [u8; $size]) (sure, it does the same but I find the former easier to comprehend).
Similar story with X as u32 versus u32::from(X). I understand it helps catching the cases where X changes type and cannot be converted to destination type losslessly any more. But it breaks formatting and makes it unnatural (and close to C++) in the sense that previously the code was read like “calculate X and cast it to type” and now it’s “convert to type from calculated X”. And by breaking formatting I mean that I have a lot of code in form
let a = br.read(4)? as usize;
let b = br.read(6)?;
let c = br.read_bool()?;
let d = br.read(3)? as u64;
and by using u64::from(br.read(3)?) I’d break nice tabular form for no obvious merit.
di
Another cases is when rust-clippy suggests refactoring conditions. When I write
if is_intra {
if block.coded { do a }
else { do b }
} else {
if block.coded { do c }
else { do d }
}
I format it this way because it reflects the logic of the code and I do not want to make it into } else if block.coded { do c } else { do d }. Similarly if I don’t join let a; if foo { do stuff; a = val; } else { a = val2; } into single let a = if foo { ... val } else { val2 }; it’s because I have enough code in that condition that I don’t want to indent it even further (sure, make fun of me because I worked in text mode and still don’t like lines longer than 80-100 characters).
There’s more like “you should implement Default trait for each structure with new() method” and “you should write constants like 0x00_0001” and “enums should not have different prefix in the name, you should consider removing Mode from Mode16k, Mode5_5k and Mode8k” but let’s move to counterproductive suggestions already.
First issue is that rust-clippy interacts with macros quite poorly. For example I have a macros that expands 4-byte array into 32-bit tag and it suggests using u32::from(arr[0]) instead of (arr[0] as u32) but if you do that it will have an error instead of warning because it can’t check types in macro. Also here’s one of the WTFiest examples: I have a macro for filter implementation that takes coefficients as arguments and calculates the value. It has a code -$c1 * $src[$off + 1] and when $c1=-1 you have a warning “about –x is double negation”. Actually I remember that example from Feuer’s C. Puzzle Book. where #define NEG(x) -x NEG(-x) expanded to –x (and thus I put all macro arguments into parentheses) but I expected Rust to handle that correctly (otherwise this WTF goes to linter). In either case -(-1) is what I meant in that case and that’s how it works.
And now the most infuriating example that the tool treats like a warning or an error depending on situation.
As I write video decoders, I have a need to apply a filter or transform on some data. And IMO this form:
dst[off - 2 * step] = a;
dst[off - 1 * step] = b;
dst[off + 0 * step] = c;
dst[off + 1 * step] = d;
I think this conveys the meaning and looks much better than
dst[off - 2 * step] = a;
dst[off - step] = b
dst[off] = c;
dst[off + step] = d;
with whatever spaces you insert there (and it’s not possible to use iterators here quite often too). But rust-clippy treats that as a warning if you’re multiplying by one, adding zero or constant multiplied by zero and gives outright error on variable multiplied by zero or an expression like N - N (which happens when you have constant base and one of the offsets equal to it).
So my experience with this linter was mostly negative—all those neat things were out-weighted by the sheer amount of useless warnings (and I don’t like to add dozens of #[allow(clippy::useless_warning)] to lib.rs for many modules) and some stuff that would simply break code clarity for no obvious benefit. Overall, it’s a nice tool and though I got something useful from it, it may be a beginner in the language aka ordinary M$ Office user who would benefit the most from it and it will mostly hinder people writing advanced code unless you turn off a lot of warnings and even some errors.
Since the clean-up work on NihAV is done and I progress with Truemotion VP3 decoder, it’s a good time to talk about what I’ve actually done—there’s even more material to write waiting in the queue.
The intent was to make all frame-related stuff thread-safe and improve efficiency a bit. In order to do the former I had to replace most of the references from Rc<RefCell<T>> to Arc<T> and while doing it I introduced aliases like type NAFrameRef = Arc<NAFrame> and .into_ref() methods to convert object into ref-counted version. This helped when I tried switching from one implementation of reference counter to another and will make it easy to switch again if I ever need that (hopefully not). Now about improved efficiency and how it’s related to the ref-counting.
There’s a straightforward way of dealing with frames: you allocate the picture, fill it, dispose, allocate a new one, etc etc. And there’s a more effective way: you allocate several pictures at once, select an unused one, fill, return to the pool when it’s not needed any more. That is where reference counting comes into play and where Rust default structures don’t help. Frame pool owns the reference and decoder gets a second copy. And Rust Arc is intended for single ownership: when you try to access the shared object it will simply clone it so you end up working with a copy (which defies the purpose). So I had to NIH my own NABufferRef<T> which keeps reference counts and still allows shared access even for writing (currently it does that in all cases but if I need to add some guards the API won’t have to be changed for that). The implementation is very simple: the structure contains a raw pointer to a structure that contains actual object and AtomicUsize counter. The whole implementation is ~2.2kB relying just on std crate.
And finally I’ve made a picture pool. The difference between picture and frame is all additional metadata picture should not care about (like timestamps, stream information and such). Because of the design decisions I have three different picture formats (implemented for 8-, 16- and 32-bit element sizes, Rust does not like aliasing after all), which means I need to provide decoder with all three picture pools because we can’t say in advance which one codec will use (if at all—the option to allocate new non-pooled pictures is still there). Also I want to keep those pools external in case the code around it wants to do keep more pictures in it (e.g. 2-3 pictures required by decoder and 25 pictures pre-buffered for the display). This resulted in a structure called NADecoderSupport that contains picture pools and may have something else added late. Of course people might argue that it’s much better to have AVCodecContext with a myriad of fields you can set directly or via utility functions but I’d rather not have one single structure. Though it might be a good place to put various decoder options there (so that decoder can ignore them at its leisure).
Since I said I did it to increase efficiency I should probably give some numbers too: RealVideo 3/4/6 decoders now use buffer pool (for three frames obviously) and reallocate it on format change. Decoding time got reduced by 4-5% from using the pool. Currently I don’t care about speed much but I may convert more decoders to it if the need arises.
In conclusion I want to say that even I did not enjoy doing that work much, it was needed and gave me some experience plus some improvements in code and design. So it was not a wasted effort.
P.S. I also installed rust-clippy since it’s in stable now and tried to fix errors and warnings it reported. But that is a story for another post.
Today I want to talk about local dynasty that was rather short-lived but left quite an impressive legacy.
So, about a thousand years ago in the South-Westernmost corner of Germany there was a noble lord by the name of Berthold who started a dynasty of Zähringen and his eldest son founded the line of margraves of Baden. And while the dynasty of Zähringen has only six rulers (Berthold, Berthold, Berthold, Konrad, Berthold and Berthold) it’s known for building and rebuilding a dozen of towns (yes, exactly twelve), some of which are important even to this day. Since I have nothing better to do at the weekends I decided to visit them all.
Berthold I founded Weilheim an der Teck, the only Zähringer town founded in Württemberg.
Berthold II is known for founding the Abbey of St. Peter (and a town forming near it), Freiburg (in Breisgau of course) and reforming a Swiss company Rheinfelden AG (though Swiss claim that AG stands for canton Aargau and not joint-stock company).
Berthold III and Konrad I somehow evaded the traditional duty and built nothing. Though Freiburg still honours Berthold III for developing Freiburg into proper city and not a mere settlement by the castle.
Berthold IV, son of Konrad I, decided to follow the suit of his grandfather and founded Freiburg (this time in Üechtland). He also rebuilt Murten, founded Burgdorf and Neuenburg (in Baden, don’t confuse it with Neuchâtel).
Berthold V probably remembered that his ancestors were margraves of Dietrichsbern (or Verona as locals call it) and decided to found his own Bern (in Üechtland). He also rebuilt and developed Thun (the castle he built there is still one the best places there), Bräunlingen and Villingen.
Since Berthold V had no offspring, the lands of duchy of Zähringen went to different other noble houses (including dukes of Fürstenberg who are famous for building a lot of castles in Czechia) and ended in hands of Swiss house of Habsburg (who, obviously, are famous for ruling Austria). Thus one can see former Austrian towns even on Rhine.
Fun fact: some of those places have suspiciously similar coats of arms (all images are from Wickedpedia).
Bern: 
Neuenburg: 
Karlsruhe: 
Would you believe it has nothing to do with heraldic colours of Zähringen being red and yellow (also adopted by margraves of Baden, one of which founded Karlsruhe)?
Now, about the towns and my impressions on them.
Another interesting thing is how much the founders are honoured in all those places. Most of them have Zähringerstraße or Zähringerplatz (and Zürich has both despite Berthold IV being just a vogt there). There are exceptions though: Freiburg in Breisgau has Zähringerstraße in a suburb called Zähringen and actual city has Bertoldstraße instead; Weilheim simply mentions that it’s one of Zähringerstädte and has their coat of arms near the church; Freiburg im Üechtland has its main bridge named after Zähringen; Thun and Murten do not seem to have any mentions. While many other places in Baden have Zähringerstraße (like Karlsruhe or Heidelberg) probably because of the relation to the house of Baden, I think they deserve recognition for building new towns. It’s a pity there were not so many nobles like them.
Since obviously I have nothing better to do, here’s a description of loop filter in Bink2 as much as I understand it (i.e. not much really).
First, the loop filter makes decision on two factors: motion vector difference between adjacent blocks is greater than two or it selects filter strength depending on number of coefficients coded in the block (that one I don’t remember seeing before). The filter is the same in all cases (inter/intra, luma/chroma, edge/inside macroblock), only the number of pixels filtered varies between zero and two on each side (more on that later). This is nice and elegant design IMO.
Second, filtering is done after each macroblock, horizontal edges first, vertical edges after that—but not necessarily for all macroblocks. Since BIKi or BIKj encoder can signal “do not deblock macroblocks in these rows and columns” by transmitting set of flags for columns and rows.
Third, in addition to normal filtering decoder can do something that I still don’t understand but it looks like whole-block overlapping in both directions (and it is performed in actual decoding but I don’t know what happens with the result of it).
And the filter itself is not that interesting (assuming we filter buf[0] buf[1] | buf[2] buf[3]:
diff0 = buf[2] - buf[1] + 8 >> 4;
diff1 = diff0 * 4 + 8 >> 4;
if (left_strength >= 2)
buf[0] = clip8(buf[0] + diff0);
if (left_strength >= 1)
buf[1] = clip8(buf[1] + diff1);
if (right_strength >= 1)
buf[2] = clip8(buf[2] - diff1);
if (right_strength >= 2)
buf[3] = clip8(buf[3] - diff0);
Strength is determined like this: 0 — more than 8 coefficients coded, 1 – MV difference or 4-7 coefficients coded, 2 — 1-3 coefficients coded, 3 — no coefficients coded.
Overall, the loop filter is nice and simple if you ignore the existence of some additional filter functions and very optimised implementation that is not that much fun to untangle.
Update: the alternative function seems to be some kind of block reconstruction based on DCs. In case it’s intra block with less than four coefficients coded it will take all neighbouring DCs, select those not differing by more than a frame-defined threshold and smooth the differences. I still don’t understand its purpose in full though.
I’m proud to say that NihAV got TrueMotion 2X support. For now only intra frames are supported but 75% of the samples I have (i.e. three samples) have just intra frames. At least I could check that it works as supposed.
First, here’s codec description after I managed to write a working decoder for it. TrueMotion 2X is another of those codecs that’s closer to TrueMotion 1 in design. It still uses the same variable-length codebook instead of Huffman coding (actually only version 5 of this codec uses bit reading for anything). It also uses “apply variable amount of deltas per block” approach but instead of old fixed scheme it now defines twenty-something coding approaches and tells decoder which ones to use in current frame. That is done because block size now can be variable too (but it’s always 8 in all files I’ve seen). And blocks are grouped in tiles (usually equivalent to one row of blocks but again, it may vary). The frame data obfuscation that XORs chunks inside the frame with a 32-bit key derived in a special way is not worth mentioning.
Second, the reference is quite peculiar too. It decodes frame data by filling an array of pointers to the functions that decode each line segment with proper mode, move to the next line and repeat. And those functions are in handwritten assembly—they use stack pointer register for decoder context pointer (that has original ESP saved somewhere inside), which also means they do not use stack space for anything and instead of returning they simply jump to the next routine until the final one restores the stack and returns properly. Thankfully Ghidra allows to assign context argument to ESP and while decompile still looks useless, assembly has proper references in the form mov EDX, dword ptr [ctx->luma_pred + ESP].
And finally, I could not check what binary specification really does because MPlayer could not run it. At first I tried running working combination of WMP+Win98 under OllyDbg in QEMU but it was painfully slow and even more painful to look at the memory state. In result I’ve managed to run TM2X decoder in MPlayer which then served as a good reference. The trick is that you should not try to run tm2X.dll (it’s really hopeless) but rather to take tm2Xdec.ax (or deceptively named tm20dec.ax from the same distribution that can handle TM2X unlike its earlier versions), patch one byte for check in DLL init and it works surprisingly well after that.
So what’s next? Probably I’ll just add missing features for the second TM2X sample (the other two samples are TM2A), maybe add Bink2 deblocking feature—since I’d rather have that decoder complete—and move to improving overall NihAV design. Frame management needs proper rework before I add more codecs—I want to change into a thread-safe version before I add more decoders. Plus I’ll need to add some missing bits for a player. There’s a lot of work still to do but I’m pleased that I still managed to do something.
So NihAV finally got Discworld Noir BMV support and I’ve tested it on all samples from the game to see if it works correctly. Here’s a sample frame:
(I still remember the song Samael plays there and have it somewhere ripped in its full MPEG audio layer II glory).
Now I want to talk about the format since it’s quite different from anything else. BMV used in Discworld II was simple but with two quirks: it employed integer coding using variable amount of nibbles (that were read as bytes but a nibble could be saved and used later) and it could decode frame either from the beginning to end or from end to beginning (reading frame data from the end too!). DW3 BMV is even stranger and let’s start with audio part. Audio codec is very simple: you have 41-byte block with one byte signalling which quantised values tables should be used for both channels and 32 indices for each channel packed into 16-bit words. The main peculiarity is that data is aligned to 16-bit and mode byte can be either in the beginning or at the end of the block. That’s a bit unusual but not strange. Well, it turns out it aligns for the absolute position in a file so my demuxer has to signal whether audio data was at even or odd position. And video is even stranger.
As I wrote previously, video codec is 16-bit now and still employs nibble variable integer coding and copy/repeat/new pixels mode. Luckily there is no backwards decoding mode yet the codec is tricky without it. First of all, where previously there were just three plain modes now we have combinations of those with bytes or nibbles signalling what should be done (i.e. copy/repeat/put new pixels fixed amount of times and then do the other operation another fixed amount of times). And they have different meaning depending on what was the last operation (copy/repeat/put for fixed amount or with arbitrary large one). And if there’s a nibble left unread after last operation or not. But that’s not all! While previously reading new pixels meant just reading a byte, 16-bit pixels can be compressed a bit more. In result we read 1-3 bytes per pixel: first read index byte, remap it, if it’s in range 00..F7 then return pixel in an array, if it’s in range F8..FE then read another byte and use it as an index in one of seven secondary “palettes”; if it’s FF then simply read explicit 2-byte value from the stream. The reference simply used an array pointing to the various functions performing this. And of course palettes can be updated in the beginning of each frame.
Surprisingly, decoder implementation takes about 28kB with a quarter of it being tables. That’s for both audio and video decoder. This is on par with other game decoders (GDV, Smacker and VMD) and feels significantly smaller than the reference (which is about 30kB in a stripped assembled .o file and over 200kB as assembly).
Overall it was hard to comprehend and tricky to debug too. Nevertheless now it’s over and I can probably move to TrueMotion 2X. Or whatever I decide to do when I’m bored enough.
I’ve made some significant progress on REing Discworld Noir BMV.
First, I put opcodes meaning into table (it’s probably the only case when I had to use spreadsheet for REing) to figure out the meaning.
Normal mode of operation have these opcodes:
00**00*0 — perform extra-long copy;00**00*1 — perform extra-long invoking of pixel functions;00**xxx0 — copy xxx-1 pixels;00**xxx1 — invoke pixel function xxx-1 times;xxxx000y — copy xxxx+3+y pixels;xxxx001y — invoke pixel function xxxx+3+y times;< xxx0yyy0 — copy yyy-1 pixels and then invoke pixel function xxx-3 times;xxx0yyy1 — invoke pixel function yyy-1 times and then repeat last value xxx-3 times;xxx1yyy0 — copy yyy-1 pixels and then repeat last value xxx-3 times;xxx1yyy1 — invoke pixel function yyy-1 times and then copy xxx-3 pixels.Then depending on last operation performed mode is changed to: something special for 00****** opcodes, no change for repeat, “after copy mode” and “after pixel func mode” for obvious cases.
After copy mode opcodes:
xxxxxxx0 — invoke normal mode opcode xxxxxxx1;00**00** — extended repeat;00**xxx1 — repeat last value xxx-1 times;xxxx00y1 — repeat last value xxxx+3+y times;xxx0yyy0 — repeat last value yyy-1 times and copy xxxx-3 pixels;xxx0yyy1 — repeat last value yyy-1 times and copy xxxx-3 pixels;xxx1yyy1 — repeat last value yyy-1 times and invoke pixels function xxxx-3 times.After pixel function mode is simple: xxxxxxx0 opcode is after copy mode xxxxxxx1 opcode and xxxxxxx1 opcode is normal mode xxxxxxx0 opcode.
The special modes may have secondary opcodes that usually boil down to the same thing: either do some more of the same and proceed normally or calculate next opcode instead of reading it from the stream.
And now the second thing: I’ve managed to rip the relevant code from the disassembly, fix it for NASM to handle plus added some fixed to make it possible to invoke externally and linked it against my own small program that parses BMV file, decodes the first video frame and dumps it into file. That approach works so I have something to test my new implementation against. Also since NASM makes all labels visible it’s easy to make debugger report each opcode as it gets called.
To summarise, I have good understanding of the algorithm and I have a working binary specification. This should be enough to finish it soon (unless I get distracted by something else of course).
As I’ve mentioned before, NihAV now can decode Bink2 files more or less decently. I don’t have many samples but I can decode all samples I could find from KB2f to KB2j quite well (the only exception is KB2a — there’s only one partial sample known with no indication which game uses it and no version of RAD tools understands it either).
I’ve omitted support for full-resolution Bink2 files (no samples) and reconstruction is not perfect because there’s an in-loop deblocking filter with some additional crazy functions invoked in some cases. It’s messy and does not affect actual bitstream decoding so I’m not going to work on it now. Maybe if I get some inspiration later…
Anyway, I moved to REing Discworld Noir BMV format instead. While the game is nice, it’s hard to run on any modern OS and there’s almost no hope for its engine being reimplemented. So maybe I’ll be able to re-watch cutscenes from it… I’ve figured out container and audio long time ago, video is not that easy. It seems to be an upgrade of Discworld II BMV that outputs 16-bit video but the way it’s implemented is baffling.
While locating the functions responsible for the decoding was easy, understanding them was hard. For the disassembler. No, the code was recognized fine but its flow makes even disassembler freak out. Here’s how it works:
00-3F, then you usually have to construct length word, perform some pixel output loop and then jump to another opcode handler which performs some operation and then jump to the address calculated by the previous operation;Anyway, I’ve made some progress and I reckon it will be possible to support this format in NihAV though maybe not soon enough.
After managing to decode the first frame of KB2g variant I had three options: try to decode the other frames, try to decode other variants or do nothing. While the third option is the most appealing and the first option is the most logical, I stuck with the second one. So now I can decode the first frame of KB2f variant of Bink2 as well. Unfortunately the only (partial) KB2a sample I know is not supported, probably it’s a beta version that was tried on one game like Bink version b. Beside a small surprise in one place bitstream decoding was rather simple. Inter-frame support should not be that hard but it might get messy because of the DC and MV prediction.
And while talking about REing Bink I should mention that I’ve tried Ghidra while doing KB2f work. It is a nice tool that sucks in some places (not having a good highlight for variables, decompiling SIMD code results in very questionable output, the system being Java-based and requiring recent JDK—that’s the worst issue really) it works and produces decent results (including the decompiler). Also since it has 16-bit decompiler support maybe I’ll manage to figure out how those clips in Monty Python & the Quest for the Holy Grail are stored.
I should start documenting it too.
Insignificant update: okay, now it decodes inter-frame data correctly too and the only thing left is to make it reconstruct them correctly. Also I’ve updated codec information on Multimedia Wiki. Actually now it works quite okay so I’m not going to pursue it further. I have no real interest in Bink2 decoding after all.
It took a long time but finally I can decode the first frame of Bink2 video (just KB2g flavour though but it’s a start).
At least the initial observations were correct: Bink2 codes data in 32×32 macroblocks, two codebooks for AC zero runs, one codebook for motion vector components, simple codes with unary prefix for the rest.
If you wonder why it took so long—that’s because I’m lazy and spend an hour or less a day on it. Also while the codec is simple in design it’s a bit complicated in implementation. While previous version related on format sub-version to decide which feature to use, Bink2 uses frame flags to decide which feature to use. For instance, flag 0x1000 signals that there are two bit arrays coded that tell when to read an additional flag during CBP decoding that tells which one of two codebooks should be used during AC decoding later. And flag 0x2000 essentially tells to use different bitstream decoding (like motion vectors decoding or block type decoding). Or the fact that it employs DC and MV prediction that usually has four cases (top-left macroblock, top block, left block, some block inside) plus WMV1-like handling of DC prediction in inter-frames (i.e. it calculated DC for inter blocks and uses them for prediction). And of course DC prediction for inter blocks works a bit different. Plus it tries to track internal state by packing all flags into 32-bit word and updating it for each block (two bits are for signalling top row, one—for macroblock being the leftmost one, some bits are copied from frame flags etc etc). So there’s a lot of nuances to take care of.
And that’s not counting the fact that current Bink2 player can’t decode versions prior to KB2g at all. Since I have some KB2f samples along with an old Bink player that can handle them, I guess I’ll support them eventually.
Okay, I’ve finally done all the low-hanging fruits and now the progress is blocked by the need of reverse engineering (namely, Bink 2, Discworld Noir BMV and TrueMotion 2X) so I don’t expect anything major happening to the project in near time.
Meanwhile it’s a good opportunity to talk about how NihAV is (mis)designed and how it should work in principle.
In principle there are three scenarios: playing the file, encoding raw input and transcoding. And first two can be considered to be transcoding with known raw inputs or outputs plus specific requirements to them.
So, what needs to be done if we want to transcode something into something else:
If this sounds obvious—that’s because it is. If it sounds trivial then you miss wonderful complications that may occur at every stage.
Detecting input format may be not trivial because not all containers have proper markers and you can either rely on extension or try every possible demuxer to see if it works. NihAV allows flexibility in this choice: there is a standalone nihav_core::detect module that allows to probe file for known container format and it will return a most probable container name and score (either extension match or marker check match). This way you can find out format even if you don’t have demuxer registered for it. And if you’re desperate you can try opening that file with all registered demuxers to see which one fit (my tools won’t do that probably but it’s easy to do even with current API).
Then you have input streams. In some cases you have proper streams with packets, in other cases you have raw stream which you have to split into packets yourself. One solution would be to mark those streams as requiring parsing and automatically insert a parser as well. NihAV design (not yet implemented since I don’t need it yet) tells to report elementary stream as its own type and return data that should be parsed elsewhere. The same applies for files that are raw elementary streams. If you want to be able to decode /dev/random as MPEG Audio Layer I you should force the proper parser and decoder yourself.
Actual decoding should be done in traditional way: a packet goes it, a frame comes out (in the exceptional cases it should report DecoderError::TryAgain but it’s really for exceptional cases; previously I thought about inserting queues of packets/frames between processing units but finally decided against it). Decoders should do no reordering. If you wonder how the caller should be able to reconstruct the proper order, it’s simple enough. There is a global nihav_core::registry (as well as the format detection module it may be moved into separate crate in the future) that contains information about all known codecs. And knowing the codec information you can create a proper frame reshuffler (none for many formats, the one that reorders using frame type for I/P/B-frame codec and something more complex for codecs with pyramid referencing scheme like H.264 or H.EVC).
While I don’t have this functionality yet it’s not that bad since currently I simply output frames into single images and sleepsort works fine (i.e. if frame00004.ppm is created before frame000002.ppm it’s no big deal for image viewer).
Encoders are something I have not considered yet.
Format negotiation is currently implemented only for video frames in nihav_core::scale module. It works as I described it three years ago: converter adds necessary stages for frame unpacking, rescaling, format conversion and packing. Of course if some stage is not needed it’s not added. The only annoying thing is that internally I use typed frames, so frames with an array of bytes, an array of 16-bit ints and 32-bit ints should be handled separately. At least that means I internally always work with native-endian data and the actual converting is done only when the data is read or written. And of course this is mostly just proof of concept than fully-working solution but it can convert e.g. RGB565 into YUV410 while rescaling it too.
And it’s worth mentioning that another fun issue is timestamps and how to interpolate them when they are missing. Since there’s no good universal solution I’d not try to invent something complex either and I’ll probably stick to the laziest approach.
Overall, I intend to create nihav-pipeline crate (that’s a tentative name) for handling some of the middle-level stuff like frame reordering and timestamp related issues that can be served as a base for both transcoder (definitely not coming this year) and player (this one might actually appear this year). But for now nihav-tool that decodes files into sequences of images and wave files is enough for experiments. I care mostly about decoding after all.
As you can guess from the title NihAV got some support for Duck formats, namely TrueMotion 1 and TrueMotion RT. The implementation was rather straightforward except that it took some additional work to support 16-bit video buffers.
Of course I made sure my new TM1 decoder supports decoding sprites. Here’s an example of such sprite picture:
The hardest part was finding a sample.
I can’t sanely support transparency though since it uses 6-bit alpha with RGB555 image and while I can support such format quite easily I’d rather not.
If you wonder about the details of sprite support, it’s almost the same as ordinary inter-coded 16-bit TM1 with some nuances:
00 mean the next four pixels should be skipped (and predictor reset to zero), bits 01 mean it’s opaque sprite data and bits 10 mean it’s sprite data with transparency info present;That information comes from our old trusty source of information called VPVision source code dump (which was used to understand TrueMotion 1, 2 and probably DK3/DK4 ADPCM (and maybe VP3 but I’m not sure at all). Also it turns out to contain TrueMotion RT encoder source code as well (which could be used to reconstruct the decoder but I forgot about it at the time and used the binary specification instead).
And now I’d like to talk about Duck codecs in general.
The codecs from this family can be divided into three groups:
Those codecs were used mostly in video games but TM1 was also licensed to Horizons Technology.
The idea behind TM1 is very simple: you split video into 4×4 blocks, predict each pixel from top and pack using quantised deltas and fixed codebook looking more like Tunstall codes (i.e. output code is always a fixed length of one byte but it may correspond to a variable length sequence of input codes). Also depending on quality frame blocks have different number of colour difference deltas per block (1, 2 or 4).
TrueMotion RT is an adaptation of TM1 for real-time video capturing (hence the name). In this case video is coded as planar YUV410 using fixed set of deltas with index taking 2, 3 or 4 bits. But the general coding idea (top and left prediction, delta quantisation and coding its index) remains the same. It uses the same frame header obfuscation so it’s probably an elder sibling of TrueMotion 2 (and its name is more like TrueMotion RT version 2.0 and not TrueMotion 2 RT but the details are unclear). There are different versions of the codec, for example Star Control II: The Ur-Quan Masters on 3DO used a special TM1 format split into several files: .hdr for global information (including quantised delta sets), .tbl with codebook definition, .duk with actual frame data and .frm with the frame offsets for .duk file. It’s a pity I can’t support it without very special handling.
TrueMotion 2 gets rid of single static codebook and packs appropriate data (deltas for different-resolution blocks, motion vector data, actual block types etc etc) in separate segments with their own Huffman codes. There are many improvements but the codec still operates on 4×4 blocks with horizontal and vertical prediction of each symbol.
There is not much known about TrueMotion 2X but so far it looks like maybe slightly improved TM2. Hopefully it will be clearer if I manage to implement a decoder for it.
And finally there were two simple ADPCM codecs accompanying video (usually TM2), there’s nothing much to say about those.
This was the age when Duck codecs became widely known and accepted, when various companies licensed them for their own needs and when it was really the golden age for them.
It all starts with TrueMotion VP3 that set the standard for the following codecs. It employed the a bit non-standard 8×8 DCT, referencing last intra frame as an alternative to referencing just the previous frame (later knows as golden frame), with various types of information grouped together instead of interleaving it all, and with coefficients coded as tokens (EOB, zero run, plus-minus one, plus-minus two, large coefficient token and such). The same approach would be used for subsequent codecs as well. Of course it briefly enjoyed the renaissance when Duck decided to put it into open-source and Xiph Theora was created on its base (and since there were no other free and open-source video codec alternatives it was destined to have popularity and success before something better comes).
TrueMotion VP4 was mostly the same but with different coding method for some data types. Maybe it was the first codec to move edge loop filtering from being performed on the frame to being performed on temporary block used in motion compensation but I’m not entirely sure.
TrueCast VP5 was the first in the series to employ their own version of static binary arithmetic coder mostly known as bool coder. That means that instead of updating bit probability after each decoding using that context as CABAC does, frame header encodes fixed probabilities (or just updates from the probabilities in the previous frame) and uses them for decoding.
VP6. Probably the most famous of them all since it was used in Flash videos. From technical point of view it’s just small improvement in some details over VP5. I suspect this was the first codec in the series that introduced selecting random frame as the next golden frame (previously it was just last intra frame, now any inter frame can signal that it should become golden).
VP7. This is the first installation in the series that was based on H.264 ideas like 4×4 transform and spatial prediction.
And of course there’s AVS, an audio codec inspired by AAC LC that accompanied some VP5-VP7 videos.
While the design direction has not changed much, the codecs themselves mostly belong to the niche provided by their current owner and hardly used anywhere else. For now we have VP8, VP9 and VP10 (aka AV1).
I hope there will be more to write about those after I write decoders for the rest of them and learn the shameful details of their design in the process.
I have not written anything about NihAV for the last two months. If you thought it’s because nothing was done you’d be mostly correct (but not in thinking about the project at all—this is pointless). And yet there has been some progress recently (but before that I spent a whole month not doing anything NihAV-related).
First thing, I’ve finally split NihAV into smaller crates. This was done in order to reduce compilation and testing time for separate components. It started to feel a lot like FFmpeg and Libav times except that I don’t have to rerun configure and recompile everything in case I want to add a new decoder. Now codecs reside in standalone crates that contain just relevant decoders and demuxers so “write code—fail to compile—fix—run tests” cycle goes faster now.
And here’s the new structure:
nihav-core — the core. It contains the definitions for basic stuff like packets and video/audio buffers, I/O routines (byte- and bitreader), DSP and common video stuff for H.263-based decoders;nihav-commonfmt — for various common formats. Currently it has all leftover codecs I could not move elsewhere (and AVI demuxer). Maybe in the future it will grow too large so I’ll split out some groups like speech codecs into new crates;nihav-duck — for Duck game codecs up to VP7;nihav-game — for various game codecs and containers;nihav-indeo — for Indeo video and audio codecs;nihav-rad — for totally RAD formats like Smacker and Bink;nihav-realmedia — full RealMedia support (except for common decoders in nihav-commonfmt);nihav-allstuff — the crate that binds them all so you don’t have to search for decoders in separate crates.Of course there’s nihav-tool to perform the actual decoding and test whether the code works but it’s always been a standalone crate.
This split required some changes in the decoder and demuxer handling. Previously I could get away with one table where all decoders (or demuxers) were registered and then search for an appropriate decoder there. Now it is not possible since there are many crates with individual codecs enabled or disabled in configuration. In result I had to introduce a structure called RegisteredDecoders and have crate-specific function for registering all decoders featured in crate in it. And nihav_allstuff::nihav_register_all_codecs() simply invokes them all for convenience. It’s exactly the same with demuxers too.
N.B.: I should probably write a separate post on overall NihAV design, missing parts and excuses for certain decisions.
Second, as you could spot from the crate names above, instead of trying to make NihAV do something useful (you have rust-av for that though) I decided to focus more on supporting various game formats.
So far I want to support the following formats:
ScummVM where they did not understand how decoding lengths works and the code still looks like a beautified assembly. For libavcodec decoder I tried my best to simplify the code and give better names to the variables. And when implementing the decoder for nihav-game I found out that “read byte, advance pointer, output nibble and save another one for later” leads to the same results but with a much cleaner code;After that it would be nice to work on actual player for the supported formats, which opens a new can of worms like proper frame reordering and format conversion, but I guess I’ll have to deal with that one day.
Anyway, back to doing nothing.
In order to celebrate the fact all important Dingo Pictures works were found (there might be more though) I decided to visit the birthplace of those masterpieces. You can find their address at the website or at Baidu Maps (with some reviews, mostly in English and Swedish).
Without further ado I present to you the studio:
Yes, this is really the place of people who gave us (or at least me) many hours of fun and joy.
Here’s a postbox with some familiar names if you don’t believe it.
And if you search local news you might learn that those people were local legends probably even before they were immortalised by cartoons. But before only the area around Frankfurt on Main knew about their talents and now the whole world does. And I’m grateful for having an opportunity to discover this whole new genre of German anime which would not be possible without them.
So this year for me started in Sweden as usual and since I had nothing better to do on 1st of January I decided to visit Örebro. And there I got reminded that there is such thing as Solar system model on real landscape. So today I want to talk about those a bit.
The first time I actually spotted one was when I went to that special place in Germany known for one curious fact: once upon a time a colonel sent the King to serve in an army and he served his time there. Yes, I’m talking about Bad Nauheim.
There is a very simplified model of Solar system there—just the outer planets.
Saturn, right at the rail station
You can see the photos for all planets and in better quality in in German Wickedpedia.
But there is at least one of the problems with it: even the distances do not seem to be in proper scale let alone planet sizes. And that’s where Sweden comes to mind.
In Sweden they tried to make proper scale of Solar system. They took Globen as the reference (it was the largest hemispherical building available after all) and tried to install models of various objects around while maintaining the proper scale in both sizes and distances.
Here’s an incomplete list of object on the model. Jupiter and Saturn are not present in the model since they’d need spheres more than six metres in diameter to represent.
;
;
;There are other objects (asteroids, comets, dwarf planets) installed in various places, including even Southern Sweden. The model is large enough to accommodate various objects while keeping the scale.
And as I mentioned in the beginning, there’s own Solar system model in Örebro (or rather, Solar system in Örebro format as it’s called properly). They also try to keep scale both in size and distance. This model is located along one side of the same street (actually two or three but they merge seamlessly). The model contains only the Sun (at Olof Palmes Torg) and planets, Neptune being located around their famous water-tower Svampen (since they’ve made this model in 2009 they did not have to care about Pluto any more). Planets smaller than Earth are represented by plaques in the pavement, the rest are made of metal and put on pedestals:
There are various models of Solar system but there’s none better than Swedish one. Also it gives me a pretext to visit various Swedish places again and again.
Somebody has managed to found Dingo Pictures media conceptual work Perseus in German and some of their video stories were discovered too—Siegfried and some Christmas story with a subtitle Max’s Wonderful Present (somebody even notified me of this one earlier). All of them are available at the usual video hosting.
I still want to see Humpie the Whale Explores the World one day though.
In late September 2017 I’ve started to work on RealMedia support in NihAV with an intent to have full support for RealMedia. So more than a year later I’ve reached that goal.
In order to do that I had to reverse engineer one and a half codecs and one format. Here’s the full list.
Supported formats:
Supported audio codecs:
And video codecs:
And here are some words about IVR that I had to RE this week.
Update: it turns out Paul had reverse engineered the format before NihAV came to existence but his implementation is even sketchier than mine unfortunately.
There are actually two formats there, one with magic .REC that contains actual recording and another one with magic .R1M that may contain several of those .rec embedded. Both formats internally reminded me more of Flash than RealMedia because both files are composed of records that can be distinguished by the first byte (yes, I still remember RTMP and how I had to parse it). R1M has two kinds of records: 0x01—recording metadata it seems, 0x02 contains actual REC.
REC files (or sub-entries in R1M) have defined amount of global properties followed by stream specific properties followed by (optional) stream seek tables followed by actual packets. All numbers are big-endian and 32-bit (seek table offsets seem to be 64-bit). Strings are coded as string length (including terminating zero) followed by string data and zero terminator.
REC record types:
There may be several RJMx chunks at the end of IVR with additional metadata but they posed no interest to me.
I had some trouble with IVR packets since I expected them to be exactly the same as in RM but it turned out to be the same payload format but with different header:
Uranus, in the KurparkMLTI substreams, keyframe information and such but I could not find a proper pattern valid for all three samples (and demuxer works fine without it too);I am fully aware that my current implementation has bugs and flaws and might not decode all files perfectly but it decodes all kinds of files and that’s good for me. Also what to expect from software written by one lazy guy in his free time for himself?
Next is probably Duck type of codecs or totally RAD ones. Or maybe I’ll waste time on making NihAV conform to Rust 2018 edition. This seems to be a task about half as hard as porting code from K&R C to ANSI C (from a quick glance you have to change at least imports from inside the crate, traits now require word dyn and there may be more). Or it may be NAScale for all I care (and I don’t care at all). The time will show.
I wanted to write this post for a while but I guess AV1.1 is that cherry on top of the whole mess called AV1 that made me finally do this.
Since the time I first heard about AV1 I tried to follow its development as much as it’s possible for a person not subscribed to their main mailing list. And unfortunately while we all expected great new codec with cool ideas we got VP10 instead (yes, I still believe that AV1 is short for “A Vp 1(0) codec”). Below I try to elaborate my view and present what facts I know that formed my opinion.
It all started with ITU H.EVC and its licensing—or rather its inability to be licensed. In case you forgot the situation here’s a summary: there are at least three licensing entities that claim to have patents on HEVC that you need to license in order to be using HEVC legally. Plus the licensing terms are much greedier than what we had for H.264 to the point where some licensing pool wanted to charge fees per streaming IIRC.
So it was natural that major companies operating video in Internet wanted to stay out of this and use some common license-free codec. Resorting to creating one if the need arises.
That was a noble goal that only HEVC patent holders may object to, so the Alliance for Open Media (or AOM for short) was formed. I am not sure about the details but IIRC only organisations could join and they had to pay entrance fee (or be sponsored—IIRC VideoLAN got sponsored by Mozilla) and the development process was coordinated via members-only mailing list (since I’m not a member I cannot say what actually happened there or how and have to rely on second- or third-hand information). And that is the first thing that I did not like—the process not being open enough. I understand that they might not wanted some ideas leaked out to the competitors but even people who were present on that list claim some decisions were questionable at best.
In the beginning there were three outlooks on how it will go:
And looks like all those opinions were wrong. AV1 is not that great especially considering its complexity (we’ll talk about it later); its features were not always selected based on the merit (so most of Daala stuff was thrown out in the end—but more about it later); and looks like the main goal was to interest hardware manufacturers in its acceptance (again, more on it later).
Anyway, let’s look what main feature proposals were (again, I could not follow it so maybe there was more):
libvpx with current development snapshot of VP10;But it started with a scandal since Baidu tried to patent ANS-based video codec (essentially just an idea of video codec that uses ANS coding) after accepting ANS inventor’s help and ignoring his existence or wishes afterwards.
And of course they had to use libvpx as the base because. Just because.
So after the initial gathering of ideas it was time to put them all to test to see which ones to select and which ones to reject.
Of course since organisations are not that happy with trying something radically new, AV1 was built on the existing ideas with three main areas where new ideas were compared:
I don’t think there were attempts to change the overall codec structure. To clarify: ITU ITU H.263 used 8×8 DCT and intra prediction consisted of copying top row or left column of coefficients from the reference block, ITU H.264 used 4×4 integer transform and tried to fill block from its neighbours already reconstructed pixel data, ITU H.265 used variable size integer transform (from 4×4 to 32×32), different block scans and quadree coding of the blocks. On the other hand moving from VP9 to AV1 did not involve such significant changes.
So, for prediction there was one radically new thing: combining Thor and Daala filter into single constrained directional enhancement filter (or CDEF for short). It works great, it gives image quality boost at small cost. And another interesting tool is predicting chroma from luma (or CfL for short) that was a rejected idea for ITU H.EVC but later was tried both in Thor and Daala and found good enough (the history is mentioned in the paper describing it). This makes me think that if Cisco joined efforts with Xiph foundation they’d be able to produce a good and free video codec without any other company. Getting it accepted by others though…
Now coefficient coding. There were four approaches initially:
ANS-based coding was rejected probably because of the scandal and that it requires data to be coded in reverse direction (the official reasoning is that while it was faster on normal CPU it was slower in some hardware implementations—that is a common reason for rejecting a feature in AV1).
Daala approach won, probably because it’s easier to manipulate a multi-symbol model than try to code everything as context-dependent binarisation of the value (and you’ll need to store and/or code a lot of context probabilities that way). In any case it was clear winner.
Now, transforms. Again, I cannot tell how it went exactly but all stories I heard were that Daala transforms were better but then Baidu had to intervene citing hardware implementation reasons (something in the lines that it’s hard to implement new transforms and why do that since we have working transforms for VP9 with tried hardware design) so VP9 transforms had been chosen in the end.
In April 2018 AOM has announced long-awaited bitstream freeze which came as a surprise to the developers.
In June it was actually frozen and AV1.0 was released along with the spec. Fun fact: the repository for av1-spec on baidusource.com that once hosted it (there are even snapshots of it from June in the Web Archive) now is completely empty.
And of course because of some hardware implementation difficulties (sounds familiar yet?) now we have AV1.1 which is not fully compatible with AV1.0.
This all started as a good intent but in the process of developing AV1.x it raised so many flags that I feel suspicious about it:
And by “not very good result” I mean that the codec is monstrous in size (tables alone take more than megabyte in source form and there’s even more code than tables) and its implementation is slow as pitch drop experiment.
Usually people trying to defend it say the same two arguments: “but it’s just a reference model, look at JM or HM” and “codecs are not inherently complex, you can write a fast encoder”. Both of those are bullshit.
First, comparing libaom to the reference software of H.264 or H.265. While formally it’s also the reference software there’s one huge difference. JM/HM were the plain C/C++ implementations with no optimisation tricks (beside transform speed-up by decomposition in HM) while libaom has all kinds optimisations including SIMD for ARM, POWER and x86. And dav1d decoder with rather full set of AVX optimisations is just 2-3 times faster (more when it can use threading). For H.264 optimised decoders were tens of times faster than JM. I expect similar range for HM too but two-three times faster is very bad result for unoptimised reference (which libaom is not).
Second, claiming that codecs are not inherently complex and thus you can write fast encoder even is the codec is more complex than its predecessor. Well, it is partly true in the sense that you’re not forced to use all possible features and thus can avoid some of combinatorial explosion by not trying some coding tools. Well, there is certain expectation built in into any codec design (i.e. that you use certain coding tools in certain sequence omitting them only in certain corner cases) and there are certain expectations on compression level/quality/speed.
For example, let’s get to the basics and make H.EVC encoder encode raw video. Since you’re not doing intra prediction, motion compensation or transforms it’s probably the fastest encoder you can get. But in order to do that you still have to code coding quadtrees and transmit flags that it has PCM data. In result your encoder will beat any other on speed but it will still lose to memcpy() because it does not have to invoke arithmetic coder for mandatory flags for every coded block (which also take space along with padding to byte boundary, so it loses in compression efficiency too). That’s not counting the fact that such encoder is useless for any practical purpose.
Now let’s consider some audio codecs—some of them use parametric bit allocation in both encoder and decoder (video codecs might start to use the same technique one day, Daala has tried something like that already) so such codec needs to run it regardless on how you try to compute better allocation—you have to code it as a difference to the implicitly calculated one. And of course such codec is more complex than the one that transmits bit allocation explicitly for each sample or sample group. But it gains in compression efficiency and that’s the whole point of having more complex codec in the first place.
Hence I cannot expect of AV1 decoders magically being ten times faster than libaom and similarly while I expect that AV1 encoders will become much faster they’ll still either measure encoding speed in frames per month minute or be on par with x265 in terms on compression efficiency/speed (and x265 is also not the best possible H.265 encoder in my opinion).
Late Sir Terence Pratchett (this world is truly sadder place without his presence) used a phrase “ladies of negotiable hospitality” to describe certain profession in Discworld. And to me it looks like AV1 is a codec of negotiated design. In other words, first they tried to design the usual general purpose codec but then (probably after seeing how well it performs) they decided to bet on hardware manufacturers (who else would make AV1 encoders and more importantly decoders perform fast enough especially for mobile devices?). And that resulted in catering to all possible objections any hardware manufacturer of the alliance had (to the point of AV1.1).
This is the only way I can reasonably explain what I observe with AV1. If somebody has a different interpretation, especially based on facts I don’t know or missed, I’d like to hear it and know the truth. Meanwhile, I hope I made my position clear.
I guess the title shows how stupid marketing something can be if there’s just one contestant in the category—so in order to win you merely need to exist. Like in this case: NihAV can barely decode data and it’s not correct but the images are recognizable (some examples below) and that’s enough to justify this post title. Also I have just one sample clip to test my decoder on but at least RV6 is not a format with many features so only one feature of it is not tested (type 3 frames).
Anyway, here’s the first frame—and it is reconstructed perfectly:
The rest of frames is of significantly worse quality but with more details.
The next I-frame:
The first P-frame after it:
And one of B-frames between those two:
The sample is just short clips of similar sights with some Korean company logo at the end. At least I could recognize most of them without straining imagination much (especially after I fixed a typo in my 4×4 transform implementation).
Overall I’d say that RealVideo 6 decoder is simpler than RealVideo 4, it needs less code (now codebase for RV4 is more than twice as large as RV6 decoder codebase) and many things are done simpler too.
For example, RV3 and RV4 had context-adaptive coding for intra prediction direction, RV6 simply has fixed scheme (one bit for selecting if it’s shortlist index or explicit mode, then 1-2 bits for index or 5 bits for mode; final intra mode is derived like in H.EVC but a bit simpler). While RV3 and RV4 used weighted motion compensation for B-frames, RV6 uses just averaging. RV1-4 had arbitrary slices, RV6 codes each row of 64×64 blocks in a separate chunk of frame that can be started to decode when enough context is there (and H.EVC scheme is too flexible to my taste, I’d rather stick to one scheme instead of having to support both tiles and slices that may or may not code rows independently).
I’d argue this makes a lot of sense: if you want to offer your codec you should make sure it decodes in software with good speed (not looking at you, AV1) because hardware implementation do not follow immediately (if at all) so you’d better deliver something that works here and now. And if you target “high-definition” you have to deal with a lot of input data regardless how well you compress it. And compression gains come mostly from employing larger blocks instead of smarter coding methods (e.g. if you compare H.264 and H.265 you’ll see they use the same coefficient encoding method, just larger blocks for transform and prediction). Considering all that, employing simpler but tried scheme that can be used for parallel decoding makes sense. Of course if you rely on magic hardware to do all decoding for you then you can sacrifice simplicity for compression efficiency.
And there’s another thing from implementer’s point of view. In RealVideo 6 there’s a separation between coded bitstream and its reconstructed form, i.e. if you don’t need to reconstruct blocks correctly in order to be able to decode upcoming blocks. In H.EVC you have arithmetic coder and almost everything is coded with an adaptive context that is selected depending on the decoded values for its neighbours. From reverse engineering point of view that means if you don’t get all details right you won’t be able to decode anything properly past the point where you made a mistake. Here’s a simple example: in order to decode split flag in RealVideo 6 you simply read a bit; in ITU H.265 you need to check whether top and left neighbours are split more than current block and depending on that information (none/one/both of them is split more) select one of three contexts that will be used to decode actual split bit value (and obviously arithmetic decoder state is updated using probability from the bit context). It is not that bad at this level but decoding coded block patterns and individual coefficients might get painful. Especially if all you have is a bad binary specification.
Having said all that, I still have quite a bit to implement in order to have proper decoding:
CRefQueue<std::shared_ptr<CFrame>>::GetRefList(...) and std::vector<std::shared_ptr<CFrame>,std::allocator<std::shared_ptr<CFrame>>>::_M_emplace_back_aux<std::shared_ptr<CFrame> const&> I decided not to dig deeper);I hope it won’t take that much time though—this decoder is for completeness sake only but I don’t want to leave it in half-working state. And I should start documenting it too.
There’s still a lot to do but some unset milestone has been reached and it’s just debugging work ahead. And since the only sample I have (containing a bit more than 1750 frames with 352×288 resolution) decodes in about two seconds I don’t feel a need to make it even faster.
I wonder what to do next—ducklings of varying beauty (from TM1 to VP7) or totally RAD family of codecs (Smacker and Bink)—but I’ll decide on the order later. Back to doing nothing!
This week I’ve started reading the binary specification for RealVideo 6 again (yes, I had a post on RV6 technical description a year ago) since now I have a goal of implementing a decoder for it. So here I’ll try to document the format with as many details as possible.
Small update: obviously I’m working on a decoder so when it’s more or less ready I’ll document it on The Wiki.
RV6 seem to have four kinds of frames probably corresponding to the conventional I-/P-/B-frames. Every frame is split into horizontal stripes of 64×64 coding units (or CUs for short), each line of them being coded separately: there is a table with slice sizes right after general frame header. Probably you can decode them in the same way as Wavefront Parallel Processing in ITU H.EVC. The codec does not use arithmetic coding but rather fixed codebooks and Elias Gamma codes.
Each coding unit can be recursively split into smaller square coding blocks and then each block can be either spatially predicted from its neighbours or from one of the reference frames. And then also the difference between the prediction and actual data can be added too. It’s the same scheme employed in H.EVC, just simpler.
The codec has deblocking which can be performed after each CU is decoded and there’s an optional post-processing.
Oh, and the codec still outputs only YUV420 8-bit frames.
As said above, only static codebooks and integer codes are used. This makes decoding faster and less painful to implement. Elias Gamma codes are used for motion vector differences and static codebooks are used to code coding block patterns and actual coefficients.
Static codebooks are stored in encrypted form—XORed (or EORed) with 32-byte key that contains the name and mail address of the person responsible for them (presumably). The codes are stored as length minus one in each nibble—exactly like in RALF and probably for the same reason: there are 5 intra codebook sets and 7 inter codebook sets with 7 or 23 tables each, some tables having 864 entries. In other words, a lot of information that should be packed somehow.
Coding unit is split into 8×8 or 16×16 clusters that have own CBPs and there’s CBP for such clusters too. The actual coefficient coding is done the same way as it was in RealVideo 4—in 4×4 blocks with sets selected depending whether it’s intra or inter block, if it’s the first cluster of 4×4 blocks in coded unit and quantiser.
Also it’s worth noting that each slice starts with delta for frame quantiser, so the whole slice is coded with the same quantiser but different slices may have different quantisers.
I have not got thoroughly through this part so only some notes for now.
Coding unit types:
Prediction unit types:
Intra (aka spatial) block prediction can be done either from one of 35 angles or with plane prediction.
Motion compensation uses 1/4-pel interpolation with filter 1, -5, 20, 20, -5, 1 for luma and 1/4-pel linear interpolation for chroma.
There are 4×4, 8×8 and 16×16 transforms that seem to differ from H.EVC. Here’s 8×8 transform matrix for example:
37, 37, 37, 37, 37, 37, 37, 37,
51, 43, 29, 10, -10, -29, -43, -51,
48, 20, -20, -48, -48, -20, 20, 48,
43, -10, -51, -29, 29, 51, 10, -43,
37, -37, -37, 37, 37, -37, -37, 37,
29, -51, 10, 43, -43, -10, 51, -29,
20, -48, 48, -20, -20, 48, -48, 20,
10, -29, 43, -51, 51, -43, 29, -10
Quantisation still uses two quantisers: for DC and AC.
Complete frame reconstruction is obvious: decode CU from the bitstream, if it’s intra then derive intra prediction mode and apply it, otherwise derive motion vector and apply motion compensation. After that dequantise, transform and add decoded coefficients. Do deblocking if requested. Repeat for the rest of coded units. Output the whole frame.
Overall, RealVideo 6 (or 11; or HD) seems to be an evolution of RealVideo 4 with coding approaches ripped off from ITU H.EVC but with a clear goal and trying to keep codec simple enough (i.e. more like Thor and less like AV1). And while knowing the standard won’t hurt, knowing RealVideo 4 would be more useful for REing this codec—at least it was to me.
Having said that, I’m probably not going to have much to say about it until the decoder is ready (hopefully there will be more samples by then too) so this should be it for this year unless I’ll finally write my thoughts on AV1 and why it’s not a thing to be overexcited about.
I am working on PowerPC SIMD optimizations for x264. I was playing with SAD functions and was thinking it would be nice to have something similar to x86 PSADBW for computing the sum of absolute differences. Luca suggested me to try using the #power9 vec_absd. Single vec_absd( fenc, pix0v ) replaces vec_sub( vec_max( fencv, pix0v ), vec_min( fencv, pix0v) ). My patch can be found here. To make it work -mcpu=power9 must be set. The patch contains the macro that makes its code backward compatible with POWER8:#ifndef __POWER9_VECTOR__
#define vec_absd(a, b) vec_sub(vec_max(a, b), vec_min(a, b))
#endif
vec_absd (the numbers are ratios of AltiVec/C checkasm timings):
vec_absd for SAD and SSD functions makes 8% improvement. Which is amazing for such a small change.I decided to write SIMD optimizations for HEVC decoder inverse transform (which is IDCT approximation) for ARMv7. (Here is an interesting post about DCT.) The inverse transform for HEVC operates on 4x4, 8x8, 16x16 and 32x32 blocks and I have finished them recently. For each block there are 2 functions, one for 8 bitdepth and the other for 10 bitdepth:
vpush/vpop {q4-q7} ) when one wants to use them. VPUSH/VPOP pushes and pops to/from stack. pop lr and then bx lr to return but it's better to return with simply pop {pc} . mov rx, sp and rx, sp, #15 add rx, rx, #buffer_size sub sp, sp, rx sp now points to the buffer. After using the buffer, the stack pointer has to be restored with add sp, sp, rx . I tried my skills at optimising HEVC. My SIMD IDCT (Inverse Discrete Cosine Transform) for HEVC decoder was merged lately. What I did was 4x4, 8x8, 16x16 and 32x32 IDCTs for 8 and 10 bitdepths. Both 4x4 and 8x8 are supported on 32-bit CPUs but 16x16 and 32x32 are 64-bit only.
The larger transforms calls the smaller ones, 32 calls 16, 16 calles 8 and so on, so 4x4 is used by all the other transforms. Here is how the actual assembly looks:
; void ff_hevc_idct_4x4__{8,10}_(int16_t *coeffs, int col_limit) *coeffs is a pointer to coefficients I want to transform. They are loaded to XMM registers and then
; %1 = bitdepth
%macro IDCT_4x4 1
cglobal hevc_idct_4x4_%1, 1, 1, 5, coeffs
mova m0, [coeffsq]
mova m1, [coeffsq + 16]
TR_4x4 7, 1, 1
TR_4x4 20 - %1, 1, 1
mova [coeffsq], m0
mova [coeffsq + 16], m1
RET
%endmacroTR_4x4 macro is called. This macro transforms the coeffs according to the following equations: res00 = 64 * src00 + 64 * src20 + 83 * src10 + 36 * src30
res10 = 64 * src01 + 64 * src21 + 83 * src11 + 36 * src31
res20 = 64 * src02 + 64 * src23 + 83 * src12 + 36 * src32
res30 = 64 * src03 + 64 * src23 + 83 * src13 + 36 * src33
Because the transformed coefficients are written back to the same place, "res" (as residual) is used for the results and "src" for the initial coefficients. The results from the calculations are then scaled res = (res + add_const) >> shift and the (4x4) block of the results is transposed. The macro is called again to perform the same transform but this time to rows.
; %1 - shift
; %2 - 1/0 - SCALE and Transpose or not
; %3 - 1/0 add constant or not
%macro TR_4x4 3
; interleaves src0 with src2 to m0
; and src1 with scr3 to m2
; src0: 00 01 02 03 m0: 00 20 01 21 02 22 03 23
; src1: 10 11 12 13 -->
; src2: 20 21 22 23 m1: 10 30 11 31 12 32 13 33
; src3: 30 31 32 33
SBUTTERFLY wd, 0, 1, 2
pmaddwd m2, m0, [pw_64] ; e0
pmaddwd m3, m1, [pw_83_36] ; o0
pmaddwd m0, [pw_64_m64] ; e1
pmaddwd m1, [pw_36_m83] ; o1
%if %3 == 1
%assign %%add 1 << (%1 - 1)
mova m4, [pd_ %+ %%add]
paddd m2, m4
paddd m0, m4
%endif
SUMSUB_BADC d, 3, 2, 1, 0, 4
%if %2 == 1
psrad m3, %1 ; e0 + o0
t psrad m1, %1 ; e1 + o1
psrad m2, %1 ; e0 - o0
psrad m0, %1 ; e1 - o1
;clip16
packssdw m3, m1
packssdw m0, m2
; Transpose
SBUTTERFLY wd, 3, 0, 1
SBUTTERFLY wd, 3, 0, 1
SWAP 3, 1, 0
%else
SWAP 3, 2, 0
%endif
%endmacro
The larger transforms are a bit more complicated but they works in a similar way.
There are the results benchmarked by checkasm bench_new() function for the bitdepth 8 (the results are similar for bitdepth 10). Checkasm can benchmark SIMD functions with --bench option, in my case the full command was:
I asked Kostya Shishkov, an experienced ARM developer, to check my basic NEON knowledge. So here are his questions and my answers to them:
vmov.i32 q0, #42 - move immediate constant 42 to q0 SIMD register, suffix i32 specifies the data type, 32-bit integer in this case, as q0 is 128-bit register, there will be 4 32-bit 42 constants mov r1, #16 - move number 16 to r1 GPR register add r0, r1 - add 16 bytes to the address stored in r0 vst1.s16 {q0}, [r0] - store the content of q0 to the address stored in r1r0 and store the result in r1mov r2, \step - move the constant step to r2 vld1.s16 {d0}, [r1], r2 - store d0 content to the address in r1, then update r1 = r1 + r2 vmov d0[0], r1I decided to organise the Libav sprint again, this time in a small village near Pelhřimov. The participants:
I rented a cosy cottage for the sprint. It was surprisingly warm for the end of September and we enjoyed a nice garden sitting not only with the table and chairs but even with a couch. The sitting was under the roof and it was possible to work outside which was really pleasant for me. There was also a grill so we grilled some sausages for one of our dinners. It started to rain the last day and making a fire in the fireplace made a nice feeling.
Some time ago Niels Möller proposed a new method of bitreading that should be faster then the current one (here). It is an interesting idea and I decided to try it. Luca Barbato considered it to be a good idea and had his company sponsored this work. The new bitstream reader (bitstream.h) is faster in many cases and is never slower than the existing one (get_bits.h).
static inline unsigned int get_bits(GetBitContext *s, int n){ register int tmp; OPEN_READER(re, s);
UPDATE_CACHE(re, s); tmp = SHOW_UBITS(re, s, n); LAST_SKIP_BITS(re, s, n); CLOSE_READER(re, s); return tmp;} The new bitstream reader is written to be easier to use, more consistent and to be easier to follow. It is better documented, the functions are named according to the current naming convetions and to be consistent with the bytestream reader naming. bitstream_read_32() reads bits from the 0-32 range and replaces get_bits() get_bits_long() get_bitsz() bitstream_peek_32() replaces show_bits() show_bits_long() show_bits1() bitstream_skip() replaces skip_bits1() skip_bits() skip_bits_long() Sometimes it's very useful to print out how some parameters changes during the program execution.
When writing the new version of some piece of code one usually needs to compare it with the old one to be sure it behaves the same in every case. Especially the corner cases might be tricky and I spent a lot of time with them while my code worked fine in general.
For example when I was working on my ASF demuxer, I was happy there's an old demuxer and I can compare their behaviour. When debugging the ASF, I wanted to know the state of I/O context. In that time lu_zero (who was mentoring me) created a set of macros which printed logs for every I/O function (here). For example there's the macro for avio_seek() function (which is equivalent to fseek()).
#define avio_seek(s, o, w) ({ \
int64_t _ret = avio_seek(s, o, w); \
int64_t _pos = avio_tell(s); \
av_log(NULL, AV_LOG_VERBOSE|AV_LOG_C(154), "0x%08"PRIx64" - %s:%d seek %p %"PRId64" %d -> %"PRId64"\n", \
_pos, __FUNCTION__, __LINE__, s, o, w, _ret); \
_ret; \
}) When such a macro was present in my demuxer, for all the calls of avio_seek the following information was printed _pos = avio_tell(s): the offset in the demuxed file __FUNCTION__ : preprocessor define that contains the name of the function being compiled to know which function called avio_seek __LINE__ : preprocessor define that contains the line number of the original source file that is being compiled to know from what line avio_seek was called from s, o, w : the values of the parameters avio_seek was called with _ret: the avio_seek return value __FILE__: preprocessor define contains the name of the file being compiled (this one was not used in the example but might be useful when one needs more complex log). _ret; as the last statement in this macro because its value serves as the value of the entire construct. If the last _ret; would be omitted in my example, the return value of this macro expression would be printf return value. The underscores in _ret or _pos variables are used to be sure it does not shadow some other variables with the same names.lu_zero for teaching me about it. The support from the more experienced developers is the thing I really love about Libav.
I made a split my complex dcadec bit-exact patch (https://patches.libav.org/patch/59013/) to the several parts. The first part which contains changing the dcadec core to work with integer coefficients instead of converting the coefficients to floats just after reading them was sent to the mailing list (https://patches.libav.org/patch/59141/). Such a change was expected to slow down the decoding process. Therefore I made some measurements to examine how much slower decoding is after my patch.
I decoded this sample: samples.libav.org/A-codecs/DTS/dts/dtswavsample16.wav 10 times and measured the user time between invocation and termination with the "time" command:
time ./avconv -f dts -i dtswavsample16.wav -f null -c pcm_f32le null,counted the average real time of avconv run and repeated everything for the master branch. The duration of the dtswavsample16.wav is ~4 mins and I wanted to look at the slow down for the longer files. Hence I used relatively new loop option for the avconv (http://sasshkas.blogspot.cz/2015/08/the-loop-option-for-avconv.html) to create ~24 mins long file from the initial file by looping it 6x with
./avconv -loop 6 -i dtswavsample16.wav -c copy dts_long.wav.I decoded this longer dts file 10x again for both new integer and old float coefficients core and counted the averages.
When playing a multimedia file, one usually wants to seek to reach different parts of a file. Mostly, containers allows this feature but it might be problem for streamed files.
Withing the libavformat, seeking is performed with function (inside the demuxer) called read_seek. This function tries to find matching timestamp for the requested position (offset) in the played file.
There are 2 ways to seek through the file. One of them is when file contains some kind of index, which matches positions with appropriate timestamps. In this case index entries are created by calling av_add_index_entry. If index entries are present, av_index_search_timestamp, which is called inside the read_seek, looks for the closest timestamp for the requested position. When the file does not provide such entries, one can look for the requested position with ff_seek_frame_binary. For doing so, read_timestamp function has to be created inside the demuxer.
Read_timestamp takes required position and stream index and then tries to find offset of the beginning of the closest packet wich is key frame with matching stream index. While doing this, read_timestamp reads timestamps for all the packets after given position and creates index entries. When the key frame with matching stream index is found, read_timestamp upgrades required position and returns timestamp matching to it.
I was told to test my ASF demuxer with the zzuf utility. Zuff is a fuzzer, it changes random bits in the program's input which simulates damaged file or unexpected data.
For testing ASF's behaviour I want to feed avconv with some corrupted wmv files and see what will happen. Because I want to fuzz in several different ways I want to vary seed (the initial value of zzuf’s random number generator). I'll do this with command:
while true; SEED=$RANDOM; for file *wmv; do zzuf -M -l -r 0.00001 -q -U 60 -s $SEED ./avconv -i "file" -f null -c copy - || echo $SEED $file >> fuzz; done; done;.
I got the file fuzz which is the list of seed paired with filename. Now I need to use zzuf for creating damaged files to check the problem with valgrind. I'll use the list to determine the seed which caused some crash for creating my damaged file:
zzuf -M -l -r 0.00001 -q -U 60 -s myseed < somefile.wmv | cat out.asf.
Now I'll just use valgrind to find out what happened:
valgrind ./avconv -i out.asf -f null -c copy -.
I tried to test the ASF demuxer with different tricky samples and with FATE
and the demuxer behaved well but testing with zzuf detected several new crashes. Mainly it was insane values sizes and it was easy to fix them by adding some more checks. Zzuf is a great thing for testing.
Pelhřimov is small but very nice town in Czech Republic approximately 120 km from the capital Prague and I decided to organize a small but nice Libav sprint in it.
The participants and the topics were:
dca_decode_frame to handle all the extensions and working with the new options for them more systematicly. I decided to improve the Libav DTS decoder - dcadec. Here I want to explain what are its problems now and what I would like to do about them.
DTS encoded audio stream consists of core audio and may contain extended audio. Dcadec supports XCH and XLL extensions but X96, XXCH and XBR extensions are waiting to be implemented - I'd like to implement them later.
For the DTS lossless extension - XLL, the decoded output audio should be a bit for bit accurate reproduction of the encoded input. However there are some problems:
dequantization (with int -> float conversion)I'm working now on modifying the core to work with integer coefficients and then convert them to floats before QMF filtering for lossy output but use bitexact QMF (intermediate LFE coefficients should be always integers and I think it's not correct in the current version) for lossless output. Also I added an option called
↓
inverse ADPCM (when needed)
↓
VQ decoding (when needed)
↓
filtering: QMF, LFE, downmixing (when needed)
↓
float output.
-force_fixed to force fixed-point reconstruction for any kind of input.XLL extension is not detected sometimes and the core audio only is decoded in this case. I want to fix this issue as well.I'd like to add the loop option to avconv. This option allows to repeat an input file given number of times, so the output contains specified number of inputs. The command is ./avconv -loop n -i infile outfile, n specifies how many times the input file should be looped in the output.
How does this work?
After processing the input file for the first time, avconv calls new seek_to_start function to seek back to the beginning of the file. av_seek_frame is called to perform seeking itself but there are other things needed for loop option to work.
1) flush
Flush decoder buffers to take out delayed frames. In avconv this is done by calling process_input_file with NULL as frame, process_input_packet had to be modified a little to not to signal EOF on the filters when seeking.
2) timestamps (ts)
To have correct timestamps in the "after seeking" part of the output stream they have to be corrected with ts = ts_{from the demuxer} + n * (duration of the input stream), n is number of times the input stream was processed so far . This duration is the duration of the longest stream in a file because all the streams have to be processed (or played) before starting the next loop. The duration of the stream is the last timestamp - the first timestamp + duration of the last frame. For the audio streams one "frame" is usually a constant number of samples and its duration is number of samples/sample rate. Video frames on the other side are displayed unevenly so their average framerate can be used for the last frame duration if available or if the average framerate is not known the last frame duration is just 1 (in the current time base).
https://github.com/sasshka/libav/commit/90f2071420b6fd50eea34982475819248e5f6c8f
I am hearing a lot of persons interested in open-source and giving back to the community. I think it can be an exciting experience and it can be positive in many different ways: first of all more contributors mean better open-source software being produced and that is great, but it also means that the persons involved can improve their skills and they can learn more about how successful projects get created.
So I wondered why many developers do not do the first step: what is stopping them to send the first patch or the first pull-request? I think that often they do not know where to start or they think that contributing to the big projects out there is intimidating, something to be left to an alien form of life, some breed of extra-good programmers totally separated by the common fellows writing code in the world we experience daily.
I think that hearing the stories of a few developers that have given major contributions to top level project could help to go over these misconceptions. So I asked a few questions to this dear friend of mine, Luca Barbato, who contributed among the others to Gentoo and VLC.
Let’s start from the beginning: when did you start programming?
I started dabbling stuff during high school, but I started doing something more consistent at the time I started university.
What was your first contribution to an open-source project?
I think either patching the ati-drivers to work with the 2.6 series or hacking cloop (a early kernel module for compressed loops) to use lzo instead of gzip.
What are the main projects you have been involved into?
Gentoo, MPlayer, Libav, VLC, cairo/pixman
How did you started being involved in Gentoo? Can you explain the roles you have covered?
Daniel Robbins invited me to join, I thought “why not?
During the early times I took care of PowerPC and [Altivec](http://en.wikipedia.org/wiki/AltiVec), then I focused on the toolchain due the fact it gcc and binutils tended to break software in funny ways, then multimedia since altivec was mainly used there. I had been part of the Council a few times used to be a recruiter (if you want to join Gentoo feel free to contact me anyway, we love to have more people involved) and I’m involved with community relationship lately.
Note: Daniel Robbins is the creator of Gentoo, a Linux distribution.
Are there other less famous projects you have contributed to?
I have minor contributions in quite a bit of software due. The fact is that in Gentoo we try our best to upstream our changes and I like to get back fixes to what I like to use.
What are your motivations to contribute to open-source?
Mainly because I can =)
Who helped you to start contributing? From who you have learnt the most?
Daniel Robbins surely had been one of the first asking me directly to help.
You learn from everybody so I can’t name a single person among all the great people I met.
How did you get to know Daniel Robbins? How did he helped you?
I was a gentoo user, I happened to do stuff he deemed interesting and asked me to join.
He involved me in quite a number of interesting projects, some worked (e.g. Gentoo PowerPC), some (e.g. Gentoo Games) not so much.
Do your contributions to open-source help your professional life?
In some way it does, contrary to the assumption I’m just seldom paid to improve the projects I care about the most, but at the same time having them working helps me when I need them during the professional work.
How do you face disagreement on technical solutions?
I’m a fan of informed consensus, otherwise prototypes (as in “do, test and then tell me back”) work the best.
To contribute to OSS are more important the technical skills or the diplomatic/relation skills?
Both are needed at different time, opensource is not just software, you MUST get along with people.
Have you found different way to organize projects? What works best in your opinion? What works worst?
Usually the main problem is dealing with poisonous people, doesn’t matter if it is a 10-people project or a 300+-people project. You can have a dictator, you can have a council, you can have global consensus, poisonous people are what makes your community suffer a lot. Bonus point if the poisonous people get clueless fan giving him additional voices.
Did you ever sent a patch for the Linux kernel?
Not really, I’m not fond of that coding style so usually other people correct the small bugs I stumble upon before I decide to polish my fix so it is acceptable =)
Do you have any suggestions for people looking to get started contributing to open-source?
Pick something you use, scratch your own itch first, do not assume other people are infallible or heroes.
ME: I certainly agree with that, it is one of the best advices. However if you cannot find anything suitable at the end of this post I wrote a short list of projects that could use some help.
Can you tell us about your best and your worst moments with contribution to OSS?
The best moment is recurring and it is when some user thanks you since you improved his or her life.
The worst moment for me is when some rabid fan claims I’m evil because I’m contributing to Libav and even praises FFmpeg for something originally written in Libav in the same statement, happened more than once.
What are you working on right now and what plans do you have for the future?
Libav, plaid, bmdtools, commonmark. In the future I might play a little more with [rust](http://www.rust-lang.org/).
Thanks Luca! I would be extremely happy if this short post could give to someone the last push they need to contribute to an existing open-source project or start their own: I think we could all use more, better, open-source software. So let’s write it.
One thing I admire in Luca is that he is always curious and ready to jump on the next challenge. I think this is the perfect attitude to become an OSS contributor: just start play around with the things you like and talk to people, you could find more possibilities to contribute that you could imagine.
…and one final thing: Luca is also the author of open-source recipes: he created the recipes of two types of chocolate bars dedicated to Libav and VLC. You can find them on the borgodoro website.
I suggest to take a look at his blog.
Well, just in case you are eager to start writing some code and you are looking for some projects to contribute to here there are a few, written with different technologies. If you want to start contributing to any of those and you need directions just drop me a line (federico at tomassetti dot me) and I would be glad to help!
If you are interested in contributing to Libav, you can take a look at this post: there I explained how I submitted my first patch (approved in the meantime!). It is written in C.
You could be also interested in plaid: it is a Python web application to manage git patches sent by e-mail (there are a few projects using this model like libav or the linux kernel)
WorldEngine, it is a world generator written in Python
Plate-tectonics, it is a library for plate tectonics simulation. It is written in C++
JavaParser a Java parser, written in Java
Incremental Java parser, an incremental Java parser, written in Scala
The post How people get started contributing to open-source? A few questions to Luca Barbato, contributor to Gentoo, MPlayer, Libav, VLC, cairo/pixman appeared first on Federico Tomassetti - Consultant Software Engineer.
I happened to have a few hours free and I was looking for some coding to do. I thought about VLC, the media player which I have enjoyed so much using over the years and I decided that I wanted to contribute in some way.
To start helping in such a complex process there are a few steps involved. Here I describe how I got my first patched accepted. In particular I wrote a patch for libav, the library behind VLC.
I started by reading the wiki. It is a very helpful starting point but the process to setup the environment and send a first patch was not yet 100% clear to me so I got in touch with some of the developers of libav to understand how they work and how I could start lending an hand with something simple. They explained me that the easier way to start is by solving issues reported by static analysis tools and style checkers. They use uncrustify to verify that the code is adhering to their style guidelines and they run coverity to check for potential issues like memory leaks or null deferences. So I:
After a few minutes the patch was approved by a committer, ready to be merged. The day after it made its way to the master branch. Yeah!
First of all, let’s clone the git repository:
git clone git://git.libav.org/libav.git
Alternatively you could use the GitHub mirror, if you want to.
At this point you may want to install all the dependencies. The instructions are platform specific, you can find them here. If you have Mac Os-X be sure to have installed yasm, because nasm does not work. If you have installed both configure will pick up yasm (correctly). Just be sure to run configure after installing yasm.
If everything goes well you can now build libav by running:
./configure make
Note that it is fine to build in-tree (no need to build in a separate directory).
Now it is time to run the tests. You will have to specify one directory where to download some samples, later used by tests. Let’s assume you wanted to put your samples under ~/libav-samples:
mkdir ~/libav-samples # This download the samples make fate-rsync SAMPLES=~/libav-samples # This run the tests make fate
Did everything run fine? Good! Let’s start to patch then!
First of all we need to find an open issue. Visit Coverity page for libav at https://scan.coverity.com/projects/106. You will have to ask for access and wait that someone grants it to you. When you will be able to login you will encounter a screen like this:
Here, this seems an easy one! The variable oggstream has been allocated by av_mallocz (basically a wrapper for malloc) but the result values has not been checked. If the allocation fails a NULL pointer is returned and when we will try to access it at the next line things are going end up unpleasantly. What we need to do is to check the return value of av_mallocz and if it is NULL we should return an error. The appropriate error to return in this case is AVERROR(ENOMEM). To get this information… you have to start reading code, getting familiar with the way of doing business of this codebase.
Libav follows strict rules about the comments in git commits: use git log to look at previous commits and try to use the same style.
I think many of you are familiar with GitHub and the whole process of submitting a patch for revision. GitHub is great because it made that process so easy. However there are some projects (notably including the Linux kernel) which adopts another approach: they receive patches by e-mail.
Git has a functionality that permits to submit a patch by e-mail with a simple command. The patch will be sent to the mailing list, discussed, and if approved the e-mail will be downloaded, processed through git and committed in the official repository. Does it sound cumbersome? Well, it sounds to me, spoiled as I am by GitHub and similar tools but, you know, if you go in Rome you should behave as the Romans do, so…
# This install the git extension for sending patches through e-mail sudo apt install git-email # This submit a patch built using the last commit git send-email -1 --to libav-devel@libav.org
Now, many of you are using gmail and many of you have enable 2-factor authentication (right? If not, you should). If this is you case you will get an error along this lines:
Password for 'smtp://f.tomassetti@gmail.com@smtp.gmail.com:587': 5.7.9 Application-specific password required. Learn more at 5.7.9 http://support.google.com/accounts/bin/answer.py?answer=185833 cj12sm14743233wjb.35 - gsmtp
Here you can find how to create a password for this goal: https://support.google.com/accounts/answer/185833 The name of the application that I had to create was smtp://f.tomassetti@gmail.com@smtp.gmail.com:587. Note that I used the same name specified in the previous error message.
If things go well an e-mail with your patch will be sent to the mailing-list, someone will look at it and accept it. Most of the times you will receive suggestions about possible adjustments to be done to improve your password. When it happens you want to submit a new version of your patch in the same thread which contains your first version of the patch and the e-mails commenting it.
To do that you want to update your patch (typically using git commit –amend) and then run something like:
git send-email -1 --to libav-devel@libav.org --in-reply-to Message-ID: 54E0F459.3090707@gentoo.org
Of course you need to find out the message-id of the e-mail to which you want to reply. To do that in gmail select the “Show original” item from the contextual menu for the message and in the screen opened look for the Message-Id header.
There are also web applications which are used to manage the patches sent by e-mail. Libav is currently using Patchwork for managing patches. You can see it deployed at: https://patches.libav.org/project/libav-devel/list/. Currently another tool has been developed to replace patchwork. It is named Plaid and I tried to help a little bit with that also 
Mine has been a very small contribution, and in the future I hope to be able to do more. But being a maintainer of other open-source projects I learned that also small help is useful and appreciated, so for today I feel good.
Please, if I am missing something help me correct this post
The post How to contribute to Libav (VLC): just got my first patch approved appeared first on Federico Tomassetti - Consultant Software Engineer.
I've participated to last Libav sprint in Torino. I made new ASF demuxer for Libav, but during the testing problems with rtsp a mms protocols has appeared. Therefore, my main task during the sprint was to fix these issues.
It was second time I was at such sprint and also my second Torino visit and the sprint was even better than I expected. It's really nice to see people I'm communicating throught the irc channel in person, the thing I like about Libav a lot is its friendly community. But the most important thing for me as the most unexperienced person among skilled developers was naturally their help. My mentors from OPW participated the sprint and as a result all the issues was fixed and patch was sent to the ML (https://patches.libav.org/patch/55682/). Also, these personal consultations can be very productive in learning new things and because I'm not native English speaker I realized few days I have to speak or even think in English are really helpful for getting better in it.
The last day of the sprint we had a trip to a really magical place called Sacra di San Michele (http://www.sacradisanmichele.com/).
After my challenge with the fused multiply-add instructions I managed to find some time to write a new test utility. It’s written ad hoc for unpaper but it can probably be used for other things too. It’s trivial and stupid but it got the job done.
What it does is simple: it loads both a golden and a result image files, compares the size and format, and then goes through all the bytes to identify how many differences are there between them. If less than 0.1% of the image surface changed, it consider the test a pass.
It’s not a particularly nice system, especially as it requires me to bundle some 180MB of golden files (they compress to just about 10 MB so it’s not a big deal), but it’s a strict improvement compared to what I had before, which is good.
This change actually allowed me to explore one change that I abandoned before because it resulted in non-pixel-perfect results. In particular, unpaper now uses single-precision floating points all over, rather than doubles. This is because the slight imperfection caused by this change are not relevant enough to warrant the ever-so-slight loss in performance due to the bigger variables.
But even up to here, there is very little gain in performance. Sure some calculation can be faster this way, but we’re still using the same set of AVX/FMA instructions. This is unfortunate, unless you start rewriting the algorithms used for searching for edges or rotations, there is no gain to be made by changing the size of the code. When I converted unpaper to use libavcodec, I decided to make the code simple and as stupid as I could make it, as that meant I could have a baseline to improve from, but I’m not sure what the best way to improve it is, now.
I still have a branch that uses OpenMP for the processing, but since most of the filters applied are dependent on each other it does not work very well. Per-row processing gets slightly better results but they are really minimal as well. I think the most interesting parallel processing low-hanging fruit would be to execute processing in parallel on the two pages after splitting them from a single sheet of paper. Unfortunately, the loops used to do that processing right now are so complicated that I’m not looking forward to touch them for a long while.
I tried some basic profile-guided optimization execution, just to figure out what needs to be improved, and compared with codiff a proper release and a PGO version trained after the tests. Unfortunately the results are a bit vague and it means I’ll probably have to profile it properly if I want to get data out of it. If you’re curious here is the output when using rbelf-size -D on the unpaper binary when built normally, with profile-guided optimisation, with link-time optimisation, and with both profile-guided and link-time optimisation:
% rbelf-size -D ../release/unpaper ../release-pgo/unpaper ../release-lto/unpaper ../release-lto-pgo/unpaper
exec data rodata relro bss overhead allocated filename
34951 1396 22284 0 11072 3196 72899 ../release/unpaper
+5648 +312 -192 +0 +160 -6 +5922 ../release-pgo/unpaper
-272 +0 -1364 +0 +144 -55 -1547 ../release-lto/unpaper
+7424 +448 -1596 +0 +304 -61 +6519 ../release-lto-pgo/unpaper
It’s unfortunate that GCC does not give you any diagnostic on what it’s trying to do achieve when doing LTO, it would be interesting to see if you could steer the compiler to produce better code without it as well.
Anyway, enough with the microptimisations for now. If you want to make unpaper faster, feel free to send me pull requests for it, I’ll be glad to take a look at them!
RealAudio files have several possible interleavers. The simplest is “Int0”, which means that the packets are in order. Today, I was contrasting “Int4” and “genr”. They both require rearranging data, in highly similar but not identical ways. “genr” is slightly more complex than “Int4”.
A typical Int4 pattern, writing to subpacket 0, 1, 2, 3, etc, would read data from subpacket 0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66, 1, 7, 13, etc, in that order – assuming subpkt_h is 12, as it was in one sample file. It is effectively subpacket_h rows of subpacket_h / 2 columns, counting up by subpacket_h / 2 and wrapping every two rows.
A typical genr pattern is a little trickier. For subpacket_h = 14, and the same 6 columns per row as above, the pattern to read from looks like 0, 12, 24, 36, 48, 60, 72, 6, 18, 30, 42, 54, 66, 78, 1, etc.
I spent most of today implementing genr, carefully working with a paper notebook, pencil, Python, and a terse formula from the old implementation:
case DEINT_ID_GENR:
for (x = 0; x < w/sps; x++) avio_read(pb, ast->pkt.data+sps*(h*x+((h+1)/2)*(y&1)+(y>>1)), sps);
After various debug printfs, a lot of quality time in GDB running commands like x /94x (pkt->data + 14 * 94), a few interestingly garbled bits of audio playback, and a mentor pointing out I have some improvements to make on header parsing, I can play (some) genr files.
I have also recently implemented SIPR support, and it works in both RA and RM files. RV10 video also largely works.
I've solved lost packets problems and finally my ASF demuxer started to work right at "ideal samples in vacuum". So the time for fixing memory leaks had come and valgrind helped me a lot with this issue. After memory leaks was solved I had to start testing my demuxer on various samples of ASF format multimedia files. As expected, I've found many samples my demuxer failed for. The reasons was different - mostly it was my mistakes, misunderstood or overlooked parts of specs, but I think I found a case that needed unusual handling specs didn't mention about.
some of problems was caused for example by
* improper subpayloads handling - one should be really careful while reading specs to avoid problems for less common cases like one subpayload is inside single payload and there's is padding inside payload itself (while padding after payload is 0), but there was other problems too
* I had to revise padding handling for all possible cases
* ASF file has 3 places where ASF packet size is told - twice in the header objects and once in a packet itself, and specs are not specifying what should one do when they differs or at least I didn't found it
* some stupid mistakes like when I just forgot to do something after adding new block to my code was really annoying
Funny thing was when I fixed my demuxer for one group of samples and another one that worked before started to fail, I fixed this new group and third group failed. I was so much annoyed by this, but many mistakes I did was caused by my inexperience and I think one (at least me) just have to do all of these mistakes to get better.
Finally, all basic parts of ASF demuxer seems to work somehow.
At last two weeks I fixed various bugs in my code and I hope packets handling is correct now. Only problem is that few packets at the end of the Data Object are still lost. Because I wanted a small break from this problem, my mentors allowed me to implement basic seeking first. ASF demuxer can now read index entries from Simple Index Object and adds them with av_add_index_entry to AVStream. So when Simple Index Object is present in an ASF file, my demuxer can seek to the requested time.
Skeleton of the new ASF demuxer was written, but only audio was demuxed properly now. Problem is complicated video frames handling in ASF format. I hope I finally found out how to process packets properly. ASF packet can contain single payload, single payload with subpayloads, multiple payloads or multiple payloads with subpayloads inside some of them. Every subpayload is always one frame, but single payload can be whole frame or just part of it. When ASF packet contains multiple payloads inside it, each of them can be one frame but it can be just fragment of it as well. When one of mulptiple payloads contains subpayloads, each of subpayload is one frame and it can be processed as AVPacket.
For the case of fragmented frame in ASF packet I have to store several unfinished frames in ASFPacket structures that I've created for this purpose. There should not be more than one unfinished frames per stream, so I have one ASFPacket in each ASFStream (ASFStream is structure for storing ASF stream properties). ASFPacket contains pointer to AVBufferRef where unfinished frame is stored. When frame is finished I can forward pointer to buffer with data to AVPacket, set its properties like size, timestamps and others and finally return AVPacket.
I introduced many bugs to my code that was working (at least ASF packets was parsed right and audio worked) and now I'm working on fixing all of them.
I was accepted for OPW, May - August 2014 round with project "Rewrite the ASF demuxer". First task from my mentors was to create wiki page about ASF (Advanced Streaming Format), it was created at https://wiki.libav.org/ASF.
Interesting notes about other containers: http://codecs.multimedia.cx/?p=676.
Next task from my mentors was to write simple program which reads asf file and prints its structure, i.e. list of asf objects, metadata and codec information. ASF file consists of so called ASF Objects. There're 3 top-level objects - Header Object, Data Object and Index Object. Especially Header Object can contain many other objects to provide different asf features, for example Codec List Object for codec information or Metadata object for metadata. One can recognise object with GUID, which is 16 byte array (each byte is number) that identifies object type. I was confused about the fact the GUID number you read from the file is not matching the GUID from specs. For some historical reasons one have to modify GUIDs from specs (reorder the numbers) for match GUID read from the file.
My program is working now and can list objects, codecs and metadata info, but it ignores Index Objects by then. I hope I'll add support for them soon. Also I want to print offsets for each object and read Data Object deeper.
Today, I learned how to use framecrc as a debug tool. Many Libav tests use framecrc to compare expected and actual decoding. While rewriting existing code, the output from the old and new versions of the code on the same sample can be checked; this makes a lot of mistakes clear quickly, including ones that can be quite difficult to debug otherwise.
Checking framecrcs interactively is straightforward: ./avconv -i somefile -c:a copy -f framecrc -. The -c:a copy specifies that the original, rather than decoded, packet should be used. The - at the end makes the output go to stdout, rather than a named file.
The output has several columns, for the stream index, dts, pts, duration, packet size, and crc:
0, 0, 0, 192, 2304, 0xbf0a6b45
0, 192, 192, 192, 2304, 0xdd016b78
0, 384, 384, 192, 2304, 0x18da71d6
0, 576, 576, 192, 2304, 0xcf5a6a07
0, 768, 768, 192, 2304, 0x3a84620a
It is also unusually simple to find out what the fields are, as libavformat/framecrcenc.c spells it out quite clearly:
static int framecrc_write_packet(struct AVFormatContext *s, AVPacket *pkt)
{
uint32_t crc = av_adler32_update(0, pkt->data, pkt->size);
char buf[256];
snprintf(buf, sizeof(buf), “%d, %10″PRId64″, %10″PRId64″, %8d, %8d, 0x%08″PRIx32″\n”,
pkt->stream_index, pkt->dts, pkt->pts, pkt->duration, pkt->size, crc);
avio_write(s->pb, buf, strlen(buf));
return 0;
}
Keiler, one of my Libav mentors, patiently explained the above; I hope documenting it helps other people who are starting with Libav development.
Most recently, I have been adding documentation to Libav. Today, my work included writing a demuxer howto. In the last couple of weeks, I have also reimplemented RealAudio 1.0 support (2.0 is in progress), and learned more about Coccinelle and undefined behavior in C. Blog posts on these topics are pending.
My first patch for undefined behavior eliminates left shifts of negative numbers, replacing a << b (where a can be negative) with a * (1 << b). This change fixes bug686, at least for fate-idct8x8 and libavcodec/dct-test -i (compiled with ubsan and fno-sanitize-recover). Due to Libav policy, the next step is to benchmark the change. I was also asked to write a simple benchmarking HowTo for the Libav wiki.
First, I installed perf: sudo aptitude install linux-tools-generic
I made two build directories, and built the code with defined behavior in one, and the code with undefined behavior in the other (with ../configure && make -j8 && make fate). Then, in each directory, I ran:
perf stat --repeat 150 ./libavcodec/dct-test -i > /dev/null
The results were somewhat more stable than with –repeat 30, but it still looks much more like noise than a meaningful result. I ran the command with –repeat 30 for both before the recorded 150 run, so both would start on equal footing. With defined behavior, the results were “0.121670022 seconds time elapsed ( +- 0.11% )”; with undefined behavior, “0.123038640 seconds time elapsed ( +- 0.15% )”. The best of a further three runs had the opposite result, shown below:
% cat undef.150.best
perf stat –repeat 150 ./libavcodec/dct-test -i > /dev/null
Performance counter stats for ‘./libavcodec/dct-test -i’ (150 runs):
120.427535 task-clock (msec) # 0.997 CPUs utilized ( +- 0.11% )
21 context-switches # 0.178 K/sec ( +- 1.88% )
0 cpu-migrations # 0.000 K/sec ( +-100.00% )
226 page-faults # 0.002 M/sec ( +- 0.01% )
455’393’772 cycles # 3.781 GHz ( +- 0.05% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
1’306’169’698 instructions # 2.87 insns per cycle ( +- 0.00% )
89’674’090 branches # 744.631 M/sec ( +- 0.00% )
1’144’351 branch-misses # 1.28% of all branches ( +- 0.18% )
0.120741498 seconds time elapse
% cat def.150.best
Performance counter stats for ‘./libavcodec/dct-test -i’ (150 runs):
120.838976 task-clock (msec) # 0.997 CPUs utilized ( +- 0.11% )
21 context-switches # 0.172 K/sec ( +- 1.98% )
0 cpu-migrations # 0.000 K/sec
226 page-faults # 0.002 M/sec ( +- 0.01% )
457’077’626 cycles # 3.783 GHz ( +- 0.08% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
1’306’321’521 instructions # 2.86 insns per cycle ( +- 0.00% )
89’673’780 branches # 742.093 M/sec ( +- 0.00% )
1’148’393 branch-misses # 1.28% of all branches ( +- 0.11% )
0.121162660 seconds time elapsed ( +- 0.11% )
I also compared the disassembled code from jrevdct.o, before and after the changes to have defined behavior (using gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2 on x86_64).
In the build directory for the code with defined behavior:
objdump -d libavcodec/jrevdct.o > def.dis
sed -e 's/^.*://' def.dis > noline.def.dis
In the build directory for the code with undefined behavior:
objdump -d libavcodec/jrevdct.o > undef.dis
sed -e 's/^.*://' undef.dis > noline.undef.dis
Leaving aside difference in jump locations (despite the fact that they can impact performance), there are two differences:
diff -u build_benchmark_undef/noline.undef.dis build_benchmark_def/noline.def.dis
– 0f bf 50 f0 movswl -0x10(%rax),%edx
+ 0f b7 58 f0 movzwl -0x10(%rax),%ebxi
It’s switched to using a zero-extension rather than sign-extension in one place.
– 74 1c je 40 <ff_j_rev_dct+0x40>
– c1 e2 02 shl $0x2,%edx
– 0f bf d2 movswl %dx,%edx
– 89 d1 mov %edx,%ecx
– 0f b7 d2 movzwl %dx,%edx
– c1 e1 10 shl $0x10,%ecx
– 09 d1 or %edx,%ecx
– 89 48 f0 mov %ecx,-0x10(%rax)
– 89 48 f4 mov %ecx,-0xc(%rax)
– 89 48 f8 mov %ecx,-0x8(%rax)
– 89 48 fc mov %ecx,-0x4(%rax)
+ 74 19 je 3d <ff_j_rev_dct+0x3d>
+ c1 e3 02 shl $0x2,%ebx
+ 89 da mov %ebx,%edx
+ 0f b7 db movzwl %bx,%ebx
+ c1 e2 10 shl $0x10,%edx
+ 09 da or %ebx,%edx
+ 89 50 f0 mov %edx,-0x10(%rax)
+ 89 50 f4 mov %edx,-0xc(%rax)
+ 89 50 f8 mov %edx,-0x8(%rax)
+ 89 50 fc mov %edx,-0x4(%rax)
Leaving aside differences in register use:
– 0f bf d2 movswl %dx,%edx
There is one extra movswl instruction in the version with undefined behavior, at least with the particular version of the particular compiler for the particular architecture checked.
This is an example of a null result while benchmarking; neither version performs better, although any given benchmark has one or the other come out ahead, generally by less than the variance within the run. If this were a suggested performance change, it would not make sense to apply it. However, the point of this change was correctness; a performance increase is not expected, and the lack of a performance penalty is a bonus.
One of my fantastic OPW mentors prepared a “Welcome task package”, of self-contained, approachable, useful tasks that can be done while getting used to the code, and with a much smaller scope than the core objective. This is awesome. To any mentors reading this: consider making a welcome package!
Step one of it is to use ubsan with gdb. This turned out to be somewhat intricate, so I have decided to supplement the wiki’s documentation with a step-by-step guide for Ubuntu 14.04.
1) Install clang-3.5 (sudo aptitude install clang-3.5), as Ubuntu 14.04 comes with gcc 4.8, which does not support -fsanitize=undefined.
2) Under libav, mkdir build_ubsan && cd build_ubsan && ../configure --toolchain=clang-usan --extra-cflags=-fno-sanitize-recover (alternatively, –cc=clang –extra-cflags=-fsanitize=undefined –extra-ldflags=-fsanitize=undefined can be used instead of –toolchain=clang-usan).
3) make -j8 && make fate
4) Watch where the tests die (they only die if –extra-cflags=-fno-sanitize-recover is used). For me, they died on TEST idct8x8. Running make V=1 fate and asking my mentors pointed me towards libavcodec/dct-test -i, which is dying on jrevdct.c:310:47: with “runtime error: left shift of negative value -14”. If you really want to err on the side of caution, make a second build dir, and ./configure --cc=clang && make -j8 && make fate in it, making sure it does not fail… this confirms that the problem is related to configuring with –toolchain=clang-usan (and, it turns out, with -fsanitize=undefined).
5) It’s time to use the information my mentor pointed out on the wiki about ubsan at https://wiki.libav.org/Security/Tools – specifically, the information about useful gdb breakpoints. I put a modified version of the b_u definitions into ~/.gdbinit. The wiki has been updated now, but was originally missing a few functions, including one that turns out to be relevant: __ubsan_handle_shift_out_of_bounds
6 Run gdb ./libavcodec/dct-test, then at the gdb prompt, set args -i to set the arguments dct-test was being run with, and then b_u to load the ubsan breakpoints defined above. Then start the program: type run at the gdb prompt.
7) It turns out that a problem can be found, and the program stops running. Get a backtrace with bt.
680 in __ubsan_handle_shift_out_of_bounds ()
#1 0x000000000048ac96 in __ubsan_handle_shift_out_of_bounds_abort ()
#2 0x000000000042c074 in row_fdct_8 (data=<optimized out>) at /home/me/opw/libav/libavcodec/jfdctint_template.c:219
#3 ff_jpeg_fdct_islow_8 (data=<optimized out>) at /home/me/opw/libav/libavcodec/jfdctint_template.c:273
#4 0x0000000000425c46 in dct_error (dct=<optimized out>, test=<optimized out>, is_idct=<optimized out>, speed=<optimized out>) at /home/me/opw/libav/libavcodec/dct-test.c:246
#5 main (argc=<optimized out>, argv=<optimized out>) at /home/me/opw/libav/libavcodec/dct-test.c:522
It would be nice to see a bit more detail, so I wanted to compile the project so that less would be optimized out, and eventually settled on -O1 because compiling with ubsan and without optimizations failed (which I reported as bug 683). This led to a slightly better backtrace:
#0 0x0000000000491a70 in __ubsan_handle_shift_out_of_bounds ()
#1 0x0000000000492086 in __ubsan_handle_shift_out_of_bounds_abort ()
#2 0x0000000000434dfb in ff_j_rev_dct (data=<optimized out>) at /home/me/opw/libav/libavcodec/jrevdct.c:275
#3 0x00000000004258eb in dct_error (dct=0x4962b0 <idct_tab+64>, test=1, is_idct=1, speed=0) at /home/me/opw/libav/libavcodec/dct-test.c:246
#4 0x00000000004251cc in main (argc=<optimized out>, argv=<optimized out>) at /home/me/opw/libav/libavcodec/dct-test.c:522
It is possible to work around the problem by modifying the source code rather than the compiler flags: FFmpeg did so within hours of the bug report – the commit is at http://git.videolan.org/?p=ffmpeg.git;a=commit;h=bebce653e5601ceafa004db0eb6b2c7d4d16f0c0 ! Both FFmpeg and Libav have also merged my patch to work around the problem (FFmpeg patch, Libav patch). The workaround of using -O1 was suggested by one of my mentors, lu_zero; –disable-optimizations does not actually disable all optimizations (in practice, it leaves in ones necessary for compilation), and it does not touch the -O1 that –toolchain=clang-usan now sets.
Wanting a better backtrace leads to the next post: a detailed guide to narrowing down a bug in a the C compiler, Clang. Yes, I know, the problem is never a bug in the C compiler – but this time, it was.
What’s the fun of only running code on platforms you physically have? Portability is important, and Libav actively targets several platforms. It can be useful to be able to try out the code, even if the hardware is totally unavailable.
Here is how to run Libav’s tests under aarch64, on x86_64 hardware and Ubuntu 14.04. This guide is provided in the hopes that it saves someone else 20 hours or more: there is a lot of once-excellent information which has become misleading, because a lot of progress has been made in aarch64 support. I have tried three approachs – building with Linaro’s cross-compiler, building under QEMU user emulation, and building under QEMU system emulation, and cross-compiling. Building with a cross-compiler is the fastest option. Building under user emulation is about ten times slower. Building under system emulation is about a hundred times slower. There is actually a fourth option, using ARM Foundation Model, but I have not tried it. Running under QEMU user emulation is the only approach I managed to make entirely work.
For all three approaches, you will want a rootfs; I used Ubuntu Core. You can download Ubuntu Core for aarch64 (a minimal rootfs; see https://wiki.ubuntu.com/Core to learn more), and untar it (as root) into a new directory. Then, set an environment variable that the rest of this guide/set of notes uses frequently, changing the path to match your system:
export a64root=/path/to/your/aarch64/rootdir
Approach 1 – build under QEMU’s user emulation.
Step 1) Set up QEMU. The days when using SUSE branches were necessary are over, but it still needs to be statically linked, and not all QEMU packages are. Ubuntu has a static QEMU:
sudo aptitude install qemu-user-static
This package also sets up binfmt for you. You can delete broken or stale binfmt information by running:
echo -1 > /proc/sys/fs/binfmt_misc/archnamehere – this can be useful, especially if you have previously installed QEMU by hand.
Step 2) Copy your QEMU binary into the chroot, as root, with:
cp `which qemu-aarch64-static` $a64root/usr/bin/
Step 3) As root, set up the aarch64 image so it can do DNS resolution, so you can freely use apt-get:
echo 'nameserver 8.8.8.8' > $a64root/etc/resolv.conf
Step 4) Chroot into your new system. Run chroot $a64root /bin/bash as root.
At this point, you should be able to run an aarch64 version of ls, and confirm with file /bin/ls that it is an aarch64 binary.
Now you have a working, emulated, minimal aarch64 system.
On x86, you would run aptitude build-dep libav, but there is no such package for aarch64 yet, so outside of the chroot, on the normal system, I installed apt-rdepends and ran:
apt-rdepends --build-depends --follow=DEPENDS libav
With version information stripped out, the following packages are considered dependencies:
debhelper frei0r-plugins-dev libasound2-dev libbz2-dev libcdio-cdda-dev libcdio-dev libcdio-paranoia-dev libdc1394-22-dev libfreetype6-dev libgnutls-dev libgsm1-dev libjack-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libopenjpeg-dev libopus-dev libpulse-dev libraw1394-dev librtmp-dev libschroedinger-dev libsdl1.2-dev libspeex-dev libtheora-dev libtiff-dev libtiff5-dev libva-dev libvdpau-dev libvo-aacenc-dev libvo-amrwbenc-dev libvorbis-dev libvpx-dev libx11-dev libx264-dev libxext-dev libxfixes-dev libxvidcore-dev libxvmc-dev texi2html yasm zlib1g-dev doxygen
Many of the libraries do not have current aarch64 Ubuntu packages, and neither does frei0r-plugins-dev, but running aptitude install on the above list installs a lot of useful things – including build-essential. The full list is in the command below; the missing packages are non-essential.
Step 5) Set it up: apt-get install aptitude
aptitude install git debhelper frei0r-plugins-dev libasound2-dev libbz2-dev libcdio-cdda-dev libcdio-dev libcdio-paranoia-dev libdc1394-22-dev libfreetype6-dev libgnutls-dev libgsm1-dev libjack-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libopenjpeg-dev libopus-dev libpulse-dev libraw1394-dev librtmp-dev libschroedinger-dev libsdl1.2-dev libspeex-dev libtheora-dev libtiff-dev libtiff5-dev libva-dev libvdpau-dev libvo-aacenc-dev libvo-amrwbenc-dev libvorbis-dev libvpx-dev libx11-dev libx264-dev libxext-dev libxfixes-dev libxvidcore-dev libxvmc-dev texi2html yasm zlib1g-dev doxygen
Now it is time to actually build libav.
Step 6) Create a user within your chroot: useradd -m auser, and switch to running as that user: sudo -u auser bash, and type cd to go to the home directory.
Step 7) Run git clone git://git.libav.org/libav.git, then ./configure --disable-pthreads && make -j8 (change the 8 to approximately the number of CPU cores you have).
On my hardware, this takes 10-11 minutes, and ‘make fate’ takes about 16. Disabling pthreads is essential, as qemu-user does not handle threads well, and running the tests hangs randomly without it.
Approach 2: cross-compile (warning: I do not have the tests working with this approach).
1) Start by getting an aarch64 compiler. A good place to get one is http://releases.linaro.org/latest/components/toolchain/binaries/; I am using http://releases.linaro.org/latest/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.8-2014.04_linux.tar.xz . Untar it, and add it to your path:
export PATH=$PATH:/path/to/your/linaro/tools/bin
2) Make the cross-compiler work. Run aptitude install lsb lib32stdc++6. Without this, invoking the compiler will say “No such file or directory”. See http://lists.linaro.org/pipermail/linaro-toolchain/2012-January/002016.html.
3) Under the libav directory (run git clone git://git.libav.org/libav.git if you do not have one), type mkdir a64crossbuild; cd a64crossbuild. Make sure the libav directory is somewhere under $a64root (it should simplify running the tests, later).
4)./configure --arch=aarch64 --cpu=generic --cross-prefix=aarch64-linux-gnu- --cc=aarch64-linux-gnu-gcc --target-os=linux --sysroot=$a64root --target-exec="qemu-aarch64-static -L $a64root" --disable-pthreads
This is a minimal variant of Jannau’s configuration – a developer who has recently done a lot of libav aarch64 work.
5) Run make -j8. On my hardware, it takes just under a minute.
6) Run make fate. Unfortunately, both versions of QEMU I tried hung on wait4 at this point (in fft-test, fate-fft-4), and used an extra couple of hundred megabytes of RAM per second until I stopped QEMU, even if I asked it to wait for a remote GDB. For anyone else trying this, https://lists.libav.org/pipermail/libav-devel/2014-May/059584.html has several useful tips for getting the tests to run after cross-compilation.
Approach 3: Use QEMU’s system emulation. In theory, this should allow you to use pthreads; in practice, the tests hung for me. The following May 9th post describes what to do: http://www.bennee.com/~alex/blog/2014/05/09/running-linux-in-qemus-aarch64-system-emulation-mode/. In short: git clone git://git.qemu.org/qemu.git qemu.git && cd qemu.git && ./configure --target-list=aarch64-softmmu && make, then
./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -fsdev local,id=r,path=$a64root,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r
Then, under the buildroot system, log in as root (no password), and type mkdir /mnt/core && mount -t 9p -o trans=virtio r /mnt/core. At this point, you can run chroot /mnt/core /bin/bash, and follow the approach 1 instructions from useradd onwards, except that ./configure without –disable-pthreads should theoretically work. On my system, ./configure takes a bit over 5 minutes with this approach. Running make is quite slow; time make took 113 minutes. Do not use -j – you are limited to a single core, so -j would slow compilation down slightly. However, make fate consistently hung on acodec-pcm-alaw, and I have not yet figured out why.
Things not to do:
Applying to OPW requires an initial contribution. The Libav IRC channel suggested porting the asettb filter from FFmpeg, so I did (version 5 of the patch was merged upstream, in two parts: a rename patch and a content patch; the FFmpeg author was credited as author for the latter, while I did a signed-off-by). I also contributed a 3000+ line documentation patch, standardizing the libavfilter documentation and removing numerous English errors, and triaged a few bugs, git bisecting the one that was reproducible.
And how it nearly ruined another video coding standard.
Everyone knows that interlacing was a trick in the '80s for pseudo motion compensation with analogue video. This more or less worked because it mimicked how television worked back then. This technique was preserved when flat panels for pc and tv were introduced, for a mix of backward compatibility and technical limitations, and video coding features interlacing in MPEG2 and H264 and similar.
However as with black and white, TACS and Gopher, old technology has to be replaced with modern and efficient technology, as a trade off of users' interests and technology providers' market prospects. In case you are not familiar, interlacing is a mess to support, makes decoding slower and heavily degrades quality. People saying that interlacing saves bandwidth do not know much about video coding and bad marketing claiming that higher resolution is better than higher framerate has an effect too.
So, when ITU and then MPEG set out to establish the mandates for a new video standard capable of superseding H264, it was decided that interlacing was old enough, did more harm than good and it was time for retirement: HEVC was going to be the first video codec to officially deprecate interlacing.
Things went pretty swell during its development, until a few months before the completion of the standard. A group of US companies complained that the proposed tools were not sufficient (a set of SEI messages and treating fields like progressive frames) and heavily protested with both standardisation bodies. ITU firmly rejected the idea (with the video group chair threatening to step down) while MPEG set out to understand the needs of the industry and see if there was anything that could be done.
An ad-hoc group was established to see if there was any evidence that interlaced coding tool would have improved the situation. Things looked really shady, the Requirements group even mentioned that it was the first time that an AhG was established to look for evidence, instead of establishing an AhG because there was evidence. Several liasons from EBU and other DVB members tried to point out this absurdity while the threat of adding interlacing back in HEVC became real. Luckily the first version of the specifications got published in the meantime, so this decision didn't slow down the standardisation process.
Why so much love towards interlacing? Well in the "rebellious" group defence, it is true that interlaced content in HEVC is less performant than in H264; however it is also true that such deinterlaced content in HEVC outperforms H264 in any configuration. Truth is that mass marketed deinterlacers (commonly found in televisions for example) bring a lot of royalty income, so it is normal that companies with vested interests would prefer to have interlacing in a soon-popular video standard like HEVC. Also in markets like US where the network operator (which has control on the encoding but not on the video source) might differ from the content provider, it could be politically difficult to act as a carrier only if you have to deinterlace a video.
However these problems are actually not enough for forcing every encoder, decoder, analyser to support a deprecated technology like interlacing. Technical problems can be solved with good deinterlacers at the top of the distribution chain, while political ones can be solved amending contracts. Plus having progressive only video will definitely improve quality and let the industry concentrate on other delicate subjects, like bit depth, both properties going in favour of users' interests.
At the last MPEG meeting, the "rebellious" group which had been working on reintroducing interlacing for a year provided no real evidence that interlaced coding tools would improve HEVC at all. The only sensible solution was to disband the group over this wasted effort and support progressive video only, which is what happened luckily. So now both ITU and MPEG support progressive video only and this has finally nailed it.
Interlacing is dead, long live progressive.
Written by Vittorio Giovara (projectsymphony@gmail.com)
Published under a CC-BY-SA 3.0 license.
I am very glad to announce that Libav 10 has been released!
This has a bunch of features that I contributed to, in particular regarding stereoscopic video and interlaced filtering, but more importantly this release has the work of an awesome group of people which has been carried out for a whole year. This is the magic of open source!
I joined the group more or less one year ago, with some patches regarding an obscure H.264 specification which I then later reimplemented in HEVC and then I wrote a few filters I needed and then designed an API and then, wow! A whole year passed without me noticing, and I am still around, sending patches to the same group of people who welcomed someone who had problems with shifting values (sad but true story)!
I met the team both at VDD and FOSDEM and they've been the most exciting conferences I ever went to (and I went to a lot of them). I couldn't believe I was with the devteam of my favourite multimeida opensource projects I've been following since I was a kid! Until a year ago, I saw the names from the commits and the blogposts from both VideoLAN and Libav projects and I had been thinking "Oh wouldn't it be so cool to be like one of them".
The answer is yes, it definitely would, and it's something that can happen if one is really committed in it! The Libav Info page states "Being a committer is a duty, not a privilege", but it sure does feel like one.
Thanks for this exciting year guys, I look forward to the next ones.
...using latest modern tools!
X264 and VLC are two of the most awesomest opensource software you can find on-line and of course the pose no problem when you compile them on a Unix environment. Too bad that sometimes you need to think of Windowze as well, so we need a way to crosscompile that software: in this blogpost, I'll describe how to achieve that, using modern tools on a Ubuntu 12.04 installation.
[0] Sources
It goes without saying that without the following guides, I'd have had a much harder time!
http://alex.jurkiewi.cz/blog/2010/cross-compiling-x264-for-win32-on-ubuntu-linux
https://bbs.archlinux.org/viewtopic.php?id=138128
http://wiki.videolan.org/Win32Compile
http://forum.videolan.org/viewtopic.php?f=32&t=101489
So a big thanks to all the original authors!
[1] Introduction
When you crosscompile you just use the same tools and toolchains that you are used to, gcc, ld and so on, but configured (and compiled) so that they produce executable code for a different platform. This platform can vary both in software and in hardware and it is usually identified by a triplet: the processor architecture, the ABI and the operating system.
What we are going to use here is i686-w64-mingw32, which identifies any x86 cpu since the Pentium III, the w64 ABI used on modern Windows NT systems (if I'm not wrong), and the mingw32 architecture, that is the Windows gcc variant.
[2] Prerequisites
Note that the name of the packages might be slightly different according to your distribution. We are going to need a quite recent mingw-runtime for VLC (>=3.00) which has not yet landed on Ubuntu, so we'll take it from our Debian cousins.
Execute this command$ wget http://ftp.jp.debian.org/debian/pool/main/m/mingw-w64/mingw-w64-dev_3.0~svn4933-1_all.deb
$ sudo dpkg -i mingw-w64-dev_3.0~svn4933-1_all.deb
and then install stock dependencies
$ sudo dpkg -i gcc-mingw-w64 g++-mingw-w64$ sudo dpkg -i pkg-config yasm subversion cvs git-core $ mkdir -p ~/win32-cross/{src,lib,include,share,bin}#!/bin/shPlease not the use of the CFLAGS variables: without all the static parameters, the executable will dynamically link gcc, so you'll need to bundle the equivalent dll. I prefer to have one single exe, so everything goes static, but I'm not really sure which flag is actually needed. If you have any idea, please drop me a line.
TRIPLET=i686-w64-mingw32
export CC=$TRIPLET-gcc
export CXX=$TRIPLET-g++
export CPP=$TRIPLET-cpp
export AR=$TRIPLET-ar
export RANLIB=$TRIPLET-ranlib
export ADD2LINE=$TRIPLET-addr2line
export AS=$TRIPLET-as
export LD=$TRIPLET-ld
export NM=$TRIPLET-nm
export STRIP=$TRIPLET-strip
export PATH="/usr/i586-mingw32msvc/bin:$PATH"
export PKG_CONFIG_PATH="$HOME/win32-cross/lib/pkgconfig/"
export CFLAGS="-static -static-libgcc -static-libstdc++ -I$HOME/win32-cross/include -L$HOME/win32-cross/lib -I/usr/$TRIPLET/include -L/usr/$TRIPLET/lib"
export CXXFLAGS="$CFLAGS"
exec "$@"
$ cd ~/win32-cross/src
$ wget -qO - http://zlib.net/zlib-1.2.7.tar.gz | tar xzvf -
$ cd zlib-1.2.7
$ ../../mingw ./configure
$ sed -i"" -e 's/-lc//' Makefile
$ make
$ DESTDIR=../.. make install prefix=$ cd ~/win32-cross/src
$ git clone git://git.libav.org/libav.git
$ cd libav
$ ./configure \
--target-os=mingw32 --cross-prefix=i686-w64-mingw32- --arch=x86 --prefix=../.. \
--enable-memalign-hack --enable-gpl --enable-avisynth --enable-runtime-cpudetect \
--disable-encoders --disable-muxers --disable-network --disable-devices
$ make
$ make install$ cd ~/win32-cross/src
$ svn checkout http://ffmpegsource.googlecode.com/svn/trunk/ ffms
$ cd ffms
$ ../../mingw ./configure --host=mingw32 --with-zlib=../.. --prefix=$HOME/win32-cross
$ ../../mingw make
$ make install$ cd $HOME/win32-x264/src
# Create a CVS auth file on your machine
$ cvs -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac login
$ cvs -z3 -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac co -P gpac
$ cd gpac
$ chmod +rwx configure src/Makefile
# Hardcode cross-prefix
$ sed -i'' -e 's/cross_prefix=""/cross_prefix="i686-w64-mingw32-"/' configure
$ ../../mingw ./configure --static --use-js=no --use-ft=no --use-jpeg=no \ --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no \ --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no \ --disable-ssl --disable-opengl --disable-wx --disable-oss-audio \ --disable-x11-shm --disable-x11-xv --disable-fragments--use-a52=no \ --disable-xmlrpc --disable-dvb --disable-alsa --static-mp4box \ --extra-cflags="-I$HOME/win32-cross/include -I/usr/i686-w64-mingw32/include" \ --extra-ldflags="-L$HOME/win32-cross/lib -L/usr/i686-w64-mingw32/lib"
# Fix pthread lib name
$ sed -i"" -e 's/pthread/pthreadGC2/' config.mak
# Add extra libs that are required but not included
$ sed -i"" -e 's/-lpthreadGC2/-lpthreadGC2 -lwinmm -lwsock32 -lopengl32 -lglu32/' config.mak
$ make
# Make will fail a few commands after building libgpac_static.a# (i586-mingw32msvc-ar cr ../bin/gcc/libgpac_static.a ...).# That's fine, we just need libgpac_static.a i686-w64-mingw32-ranlib bin/gcc/libgpac_static.a $ cp bin/gcc/libgpac_static.a ../../lib/
$ cp -r include/gpac ../../include/
Finally we can compile x264 at full power! The configure script will provide a list of what features have been activated, make sure everything you need is there!$ cd ~/win32-cross/src
$ git clone git://git.videolan.org/x264.git
$ cd x264
$ ./configure --cross-prefix=i686-w64-mingw32- --host=i686-w64-mingw32 \ --extra-cflags="-static -static-libgcc -static-libstdc++ -I$HOME/win32-cross/include" \ --extra-ldflags="-static -static-libgcc -static-libstdc++ -L$HOME/win32-cross/lib" \ --enable-win32thread
$ make$ git clone git://git.videolan.org/vlc.git vlc$ cd vlc And let's get the dependencies through the contrib scripts, qt4 needs to be compiled by hand as the version in Ubuntu repositories doesn't cope well with the rest of the process. I also had to remove some of the files because they were of the wrong architecture (mileage might vary here) .$ mkdir -p contrib/win32
$ cd contrib/win32
$ ../bootstrap --host=i686-w64-mingw32
$ make prebuilt
$ make .qt4$ rm ../i686-w64-mingw32/bin/{moc,uic,rcc}$ cd -$ ./bootstrap$ mkdir win32 && cd win32$ ../extras/package/win32/configure.sh --host=i686-w64-mingw32$ ./compile$ make package-win-commonFor future reference, all (or most of) functions and structs in libav have a prefix that indicates the exposure of that functions. Those are
Well, I've finished the new audio decoding API, which has been merged into Libav master. The new audio encoding API is basically done, pending a (hopefully final) round of review before committing.
Next up is audio timestamp fixes/clean-up. This is a fairly undefined task. I've been collecting a list of various things that need to be fixed and ideas to try. Plus, the audio encoding API revealed quite a few bugs in some of the demuxers. Today I started a sort of TODO list for this stage of the project. I'll be editing it as the project continues to progress.
For the past few weeks I've been working on a new project sponsored by FFMTech. The entire project involves reworking much of the existing audio framework in libavcodec.
Part 1 is changing the audio decoding API to match the video decoding API. Currently the audio decoders take packet data from an AVPacket and decode it directly to a sample buffer supplied by the user. The video decoders take packet data from an AVPacket and decode it to an AVFrame structure with a buffer allocated by AVCodecContext.get_buffer(). My project will include modifying the audio decoding API to decode audio from an AVPacket to an AVFrame, as is done with video.
AVCODEC_MAX_AUDIO_FRAME_SIZE puts an arbitrary limit on the amount of audio data returned by the decoder. For example, each FLAC frame can hold up to 65536 samples for 8 channels at 32-bit sample depth, which is 2097152 bytes of raw audio, but AVCODEC_MAX_AUDIO_FRAME_SIZE is only 192000. Using get/release_buffer() for audio decoding will solve this problem. It will, however, require changes to every audio decoder. Most of those changes are trivial since the frame size is known prior to decoding the frame or is easily parsed. Some of the changes are more intrusive due to having to determine the frame size prior to allocating and writing to the output buffer.
As part of the preparation for the new API, I have been cleaning up all the audio decoder, which has been quite tedious. I've found some pretty surprising bugs along the way. I'm getting close to finishing that part so I'll be able to move on to implementing the new API in each decoder.
So, I've moved on from AHT now, and it's on to Spectral Extension (SPX). I got the full syntax working yesterday, now I just need to figure out how to calculate all the parameters. I have a feeling this will help quality quite a bit, especially when used in conjunction with variable bandwidth/coupling. My vision for automatic bandwidth adjustment is starting to come together.
SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement. Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors. The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream). For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors. Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.
Once SPX encoding is working properly, I'll revisit variable bandwidth. However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate. This could also be used in a VBR mode with constrained bitrate limits.
If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx
I finally got the complete AHT syntax working properly. Unfortunately, the quality seems to be lower at all bitrates than with the normal AC-3 quantization. I'm hoping that I just need to pick better gain values, but I have a suspicion that some of the difference is related to vector quantization, which the encoder has no control over (a basic 6-dimensional VQ minimum distance search is the best it can do).
My first step is to find out for sure if choosing better gain values will help. One problem is that the bit allocation model is saying we need X number of bits for each mantissa. Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions. So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations. Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT). This is probably the point where I need a more accurate bit allocation model...
edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range. They are nearly identical. So the difference is clearly in VQ. Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.
edit2: dct+VQ is nearly always worse than linear quantization... I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off. I don't know what to do at this point... *sigh*
note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.
edit3: It turns out that the majority of the quality difference was in the 6-point DCT. If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better. I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation... If not then I'm at another dead-end.
edit4: I'm giving up on AHT for now. The DCT is definitely correct and is very certainly causing the quality decrease. If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this. Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.
Well, I finally got a working E-AC-3 encoder committed to Libav. The bitstream format does save a few bits here and there, but the overall quality difference is minimal. However, it will be the starting point for adding more E-AC-3 features that will improve quality.
The first feature I completed was support for higher bit rates. This is done in E-AC-3 by using fewer blocks per frame. A normal AC-3 frame has 6 blocks of 256 samples each, but E-AC-3 can reduce that to 1, 2, or 3 blocks. This way a small range can be used for the per-frame bit rate, but it still allow for increasing the per-second bit rate. For example, 5.1-channel E-AC-3 content on HD-DVDs was typically encoded at 1536 kbps using 1 block per frame.
Currently I am working on implementing AHT (adaptive hybrid transform). The AHT process uses a 6-point DCT on each coefficient across the 6 blocks in the frame. It basically uses the normal AC-3 bit allocation process to determine quantization of each DCT-processed "pre-mantissa" but it uses a finer resolution for quantization and different quantization methods. I have the 6-point DCT working and one of the two quantization methods. Now I just need to finish the other quantization method and implement mantissa bit counting and bitstream output.
| Feed | RSS | Last fetched |
|---|---|---|
| Ambient Language | XML | 2019-05-23 00:31 |
| Kostya's Boring Codec World | XML | 2019-05-23 00:31 |
| libav – alpaastero | XML | 2019-05-23 00:31 |
| Libav – Federico Tomassetti – Consultant Software Engineer | XML | 2019-05-23 00:31 |
| libav – Luca Barbato | XML | 2019-05-23 00:31 |
| Multimedia – Flameeyes's Weblog | XML | 2019-05-23 00:31 |
| Project Symphony | XML | 2019-05-23 00:31 |
| Sasshka's | XML | 2019-05-23 00:31 |
| Scarabeus' blag | XML | 2019-05-23 00:31 |
If you want your blog to be added, send an email to Luca Barbato.
Planet service provided by Luminem.