Into the Depths: The Technical Details behind AV1 by Nathan Egge

Basis Universal : Supercompressed GPU Video Texture Codec

I was poking about inside Mozilla's big WebXR demo to see how it was made.
 https://mixedreality.mozilla.org/hello-webxr/

Most of the assets are xxx.basis files.  ".basis"  I had never heard of a dot basis file and not much came up on the internet. 

Digging deeper I found a compiled Web Assembly Library and a javascript library wrapper for it.

  60935   basis_transcoder.js
 367268   basis_transcoder.wasm

https://github.com/BinomialLLC/basis_universal

Paper: GST: GPU-decodable Supercompressed Textures

They also used Draco which is also interesting for 3D object compression. 

In further searching it's starting to look like this basis codec is going to be a standard part of three.js and therefore aframe.io for web based VR development. 

Basis Universal GPU Texture Compression

texture video compression system that outputs a highly compressed intermediate file format (.basis) that can be quickly transcoded to a wide variety of GPU texture compression formats.

basis_universal

Basis Universal Supercompressed GPU Texture Codec

Basis Universal is a "supercompressed" GPU texture compression system that outputs a highly compressed intermediate file format (.basis) that can be quickly transcoded to a very wide variety of GPU compressed and uncompressed pixel formats: ASTC 4x4 L/LA/RGB/RGBA, PVRTC1 4bpp RGB/RGBA, PVRTC2 RGB/RGBA, BC7 mode 6 RGB, BC7 mode 5 RGB/RGBA, BC1-5 RGB/RGBA/X/XY, ETC1 RGB, ETC2 RGBA, ATC RGB/RGBA, ETC2 EAC R11 and RG11, FXT1 RGB, and uncompressed raster image formats 8888/565/4444.

The system now supports two modes: a high quality mode which is internally based off the UASTC compressed texture format, and the original lower quality mode which is based off a subset of ETC1 called "ETC1S". UASTC is for extremely high quality (similar to BC7 quality) textures, and ETC1S is for very small files. The ETC1S system includes built-in data compression, while the UASTC system includes an optional Rate Distortion Optimization (RDO) post-process stage that conditions the encoded UASTC texture data in the .basis file so it can be more effectively LZ compressed by the end user. More technical details about UASTC integration are here.

Basis files support non-uniform texture arrays, so cubemaps, volume textures, texture arrays, mipmap levels, video sequences, or arbitrary texture "tiles" can be stored in a single file. The compressor is able to exploit color and pattern correlations across the entire file, so multiple images with mipmaps can be stored very efficiently in a single file.

The system's bitrate depends on the quality setting and image content, but common usable ETC1S bitrates are .3-1.25 bits/texel. ETC1S .basis files are typically 10-25% smaller than using RDO texture compression of the internal texture data stored in the .basis file followed by LZMA. For UASTC files, the bitrate is fixed at 8bpp, but with RDO post-processing and user-provided LZ compression on the .basis file the effective bitrate can be as low as 2bpp for video or for individual textures approximately 4-6bpp.

The transcoder has been fuzz tested using zzuf.

So far, we've compiled the code using MSVS 2019, under Ubuntu x64 using cmake with either clang 3.8 or gcc 5.4, and emscripten 1.35 to asm.js. (Be sure to use this version or later of emcc, as earlier versions fail with internal errors/exceptions during compilation.) The compressor is multithreaded by default, but this can be disabled using the -no_multithreading command line option. The transcoder is currently single threaded.

Basis Universal supports "skip blocks" in ETC1S compressed texture arrays, which makes it useful for basic compressed texture video applications. Note that Basis Universal is still at heart a GPU texture compression system, not a video codec, so bitrates will be larger than even MPEG1.

Important Usage Notes

Probably the most important concept to understand about Basis Universal before using it: The system supports two very different universal texture modes: The original "ETC1S" mode is low/medium quality, but the resulting file sizes are very small because the system has built-in compression for ETC1S texture format files. This is the command line encoding tool's default mode. ETC1S textures work best on images, photos, map data, or albedo/specular/etc. textures, but don't work as well on normal maps. There's the second "UASTC" mode, which is significantly higher quality (near-BC7 grade), and is usable on all texture types including complex normal maps. UASTC mode purposely does not have built-in file compression like ETC1S mode does, so the resulting files are quite large (8-bits/texel - same as BC7) compared to ETC1S mode. The UASTC encoder has an optional Rate Distortion Optimization (RDO) encoding mode (implemented as a post-process over the encoded UASTC texture data), which lowers the output data's entropy in a way that results in better compression when UASTC .basis files are compressed with Deflate/Zstd, etc. In UASTC mode, you must losslessly compress the file yourself.

Basis Universal is not an image compression codec. It's a texture compression codec. It can be used just like an image compression codec, but that's not the only use case. Here's a good intro to GPU texture compression. If you're looking to primarily use the system as an image compression codec on sRGB photographic content, use the default ETC1S mode, because it has built-in compression.

The "-q X" option controls the output quality in ETC1S mode. The default is quality level 128. "-q 255" will increase quality quite a bit. If you want even higher quality, try "-max_selectors 16128 -max_endpoints 16128" instead of -q. -q internally tries to set the codebook sizes (or the # of quantization intervals for endpoints/selectors) for you. You need to experiment with the quality level on your content.

For tangent space normal maps, you should separate X into RGB and Y into Alpha, and provide the compressor with 32-bit/pixel input images. Or use the "-separate_rg_to_color_alpha" command line option which does this for you. The internal texture format that Basis Universal uses (ETC1S) doesn't handle tangent space normal maps encoded into RGB well. You need to separate the channels and recover Z in the pixel shader using z=sqrt(1-x^2-y^2).

3rd party code dependencies

The stand-alone transcoder (in the "transcoder" directory) is a single .cpp source file library which has no 3rd party code dependencies.

The encoder uses lodepng for loading and saving PNG images, which is Copyright (c) 2005-2019 Lode Vandevenne. It uses the zlib license. It also uses apg_bmp for loading BMP images, which is Copyright 2019 Anton Gerdelan. It uses the Apache 2.0 license.

The encoder uses tcuAstcUtil.cpp, from the Android drawElements Quality Program (deqp) Testing Suite, for unpacking the transcoder's ASTC output for testing/validation purposes. This code is Copyright 2016 The Android Open Source Project, and uses the Apache 2.0 license. We have modified the code so it has no external dependencies, and disabled HDR support.

https://git.ucsc.edu/jlao3/CMPM163Labs/-/tree/b34f42b1de294d35f52401c1eaff4f86843dd9fb/three.js-master/examples/js/libs/basis

This package uses BASIS and discusses is a little. 

https://www.npmjs.com/package/pixi-compressed-textures?activeTab=readme

H.266 (VVC) New Codec Announced: Laying the Groundwork for 8K Streaming

https://ymcinema.com/2020/07/10/h-266-vvc-new-codec-announced-laying-the-groundwork-for-8k-streaming/


The Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (HHI) has announced the standardization of a new codec called H.266 or VVC (Versatile Video Coding) which allows half of the bit rate compared to H.265 (HEVC) without a reduction in quality. The goal: To lay the groundwork for 8K streaming.

Support ultra-high-resolution: Up to 16K
The VVC should support resolutions from 4K to 16K as well as 360° videos, as well as support YCbCr 4:4:4, 4:2:2, and 4:2:0 with 10 to 16 bits per component




The Versatile Video Coding (VVC) Standard on the Final Stretch by Benjamin Bross

https://news.ycombinator.com/item?id=23747626

Google Finishes Defining Its VP9 Video Codec, Adds it to Chromium

http://thenextweb.com/google/2013/06/17/google-finishes-defining-its-vp9-video-codec-adds-it-to-chromium-ahead-of-chrome-and-youtube-rollout/

Last month, Google revealed that it was planning to finish defining its VP9 video codec on June 17 (today), after which it will start using the next-generation compression technology in Chrome and on YouTube. The company is wasting no time: it has already enabled the free video compression standard by default in the latest Chromium build.

The addition was first noted by developer and Google open-source Chromium evangelist François Beaufort. He points to a Chromium code review with the following short but succinct description: “Remove VP9 flag, and enable VP9 support by default.”

VP9 is the successor to VP8, both of which fall under Google’s WebM project of freeing Web codecs from royalty constraints. Despite the fact that Google unveiled WebM three years ago at its I/O conference, VP8 is still rarely used when compared to H.264, today’s most popular video codec.

As Beaufort explains, the main advantage of VP9 for users is that it’s 50 percent more efficient than H.264, meaning that you’ll use half the bandwidth on average when watching a video on the internet. Yet that doesn’t take H.265 into account, the successor to H.264 that offers comparable video quality at half the number of bits per second and also requires its implementers to pay patent royalties.

Although VP9 transmits video more efficiently than its predecessor, VP8 still saw patent-infringement claims from Nokia, and there’s no reason to assume VP9 won’t as well. Presumably, Google has learned and will make an effort to avoid such problems again, especially given that VP9 is meant to become part of WebRTC, an open project that lets users communicate in real-time via voice and video sans plugins, later this year.

As for its own products, Google has previously said it wants to build VP9 into Chrome, and today’s Chromium integration shows the company is putting its money where its mouth is. Furthermore, YouTube said last month it plans to add support for VP9 once the video codec lands in Chrome.

Chromium is the open source Web browser project that shares much of the same code as Google Chrome, and new features are often added there first. It’s not clear when VP9 will land in Chrome, but we’d wager it’s likely to trickle into the various channels during the next few months and arrive in the stable release by default before the end of the year.

WebRTC - bringing real time communications to the web.

WebRTC (Real-Time Communications) is an open-source project supported by Google. WebRTC components come from Google’s acquisition of GIPS (Global IP Solutions)  formerly "Global IP Sound".

It's been released with a Google License that's similar to a BSD license.

Some experiments I was doing with P2P between browsers back in 1996/7 using Java had mixed results. UDP in java wasn't always available or would act like it's working but never send or recieve or just revieve.  There was no way to get TCP/IP through NAT's and Firewalls at the time, I did fine a method that did worked that I published in 2002.  I was stunned when I found out Skype was using my firewall bypass idea. See posts on Slashdot.

I have a feeling WebRTC is going to be getting a lot more attention over time. The GIPS (Global IP Solutions) product was using by Citrix Online, Cisco’s WebEX, Yahoo, IBM, AOL and many others in particular Skype itself. 


From GIPS orignial Marketing Materials:

VoiceEngine
The VoiceEngine family consists of comprehensive, packaged solutions that handle all the necessary voice processing tasks for IP networks, providing superior voice quality even under adverse network conditions. VoiceEngine allows application developers, service providers and hardware manufacturers the ability to easily build complex voice processing technology into their solutions, without worrying about the effects of delay, jitter, packet loss, background noise and echo. These products have been optimized for high performance on a number of platforms- from PCs to Mobile devices- allowing customers the flexibility to design almost any application or device around Global IP Solutions’ cutting-edge technology.

VideoEngine
Global IP Solutions’ VideoEngine products deliver world-class voice and video quality to PC softphones, Unified Communications applications and mobile devices. The VideoEngine suite consists of flexible, easy to integrate media processing frameworks that overcome the limitations of IP networks to provide a clear, consistent user experience.

• Flexible solution delivers high quality voice and video to almost any application
• High-level API for easy integration and accelerated time to market
• Field-proven, best in class voice and video quality
• Optimized to provide synchronized audio and video
• Advanced techniques to handle jitter, packet loss, CPU and bandwidth constraints
• Supports an array of audio and video codecs

Posted on Slashdot 6/2/2011:
Google WebRTC: Can It Replace Skype?

"Google WebRTC, all open source, is part of the web revolution that allows one browser to talk directly to another without the need for a server getting involved. WebRTC is an API that used the new P2P web API to allow developers to implement audio and video communications using direct P2P links between browsers. This really is a game changer."

And, while this feature doesn't seem to have gotten a lot of attention so far, Google Voice can call landline and cell phones for a small fee, just like Skype.

The GIPS wideband iSAC codec was used for the great audio quality on Skype calls before they parted ways in 2007 with the Skype 3.2 release.


There was a post to the main Skype blog that included this (my emphasis in bold):

And because we’ve replaced our audio engine in our most recent releases — it’s now fully built in-house — it’s worth bearing in mind that you may run into some bumps when a call is placed from an older version of Skype to newer versions.

Looks like Skype really shot them selves in the foot on this one. With Google opensourcing it with a BSD style license soon Skype may be history.
Which may explain why they sold off to Microsoft reciently for $8.5 billion.

Tester(591) on Slashdot said:

The WebRTC code that was released is missing many important bits that are required to compete against Skype. The most important is probably a bandwidth management engine, the code that's currently public just sends at a pre-configured bitrate. That means it can only do low resolution video with a shitty quality.
That said, Google Talk in GMail and Android have a dynamic bitrate stuff, and I expect they will be released at some point. I should also mention that Farsight2/Farstream using in Empathy and Pidgin are currently gaining the same kind of bandwidth management that Google is doing. So we should get at least two independent open implementations soon.

WebRTC FAQ


Why Google bought Global IP Solutions ZDNet 5/2010


I did an article on my other blog on this:
WebRTC - bringing real time communications to the web.

Tandberg Attempts To Patent Open Source Code

From Slashdot

"As if the current situation with software patents wasn't bad enough, it appears a new phenomenon is emerging: companies are watching the commit logs of open source projects for ideas to patent. In this case, Tandberg filed a patent that was step-by-step identical to an algorithm developed by the x264 project — a mere two months after the original commit. The particular algorithm is a useful performance optimization in a wide variety of video encoders, including Theora."

WebM - Video Licensing Problems Resolved

WebM is a new free video codec based on Google (formerly On2) VP8 with the Vorbis open-source audio codec.

If you want to know more read:   Google shares VP8 code to create new video format



From Slashdot:

"The WebM licensing problems have been resolved. The copyright license is straight BSD now, and the patent license is separate and has no impact on the copyright license. Quoting Chris DiBona: 'As it was originally written, if a patent action was brought against Google, the patent license terminated. This provision itself is not unusual in an OSS license, and similar provisions exist in the 2nd Apache License and in version 3 of the GPL. The twist was that ours terminated "any" rights and not just rights to the patents, which made our license GPLv3 and GPLv2 incompatible. Also, in doing this, we effectively created a potentially new open source copyright license, something we are loath to do. Using patent language borrowed from both the Apache and GPLv3 patent clauses, in this new iteration of the patent clause we've decoupled patents from copyright, thus preserving the pure BSD nature of the copyright license. This means we are no longer creating a new open source copyright license, and the patent grant can exist on its own.'"

Wikipedia's Assault On Patent-Encumbered Codecs

From Slashdot:
Wikipedia's Assault On Patent-Encumbered Codecs

"The Open Video Alliance is launching a campaign today called Let's Get Video on Wikipedia, asking people to create and post videos to Wikipedia articles. (Good, encyclopedia-style videos only!) Because all video must be in patent-free codecs (theora for now), this will make Wikipedia by far the most likely site for an average internet user to have a truly free and open video experience. The campaign seeks to 'strike a blow for freedom' against a wave of h.264 adoption in otherwise open HTML5 video implementations."

Google Acquires On2

Here are a few links to the story:

Google Acquires Video Compression Technology Company On2 For $106 Million

Google to Acquire On2 Technologies

Flash which dominates web video today only uses to video codecs to stream video. H.264 (AKA MPEG4) and ON2 VP7. Technically you need to pay licensing fee's to use either.

H.264 is an open standard, which is still not free through.  There are a number of opensource versions of  the H.264 codec, most notably X.264 and JM10 (the original example code)



From Slashdot:
Free Software Foundation Urges Google To Free VP8

"The FSF have written an open letter to Google urging them to free the VP8 codec with an irrevocable royalty-free licence: 'With its purchase of the On2 video compression technology company having been completed on Wednesday February 16, 2010, Google now has the opportunity to make free video formats the standard, freeing the web from both Flash and the proprietary H.264 codec.'" Also from the letter: "The world would have a new free format unencumbered by software patents. Viewers, video creators, free software developers, hardware makers -- everyone -- would have another way to distribute video without patents, fees, and restrictions. The free video format Ogg Theora was already at least as good for web video (see a comparison) as its nonfree competitor H.264, and we never did agree with your objections to using it. But since you made the decision to purchase VP8, presumably you're confident it can meet even those objections, and using it on YouTube is a no-brainer."

A little more background
Flash and H.264: Together At Last

Developers: Money For Nothing and the Codecs For Free

From SlashDot:

"In an in depth discussion on the codec industry, CoreCodec CEO and Matroska Foundation board member Dan Marlin shares his thoughts on the growing popularity of the MKV container, confusion in the marketplace between X.264/MKV and DivXHD and weighs in on a controversial decision by Microsoft to block third party filter support in future versions of Windows media player. His interview offers a behind the scenes look at an important piece of technology that is helping to power the P2P movement. It also raises the prickly question of whether or not Microsoft is abusing their OS monopoly, in order to rein in competition within the codec industry."

AMR codec source code and Standards docs.

Adaptive Multi-Rate (AMR) is a patented audio data compression scheme optimized for speech coding. AMR was adopted as the standard speech codec by 3GPP in October 1998 and is now widely used in GSM and UMTS mobile phone standards.
AMR is also a file format for storing spoken audio using the AMR codec. Many modern mobile telephone handsets will allow you to store short recordings in the AMR format, both open source (see the external links) and commercial programs exist to convert between this and other formats such as MP3, although it should be remembered that AMR is a speech format and is unlikely to give ideal results for other audio. The common filename extension is .amr.
3GPP - 3rd Generation Partnership Project
Technical Specification Group Services and System Aspects
Performance characterization of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec

Location of Standards Docs and Codec source code:

• http://www.3gpp.org/ftp/Specs/latest/Rel-5/26_series
• http://www.3gpp.org/ftp/Specs/archive/26_series/26.204/