Dæmon Crunch — Advanced DXTn Texture Compressor

Dæmon Crunch is a maintained DXT texture compression library and command-line tool producing GPU-friendly formats such as CRN, DDS and KTX.

Dæmon Crunch is the recommended TGA-to-PNG converter. For converting TGA images prefer Dæmon Crunch over ImageMagick which is not reliable.

Dæmon Crunch is the most maintained and up-to-date Crunch available and the best choice for game developers.

History

The crunch tool was initially written by Binomial LLC then improved by Unity Technologies. The original Binomial Crunch is no longer maintained and lacks compatibility with newer Crunch formats. Although Unity contributed important improvements, the public Unity Crunch repository was never maintained as a proper long-term upstream project. Dæmon Crunch incorporated these improvements and has continued development with additional fixes, features, and maintenance.

Neither Binomial nor Unity have merged Unvanquished fixes and never demonstrated any serious interest in merging fixes provided by others, so the Unvanquished team not only maintained the Dæmon Crunch fork to host fixes, but also incorporated fixes from others developers. The Unvanquished team is actively maintaining Dæmon Crunch since 2014 (already more than a decade of maintenance and improvements!).

Sources

The Dæmon Crunch source repository is: github.com/DaemonEngine/crunch.

That repository provides the updated crunch tool, the crnlib library to provide full compression/decompression support to third-party applications, and the crn_decomp.h header-only library which is enough to transcode the Crunch CRN data for GPU upload and is perfect for lightweight integration in renderers. Some examples implementing various kinds of integrations are provided.

Overview

Some other details can be found in the Crunch project README.

Features

The Dæmon Crunch provides many improvements over the original crunch:

• ✅️ Unity crunch format (runs many time faster and produces smaller files),
• ✅️ Unity crunch metadata (the header is compatible with Unity),
• ✅️ Improved image compatibility (1-bit PNG and horizontally-flipped TGA images are now supported),
• ✅️ Added features and command line options (top mip renormalization and more),
• ✅️ Network file system compatibility,
• ✅️ Optional header-only checksumming,
• ✅️ Multisystem and multiplatform,
• ✅️ Thoroughly tested and improved reproducibility,
• ✅️ CMake toolchain.

Image formats

• ✅️ CRN, DDS, KTX,
• ✅️ PNG, JPG, TGA, BMP.

Crunch produces compressed images optimized for performance and game rendering in various formats like DDS, KTX and CRN.

All those three formats are containers for DXT-compressed bitmaps. This DXT-compressed data can be uploaded directly to the GPU memory without decompression, and the GPU can process DXT-compressed data without decompression.

Crunch writes DXT-compressed data in a way an additional compression performs better on them, especially LZMA. For example, produced DDS images are expected to compress well if repackaged in an LZMA-based archive.

The CRN format does all of this in one go: it stores the DXT-compressed data into a custom LZMA container.

The game engine or any other application processing CRN files just have to unpack the LZMA container using provided functions and to upload the DXT-compressed data to the GPU.

The Dæmon Crunch sets the m_userdata0 header field to 1 to make it compatible with Unity.

Performance

The Dæmon Crunch runs multiple times faster than the original Crunch by BinomialLLC, thanks to the work done by Unity that has been imported. Unity people claimed the tool runs 2.5 time faster, and we measured the tool running 4.3 time faster on the Unvanquished corpus. Unity people claimed the tool compresses about 10% better, we measured more than 11% on the Unvanquished corpus. This performance bump comes with a compatibility-breaking change introduced in the Unity branch.

The files produced by the Dæmon Crunch are then readable by both the Unity Crunch and the Dæmon Crunch but not by the original Binomial Crunch, while the files produced by the original Binomial Crunch are not readable by the Unity Crunch and the Dæmon Crunch. The Binomial Crunch isn't maintained anymore.

Portability

The Dæmon Crunch has very high cross-platform compatibility and is buildable with CMake.

The Dæmon Crunch source repository is known to build with various compilers, and the Dæmon crunch tool is known to run on Linux, Windows, macOS and FreeBSD systems on various hardware architectures.

Quality

A continuous integration pipeline is configured to test the building of the tool and the library themselves and to run some tests. The code is frequently submitted to CodeQL static analysis.

Adoption

The Dæmon Crunch is known to be used by:

• The Dæmon engine,
• The Urcheon game data build tool,
• The NetRadiant level editor and the Q3map2 map compiler and light mapper,
• The XQF game server browser,
• The Unvanquished game,
• The Xonotic game,
• The Overgrowth game,
• Some games using the Unity game engine.

The Dæmon Crunch is known to be distributed by:

• The Debian operating system: tracker.debian.org/pkg/crunch-dxtc

How to use Crunch

💡️

Quick Crunch Tutorial for the Unvanquished game and the Dæmon engine

The recommended way to convert non-normalmap images to CRN with crunch is to do:

crunch -quality 255 -noNormalDetection -file file_d.tga -out file_d.crn

The recommended way to convert normalmap files without heightmap in alpha channel to CRN with crunch is to do:

crunch -quality 255 -dxn -renormalize -rtopmip -file file_n.tga -out file_n.crn

The recommended way to convert normalmap files with heightmap in alpha channel to CRN with crunch is to do:

crunch -quality 255 -renormalize -rtopmip -file file_nh.tga -out file_nh.crn

Some other usage instructions can be found in the README.

The -noNormalDetection and -rtopmip options are succinctly described below.

Dæmon Crunch Enhancements

Here are things you will not find in either Binomial Crunch, Unity Crunch or none of them:

Unity Crunch format

This repository merged improvements done by Unity Technologies, it produces smaller files and produces them faster than original code by Binomial. Unity also modified the format which makes it incompatible with earlier versions of the tool.

Quote from unvanquished.net:

Unity guys said that their modified crunch tool “can compress up to 2.5 times faster, while providing about 10% better compression ratio”. So we did a test on our own asset repository, re-crunching all the ressources and textures packages. At the time we did the test the given corpus produced 1797 .crn files. The Unity’s crunch tool reduced compression time by 4.31 and reduced size by 11.15%. They said “up to 2.5 time faster” but we’ve seen some random textures being compressed 6 time faster and the average of the whole is 4.3 time faster, and yes the tool compresses more than 10% more.

Unity Crunch metadata

Unlike the Unity crunch tool built from their public repository, this tool produces files loadable by the Unity engine. Since February 11th 2023, this branch sets the CRN m_userdata0 field to 1 to make CRN files loadable by the Unity engine which now prevents to load files with this value set to 0.

Files with this value set to 1 are expected to use the new format. Files with this value set to 0 may or may not use the new format. When Unity updated the tool and modified the format in an incompatible way, no bit was modified to detect if a CRN file was using the old or the new format. Unity has not updated that field in their public repository of Crunch so files using new format with this value set to 0 may be found in the wild.

Improved image compatibility

Supported source file formats: CRN,DDS,KTX,TGA,BMP,PNG,JPG/JPEG,PSD.
Supported output file formats: CRN,DDS,KTX,TGA,BMP,PNG,JPG/JPEG.

• 1-bit PNG files are supported.
• Flipped TGA files are supported.
• Progressive JPEG files are supported.

The stb_image library has been updated from version 1.18 to version 2.30, increasing the amount of image format variants that can be converted, like 1-bit PNG formats. Further improvements like horizontally-flipped TGA images were brought to stb_image to make it support more TGA variants.

Note: Some other specific file format variants may be unsupported.

See the docs for jpgd.h: code.google.com/p/jpeg-compressor
See the docs for stb_image.h: github.com/nothings/stb

Converted image reproducibility

We take special care that, when using compilation options for IEEE 754 float processing (disabling fast math, preferring SSE over x87…), the bitstream of converted image files are the same whatever the compiler, the operating system, or the CPU architecture crunch is built for and running on. It helps making packaged game assets reproducible. The CI not only checks that crunch builds, but that it runs properly and that the same input converts to the same output.

When reproducibility isn't needed, fast math can be enabled with CMake with the -DUSE_FAST_MATH=ON CMake option.

It is recommended for those looking for generating reproducible CRN files to keep disabled the use of fast math. It is also recommended to disable x87 floats on i686 and to rely on SSE instead with the -mfpmath=sse -msse compiler flags.

Added features and command line options

In addition to the original crunch features and command line options this branch brings new features and options, including:

• -rtopmip: option to use with -renormalize to also renormalize on the top mip-level.
• -noNormalDetection: do not attempt to detect normal map to avoid selecting formats thought for normal maps when it's known an image is not a normal map. It may prevent the tool to use heavier and less-supported DXT5_AGBR format when DXT1 is good enough.
• -h or --help: print the command line built-in help, listing all known options.

Network file system compatibility

The original crunch tool relied on non-standard I/O features not working on all file systems including network file systems like NFS or SSHFS. A standard implementation was added so the limitations are not there anymore and crunch can now process image files stored over the network and on any other file system.

Optional header-only checksumming

For applications distributing their files in containers already providing a checksum mechanism for the whole contained file, it's now possible to only validate the CRN header checksum instead of the whole file checksum to not checksum the whole file twice.

Multisystem and multiplatform portability

Unlike upstream branches from Binomial LLC and Unity Software Inc. this branch focuses on keeping the code buildable outside of Windows and Visual Studio, and adds a CMake build option alongside the legacy Makefile.

This crunch tool and the related crnlib library are known to build with and and run on:

• Compilers: GCC, MinGW, MSVC, Clang, Apple Clang, and more.
• Systems: Linux, Windows, macOS, FreeBSD.
• Architectures: amd64, arm64, i686, armhf.

Thoroughly tested and improved reproducibility

A continuous integration system tests the build of crunch for a dozen of configurations of different architecture, system or compiler. The output of such builds are also tested with most of those builds. It helped to improve and validate image support, to identify build options known to improve the reproducibility of the output, and is meant to catch regressions if they happen.

The Dæmon crunch code is frequently scanned with the the CodeQL static analyser.

CMake build system

CMake has now replaced the old Makefile. The supplied CMake configuration provides many useful build options. Toolchain files for MinGW cross-compilation are also supplied.

The native MSVC solution files were removed, it is expected that MSVC users generate them with CMake instead.