DWARFS

From Just Solve the File Format Problem
(Redirected from DwarFS)
Jump to: navigation, search
File Format
Name DWARFS
Ontology
Compression lossless, optional
Magic Bytes 44 57 41 52 46 53
Developed By Marcus Holland-Moritz
Released 2020 [1]

Deduplicating Warp-speed Advanced Read-only File System (DWARFS) is a read-only filesystem that optionally supports no compression (via --compress-level=0 or --compression=null for instance) and no deduplication (via --file-hash none).

It is developed by Marcus Holland-Moritz and when compared to Squashfs, DWARFS also offers a choice of hashing algorithms,[2] as well as a tool for checking DWARFS image via dwarfsck.

At maximum compression levels using LZMA, DWARFS (using --compression=lzma:level=9:extreme) can produce smaller file size compared to SquashFS with the rough equivalent of using XZ as -comp xz.

The software is currently available for both Unix-like (generally Linux) and Microsoft Windows, with the caveat that, "Support for the Windows operating system is currently experimental. Having worked pretty much exclusively in a Unix world for the past two decades, my experience with Windows development is rather limited and I'd expect there to definitely be bugs and rough edges in the Windows code."[3] There is even homebrew formulae for macOS.

See also Squashfs.

Contents

Discussion

The aim with this software project is to ultimately create a compressed, deduplicated, read-only file system. While it is possible to disable all except for read-only, disabling it defeats the purpose.

DWARFS software project also intends to compete (performance-wise) with SquashFS at file system creation, in that a given directory, folder, or path is scanned, hashed, (optionally categorized) before the contents are compressed, adding only the unique copies. SquashFS tends to add and hash files as it creates the file system, similar to how 7-Zip or ZIP files for instance are created, with the apparent ability to detect and avoid adding duplicate files during creation process that is unique to the likes of SquashFS and DWARFS, which however is considered slower, generally double-handling compared to DWARFS which does these during the scanning phase.

The developer's original motivation was with, "several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space" and that there, "was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them."[4] This may put the project in line for competing with git, which is a Versioning Control System (VCS), that is used to record changes to a given project at a more atomic level, compressing (and using zlib) changes with each commit, however, unlike git which introduces rather steep learning curves, contrasting to just adding several versions of the same software project into an "archive" is arguably more trivial to do. Ultimately, DWARFS, like SquashFS has many other potential use-cases, rather than whatever they were intentionally designed for, making it a somewhat popular choice in the race against other forms of archivers and software/file distribution methods, without requiring the use of bespoke compression software that happens to be ranked among the top in Matt Mahoney's data compression benchmarks.

Identification

DWARFS files begins with the hexadecimal 44 57 41 52 46 53 which translates to "DWARFS" in ASCII.[5]

Examples

Create an extremely compressed DWARFS image, without history, all root owned contents, sha3-512 hash algorithm, idle task (lowest priority) from present directory to example.dwarfs:

 mkdwarfs --input . --output=example.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --compress-niceness=10 --schema-compression=lzma:level=9:extreme \
 --metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --set-owner=0 --set-group=0 --no-history-timestamp \
 --no-create-timestamp --no-history-command-line

mkdwarfs only features

The following below illustrates features that are only available upon creating the DWARFS file. Once the DWARFS file has been created, the only way to change these is to use --recompress=all.

Choice of available compression

As of version 0.12.4, the following compressors are available:

Switch Comments
--compression=null No compression.[6]
--compression=lzma liblzma compression[7], under modern Linux environments, XZ's liblzma is used.
--compression=zstd libzstd compression[8]
--compression=lz4 liblz4 compression,[9] choice of either LZ4HC or LZ4 compression are available.
--compression=brotli Brotli compressor[10]
--compression=flac FLAC compression[11]
--compression=ricepp RICEPP compression, likely Golomb coding[12][13]

Choice of available hash for duplication detection

Since 0.7.0-RC1 introduced alternatives,[14] as of version 0.12.4, the following hash algorithms are available:

Switch Remarks
--file-hash=none Disable file deduplication checks.[15]
--file-hash=blake2b512
--file-hash=blake2s256
--file-hash=md5 "As of 2019, MD5 continues to be widely used, despite its well-documented weaknesses and deprecation by security experts."[16] Vulnerabilities include collision attacks and preimage vulnerability.
--file-hash=md5-sha1
--file-hash=ripemd160 Appeared as a checksum test in git hash de5ec99.[17]
--file-hash=sha1 Formerly used hash algorithm.[18]
Vulnerable to collision attacks.[19]
--file-hash=sha224
--file-hash=sha256
--file-hash=sha384
--file-hash=sha512
--file-hash=sha3-224
--file-hash=sha3-256
--file-hash=sha3-384
--file-hash=sha3-512
--file-hash=sha512-224
--file-hash=sha512-256
--file-hash=shake128 Disabled by author in git hash afbd85e.[20]
--file-hash=shake256 Disabled by author in git hash afbd85e.[21]
--file-hash=sm3
--file-hash=xxh3-64 Added by author in git hash 7ded26d.[22]
--file-hash=xxh3-128 Current default choice when not specified.[23]

Links

References

  1. Release 0.1.0 - GitHub
  2. Choice of hash for duplicate detection #92 - GitHub
  3. Windows Support section of README.md - DWARFS - GitHub
  4. History section - DWARFS - GitHub
  5. 0000449: Add magic for the DWARFS compressed file system format - bugs.astron.com
  6. null.cpp (line 110) - DWARFS - GitHub
  7. lzma.cpp (line 413) - DWARFS - GitHub
  8. zstd.cpp (line 179) - DWARFS - GitHub
  9. lz4.cpp (line 186) - DWARFS - GitHub
  10. brotli.cpp (line 169) - DWARFS - GitHub
  11. flac.cpp (line 496) - DWARFS - GitHub
  12. rice.hpp (lines 7-9) - compression-algorithms - GitHub
  13. ricepp.cpp (line 260) - DWARFS - GitHub
  14. Choice of hash for duplicate detection #92 post #5 - DWARFS - GitHub
  15. Choice of hash for duplicate detection #92, post #5 - DWARFS - GitHub
  16. Security section of MD5 - Wikipedia
  17. Commit de5ec99 - test/checksum_test.cpp (line 55) - test(checksum): add checksum tests - DWARFS - GitHub
  18. Choice of hash for duplicate detection #92, post #1 - DWARFS - GitHub
  19. Attacks section of SHA-1 - Wikipedia
  20. Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub
  21. Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub
  22. Commit 7ded26d - feat(checksum): add hexdigest() method - DWARFS - GitHub
  23. v0.7.0-RC1 - DWARFS releases by tag - GitHub
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox