Compression
From Just Solve the File Format Problem
(Difference between revisions)
Dan Tobias (Talk | contribs) (→Stream compression formats) |
(→Compression algorithms and compressed data formats) |
||
| (144 intermediate revisions by 9 users not shown) | |||
| Line 2: | Line 2: | ||
|formattype=electronic | |formattype=electronic | ||
|thiscat=Compression | |thiscat=Compression | ||
| + | |image=Vise.png | ||
}} | }} | ||
| Line 7: | Line 8: | ||
A stream format takes a stream of bytes, and outputs a different, hopefully smaller, stream of bytes. These compression formats are often used internally in other data structures to compress data, as well as in network protocols, such as http. Used stand-alone, a stream compression format does not offer archiving capability, however in the UNIX doctrine, an archiver like [[tar]] can be combined with an archive format to produce a proper compressed archive. | A stream format takes a stream of bytes, and outputs a different, hopefully smaller, stream of bytes. These compression formats are often used internally in other data structures to compress data, as well as in network protocols, such as http. Used stand-alone, a stream compression format does not offer archiving capability, however in the UNIX doctrine, an archiver like [[tar]] can be combined with an archive format to produce a proper compressed archive. | ||
| + | |||
| + | == Compression algorithms and compressed data formats == | ||
| + | |||
| + | (excluding formats usually used as file formats) | ||
| + | |||
| + | * [[Arithmetic coding]] | ||
| + | * [[Brotli]] | ||
| + | * [[Burrows–Wheeler transform]] | ||
| + | * [[DEFLATE]] | ||
| + | * [[DivANS]] | ||
| + | * [[EXEPACK2]] | ||
| + | * [[FastLZ]] | ||
| + | * [[Fractal compression]] | ||
| + | * [[HPACK (HTTP)|HPACK]] (used to compress HTTP headers) | ||
| + | * [[Huffman coding]] | ||
| + | ** [[Adaptive Huffman coding]] | ||
| + | ** [[Canonical Huffman code]] | ||
| + | * [[Lempel–Ziv–Stac]] (LZS) | ||
| + | * [[LZ4]] | ||
| + | * [[LZ77]] | ||
| + | * [[LZ77 with Huffman coding]] | ||
| + | * [[LZ78]] | ||
| + | * [[LZFSE]] | ||
| + | * [[LZHAM]] | ||
| + | * [[LZMA]] | ||
| + | * [[LZMA2]] | ||
| + | * [[LZMW]] | ||
| + | * [[LZO]] (Lempel–Ziv–Oberhumer) | ||
| + | * [[LZRW]] | ||
| + | * [[LZSS]] (Lempel–Ziv–Storer–Szymanski) | ||
| + | * [[LZW]] | ||
| + | * [[LZX]] | ||
| + | ** [[LZX Delta]] | ||
| + | * [[MSZIP]] | ||
| + | * [[PKWARE DCL Implode]] | ||
| + | * [[PPMd]] | ||
| + | * [[Prediction by partial matching]] (PPM) | ||
| + | * [[Run-length encoding]] | ||
| + | ** [[PackBits]] | ||
| + | ** [[RLE90]] | ||
| + | * [[Shannon–Fano coding]] | ||
| + | * [[Snappy]] (Google-created compression also used by Apple in iWork formats) | ||
| + | * [[zlib]] | ||
| + | * [[Zstandard]] (Zstd) | ||
| + | ** [[Zstandard dictionary]] | ||
| + | |||
| + | === Graphics compression === | ||
| + | See [[Graphics#Compression]] for compression formats used primarily with graphics. | ||
| + | |||
| + | == Specific file formats/programs == | ||
| + | |||
| + | (multi-file compressors are in [[Archiving]]) | ||
* [[9CDR]] (Amiga FileImploder Clone) | * [[9CDR]] (Amiga FileImploder Clone) | ||
| + | * [[ABC (Advanced Blocksorting Compressor)]] | ||
| + | * [[ABComp]] (.abp) | ||
| + | * [[Aldus PKZP]] | ||
| + | * [[BARF]] (.x, .x??) | ||
| + | * [[BGZF]] | ||
| + | * [[BWTC32Key]] (.B3K) | ||
| + | * [[bzip]] (.bz) | ||
* [[bzip2]] (.bz2) | * [[bzip2]] (.bz2) | ||
| − | * [[ | + | * [[BZip3]] (.bz3) |
| + | * [[CAZIP]] | ||
| + | * [[ChiefLZ]] (.LZZ) | ||
| + | * [[compact (Gene Olson)]] (.W) | ||
| + | * [[compact (Unix)]] (.C) | ||
| + | * [[compress (Unix)]] (.Z) | ||
| + | * [[CMP (NEW-DOS)]] | ||
* [[CrLZH]] (.?y?) | * [[CrLZH]] (.?y?) | ||
* [[CRN]] (.crn) - compressed text files used for PC-Write manual | * [[CRN]] (.crn) - compressed text files used for PC-Write manual | ||
* [[Crunch]] (.?z?) | * [[Crunch]] (.?z?) | ||
| + | * [[Crunch-Mania]] | ||
| + | * [[DACT]] | ||
| + | * [[DitPack]] | ||
* [[DUPA]] (Amiga FileImploder Clone) | * [[DUPA]] (Amiga FileImploder Clone) | ||
| + | * [[EDI Install archive]] | ||
| + | * [[EDI Install packed file]] | ||
| + | * [[EDI LZSSLib]] | ||
| + | * [[EmmaCompress]] | ||
* [[Error Code Modeler]] (.ecm) | * [[Error Code Modeler]] (.ecm) | ||
* [[File Imploder]] (Amiga) (.imp) | * [[File Imploder]] (Amiga) (.imp) | ||
| + | * [[FlySpeed st/exp]] | ||
| + | * [[Freeze/Melt]] (Unix) (.F) | ||
* [[gzip]] (.gz) | * [[gzip]] (.gz) | ||
| − | * [[ | + | * [[Hammer (compression)|Hammer]] |
| − | * [[LZMA]] (.lzma) | + | * [[HAP]] (v2.x) |
| − | * [[ | + | * [[ICE (Michael Lamanuzzi)|ICE]] (.ice, by Michael Lamanuzzi) |
| − | * [[ | + | * [[Inflate]] (.infl) - parody format that actually increases size |
| + | * [[InstallIt!]] | ||
| + | * [[JAM (W. Jiang)]] | ||
| + | * [[The JAM Packer|JAM Packer, The]] | ||
| + | * [[JCalG1]] (.jc, Commodore Amiga) | ||
| + | * [[Kboom11]] | ||
| + | * [[lrzip]] (.lrz) | ||
| + | * [[LZ5 and Lizard]] | ||
| + | * [[LZARI]] | ||
| + | * [[LZHUF]] | ||
| + | * [[Lzip]] (.lz) | ||
| + | * [[LZMA Alone|LZMA_Alone]] (.lzma) | ||
| + | * [[lzop]] (.lzo) | ||
| + | * [[LZS221]] | ||
| + | * [[LZSS (Haruhiko Okumura)]] | ||
| + | * [[LZWCOM]] | ||
| + | * [[LzwPak]] | ||
| + | * [[Microsoft KWAJ installation compression]] | ||
| + | * [[Microsoft SZ installation compression]] | ||
| + | * [[Mozilla LZ4]] (.jsonlz4) | ||
| + | * [[MS-DOS installation compression]] (.??_) | ||
| + | * [[MULTIKIT COMPRESS]] | ||
| + | * [[NetWare Packed File]] | ||
| + | * [[NOAH]] | ||
| + | * [[Nova Compress]] | ||
| + | * [[NPack]] | ||
| + | * [[NRV (Not Really Vanished)]] | ||
| + | * [[Oodle Data Compression]] | ||
| + | * [[pack (Unix)]] (.z) | ||
| + | * [[PGNPack]] | ||
| + | * [[PowerPacker]] (.pp) | ||
| + | * [[PPMZ]] | ||
| + | * [[PPMZ2]] | ||
| + | * [[Pro-Pack]] | ||
| + | * [[PUT and GET (Scott Jennings)]] | ||
| + | * [[Quasijarus Strong Compression]] | ||
| + | * [[RAC]] | ||
| + | * [[Reduq]] | ||
| + | * [[RKUC]] | ||
| + | * [[rzip]] (.rz) | ||
| + | * [[S2]] (.s2, .snappy) | ||
| + | * [[SCO compress LZH]] | ||
| + | * [[SEMONE]] (.one) | ||
| + | * [[SFDN Packer]] | ||
| + | * [[SHARC]] (density) | ||
* [[Softdisk Text Compressor]] (.ctx) | * [[Softdisk Text Compressor]] (.ctx) | ||
| + | * [[Splint]] | ||
| + | * [[SQ2]] | ||
* [[Squash (RISC OS)|Squash]] - single file compression on RISC OS | * [[Squash (RISC OS)|Squash]] - single file compression on RISC OS | ||
* [[Squeeze|Squeeze/SQ]] (.?q?) | * [[Squeeze|Squeeze/SQ]] (.?q?) | ||
| + | * [[SQUEEZE (AJP Systems)]] | ||
| + | * [[squeeze (Daniel Bernstein)]] (.MW) | ||
| + | * [[Squish (Mike Albert)]] | ||
| + | * [[Sunzip]] (.sz) | ||
| + | * [[TERSE]] | ||
| + | * [[Top4]] | ||
| + | * [[TTCOMP]] | ||
| + | * [[TTW]] | ||
| + | * [[tzip]] | ||
| + | * [[Tzip (Text ZIP)]] | ||
| + | * [[UHBC]] | ||
| + | * [[WWPACK]] | ||
| + | * [[XPK]] | ||
* [[XZ]] (.xz) | * [[XZ]] (.xz) | ||
| + | * [[Yaz0]] | ||
| + | * [[zchunk]] | ||
| + | * [[Zoo filter format]] | ||
| + | * [[Zoo Z format]] | ||
| + | * [[ZSQ (LZW compression)]] | ||
| + | |||
| + | == Transparent file compression == | ||
| + | (A mostly-obsolete technique in which a background process hooks into the OS and decompresses files transparently when they are read. ''Compression'' might be done manually. The line between this and ''Disk compression'' can be blurry.) | ||
| + | |||
| + | * [[AFX (Atari ST)]] | ||
| + | * [[CFS (Computer Concepts Compression)]] | ||
| + | * [[DIET (compression)|DIET]] | ||
| + | * [[Diet Disk]] | ||
| + | * [[Pack-Ice]] | ||
| + | * [[SLIM (Dominic Herity)|SLIM]] | ||
| + | * [[Turbo Packer]] | ||
| + | * [[XPACK]] | ||
| + | |||
| + | == Disk compression == | ||
| + | (Transparent disk compression software and the like, excluding filesystem formats.) | ||
| + | |||
| + | * [[DiskDoubler]] | ||
| + | * [[DriveSpace]] and DoubleSpace | ||
| + | * [[JAM (disk compression)]] | ||
| + | * [[Stacker]] | ||
| + | |||
| + | For filesystem formats that feature compression, see [[Filesystem]] (especially [[Filesystem#Compressed filesystems]]). | ||
| + | |||
| + | == Executable compression == | ||
| + | |||
| + | See [[Executable compression]]. | ||
== See also == | == See also == | ||
| − | * [[Archiving]] | + | * [[Compression and archiving software]] |
| + | * [[Compression research]] | ||
| + | * [[Archiving]] (for formats that place multiple files together in one file, with or without compression) | ||
| + | * [[Graphics#Compression|Graphics]] (for image-specific compression) | ||
| + | * [[Video]] (for video-specific compression) | ||
| + | * [[Audio and Music]] (for audio-specific compression) | ||
| + | |||
| + | == Links == | ||
| + | * [http://boingboing.net/2014/03/04/data-compression-with-playing.html Data-compression with playing cards] | ||
| + | * [http://www.quora.com/What-is-the-most-compressed-file-ever/answer/Kelly-Martin?srid=uejz&share=1 What is the most compressed file ever?] | ||
| + | * [http://ieeeghn.org/wiki/index.php/History_of_Lossless_Data_Compression_Algorithms History of Lossless Data Compression Algorithms] | ||
Revision as of 13:57, 3 March 2024
Contents |
Stream compression formats
A stream format takes a stream of bytes, and outputs a different, hopefully smaller, stream of bytes. These compression formats are often used internally in other data structures to compress data, as well as in network protocols, such as http. Used stand-alone, a stream compression format does not offer archiving capability, however in the UNIX doctrine, an archiver like tar can be combined with an archive format to produce a proper compressed archive.
Compression algorithms and compressed data formats
(excluding formats usually used as file formats)
- Arithmetic coding
- Brotli
- Burrows–Wheeler transform
- DEFLATE
- DivANS
- EXEPACK2
- FastLZ
- Fractal compression
- HPACK (used to compress HTTP headers)
- Huffman coding
- Lempel–Ziv–Stac (LZS)
- LZ4
- LZ77
- LZ77 with Huffman coding
- LZ78
- LZFSE
- LZHAM
- LZMA
- LZMA2
- LZMW
- LZO (Lempel–Ziv–Oberhumer)
- LZRW
- LZSS (Lempel–Ziv–Storer–Szymanski)
- LZW
- LZX
- MSZIP
- PKWARE DCL Implode
- PPMd
- Prediction by partial matching (PPM)
- Run-length encoding
- Shannon–Fano coding
- Snappy (Google-created compression also used by Apple in iWork formats)
- zlib
- Zstandard (Zstd)
Graphics compression
See Graphics#Compression for compression formats used primarily with graphics.
Specific file formats/programs
(multi-file compressors are in Archiving)
- 9CDR (Amiga FileImploder Clone)
- ABC (Advanced Blocksorting Compressor)
- ABComp (.abp)
- Aldus PKZP
- BARF (.x, .x??)
- BGZF
- BWTC32Key (.B3K)
- bzip (.bz)
- bzip2 (.bz2)
- BZip3 (.bz3)
- CAZIP
- ChiefLZ (.LZZ)
- compact (Gene Olson) (.W)
- compact (Unix) (.C)
- compress (Unix) (.Z)
- CMP (NEW-DOS)
- CrLZH (.?y?)
- CRN (.crn) - compressed text files used for PC-Write manual
- Crunch (.?z?)
- Crunch-Mania
- DACT
- DitPack
- DUPA (Amiga FileImploder Clone)
- EDI Install archive
- EDI Install packed file
- EDI LZSSLib
- EmmaCompress
- Error Code Modeler (.ecm)
- File Imploder (Amiga) (.imp)
- FlySpeed st/exp
- Freeze/Melt (Unix) (.F)
- gzip (.gz)
- Hammer
- HAP (v2.x)
- ICE (.ice, by Michael Lamanuzzi)
- Inflate (.infl) - parody format that actually increases size
- InstallIt!
- JAM (W. Jiang)
- JAM Packer, The
- JCalG1 (.jc, Commodore Amiga)
- Kboom11
- lrzip (.lrz)
- LZ5 and Lizard
- LZARI
- LZHUF
- Lzip (.lz)
- LZMA_Alone (.lzma)
- lzop (.lzo)
- LZS221
- LZSS (Haruhiko Okumura)
- LZWCOM
- LzwPak
- Microsoft KWAJ installation compression
- Microsoft SZ installation compression
- Mozilla LZ4 (.jsonlz4)
- MS-DOS installation compression (.??_)
- MULTIKIT COMPRESS
- NetWare Packed File
- NOAH
- Nova Compress
- NPack
- NRV (Not Really Vanished)
- Oodle Data Compression
- pack (Unix) (.z)
- PGNPack
- PowerPacker (.pp)
- PPMZ
- PPMZ2
- Pro-Pack
- PUT and GET (Scott Jennings)
- Quasijarus Strong Compression
- RAC
- Reduq
- RKUC
- rzip (.rz)
- S2 (.s2, .snappy)
- SCO compress LZH
- SEMONE (.one)
- SFDN Packer
- SHARC (density)
- Softdisk Text Compressor (.ctx)
- Splint
- SQ2
- Squash - single file compression on RISC OS
- Squeeze/SQ (.?q?)
- SQUEEZE (AJP Systems)
- squeeze (Daniel Bernstein) (.MW)
- Squish (Mike Albert)
- Sunzip (.sz)
- TERSE
- Top4
- TTCOMP
- TTW
- tzip
- Tzip (Text ZIP)
- UHBC
- WWPACK
- XPK
- XZ (.xz)
- Yaz0
- zchunk
- Zoo filter format
- Zoo Z format
- ZSQ (LZW compression)
Transparent file compression
(A mostly-obsolete technique in which a background process hooks into the OS and decompresses files transparently when they are read. Compression might be done manually. The line between this and Disk compression can be blurry.)
Disk compression
(Transparent disk compression software and the like, excluding filesystem formats.)
- DiskDoubler
- DriveSpace and DoubleSpace
- JAM (disk compression)
- Stacker
For filesystem formats that feature compression, see Filesystem (especially Filesystem#Compressed filesystems).
Executable compression
See also
- Compression and archiving software
- Compression research
- Archiving (for formats that place multiple files together in one file, with or without compression)
- Graphics (for image-specific compression)
- Video (for video-specific compression)
- Audio and Music (for audio-specific compression)
