View source for Talk:EBZip

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
  mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history \
  --pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
  mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| 
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebuzip -z -l 5</code><br />(zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebuzip -z -l 5</code><br />(libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code><br />(liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code><br />(liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code><br />(liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code><br />(liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with poor compression results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could by default take advantage of multi-threading, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.