The joys of ZFS: transparent compression
One of the great things of ZFS is its built-in compression capability. Enabling compression on a ZFS filesystem allows you to save a significant amount of space, especially in a field like ours where one sometimes needs to use large plain text files. It is completely transparent: the user doesn’t notice anything. Well… unless they check the output of the
ls command carefully, of course. I just noticed the most extreme example I have seen to date:
▶ ls -lh total 640G -rw-r----- 1 user1 group1 12T May 15 00:27 chr10impv1.gen -rw-r----- 1 user1 group1 695G May 13 15:01 chr22impv1.gen -rw-r----- 1 user1 group1 1.1K May 15 12:53 fcgene_out_fcgene.log -rw-r----- 1 user1 group1 2.7M May 14 17:40 file1.sample -rw-r----- 1 user1 group1 3.2M May 14 17:46 file2.sample
ls says that the first
.gen file is 12TB in size, but the actually used total disk space (see the first line) is 640GB!
An alternative way to check this is using the
du command to show disk space usage:
▶ du -bh 13T . ▶ du -h 640G .
This ZFS filesystem uses the default LZ4 compression, and as can be seen below, the compression ratio on that filesystem is quite good:
▶ sudo zfs get compression,compressratio storage/projects/Project1 NAME PROPERTY VALUE SOURCE storage/projects/Project1 compression lz4 inherited from storage storage/projects/Project1 compressratio 7.70x -
More information on ZFS compression can be found at, for example, the following to blogs: