Date: Thu, 08 Mar 2018 10:21:16 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 178504] [patch] switch logs compression from bzip2 to xz Message-ID: <bug-178504-8-0CajX4y63M@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-178504-8@https.bugs.freebsd.org/bugzilla/> References: <bug-178504-8@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178504 David Chisnall <theraven@FreeBSD.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |theraven@FreeBSD.org --- Comment #5 from David Chisnall <theraven@FreeBSD.org> --- Do we have any measurements regarding the CPU / RAM overhead here? Can thi= s be addressed by using a smaller block size without impacting compression ratios too much? A few measurements from my largest maillog file (852K uncompressed, much smaller than on a large installation): bzip2 baseline:=20 7948 maximum resident set size File size: 77KB. xz --fast: 4328 maximum resident set size File size: 80K xz --best: 59848 maximum resident set size File size: 70KB xz -5: 26196 maximum resident set size File size: 74KB xz -block-size=3D1M: 26160 maximum resident set size 70K The default appears to be --best.=20=20 If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get = this bzip2 baseline:=20 8316 maximum resident set size File size: 443K. xz --fast: 4436 maximum resident set size File size: 447K xz --best: 110092 maximum resident set size File size: 382K xz -5: 61524 maximum resident set size File size: 412K xz -block-size=3D1M 27920 maximum resident set size File size: 396K It looks as if setting the block size to 1MB prevents too much blowup in me= mory for very large files, gives us better compression that bzip2, though at the cost of more CPU and RAM. I think that the correct solution for this is probably: * Make the block size a configurable parameter * Make the default 1M * Make xz the default This should let us get the benefit from better compression, but without let= ting attackers consume too much memory (though they can consume CPU in proportio= n to the size of the input file, with a larger constant multiplier for xz than bzip2). --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-178504-8-0CajX4y63M>