Date: Thu, 08 Mar 2018 10:21:16 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 178504] [patch] switch logs compression from bzip2 to xz Message-ID: <bug-178504-8-0CajX4y63M@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-178504-8@https.bugs.freebsd.org/bugzilla/> References: <bug-178504-8@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178504 David Chisnall <theraven@FreeBSD.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |theraven@FreeBSD.org --- Comment #5 from David Chisnall <theraven@FreeBSD.org> --- Do we have any measurements regarding the CPU / RAM overhead here? Can this be addressed by using a smaller block size without impacting compression ratios too much? A few measurements from my largest maillog file (852K uncompressed, much smaller than on a large installation): bzip2 baseline: 7948 maximum resident set size File size: 77KB. xz --fast: 4328 maximum resident set size File size: 80K xz --best: 59848 maximum resident set size File size: 70KB xz -5: 26196 maximum resident set size File size: 74KB xz -block-size=1M: 26160 maximum resident set size 70K The default appears to be --best. If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get this bzip2 baseline: 8316 maximum resident set size File size: 443K. xz --fast: 4436 maximum resident set size File size: 447K xz --best: 110092 maximum resident set size File size: 382K xz -5: 61524 maximum resident set size File size: 412K xz -block-size=1M 27920 maximum resident set size File size: 396K It looks as if setting the block size to 1MB prevents too much blowup in memory for very large files, gives us better compression that bzip2, though at the cost of more CPU and RAM. I think that the correct solution for this is probably: * Make the block size a configurable parameter * Make the default 1M * Make xz the default This should let us get the benefit from better compression, but without letting attackers consume too much memory (though they can consume CPU in proportion to the size of the input file, with a larger constant multiplier for xz than bzip2). -- You are receiving this mail because: You are the assignee for the bug.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-178504-8-0CajX4y63M>