From owner-freebsd-bugs@freebsd.org Thu Mar 8 10:21:18 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F163FF49590 for ; Thu, 8 Mar 2018 10:21:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 328C372D85 for ; Thu, 8 Mar 2018 10:21:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 6E854210C3 for ; Thu, 8 Mar 2018 10:21:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w28ALGiD092437 for ; Thu, 8 Mar 2018 10:21:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w28ALGXV092436 for freebsd-bugs@FreeBSD.org; Thu, 8 Mar 2018 10:21:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 178504] [patch] switch logs compression from bzip2 to xz Date: Thu, 08 Mar 2018 10:21:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: theraven@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Mar 2018 10:21:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178504 David Chisnall changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |theraven@FreeBSD.org --- Comment #5 from David Chisnall --- Do we have any measurements regarding the CPU / RAM overhead here? Can thi= s be addressed by using a smaller block size without impacting compression ratios too much? A few measurements from my largest maillog file (852K uncompressed, much smaller than on a large installation): bzip2 baseline:=20 7948 maximum resident set size File size: 77KB. xz --fast: 4328 maximum resident set size File size: 80K xz --best: 59848 maximum resident set size File size: 70KB xz -5: 26196 maximum resident set size File size: 74KB xz -block-size=3D1M: 26160 maximum resident set size 70K The default appears to be --best.=20=20 If I concatenate multiple rolled-over maillogs to give a 4.6MB file, I get = this bzip2 baseline:=20 8316 maximum resident set size File size: 443K. xz --fast: 4436 maximum resident set size File size: 447K xz --best: 110092 maximum resident set size File size: 382K xz -5: 61524 maximum resident set size File size: 412K xz -block-size=3D1M 27920 maximum resident set size File size: 396K It looks as if setting the block size to 1MB prevents too much blowup in me= mory for very large files, gives us better compression that bzip2, though at the cost of more CPU and RAM. I think that the correct solution for this is probably: * Make the block size a configurable parameter * Make the default 1M * Make xz the default This should let us get the benefit from better compression, but without let= ting attackers consume too much memory (though they can consume CPU in proportio= n to the size of the input file, with a larger constant multiplier for xz than bzip2). --=20 You are receiving this mail because: You are the assignee for the bug.=