From owner-freebsd-hackers@freebsd.org Mon Nov 20 05:34:12 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD032DE22AF for ; Mon, 20 Nov 2017 05:34:12 +0000 (UTC) (envelope-from andre@fbsd.e4m.org) Received: from mail.g66.org (mail.g66.org [85.10.206.112]) by mx1.freebsd.org (Postfix) with ESMTP id 62CCA71690; Mon, 20 Nov 2017 05:34:11 +0000 (UTC) (envelope-from andre@fbsd.e4m.org) Received: from x4e35ecc2.dyn.telefonica.de (x4e35ecc2.dyn.telefonica.de [78.53.236.194]) (authenticated bits=128) by mail.g66.org (8.15.2/8.15.2) with ESMTPA id vAK5YAQM013383; Mon, 20 Nov 2017 06:34:10 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Received: from submit.client ([127.0.0.1]) by gate.local (8.15.2/8.15.2) with ESMTP id vAK5Y9Ps000616; Mon, 20 Nov 2017 06:34:10 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Received: (from user@localhost) by gate.local (8.15.2/8.15.2/Submit) id vAK5Y9xv000614; Mon, 20 Nov 2017 06:34:09 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Date: Mon, 20 Nov 2017 06:34:09 +0100 From: Andre Albsmeier To: Mark Johnston Cc: Andre Albsmeier , freebsd-hackers@freebsd.org Subject: Re: gmirror synchronising is very slow due to frequent metadata updates Message-ID: <20171120053409.GA57536@gate> References: <20171119103241.GA20588@voyager> <20171120033828.GA1959@bish> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171120033828.GA1959@bish> User-Agent: Mutt/1.7.2 (2016-11-26) X-Virus-Scanned: clamav-milter 0.99.2 at colo X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 05:34:13 -0000 On Sun, 19-Nov-2017 at 22:38:28 -0500, Mark Johnston wrote: > On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote: > > I am running a 2 TB gmirror with two disks. I replaced one disk and the > > mirror started to synchronise. I noticed that the sync rate was about > > 50 MB/s which is a lot lower than the more than 120 MB/s which are > > possible with this hardware. > > > > The reason for this are the very frequently happening metadata updates > > done in sys/geom/mirror/g_mirror.c: > > > > if (sync->ds_offset_done + (MAXPHYS * 100) < offset) { > > /* Update offset_done on every 100 blocks. */ > > sync->ds_offset_done = offset; > > g_mirror_update_metadata(disk); > > > > With MAXPHYS being 128kB this would mean that metadata is updated every > > 12 MB -- or 4 times a second @ 50 MB/s. > > > > Out of curiosity I changed this to MAXPHYS * 2000 which means that the > > updating is done every 240 MB. Now the disks are synchronising with > > approx. 105 MB/s... > > > > Is there any reason why these metadata updates are done so often and > > what are the risks having them more infrequently (as I do now)? > > I believe that the metadata updates are done only so that it's possible > to resume a synchronization after a crash. In particular, ds_offset_done That's what I thought I had figured out as well when looking at the code. > is translated into the sync_offset field in the metadata block of a > synchronizing disk. Therefore, there is no real risk in making the > updates less frequent; it'll just take marginally more time to complete > an interrupted rebuild of a mirror (which would hopefully be a rare > event anyway). > > We should probably decrease the update interval based on the size of a > mirror. For mirrors larger than say, 1GB, we might just update the > metadata block once per 1% of the synchronization operation's progress. I think best would be to have it updated every seconds. I think of very fast or very slow drives, or drives which are used heavily during rebuild (where the effective rebuild speed is quite low)...