From owner-freebsd-hackers@freebsd.org Sun Nov 19 10:34:33 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08D21DF1626 for ; Sun, 19 Nov 2017 10:34:33 +0000 (UTC) (envelope-from andre@fbsd.e4m.org) Received: from mail.g66.org (mail.g66.org [85.10.206.112]) by mx1.freebsd.org (Postfix) with ESMTP id 9076B73397 for ; Sun, 19 Nov 2017 10:34:31 +0000 (UTC) (envelope-from andre@fbsd.e4m.org) Received: from x55b535d2.dyn.telefonica.de (x55b535d2.dyn.telefonica.de [85.181.53.210]) (authenticated bits=128) by mail.g66.org (8.15.2/8.15.2) with ESMTPA id vAJAWfCx073117 for ; Sun, 19 Nov 2017 11:32:42 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Received: from stationary.client ([192.168.128.2]) by gate.local (8.15.2/8.15.2) with ESMTP id vAJAWfTu003358; Sun, 19 Nov 2017 11:32:41 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Received: from submit.client ([127.0.0.1]) by voyager.local (8.15.2/8.15.2) with ESMTP id vAJAWfbN025399; Sun, 19 Nov 2017 11:32:41 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Received: (from user@localhost) by voyager.local (8.15.2/8.15.2/Submit) id vAJAWf9S025398; Sun, 19 Nov 2017 11:32:41 +0100 (CET) (envelope-from andre@fbsd.e4m.org) Date: Sun, 19 Nov 2017 11:32:41 +0100 From: Andre Albsmeier To: freebsd-hackers@freebsd.org Cc: andre@fbsd.e4m.org Subject: gmirror synchronising is very slow due to frequent metadata updates Message-ID: <20171119103241.GA20588@voyager> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.7.2 (2016-11-26) X-Virus-Scanned: clamav-milter 0.99.2 at colo X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Nov 2017 10:34:33 -0000 I am running a 2 TB gmirror with two disks. I replaced one disk and the mirror started to synchronise. I noticed that the sync rate was about 50 MB/s which is a lot lower than the more than 120 MB/s which are possible with this hardware. The reason for this are the very frequently happening metadata updates done in sys/geom/mirror/g_mirror.c: if (sync->ds_offset_done + (MAXPHYS * 100) < offset) { /* Update offset_done on every 100 blocks. */ sync->ds_offset_done = offset; g_mirror_update_metadata(disk); With MAXPHYS being 128kB this would mean that metadata is updated every 12 MB -- or 4 times a second @ 50 MB/s. Out of curiosity I changed this to MAXPHYS * 2000 which means that the updating is done every 240 MB. Now the disks are synchronising with approx. 105 MB/s... Is there any reason why these metadata updates are done so often and what are the risks having them more infrequently (as I do now)? -Andre