From owner-freebsd-hackers@freebsd.org Mon Nov 20 02:38:27 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D166DDE434 for ; Mon, 20 Nov 2017 02:38:27 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg0-x230.google.com (mail-pg0-x230.google.com [IPv6:2607:f8b0:400e:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C0006C7C2 for ; Mon, 20 Nov 2017 02:38:27 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg0-x230.google.com with SMTP id 70so6178648pgf.6 for ; Sun, 19 Nov 2017 18:38:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=PtQwXmz7BFM1ZNyZb9oikPRn6SnUuujkla1dML1t4+I=; b=gWnrtpNMoVgjDsP1E+87aiKA0wZL2RSVhXYibprIlVqjDHp/fWZUcbMlwRV2hla67m rpL5p0gOkhRKMAqJcD8Dxsj0O1/XEGFv5EDoEK26NMXM9D0deUzCCUHaWPAdNMuqtk/F I2v2Z9hwse3zd7znMCPnJV1w1Qmw7e6OLXY/r2H6uA2NKQBYHLw/9BDojVZejVy9ONP4 86eMK8np9DpNtaQkM8oG8XLPi/l2O1fDHag5sjoCWJBlzDDkIqQBKsVneKz3vLWOB6OJ Cc/Fo0hA80DebxZMxz1SNxbhaLEnUkVDTSmPB5pDI8v7cpfEWhZdIrpVCtIWIAxyxNjF /HkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=PtQwXmz7BFM1ZNyZb9oikPRn6SnUuujkla1dML1t4+I=; b=Padm8uK0hxCV9qUKh4aXNQp/Ff+DfU1MYkNvNzqWsBkHH3rOJni0UR11GErQ9xrnsj ZWDhtfWtksd4K1Ytx5sL/ozlPCTfcNh9vIxgxXtl2/upA5aj7LwBRz3NaHZOVPgTYHi4 U4QLhlO/vN91/77eKsX8G5RJT1ZySzKlnQJ5EW9Q919RdYeIivpC21E+2uIO187AGzDy TvrnnKUNQKUG1lcysqPmm0tl++9KgFLPUjXa3xJZfujz8hxL0EreUIEx21thGBIMquN2 eiGb+i8CHcundMotNsyHaj7x4zLil1vsPxxRmX0mNjf5Mwr5ZE3Jz+duAW+3NkTwkrMC zlDA== X-Gm-Message-State: AJaThX6DjgqZXKtg8ti15py/PgR12scPGgMCBTUXKuqh1IJ1JjL+upH2 T1OFAW+MvcD0zJ6uUznoyBkRlg== X-Google-Smtp-Source: AGs4zMaeLYQn4i/RMeMC8xT7yvDGjuuVcfFjujDJ1dzHjf010e0AFpWYme0KOAigVEhaEOC8PE+eeg== X-Received: by 10.101.90.193 with SMTP id d1mr11980280pgt.423.1511145506532; Sun, 19 Nov 2017 18:38:26 -0800 (PST) Received: from bish ([204.239.253.216]) by smtp.gmail.com with ESMTPSA id k12sm14546077pgo.5.2017.11.19.18.38.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 Nov 2017 18:38:25 -0800 (PST) Sender: Mark Johnston Date: Sun, 19 Nov 2017 22:38:28 -0500 From: Mark Johnston To: Andre Albsmeier Cc: freebsd-hackers@freebsd.org Subject: Re: gmirror synchronising is very slow due to frequent metadata updates Message-ID: <20171120033828.GA1959@bish> References: <20171119103241.GA20588@voyager> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171119103241.GA20588@voyager> User-Agent: Mutt/1.8.3 (2017-05-23) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 02:38:27 -0000 On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote: > I am running a 2 TB gmirror with two disks. I replaced one disk and the > mirror started to synchronise. I noticed that the sync rate was about > 50 MB/s which is a lot lower than the more than 120 MB/s which are > possible with this hardware. > > The reason for this are the very frequently happening metadata updates > done in sys/geom/mirror/g_mirror.c: > > if (sync->ds_offset_done + (MAXPHYS * 100) < offset) { > /* Update offset_done on every 100 blocks. */ > sync->ds_offset_done = offset; > g_mirror_update_metadata(disk); > > With MAXPHYS being 128kB this would mean that metadata is updated every > 12 MB -- or 4 times a second @ 50 MB/s. > > Out of curiosity I changed this to MAXPHYS * 2000 which means that the > updating is done every 240 MB. Now the disks are synchronising with > approx. 105 MB/s... > > Is there any reason why these metadata updates are done so often and > what are the risks having them more infrequently (as I do now)? I believe that the metadata updates are done only so that it's possible to resume a synchronization after a crash. In particular, ds_offset_done is translated into the sync_offset field in the metadata block of a synchronizing disk. Therefore, there is no real risk in making the updates less frequent; it'll just take marginally more time to complete an interrupted rebuild of a mirror (which would hopefully be a rare event anyway). We should probably decrease the update interval based on the size of a mirror. For mirrors larger than say, 1GB, we might just update the metadata block once per 1% of the synchronization operation's progress.