Date: Sun, 19 Nov 2017 22:38:28 -0500 From: Mark Johnston <markj@FreeBSD.org> To: Andre Albsmeier <andre@fbsd.e4m.org> Cc: freebsd-hackers@freebsd.org Subject: Re: gmirror synchronising is very slow due to frequent metadata updates Message-ID: <20171120033828.GA1959@bish> In-Reply-To: <20171119103241.GA20588@voyager> References: <20171119103241.GA20588@voyager>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote:
> I am running a 2 TB gmirror with two disks. I replaced one disk and the
> mirror started to synchronise. I noticed that the sync rate was about
> 50 MB/s which is a lot lower than the more than 120 MB/s which are
> possible with this hardware.
>
> The reason for this are the very frequently happening metadata updates
> done in sys/geom/mirror/g_mirror.c:
>
> if (sync->ds_offset_done + (MAXPHYS * 100) < offset) {
> /* Update offset_done on every 100 blocks. */
> sync->ds_offset_done = offset;
> g_mirror_update_metadata(disk);
>
> With MAXPHYS being 128kB this would mean that metadata is updated every
> 12 MB -- or 4 times a second @ 50 MB/s.
>
> Out of curiosity I changed this to MAXPHYS * 2000 which means that the
> updating is done every 240 MB. Now the disks are synchronising with
> approx. 105 MB/s...
>
> Is there any reason why these metadata updates are done so often and
> what are the risks having them more infrequently (as I do now)?
I believe that the metadata updates are done only so that it's possible
to resume a synchronization after a crash. In particular, ds_offset_done
is translated into the sync_offset field in the metadata block of a
synchronizing disk. Therefore, there is no real risk in making the
updates less frequent; it'll just take marginally more time to complete
an interrupted rebuild of a mirror (which would hopefully be a rare
event anyway).
We should probably decrease the update interval based on the size of a
mirror. For mirrors larger than say, 1GB, we might just update the
metadata block once per 1% of the synchronization operation's progress.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171120033828.GA1959>