Date: Mon, 20 Nov 2017 06:34:09 +0100 From: Andre Albsmeier <andre@fbsd.e4m.org> To: Mark Johnston <markj@FreeBSD.org> Cc: Andre Albsmeier <andre@fbsd.e4m.org>, freebsd-hackers@freebsd.org Subject: Re: gmirror synchronising is very slow due to frequent metadata updates Message-ID: <20171120053409.GA57536@gate> In-Reply-To: <20171120033828.GA1959@bish> References: <20171119103241.GA20588@voyager> <20171120033828.GA1959@bish>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 19-Nov-2017 at 22:38:28 -0500, Mark Johnston wrote:
> On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote:
> > I am running a 2 TB gmirror with two disks. I replaced one disk and the
> > mirror started to synchronise. I noticed that the sync rate was about
> > 50 MB/s which is a lot lower than the more than 120 MB/s which are
> > possible with this hardware.
> >
> > The reason for this are the very frequently happening metadata updates
> > done in sys/geom/mirror/g_mirror.c:
> >
> > if (sync->ds_offset_done + (MAXPHYS * 100) < offset) {
> > /* Update offset_done on every 100 blocks. */
> > sync->ds_offset_done = offset;
> > g_mirror_update_metadata(disk);
> >
> > With MAXPHYS being 128kB this would mean that metadata is updated every
> > 12 MB -- or 4 times a second @ 50 MB/s.
> >
> > Out of curiosity I changed this to MAXPHYS * 2000 which means that the
> > updating is done every 240 MB. Now the disks are synchronising with
> > approx. 105 MB/s...
> >
> > Is there any reason why these metadata updates are done so often and
> > what are the risks having them more infrequently (as I do now)?
>
> I believe that the metadata updates are done only so that it's possible
> to resume a synchronization after a crash. In particular, ds_offset_done
That's what I thought I had figured out as well when looking at the code.
> is translated into the sync_offset field in the metadata block of a
> synchronizing disk. Therefore, there is no real risk in making the
> updates less frequent; it'll just take marginally more time to complete
> an interrupted rebuild of a mirror (which would hopefully be a rare
> event anyway).
>
> We should probably decrease the update interval based on the size of a
> mirror. For mirrors larger than say, 1GB, we might just update the
> metadata block once per 1% of the synchronization operation's progress.
I think best would be to have it updated every <whatever> seconds.
I think of very fast or very slow drives, or drives which are used
heavily during rebuild (where the effective rebuild speed is quite
low)...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171120053409.GA57536>