Date: Mon, 17 Dec 2007 23:54:14 -0700 From: Scott Long <scottl@samsco.org> To: Bruce Evans <brde@optusnet.com.au> Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Packet loss every 30.999 seconds Message-ID: <47676E96.4030708@samsco.org> In-Reply-To: <20071218170133.X32807@delplex.bde.org> References: <D50B5BA8-5A80-4370-8F20-6B3A531C2E9B@eng.oar.net> <20071217103936.GR25053@tnn.dglawrence.com> <20071218170133.X32807@delplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Bruce Evans wrote:
> On Mon, 17 Dec 2007, David G Lawrence wrote:
>
>> One more comment on my last email... The patch that I included is not
>> meant as a real fix - it is just a bandaid. The real problem appears to
>> be that a very large number of vnodes (all of them?) are getting synced
>> (i.e. calling ffs_syncvnode()) every time. This should normally only
>> happen for dirty vnodes. I suspect that something is broken with this
>> check:
>>
>> if (vp->v_type == VNON || ((ip->i_flag &
>> (IN_ACCESS | IN_CHANGE | IN_MODIFIED | IN_UPDATE)) == 0 &&
>> vp->v_bufobj.bo_dirty.bv_cnt == 0)) {
>> VI_UNLOCK(vp);
>> continue;
>> }
>
> Isn't it just the O(N) algorithm with N quite large? Under ~5.2, on
> a 2.2GHz A64 UP in 32-bit mode, I see a latency of 3 ms for 17500 vnodes,
> which would be explained by the above (and the VI_LOCK() and loop
> overhead) taking 171 ns per vnode. I would expect it to take more like
> 20 ns per vnode for UP and 60 for SMP.
>
> The comment before this code shows that the problem is known, and says
> that a subroutine call cannot be afforded unless there is work to do,
> but the, the locking accesses look like subroutine calls, have subroutine
> calls in their internals, and take longer than simple subroutine calls
> in the SMP case even when they don't make subroutine calls. (IIRC, on
> A64 a minimal subroutine call takes 4 cycles while a minimal locked
> instructions takes 18 cycles; subroutine calls are only slow when their
> branches are mispredicted.)
>
> Bruce
Right, it's a non-optimal loop when N is very large, and that's a fairly
well understood problem. I think what DG was getting at, though, is
that this massive flush happens every time the syncer runs, which
doesn't seem correct. Sure, maybe you just rsynced 100,000 files 20
seconds ago, so the upcoming flush is going to be expensive. But the
next flush 30 seconds after that shouldn't be just as expensive, yet it
appears to be so. This is further supported by the original poster's
claim that it takes many hours of uptime before the problem becomes
noticeable. If vnodes are never truly getting cleaned, or never getting
their flags cleared so that this loop knows that they are clean, then
it's feasible that they'll accumulate over time, keep on getting flushed
every 30 seconds, keep on bogging down the loop, and so on.
Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47676E96.4030708>