Date: Sun, 20 Dec 2009 13:46:54 +0000 From: "Robert N. M. Watson" <rwatson@FreeBSD.org> To: Harti Brandt <harti@freebsd.org> Cc: =?iso-8859-1?Q?Ulrich_Sp=F6rlein?= <uqs@spoerlein.net>, Hans Petter Selasky <hselasky@c2i.net>, freebsd-arch@freebsd.org Subject: Re: network statistics in SMP Message-ID: <5230C2B2-57A5-4982-928A-43756BF8C1C4@FreeBSD.org> In-Reply-To: <20091220134738.V46221@beagle.kn.op.dlr.de> References: <20091215103759.P97203@beagle.kn.op.dlr.de> <200912151313.28326.jhb@freebsd.org> <20091219112711.GR55913@acme.spoerlein.net> <200912191244.17803.hselasky@c2i.net> <20091219232119.L1555@besplex.bde.org> <20091219164818.L1741@beagle.kn.op.dlr.de> <alpine.BSF.2.00.0912201202520.73550@fledge.watson.org> <20091220134738.V46221@beagle.kn.op.dlr.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On 20 Dec 2009, at 13:19, Harti Brandt wrote: > RW>Frequent writes to the same cache line across multiple cores are remarkably > RW>expensive, as they trigger the cache coherency protocol (mileage may vary). > RW>For example, a single non-atomically incremented counter cut performance of > RW>gettimeofday() to 1/6th performance on an 8-core system when parallel system > RW>calls were made across all cores. On many current systems, the cost of an > RW>"atomic" operation is now fairly reasonable as long as the cache line is held > RW>exclusively by the current CPU. However, if we can avoid them that has > RW>value, as we update quite a few global stats on the way through the network > RW>stack. > > Hmm. I'm not sure that gettimeofday() is comparable to forwarding an IP > packet. I would expect, that a single increment is a good percentage of > the entire processing (in terms of numbers of operations) for > gettimeofday(), while in IP forwarding this is somewhere in the noise > floor. In the simples case the packet is acted upon by the receiving > driver, the IP input function, the IP output function and the sending > driver. Not talking about IP filters, firewalls, tunnels, dummynet and > what else. The relative cost of the increment should be much less. But, I > may be wrong of course. If processing is occurring on multiple CPUs -- for example, you are receiving UDP from two ithreads -- then 4-8 cache lines being contended due to stats is a lot. Our goal should be (for 9.0) to avoid having any contended cache lines in the common case when processing independent streams on different CPUs. > I would really like to sort that out before any kind of ABI freeze > happens. Ideally all the statistics would be accessible per sysctl(), have > a version number and have all or most of the required statistics with a > simple way to add new fields without breaking anything. Also the field > sizes (64 vs. 32 bit) should be correct on the kernel - user interface. > > My current feeling after reading this thread is that the low-level kernel > side stuff is probably out of what I could do with the time I have and > would sidetrack me too far from the work on bsnmp. What I would like to do > is to fix the kernel/user interface and let the people that now how to do > it handle the low-level side. > > I would really not like to have to deal with a changing user/kernel > interface in current if we go in several steps with the kernel stuff. I think we should treat the statistics gathering and statistics reporting interfaces as entirely separable problems. Statistics are updated far more frequently than they are queried, so making the query process a bit more expensive (reformatting from an efficient 'update' format to an application-friendly 'report' format) should be fine. One question to think about is whether or not simply cross-CPU summaries are sufficient, or whether we actually also want to be able to directly monitor per-CPU statistics at the IP layer. The former would maintain the status quo making per-CPU behavior purely part of the 'update' step; the latter would change the 'report' format as well. I've been focused primarily on 'update', but at least for my work it would be quite helpful to have per-CPU stats in the 'report' format as well. > I will try to come up with a patch for the kernel/user interface in the > mean time. This will be for 9.x only, obviously. Sounds good -- and the kernel stats capture can "grow into" the full report format as it matures. > Doesn't this help for output only? For the input statistics there still > will be per-ifnet statistics. Most ifnet-layer stats should really be per-queue, both for input and output, which may help. > An interesting question from the SNMP point of view is, what happens to > the statistics if you move around interfaces between vimages. In any case > it would be good if we could abstract from all the complications while > going kernel->userland. At least for now, the interface is effectively recreated when it moves vimage, and only the current vimage is able to monitor it. That could be considered a bug but it might also be a simplifying assumption or even a feature. Likewise, it's worth remembering that the ifnet index space is per-vimage. Robert
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5230C2B2-57A5-4982-928A-43756BF8C1C4>