From owner-freebsd-arch@FreeBSD.ORG Tue Dec 15 17:45:17 2009 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 678A81065670 for ; Tue, 15 Dec 2009 17:45:17 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp3.dlr.de (smtp3.dlr.de [129.247.252.33]) by mx1.freebsd.org (Postfix) with ESMTP id F238F8FC16 for ; Tue, 15 Dec 2009 17:45:16 +0000 (UTC) Received: from beagle.kn.op.dlr.de ([129.247.178.136]) by smtp3.dlr.de over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959); Tue, 15 Dec 2009 18:45:15 +0100 Date: Tue, 15 Dec 2009 18:45:13 +0100 (CET) From: Harti Brandt X-X-Sender: brandt_h@beagle.kn.op.dlr.de To: John Baldwin In-Reply-To: <200912150812.35521.jhb@freebsd.org> Message-ID: <20091215183859.S53283@beagle.kn.op.dlr.de> References: <20091215103759.P97203@beagle.kn.op.dlr.de> <200912150812.35521.jhb@freebsd.org> X-OpenPGP-Key: harti@freebsd.org MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-OriginalArrivalTime: 15 Dec 2009 17:45:15.0071 (UTC) FILETIME=[5BC21CF0:01CA7DAE] Cc: freebsd-arch@freebsd.org Subject: Re: network statistics in SMP X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Harti Brandt List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2009 17:45:17 -0000 On Tue, 15 Dec 2009, John Baldwin wrote: JB>On Tuesday 15 December 2009 4:38:04 am Harti Brandt wrote: JB>> Hi all, JB>> JB>> I'm working on our network statistics (in the context of SNMP) and wonder, JB>> to what extend we want them to be correct. I've re-read part of the past JB>> discussions about 64-bit counters on 32-bit archs and got the impression, JB>> that there are users that would like to have almost correct statistics JB>> (for accounting, for example). If this is the case I wonder whether the JB>> way we do the statistics today is correct. JB>> JB>> Basically all statistics are incremented or added to simply by a += b oder JB>> a++. As I understand, this worked fine in the old days, where you had JB>> spl*() calls at the right places. Nowadays when everything is SMP JB>> shouldn't we use at least atomic operations for this? Also I read that on JB>> architectures where cache coherency is not implemented in hardware even JB>> this does not help (I found a mail from jhb why for the mutex JB>> implementation this is not a problem, but I don't understand what to do JB>> for the += and ++ operations). I failed to find a way, though, to JB>> influence the caching policy (is there a function one can call to JB>> change the policy?). JB> JB>Atomic ops will always work for reliable statistics. However, I believe JB>Robert is working on using per-CPU statistics for TCP, UDP, etc. similar to JB>what we do now for many of the 'cnt' stats (context switches, etc.). For JB>'cnt' each CPU has its own count of stats that are updated using non-atomic JB>ops (since they are CPU local). sysctl handlers then sum up the various per- JB>CPU counts to report global counts to userland. I see. I was also thinking along these lines, but was not sure whether it is worth the trouble. I suppose this does not help to implement 64-bit counters on 32-bit architectures, though, because you cannot read them reliably without locking to sum them up, right? harti