From owner-freebsd-net@FreeBSD.ORG Tue Jan 15 12:33:08 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AADF316A419 for ; Tue, 15 Jan 2008 12:33:08 +0000 (UTC) (envelope-from ianf@clue.co.za) Received: from munchkin.clue.co.za (munchkin.clue.co.za [66.219.59.160]) by mx1.freebsd.org (Postfix) with ESMTP id 9003E13C442 for ; Tue, 15 Jan 2008 12:33:08 +0000 (UTC) (envelope-from ianf@clue.co.za) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=20070313; d=clue.co.za; h=Received:Received:Received:To:cc:From:Subject:In-Reply-To:X-Attribution:Date:Message-Id; b=cdZFh4ef7gB2K8vzP3n15TRkzjtbC+lKbgVGcVEzVIJ+91m9+aT8S1M3CsrFenCvUJ+ke+c1Xt4DQiwwJ6YcwZq49Pl1MKXWqqzf2R37h5Ze1xZq5ZB5jvgUMEVWg5qqmYdG+CnDxk4Mw+ACSsWNfE9uYRg2ozsDPbznERuej1A/sA4GU/tSESHkOEnLicb+xRzApheZ1TDOE4MRz59OoVxeWQrbMi0iRFpT2qj6hRsUXqWYPqx8W0S1GXZj8kGf; Received: from uucp by munchkin.clue.co.za with local-rmail (Exim 4.67) (envelope-from ) id 1JEkcJ-00041U-EV; Tue, 15 Jan 2008 12:10:07 +0000 Received: from dsl-241-38-20.telkomadsl.co.za ([41.241.38.20] helo=clue.co.za) by urchin.clue.co.za with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JEkbS-0004vA-Dc; Tue, 15 Jan 2008 12:09:14 +0000 Received: from localhost ([127.0.0.1] helo=clue.co.za) by clue.co.za with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JEkbQ-0001bz-Of; Tue, 15 Jan 2008 14:09:12 +0200 To: Stefan Lambrev From: Ian FREISLICH In-Reply-To: Message from Stefan Lambrev of "Tue, 15 Jan 2008 10:06:54 +0200." <478C699E.8080105@moneybookers.com> X-Attribution: BOFH Date: Tue, 15 Jan 2008 14:09:12 +0200 Message-Id: Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, thierry@herbelot.com Subject: Re: Linux SMP network performance measurements X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2008 12:33:08 -0000 Stefan Lambrev wrote: > Thierry Herbelot wrote: > > gives some measurements on various tweakings of an SMP machine with > > 4 Xeon processors (it *shows* a nice improvement when using more > > CPUs and more bonded Ethernet interfaces). > > > > Has some the machine (and the time, obviously) to make some of the > > same measurements with the latest FreeBSD versions ? > > I'm planning to test network performance on FreeBSD + bridged > interfaces, very soon, but my test servers are not so powerful as > the server from this page :) Best that I'll have is 1x quad core > processor, 4 port gigabit intel network card and 2GB RAM. I did some testing about a year or two ago and with a recent current of about 2 months ago. I found that generally SMP was a performance regression for the workload I tested - forwarding and filtering. The single most significant contributer to network performance is level 1 cache size. I believe that the extreme cost of mutex acquisition on Intel cpus is the main culprit for SMP network performance regression. Coupled with miniscule L1 cache size on the entire Intel CPU product line gives pretty poor network performance. Forwarding (routing between multiple interfaces) and filtering (ipfw) IIRC with quad Intel e1000 NIC: Dual Intel Xeon 2.8GHz: 240Kpps 12k L1 cache Single Intel Xeon 2.8GHz: 380Kpps 12k L1 cache Core 2 Duo 1.8Ghz: 420kpps 12k L1 cache Single Pentium-M 1.8GHz: 550Kpps 32k L1 cache Dual AMD opteron 2GHz: 890Kpps 64k L1 cache Single AMD opteron 2GHz: 970Kpps 64k L1 cache All these hosts had 255 vlan interfaces with about 3000 routes and about 30000 firewall rules, with a good spread of packets between the interfaces with polling and fastforwarding. I struggled to generate enough packets to load the AMD routers. I was interested in SMP due to additional processing for netflow accounting and packet rate monitoring for DDoS detection and mitigation. I recomend to anyone using FreeBSD as a router or for any serious workload to just plain forget about using Intel CPUs. Ian -- Ian Freislich