From owner-freebsd-questions@FreeBSD.ORG Sat Jun 3 16:58:36 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7178F16A537 for ; Sat, 3 Jun 2006 16:58:36 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.208.78.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9AC5843D60 for ; Sat, 3 Jun 2006 16:58:25 +0000 (GMT) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.13.6/8.13.6) with ESMTP id k53Gw6Zw036740; Sat, 3 Jun 2006 09:58:06 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.13.6/8.13.6/Submit) id k53Gw66t036739; Sat, 3 Jun 2006 09:58:06 -0700 (PDT) (envelope-from sgk) Date: Sat, 3 Jun 2006 09:58:06 -0700 From: Steve Kargl To: Danial Thom Message-ID: <20060603165806.GA36639@troutmask.apl.washington.edu> References: <20060603153847.GA36267@troutmask.apl.washington.edu> <20060603162842.24827.qmail@web33307.mail.mud.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060603162842.24827.qmail@web33307.mail.mud.yahoo.com> User-Agent: Mutt/1.4.2.1i Cc: freebsd-questions@freebsd.org Subject: Re: Tuning GigE network for cluster computing? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jun 2006 16:58:37 -0000 On Sat, Jun 03, 2006 at 09:28:42AM -0700, Danial Thom wrote: > > wrote: > > > First, I'm running FreeBSD 6.1-stable on a > > small cluster > > with 6 nodes that contain Tyan motheriboards. > > These broads > > have Broadcom GigE NICs that use the bge > > device. > > > > The cluster will be using MPI to possibly shove > > large data > > sets through a GigE switch, so I'm trying to > > determine how > > best to optimize the transfer. > > > > Would DEVICE_POLLING and/or jumbo frames likely > > methods for > > enhanced speeds? > > > Firstly we've been discussing how bad and > inefficient the broadcom controllers are. If you > have a tyan MB with onboard controllers they are > incredibly slow (ie inefficient), as well as > being quirky. If you have a PCI-X slot put in an > intel card. They have built-in interrupt > moderation so you don't have dick around with > polling. A 133Mhz intel controller (make sure > they have the GB chips on them and not the EB) > will use half the cpu of the on-board broadcoms. > If you don't believe try it with one box and > measure the cpu usage before and after. Its worth > the $50. investment in the card, believe me. I'll look into the Intel cards, but unfortunately I'm probably stuck with the onboard broadcom devices for the immediate future. There is one expansion slot (whether its PCI-X, I don't know). I was actually planning to use the slot for infiniband, myrinet, or the 10 GiGE cards that Drew Gallatin has mentioned. > While polling may marginally decrease the cpu > load (depending on how bad FreeBSD 6.1 is on > interrupt overhead), its also going to add > latency to the processing of packets, which is > the opposite of what you want to do. This is the conclusion that I reached in reading i386/conf/NOTES. Thanks for confirming my suspicions. -- Steve