From owner-freebsd-performance@FreeBSD.ORG Thu May 26 17:47:50 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2EFE616A41C for ; Thu, 26 May 2005 17:47:50 +0000 (GMT) (envelope-from pfak@telus.net) Received: from priv-edtnes51.telusplanet.net (outbound04.telus.net [199.185.220.223]) by mx1.FreeBSD.org (Postfix) with ESMTP id C95EF43D48 for ; Thu, 26 May 2005 17:47:49 +0000 (GMT) (envelope-from pfak@telus.net) Received: from localhost ([199.185.220.240]) by priv-edtnes51.telusplanet.net (InterMail vM.6.01.04.04 201-2131-118-104-20050224) with ESMTP id <20050526174748.TFKI4603.priv-edtnes51.telusplanet.net@localhost> for ; Thu, 26 May 2005 11:47:48 -0600 Received: from 64.180.103.26 ( [64.180.103.26]) as user a1b26634@192.168.200.1 by webmail.telus.net with HTTP; Thu, 26 May 2005 10:47:48 -0700 Message-ID: <1117129668.42960bc4b751b@webmail.telus.net> Date: Thu, 26 May 2005 10:47:48 -0700 From: Peter Kieser To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1-cvs X-Originating-IP: 64.180.103.26 Subject: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 17:47:50 -0000 Hello guys, I'm not quite sure if this is the right list to address this to, as it's partly a performance problem and partly otherwise. I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of traffic (lot's of small packets, about 45,000 pps)), however I'm currently running into issues where one, or both of the NICs will stop transmitting traffic. When I go onto the machine, and try to ping something I get "No buffer space available" The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no firewall or anything): Here's my /etc/sysctl.conf: net.inet.ip.rtexpire=1800 net.inet.ip.rtminexpire=1800 kern.maxfiles=32768 kern.maxfilesperproc=32768 kern.ipc.somaxconn=32767 net.inet.tcp.sendspace=256000 net.inet.tcp.recvspace=256000 kern.ipc.maxsockbuf=2097152 net.inet.ip.fastforwarding=1 net.inet.tcp.rfc1323=1 net.link.ether.inet.max_age=600 net.inet.tcp.msl=7500 net.inet.ip.fw.dyn_udp_lifetime=10 net.inet.ip.fw.dyn_buckets=1024 And here's my /boot/loader.conf: kern.ipc.maxsockets="163840" kern.maxusers="2048" Is there anythign I'm overlooking that would be causing the machine to lockup like this? --Peter From owner-freebsd-performance@FreeBSD.ORG Thu May 26 17:58:53 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 68CF516A427 for ; Thu, 26 May 2005 17:58:53 +0000 (GMT) (envelope-from gollum123@free.fr) Received: from postfix4-2.free.fr (postfix4-2.free.fr [213.228.0.176]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF77C43D48 for ; Thu, 26 May 2005 17:58:52 +0000 (GMT) (envelope-from gollum123@free.fr) Received: from BIGBOY (tui75-2-82-229-178-102.fbx.proxad.net [82.229.178.102]) by postfix4-2.free.fr (Postfix) with ESMTP id 4CCD5319387; Thu, 26 May 2005 19:58:51 +0200 (CEST) Date: Thu, 26 May 2005 19:58:52 +0200 From: Mathieu CHATEAU X-Mailer: The Bat! (v3.0) Professional X-Priority: 3 (Normal) Message-ID: <406837405.20050526195852@free.fr> To: Peter Kieser In-Reply-To: <1117129668.42960bc4b751b@webmail.telus.net> References: <1117129668.42960bc4b751b@webmail.telus.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mathieu CHATEAU List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 17:58:54 -0000 Hello Peter, in your kernel, try : options VM_KMEM_SIZE_SCALE options DEVICE_POLLING options HZ=1000 You say that you do not have any firewall, so why do you use: net.inet.ip.fw.dyn_udp_lifetime=10 net.inet.ip.fw.dyn_buckets=1024 you may add: kern.polling.enable=1 kern.polling.user_frac=50 best regards, Mathieu CHATEAU Thursday, May 26, 2005, 7:47:48 PM, you wrote: PK> Hello guys, PK> I'm not quite sure if this is the right list to address this to, as it's partly PK> a performance problem and partly otherwise. PK> I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of PK> traffic (lot's of small packets, about 45,000 pps)), however I'm currently PK> running into issues where one, or both of the NICs will stop transmitting PK> traffic. When I go onto the machine, and try to ping something I get "No buffer PK> space available" PK> The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no PK> firewall or anything): PK> Here's my /etc/sysctl.conf: PK> net.inet.ip.rtexpire=1800 PK> net.inet.ip.rtminexpire=1800 PK> kern.maxfiles=32768 PK> kern.maxfilesperproc=32768 PK> kern.ipc.somaxconn=32767 PK> net.inet.tcp.sendspace=256000 PK> net.inet.tcp.recvspace=256000 PK> kern.ipc.maxsockbuf=2097152 PK> net.inet.ip.fastforwarding=1 PK> net.inet.tcp.rfc1323=1 PK> net.link.ether.inet.max_age=600 PK> net.inet.tcp.msl=7500 PK> net.inet.ip.fw.dyn_udp_lifetime=10 PK> net.inet.ip.fw.dyn_buckets=1024 PK> And here's my /boot/loader.conf: PK> kern.ipc.maxsockets="163840" PK> kern.maxusers="2048" PK> Is there anythign I'm overlooking that would be causing the machine to lockup PK> like this? PK> --Peter PK> _______________________________________________ PK> freebsd-performance@freebsd.org mailing list PK> http://lists.freebsd.org/mailman/listinfo/freebsd-performance PK> To unsubscribe, send any mail to PK> "freebsd-performance-unsubscribe@freebsd.org" -- Best regards, Mathieu mailto:gollum123@free.fr From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:16:39 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D109716A41C for ; Thu, 26 May 2005 18:16:39 +0000 (GMT) (envelope-from roo@seattlefenix.net) Received: from strawberry.seattlefenix.net (dsl231-061-109.sea1.dsl.speakeasy.net [216.231.61.109]) by mx1.FreeBSD.org (Postfix) with ESMTP id 904C043D49 for ; Thu, 26 May 2005 18:16:39 +0000 (GMT) (envelope-from roo@seattlefenix.net) Received: from strawberry.seattlefenix.net (localhost [127.0.0.1]) by strawberry.seattlefenix.net (Postfix) with ESMTP id B9E251B5A6; Thu, 26 May 2005 11:17:47 -0700 (PDT) Received: (from roo@localhost) by strawberry.seattlefenix.net (8.12.11/8.12.11/Submit) id j4QIHlp7058518; Thu, 26 May 2005 11:17:47 -0700 (PDT) (envelope-from roo) Date: Thu, 26 May 2005 11:17:47 -0700 From: Benjamin Krueger To: Peter Kieser Message-ID: <20050526181747.GF62730@strawberry.seattlefenix.net> References: <1117129668.42960bc4b751b@webmail.telus.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1117129668.42960bc4b751b@webmail.telus.net> User-Agent: Mutt/1.4.2.1i Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Benjamin Krueger List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:16:39 -0000 * Peter Kieser (pfak@telus.net) [050526 10:49]: > Hello guys, > > I'm not quite sure if this is the right list to address this to, as it's partly > a performance problem and partly otherwise. > > I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of > traffic (lot's of small packets, about 45,000 pps)), however I'm currently > running into issues where one, or both of the NICs will stop transmitting > traffic. When I go onto the machine, and try to ping something I get "No buffer > space available" > > The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no > firewall or anything): > > Here's my /etc/sysctl.conf: > > net.inet.ip.rtexpire=1800 > net.inet.ip.rtminexpire=1800 > kern.maxfiles=32768 > kern.maxfilesperproc=32768 > kern.ipc.somaxconn=32767 > net.inet.tcp.sendspace=256000 > net.inet.tcp.recvspace=256000 > kern.ipc.maxsockbuf=2097152 > net.inet.ip.fastforwarding=1 > net.inet.tcp.rfc1323=1 > net.link.ether.inet.max_age=600 > net.inet.tcp.msl=7500 > net.inet.ip.fw.dyn_udp_lifetime=10 > net.inet.ip.fw.dyn_buckets=1024 > > And here's my /boot/loader.conf: > > kern.ipc.maxsockets="163840" > kern.maxusers="2048" > > Is there anythign I'm overlooking that would be causing the machine to lockup > like this? > > --Peter What does 'netstat -m' say? Is there any reason you explicitely set maxusers instead of letting the kernel auto-tune the setting? I would watch your mbuf usage and when you find your average and peak usage, configure your mbuf allocation at boot and see if that helps. I'd also consider letting FreeBSD auto-tune your MAXUSERS. -- Benjamin Krueger SysAdmin, CarDomain Network 92 Toyota Turbo MR2 78 Datsun B-210 91 freakin Geo Prizm From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:27:52 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1E3EA16A41C for ; Thu, 26 May 2005 18:27:52 +0000 (GMT) (envelope-from pfak@telus.net) Received: from priv-edtnes56.telusplanet.net (outbound01.telus.net [199.185.220.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id AD06443D1D for ; Thu, 26 May 2005 18:27:51 +0000 (GMT) (envelope-from pfak@telus.net) Received: from localhost ([199.185.220.240]) by priv-edtnes56.telusplanet.net (InterMail vM.6.01.04.04 201-2131-118-104-20050224) with ESMTP id <20050526182751.DSDA16662.priv-edtnes56.telusplanet.net@localhost>; Thu, 26 May 2005 12:27:51 -0600 Received: from 64.180.103.26 ( [64.180.103.26]) as user a1b26634@192.168.200.1 by webmail.telus.net with HTTP; Thu, 26 May 2005 11:27:50 -0700 Message-ID: <1117132070.42961526e0922@webmail.telus.net> Date: Thu, 26 May 2005 11:27:50 -0700 From: Peter Kieser To: Benjamin Krueger References: <1117129668.42960bc4b751b@webmail.telus.net> <20050526181747.GF62730@strawberry.seattlefenix.net> In-Reply-To: <20050526181747.GF62730@strawberry.seattlefenix.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1-cvs X-Originating-IP: 64.180.103.26 Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:27:52 -0000 -- Peter Kieser pfak@telus.net Quoting Benjamin Krueger : > * Peter Kieser (pfak@telus.net) [050526 10:49]: > > Hello guys, > > > > I'm not quite sure if this is the right list to address this to, as it's > partly > > a performance problem and partly otherwise. > > > > I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of > > traffic (lot's of small packets, about 45,000 pps)), however I'm currently > > running into issues where one, or both of the NICs will stop transmitting > > traffic. When I go onto the machine, and try to ping something I get "No > buffer > > space available" > > > > The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no > > firewall or anything): > > > > Here's my /etc/sysctl.conf: > > > > net.inet.ip.rtexpire=1800 > > net.inet.ip.rtminexpire=1800 > > kern.maxfiles=32768 > > kern.maxfilesperproc=32768 > > kern.ipc.somaxconn=32767 > > net.inet.tcp.sendspace=256000 > > net.inet.tcp.recvspace=256000 > > kern.ipc.maxsockbuf=2097152 > > net.inet.ip.fastforwarding=1 > > net.inet.tcp.rfc1323=1 > > net.link.ether.inet.max_age=600 > > net.inet.tcp.msl=7500 > > net.inet.ip.fw.dyn_udp_lifetime=10 > > net.inet.ip.fw.dyn_buckets=1024 > > > > And here's my /boot/loader.conf: > > > > kern.ipc.maxsockets="163840" > > kern.maxusers="2048" > > > > Is there anythign I'm overlooking that would be causing the machine to > lockup > > like this? > > > > --Peter > > What does 'netstat -m' say? Is there any reason you explicitely set maxusers > instead of > letting the kernel auto-tune the setting? I would watch your mbuf usage and > when you > find your average and peak usage, configure your mbuf allocation at boot and > see if that > helps. I'd also consider letting FreeBSD auto-tune your MAXUSERS. > > -- > Benjamin Krueger > SysAdmin, CarDomain Network > 92 Toyota Turbo MR2 > 78 Datsun B-210 > 91 freakin Geo Prizm > I forgot to mention that mbuf's aren't exhausted, here's the reporting on mbuf's when the machine's NICs fail to work: $ netstat -m 450 mbufs in use 448/132096 mbuf clusters in use (current/max) 0/2/33280 sfbufs in use (current/peak/max) 1008 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:44:36 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 820CB16A41C for ; Thu, 26 May 2005 18:44:36 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 51DF543D1F for ; Thu, 26 May 2005 18:44:36 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from mac.com (smtpin01-en2 [10.13.10.146]) by smtpout.mac.com (Xserve/8.12.11/smtpout06/MantshX 4.0) with ESMTP id j4QIiZtQ016040; Thu, 26 May 2005 11:44:35 -0700 (PDT) Received: from [192.168.1.6] (pool-68-161-53-96.ny325.east.verizon.net [68.161.53.96]) (authenticated bits=0) by mac.com (Xserve/smtpin01/MantshX 4.0) with ESMTP id j4QIiYL4001817; Thu, 26 May 2005 11:44:35 -0700 (PDT) In-Reply-To: <1117129668.42960bc4b751b@webmail.telus.net> References: <1117129668.42960bc4b751b@webmail.telus.net> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> Content-Transfer-Encoding: 7bit From: Charles Swiger Date: Thu, 26 May 2005 14:44:29 -0400 To: Peter Kieser X-Mailer: Apple Mail (2.730) Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:44:36 -0000 On May 26, 2005, at 1:47 PM, Peter Kieser wrote: > The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE > installation (no > firewall or anything): > > Here's my /etc/sysctl.conf: > > net.inet.ip.rtexpire=1800 > net.inet.ip.rtminexpire=1800 > kern.maxfiles=32768 > kern.maxfilesperproc=32768 > kern.ipc.somaxconn=32767 > net.inet.tcp.sendspace=256000 > net.inet.tcp.recvspace=256000 > kern.ipc.maxsockbuf=2097152 > net.inet.ip.fastforwarding=1 > net.inet.tcp.rfc1323=1 > net.link.ether.inet.max_age=600 > net.inet.tcp.msl=7500 > net.inet.ip.fw.dyn_udp_lifetime=10 > net.inet.ip.fw.dyn_buckets=1024 > > And here's my /boot/loader.conf: > > kern.ipc.maxsockets="163840" > kern.maxusers="2048" > > Is there anythign I'm overlooking that would be causing the machine > to lockup > like this? Your settings are tuned so high that you may be running out of KVA memory. I bet the system would be happier if you let maxusers autotune itself (and then maybe adjust it by a factor of 2 from there), and if you reduced kern.maxfilesperproc to 2048 or less, and reduced the TCP sendspace to 65K. [ As a last resort, you might even try nixing the fastforwarding option. It's well-suited for your task, and you do want it on if it is working right, but maybe try running without it for a test. ] -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:52:53 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 302AC16A41C for ; Thu, 26 May 2005 18:52:53 +0000 (GMT) (envelope-from pfak@telus.net) Received: from priv-edtnes51.telusplanet.net (outbound04.telus.net [199.185.220.223]) by mx1.FreeBSD.org (Postfix) with ESMTP id BEAE743D49 for ; Thu, 26 May 2005 18:52:52 +0000 (GMT) (envelope-from pfak@telus.net) Received: from localhost ([199.185.220.240]) by priv-edtnes51.telusplanet.net (InterMail vM.6.01.04.04 201-2131-118-104-20050224) with ESMTP id <20050526185251.XTEK4603.priv-edtnes51.telusplanet.net@localhost>; Thu, 26 May 2005 12:52:51 -0600 Received: from 64.180.103.26 ( [64.180.103.26]) as user a1b26634@192.168.200.1 by webmail.telus.net with HTTP; Thu, 26 May 2005 11:52:51 -0700 Message-ID: <1117133571.42961b03ca6fd@webmail.telus.net> Date: Thu, 26 May 2005 11:52:51 -0700 From: Peter Kieser To: Charles Swiger References: <1117129668.42960bc4b751b@webmail.telus.net> <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> In-Reply-To: <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1-cvs X-Originating-IP: 64.180.103.26 Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:52:53 -0000 Quoting Charles Swiger : > On May 26, 2005, at 1:47 PM, Peter Kieser wrote: > > The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE > > installation (no > > firewall or anything): > > > > Here's my /etc/sysctl.conf: > > > > net.inet.ip.rtexpire=1800 > > net.inet.ip.rtminexpire=1800 > > kern.maxfiles=32768 > > kern.maxfilesperproc=32768 > > kern.ipc.somaxconn=32767 > > net.inet.tcp.sendspace=256000 > > net.inet.tcp.recvspace=256000 > > kern.ipc.maxsockbuf=2097152 > > net.inet.ip.fastforwarding=1 > > net.inet.tcp.rfc1323=1 > > net.link.ether.inet.max_age=600 > > net.inet.tcp.msl=7500 > > net.inet.ip.fw.dyn_udp_lifetime=10 > > net.inet.ip.fw.dyn_buckets=1024 > > > > And here's my /boot/loader.conf: > > > > kern.ipc.maxsockets="163840" > > kern.maxusers="2048" > > > > Is there anythign I'm overlooking that would be causing the machine > > to lockup > > like this? > > Your settings are tuned so high that you may be running out of KVA > memory. I bet the system would be happier if you let maxusers > autotune itself (and then maybe adjust it by a factor of 2 from > there), and if you reduced kern.maxfilesperproc to 2048 or less, and > reduced the TCP sendspace to 65K. > > [ As a last resort, you might even try nixing the fastforwarding > option. It's well-suited for your task, and you do want it on if it > is working right, but maybe try running without it for a test. ] > > -- > -Chuck > > I've tried without fast forwarding, I've tried without the TCP sendspace as well as reducing it to 65K, I let maxusers auto tune itself and I've even tried uping the KVA space. I'm at a loss, what would be the ideal sysctl's/loader.conf for a router thats doing a fair amount of traffic? From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:57:02 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9AF2916A41C for ; Thu, 26 May 2005 18:57:02 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from smtp103.rog.mail.re2.yahoo.com (smtp103.rog.mail.re2.yahoo.com [206.190.36.81]) by mx1.FreeBSD.org (Postfix) with SMTP id 1CC4E43D1D for ; Thu, 26 May 2005 18:57:01 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from unknown (HELO 172.16.0.1) (mikej@69.193.222.195 with login) by smtp103.rog.mail.re2.yahoo.com with SMTP; 26 May 2005 18:57:00 -0000 Received: from 172.16.0.199 (SquirrelMail authenticated user mikej) by 172.16.0.1 with HTTP; Thu, 26 May 2005 14:56:58 -0400 (EDT) Message-ID: <2416.172.16.0.199.1117133818.squirrel@172.16.0.1> In-Reply-To: <1117133571.42961b03ca6fd@webmail.telus.net> References: <1117129668.42960bc4b751b@webmail.telus.net> <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> <1117133571.42961b03ca6fd@webmail.telus.net> Date: Thu, 26 May 2005 14:56:58 -0400 (EDT) From: "Mike Jakubik" To: "Peter Kieser" User-Agent: SquirrelMail/1.5.1 [CVS] MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:57:02 -0000 On Thu, May 26, 2005 2:52 pm, Peter Kieser said: > I've tried without fast forwarding, I've tried without the TCP sendspace > as well as reducing it to 65K, I let maxusers auto tune itself and I've > even tried uping the KVA space. > > I'm at a loss, what would be the ideal sysctl's/loader.conf for a router > thats doing a fair amount of traffic? Have you tried different network cards? From owner-freebsd-performance@FreeBSD.ORG Thu May 26 18:58:33 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8A4B416A41C for ; Thu, 26 May 2005 18:58:33 +0000 (GMT) (envelope-from pfak@telus.net) Received: from priv-edtnes56.telusplanet.net (outbound01.telus.net [199.185.220.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25AAC43D1D for ; Thu, 26 May 2005 18:58:32 +0000 (GMT) (envelope-from pfak@telus.net) Received: from localhost ([199.185.220.240]) by priv-edtnes56.telusplanet.net (InterMail vM.6.01.04.04 201-2131-118-104-20050224) with ESMTP id <20050526185832.FWZR16662.priv-edtnes56.telusplanet.net@localhost>; Thu, 26 May 2005 12:58:32 -0600 Received: from 64.180.103.26 ( [64.180.103.26]) as user a1b26634@192.168.200.1 by webmail.telus.net with HTTP; Thu, 26 May 2005 11:58:32 -0700 Message-ID: <1117133912.42961c587e09d@webmail.telus.net> Date: Thu, 26 May 2005 11:58:32 -0700 From: Peter Kieser To: Mike Jakubik References: <1117129668.42960bc4b751b@webmail.telus.net> <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> <1117133571.42961b03ca6fd@webmail.telus.net> <2416.172.16.0.199.1117133818.squirrel@172.16.0.1> In-Reply-To: <2416.172.16.0.199.1117133818.squirrel@172.16.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1-cvs X-Originating-IP: 64.180.103.26 Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 18:58:33 -0000 Quoting Mike Jakubik : > On Thu, May 26, 2005 2:52 pm, Peter Kieser said: > > > I've tried without fast forwarding, I've tried without the TCP sendspace > > as well as reducing it to 65K, I let maxusers auto tune itself and I've > > even tried uping the KVA space. > > > > I'm at a loss, what would be the ideal sysctl's/loader.conf for a router > > thats doing a fair amount of traffic? > > Have you tried different network cards? > > > Yes, I have -- some fxp and ed cards. From owner-freebsd-performance@FreeBSD.ORG Thu May 26 19:07:45 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 288AD16A41C for ; Thu, 26 May 2005 19:07:45 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBCA343D49 for ; Thu, 26 May 2005 19:07:44 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from mac.com (smtpin07-en2 [10.13.10.152]) by smtpout.mac.com (Xserve/8.12.11/smtpout08/MantshX 4.0) with ESMTP id j4QJ7iDB013414; Thu, 26 May 2005 12:07:44 -0700 (PDT) Received: from [192.168.1.6] (pool-68-161-53-96.ny325.east.verizon.net [68.161.53.96]) (authenticated bits=0) by mac.com (Xserve/smtpin07/MantshX 4.0) with ESMTP id j4QJ7gn8007218; Thu, 26 May 2005 12:07:43 -0700 (PDT) In-Reply-To: <1117133571.42961b03ca6fd@webmail.telus.net> References: <1117129668.42960bc4b751b@webmail.telus.net> <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> <1117133571.42961b03ca6fd@webmail.telus.net> Mime-Version: 1.0 (Apple Message framework v730) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <25049451-CF59-48B6-A3ED-B154A36F35D1@mac.com> Content-Transfer-Encoding: 7bit From: Charles Swiger Date: Thu, 26 May 2005 15:07:43 -0400 To: Peter Kieser X-Mailer: Apple Mail (2.730) Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 19:07:45 -0000 On May 26, 2005, at 2:52 PM, Peter Kieser wrote: > I've tried without fast forwarding, I've tried without the TCP > sendspace as well > as reducing it to 65K, I let maxusers auto tune itself and I've > even tried uping > the KVA space. OK. You didn't mention whether this helped or made no difference. :-) > I'm at a loss, what would be the ideal sysctl's/loader.conf for a > router thats > doing a fair amount of traffic? Using FreeBSD as a router is a very common task: you generally don't need to tune anything to have it work pretty well, if the hardware underneath is OK. Can you confirm that you're not losing the connections because a NIC freaks out and drops the carrier, or something along those lines? What does top look like, and "netstat -m" and "netstat -i" look like during a lockup...? Anything in the logs or dmesg? -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Thu May 26 19:08:36 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 54DC116A41C for ; Thu, 26 May 2005 19:08:36 +0000 (GMT) (envelope-from hroi@ngdc.net) Received: from quentin.asdf.dk (quentin.asdf.dk [217.116.240.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id E442B43D53 for ; Thu, 26 May 2005 19:08:35 +0000 (GMT) (envelope-from hroi@ngdc.net) Received: from localhost (localhost [127.0.0.1]) by quentin.asdf.dk (Postfix) with ESMTP id 0BF75B81E for ; Thu, 26 May 2005 19:08:32 +0000 (UTC) Received: from quentin.asdf.dk ([127.0.0.1]) by localhost (quentin.asdf.dk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 92855-02 for ; Thu, 26 May 2005 19:08:26 +0000 (UTC) Received: from [10.12.1.133] (city-office.ngdc.dk [195.190.153.193]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by quentin.asdf.dk (Postfix) with ESMTP id 39B3FB81C for ; Thu, 26 May 2005 19:08:26 +0000 (UTC) Message-ID: <42961EA9.2010700@ngdc.net> Date: Thu, 26 May 2005 21:08:25 +0200 From: Hroi Sigurdsson Organization: NetGroup Data Center A/S User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <1117129668.42960bc4b751b@webmail.telus.net> <9827650D-A61E-461A-85FC-A45413B42FEA@mac.com> <1117133571.42961b03ca6fd@webmail.telus.net> <2416.172.16.0.199.1117133818.squirrel@172.16.0.1> <1117133912.42961c587e09d@webmail.telus.net> In-Reply-To: <1117133912.42961c587e09d@webmail.telus.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: amavisd-new at quentin.asdf.dk Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 19:08:36 -0000 Peter Kieser wrote: > Yes, I have -- some fxp and ed cards. Have you tried without "optimizing"? Most of your optimizations are for socket and TCP buffers, only servers need that. A router just forwards packets. What does netstat -m say and the "CPU Usage:" column in top? Check the interrupt load. Try if polling(4) helps. -- Hroi Sigurdsson · NetGroup DataCenter A/S St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1544 · Fax: +45 7025 2687 From owner-freebsd-performance@FreeBSD.ORG Thu May 26 21:15:16 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B51F216A41C for ; Thu, 26 May 2005 21:15:16 +0000 (GMT) (envelope-from tec@mega.net.br) Received: from msrv.matik.com.br (msrv.matik.com.br [200.152.83.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id A6C2943D1F for ; Thu, 26 May 2005 21:15:14 +0000 (GMT) (envelope-from tec@mega.net.br) Received: from [200.152.82.190] ([200.152.82.190]) by msrv.matik.com.br (8.13.1/8.12.11) with ESMTP id j4QLFdF5045253 for ; Thu, 26 May 2005 18:15:39 -0300 (BRST) (envelope-from tec@mega.net.br) From: "T.D.Wipnet (H.M.)" To: freebsd-performance@freebsd.org Date: Thu, 26 May 2005 18:14:59 -0300 User-Agent: KMail/1.8 References: <1117129668.42960bc4b751b@webmail.telus.net> In-Reply-To: <1117129668.42960bc4b751b@webmail.telus.net> Organization: Wipnet Telecom Ltda MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200505261815.02909.tec@mega.net.br> X-Filter-Version: 1.11a (msrv.matik.com.br) X-Virus-Scanned: ClamAV version 0.83, clamav-milter version 0.83 on msrv.matik.com.br X-Virus-Status: Clean Subject: Re: Performance/lockup issues with FreeBSD as a router X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: tec@mega.net.br List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2005 21:15:16 -0000 On Thursday 26 May 2005 14:47, Peter Kieser wrote: > however I'm currently running into issues where one, or both of the > NICs will stop transmitting traffic. When I go onto the machine, > and try to ping something I get "No buffer space available" > stopping transmitting anything or only your ping tries? > net.inet.ip.rtexpire=1800 > net.inet.ip.rtminexpire=1800 > kern.maxfiles=32768 > kern.maxfilesperproc=32768 > kern.ipc.somaxconn=32767 > net.inet.tcp.sendspace=256000 > net.inet.tcp.recvspace=256000 > kern.ipc.maxsockbuf=2097152 > net.inet.ip.fastforwarding=1 > net.inet.tcp.rfc1323=1 > net.link.ether.inet.max_age=600 > net.inet.tcp.msl=7500 > net.inet.ip.fw.dyn_udp_lifetime=10 > net.inet.ip.fw.dyn_buckets=1024 > > And here's my /boot/loader.conf: > > kern.ipc.maxsockets="163840" > kern.maxusers="2048" > if your pc is really only a router you probably have a lot of unessessary stuff in here, your extremly high settings for maxuser, maxfiles and somaxcons is somewhat useless here and could slow your machine down, anyway, this is not a p-III 600 with 128MB of ram isn't it? "no buffer space available" appears often when the buffer is used so that means that if your connections are bad the buffer gets filled because if your connections are good why the buffer should be used? - then it goes straight out ... I guess that probably your Nics, cables, hubs or even the "where it goes" may have a problem Hans -- WIPNET Telecom Ltda. http://wip.mega.net.br http://wip.mega.net.br/tec.asc From owner-freebsd-performance@FreeBSD.ORG Fri May 27 01:44:29 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5241E16A41C for ; Fri, 27 May 2005 01:44:29 +0000 (GMT) (envelope-from pfak@telus.net) Received: from priv-edtnes46.telusplanet.net (defout.telus.net [199.185.220.240]) by mx1.FreeBSD.org (Postfix) with ESMTP id E77FA43D1D for ; Fri, 27 May 2005 01:44:28 +0000 (GMT) (envelope-from pfak@telus.net) Received: from [192.168.1.151] (really [64.180.103.26]) by priv-edtnes46.telusplanet.net (InterMail vM.6.01.04.04 201-2131-118-104-20050224) with ESMTP id <20050527014427.QDTV22705.priv-edtnes46.telusplanet.net@[192.168.1.151]> for ; Thu, 26 May 2005 19:44:27 -0600 Message-ID: <42967B7E.1020607@telus.net> Date: Thu, 26 May 2005 18:44:30 -0700 From: Peter Kieser User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <1117129668.42960bc4b751b@webmail.telus.net> In-Reply-To: <1117129668.42960bc4b751b@webmail.telus.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Performance/lockup issues with FreeBSD as a router (followup) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2005 01:44:29 -0000 Hello, I fixed the problem. It appears that the kernel was running out of memory to allocate the network sockets, I took your guys suggestions (along with the kernel configurations and such), and the problem is now resolved. I also replaced the physical machine, but am using the same NICs for even better performance. If anyone is interested in the exact configuration I've used, I'll be happy to post. --Peter Peter Kieser wrote: >Hello guys, > >I'm not quite sure if this is the right list to address this to, as it's partly >a performance problem and partly otherwise. > >I have a FreeBSD machine acting as a router (doing approx. 15-25Mbit/s of >traffic (lot's of small packets, about 45,000 pps)), however I'm currently >running into issues where one, or both of the NICs will stop transmitting >traffic. When I go onto the machine, and try to ping something I get "No buffer >space available" > >The nics are if_dc, this is a stock FreeBSD 5.4-RELEASE installation (no >firewall or anything): > >Here's my /etc/sysctl.conf: > >net.inet.ip.rtexpire=1800 >net.inet.ip.rtminexpire=1800 >kern.maxfiles=32768 >kern.maxfilesperproc=32768 >kern.ipc.somaxconn=32767 >net.inet.tcp.sendspace=256000 >net.inet.tcp.recvspace=256000 >kern.ipc.maxsockbuf=2097152 >net.inet.ip.fastforwarding=1 >net.inet.tcp.rfc1323=1 >net.link.ether.inet.max_age=600 >net.inet.tcp.msl=7500 >net.inet.ip.fw.dyn_udp_lifetime=10 >net.inet.ip.fw.dyn_buckets=1024 > >And here's my /boot/loader.conf: > >kern.ipc.maxsockets="163840" >kern.maxusers="2048" > >Is there anythign I'm overlooking that would be causing the machine to lockup >like this? > >--Peter > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > From owner-freebsd-performance@FreeBSD.ORG Fri May 27 12:44:04 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7D78F16A41F for ; Fri, 27 May 2005 12:44:04 +0000 (GMT) (envelope-from ivoras@fer.hr) Received: from pinus.cc.fer.hr (pinus.cc.fer.hr [161.53.73.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id B949F43D48 for ; Fri, 27 May 2005 12:44:03 +0000 (GMT) (envelope-from ivoras@fer.hr) Received: from [161.53.72.113] (lara.cc.fer.hr [161.53.72.113]) by pinus.cc.fer.hr (8.12.2/8.12.2) with ESMTP id j4RCkKpq006294; Fri, 27 May 2005 14:46:20 +0200 (MEST) Message-ID: <429715D9.6090204@fer.hr> Date: Fri, 27 May 2005 14:43:05 +0200 From: Ivan Voras User-Agent: Mozilla Thunderbird 1.0 (X11/20041213) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Kieser References: <1117129668.42960bc4b751b@webmail.telus.net> <42967B7E.1020607@telus.net> In-Reply-To: <42967B7E.1020607@telus.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: Performance/lockup issues with FreeBSD as a router (followup) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2005 12:44:04 -0000 Peter Kieser wrote: > Hello, > > If anyone is interested in the exact configuration I've used, I'll be > happy to post. Please do post, I'll need to setup a similar configuration in a few weeks so it might be useful :) (Or maybe just post the differences from your initial configuration) From owner-freebsd-performance@FreeBSD.ORG Fri May 27 13:50:38 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22E0516A41C for ; Fri, 27 May 2005 13:50:38 +0000 (GMT) (envelope-from uli.ruffler@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id D30D343D49 for ; Fri, 27 May 2005 13:50:37 +0000 (GMT) (envelope-from uli.ruffler@gmail.com) Received: by zproxy.gmail.com with SMTP id 16so724882nzp for ; Fri, 27 May 2005 06:50:37 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=U2435sV3yOe+N+Vt4y6g8vIpdzwNhNQyAHuhziZaa3kuZp1teK5b1uUdLK9gGiQrNNkrKkBzWVsRbYFPwR/xK87a+7abjINZl1YpukmEl90TMXfeD04J0yjHg9BlZvlvyTUI6N7zHXHPy3IB5nouv1n8C4bVeuwnfT1y15iJ+ng= Received: by 10.36.34.18 with SMTP id h18mr988080nzh; Fri, 27 May 2005 06:50:37 -0700 (PDT) Received: by 10.36.46.10 with HTTP; Fri, 27 May 2005 06:50:37 -0700 (PDT) Message-ID: Date: Fri, 27 May 2005 15:50:37 +0200 From: Uli Ruffler To: freebsd-performance@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Subject: Boot process hangs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Uli Ruffler List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2005 13:50:38 -0000 Hi, I have freeBSD 5.3-RELEASE-p10 installed on my notebook. Since I have compiled a new kernel with "device atapicam" the boot process hangs after detecting the acd0 device for about 30 seconds. During this time the system is not reacting on anything, i. e. key strokes. Can you please help me? Regards Uli From owner-freebsd-performance@FreeBSD.ORG Fri May 27 15:19:59 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5488B16A41C for ; Fri, 27 May 2005 15:19:59 +0000 (GMT) (envelope-from mike@sentex.net) Received: from pumice6.sentex.ca (pumice6.sentex.ca [64.7.153.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E7A3443D1D for ; Fri, 27 May 2005 15:19:58 +0000 (GMT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by pumice6.sentex.ca (8.13.3/8.13.3) with ESMTP id j4RFJw3O084877 for ; Fri, 27 May 2005 11:19:58 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.13.3) with ESMTP id j4RFJq2t098042 for ; Fri, 27 May 2005 11:19:52 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050527111809.0704d4e0@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Fri, 27 May 2005 11:20:22 -0400 To: freebsd-performance@freebsd.org From: Mike Tancsa Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Scanned-By: MIMEDefang 2.51 on 64.7.153.21 Subject: FreeBSD router performance and PCI-Express NICs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2005 15:19:59 -0000 While on the topic, has anyone tried http://www.dlink.com/products/?pid=406 on FreeBSD ? ---Mike -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike From owner-freebsd-performance@FreeBSD.ORG Sat May 28 03:17:33 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0656816A41C for ; Sat, 28 May 2005 03:17:33 +0000 (GMT) (envelope-from mv@roq.com) Received: from p4.roq.com (ns1.ecoms.com [207.44.130.137]) by mx1.FreeBSD.org (Postfix) with ESMTP id C627D43D1D for ; Sat, 28 May 2005 03:17:30 +0000 (GMT) (envelope-from mv@roq.com) Received: from p4.roq.com (localhost.roq.com [127.0.0.1]) by p4.roq.com (Postfix) with ESMTP id 511F64D2EC for ; Sat, 28 May 2005 03:17:53 +0000 (GMT) Received: from [10.0.1.8] (adsl-143-85.swiftdsl.com.au [218.214.143.85]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by p4.roq.com (Postfix) with ESMTP id ADBFC4D326 for ; Sat, 28 May 2005 03:17:52 +0000 (GMT) Message-ID: <4297E2C4.1030505@roq.com> Date: Sat, 28 May 2005 13:17:24 +1000 From: Michael VInce User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.7.8) Gecko/20050524 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Subject: High usage of mbufs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 May 2005 03:17:33 -0000 Hi guys On one of my web servers I have a really high usage of mbuf clusters in use on a web server that does about 3million hits a day. 4294914731/262144 mbuf clusters in use (current/max) Does any one know why this is? I was thought it may of been hit by a small syn flood that went un noticed and that the current mbuf stat is failing to update its real usage, but the number does move every time I run netstat -m This is a dual P4 machine with 4gigs of ram that was setup before 5.4 was release, so it could be considered 5.3 stable. 5.4-PRERELEASE #0: Mon Feb 28 20:01:43 EST 2005 netstat -m 352955 mbufs in use 4294914731/262144 mbuf clusters in use (current/max) 0/27/6656 sfbufs in use (current/peak/max) 4177412 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 415 requests for I/O initiated by sendfile 2300 calls to protocol drain routines From owner-freebsd-performance@FreeBSD.ORG Sat May 28 17:45:33 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1151516A41C for ; Sat, 28 May 2005 17:45:33 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: from mail.geeks.org (jacobs.geeks.org [204.153.247.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id CCE5543D48 for ; Sat, 28 May 2005 17:45:32 +0000 (GMT) (envelope-from drechsau@Geeks.ORG) Received: by mail.geeks.org (Postfix, from userid 1000) id 0B6F0159029; Sat, 28 May 2005 12:45:32 -0500 (CDT) Date: Sat, 28 May 2005 12:45:31 -0500 From: Mike Horwath To: Michael VInce Message-ID: <20050528174531.GA38323@octanews.net> References: <4297E2C4.1030505@roq.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4297E2C4.1030505@roq.com> X-PGP-Fingerprint: D8 24 CC E6 47 5F E4 60 BF B7 6E FA BF C7 6E C5 X-GPG-Fingerprint: 6A89 E78A B8B1 69D9 8CDB E966 4A5A C3F9 A1B0 C381 User-Agent: Mutt/1.5.9i Cc: freebsd-performance@freebsd.org Subject: Re: High usage of mbufs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 May 2005 17:45:33 -0000 On Sat, May 28, 2005 at 01:17:24PM +1000, Michael VInce wrote: > Hi guys > On one of my web servers I have a really high usage of mbuf clusters in > use on a web server that does about 3million hits a day. > 4294914731/262144 mbuf clusters in use (current/max) > Does any one know why this is? I was thought it may of been hit by a > small syn flood that went un noticed and that the current mbuf stat is > failing to update its real usage, but the number does move every time I > run netstat -m > This is a dual P4 machine with 4gigs of ram that was setup before 5.4 > was release, so it could be considered 5.3 stable. > 5.4-PRERELEASE #0: Mon Feb 28 20:01:43 EST 2005 > > netstat -m > 352955 mbufs in use > 4294914731/262144 mbuf clusters in use (current/max) > 0/27/6656 sfbufs in use (current/peak/max) > 4177412 KBytes allocated to network > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 415 requests for I/O initiated by sendfile > 2300 calls to protocol drain routines Yah, I don't know, haven't had an issue on my news feeder box, but here is my output: [12:43pm] 6 [~]:canary% netstat -m 4107409126 mbufs in use 4159799080/25600 mbuf clusters in use (current/max) 0/6/6656 sfbufs in use (current/peak/max) 1541129 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 743625 calls to protocol drain routines [12:43pm] 7 [~]:canary% uname -a FreeBSD canary.octanews.net 5.4-STABLE FreeBSD 5.4-STABLE #0: Thu Apr 28 20:05:25 CDT 2005 root@canary.octanews.net:/usr/src/sys/i386/compile/CANARY i386 1.5GB allocated for network? Considering I am only seeing 177MB wired, that seems kind of odd. -- Mike Horwath, reachable via drechsau@Geeks.ORG From owner-freebsd-performance@FreeBSD.ORG Sat May 28 22:10:29 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3459B16A41F for ; Sat, 28 May 2005 22:10:29 +0000 (GMT) (envelope-from fullermd@over-yonder.net) Received: from mortis.over-yonder.net (adsl-144-221-242.jan.bellsouth.net [70.144.221.242]) by mx1.FreeBSD.org (Postfix) with ESMTP id E182B43D1D for ; Sat, 28 May 2005 22:10:28 +0000 (GMT) (envelope-from fullermd@over-yonder.net) Received: by mortis.over-yonder.net (Postfix, from userid 100) id CB21921051; Sat, 28 May 2005 17:10:27 -0500 (CDT) Date: Sat, 28 May 2005 17:10:27 -0500 From: "Matthew D. Fuller" To: Michael VInce Message-ID: <20050528221027.GB47448@over-yonder.net> References: <4297E2C4.1030505@roq.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4297E2C4.1030505@roq.com> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.9i-fullermd.2 Cc: freebsd-performance@freebsd.org Subject: Re: High usage of mbufs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 May 2005 22:10:29 -0000 On Sat, May 28, 2005 at 01:17:24PM +1000 I heard the voice of Michael VInce, and lo! it spake thus: > On one of my web servers I have a really high usage of mbuf clusters in > use on a web server that does about 3million hits a day. > 4294914731/262144 mbuf clusters in use (current/max) No it doesn't (have really high usage, that is). It just has messed up statistics, which is standard on SMP machines; see the archives. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-performance@FreeBSD.ORG Sat May 28 23:59:50 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 287C216A41C for ; Sat, 28 May 2005 23:59:50 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id A810D43D1D for ; Sat, 28 May 2005 23:59:49 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j4T01MJF057621; Sat, 28 May 2005 18:01:23 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <429905A3.9050607@samsco.org> Date: Sat, 28 May 2005 17:58:27 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Tancsa References: <6.2.1.2.0.20050527111809.0704d4e0@64.7.153.2> In-Reply-To: <6.2.1.2.0.20050527111809.0704d4e0@64.7.153.2> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD router performance and PCI-Express NICs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 May 2005 23:59:50 -0000 Mike Tancsa wrote: > While on the topic, has anyone tried > > http://www.dlink.com/products/?pid=406 > > on FreeBSD ? > > ---Mike > I've seen the marketting literature on the D-Link website for this card for many months, but I'm having a hard time finding a seller in the USA (i.e. a significant mail-order distributor). I would imagine the the card would 'just work' as long as it has a programming interface that is compatible with one of our drivers. We don't have MSI support yet so interrupt latency might still be a small issue (though it's debatable whether MSI gives better or worse latency than legacy PCI when you're dealing with a heavily loaded root complex), but interrupts should still work fine in legacy mode. I have several machines that can take a card like this and I wouldn't mind having one just for validation purposes, but it's probably not a big deal. Scott