Date: Mon, 7 May 2001 08:25:41 +0200 (CEST) From: Luigi Rizzo <luigi@info.iet.unipi.it> To: Pekka Savola <pekkas@netcore.fi> Cc: freebsd-stable@freebsd.org, luigi@freebsd.org Subject: Re: 4.3-S: No buffer space available [SOLVED: dummynet] Message-ID: <200105070625.IAA93694@info.iet.unipi.it> In-Reply-To: <Pine.LNX.4.33.0105070740270.23569-100000@netcore.fi> from Pekka Savola at "May 7, 2001 07:51:41 am"
next in thread | previous in thread | raw e-mail | index | archive | help
> On Sat, 5 May 2001, Pekka Savola wrote: > > Ok. This was caused by the dummynet rule, either directly or indirectly. > > The symptoms: > 1) 'No buffer space available' > 2) hard crashes: does not respond to ping, closole goes blank, ethernet > link led in the switch goes off etc (about 3-4 times in 48h) > 3) soft crashes: responds to ping, traceroute, tcp establishment ok, but > the userland is dead (2 times in 48h) > > The first rule was: > --- > $fwcmd pipe 1 config delay 0 plr 0 bw 20000Kbit/s queue 100 > $fwcmd add pipe 1 ip from any to any > --- > In addition, net.inet.ip.fw.one_pass=0 and 50-500 ipfw rules (assigned > using external signals). dummynet consumes 1 mbuf/cluster per queue entry, so if you only have one pipe as above that should not be a major problem. I can only think of some leak, e.g. caused by mishandling of ipfw reconfigurations, but have no specific example in mind -- in any case, if the problem is there, you should see the number of mbufs in use go up with time. I have a comment on your configuration though -- it sounds kind of strange to have a single pipe limiting everyghing (in and out) upfront, and then do the actual filtering afterwards. It would be much more efficient to do the filtering first, and replace all "accept" rules with a "skipto 60000" where at rule 60000 you have the pipe limiting outgoing traffic (and maybe a different one for limiting input traffic). This way you would also not need the net.inet.ip.fw.one_pass=0 setting. Also, the 50-500 rules sound strange, wouldn't you be able to replace a fair amount of them with dynamic (aka stateful) rules ? cheers luigi > I recompiled the kernel with 100% same options, only leaving dummynet out. > Everything wortked like a charm. Other things had also been tried, like > removing SMP support, no luck. > > The traffic being shaped to 20Mbit/s ranged from 25-35 Mbit/s (steady), > mostly outgoing. > > Has dummynet been tested in this kind of heavy environment? > > Is there a better value for 'queue', e.g. 1000Kbytes in this scenario? > > Regards, > Pekka Savola > > > Running on 4.3-S on Dual P3/866 (self-compiled for SMP, dummynet etc.): > > > > [root@xxx /root] # snmpwalk xxx yyy > > snmpwalk: Failure in sendto (No buffer space available) > > > > [root@xxx /root] # netstat -m > > 8012/8576/65536 mbufs in use (current/peak/max): > > 6581 mbufs allocated to data > > 1431 mbufs allocated to packet headers > > 6300/6682/16384 mbuf clusters in use (current/peak/max) > > 15508 Kbytes allocated to network (31% of mb_map in use) > > 0 requests for memory denied > > 0 requests for memory delayed > > 0 calls to protocol drain routines > > > > [root@xxx /root] # sysctl kern | grep files > > kern.maxfiles: 16424 > > kern.maxfilesperproc: 16424 > > kern.openfiles: 2709 > > > > --- > > last pid: 84147; load averages: 0.52, 0.66, 0.61 up 0+01:38:21 13:12:34 > > 713 processes: 12 running, 688 sleeping, 13 stopped > > CPU states: 14.1% user, 0.0% nice, 20.7% system, 4.0% interrupt, 61.3% idle > > Mem: 516M Active, 316M Inact, 134M Wired, 37M Cache, 112M Buf, 1664K Free > > Swap: 1024M Total, 92K Used, 1024M Free > > --- > > > > Pushing through 20 Mbit/s steady as we speak. > > > > This seems to have been mentioned in the beginning of April with no clear > > resolution. > > > > A few kernel options mentioned in the posts: > > --- > > cpu I686_CPU > > maxusers 512 > > options NMBCLUSTERS=16384 > > options SMP # Symmetric MultiProcessor Kernel > > options APIC_IO # Symmetric (APIC) I/O > > --- > > This is a system with 1 GB of RAM. Network card is: > > > > fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0xccc0-0xccff mem > > 0xf9000000-0xf90fffff,0xf9100000-0xf9100fff irq 10 at device 8.0 on pci1 > > > > > > What's wrong? There definitely should be enough buffers. > > > > Also: userspace froze earlier today for some odd reason; ping and > > traceroute responded, ipfw worked, tcp connection could be established but > > the daemon listening to the port never replied. Nothing in the log or the > > console. Ideas? > > > > Please Cc:. > > > > > > -- > Pekka Savola "Tell me of difficulties surmounted, > Netcore Oy not those you stumble over and fall" > Systems. Networks. Security. -- Robert Jordan: A Crown of Swords > > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200105070625.IAA93694>