Date: Mon, 6 May 2002 22:15:58 -0500 From: "Maildrop" <maildrop@qwest.net> To: "Maildrop" <maildrop@qwest.net>, "Benjamin Krueger" <benjamin@macguire.net>, "Bill Moran" <wmoran@potentialtech.com> Cc: questions@freebsd.org Subject: RE: Networking Buffers Message-ID: <NGBBIILBAKIFGHHCHOHPOELICOAA.maildrop@qwest.net> In-Reply-To: <NGBBIILBAKIFGHHCHOHPMEILCOAA.maildrop@qwest.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I think I hit on something. I unpluged everything from this server expect
for one workstation. After a forced reboot, there was 12 established
connections, after about 45 minutes there was over 300 connections
established to the server. I rebooted the workstation, expect the
connections stayed! I changed keepalive from 1 to 0 and let the workstation
back on and it grew to 600 in about 30 minutes. I shutdown the workstation
and the connections stayed. I changed back from 0 to 1 on keepalive and let
the workstation back on, within 5-10 minutes there was over 700 connections!
bud@hydra:/home/bud/bin/ --> netstat -s | grep con; netstat -m
729 control packets
11 connection requests
715 connection accepts
1 bad connection attempt
726 connections established (including accepts)
722 connections closed (including 573 drops)
12 connections updated cached RTT on close
12 connections updated cached RTT variance on close
1 connection updated cached ssthresh on close
0 embryonic connections dropped
0 connections dropped by rexmit timeout
0 connections dropped by persist timeout
0 connections dropped by keepalive
153/464/131072 mbufs in use (current/peak/max):
143 mbufs allocated to data
10 mbufs allocated to packet headers
141/250/32768 mbuf clusters in use (current/peak/max)
616 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
It is like it is not "deallocating" the connection correctly? Even after
doing a clean shutdown or even pulling the cable from the network card. Any
settings that would force these connections to be deallocated when not in
use (though it was net.inet.tcp.always_keepalive, but I guess that is
wrong).
Is there anyway to tell what is eating these connections? (ie. connection
1230 is allocated to port 22)
It appears that the mbufs are deallocating correctly as I saw them go up and
down during file copies, etc
Regards,
Jack
> -----Original Message-----
> From: Maildrop [mailto:maildrop@qwest.net]
> Sent: Sunday, May 05, 2002 10:46 PM
> To: Benjamin Krueger; Maildrop; Bill Moran
> Cc: questions@freebsd.org
> Subject: RE: Networking Buffers
>
>
>
> I do appericate your suggestions, I apoligize if my tone was a
> little rough in the last posts, the frustration and lack of sleep
> had got the better of me :O
>
> I changed NMBCLUSTERS to 32768, 65535 and also tried using
> net.inet.tcp.always_keepalive to both 1 and 0. I tried also
> kern.ipc.somaxconn at default (100 something?), 1024, 2048 and
> 10,000,000. None of this options have had a positive effect.
>
> I tried a differant network card. No good. I rebuilt then
> entire system and the default 4.5-rel out of the box (ie.
> Generic, no patches) crashes even more than this and decides to
> take the console with it. :( I patched back up to 4.5-Stable and
> have generic kernel, expect for NMBCLUSTERS (which is set to 32768).
>
> I tried a differant model network card. No good.
>
> I think it has something to do with coping more than one file at
> the same time (for example the last time it crashed it was apox 6
> files copies, 2 over smb, 4 over nfs and the average file size
> was 500 megs. This was take litteraly 20 seconds before it
> crashed (expect for all.log, that is 5 minutes after I got the
> machine back up (did a ifconfig down; ifconfig up)
>
>
> bud@hydra:/usr/src/sys/i386/conf/ --> netstat -m
> 133/704/131072 mbufs in use (current/peak/max):
> 131 mbufs allocated to data
> 2 mbufs allocated to packet headers
> 129/288/32768 mbuf clusters in use (current/peak/max)
> 752 Kbytes allocated to network (0% of mb_map in use)
> 0 requests for memory denied
> 0 requests for memory delayed
> 0 calls to protocol drain routines
> bud@hydra:/usr/src/sys/i386/conf/ --> netstat -i
> Name Mtu Network Address Ipkts Ierrs Opkts
> Oerrs Coll
> fxp0 1500 <Link#1> 00:03:47:40:2c:93 67561 0 114108
> 0 0
> fxp0 1500 192.168.17 hydra 35325 - 74227
> - -
> dc0 1500 <Link#2> 00:03:6d:1a:5a:31 331655 0 235408
> 0 0
> dc0 1500 64.231.238.22 64.231.238.225 291355 - 235387
> - -
> dc0 1500 64.231.238.22 64.231.238.226 3 - 0
> - -
> dc0 1500 64.231.238.22 64.231.238.227 436 - 0
> - -
> dc0 1500 64.231.238.22 64.231.238.228 10 - 0
> - -
> dc0 1500 64.231.238.22 64.231.238.229 9 - 0
> - -
> lo0 16384 <Link#3> 12675 0 12675
> 0 0
> lo0 16384 your-net localhost 12519 - 12519
> - -
> bud@hydra:/usr/src/sys/i386/conf/ --> ifconfig -a
> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 192.168.17.1 netmask 0xffffff00 broadcast 192.168.17.255
> ether 00:03:47:40:2c:93
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> dc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 64.231.238.225 netmask 0xfffffff8 broadcast 64.231.238.231
> inet 64.231.238.226 netmask 0xffffffff broadcast 64.231.238.226
> inet 64.231.238.227 netmask 0xffffffff broadcast 64.231.238.227
> inet 64.231.238.228 netmask 0xffffffff broadcast 64.231.238.228
> inet 64.231.238.229 netmask 0xffffffff broadcast 64.231.238.229
> ether 00:03:6d:1a:5a:31
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
> inet 127.0.0.1 netmask 0xff000000
>
> bud@hydra:/usr/src/sys/i386/conf/ --> netstat -s | grep con
> 82 control packets
> 13 connection requests
> 64 connection accepts
> 0 bad connection attempts
> 76 connections established (including accepts)
> 75 connections closed (including 37 drops)
> 13 connections updated cached RTT on close
> 13 connections updated cached RTT variance on close
> 6 connections updated cached ssthresh on close
> 1 embryonic connection dropped
> 0 connections dropped by rexmit timeout
> 0 connections dropped by persist timeout
> 0 connections dropped by keepalive
>
> bud@hydra:/usr/src/sys/i386/conf/ --> tail /var/log/all.log
> May 5 21:55:00 hydra /usr/sbin/cron[960]: (bud) CMD
> (/home/bud/bin/mrtg.sh)
> May 5 21:58:18 hydra su: bud to root on /dev/ttyp7
> May 5 21:58:18 hydra /kernel: May 5 21:58:18 hydra su: bud to
> root on /dev/ttyp7
> May 5 22:00:00 hydra /usr/sbin/cron[998]: (root) CMD (newsyslog -v )
> May 5 22:00:00 hydra /usr/sbin/cron[999]: (root) CMD
> (/usr/libexec/atrun)
> May 5 22:00:00 hydra /usr/sbin/cron[1000]: (bud) CMD
> (/home/bud/bin/mrtg.sh)
> May 5 22:00:01 hydra sendmail[1006]:
> gethostbyaddr(64.231.238.225) failed: 1
> May 5 22:00:01 hydra sendmail[1006]:
> gethostbyaddr(64.231.238.226) failed: 1
> May 5 22:00:01 hydra sendmail[1006]:
> gethostbyaddr(64.231.238.227) failed: 1
> May 5 22:00:01 hydra sendmail[1006]:
> gethostbyaddr(64.231.238.228) failed: 1
> May 5 22:00:01 hydra sendmail[1006]:
> gethostbyaddr(64.231.238.229) failed: 1
> May 5 22:00:01 hydra sendmail[1006]: g46301p01006: from=root,
> size=1180, class=0, nrcpts=1,
> msgid=<200205060300.g46301p01006@63.231.238.225>, relay=root@localhost
> May 5 22:00:02 hydra sendmail[1006]: g46301p01006: to=bud,
> ctladdr=root (0/0), delay=00:00:01, xdelay=00:00:01,
> mailer=local, pri=31180, relay=local, dsn=2.0.0, stat=Sent
> May 5 22:03:41 hydra inetd[1013]: connection from jenny, service
> qpopper (tcp)
> May 5 22:05:00 hydra /usr/sbin/cron[1016]: (root) CMD
> (/usr/libexec/atrun)
> May 5 22:05:00 hydra /usr/sbin/cron[1017]: (bud) CMD
> (/home/bud/bin/mrtg.sh)
> May 5 22:10:00 hydra /usr/sbin/cron[1027]: (bud) CMD
> (/home/bud/bin/mrtg.sh)
> May 5 22:10:00 hydra /usr/sbin/cron[1026]: (root) CMD
> (/usr/libexec/atrun)
> May 5 22:13:25 hydra su: bud to root on /dev/ttyp7
> May 5 22:13:25 hydra /kernel: May 5 22:13:25 hydra su: bud to
> root on /dev/ttyp7
>
>
>
> The current settings before the crash where 32768 NBMCLUSTERS,
> keepalive = 0, maxconn = 1024
>
> Everything looks normal, do you guys see anything?
>
> Also, I look those the documents you guys suggested and I still
> can't fine a complete and update info on sysctl and kernel
> options. The pages you sent where helpful, but has less than 10%
> of all the options avaiable and it scattered around. Maybe I am
> looking at the wrong thing?
>
> Any insight on what might be going on?
>
> Regards,
> Jack
>
>
>
>
>
>
>
>
>
>
>
>
> > -----Original Message-----
> > From: Benjamin Krueger [mailto:benjamin@macguire.net]
> > Sent: Friday, May 03, 2002 2:20 AM
> > To: Maildrop
> > Subject: Re: Networking Buffers
> >
> >
> > * Maildrop (maildrop@qwest.net) [020502 23:47]:
> > >
> > > WTF, now this is happening on both interfaces :( sigh
> > >
> > > ping: socket: No buffer space available
> > > telnet: socket: No buffer space available
> > >
> > > Anyone know why FreeBSD is starting to suck so bad when it comes to
> > > networking?
> > >
> > > J
> >
> > My mail server is borked right now, so I can't send to the list.
> > Basically,
> > this is user config error. ;) You need to increase the
> > available amount of
> > network buffers your server has. When you run out of mbufs, thats
> > it. Until
> > those network connections time out, you're just going to wait.
> >
> > Here's the scoop. When you compiled the kernel, you set an option called
> > NMBCLUSTERS. The default is likely too small. Run netstat -m and
> > you'll likely
> > see that all available mbufs are being used. You can adjust the
> number of
> > available nmbclusters using the kern.ipc.nmbclusters sysctl oid.
> >
> > The number you will want to use will be highly dependant on what
> > kind of load
> > you're running. See tuning(7) and search for mbuf. It has a short
> > and simple
> > guide to figuring out a decent number to enter for nmbclusters.
> > Once you get
> > that squared away and tuned, your FreeBSD server should run
> > happily again. You
> > might consider investing some time into reading some of the
> > advanced tutorials
> > in the documentation, as well as the entire tuning(7) page. They
> > all make for
> > excellent reading.
> >
> > -ben
> >
> > > > The issuse I am having on it, is it seems I am "blowing out"
> > one of the
> > > > nics.
> > > > I am sure this FreeBSD related, because this box use to run
> W2K with a
> > > > higher
> > > > network load and didn't expeirence any issuses.
> > > >
> > > > After a high spike it network load (when we backup all data in
> > > > the middle of
> > > > the night) one
> > > > of the ip address will stop responding. I log in locally and
> > the network
> > > > tables are fine, but
> > > > when I try to ping out of that network card it gives the
> error message
> > > > "ping: no buffers avaiable"
> > > >
> > > > Taking the nic up then back down (`ifconfig fxp0 down;
> > ifconfig fxp0 up`)
> > > > will fix the issuse, but
> > > > is a huge problem because it is very distrubating to network
> > operation,
> > > > espically during nightly
> > > > backups/data merges when no one is onsite. I replaced the nic
> > > > with a brand
> > > > new one that is the same
> > > > model, and also with a new one that is a differant
> > brand/model and it is
> > > > still having issuses.
> > > >
> > > > In sysctl I changed (IIRC) "net.inet.tcp.sendspace" and
> > > > "net.inet.recvspace"
> > > > both to 8192 (thinking it is Kb?),
> > > > that still had issuses, so I change it to 8192000 (thinking
> > it might be
> > > > bytes) and also tried 8192000000.
> > > > All these settings still cause FreeBSD to drop the nic. What is
> > > > the size of
> > > > these variables? (ie. mb, kb, etc ?)
> > > >
> > > > Also, are these the correct settings? Is network buffers
> > configured some
> > > > where else? I have yet to find
> > > > complete and full documentation on sysctl, anyone know where
> > I could get
> > > > these docs, freebsd.com has very
> > > > incomplete and out of data documentation in regards to sysctl
> > > > variables and
> > > > kernel tweaking in general...
> > > >
> > > > Also is there any type of intelligent networking buffering
> in FreeBSD?
> > > > Something that would
> > > > say "If network buffers are full, pause network traffic,
> > flush buffers,
> > > > continue with network
> > > > traffic" ? This way, if the buffers are full, it won't take
> > out the nic
> > > > completly,
> > > > but rather hang it, and then continue after the buffers had
> > be purged? I
> > > > think this would be a more
> > > > graceful method of handling this, instead of "Opps, buffers are
> > > > full, sorry
> > > > but your server is useless now"...
> > > > Anyone know how to enable this?
> > > >
> > > > Also it is very strange that something this critical isn't logged
> > > > to syslog,
> > > > what do I have to have in
> > > > /etc/syslog.conf to get this info, I basically turned on
> everything to
> > > > debug, and didn't see anything to
> > > > messages, console or all.log.
> > > >
> > > > It is kinda funny, this server went down again when I was
> typing this
> > > > email.. I can tell when it goes down
> > > > cause, winamp disconnects from server and then I hear the
> > solaris admin 4
> > > > rooms down swearing at freebsd :)
> > > >
> > > > I got to get this fixed soon or will be forced to move it
> > back to W2K, any
> > > > ideas?
> > > >
> > > > Regards,
> > > > Jack
> >
> > --
> > Benjamin Krueger
> >
> > "Life is far too important a thing ever to talk seriously about."
> > - Oscar Wilde (1854 - 1900)
> > ----------------------------------------------------------------
> > Send mail w/ subject 'send public key' or query for (0x251A4B18)
> > Fingerprint = A642 F299 C1C1 C828 F186 A851 CFF0 7711 251A 4B18
> >
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?NGBBIILBAKIFGHHCHOHPOELICOAA.maildrop>