From owner-freebsd-questions Wed Jul 30 15:53:00 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.5/8.8.5) id PAA10325 for questions-outgoing; Wed, 30 Jul 1997 15:53:00 -0700 (PDT) Received: from badger.tltodd.com (badger.tltodd.com [208.133.92.209]) by hub.freebsd.org (8.8.5/8.8.5) with ESMTP id PAA10306 for ; Wed, 30 Jul 1997 15:52:51 -0700 (PDT) Received: (from tlt@localhost) by badger.tltodd.com (8.8.3/8.8.3) id SAA09724; Wed, 30 Jul 1997 18:01:17 -0500 (CDT) Date: Wed, 30 Jul 1997 18:01:17 -0500 (CDT) From: Terry Todd Message-Id: <199707302301.SAA09724@badger.tltodd.com> To: freebsd-questions@FreeBSD.ORG, tlt@badger.tltodd.com Subject: No buffer space available Sender: owner-freebsd-questions@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk I posted this once before and didn't hear anything back but that may have been because various people were on vacation. This happened again last week and I had maxconcur set to 8. My system is set up with a dedicated 28.8 dial up using pppd. 99% of the time it just sits there and runs happily along. I really like FreeBSD. Here's the main symptom that something is wrong: ping: sendto: No buffer space available After it gets in this state it will never recover by itself. Is there something else that I need to do if it happens again? from before: My system got sick again. The symptoms were as before where no TCP/IP traffic was going out or coming in via ppp. This time I was able to capture a fair amount of data. I am running FreeBSD 2.1.6 and using it to run several mailing lists. I am using the procmail/SmartList package. I have maxconcur set to 16 in rc.init for SmartList. I was able to kill -1 on the pid of pppd and get it to restart this time. This leads me to believe it may be a bug in ppp. There were a couple of other commands that were suggested I run if this happened again which I did but they just sat there and produced no output so I ^C'ed out. I think the chances of the problem happening again can be decreased by decreasing maxconcur but the OS/ppp should be able to recover from this by itself. Any pointers or help with this problem would be appreciated. Thanks, Terry Todd Here's the result of running ping. just before it went totally offline... tlt@badger> ping ns1.megsinet.net PING ns1.megsinet.net (208.150.60.2): 56 data bytes ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 64 bytes from 208.150.60.2: icmp_seq=0 ttl=252 time=16395.492 ms 64 bytes from 208.150.60.2: icmp_seq=1 ttl=252 time=15433.823 ms 64 bytes from 208.150.60.2: icmp_seq=2 ttl=252 time=14434.907 ms 64 bytes from 208.150.60.2: icmp_seq=3 ttl=252 time=13450.170 ms 64 bytes from 208.150.60.2: icmp_seq=4 ttl=252 time=12814.616 ms 64 bytes from 208.150.60.2: icmp_seq=5 ttl=252 time=11818.169 ms 64 bytes from 208.150.60.2: icmp_seq=6 ttl=252 time=10819.124 ms 64 bytes from 208.150.60.2: icmp_seq=7 ttl=252 time=9846.337 ms ^C --- ns1.megsinet.net ping statistics --- 18 packets transmitted, 8 packets received, 55% packet loss round-trip min/avg/max = 9846.337/13126.579/16395.492 ms tlt@badger> and after it was dead.... (after it gets in this state it will never recover) tlt@badger> ping ns1.megsinet.net PING ns1.megsinet.net (208.150.60.2): 56 data bytes ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ping: sendto: No buffer space available ping: wrote ns1.megsinet.net 64 chars, ret=-1 ^C --- ns1.megsinet.net ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss tlt@badger> Here's the result of various netstat commands... netstat -m 381 mbufs in use: 151 mbufs allocated to data 156 mbufs allocated to packet headers 68 mbufs allocated to protocol control blocks 6 mbufs allocated to socket names and addresses 65/124 mbuf clusters in use 295 Kbytes allocated to network (60% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 208.133.80.1 UGc 114 61968 ppp0 127.0.0.1 127.0.0.1 UH 3 659000 lo0 208.133.80.1 208.133.92.209 UH 114 0 ppp0 208.133.92.208/29 link#1 UC 0 0 208.133.92.211 0:60:97:26:50:b5 UHLW 2 4684 ed0 1104 208.133.92.212 0:20:af:d8:a:4e UHLW 0 11123 ed0 1066 netstat -nrs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 546 destinations found unreachable 0 uses of a wildcard route netstat -s ip: 2275262 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 172 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 86 packets reassembled ok 2141054 packets for this host 14278 packets for unknown/unsupported protocol 118491 packets forwarded 940 packets not forwardable 0 redirects sent 2135189 packets sent from this host 39 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 272 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented icmp: 5296 calls to icmp_error 59 errors not generated 'cuz old message was icmp Output histogram: echo reply: 731 destination unreachable: 3960 source quench: 858 time exceeded: 413 77 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 120 destination unreachable: 13277 source quench: 8 echo: 731 time exceeded: 1012 731 message responses generated igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent tcp: 1322206 packets sent 785862 data packets (271454502 bytes) 42641 data packets (26059172 bytes) retransmitted 2991 resends initiated by MTU discovery 354992 ack-only packets (98931 delayed) 0 URG only packets 1374 window probe packets 3386 window update packets 156471 control packets 1410062 packets received 853894 acks (for 272958170 bytes) 210110 duplicate acks 0 acks for unsent data 600327 packets (45329669 bytes) received in-sequence 92607 completely duplicate packets (4754201 bytes) 0 old duplicate packets 376 packets with some dup. data (36534 bytes duped) 71558 out-of-order packets (635751 bytes) 5 packets (0 bytes) of data after window 0 window probes 3764 window update packets 322 packets received after close 753 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 82635 connection requests 7639 connection accepts 43 bad connection attempts 0 listen queue overflows 74818 connections established (including accepts) 90155 connections closed (including 3183 drops) 2049 connections updated cached RTT on close 2049 connections updated cached RTT variance on close 636 connections updated cached ssthresh on close 14961 embryonic connections dropped 728447 segments updated rtt (of 754105 attempts) 47182 retransmit timeouts 62 connections dropped by rexmit timeout 1428 persist timeouts 0 connections dropped by persist timeout 6441 keepalive timeouts 11 keepalive probes sent 6430 connections dropped by keepalive 90173 correct ACK header predictions 263041 correct data packet header predictions udp: 730122 datagrams received 0 with incomplete header 1 with bad data length field 3 with bad checksum 3943 dropped due to no socket 7200 broadcast/multicast datagrams dropped due to no socket 0 dropped due to full socket buffers 403651 not for hashed pcb 718975 delivered 737376 datagrams output and the result of running ifconfig... ifconfig -a ed0: flags=8863 mtu 1500 inet 208.133.92.210 netmask 0xfffffff8 broadcast 208.133.92.215 lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000 ppp0: flags=8051 mtu 1524 inet 208.133.92.209 --> 208.133.80.1 netmask 0xfffffff8 sl0: flags=c010 mtu 552 tun0: flags=8010 mtu 1500 and the result of pstat.... pstat -s Device 512-blocks Used Avail Capacity Type /dev/wd0s1b 286624 0 286496 0% Interleaved /dev/vn0c 262144 0 262016 0% Interleaved Total 548512 0 548512 0% here's vmstat.... vmstat -s 5427559 cpu context switches 273249653 device interrupts 3525240 software interrupts 20682765 traps 60302738 system calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 417 vnode pager pageins 1775 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 0 page daemon wakeups 0 pages examined by the page daemon 264612 pages reactivated 4582333 copy-on-write faults 5736908 zero fill pages zeroed 2 intransit blocking page faults 21349301 total VM faults taken 18359250 pages freed 0 pages freed by daemon 8548611 pages freed by exiting processes 11884 pages active 558 pages inactive 9615 pages in VM cache 3387 pages wired down 2241 pages free 4096 bytes per page 5926643 total name lookups cache hits (76% pos + 4% neg) system 3% per-process deletions 2%, falsehits 0%, toolong 0% here's what was running at the time... ps aux USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND tlt 27264 0.0 0.3 500 312 p0 R+ 10:25PM 0:00.01 ps -aux root 1 0.0 0.2 436 208 ?? Is 29Jun97 0:10.54 /sbin/init -- root 2 0.0 0.0 0 12 ?? DL 29Jun97 0:00.00 (pagedaemon) root 3 0.0 0.0 0 12 ?? DL 29Jun97 0:00.00 (vmdaemon) root 4 0.0 0.0 0 12 ?? DL 29Jun97 19:19.57 (update) root 22 0.0 0.1 216 80 ?? Is 29Jun97 0:00.01 adjkerntz -i root 54 0.0 0.2 192 244 ?? Is 29Jun97 0:09.64 routed -q root 70 0.0 0.3 192 324 ?? Ss 29Jun97 1:30.04 syslogd daemon 78 0.0 0.2 176 240 ?? Is 29Jun97 0:00.01 portmap root 88 0.0 0.3 220 308 ?? Is 29Jun97 0:18.43 inetd root 95 0.0 0.4 432 472 ?? Ss 29Jun97 3:42.17 /usr/local/htt nobody 97 0.0 0.6 484 612 ?? I 29Jun97 0:00.22 /usr/local/htt nobody 98 0.0 0.6 484 612 ?? I 29Jun97 0:00.20 /usr/local/htt nobody 99 0.0 0.5 484 608 ?? I 29Jun97 0:00.24 /usr/local/htt nobody 100 0.0 0.6 496 612 ?? I 29Jun97 0:00.27 /usr/local/htt nobody 101 0.0 0.6 496 616 ?? I 29Jun97 0:00.24 /usr/local/htt root 102 0.0 0.3 268 320 ?? Is 29Jun97 0:18.57 cron root 109 0.0 0.3 560 340 ?? Is 29Jun97 0:20.92 sendmail: acce root 155 0.0 0.4 156 480 v2 Is+ 29Jun97 0:00.02 /usr/libexec/g root 198 0.0 0.5 260 600 a2 Is+ 29Jun97 0:00.06 /usr/sbin/pppd nobody 11830 0.0 0.6 496 620 ?? I 29Jun97 0:00.27 /usr/local/htt nobody 1329 0.0 0.6 484 612 ?? I 1Jul97 0:00.18 /usr/local/htt nobody 11879 0.0 0.5 484 604 ?? I Mon12AM 0:00.09 /usr/local/htt nobody 11880 0.0 0.6 496 616 ?? I Mon12AM 0:00.13 /usr/local/htt root 4319 0.0 1.1 1176 1192 ?? Ss Tue06AM 1:58.25 named -b /etc/ root 4228 0.0 0.5 156 524 v0 Is+ Wed07PM 0:00.02 /usr/libexec/g tlt 8188 0.0 0.6 344 708 v1 Is+ 10:48PM 0:00.32 -ksh (ksh) root 22118 0.0 1.0 752 1148 ?? I 5:44PM 0:00.99 sendmail: RAA2 root 22213 0.0 1.0 760 1140 ?? I 5:50PM 0:00.98 sendmail: RAA2 root 22255 0.0 1.0 736 1140 ?? I 5:50PM 0:00.56 sendmail: RAA2 root 22715 0.0 1.0 764 1124 ?? I 6:46PM 0:00.82 sendmail: SAA2 root 22717 0.0 1.0 752 1128 ?? I 6:46PM 0:00.81 sendmail: SAA2 root 23139 0.0 1.0 760 1144 ?? I 6:59PM 0:00.80 sendmail: SAA2 root 23141 0.0 1.0 744 1140 ?? I 6:59PM 0:00.70 sendmail: SAA2 root 23280 0.0 1.0 760 1120 ?? S 7:10PM 0:00.83 sendmail: TAA2 root 23282 0.0 1.0 744 1120 ?? I 7:10PM 0:00.70 sendmail: TAA2 root 23530 0.0 1.0 756 1140 ?? I 7:13PM 0:00.72 sendmail: TAA2 root 23532 0.0 1.0 744 1140 ?? I 7:13PM 0:00.71 sendmail: TAA2 root 23631 0.0 1.0 760 1128 ?? S 7:22PM 0:00.76 sendmail: TAA2 root 23633 0.0 1.0 744 1108 ?? I 7:22PM 0:00.64 sendmail: TAA2 root 23727 0.0 1.0 756 1140 ?? S 7:28PM 0:00.69 sendmail: TAA2 root 23729 0.0 1.0 744 1140 ?? I 7:28PM 0:00.69 sendmail: TAA2 root 23873 0.0 1.0 744 1120 ?? S 7:47PM 0:00.63 sendmail: TAA2 root 23876 0.0 1.0 736 1112 ?? I 7:47PM 0:00.60 sendmail: TAA2 root 24331 0.0 1.0 740 1140 ?? I 8:17PM 0:00.51 sendmail: UAA2 root 24334 0.0 1.0 736 1140 ?? S 8:17PM 0:00.45 sendmail: UAA2 root 24335 0.0 1.0 712 1104 ?? S 8:17PM 0:00.34 sendmail: UAA2 root 24360 0.0 1.0 720 1128 ?? I 8:17PM 0:00.41 sendmail: UAA2 root 24584 0.0 1.0 756 1096 ?? I 8:24PM 0:00.57 sendmail: UAA2 root 24586 0.0 1.0 744 1116 ?? S 8:24PM 0:00.52 sendmail: UAA2 root 24666 0.0 0.8 776 860 ?? I 8:41PM 0:00.52 sendmail: UAA2 root 24685 0.0 1.0 756 1124 ?? I 8:43PM 0:00.57 sendmail: UAA2 root 24687 0.0 1.0 744 1104 ?? S 8:43PM 0:00.49 sendmail: UAA2 root 25117 0.0 1.0 700 1076 ?? I 9:01PM 0:00.28 sendmail: VAA2 root 25128 0.0 1.0 732 1104 ?? I 9:01PM 0:00.41 sendmail: VAA2 root 25131 0.0 1.0 732 1128 ?? I 9:01PM 0:00.35 sendmail: VAA2 root 25132 0.0 1.0 708 1088 ?? I 9:01PM 0:00.28 sendmail: VAA2 root 25173 0.0 1.0 700 1076 ?? I 9:01PM 0:00.27 sendmail: VAA2 root 25397 0.0 1.0 756 1124 ?? I 9:08PM 0:00.52 sendmail: VAA2 root 25399 0.0 1.0 744 1100 ?? S 9:08PM 0:00.48 sendmail: VAA2 root 25464 0.0 0.7 744 820 ?? I 9:11PM 0:00.38 sendmail: VAA2 root 25505 0.0 1.0 736 1108 ?? S 9:21PM 0:00.46 sendmail: VAA2 root 25508 0.0 1.0 736 1108 ?? I 9:21PM 0:00.34 sendmail: VAA2 root 25509 0.0 0.9 704 1060 ?? S 9:21PM 0:00.25 sendmail: VAA2 root 25901 0.0 1.0 732 1100 ?? I 9:27PM 0:00.37 sendmail: VAA2 root 25904 0.0 1.0 732 1104 ?? I 9:27PM 0:00.33 sendmail: VAA2 root 25905 0.0 0.9 704 1056 ?? S 9:27PM 0:00.25 sendmail: VAA2 root 25942 0.0 1.0 700 1064 ?? I 9:27PM 0:00.23 sendmail: VAA2 root 26150 0.0 0.9 700 1052 ?? S 9:27PM 0:00.22 sendmail: VAA2 root 26153 0.0 1.0 748 1112 ?? S 9:27PM 0:00.42 sendmail: VAA2 root 26155 0.0 1.0 740 1100 ?? I 9:27PM 0:00.40 sendmail: VAA2 root 26255 0.0 1.0 736 1108 ?? S 9:35PM 0:00.42 sendmail: VAA2 root 26258 0.0 1.0 736 1108 ?? I 9:35PM 0:00.34 sendmail: VAA2 root 26259 0.0 0.9 700 1052 ?? S 9:35PM 0:00.24 sendmail: VAA2 root 26654 0.0 0.8 776 860 ?? S 9:41PM 0:00.56 sendmail: VAA2 root 26688 0.0 0.9 700 1060 ?? I 9:51PM 0:00.20 sendmail: VAA2 root 26691 0.0 1.0 752 1096 ?? I 9:51PM 0:00.43 sendmail: VAA2 root 26693 0.0 1.0 736 1112 ?? S 9:51PM 0:00.35 sendmail: VAA2 root 26694 0.0 0.9 644 1020 ?? I 9:51PM 0:00.13 sendmail: VAA2 root 26695 0.0 1.0 700 1072 ?? I 9:51PM 0:00.21 sendmail: VAA2 root 26825 0.0 0.9 700 1052 ?? S 10:10PM 0:00.18 sendmail: WAA2 root 26827 0.0 0.9 644 1008 ?? I 10:10PM 0:00.13 sendmail: WAA2 root 26832 0.0 0.9 644 1016 ?? S 10:10PM 0:00.13 sendmail: WAA2 root 26835 0.0 0.9 644 1016 ?? I 10:10PM 0:00.14 sendmail: WAA2 root 26836 0.0 0.9 680 1044 ?? I 10:10PM 0:00.31 sendmail: WAA2 root 26838 0.0 0.9 696 1044 ?? I 10:10PM 0:00.15 sendmail: WAA2 root 26839 0.0 1.0 676 1072 ?? I 10:10PM 0:00.25 sendmail: WAA2 root 26840 0.0 0.9 700 1048 ?? I 10:10PM 0:00.17 sendmail: WAA2 root 26867 0.0 0.9 648 1008 ?? I 10:10PM 0:00.15 sendmail: WAA2 root 27068 0.0 0.8 764 852 ?? S 10:11PM 0:00.36 sendmail: VAA2 root 27069 0.0 0.9 644 1016 ?? I 10:11PM 0:00.13 sendmail: WAA2 root 27072 0.0 0.5 216 592 ?? S 10:13PM 0:10.99 telnetd tlt 27073 0.0 0.6 348 680 p0 Ss 10:13PM 0:00.58 -ksh (ksh) root 27114 0.0 0.4 600 472 ?? I 10:22PM 0:00.02 sendmail: serv root 27256 0.0 0.4 600 484 ?? I 10:24PM 0:00.02 sendmail: serv root 0 0.0 0.0 0 0 ?? DLs - 0:00.00 (swapper) Here's my mailq shortened for the sake of brevity here... Mail Queue (309 requests) --Q-ID-- --Size-- -----Q-Time----- ------------Sender/Recipient------------ WAA27565* 318 Thu Jul 10 22:30 rcg-forum@tltodd.com Eric.Walter@quickmail.llnl.gov WAA26833 230 Thu Jul 10 22:10 rcg-forum@tltodd.com (host map: lookup (hyphe.com): deferred) jutson@hyphe.com VAA26253* 318 Thu Jul 10 21:35 rcg-forum@tltodd.com (wat@uniserve.com,ctolmie@uniserve.com... timeout waiting for) ctolmie@uniserve.com wat@uniserve.com VAA26257* 318 Thu Jul 10 21:35 rcg-forum@tltodd.com (dbedwell@shastalink.k12.ca.us... timeout waiting for input d) dbedwell@shastalink.k12.ca.us jhendric@cabin.llcc.cc.il.us .........much more snipped off here.....