Date: Fri, 11 Jul 1997 07:54:34 -0500 (CDT) From: Terry Todd <tlt@badger.tltodd.com> To: freebsd-questions@freebsd.org Subject: It did it again. Message-ID: <199707111254.HAA09444@badger.tltodd.com>
next in thread | raw e-mail | index | archive | help
My system got sick again. The symptoms were as before where no
TCP/IP traffic was going out or coming in. This time I was able
to capture a fair amount of data. I am running FreeBSD 2.1.6
and using it to run several mailing lists. I am using the
procmail/SmartList package. I have maxconcur set to 16 in
rc.init for SmartList. I was able to kill -1 on the pid of
pppd and get it to restart this time. This leads me to believe
it may be a bug in pppd. There were a couple of other commands
that were suggested I run if this happened again which I did but
they just sat there and produced no output so I ^C'ed out. I think
the chances of the problem happening again can be decreased by
decreasing maxconcur but the OS/pppd should be able to recover
from this by itself. Any pointers or help with this problem would
be appreciated.
Thanks,
Terry Todd
Here's the result of running ping.
just before it went totally offline...
tlt@badger> ping ns1.megsinet.net
PING ns1.megsinet.net (208.150.60.2): 56 data bytes
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
64 bytes from 208.150.60.2: icmp_seq=0 ttl=252 time=16395.492 ms
64 bytes from 208.150.60.2: icmp_seq=1 ttl=252 time=15433.823 ms
64 bytes from 208.150.60.2: icmp_seq=2 ttl=252 time=14434.907 ms
64 bytes from 208.150.60.2: icmp_seq=3 ttl=252 time=13450.170 ms
64 bytes from 208.150.60.2: icmp_seq=4 ttl=252 time=12814.616 ms
64 bytes from 208.150.60.2: icmp_seq=5 ttl=252 time=11818.169 ms
64 bytes from 208.150.60.2: icmp_seq=6 ttl=252 time=10819.124 ms
64 bytes from 208.150.60.2: icmp_seq=7 ttl=252 time=9846.337 ms
^C
--- ns1.megsinet.net ping statistics ---
18 packets transmitted, 8 packets received, 55% packet loss
round-trip min/avg/max = 9846.337/13126.579/16395.492 ms
tlt@badger>
and after it was dead....
(after it gets in this state it will never recover)
tlt@badger> ping ns1.megsinet.net
PING ns1.megsinet.net (208.150.60.2): 56 data bytes
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
ping: sendto: No buffer space available
ping: wrote ns1.megsinet.net 64 chars, ret=-1
^C
--- ns1.megsinet.net ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
tlt@badger>
Here's the result of various netstat commands...
netstat -m
381 mbufs in use:
151 mbufs allocated to data
156 mbufs allocated to packet headers
68 mbufs allocated to protocol control blocks
6 mbufs allocated to socket names and addresses
65/124 mbuf clusters in use
295 Kbytes allocated to network (60% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 208.133.80.1 UGc 114 61968 ppp0
127.0.0.1 127.0.0.1 UH 3 659000 lo0
208.133.80.1 208.133.92.209 UH 114 0 ppp0
208.133.92.208/29 link#1 UC 0 0
208.133.92.211 0:60:97:26:50:b5 UHLW 2 4684 ed0 1104
208.133.92.212 0:20:af:d8:a:4e UHLW 0 11123 ed0 1066
netstat -nrs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
546 destinations found unreachable
0 uses of a wildcard route
netstat -s
ip:
2275262 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
172 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
86 packets reassembled ok
2141054 packets for this host
14278 packets for unknown/unsupported protocol
118491 packets forwarded
940 packets not forwardable
0 redirects sent
2135189 packets sent from this host
39 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
272 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
icmp:
5296 calls to icmp_error
59 errors not generated 'cuz old message was icmp
Output histogram:
echo reply: 731
destination unreachable: 3960
source quench: 858
time exceeded: 413
77 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
echo reply: 120
destination unreachable: 13277
source quench: 8
echo: 731
time exceeded: 1012
731 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
tcp:
1322206 packets sent
785862 data packets (271454502 bytes)
42641 data packets (26059172 bytes) retransmitted
2991 resends initiated by MTU discovery
354992 ack-only packets (98931 delayed)
0 URG only packets
1374 window probe packets
3386 window update packets
156471 control packets
1410062 packets received
853894 acks (for 272958170 bytes)
210110 duplicate acks
0 acks for unsent data
600327 packets (45329669 bytes) received in-sequence
92607 completely duplicate packets (4754201 bytes)
0 old duplicate packets
376 packets with some dup. data (36534 bytes duped)
71558 out-of-order packets (635751 bytes)
5 packets (0 bytes) of data after window
0 window probes
3764 window update packets
322 packets received after close
753 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
82635 connection requests
7639 connection accepts
43 bad connection attempts
0 listen queue overflows
74818 connections established (including accepts)
90155 connections closed (including 3183 drops)
2049 connections updated cached RTT on close
2049 connections updated cached RTT variance on close
636 connections updated cached ssthresh on close
14961 embryonic connections dropped
728447 segments updated rtt (of 754105 attempts)
47182 retransmit timeouts
62 connections dropped by rexmit timeout
1428 persist timeouts
0 connections dropped by persist timeout
6441 keepalive timeouts
11 keepalive probes sent
6430 connections dropped by keepalive
90173 correct ACK header predictions
263041 correct data packet header predictions
udp:
730122 datagrams received
0 with incomplete header
1 with bad data length field
3 with bad checksum
3943 dropped due to no socket
7200 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
403651 not for hashed pcb
718975 delivered
737376 datagrams output
and the result of running ifconfig...
ifconfig -a
ed0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 208.133.92.210 netmask 0xfffffff8 broadcast 208.133.92.215
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet 127.0.0.1 netmask 0xff000000
ppp0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1524
inet 208.133.92.209 --> 208.133.80.1 netmask 0xfffffff8
sl0: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 552
tun0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
and the result of pstat....
pstat -s
Device 512-blocks Used Avail Capacity Type
/dev/wd0s1b 286624 0 286496 0% Interleaved
/dev/vn0c 262144 0 262016 0% Interleaved
Total 548512 0 548512 0%
here's vmstat....
vmstat -s
5427559 cpu context switches
273249653 device interrupts
3525240 software interrupts
20682765 traps
60302738 system calls
0 swap pager pageins
0 swap pager pages paged in
0 swap pager pageouts
0 swap pager pages paged out
417 vnode pager pageins
1775 vnode pager pages paged in
0 vnode pager pageouts
0 vnode pager pages paged out
0 page daemon wakeups
0 pages examined by the page daemon
264612 pages reactivated
4582333 copy-on-write faults
5736908 zero fill pages zeroed
2 intransit blocking page faults
21349301 total VM faults taken
18359250 pages freed
0 pages freed by daemon
8548611 pages freed by exiting processes
11884 pages active
558 pages inactive
9615 pages in VM cache
3387 pages wired down
2241 pages free
4096 bytes per page
5926643 total name lookups
cache hits (76% pos + 4% neg) system 3% per-process
deletions 2%, falsehits 0%, toolong 0%
here's what was running at the time...
ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
tlt 27264 0.0 0.3 500 312 p0 R+ 10:25PM 0:00.01 ps -aux
root 1 0.0 0.2 436 208 ?? Is 29Jun97 0:10.54 /sbin/init --
root 2 0.0 0.0 0 12 ?? DL 29Jun97 0:00.00 (pagedaemon)
root 3 0.0 0.0 0 12 ?? DL 29Jun97 0:00.00 (vmdaemon)
root 4 0.0 0.0 0 12 ?? DL 29Jun97 19:19.57 (update)
root 22 0.0 0.1 216 80 ?? Is 29Jun97 0:00.01 adjkerntz -i
root 54 0.0 0.2 192 244 ?? Is 29Jun97 0:09.64 routed -q
root 70 0.0 0.3 192 324 ?? Ss 29Jun97 1:30.04 syslogd
daemon 78 0.0 0.2 176 240 ?? Is 29Jun97 0:00.01 portmap
root 88 0.0 0.3 220 308 ?? Is 29Jun97 0:18.43 inetd
root 95 0.0 0.4 432 472 ?? Ss 29Jun97 3:42.17 /usr/local/htt
nobody 97 0.0 0.6 484 612 ?? I 29Jun97 0:00.22 /usr/local/htt
nobody 98 0.0 0.6 484 612 ?? I 29Jun97 0:00.20 /usr/local/htt
nobody 99 0.0 0.5 484 608 ?? I 29Jun97 0:00.24 /usr/local/htt
nobody 100 0.0 0.6 496 612 ?? I 29Jun97 0:00.27 /usr/local/htt
nobody 101 0.0 0.6 496 616 ?? I 29Jun97 0:00.24 /usr/local/htt
root 102 0.0 0.3 268 320 ?? Is 29Jun97 0:18.57 cron
root 109 0.0 0.3 560 340 ?? Is 29Jun97 0:20.92 sendmail: acce
root 155 0.0 0.4 156 480 v2 Is+ 29Jun97 0:00.02 /usr/libexec/g
root 198 0.0 0.5 260 600 a2 Is+ 29Jun97 0:00.06 /usr/sbin/pppd
nobody 11830 0.0 0.6 496 620 ?? I 29Jun97 0:00.27 /usr/local/htt
nobody 1329 0.0 0.6 484 612 ?? I 1Jul97 0:00.18 /usr/local/htt
nobody 11879 0.0 0.5 484 604 ?? I Mon12AM 0:00.09 /usr/local/htt
nobody 11880 0.0 0.6 496 616 ?? I Mon12AM 0:00.13 /usr/local/htt
root 4319 0.0 1.1 1176 1192 ?? Ss Tue06AM 1:58.25 named -b /etc/
root 4228 0.0 0.5 156 524 v0 Is+ Wed07PM 0:00.02 /usr/libexec/g
tlt 8188 0.0 0.6 344 708 v1 Is+ 10:48PM 0:00.32 -ksh (ksh)
root 22118 0.0 1.0 752 1148 ?? I 5:44PM 0:00.99 sendmail: RAA2
root 22213 0.0 1.0 760 1140 ?? I 5:50PM 0:00.98 sendmail: RAA2
root 22255 0.0 1.0 736 1140 ?? I 5:50PM 0:00.56 sendmail: RAA2
root 22715 0.0 1.0 764 1124 ?? I 6:46PM 0:00.82 sendmail: SAA2
root 22717 0.0 1.0 752 1128 ?? I 6:46PM 0:00.81 sendmail: SAA2
root 23139 0.0 1.0 760 1144 ?? I 6:59PM 0:00.80 sendmail: SAA2
root 23141 0.0 1.0 744 1140 ?? I 6:59PM 0:00.70 sendmail: SAA2
root 23280 0.0 1.0 760 1120 ?? S 7:10PM 0:00.83 sendmail: TAA2
root 23282 0.0 1.0 744 1120 ?? I 7:10PM 0:00.70 sendmail: TAA2
root 23530 0.0 1.0 756 1140 ?? I 7:13PM 0:00.72 sendmail: TAA2
root 23532 0.0 1.0 744 1140 ?? I 7:13PM 0:00.71 sendmail: TAA2
root 23631 0.0 1.0 760 1128 ?? S 7:22PM 0:00.76 sendmail: TAA2
root 23633 0.0 1.0 744 1108 ?? I 7:22PM 0:00.64 sendmail: TAA2
root 23727 0.0 1.0 756 1140 ?? S 7:28PM 0:00.69 sendmail: TAA2
root 23729 0.0 1.0 744 1140 ?? I 7:28PM 0:00.69 sendmail: TAA2
root 23873 0.0 1.0 744 1120 ?? S 7:47PM 0:00.63 sendmail: TAA2
root 23876 0.0 1.0 736 1112 ?? I 7:47PM 0:00.60 sendmail: TAA2
root 24331 0.0 1.0 740 1140 ?? I 8:17PM 0:00.51 sendmail: UAA2
root 24334 0.0 1.0 736 1140 ?? S 8:17PM 0:00.45 sendmail: UAA2
root 24335 0.0 1.0 712 1104 ?? S 8:17PM 0:00.34 sendmail: UAA2
root 24360 0.0 1.0 720 1128 ?? I 8:17PM 0:00.41 sendmail: UAA2
root 24584 0.0 1.0 756 1096 ?? I 8:24PM 0:00.57 sendmail: UAA2
root 24586 0.0 1.0 744 1116 ?? S 8:24PM 0:00.52 sendmail: UAA2
root 24666 0.0 0.8 776 860 ?? I 8:41PM 0:00.52 sendmail: UAA2
root 24685 0.0 1.0 756 1124 ?? I 8:43PM 0:00.57 sendmail: UAA2
root 24687 0.0 1.0 744 1104 ?? S 8:43PM 0:00.49 sendmail: UAA2
root 25117 0.0 1.0 700 1076 ?? I 9:01PM 0:00.28 sendmail: VAA2
root 25128 0.0 1.0 732 1104 ?? I 9:01PM 0:00.41 sendmail: VAA2
root 25131 0.0 1.0 732 1128 ?? I 9:01PM 0:00.35 sendmail: VAA2
root 25132 0.0 1.0 708 1088 ?? I 9:01PM 0:00.28 sendmail: VAA2
root 25173 0.0 1.0 700 1076 ?? I 9:01PM 0:00.27 sendmail: VAA2
root 25397 0.0 1.0 756 1124 ?? I 9:08PM 0:00.52 sendmail: VAA2
root 25399 0.0 1.0 744 1100 ?? S 9:08PM 0:00.48 sendmail: VAA2
root 25464 0.0 0.7 744 820 ?? I 9:11PM 0:00.38 sendmail: VAA2
root 25505 0.0 1.0 736 1108 ?? S 9:21PM 0:00.46 sendmail: VAA2
root 25508 0.0 1.0 736 1108 ?? I 9:21PM 0:00.34 sendmail: VAA2
root 25509 0.0 0.9 704 1060 ?? S 9:21PM 0:00.25 sendmail: VAA2
root 25901 0.0 1.0 732 1100 ?? I 9:27PM 0:00.37 sendmail: VAA2
root 25904 0.0 1.0 732 1104 ?? I 9:27PM 0:00.33 sendmail: VAA2
root 25905 0.0 0.9 704 1056 ?? S 9:27PM 0:00.25 sendmail: VAA2
root 25942 0.0 1.0 700 1064 ?? I 9:27PM 0:00.23 sendmail: VAA2
root 26150 0.0 0.9 700 1052 ?? S 9:27PM 0:00.22 sendmail: VAA2
root 26153 0.0 1.0 748 1112 ?? S 9:27PM 0:00.42 sendmail: VAA2
root 26155 0.0 1.0 740 1100 ?? I 9:27PM 0:00.40 sendmail: VAA2
root 26255 0.0 1.0 736 1108 ?? S 9:35PM 0:00.42 sendmail: VAA2
root 26258 0.0 1.0 736 1108 ?? I 9:35PM 0:00.34 sendmail: VAA2
root 26259 0.0 0.9 700 1052 ?? S 9:35PM 0:00.24 sendmail: VAA2
root 26654 0.0 0.8 776 860 ?? S 9:41PM 0:00.56 sendmail: VAA2
root 26688 0.0 0.9 700 1060 ?? I 9:51PM 0:00.20 sendmail: VAA2
root 26691 0.0 1.0 752 1096 ?? I 9:51PM 0:00.43 sendmail: VAA2
root 26693 0.0 1.0 736 1112 ?? S 9:51PM 0:00.35 sendmail: VAA2
root 26694 0.0 0.9 644 1020 ?? I 9:51PM 0:00.13 sendmail: VAA2
root 26695 0.0 1.0 700 1072 ?? I 9:51PM 0:00.21 sendmail: VAA2
root 26825 0.0 0.9 700 1052 ?? S 10:10PM 0:00.18 sendmail: WAA2
root 26827 0.0 0.9 644 1008 ?? I 10:10PM 0:00.13 sendmail: WAA2
root 26832 0.0 0.9 644 1016 ?? S 10:10PM 0:00.13 sendmail: WAA2
root 26835 0.0 0.9 644 1016 ?? I 10:10PM 0:00.14 sendmail: WAA2
root 26836 0.0 0.9 680 1044 ?? I 10:10PM 0:00.31 sendmail: WAA2
root 26838 0.0 0.9 696 1044 ?? I 10:10PM 0:00.15 sendmail: WAA2
root 26839 0.0 1.0 676 1072 ?? I 10:10PM 0:00.25 sendmail: WAA2
root 26840 0.0 0.9 700 1048 ?? I 10:10PM 0:00.17 sendmail: WAA2
root 26867 0.0 0.9 648 1008 ?? I 10:10PM 0:00.15 sendmail: WAA2
root 27068 0.0 0.8 764 852 ?? S 10:11PM 0:00.36 sendmail: VAA2
root 27069 0.0 0.9 644 1016 ?? I 10:11PM 0:00.13 sendmail: WAA2
root 27072 0.0 0.5 216 592 ?? S 10:13PM 0:10.99 telnetd
tlt 27073 0.0 0.6 348 680 p0 Ss 10:13PM 0:00.58 -ksh (ksh)
root 27114 0.0 0.4 600 472 ?? I 10:22PM 0:00.02 sendmail: serv
root 27256 0.0 0.4 600 484 ?? I 10:24PM 0:00.02 sendmail: serv
root 0 0.0 0.0 0 0 ?? DLs - 0:00.00 (swapper)
Here's my mailq shortened for the sake of brevity here...
Mail Queue (309 requests)
--Q-ID-- --Size-- -----Q-Time----- ------------Sender/Recipient------------
WAA27565* 318 Thu Jul 10 22:30 rcg-forum@tltodd.com
Eric.Walter@quickmail.llnl.gov
WAA26833 230 Thu Jul 10 22:10 rcg-forum@tltodd.com
(host map: lookup (hyphe.com): deferred)
jutson@hyphe.com
VAA26253* 318 Thu Jul 10 21:35 rcg-forum@tltodd.com
(wat@uniserve.com,ctolmie@uniserve.com... timeout waiting for)
ctolmie@uniserve.com
wat@uniserve.com
VAA26257* 318 Thu Jul 10 21:35 rcg-forum@tltodd.com
(dbedwell@shastalink.k12.ca.us... timeout waiting for input d)
dbedwell@shastalink.k12.ca.us
jhendric@cabin.llcc.cc.il.us
.........much more snipped off here.....
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199707111254.HAA09444>