From owner-freebsd-questions@FreeBSD.ORG Wed Jul 30 09:32:02 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B16937B401; Wed, 30 Jul 2003 09:32:02 -0700 (PDT) Received: from mx01.bos.ma.towardex.com (a65-124-16-8.svc.towardex.com [65.124.16.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1A7B343FAF; Wed, 30 Jul 2003 09:32:01 -0700 (PDT) (envelope-from haesu@mx01.bos.ma.towardex.com) Received: by mx01.bos.ma.towardex.com (TowardEX ESMTP 3.0p11_DAKN, from userid 1001) id D3CE92F893; Wed, 30 Jul 2003 12:32:02 -0400 (EDT) Date: Wed, 30 Jul 2003 12:32:02 -0400 From: Haesu To: freebsd-isp@freebsd.org, freebsd-questions@freebsd.org Message-ID: <20030730163202.GA8118@scylla.towardex.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i Subject: sendto: No buffer space available X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2003 16:32:02 -0000 Hello, We have a FreeBSD box here that we use to route some GRE tunnels and ipv6 gif tunnels. We use zebra for dynamic routing running zebra, bgpd, ospfd, and ospf6d. We have about 12 FreeBSD boxes with exact same configuration, the only difference is just the IP address of each interface. None of them fail but this one box... Everyday, this box stops all networking. I can still console in and stuff.. When I typed 'ping 127.0.0.1' at the console after networking locked up, it says: ping: sendto: No buffer space available The only solution seems to be rebooting it everyday... It happens every 12 hours or so... This is not related with mbuf, etc either, as netstat -m doesn't show any issues. The box has one IP address and IPv6 address in addition to 127.0.0.1 on lo0 interface. It also has a ds0 interface with 10.5.5.5/30 assigned to ds0. This is exact same configuration on all other boxes, and none of them fail but this one. I've swapped out NICs with different vendors 3 times (tried, xl, dc, and now rl) I've also swapped out the whole box, and also swapped out the whole hard drive and did full reinstall. And problem still persists and it's definately not hardware as I swapped everything out... (unless the 3 NIC vendors above are all exhibiting same issue) I tried to look on Google but nothing useful that corelates to this particular issue.. Any help would be very appreciated :) Thanks, -hc The box is running FreeBSD 4.8-STABLE FreeBSD necsis 4.8-STABLE FreeBSD 4.8-STABLE #0: Tue Jul 29 13:10:11 GMT 2003 root@necsis:/usr/src/sys/compile/router i386 Following is output of netstat -s AFTER the networking locks up with no buffer space available error: tcp: 30115 packets sent 17167 data packets (1232152 bytes) 301 data packets (54320 bytes) retransmitted 0 resends initiated by MTU discovery 12416 ack-only packets (10931 delayed) 0 URG only packets 0 window probe packets 41 window update packets 280 control packets 28010 packets received 16762 acks (for 1236693 bytes) 140 duplicate acks 0 acks for unsent data 13205 packets (567038 bytes) received in-sequence 43 completely duplicate packets (818 bytes) 0 old duplicate packets 2 packets with some dup. data (38 bytes duped) 9 out-of-order packets (240 bytes) 0 packets (0 bytes) of data after window 0 window probes 31 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 252 connection requests 18 connection accepts 6 bad connection attempts 0 listen queue overflows 30 connections established (including accepts) 288 connections closed (including 10 drops) 23 connections updated cached RTT on close 23 connections updated cached RTT variance on close 11 connections updated cached ssthresh on close 164 embryonic connections dropped 16643 segments updated rtt (of 16929 attempts) 1566 retransmit timeouts 10 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 161 keepalive timeouts 0 keepalive probes sent 161 connections dropped by keepalive 96 correct ACK header predictions 10392 correct data packet header predictions 19 syncache entries added 6 retransmitted 2 dupsyn 0 dropped 18 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 1 unreach 0 zone failures 0 cookies sent 0 cookies received udp: 196 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 1 with no checksum 61 dropped due to no socket 3 broadcast/multicast datagrams dropped due to no socket 0 dropped due to full socket buffers 0 not for hashed pcb 132 delivered 132 datagrams output ip: 2154646 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 366 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 183 packets reassembled ok 1118217 packets for this host 11 packets for unknown/unsupported protocol 1031566 packets forwarded (0 packets fast forwarded) 265 packets not forwardable 3573 packets received for unknown multicast group 0 redirects sent 772758 packets sent from this host 13697 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 366 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 1075 calls to icmp_error 0 errors not generated 'cuz old message was icmp Output histogram: echo reply: 28 destination unreachable: 244 time exceeded: 831 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 21 echo: 28 time exceeded: 111 28 message responses generated 1 invalid return address 1 no return route ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 9 membership reports sent ip6: 26952 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 20061 packets for this host 35 packets forwarded 108 packets not forwardable 0 redirects sent 29834 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 82 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 108 multicast packets which we don't join Input histogram: TCP: 12122 UDP: 12 ICMP6: 6761 OSPF: 8057 Mbuf statistics: 26308 one mbuf 644 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not continuous 0 tunneling packets that can't find gif 0 packets discarded due to too may headers 0 failures of source address selection source addresses on an outgoing I/F 4 link-locals 44 globals source addresses of same scope 4 link-locals 44 globals 9346 forward cache hit 10034 forward cache miss icmp6: 0 calls to icmp_error 0 errors not generated because old message was icmp error or so 0 errors not generated because rate limitation Output histogram: multicast listener report: 49 multicast listener done: 4 neighbor solicitation: 4265 neighbor advertisement: 3239 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: unreach: 3 neighbor solicitation: 3239 neighbor advertisement: 3506 Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes -- Sincerely, Haesu C. TowardEX Technologies, Inc. WWW: http://www.towardex.com E-mail: haesu@towardex.com Cell: (978) 394-2867