Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 30 Jul 2003 12:32:02 -0400
From:      Haesu <haesu@towardex.com>
To:        freebsd-isp@freebsd.org, freebsd-questions@freebsd.org
Subject:   sendto: No buffer space available
Message-ID:  <20030730163202.GA8118@scylla.towardex.com>

next in thread | raw e-mail | index | archive | help
Hello,

We have a FreeBSD box here that we use to route some GRE tunnels and ipv6 gif tunnels. We use zebra for dynamic routing running zebra, bgpd, ospfd, and ospf6d.

We have about 12 FreeBSD boxes with exact same configuration, the only
difference is just the IP address of each interface.

None of them fail but this one box...

Everyday, this box stops all networking. I can still console in and stuff.. When
I typed 'ping 127.0.0.1' at the console after networking locked up, it says:
ping: sendto: No buffer space available

The only solution seems to be rebooting it everyday... It happens every 12 hours
or so...

This is not related with mbuf, etc either, as netstat -m doesn't show any
issues.

The box has one IP address and IPv6 address in addition to 127.0.0.1 on lo0
interface. It also has a ds0 interface with 10.5.5.5/30 assigned to ds0.
This is exact same configuration on all other boxes, and none of them fail but
this one.

I've swapped out NICs with different vendors 3 times (tried, xl, dc, and now rl)

I've also swapped out the whole box, and also swapped out the whole hard drive
and did full reinstall. And problem still persists and it's definately not
hardware as I swapped everything out... (unless the 3 NIC vendors above are all
exhibiting same issue)

I tried to look on Google but nothing useful that corelates to this particular
issue..

Any help would be very appreciated :)

Thanks,
-hc

The box is running FreeBSD 4.8-STABLE
FreeBSD necsis 4.8-STABLE FreeBSD 4.8-STABLE #0: Tue Jul 29 13:10:11 GMT 2003     root@necsis:/usr/src/sys/compile/router  i386

Following is output of netstat -s AFTER the networking locks up with no buffer
space available error:

tcp:
        30115 packets sent
                17167 data packets (1232152 bytes)
                301 data packets (54320 bytes) retransmitted
                0 resends initiated by MTU discovery
                12416 ack-only packets (10931 delayed)
                0 URG only packets
                0 window probe packets
                41 window update packets
                280 control packets
        28010 packets received
                16762 acks (for 1236693 bytes)
                140 duplicate acks
                0 acks for unsent data
                13205 packets (567038 bytes) received in-sequence
                43 completely duplicate packets (818 bytes)
                0 old duplicate packets
                2 packets with some dup. data (38 bytes duped)
                9 out-of-order packets (240 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                31 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        252 connection requests
        18 connection accepts
        6 bad connection attempts
        0 listen queue overflows
        30 connections established (including accepts)
        288 connections closed (including 10 drops)
                23 connections updated cached RTT on close
                23 connections updated cached RTT variance on close
                11 connections updated cached ssthresh on close
        164 embryonic connections dropped
        16643 segments updated rtt (of 16929 attempts)
        1566 retransmit timeouts
                10 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        161 keepalive timeouts
                0 keepalive probes sent
                161 connections dropped by keepalive
        96 correct ACK header predictions
        10392 correct data packet header predictions
        19 syncache entries added
                6 retransmitted
                2 dupsyn
                0 dropped
                18 completed
                0 bucket overflow
                0 cache overflow
                0 reset
                0 stale
                0 aborted
                0 badack
                1 unreach
                0 zone failures
        0 cookies sent
        0 cookies received
udp:
        196 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        1 with no checksum
        61 dropped due to no socket
        3 broadcast/multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        0 not for hashed pcb
        132 delivered
        132 datagrams output
ip:
        2154646 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        366 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        183 packets reassembled ok
        1118217 packets for this host
        11 packets for unknown/unsupported protocol
        1031566 packets forwarded (0 packets fast forwarded)
        265 packets not forwardable
        3573 packets received for unknown multicast group
        0 redirects sent
        772758 packets sent from this host
        13697 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        366 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        1075 calls to icmp_error
        0 errors not generated 'cuz old message was icmp
        Output histogram:
                echo reply: 28
                destination unreachable: 244
                time exceeded: 831
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                destination unreachable: 21
                echo: 28
                time exceeded: 111
        28 message responses generated
        1 invalid return address
        1 no return route
        ICMP address mask responses are disabled
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        9 membership reports sent
ip6:
        26952 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        20061 packets for this host
        35 packets forwarded
        108 packets not forwardable
        0 redirects sent
        29834 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        82 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        108 multicast packets which we don't join
        Input histogram:
                TCP: 12122
                UDP: 12
                ICMP6: 6761
                OSPF: 8057
        Mbuf statistics:
                26308 one mbuf
                644 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not continuous
        0 tunneling packets that can't find gif
        0 packets discarded due to too may headers
        0 failures of source address selection
        source addresses on an outgoing I/F
                4 link-locals
                44 globals
        source addresses of same scope
                4 link-locals
                44 globals
        9346 forward cache hit
        10034 forward cache miss
icmp6:
        0 calls to icmp_error
        0 errors not generated because old message was icmp error or so
        0 errors not generated because rate limitation
        Output histogram:
                multicast listener report: 49
                multicast listener done: 4
                neighbor solicitation: 4265
                neighbor advertisement: 3239
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Input histogram:
                unreach: 3
                neighbor solicitation: 3239
                neighbor advertisement: 3506
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 path MTU changes



-- 
Sincerely,
  Haesu C.
  TowardEX Technologies, Inc.
  WWW: http://www.towardex.com
  E-mail: haesu@towardex.com
  Cell: (978) 394-2867



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030730163202.GA8118>