Date: Wed, 5 May 2004 13:27:50 -0500 From: "adp" <dap99@i-55.com> To: <questions@freebsd.org> Cc: performance@freebsd.org Subject: Abnormal network errors? Message-ID: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf>
next in thread | raw e-mail | index | archive | help
I am working on some network performance issues. One of the first things I
inspected was netstat-s. This is for a FreeBSD 4.9-REL NFS and MySQL server
that has been up for 33 days. It seems to me that I have a lot of errors,
specifically with UDP (NFS related I would guess). The server is a dual P3
with 512MB of RAM, software RAID-1 (one of the Promise hybrid
hardware/kernel RAID cards), and FastEthernet. We are using just a
consumer'ish 10/100 switch. (Hope to rectify that soon enough.)
On this server I'm thinking I need two things:
1. More sockets available.
2. Larger sockbufs for send and recv.
Is this an accurate assessment?
What is "2432320 packets for unknown/unsupported protocol"? What
specifically does this mean? (In other words, what should I do to resolve
this?)
What about "921363 calls to icmp_error"?
Under tcp I have "481930 embryonic connections dropped". I assume that means
I don't have enough sockets available for when this server gets loaded.
Correct?
I am including 'ifconfig', 'netstat -m', 'netstat -s', 'nfsstat -s', 'sysctl
net' and
'sysctl kern.ipc' below.
Is there a way to see the peak sockets I have had opened? It doesn't look
like it.
P.S. I am Cc'ing performance@ since this is both a general administration
and performance-related question.
# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 192.168.42.70 netmask 0xffffffe0 broadcast 192.168.42.95
..multiple aliases..
ether 00:50:ba:60:4d:e5
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
# netstat -m
34/736/10112 mbufs in use (current/peak/max):
34 mbufs allocated to data
0/504/2528 mbuf clusters in use (current/peak/max)
1192 Kbytes allocated to network (15% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
# netstat -s
tcp:
332918288 packets sent
279842528 data packets (1533896593 bytes)
184127 data packets (216536937 bytes) retransmitted
2 resends initiated by MTU discovery
41010478 ack-only packets (14887613 delayed)
0 URG only packets
12197 window probe packets
4899618 window update packets
6969381 control packets
298154198 packets received
237883532 acks (for 1547823911 bytes)
5548105 duplicate acks
0 acks for unsent data
194397219 packets (612913715 bytes) received in-sequence
175092 completely duplicate packets (40526143 bytes)
386 old duplicate packets
7879 packets with some dup. data (1540154 bytes duped)
1113196 out-of-order packets (1061944624 bytes)
5645 packets (6794550 bytes) of data after window
209 window probes
3540787 window update packets
217721 packets received after close
2925 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
1450821 connection requests
5141899 connection accepts
1739 bad connection attempts
0 listen queue overflows
5676325 connections established (including accepts)
6592771 connections closed (including 153553 drops)
2325245 connections updated cached RTT on close
2325245 connections updated cached RTT variance on close
2124129 connections updated cached ssthresh on close
481930 embryonic connections dropped
232282251 segments updated rtt (of 188115355 attempts)
2843613 retransmit timeouts
829 connections dropped by rexmit timeout
25922 persist timeouts
0 connections dropped by persist timeout
528 keepalive timeouts
111 keepalive probes sent
417 connections dropped by keepalive
10677309 correct ACK header predictions
40631044 correct data packet header predictions
5143661 syncache entries added
10004 retransmitted
6951 dupsyn
3 dropped
5141899 completed
0 bucket overflow
0 cache overflow
382 reset
1327 stale
0 aborted
0 badack
53 unreach
0 zone failures
0 cookies sent
0 cookies received
udp:
272987897 datagrams received
0 with incomplete header
0 with bad data length field
870 with bad checksum
682 with no checksum
921363 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
19976574 dropped due to full socket buffers
0 not for hashed pcb
252089090 delivered
275246074 datagrams output
ip:
578001924 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
4899083 fragments received
4 fragments dropped (dup or out of space)
750 fragments dropped after timeout
842689 packets reassembled ok
571513013 packets for this host
2432320 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
197 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
609745425 packets sent from this host
1 packet sent with fabricated ip header
22 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
1914687 output datagrams fragmented
10496350 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
icmp:
921363 calls to icmp_error
0 errors not generated 'cuz old message was icmp
Output histogram:
echo reply: 347036
destination unreachable: 921363
0 messages with bad code fields
0 messages < minimum length
1183 bad checksums
0 messages with bad length
2 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echo reply: 10453
destination unreachable: 2443161
source quench: 47
echo: 347038
time exceeded: 1354
347036 message responses generated
0 invalid return addresses
0 no return routes
ICMP address mask responses are disabled
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
netstat: sysctl: net.inet.pim.stats: No such file or directory
# nfsstat -s
Server Info:
Getattr Setattr Lookup Readlink Read Write Create
Remove
501 199150 80001872 280 4426274 2330054 267945
458716
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus
Access
361427 190173 0 1977 451 1803697 0
114714817
Mknod Fsstat Fsinfo PathConf Commit GLease Vacate
Evict
0 4846498 234 0 398335 0 0
0
Server Ret-Failed
76071546
Server Faults
0
Server Cache Stats:
Inprog Idem Non-idem Misses
341182 97307 1569 209905447
Server Lease Stats:
Leases PeakL GLeases
0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved
2330050 2330054 4
# sysctl net
net.local.stream.sendspace: 8192
net.local.stream.recvspace: 8192
net.local.dgram.maxdgram: 2048
net.local.dgram.recvspace: 4096
net.local.inflight: 0
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 1024
net.inet.ip.portrange.last: 5000
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 27
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 50
net.inet.ip.intr_queue_drops: 121
net.inet.ip.accept_sourceroute: 0
net.inet.ip.fastforwarding: 0
net.inet.ip.keepfaith: 0
net.inet.ip.subnets_are_local: 0
net.inet.ip.maxfragpackets: 79
net.inet.ip.maxfragsperpacket: 16
net.inet.ip.sendsourcequench: 0
net.inet.ip.check_interface: 0
net.inet.ip.fw.enable: 1
net.inet.ip.fw.one_pass: 1
net.inet.ip.fw.debug: 1
net.inet.ip.fw.verbose: 0
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_count: 0
net.inet.ip.fw.dyn_max: 1000
net.inet.ip.fw.static_count: 5
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_grace_time: 10
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 200
net.inet.icmp.drop_redirect: 0
net.inet.icmp.log_redirect: 0
net.inet.icmp.icmplim_output: 1
net.inet.icmp.bmcastecho: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.rfc1644: 0
net.inet.tcp.mssdflt: 512
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 32768
net.inet.tcp.recvspace: 57344
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.log_in_vain: 0
net.inet.tcp.blackhole: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.reass.maxsegments: 158
net.inet.tcp.reass.cursegments: 0
net.inet.tcp.reass.overflows: 0
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.newreno: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.pcbcount: 100
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.inflight_enable: 0
net.inet.tcp.inflight_debug: 0
net.inet.tcp.inflight_min: 6144
net.inet.tcp.inflight_max: 1073725440
net.inet.tcp.inflight_stab: 20
net.inet.tcp.syncookies: 1
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncache.cachelimit: 15359
net.inet.tcp.syncache.count: 0
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.msl: 30000
net.inet.tcp.rexmit_min: 1000
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.always_keepalive: 1
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 41600
net.inet.udp.log_in_vain: 0
net.inet.udp.blackhole: 0
net.inet.accf.unloadable: 0
net.inet.raw.maxdgram: 8192
net.inet.raw.recvspace: 8192
net.link.generic.system.ifcount: 3
net.link.ether.inet.prune_intvl: 300
net.link.ether.inet.max_age: 1200
net.link.ether.inet.host_down_time: 20
net.link.ether.inet.maxtries: 5
net.link.ether.inet.useloopback: 1
net.link.ether.inet.proxyall: 0
net.link.ether.inet.log_arp_wrong_iface: 1
net.link.ether.inet.log_arp_movements: 1
net.link.ether.ipfw: 0
# sysctl kern.ipc
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 128
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 40
kern.ipc.max_hdr: 56
kern.ipc.max_datalen: 156
kern.ipc.nmbclusters: 2528
kern.ipc.msgmax: 16384
kern.ipc.msgmni: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgtql: 40
kern.ipc.msgssz: 8
kern.ipc.msgseg: 2048
kern.ipc.semmap: 30
kern.ipc.semmni: 10
kern.ipc.semmns: 60
kern.ipc.semmnu: 30
kern.ipc.semmsl: 60
kern.ipc.semopm: 100
kern.ipc.semume: 10
kern.ipc.semusz: 92
kern.ipc.semvmx: 32767
kern.ipc.semaem: 16384
kern.ipc.shmmax: 33554432
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 8192
kern.ipc.shm_use_phys: 0
kern.ipc.shm_allow_removed: 0
kern.ipc.mbuf_wait: 32
kern.ipc.mbtypes: 689 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0
kern.ipc.nmbufs: 10112
kern.ipc.m_clreflimithits: 0
kern.ipc.mcl_pool_max: 0
kern.ipc.mcl_pool_now: 0
kern.ipc.maxsockets: 4072
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01a501c432ce$adf20e30$4b0a000a>