From owner-freebsd-questions@FreeBSD.ORG Wed May 5 11:28:10 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9260F16A4CE; Wed, 5 May 2004 11:28:10 -0700 (PDT) Received: from watcher.puryear-it.com (ip-66-186-248-99.eatel.net [66.186.248.99]) by mx1.FreeBSD.org (Postfix) with ESMTP id 89A9843D1D; Wed, 5 May 2004 11:28:09 -0700 (PDT) (envelope-from dap99@i-55.com) Received: from localhost (unknown [127.0.0.1]) by watcher.puryear-it.com (Postfix) with ESMTP id D77DB34D1F; Wed, 5 May 2004 13:26:48 -0500 (CDT) Received: from watcher.puryear-it.com ([127.0.0.1]) by localhost (watcher.puryear-it.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 96954-05; Wed, 5 May 2004 13:26:47 -0500 (CDT) Received: from yourqqh4336axf (localhost [127.0.0.1]) by watcher.puryear-it.com (Postfix) with SMTP id 4D0BD34D1E; Wed, 5 May 2004 13:26:46 -0500 (CDT) Message-ID: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> From: "adp" To: Date: Wed, 5 May 2004 13:27:50 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2739.300 X-Virus-Scanned: by amavisd-new cc: performance@freebsd.org Subject: Abnormal network errors? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2004 18:28:10 -0000 I am working on some network performance issues. One of the first things I inspected was netstat-s. This is for a FreeBSD 4.9-REL NFS and MySQL server that has been up for 33 days. It seems to me that I have a lot of errors, specifically with UDP (NFS related I would guess). The server is a dual P3 with 512MB of RAM, software RAID-1 (one of the Promise hybrid hardware/kernel RAID cards), and FastEthernet. We are using just a consumer'ish 10/100 switch. (Hope to rectify that soon enough.) On this server I'm thinking I need two things: 1. More sockets available. 2. Larger sockbufs for send and recv. Is this an accurate assessment? What is "2432320 packets for unknown/unsupported protocol"? What specifically does this mean? (In other words, what should I do to resolve this?) What about "921363 calls to icmp_error"? Under tcp I have "481930 embryonic connections dropped". I assume that means I don't have enough sockets available for when this server gets loaded. Correct? I am including 'ifconfig', 'netstat -m', 'netstat -s', 'nfsstat -s', 'sysctl net' and 'sysctl kern.ipc' below. Is there a way to see the peak sockets I have had opened? It doesn't look like it. P.S. I am Cc'ing performance@ since this is both a general administration and performance-related question. # ifconfig rl0 rl0: flags=8843 mtu 1500 inet 192.168.42.70 netmask 0xffffffe0 broadcast 192.168.42.95 ..multiple aliases.. ether 00:50:ba:60:4d:e5 media: Ethernet autoselect (100baseTX ) status: active # netstat -m 34/736/10112 mbufs in use (current/peak/max): 34 mbufs allocated to data 0/504/2528 mbuf clusters in use (current/peak/max) 1192 Kbytes allocated to network (15% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines # netstat -s tcp: 332918288 packets sent 279842528 data packets (1533896593 bytes) 184127 data packets (216536937 bytes) retransmitted 2 resends initiated by MTU discovery 41010478 ack-only packets (14887613 delayed) 0 URG only packets 12197 window probe packets 4899618 window update packets 6969381 control packets 298154198 packets received 237883532 acks (for 1547823911 bytes) 5548105 duplicate acks 0 acks for unsent data 194397219 packets (612913715 bytes) received in-sequence 175092 completely duplicate packets (40526143 bytes) 386 old duplicate packets 7879 packets with some dup. data (1540154 bytes duped) 1113196 out-of-order packets (1061944624 bytes) 5645 packets (6794550 bytes) of data after window 209 window probes 3540787 window update packets 217721 packets received after close 2925 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1450821 connection requests 5141899 connection accepts 1739 bad connection attempts 0 listen queue overflows 5676325 connections established (including accepts) 6592771 connections closed (including 153553 drops) 2325245 connections updated cached RTT on close 2325245 connections updated cached RTT variance on close 2124129 connections updated cached ssthresh on close 481930 embryonic connections dropped 232282251 segments updated rtt (of 188115355 attempts) 2843613 retransmit timeouts 829 connections dropped by rexmit timeout 25922 persist timeouts 0 connections dropped by persist timeout 528 keepalive timeouts 111 keepalive probes sent 417 connections dropped by keepalive 10677309 correct ACK header predictions 40631044 correct data packet header predictions 5143661 syncache entries added 10004 retransmitted 6951 dupsyn 3 dropped 5141899 completed 0 bucket overflow 0 cache overflow 382 reset 1327 stale 0 aborted 0 badack 53 unreach 0 zone failures 0 cookies sent 0 cookies received udp: 272987897 datagrams received 0 with incomplete header 0 with bad data length field 870 with bad checksum 682 with no checksum 921363 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 19976574 dropped due to full socket buffers 0 not for hashed pcb 252089090 delivered 275246074 datagrams output ip: 578001924 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 4899083 fragments received 4 fragments dropped (dup or out of space) 750 fragments dropped after timeout 842689 packets reassembled ok 571513013 packets for this host 2432320 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 197 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 609745425 packets sent from this host 1 packet sent with fabricated ip header 22 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 1914687 output datagrams fragmented 10496350 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 921363 calls to icmp_error 0 errors not generated 'cuz old message was icmp Output histogram: echo reply: 347036 destination unreachable: 921363 0 messages with bad code fields 0 messages < minimum length 1183 bad checksums 0 messages with bad length 2 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: echo reply: 10453 destination unreachable: 2443161 source quench: 47 echo: 347038 time exceeded: 1354 347036 message responses generated 0 invalid return addresses 0 no return routes ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent netstat: sysctl: net.inet.pim.stats: No such file or directory # nfsstat -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 501 199150 80001872 280 4426274 2330054 267945 458716 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 361427 190173 0 1977 451 1803697 0 114714817 Mknod Fsstat Fsinfo PathConf Commit GLease Vacate Evict 0 4846498 234 0 398335 0 0 0 Server Ret-Failed 76071546 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 341182 97307 1569 209905447 Server Lease Stats: Leases PeakL GLeases 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 2330050 2330054 4 # sysctl net net.local.stream.sendspace: 8192 net.local.stream.recvspace: 8192 net.local.dgram.maxdgram: 2048 net.local.dgram.recvspace: 4096 net.local.inflight: 0 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 1024 net.inet.ip.portrange.last: 5000 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535 net.inet.ip.forwarding: 0 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 27 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 121 net.inet.ip.accept_sourceroute: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.keepfaith: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.maxfragpackets: 79 net.inet.ip.maxfragsperpacket: 16 net.inet.ip.sendsourcequench: 0 net.inet.ip.check_interface: 0 net.inet.ip.fw.enable: 1 net.inet.ip.fw.one_pass: 1 net.inet.ip.fw.debug: 1 net.inet.ip.fw.verbose: 0 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_count: 0 net.inet.ip.fw.dyn_max: 1000 net.inet.ip.fw.static_count: 5 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_grace_time: 10 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 200 net.inet.icmp.drop_redirect: 0 net.inet.icmp.log_redirect: 0 net.inet.icmp.icmplim_output: 1 net.inet.icmp.bmcastecho: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.rfc1644: 0 net.inet.tcp.mssdflt: 512 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 57344 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.reass.maxsegments: 158 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.overflows: 0 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.newreno: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 100 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.inflight_enable: 0 net.inet.tcp.inflight_debug: 0 net.inet.tcp.inflight_min: 6144 net.inet.tcp.inflight_max: 1073725440 net.inet.tcp.inflight_stab: 20 net.inet.tcp.syncookies: 1 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.msl: 30000 net.inet.tcp.rexmit_min: 1000 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.always_keepalive: 1 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 41600 net.inet.udp.log_in_vain: 0 net.inet.udp.blackhole: 0 net.inet.accf.unloadable: 0 net.inet.raw.maxdgram: 8192 net.inet.raw.recvspace: 8192 net.link.generic.system.ifcount: 3 net.link.ether.inet.prune_intvl: 300 net.link.ether.inet.max_age: 1200 net.link.ether.inet.host_down_time: 20 net.link.ether.inet.maxtries: 5 net.link.ether.inet.useloopback: 1 net.link.ether.inet.proxyall: 0 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.ipfw: 0 # sysctl kern.ipc kern.ipc.maxsockbuf: 262144 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 128 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 40 kern.ipc.max_hdr: 56 kern.ipc.max_datalen: 156 kern.ipc.nmbclusters: 2528 kern.ipc.msgmax: 16384 kern.ipc.msgmni: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgtql: 40 kern.ipc.msgssz: 8 kern.ipc.msgseg: 2048 kern.ipc.semmap: 30 kern.ipc.semmni: 10 kern.ipc.semmns: 60 kern.ipc.semmnu: 30 kern.ipc.semmsl: 60 kern.ipc.semopm: 100 kern.ipc.semume: 10 kern.ipc.semusz: 92 kern.ipc.semvmx: 32767 kern.ipc.semaem: 16384 kern.ipc.shmmax: 33554432 kern.ipc.shmmin: 1 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.shmall: 8192 kern.ipc.shm_use_phys: 0 kern.ipc.shm_allow_removed: 0 kern.ipc.mbuf_wait: 32 kern.ipc.mbtypes: 689 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 kern.ipc.nmbufs: 10112 kern.ipc.m_clreflimithits: 0 kern.ipc.mcl_pool_max: 0 kern.ipc.mcl_pool_now: 0 kern.ipc.maxsockets: 4072