From owner-freebsd-performance@FreeBSD.ORG Wed May 5 11:28:10 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9260F16A4CE; Wed, 5 May 2004 11:28:10 -0700 (PDT) Received: from watcher.puryear-it.com (ip-66-186-248-99.eatel.net [66.186.248.99]) by mx1.FreeBSD.org (Postfix) with ESMTP id 89A9843D1D; Wed, 5 May 2004 11:28:09 -0700 (PDT) (envelope-from dap99@i-55.com) Received: from localhost (unknown [127.0.0.1]) by watcher.puryear-it.com (Postfix) with ESMTP id D77DB34D1F; Wed, 5 May 2004 13:26:48 -0500 (CDT) Received: from watcher.puryear-it.com ([127.0.0.1]) by localhost (watcher.puryear-it.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 96954-05; Wed, 5 May 2004 13:26:47 -0500 (CDT) Received: from yourqqh4336axf (localhost [127.0.0.1]) by watcher.puryear-it.com (Postfix) with SMTP id 4D0BD34D1E; Wed, 5 May 2004 13:26:46 -0500 (CDT) Message-ID: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> From: "adp" To: Date: Wed, 5 May 2004 13:27:50 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2739.300 X-Virus-Scanned: by amavisd-new cc: performance@freebsd.org Subject: Abnormal network errors? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2004 18:28:10 -0000 I am working on some network performance issues. One of the first things I inspected was netstat-s. This is for a FreeBSD 4.9-REL NFS and MySQL server that has been up for 33 days. It seems to me that I have a lot of errors, specifically with UDP (NFS related I would guess). The server is a dual P3 with 512MB of RAM, software RAID-1 (one of the Promise hybrid hardware/kernel RAID cards), and FastEthernet. We are using just a consumer'ish 10/100 switch. (Hope to rectify that soon enough.) On this server I'm thinking I need two things: 1. More sockets available. 2. Larger sockbufs for send and recv. Is this an accurate assessment? What is "2432320 packets for unknown/unsupported protocol"? What specifically does this mean? (In other words, what should I do to resolve this?) What about "921363 calls to icmp_error"? Under tcp I have "481930 embryonic connections dropped". I assume that means I don't have enough sockets available for when this server gets loaded. Correct? I am including 'ifconfig', 'netstat -m', 'netstat -s', 'nfsstat -s', 'sysctl net' and 'sysctl kern.ipc' below. Is there a way to see the peak sockets I have had opened? It doesn't look like it. P.S. I am Cc'ing performance@ since this is both a general administration and performance-related question. # ifconfig rl0 rl0: flags=8843 mtu 1500 inet 192.168.42.70 netmask 0xffffffe0 broadcast 192.168.42.95 ..multiple aliases.. ether 00:50:ba:60:4d:e5 media: Ethernet autoselect (100baseTX ) status: active # netstat -m 34/736/10112 mbufs in use (current/peak/max): 34 mbufs allocated to data 0/504/2528 mbuf clusters in use (current/peak/max) 1192 Kbytes allocated to network (15% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines # netstat -s tcp: 332918288 packets sent 279842528 data packets (1533896593 bytes) 184127 data packets (216536937 bytes) retransmitted 2 resends initiated by MTU discovery 41010478 ack-only packets (14887613 delayed) 0 URG only packets 12197 window probe packets 4899618 window update packets 6969381 control packets 298154198 packets received 237883532 acks (for 1547823911 bytes) 5548105 duplicate acks 0 acks for unsent data 194397219 packets (612913715 bytes) received in-sequence 175092 completely duplicate packets (40526143 bytes) 386 old duplicate packets 7879 packets with some dup. data (1540154 bytes duped) 1113196 out-of-order packets (1061944624 bytes) 5645 packets (6794550 bytes) of data after window 209 window probes 3540787 window update packets 217721 packets received after close 2925 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1450821 connection requests 5141899 connection accepts 1739 bad connection attempts 0 listen queue overflows 5676325 connections established (including accepts) 6592771 connections closed (including 153553 drops) 2325245 connections updated cached RTT on close 2325245 connections updated cached RTT variance on close 2124129 connections updated cached ssthresh on close 481930 embryonic connections dropped 232282251 segments updated rtt (of 188115355 attempts) 2843613 retransmit timeouts 829 connections dropped by rexmit timeout 25922 persist timeouts 0 connections dropped by persist timeout 528 keepalive timeouts 111 keepalive probes sent 417 connections dropped by keepalive 10677309 correct ACK header predictions 40631044 correct data packet header predictions 5143661 syncache entries added 10004 retransmitted 6951 dupsyn 3 dropped 5141899 completed 0 bucket overflow 0 cache overflow 382 reset 1327 stale 0 aborted 0 badack 53 unreach 0 zone failures 0 cookies sent 0 cookies received udp: 272987897 datagrams received 0 with incomplete header 0 with bad data length field 870 with bad checksum 682 with no checksum 921363 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 19976574 dropped due to full socket buffers 0 not for hashed pcb 252089090 delivered 275246074 datagrams output ip: 578001924 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 4899083 fragments received 4 fragments dropped (dup or out of space) 750 fragments dropped after timeout 842689 packets reassembled ok 571513013 packets for this host 2432320 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 197 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 609745425 packets sent from this host 1 packet sent with fabricated ip header 22 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 1914687 output datagrams fragmented 10496350 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 921363 calls to icmp_error 0 errors not generated 'cuz old message was icmp Output histogram: echo reply: 347036 destination unreachable: 921363 0 messages with bad code fields 0 messages < minimum length 1183 bad checksums 0 messages with bad length 2 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: echo reply: 10453 destination unreachable: 2443161 source quench: 47 echo: 347038 time exceeded: 1354 347036 message responses generated 0 invalid return addresses 0 no return routes ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent netstat: sysctl: net.inet.pim.stats: No such file or directory # nfsstat -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 501 199150 80001872 280 4426274 2330054 267945 458716 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 361427 190173 0 1977 451 1803697 0 114714817 Mknod Fsstat Fsinfo PathConf Commit GLease Vacate Evict 0 4846498 234 0 398335 0 0 0 Server Ret-Failed 76071546 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 341182 97307 1569 209905447 Server Lease Stats: Leases PeakL GLeases 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 2330050 2330054 4 # sysctl net net.local.stream.sendspace: 8192 net.local.stream.recvspace: 8192 net.local.dgram.maxdgram: 2048 net.local.dgram.recvspace: 4096 net.local.inflight: 0 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 1024 net.inet.ip.portrange.last: 5000 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535 net.inet.ip.forwarding: 0 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 27 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 121 net.inet.ip.accept_sourceroute: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.keepfaith: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.maxfragpackets: 79 net.inet.ip.maxfragsperpacket: 16 net.inet.ip.sendsourcequench: 0 net.inet.ip.check_interface: 0 net.inet.ip.fw.enable: 1 net.inet.ip.fw.one_pass: 1 net.inet.ip.fw.debug: 1 net.inet.ip.fw.verbose: 0 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_count: 0 net.inet.ip.fw.dyn_max: 1000 net.inet.ip.fw.static_count: 5 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_grace_time: 10 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 200 net.inet.icmp.drop_redirect: 0 net.inet.icmp.log_redirect: 0 net.inet.icmp.icmplim_output: 1 net.inet.icmp.bmcastecho: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.rfc1644: 0 net.inet.tcp.mssdflt: 512 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 57344 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.reass.maxsegments: 158 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.overflows: 0 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.newreno: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 100 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.inflight_enable: 0 net.inet.tcp.inflight_debug: 0 net.inet.tcp.inflight_min: 6144 net.inet.tcp.inflight_max: 1073725440 net.inet.tcp.inflight_stab: 20 net.inet.tcp.syncookies: 1 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.msl: 30000 net.inet.tcp.rexmit_min: 1000 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.always_keepalive: 1 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 41600 net.inet.udp.log_in_vain: 0 net.inet.udp.blackhole: 0 net.inet.accf.unloadable: 0 net.inet.raw.maxdgram: 8192 net.inet.raw.recvspace: 8192 net.link.generic.system.ifcount: 3 net.link.ether.inet.prune_intvl: 300 net.link.ether.inet.max_age: 1200 net.link.ether.inet.host_down_time: 20 net.link.ether.inet.maxtries: 5 net.link.ether.inet.useloopback: 1 net.link.ether.inet.proxyall: 0 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.ipfw: 0 # sysctl kern.ipc kern.ipc.maxsockbuf: 262144 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 128 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 40 kern.ipc.max_hdr: 56 kern.ipc.max_datalen: 156 kern.ipc.nmbclusters: 2528 kern.ipc.msgmax: 16384 kern.ipc.msgmni: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgtql: 40 kern.ipc.msgssz: 8 kern.ipc.msgseg: 2048 kern.ipc.semmap: 30 kern.ipc.semmni: 10 kern.ipc.semmns: 60 kern.ipc.semmnu: 30 kern.ipc.semmsl: 60 kern.ipc.semopm: 100 kern.ipc.semume: 10 kern.ipc.semusz: 92 kern.ipc.semvmx: 32767 kern.ipc.semaem: 16384 kern.ipc.shmmax: 33554432 kern.ipc.shmmin: 1 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.shmall: 8192 kern.ipc.shm_use_phys: 0 kern.ipc.shm_allow_removed: 0 kern.ipc.mbuf_wait: 32 kern.ipc.mbtypes: 689 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 kern.ipc.nmbufs: 10112 kern.ipc.m_clreflimithits: 0 kern.ipc.mcl_pool_max: 0 kern.ipc.mcl_pool_now: 0 kern.ipc.maxsockets: 4072 From owner-freebsd-performance@FreeBSD.ORG Wed May 5 12:33:54 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 60A6A16A4CE; Wed, 5 May 2004 12:33:54 -0700 (PDT) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.88]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCDD243D1F; Wed, 5 May 2004 12:33:53 -0700 (PDT) (envelope-from cswiger@mac.com) Received: from mac.com (smtpin08-en2 [10.13.10.153]) by smtpout.mac.com (Xserve/MantshX 2.0) with ESMTP id i45JXrqs008698; Wed, 5 May 2004 12:33:53 -0700 (PDT) Received: from [10.1.1.193] (nfw2.codefab.com [199.103.21.225] (may be forged)) (authenticated bits=0)i45JXqg4005012; Wed, 5 May 2004 12:33:53 -0700 (PDT) In-Reply-To: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> References: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> Mime-Version: 1.0 (Apple Message framework v613) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <207CE1A8-9ECB-11D8-8DD7-003065ABFD92@mac.com> Content-Transfer-Encoding: 7bit From: Charles Swiger Date: Wed, 5 May 2004 15:33:46 -0400 To: adp X-Mailer: Apple Mail (2.613) cc: questions@freebsd.org cc: performance@freebsd.org Subject: Re: Abnormal network errors? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2004 19:33:54 -0000 On May 5, 2004, at 2:27 PM, adp wrote: > On this server I'm thinking I need two things: > > 1. More sockets available. > 2. Larger sockbufs for send and recv. > > Is this an accurate assessment? Given the application of this system, you might want to up the value of kern.ipc.nmbclusters by a factor of four or so (it's NBMCLUSTERS in the kernel config file). However, it's not essential-- your "netstat -m" is OK, and your TCP send and receive windows are reasonably sized as-is by default. > What is "2432320 packets for unknown/unsupported protocol"? What > specifically does this mean? (In other words, what should I do to > resolve > this?) It means machines are sending non-IP traffic on your network, which is normal if you have Windows protocols (NetBEUI, SPX/IPX) or Macs (AppleTalk) around. Or chatty network devices like some printers.... See /usr/include/net/ethernet.h for an idea, or maybe "tcpdump not ip" might give some idea of what's going by. > What about "921363 calls to icmp_error"? ICMP messages like responding to a ping, or people sending traffic with RFC-1918 unroutable addresses (gives "dest unreachable")... > Under tcp I have "481930 embryonic connections dropped". I assume that > means > I don't have enough sockets available for when this server gets loaded. > Correct? More likely, these are someone doing a port scan and leaving half-open connections lying around to get cleaned up. ---- It might be helpful if you gave us some idea as to what the performance problem you were seeing was? Is NFS access slow, or some such? Are you seeing errors or collisions in netstat -i or in whatever statistics your switch keeps per port? The following areas struck me as being relevant: > # ifconfig rl0 First, consider upgrading to a fxp or dc-based NIC. > udp: > 272987897 datagrams received > [ ... ] > 19976574 dropped due to full socket buffers This is high enough to represent a concern, agreed. > ip: > 578001924 total packets received [ ... ] > 4899083 fragments received > 4 fragments dropped (dup or out of space) > 750 fragments dropped after timeout > 842689 packets reassembled ok [ ... ] > 609745425 packets sent from this host > 1914687 output datagrams fragmented > 10496350 fragments created Second, you're fragmenting a relatively large number of packets going by, you ought to see what's going on with your MTU and pMTU discovery. I suppose if you're using large UDP datagrams with NFS, that might be it. [ The machines I've got around with comparible traffic volume might have 400 frags received, and 10 transmitted, or some such. ] -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Wed May 5 14:39:38 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 50C5C16A4CF; Wed, 5 May 2004 14:39:38 -0700 (PDT) Received: from watcher.puryear-it.com (ip-66-186-248-99.eatel.net [66.186.248.99]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D63B43DB9; Wed, 5 May 2004 14:38:22 -0700 (PDT) (envelope-from dap99@i-55.com) Received: from localhost (unknown [127.0.0.1]) by watcher.puryear-it.com (Postfix) with ESMTP id 0A9F134D1F; Wed, 5 May 2004 16:36:04 -0500 (CDT) Received: from watcher.puryear-it.com ([127.0.0.1]) by localhost (watcher.puryear-it.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 61052-01; Wed, 5 May 2004 16:36:02 -0500 (CDT) Received: from yourqqh4336axf (localhost [127.0.0.1]) by watcher.puryear-it.com (Postfix) with SMTP id 3078234D1E; Wed, 5 May 2004 16:36:02 -0500 (CDT) Message-ID: <015e01c432e9$1d77b650$6501a8c0@yourqqh4336axf> From: "adp" To: "Charles Swiger" References: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> <207CE1A8-9ECB-11D8-8DD7-003065ABFD92@mac.com> Date: Wed, 5 May 2004 16:34:28 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2739.300 X-Virus-Scanned: by amavisd-new cc: questions@freebsd.org cc: performance@freebsd.org Subject: Re: Abnormal network errors? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2004 21:39:38 -0000 ----- Original Message ----- From: "Charles Swiger" > On May 5, 2004, at 2:27 PM, adp wrote: > > On this server I'm thinking I need two things: > > > > 1. More sockets available. > > 2. Larger sockbufs for send and recv. > > > > Is this an accurate assessment? > > Given the application of this system, you might want to up the value of > kern.ipc.nmbclusters by a factor of four or so (it's NBMCLUSTERS in the > kernel config file). However, it's not essential-- your "netstat -m" > is OK, and your TCP send and receive windows are reasonably sized as-is > by default. Several problems. First, we are hosting a DNS server on this box. The DNS resolves domains we are auth. for very fast, or anything in its cache very fast, but anything else is SLOW or times-out. Also, our www server (another box) is responding slowly in general (4-6 seconds). > > What is "2432320 packets for unknown/unsupported protocol"? What > > specifically does this mean? (In other words, what should I do to > > resolve > > this?) > > It means machines are sending non-IP traffic on your network, which is > normal if you have Windows protocols (NetBEUI, SPX/IPX) or Macs > (AppleTalk) around. Or chatty network devices like some printers.... What is 802.1d? I am getting a lot of this: 16:21:16.788617 802.1d config 8000.00:04:27:d1:cb:d3.8019 root 8000.00:03:6c:51:a2:a7 pathcost 8 age 2 max 20 hello 2 fdelay 15 And this: 16:21:15.424508 CDP v2, ttl=180s DevID 'Six2' Addr (1): IPv4 10.2.254.62 PortID 'FastEthernet0/12' CAP 0x0a[|cdp] I'm at a colo. > See /usr/include/net/ethernet.h for an idea, or maybe "tcpdump not ip" > might give some idea of what's going by. > > > What about "921363 calls to icmp_error"? > > ICMP messages like responding to a ping, or people sending traffic with > RFC-1918 unroutable addresses (gives "dest unreachable")... That's weird. I tried 'tcpdump icmp' and see a few errors right off the bat: # tcpdump -n icmp tcpdump: listening on rl0 16:27:46.633262 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1249 unreachable 16:27:53.639237 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1280 unreachable 16:28:02.579417 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1204 unreachable 16:28:07.716527 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1510 unreachable 16:28:08.589910 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1218 unreachable 16:28:15.668697 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1327 unreachable 16:28:33.581427 192.168.42.71 > 192.168.42.76: icmp: 192.168.42.71 udp port 1355 unreachable Hmm. I am running a DNS server in a jail on the NFS server. We have been getting very slow responses times from it. Seems related. > > Under tcp I have "481930 embryonic connections dropped". I assume that > > means > > I don't have enough sockets available for when this server gets loaded. > > Correct? > > More likely, these are someone doing a port scan and leaving half-open > connections lying around to get cleaned up. We are behind a managed NetScreen firewall. I can't see how anyone is port-scanning us, unless they are just scanning the few ports we have open to the world. > It might be helpful if you gave us some idea as to what the performance > problem you were seeing was? Is NFS access slow, or some such? Are > you seeing errors or collisions in netstat -i or in whatever statistics > your switch keeps per port? > > The following areas struck me as being relevant: > > > # ifconfig rl0 > > First, consider upgrading to a fxp or dc-based NIC. Noted. > > udp: > > 272987897 datagrams received > > [ ... ] > > 19976574 dropped due to full socket buffers > > This is high enough to represent a concern, agreed. How do I fix this then? I assume I don't have enough sockets available. Is there a way to see where I am peaking? I'm thinking adding more memory will increase the system-set defaults (also read 'man tuning'). > > ip: > > 578001924 total packets received > [ ... ] > > 4899083 fragments received > > 4 fragments dropped (dup or out of space) > > 750 fragments dropped after timeout > > 842689 packets reassembled ok > [ ... ] > > 609745425 packets sent from this host > > 1914687 output datagrams fragmented > > 10496350 fragments created > > Second, you're fragmenting a relatively large number of packets going > by, you ought to see what's going on with your MTU and pMTU discovery. > I suppose if you're using large UDP datagrams with NFS, that might be > it. > > [ The machines I've got around with comparible traffic volume might > have 400 frags received, and 10 transmitted, or some such. ] Could this be related to my switch or anything else? From owner-freebsd-performance@FreeBSD.ORG Wed May 5 14:39:49 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5878C16A4CF; Wed, 5 May 2004 14:39:49 -0700 (PDT) Received: from watcher.puryear-it.com (ip-66-186-248-99.eatel.net [66.186.248.99]) by mx1.FreeBSD.org (Postfix) with ESMTP id C245F43D77; Wed, 5 May 2004 14:39:42 -0700 (PDT) (envelope-from dap99@i-55.com) Received: from localhost (unknown [127.0.0.1]) by watcher.puryear-it.com (Postfix) with ESMTP id 1D71534D25; Wed, 5 May 2004 16:37:32 -0500 (CDT) Received: from watcher.puryear-it.com ([127.0.0.1]) by localhost (watcher.puryear-it.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 51655-09; Wed, 5 May 2004 16:37:31 -0500 (CDT) Received: from yourqqh4336axf (localhost [127.0.0.1]) by watcher.puryear-it.com (Postfix) with SMTP id BA77334D1E; Wed, 5 May 2004 16:37:30 -0500 (CDT) Message-ID: <017a01c432e9$52437d60$6501a8c0@yourqqh4336axf> From: "adp" To: "Charles Swiger" References: <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf> <207CE1A8-9ECB-11D8-8DD7-003065ABFD92@mac.com> Date: Wed, 5 May 2004 16:38:23 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2739.300 X-Virus-Scanned: by amavisd-new cc: questions@freebsd.org cc: performance@freebsd.org Subject: Re: Abnormal network errors? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 May 2004 21:39:49 -0000 As an added note, I am seeing real issues with the NFS stats on the server: # nfsstat -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 502 201060 80441153 281 4569327 2420840 270703 462872 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 365160 191938 0 2009 453 1811510 0 115333847 Mknod Fsstat Fsinfo PathConf Commit GLease Vacate Evict 0 4878368 234 0 409440 0 0 0 Server Ret-Failed 76480937 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 351435 97871 1579 211262178 Server Lease Stats: Leases PeakL GLeases 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 2420836 2420840 4