From owner-freebsd-net@FreeBSD.ORG Mon Dec 14 00:08:08 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B5EE106566B for ; Mon, 14 Dec 2009 00:08:08 +0000 (UTC) (envelope-from fjo-lists@ogris.de) Received: from ns1.ogris.net (ns1.ogris.net [212.62.68.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5E9398FC0C for ; Mon, 14 Dec 2009 00:08:07 +0000 (UTC) Received: from [192.168.0.14] (p54875767.dip.t-dialin.net [84.135.87.103]) by ns1.ogris.net (Postfix) with ESMTPA id 2C3B11211A3 for ; Mon, 14 Dec 2009 00:53:48 +0100 (CET) User-Agent: Microsoft-Entourage/13.3.0.091002 Date: Mon, 14 Dec 2009 00:53:22 +0100 From: "Felix J. Ogris" To: Message-ID: Thread-Topic: tcp keepalive after fin+ack from client and server Thread-Index: Acp8T3PEW34co3FYMkyvX8dmC028ug== Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Subject: tcp keepalive after fin+ack from client and server X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Dec 2009 00:08:08 -0000 Hi, I am experiencing some strange problem where FreeBSD sometimes starts sending tcp keepalives after client and server have sent and ack'ed FINs. The server runs 7.1-RELEASE/amd64 with open-vm-tools-nox11-148847 in a VMware ESXi 4.0. The client runs a CentOS Linux 2.6.18-164.6.1.el5PAE SMP on a bare metal machine. FreeBSD houses a Apache installation with sendfile and mmap enabled. The Linux machine runs a homemade monitoring system and starts a Perl script every 5 minutes to check if Apache is still alive. I have put a tcpdump output on http://ogris.de/keepalive.txt for readability and can provide the raw tcpdump file, if needed. Client and server keep sending those keepalives for about 2 hours (yielding 300kB/s constantly) if not stopped manually by an ipfw rule. lsof shows that no user process has open the corresponding sockets anymore, whereas netstat shows established connections. FreeBSD has loaded ipfw with some keep-state rules, the Linux box has iptables disabled. kldstat: Id Refs Address Size Name 1 7 0xffffffff80100000 b4be40 kernel 2 1 0xffffffff80c4c000 7d0 accf_data.ko 3 1 0xffffffff80c4d000 14d8 accf_http.ko 4 1 0xffffffffae531000 175a vmmemctl.ko 5 1 0xffffffffae543000 1e2e vmxnet.ko 6 1 0xffffffffae548000 9dd2 ipfw.ko netstat -s -p tcp: tcp: 952726388 packets sent 9941686 data packets (12846102403 bytes) 27667 data packets (37621159 bytes) retransmitted 844 data packets unnecessarily retransmitted 10 resends initiated by MTU discovery 942517629 ack-only packets (125912 delayed) 0 URG only packets 632 window probe packets 61151 window update packets 177623 control packets 949354853 packets received 6393724 acks (for 12784848708 bytes) 217997 duplicate acks 941774559 acks for unsent data 677499 packets (448005826 bytes) received in-sequence 179120 completely duplicate packets (1862276 bytes) 178 old duplicate packets 12 packets with some dup. data (6813 bytes duped) 4934 out-of-order packets (6499234 bytes) 0 packets (0 bytes) of data after window 0 window probes 92981 window update packets 1848 packets received after close 19 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 81 discarded due to memory problems 24262 connection requests 152765 connection accepts 0 bad connection attempts 0 listen queue overflows 21854 ignored RSTs in the windows 176860 connections established (including accepts) 179292 connections closed (including 23036 drops) 111998 connections updated cached RTT on close 112122 connections updated cached RTT variance on close 43123 connections updated cached ssthresh on close 58 embryonic connections dropped 5513567 segments updated rtt (of 3368553 attempts) 17054 retransmit timeouts 751 connections dropped by rexmit timeout 803 persist timeouts 3 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 2008 keepalive timeouts 1786 keepalive probes sent 222 connections dropped by keepalive 1234887 correct ACK header predictions 434353 correct data packet header predictions 152809 syncache entries added 738 retransmitted 506 dupsyn 0 dropped 152765 completed 0 bucket overflow 0 cache overflow 144 reset 165 stale 0 aborted 0 badack 0 unreach 0 zone failures 152809 cookies sent 265 cookies received 6416 SACK recovery episodes 12522 segment rexmits in SACK recovery episodes 17837785 byte rexmits in SACK recovery episodes 62528 SACK options (SACK blocks) received 3807 SACK options (SACK blocks) sent 0 SACK scoreboard overflow sysctl net: net.local.stream.recvspace: 8192 net.local.stream.sendspace: 8192 net.local.dgram.recvspace: 4096 net.local.dgram.maxdgram: 2048 net.local.recycled: 0 net.local.taskcount: 0 net.local.inflight: 0 net.inet.ip.portrange.randomtime: 45 net.inet.ip.portrange.randomcps: 10 net.inet.ip.portrange.randomized: 1 net.inet.ip.portrange.reservedlow: 0 net.inet.ip.portrange.reservedhigh: 1023 net.inet.ip.portrange.hilast: 65535 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.first: 49152 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.forwarding: 0 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 3600 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 0 net.inet.ip.accept_sourceroute: 0 net.inet.ip.keepfaith: 0 net.inet.ip.gifttl: 30 net.inet.ip.same_prefix_carp_only: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.maxfragpackets: 1024 net.inet.ip.maxfragsperpacket: 16 net.inet.ip.fragpackets: 0 net.inet.ip.check_interface: 0 net.inet.ip.random_id: 0 net.inet.ip.sendsourcequench: 0 net.inet.ip.process_options: 1 net.inet.ip.fw.dyn_keepalive: 1 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.static_count: 22 net.inet.ip.fw.dyn_max: 4096 net.inet.ip.fw.dyn_count: 1038 net.inet.ip.fw.curr_dyn_buckets: 1024 net.inet.ip.fw.dyn_buckets: 1024 net.inet.ip.fw.default_rule: 65535 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.verbose: 0 net.inet.ip.fw.debug: 1 net.inet.ip.fw.one_pass: 1 net.inet.ip.fw.autoinc_step: 100 net.inet.ip.fw.enable: 1 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 200 net.inet.icmp.bmcastecho: 0 net.inet.icmp.quotelen: 8 net.inet.icmp.reply_from_interface: 0 net.inet.icmp.reply_src: net.inet.icmp.icmplim_output: 1 net.inet.icmp.log_redirect: 0 net.inet.icmp.drop_redirect: 0 net.inet.icmp.maskfake: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 512 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 65536 net.inet.tcp.recvspace: 65536 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.v6mssdflt: 1024 net.inet.tcp.hostcache.purge: 0 net.inet.tcp.hostcache.prune: 300 net.inet.tcp.hostcache.expire: 3600 net.inet.tcp.hostcache.count: 116 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.recvbuf_max: 262144 net.inet.tcp.recvbuf_inc: 16384 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.insecure_rst: 0 net.inet.tcp.rfc3390: 1 net.inet.tcp.rfc3042: 1 net.inet.tcp.drop_synfin: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.blackhole: 0 net.inet.tcp.log_in_vain: 0 net.inet.tcp.sendbuf_max: 262144 net.inet.tcp.sendbuf_inc: 8192 net.inet.tcp.sendbuf_auto: 1 net.inet.tcp.tso: 1 net.inet.tcp.newreno: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.reass.overflows: 81 net.inet.tcp.reass.maxqlen: 48 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.maxsegments: 2048 net.inet.tcp.sack.globalholes: 0 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.enable: 1 net.inet.tcp.inflight.stab: 20 net.inet.tcp.inflight.max: 1073725440 net.inet.tcp.inflight.min: 6144 net.inet.tcp.inflight.rttthresh: 10 net.inet.tcp.inflight.debug: 0 net.inet.tcp.inflight.enable: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.pcbcount: 168 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.log_debug: 0 net.inet.tcp.minmss: 216 net.inet.tcp.syncache.rst_on_sock_fail: 1 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.cachelimit: 15360 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncookies_only: 0 net.inet.tcp.syncookies: 1 net.inet.tcp.timer_race: 0 net.inet.tcp.finwait2_timeout: 60000 net.inet.tcp.fast_finwait2_recycle: 0 net.inet.tcp.always_keepalive: 1 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.rexmit_min: 30 net.inet.tcp.msl: 30000 net.inet.tcp.nolocaltimewait: 0 net.inet.tcp.maxtcptw: 6553 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 42080 net.inet.udp.soreceive_dgram_enabled: 0 net.inet.udp.blackhole: 0 net.inet.udp.log_in_vain: 0 net.inet.sctp.enable_sack_immediately: 0 net.inet.sctp.udp_tunneling_port: 0 net.inet.sctp.udp_tunneling_for_client_enable: 0 net.inet.sctp.mobility_fasthandoff: 0 net.inet.sctp.mobility_base: 0 net.inet.sctp.default_frag_interleave: 1 net.inet.sctp.default_cc_module: 0 net.inet.sctp.log_level: 0 net.inet.sctp.max_retran_chunk: 30 net.inet.sctp.min_residual: 1452 net.inet.sctp.strict_data_order: 0 net.inet.sctp.abort_at_limit: 0 net.inet.sctp.hb_max_burst: 4 net.inet.sctp.do_sctp_drain: 1 net.inet.sctp.max_chained_mbufs: 5 net.inet.sctp.abc_l_var: 1 net.inet.sctp.nat_friendly: 1 net.inet.sctp.auth_disable: 0 net.inet.sctp.asconf_auth_nochk: 0 net.inet.sctp.early_fast_retran_msec: 250 net.inet.sctp.early_fast_retran: 0 net.inet.sctp.cwnd_maxburst: 1 net.inet.sctp.cmt_pf: 0 net.inet.sctp.cmt_use_dac: 0 net.inet.sctp.cmt_on_off: 0 net.inet.sctp.outgoing_streams: 10 net.inet.sctp.add_more_on_output: 1452 net.inet.sctp.path_rtx_max: 5 net.inet.sctp.assoc_rtx_max: 10 net.inet.sctp.init_rtx_max: 8 net.inet.sctp.valid_cookie_life: 60000 net.inet.sctp.init_rto_max: 60000 net.inet.sctp.rto_initial: 3000 net.inet.sctp.rto_min: 1000 net.inet.sctp.rto_max: 60000 net.inet.sctp.secret_lifetime: 3600 net.inet.sctp.shutdown_guard_time: 180 net.inet.sctp.pmtu_raise_time: 600 net.inet.sctp.heartbeat_interval: 30000 net.inet.sctp.asoc_resource: 10 net.inet.sctp.sys_resource: 1000 net.inet.sctp.sack_freq: 2 net.inet.sctp.delayed_sack_time: 200 net.inet.sctp.chunkscale: 10 net.inet.sctp.min_split_point: 2904 net.inet.sctp.pcbhashsize: 256 net.inet.sctp.tcbhashsize: 1024 net.inet.sctp.maxchunks: 4096 net.inet.sctp.maxburst: 4 net.inet.sctp.peer_chkoh: 256 net.inet.sctp.strict_init: 1 net.inet.sctp.loopback_nocsum: 1 net.inet.sctp.strict_sacks: 0 net.inet.sctp.ecn_nonce: 0 net.inet.sctp.ecn_enable: 1 net.inet.sctp.auto_asconf: 1 net.inet.sctp.recvspace: 233016 net.inet.sctp.sendspace: 233016 net.inet.raw.recvspace: 9216 net.inet.raw.maxdgram: 9216 net.inet.accf.unloadable: 0 net.inet.accf.http.parsehttpversion: 1 net.link.generic.system.ifcount: 3 net.link.ether.inet.log_arp_permanent_modify: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.proxyall: 0 net.link.ether.inet.useloopback: 1 net.link.ether.inet.maxtries: 5 net.link.ether.inet.max_age: 1200 net.link.ether.ipfw: 0 net.link.gif.parallel_tunnels: 0 net.link.gif.max_nesting: 1 net.link.log_link_state_change: 1 net.link.tun.devfs_cloning: 1 net.inet6.ip6.forwarding: 0 net.inet6.ip6.redirect: 1 net.inet6.ip6.hlim: 64 net.inet6.ip6.maxfragpackets: 8192 net.inet6.ip6.accept_rtadv: 0 net.inet6.ip6.keepfaith: 0 net.inet6.ip6.log_interval: 5 net.inet6.ip6.hdrnestlimit: 15 net.inet6.ip6.dad_count: 1 net.inet6.ip6.auto_flowlabel: 1 net.inet6.ip6.defmcasthlim: 1 net.inet6.ip6.gifhlim: 30 net.inet6.ip6.kame_version: FreeBSD net.inet6.ip6.use_deprecated: 1 net.inet6.ip6.rr_prune: 5 net.inet6.ip6.v6only: 1 net.inet6.ip6.rtexpire: 3600 net.inet6.ip6.rtminexpire: 10 net.inet6.ip6.rtmaxcache: 128 net.inet6.ip6.use_tempaddr: 0 net.inet6.ip6.temppltime: 86400 net.inet6.ip6.tempvltime: 604800 net.inet6.ip6.auto_linklocal: 0 net.inet6.ip6.prefer_tempaddr: 0 net.inet6.ip6.use_defaultzone: 0 net.inet6.ip6.maxfrags: 8192 net.inet6.ip6.mcast_pmtu: 0 net.inet6.ip6.fw.enable: 1 net.inet6.ip6.fw.deny_unknown_exthdrs: 1 net.inet6.icmp6.rediraccept: 1 net.inet6.icmp6.redirtimeout: 600 net.inet6.icmp6.nd6_prune: 1 net.inet6.icmp6.nd6_delay: 5 net.inet6.icmp6.nd6_umaxtries: 3 net.inet6.icmp6.nd6_mmaxtries: 3 net.inet6.icmp6.nd6_useloopback: 1 net.inet6.icmp6.nodeinfo: 3 net.inet6.icmp6.errppslimit: 100 net.inet6.icmp6.nd6_maxnudhint: 0 net.inet6.icmp6.nd6_debug: 0 net.inet6.icmp6.nd6_maxqueuelen: 1 net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0 net.bpf.maxinsns: 512 net.bpf.maxbufsize: 524288 net.bpf.bufsize: 4096 net.isr.swi_count: 751963064 net.isr.drop: 0 net.isr.queued: 952769795 net.isr.deferred: 0 net.isr.directed: 952250693 net.isr.count: 952250693 net.isr.direct: 1 net.raw.recvspace: 8192 net.raw.sendspace: 8192 net.my_fibnum: 0 net.add_addr_allfibs: 1 net.fibs: 1 net.route.netisr_maxqlen: 256 net.wlan.recv_bar: 1 net.wlan.debug: 0 sysctl kern.ipc: kern.ipc.maxsockbuf: 262144 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 4096 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 60 kern.ipc.max_hdr: 76 kern.ipc.max_datalen: 100 kern.ipc.nmbjumbo16: 3200 kern.ipc.nmbjumbo9: 6400 kern.ipc.nmbjumbop: 12800 kern.ipc.nmbclusters: 32768 kern.ipc.piperesizeallowed: 1 kern.ipc.piperesizefail: 0 kern.ipc.pipeallocfail: 0 kern.ipc.pipefragretry: 0 kern.ipc.pipekva: 147456 kern.ipc.maxpipekva: 20971520 kern.ipc.msgseg: 2048 kern.ipc.msgssz: 8 kern.ipc.msgtql: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgmni: 40 kern.ipc.msgmax: 16384 kern.ipc.semaem: 16384 kern.ipc.semvmx: 32767 kern.ipc.semusz: 104 kern.ipc.semume: 10 kern.ipc.semopm: 100 kern.ipc.semmsl: 60 kern.ipc.semmnu: 512 kern.ipc.semmns: 1024 kern.ipc.semmni: 512 kern.ipc.semmap: 256 kern.ipc.shm_allow_removed: 0 kern.ipc.shm_use_phys: 0 kern.ipc.shmall: 32768 kern.ipc.shmseg: 128 kern.ipc.shmmni: 192 kern.ipc.shmmin: 1 kern.ipc.shmmax: 134217728 kern.ipc.maxsockets: 32768 kern.ipc.numopensockets: 188 kern.ipc.nsfbufsused: 0 kern.ipc.nsfbufspeak: 0 kern.ipc.nsfbufs: 0 TIA, Felix