From owner-freebsd-questions@FreeBSD.ORG Sun Oct 9 17:28:20 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E0141065672 for ; Sun, 9 Oct 2011 17:28:20 +0000 (UTC) (envelope-from freebsd_user@guice.ath.cx) Received: from WORKSTATION.guice.ath.cx (static-72-66-82-53.washdc.fios.verizon.net [72.66.82.53]) by mx1.freebsd.org (Postfix) with ESMTP id 6B7528FC12 for ; Sun, 9 Oct 2011 17:28:19 +0000 (UTC) Received: from wtp1.ath.cx (localhost [127.0.0.1]) by WORKSTATION.guice.ath.cx (8.14.4/8.14.4) with ESMTP id p99HSJIM095853 for ; Sun, 9 Oct 2011 13:28:19 -0400 (EDT) (envelope-from freebsd_user@guice.ath.cx) Received: from 72.66.82.53 (SquirrelMail authenticated user email) by wtp1.ath.cx with HTTP; Sun, 9 Oct 2011 13:28:19 -0400 Message-ID: Date: Sun, 9 Oct 2011 13:28:19 -0400 From: freebsd_user@guice.ath.cx To: freebsd-questions@freebsd.org User-Agent: SquirrelMail/1.4.21 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: System randomly not logging complete bi-directional traffic. X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2011 17:28:20 -0000 freebsd-questions@freebsd.org # # # FreeBSD_7-4 RELEASE # Our hardware is pristine # # What is described herein are regular, yet random occurrences; we need help. We have already performed a reinstall of FreeBSD_7-4 RELEASE (and the daemons in question); the issue remains. Below, is part of a conversation with an httpd whereby the packets (entire conversations) are randomly 'not' being logged and/or seen by either the httpd nor ipfw (logging enabled), yet both tshark and tcpdump are capturing everything. To be perfectly clear, httpd and ipfw (randomly) will not see/log anything of an 'entire conversation'. It is not like it drops certain packets of a conversation; they (httpd/ipfw) either see and log everything during a conversation, or, 'do not see' and 'do not log' any packet associated with a given conversation; all the while tshark and tcpdump are capturing everything (bidirectional); hence the connection is real. The capture below was witnessed by both tshark and tcpdump, but not logged via the httpd or the following ipfw rule: $cmd 00029 deny log logamount 0 ip from "table(1)" to me 80 The above ipfw rule functions properly from "table(1)" which contains -- ip.ip.ip.ip/32 -- one (1) ip per line. The names (below) were changed to protect the innocent; yeah right. Internet Protocol Version 4, Src: ex.ter.nal.ip (ex.ter.nal.ip), Dst: in.ter.nal.ip (in.ter.nal.ip) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 60 Identification: 0x8ce5 (36069) Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 251 Protocol: TCP (6) Header checksum: 0x9102 [correct] [Good: True] [Bad: False] Source: ex.ter.nal.ip (ex.ter.nal.ip) Destination: in.ter.nal.ip (in.ter.nal.ip) Transmission Control Protocol, Src Port: 46463 (46463), Dst Port: http (80), Seq: 0, Len: 0 Source port: 46463 (46463) Destination port: http (80) [Stream index: 19] Sequence number: 0 (relative sequence number) Header length: 40 bytes Flags: 0x02 (SYN) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...0 .... = Acknowledgement: Not set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..1. = Syn: Set [Expert Info (Chat/Sequence): Connection establish request (SYN): server port http] [Message: Connection establish request (SYN): server port http] [Severity level: Chat] [Group: Sequence] .... .... ...0 = Fin: Not set Window size value: 5840 [Calculated window size: 5840] Checksum: 0xe7f8 [validation disabled] [Good Checksum: False] [Bad Checksum: False] Options: (20 bytes) Maximum segment size: 1460 bytes TCP SACK Permitted Option: True Timestamps: TSval 309029146, TSecr 0 Kind: Timestamp (8) Length: 10 Timestamp value: 309029146 Timestamp echo reply: 0 No-Operation (NOP) Window scale: 7 (multiply by 128) Kind: Window Scale (3) Length: 3 Shift count: 7 [Multiplier: 128] Frame Number: 51 Frame Length: 74 bytes (592 bits) Capture Length: 74 bytes (592 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:tcp] Ethernet II, Src: Router_cf:gr:f0 (11:52:c3:fd:dd:f0), Dst: Goe_40:84:21 (00:15:18:40:28:41) Destination: Goe_40:84:21 (00:15:18:40:28:41) Address: Goe_40:84:21 (00:15:18:40:28:41) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: Router_cf:gr:f0 (11:52:c3:fd:dd:f0) Address: Router_cf:gr:f0 (11:52:c3:fd:dd:f0) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol Version 4, Src: ex.ter.nal.ip (ex.ter.nal.ip), Dst: in.ter.nal.ip (in.ter.nal.ip) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 60 Identification: 0x8ce5 (36069) Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 251 Protocol: TCP (6) Header checksum: 0x9102 [correct] [Good: True] [Bad: False] Source: ex.ter.nal.ip (ex.ter.nal.ip) Destination: in.ter.nal.ip (in.ter.nal.ip) Transmission Control Protocol, Src Port: 46463 (46463), Dst Port: http (80), Seq: 0, Len: 0 Source port: 46463 (46463) Destination port: http (80) [Stream index: 19] Sequence number: 0 (relative sequence number) Header length: 40 bytes Flags: 0x02 (SYN) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...0 .... = Acknowledgement: Not set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..1. = Syn: Set [Expert Info (Chat/Sequence): Connection establish request (SYN): server port http] [Message: Connection establish request (SYN): server port http] [Severity level: Chat] [Group: Sequence] .... .... ...0 = Fin: Not set Window size value: 5840 [Calculated window size: 5840] Checksum: 0xe7f8 [validation disabled] [Good Checksum: False] [Bad Checksum: False] Options: (20 bytes) Maximum segment size: 1460 bytes TCP SACK Permitted Option: True Timestamps: TSval 309029146, TSecr 0 Kind: Timestamp (8) Length: 10 Timestamp value: 309029146 Timestamp echo reply: 0 No-Operation (NOP) Window scale: 7 (multiply by 128) Kind: Window Scale (3) Length: 3 Shift count: 7 [Multiplier: 128] ================================ We first noticed this logging issue with Apache22 and thought httpd was the culprit; we installed another httpd and the problem remained. At that point, with two (2) httpds' seemingly offering the same results we decided to enable ipfw and its logging feature. Shortly thereafter we notice ipfw was also randomly 'not seeing' the exact same traffic, destined for port 80, that neither httpd's were also 'not seeing', yet Tshark and tcpdump are in fact, capturing these packets. We would like help to troubleshoot this concern rather than blindly replacing things such as syslogd.