From owner-freebsd-performance@FreeBSD.ORG Tue Apr 1 14:03:49 2008 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11239106566B; Tue, 1 Apr 2008 14:03:49 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id 8EABE8FC16; Tue, 1 Apr 2008 14:03:48 +0000 (UTC) (envelope-from ap00@mail.ru) Received: by mx0.awanti.com (Postfix, from userid 100) id D55244C3F1; Tue, 1 Apr 2008 18:03:46 +0400 (MSD) X-Spam-Flag: NO X-Spam-Checker-Version: SpamAssassin 3.1.9 on mx0.awanti.com X-Spam-Status: No, score=-2.3 required=6.5 tests=AWL,BAYES_00 autolearn=ham version=3.1.9 Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id A5F364C3DD; Tue, 1 Apr 2008 18:03:45 +0400 (MSD) Date: Tue, 1 Apr 2008 18:05:29 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <1493139437.20080401180529@mail.ru> To: Rui Paulo In-Reply-To: <20080401104128.GA1194@fnop.net> References: <1333421734.20080328201458@mail.ru> <20080401104128.GA1194@fnop.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, performance@freebsd.org Subject: Re[2]: packet delay because of blackhole X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2008 14:03:49 -0000 Hello Rui, Tuesday, April 01, 2008, 2:41:29 PM, you wrote: RP> On Fri, Mar 28, 2008 at 08:14:58PM +0300, Anthony Pankov wrote: >> Just for somebody convince. >> >> While analyzing client<->server HTTPS conversation one second delay in >> packet exchange was discovered (strongly reproducible): >> >> Sample: >> N time >> 6 0.002303 10.28.4.14 10.28.4.50 SSL Client Hello >> 7 0.106710 10.28.4.50 10.28.4.14 TCP 443 > 1447 [ACK] Seq=1 Ack=103 Win=65535 Len=0 >> 8 1.045712 10.28.4.50 10.28.4.14 TLSv1 Server Hello, Certificate, Server Hello Done >> >> Another sample: >> 10 0.011722 10.28.4.14 10.28.4.50 TLSv1 Application Data >> 11 0.115933 10.28.4.50 10.28.4.14 TCP 443 > 1442 [ACK] Seq=839 Ack=519 Win=65466 Len=0 >> 12 1.054037 10.28.4.50 10.28.4.14 TLSv1 Application Data >> >> The reason for delay is sysctl tcp.blackhole value grater than 0, much to surprise. >> >> So, turning tcp.blackhole to 0 eliminate any delay (strongly reproducible). >> >> System: FreeBSD 6_2_stable RP> I'm not sure how performance penalty can induce a cache miss and I RP> it's very processor specific. So, you're best guess is to profile the RP> kernel. RP> Regards, I'm not fully understand what cache miss do you mean. I'll try to be more clear. During client<->server HTTPS conversation there is a packet delay (see "sample" and "Another sample") about 900 ms. Delay appear one per conversation in random place (between 7-8 packet in "sample", 11-12 in "another sample"). Of course, it's not depending from SSL session cache, SSL negotiation or any other apache/mod_ssl/OpenSSL setting/performance, otherwise i should write to another maillist. I have disabled all my sysctl tuning, one by one. No effect has achieved. But when i turn tcp.blackhole to zero, all things became fine. Maximum delay between packet is 6 ms. It is strange, so i've reported to all. I suggest to keep tcp.blackhole=0 and use firewall for protection. If one would raise tcp.blackhole value, than he should dump packets and make sure that there is no strange delay between packets. It most likely FreeBSD net issue. P.S. "Another sample" is not a sequel of "Sample", it is a dump of different transaction. -- Best regards, Anthony mailto:ap00@mail.ru