From owner-freebsd-net@FreeBSD.ORG Fri Jul 15 16:58:43 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FB031065673 for ; Fri, 15 Jul 2011 16:58:43 +0000 (UTC) (envelope-from prvs=11774c4b74=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 524AA8FC19 for ; Fri, 15 Jul 2011 16:58:42 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 15 Jul 2011 17:47:18 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 15 Jul 2011 17:47:18 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014168297.msg for ; Fri, 15 Jul 2011 17:47:18 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=11774c4b74=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-net@freebsd.org Message-ID: <379885BA631F4C7787C24E00A174B429@multiplay.co.uk> From: "Steven Hartland" To: Date: Fri, 15 Jul 2011 17:47:34 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Subject: igb enable_aim or flow_control causing tcp stalls? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jul 2011 16:58:43 -0000 Been trying to identify an strange network stalling issue while using scp or rsync between two machines, initially at remote locations. The behaviour has proved quite difficult to track as it seems to require a number or factors combined before the stalls occur. These seem to be: 1. This particular target machine 2. Some load, but not much on the machine, when idle we don't see stalls. 3. Remote 9ms+ latency or high through put 50MB/s transmission speeds My current test case is copying a freebsd iso from a local machine to the potentially problematic machine's /dev/null e.g. scp FreeBSD-8.2-RELEASE-amd64-disc1.iso test1:/dev/null These machines are connected via a cisco 6509 -> supermicro blade chassis. When the failure happens we see the following:- scp FreeBSD-8.2-RELEASE-amd64-disc1.iso amsbld16:/dev/null FreeBSD-8.2-RELEASE-amd64-disc1.iso 21% 147MB 2.1MB/s - stalled - When all is well we see:- scp FreeBSD-8.2-RELEASE-amd64-disc1.iso amsbld16:/dev/null FreeBSD-8.2-RELEASE-amd64-disc1.iso 100% 691MB 53.1MB/s 00:13 This setup:- 1. Source machine 7.0-RELEASE-p2 using em0 em0@pci0:6:0:0: class=0x020000 card=0x109615d9 chip=0x10968086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'PRO/1000 EB Network Connection' class = network subclass = ethernet 2. Target (problem) machine 8.2-RELEASE using igb0 igb0@pci0:5:0:0: class=0x020000 card=0x10e715d9 chip=0x10e78086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet I've tried switching to igb1 with no change, which also changes switches and hence ports on the Cisco, so I don't at this point believe there is an issue there. Now I've just noticed that igb has at least two sysctl's which seemed interesting, enable_aim & flow_control (which is missing from the man page btw). On disabling both, the stalls seem to go away. Unfortunately re-enabling them didn't re-introduce the stalls, but this could another quirk when they don't re-enable properly? So the questions are:- 1. Could either of these settings cause tcp stalls? 2. If the nic and switch differ in flow control, what is the likely effect? 3. Any other thoughts? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.