From owner-freebsd-usb@FreeBSD.ORG Sun Oct 3 23:56:12 2010 Return-Path: Delivered-To: freebsd-usb@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49B02106564A; Sun, 3 Oct 2010 23:56:12 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 116208FC0A; Sun, 3 Oct 2010 23:56:11 +0000 (UTC) Received: by pxi17 with SMTP id 17so1485238pxi.13 for ; Sun, 03 Oct 2010 16:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=xpy4jvI/N5nr1N6pjq7QG3xeQIhWe993fjgzyGDrnsI=; b=wT/gdd9UZIWxx8LlC7M8LyUClb6ruR2RgDxcWkjMkmH3R9EjH0vBQTzhe9dtvNZUOR 7QmBjd0UIE4SUSr7eD+ZIv1J1aq7RWDhZ6wOpQUs6sRExnGVElYWr/vhOgccG2BYvq4k tDT6D4EdZBqwtgWKQoNxJtKFTJlF90d94yv6w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=wTmns4+OdNnHbwP286Kytk5h65O3RM4NPm9Pok1H/tWfW/UqxwHIkw70D3nWFq1Pi1 YsUx+4h0/dMBKB4SkStwflSWnmtrKvV9P6uVIQdJdF2HpD78BeQ/kh6YTnWcwo12Zp11 kRnZfaspUJeOkYSr+g4rOVsMSje0zVBfDXAAs= Received: by 10.114.102.20 with SMTP id z20mr10356489wab.133.1286150171418; Sun, 03 Oct 2010 16:56:11 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id r37sm8109302wak.11.2010.10.03.16.56.08 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 03 Oct 2010 16:56:09 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 3 Oct 2010 16:54:23 -0700 From: Pyun YongHyeon Date: Sun, 3 Oct 2010 16:54:23 -0700 To: Hans Petter Selasky Message-ID: <20101003235423.GB1135@michelle.cdnetworks.com> References: <20101002001100.GL10521@michelle.cdnetworks.com> <201010020841.57474.hselasky@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201010020841.57474.hselasky@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: freebsd-usb@freebsd.org Subject: Re: Network TX/RX fairness is not honored by USB stack X-BeenThere: freebsd-usb@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: FreeBSD support for USB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2010 23:56:12 -0000 On Sat, Oct 02, 2010 at 08:41:57AM +0200, Hans Petter Selasky wrote: > On Saturday 02 October 2010 02:11:00 Pyun YongHyeon wrote: > > Hi, > > > > I don't know how long it had been there but it seems current USB > > stack does not honor fairness of TX/RX on USB ethernet controller. > > Unidirectional performance test(UDP) or most-unidirectional > > performance(TCP) test works well without problems. However if heavy > > TX/RX traffic hits controller at the same time either TX or RX is > > not served at all. I'm under the impression that whenever TX work > > is done it seems USB reschedules next pending TX again instead of > > processing RX such that RX is starved to death. This can be easily > > reproduced on two hosts with the netperf performance test. > > Whenever both hosts send tiny UDP datagrams to the other host > > either TX or RX packet counters are not increasing until the end > > of the UDP torture test. The number of EHCI interrupt is about 8K/s > > while test is in progress so I think it reached its maximum > > processing limit. After netperf testing, it can still process TX/RX > > packets even though it dropped too many RX packets. But these > > dropped packets are not counted so netstat(1) shows 0 dropped > > frames even though it lost millions of packets. > > > > Hans, do you have any idea what's going on here? > > You can use the following netperf command on both hosts after > > running netserver. > > %netperf -c -H ip_addr_of_other_host -tUDP_STREAM -l 300 -- -m 1 > > > > Another odd thing I noticed is number of interrupts does not go > > down to 0 after the testing. It constantly generates 1k/s > > interrupts after that. > > Maybe we are triggering a bug. Can you enable USB debugging to figure out what > data lengths are transmitted or received. > In the middle of testing? If yes, that would be meaningless as it would generate bunch of messages. The test case generates payload size 1 UDP datagrams with full speed so enabling debug messages will change timing. Note, I'm exercising number of packets per second, not number of bytes per second. > USB EHCI uses round robin, so this is either USB device problem or a test- > program software failure. > I'm pretty sure the benchmark program is not broken, so either axe(4) or USB stack could be wrong here. I see three issues from the UDP torture test. - Either TX or RX could be starved to death. If you start TX test first, RX would be stuck. If you start RX test first, TX would be stuck. - The number of packets sent or received are much lower than expected. For TX case, the number of packets sent per second is exactly 8k which is much less than that of non-USB controllers. For gigabit controllers number of TX packets could be several hundred thousands per second. For RX packets it shows 14K/s packets with 8K/s interrupts. I thought USB ethernet controllers can send more than 8k packets per second. Because the number of interrupts per second and 8k packets per second is the same, this also make me wonder there could be some relations there. - Number of interrupts does not go back to 0 after the testing. I'll let you know if I find some clue but it may take long time as I'm not familiar with USB stack. :-( > Check the CPU usage of the host computer during the test. Do you see anything? > I didn't notice odd thing except 8k/s interrupts. > > The only way I stop that interrupts was to > > down the ue0 interface with "ifconfig ue0 down" command. > > --HPS