From owner-freebsd-net@FreeBSD.ORG Wed Dec 5 03:58:09 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0DF2F286; Wed, 5 Dec 2012 03:58:09 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 96C398FC12; Wed, 5 Dec 2012 03:58:08 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id s9so8977056iec.13 for ; Tue, 04 Dec 2012 19:58:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Az1vPZAQ9TMin+ClxwSn7WMwLgYwBzDn6RN1spKRA9M=; b=hzvzRdBngMd2IpmZtYfCAeFmeharw/RZn/LQ6D9s6l2VQgodn9PyEApeC4Vvw/DqqM AbWYR3XgpuTVRa/78Kf8Sn6WGNn9ylb2Xxgb8LmILvJj72yde2d9oD1BB8ZFW3FnlsVi ho6xzMQp7Qe/SWN6EJtIFREd4WwL07uGUAqi0GdQYRFQNuh6SkvdhBqHGWie47nwUYM+ d+uKkGVmDfcUyonDDSg5hyA7Kwr8kcE7YyTBjV+riJExY97vB+sa710VqfNxM4MKWhGJ F7rR4RuVSo+tglENGZ4yWRrdQ9qSDGTBU0J/vlC5ibq3UQsgQ1z2NJFHR+Xp3rUgJ5HQ nNCQ== Received: by 10.50.150.144 with SMTP id ui16mr503107igb.68.1354679887885; Tue, 04 Dec 2012 19:58:07 -0800 (PST) Received: from [10.0.0.130] ([24.225.136.71]) by mx.google.com with ESMTPS id uj11sm11568274igb.15.2012.12.04.19.58.06 (version=SSLv3 cipher=OTHER); Tue, 04 Dec 2012 19:58:07 -0800 (PST) Message-ID: <50BEC64B.7010906@gmail.com> Date: Tue, 04 Dec 2012 22:58:03 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Andre Oppermann Subject: Re: Latency issues with buf_ring References: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> <201212041108.17645.jhb@freebsd.org> <50BE56C8.1030804@networx.ch> In-Reply-To: <50BE56C8.1030804@networx.ch> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Adrian Chadd , John Baldwin , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Dec 2012 03:58:09 -0000 Hi, On 04/12/2012 3:02 PM, Andre Oppermann wrote: > On 04.12.2012 20:34, Adrian Chadd wrote: >> .. and it's important to note that buf_ring itself doesn't have the >> race condition; it's the general driver implementation that's racy. >> >> I have the same races in ath(4) with the watchdog programming. Exactly >> the same issue. > > Our IF_* stack/driver boundary handoff isn't up to the task anymore. > > Also the interactions are either poorly defined or understood in many > places. I've had a few chats with yongari@ and am experimenting with > a modernized interface in my branch. > > The reason I stumbled across it was because I'm extending the hardware > offload feature set and found out that the stack and the drivers (and > the drivers among themself) are not really in sync with regards to > behavior. > > For most if not all ethernet drivers from 100Mbit/s the TX DMA rings > are so large that buffering at the IFQ level doesn't make sense anymore > and only adds latency. So it could simply directly put everything into > the TX DMA and not even try to soft-queue. If the TX DMA ring is full > ENOBUFS is returned instead of filling yet another queue. However there > are ALTQ interactions and other mechanisms which have to be considered > too making it a bit more involved. I've also bumped into this 'internalization' of drbr for quite some time now. I have been toying with some ideas around a multi-queue capable ALTQ. Not unlike IFQ_* the whole class_queue_t code in ALTQ could use some freshening up. One avenue I am looking into is drbr queues (and its associated TX lock) as the back end queue implementation for ALTQ. ALTQ(9) has a concept of driver managed queues and the approach tries to keep the same paradigm but adapt it for buf_ring. In that context, It doesn't feel natural for me that drbr logic is handled so low inside the device drivers and makes system level modifications to ALTQ unnecessarily driver dependent. ALTQ is also using very coarse grained locking (using the IFQ_LOCK for everything) which doesn't make much sense in a SMP/multiqueue system but that's another story. > > I'm coming up with a draft and some benchmark results for an updated > stack/driver boundary in the next weeks before xmas. > Sounds great, can't wait to read it while drinking eggnog :)