From owner-freebsd-net@FreeBSD.ORG Tue Oct 29 21:02:40 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B67EF430; Tue, 29 Oct 2013 21:02:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x231.google.com (mail-qc0-x231.google.com [IPv6:2607:f8b0:400d:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 64751272B; Tue, 29 Oct 2013 21:02:40 +0000 (UTC) Received: by mail-qc0-f177.google.com with SMTP id u18so280575qcx.36 for ; Tue, 29 Oct 2013 14:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=869yhwA5JFgzW7eN27uMblMpWHeA+FnAVb49v/fEe20=; b=eSUvg/Rsio9unPDX7nyGqYBsAgeRR65Z6u8Yoa+VfqWR1sB27iWfAx3dHfGMoHnhC/ N/9Jjcp9orVYbAslXS3Zw5EilGz0NpNS17xG3+XWxZ1YQygcTDms39pdrav+s/Wm1bBY J92gWeZCQ8kW3+ap8NMXC5oHA5S+yC/nMJmLblix1C49rxThmuqCf2SLms6Uw0w/zj3e vhC8f0y+JQ7ty5umirJj2aVUPT/5lZLg5xUXupEYt5EUu8Mth3wngLVWfUxqp+CBln89 ucyW7rv8cyB0e3+nfHWxEywTSIiKznd2OzUiCW/wyR27cKR+lnhoeLKog2VzI8wpnPTX iQLg== MIME-Version: 1.0 X-Received: by 10.49.12.14 with SMTP id u14mr2335894qeb.74.1383080559519; Tue, 29 Oct 2013 14:02:39 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Tue, 29 Oct 2013 14:02:39 -0700 (PDT) In-Reply-To: <52701F7E.2060604@freebsd.org> References: <40948D79-E890-4360-A3F2-BEC34A389C7E@lakerest.net> <526FFED9.1070704@freebsd.org> <13BF1F55-EC13-482B-AF7D-59AE039F877D@lakerest.net> <52701F7E.2060604@freebsd.org> Date: Tue, 29 Oct 2013 14:02:39 -0700 X-Google-Sender-Auth: CM5CFZliHd3rd1Ywv_53dQLgg3I Message-ID: Subject: Re: MQ Patch. From: Adrian Chadd To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: Luigi Rizzo , Randall Stewart , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Oct 2013 21:02:40 -0000 [snip everything] ok, I've reviewed the work. TL;DR - it's a clearly correct step in the right direction, but I think we need to just think it through a tad bit more first. There have been queue discipline and queue management discussions in the past. Randall's work is a good step in that direction. I think though that we can take a step back up a little further. * In terms of queuing frames into multiple queues - yes, we absolutely should have an if_transmit() path to the driver that obeys "a" QoS field in the mbuf and pushes it into the relevant queue - with randalls work, it's in the driver, but it doesn't _have_ to be; * In terms of queue servicing and management - we likely need to have a variety of queue plugins that determine which frame from which queue gets chosen next to hand to the hardware. The hardware may have multiple queues! The hardware may have one queue! The application developer may only want to use one queue! That should be flexible and easy to plug into things. * Then we need to support dropping frames during queue and dropping frames during dequeue (ie, on its way to the hardware). That way we can implement the currently interesting kinds of queue disciplines (eg CODEL, etc.) * Should this be done at the driver layer (ie it's a library that each driver creates and owns), or as a layer above it, controlling the network device (ie, the linux queue discipline method.) So, my comments: * I don't like how it's hard-coding drbr's into the drivers. Yes, the underlying state should be a drbr for now. But I'd rather we have a queue discipline plugin API that drivers create an instance of. * It'll have methods to init/flush the rings, queue a frame into a ring, dequeue a frame from a ring, be notified of transmit completions so more work can be done, etc. * For people who do latency-sensitive things, they can just bypass this entirely and go straight to the hardware queues without going through this kind of intermediary queue layer. Randall - I think we can take your work and turn it into a net library that implements your queue management routines. That way we can start enabling people to tinker with it and replace it if they need to. What do you think?