From owner-freebsd-net@freebsd.org Thu Aug 20 08:10:34 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E3029BE82F; Thu, 20 Aug 2015 08:10:34 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1386D1832; Thu, 20 Aug 2015 08:10:33 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.15.2/8.15.2) with ESMTPS id t7K8ADUE040211 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 20 Aug 2015 11:10:13 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.15.2/8.15.2/Submit) id t7K8ACxi040210; Thu, 20 Aug 2015 11:10:12 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 20 Aug 2015 11:10:12 +0300 From: Gleb Smirnoff To: Yonghyeon PYUN Cc: Rick Macklem , Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Christopher Forgeron , Daniel Braniss Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <20150820081012.GY75813@glebius.int.ru> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> <55D333D6.5040102@selasky.org> <1325951625.25292515.1439934848268.JavaMail.zimbra@uoguelph.ca> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150820023024.GB996@michelle.fasterthan.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Aug 2015 08:10:34 -0000 Yonghyeon, On Thu, Aug 20, 2015 at 11:30:24AM +0900, Yonghyeon PYUN wrote: Y> > > >> Maybe it can be controlled by some kind of flag, if all the three TSO Y> > > >> limits should include the TCP/IP/ethernet headers too. I'm pretty sure Y> > > >> we want both versions. Y> > > >> Y> > > > Y> > > > Hmm, I'm afraid it's already complex. Drivers have to tell almost Y> > > > the same information to both bus_dma(9) and network stack. Y> > > Y> > > Don't forget that not all drivers in the tree set the TSO limits before Y> > > if_attach(), so possibly the subtraction of one TSO fragment needs to go Y> > > into ip_output() .... Y> > > Y> > Ok, I realized that some drivers may not know the answers before ether_ifattach(), Y> > due to the way they are configured/written (I saw the use of if_hw_tsomax_update() Y> > in the patch). Y> Y> I was not able to find an interface that configures TSO parameters Y> after if_t conversion. I'm under the impression Y> if_hw_tsomax_update() is not designed to use this way. Probably we Y> need a better one?(CCed to Gleb). Yes. In the projects/ifnet all the TSO stuff is configured differently. I'd really appreciate if other developers look there and review it, try it, give some input. Here is a snippet from net/if.h in projects/ifnet: /* * Structure describing TSO properties of an interface. Known both to ifnet * layer and TCP. Most interfaces point to a static tsomax in ifdriver * definition. However, vlan(4) and lagg(4) require a dynamic tsomax. */ struct iftsomax { uint32_t tsomax_bytes; /* TSO total burst length limit in bytes */ uint32_t tsomax_segcount; /* TSO maximum segment count */ uint32_t tsomax_segsize; /* TSO maximum segment size in bytes */ }; Now closer to your original question. I haven't yet converted lagg(4), so haven't yet worked on if_hw_tsomax_update(). I am convinced that it shouldn't be needed for a regular driver (save lagg(4). A proper driver should first study its hardware and only then call if_attach(). Correct me if am wrong, please. Also, I suppose, that a piece of hardware can't change its TSO maximums at runtime, so I don't see reason for changing them at runtime (save lagg(4)). -- Totus tuus, Glebius.