From owner-svn-src-all@FreeBSD.ORG Fri Jun 11 17:44:43 2010 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ACB8106566B; Fri, 11 Jun 2010 17:44:43 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id B9AA38FC13; Fri, 11 Jun 2010 17:44:42 +0000 (UTC) Received: from [127.0.0.1] (pooker.samsco.org [168.103.85.57]) (authenticated bits=0) by pooker.samsco.org (8.14.4/8.14.4) with ESMTP id o5BHidN1039948; Fri, 11 Jun 2010 11:44:39 -0600 (MDT) (envelope-from scottl@samsco.org) Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Scott Long In-Reply-To: <20100611174137.GC13776@michelle.cdnetworks.com> Date: Fri, 11 Jun 2010 11:44:39 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9F065122-7D91-42E9-A251-5AF4AAF0B4E5@samsco.org> References: <201006110300.o5B30X9q045387@svn.freebsd.org> <201006110751.40735.jhb@freebsd.org> <853068F6-D736-4DA3-859F-D946D096843D@samsco.org> <19B0DF11-5998-40F5-8095-8D2521B1C597@mac.com> <291D17F7-F387-4A13-8ED9-2898A9F7E018@mac.com> <028BA7D4-90C8-4D2D-B9A5-C1C3CF9B6D7D@samsco.org> <20100611174137.GC13776@michelle.cdnetworks.com> To: pyunyh@gmail.com X-Mailer: Apple Mail (2.1078) X-Spam-Status: No, score=-50.0 required=3.8 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on pooker.samsco.org Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, Marcel Moolenaar , src-committers@freebsd.org, John Baldwin Subject: Re: svn commit: r209026 - in head/sys/ia64: ia64 include X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jun 2010 17:44:43 -0000 On Jun 11, 2010, at 11:41 AM, Pyun YongHyeon wrote: > On Fri, Jun 11, 2010 at 11:37:36AM -0600, Scott Long wrote: >> On Jun 11, 2010, at 11:32 AM, Marcel Moolenaar wrote: >>>=20 >>> On Jun 11, 2010, at 10:21 AM, Scott Long wrote: >>>=20 >>>> On Jun 11, 2010, at 11:04 AM, Marcel Moolenaar wrote: >>>>>=20 >>>>> On Jun 11, 2010, at 9:12 AM, Scott Long wrote: >>>>>=20 >>>>>> On Jun 11, 2010, at 5:51 AM, John Baldwin wrote: >>>>>>> On Thursday 10 June 2010 11:00:33 pm Marcel Moolenaar wrote: >>>>>>>> Author: marcel >>>>>>>> Date: Fri Jun 11 03:00:32 2010 >>>>>>>> New Revision: 209026 >>>>>>>> URL: http://svn.freebsd.org/changeset/base/209026 >>>>>>>>=20 >>>>>>>> Log: >>>>>>>> Bump MAX_BPAGES from 256 to 1024. It seems that a few drivers, = bge(4) >>>>>>>> in particular, do not handle deferred DMA map load operations = at all. >>>>>>>> Any error, and especially EINPROGRESS, is treated as a hard = error and >>>>>>>> typically abort the current operation. The fact that the busdma = code >>>>>>>> queues the load operation for when resources (i.e. bounce = buffers in >>>>>>>> this particular case) are available makes this especially = problematic. >>>>>>>> Bounce buffering, unlike what the PR synopsis would suggest, = works >>>>>>>> fine. >>>>>>>>=20 >>>>>>>> While on the subject, properly implement swi_vm(). >>>>>>>=20 >>>>>>> NIC drivers do not handle deferred load operations at all (note = that=20 >>>>>>> bus_dmamap_load_mbuf() and bus_dmamap_load_mbuf_sg() enforce = BUS_DMA_NOWAIT). >>>>>>> It is common practice to just drop the packet in that case. >>>>>>>=20 >>>>>>=20 >>>>>> Yes, long ago when network drivers started being converted to = busdma, it was agreed that EINPROGRESS simply doesn't make sense for = them. Any platform that winds up making extensive use of bounce buffers = for network hardware is going to perform poorly no matter what, and = should hopefully have some sort of IOMMU that can be used instead. >>>>>=20 >>>>> Unfortunately things aren't as simple as is presented. >>>>>=20 >>>>> For one, bge(4) wedges as soon as the platform runs out of bounce >>>>> buffers when they're needed. The box needs to be reset in order to >>>>> get the interface back. I pick any implementation that remains >>>>> functional over a mis-optimized one that breaks. Deferred load >>>>> operations are more performance optimal than failure is. >>>>>=20 >>>>=20 >>>> This sounds like a bug in the bge driver. I don't see if through = casual inspection, but the driver should be able to either drop the mbuf = entirely, or requeue it on the ifq and then restart the ifq later. >>>>=20 >>>>> Also: the kernel does nothing to guarantee maximum availability >>>>> of DMA-able memory under load, so bounce buffers (or use of I/O >>>>> MMUs for that matter) are a reality. Here too the performance >>>>> argument doesn't necessarily hold because the kernel may be >>>>> busy with more than just sending and receiving packets and the >>>>> need to defer load operations is very appropriate. If the >>>>> alternative is just dropped packets, I'm fine with that too, >>>>> but I for one cannot say that *not* filling a H/W ring with >>>>> buffers is not going to wedge the hardware in some cases. >>>>>=20 >>>>> Plus: SGI Altix does not have any DMA-able memory for 32-bit >>>>> hardware. The need for an I/O MMU is absolute and since there >>>>> are typically less mapping registers than packets on a ring, >>>>> the need for deferred operation seems quite acceptable if the >>>>> alternative is, again, failure to operate. >>>>>=20 >>>>=20 >>>> I'm not against you upping the bounce buffer limit for a particular = platform, but it's still unclear to me if (given bug-free drivers) it's = worth the effort to defer a load rather than just drop the packet and = let the stack retry it. One question that would be good to answer is = wether the failed load is happening in the RX to TX path. >>>=20 >>> RX path I believe. >>>=20 >>=20 >> I'm not clear why you even need bounce buffers for RX. The chip = supports 64bit addresses with no boundary or alignment restrictions. >>=20 >=20 > Some controllers have 4G boundary bug so bge(4) restricts dma > address space. That limitation should be reflected in the boundary attribute of the = tag, not the lowaddr/highaddr attributes. Scott