Date: Fri, 28 Aug 2015 18:52:06 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: "K. Macy" <kmacy@freebsd.org> Cc: John Baldwin <jhb@freebsd.org>, Sean Bruno <sbruno@freebsd.org>, "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: Network card interrupt handling Message-ID: <00E4073A-9AF4-4FAD-8C09-B771C26A8319@gmail.com> In-Reply-To: <CAHM0Q_N65J9OSaU=znjgJ_gEiu=M-cb9q1hrxskGSvYFhxL_NQ@mail.gmail.com> References: <55DDE9B8.4080903@freebsd.org> <24017021.PxBoCiQKDJ@ralph.baldwin.cx> <CAHM0Q_N65J9OSaU=znjgJ_gEiu=M-cb9q1hrxskGSvYFhxL_NQ@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
> On Aug 28, 2015, at 18:25, K. Macy <kmacy@freebsd.org> wrote: > >> On Aug 28, 2015 12:59 PM, "John Baldwin" <jhb@freebsd.org> wrote: >> >>> On Wednesday, August 26, 2015 09:30:48 AM Sean Bruno wrote: >>> We've been diagnosing what appeared to be out of order processing in >>> the network stack this week only to find out that the network card >>> driver was shoveling bits to us out of order (em). >>> >>> This *seems* to be due to a design choice where the driver is allowed >>> to assert a "soft interrupt" to the h/w device while real interrupts >>> are disabled. This allows a fake "em_msix_rx" to be started *while* >>> "em_handle_que" is running from the taskqueue. We've isolated and >>> worked around this by setting our processing_limit in the driver to >>> -1. This means that *most* packet processing is now handled in the >>> MSI-X handler instead of being deferred. Some periodic interference >>> is still detectable via em_local_timer() which causes one of these >>> "fake" interrupt assertions in the normal, card is *not* hung case. >>> >>> Both functions use identical code for a start. Both end up down >>> inside of em_rxeof() to process packets. Both drop the RX lock prior >>> to handing the data up the network stack. >>> >>> This means that the em_handle_que running from the taskqueue will be >>> preempted. Dtrace confirms that this allows out of order processing >>> to occur at times and generates a lot of resets. >>> >>> The reason I'm bringing this up on -arch and not on -net is that this >>> is a common design pattern in some of the Ethernet drivers. We've >>> done preliminary tests on a patch that moves *all* processing of RX >>> packets to the rx_task taskqueue, which means that em_handle_que is >>> now the only path to get packets processed. >> >> It is only a common pattern in the Intel drivers. :-/ We (collectively) >> spent quite a while fixing this in ixgbe and igb. Longer (hopefully more >> like medium) term I have an update to the interrupt API I want to push in >> that allows drivers to manually schedule interrupt handlers using an >> 'hwi' API to replace the manual taskqueues. This also ensures that >> the handler that dequeues packets is only ever running in an ithread >> context and never concurrently. > > Jeff has a generalization of the net_task infrastructure used at Nokia > called grouptaskq that I've used for iflib. That does essentially what you > refer to. I've converted ixl and am currently about to test an ixgbe > conversion. I anticipate converting mlxen, all Intel drivers as well as the > remaining drivers with device specific code in netmap. The one catch is > finding someone who will publicly admit to owning re hardware so that I can > buy it from him and test my changes. > > Cheers. I have 2 re NICs in my fileserver at home (Asus went cheap on some of their MBs a while back), but the cards shouldn't cost more than $15 + shipping (look for "Realtek 8169" on Google). HTH! -NGiehome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?00E4073A-9AF4-4FAD-8C09-B771C26A8319>