Date: Mon, 31 Aug 2015 01:51:32 +0430 From: Hooman Fazaeli <hoomanfazaeli@gmail.com> To: Sean Bruno <sbruno@freebsd.org> Cc: freebsd-arch@freebsd.org Subject: Re: Network card interrupt handling Message-ID: <55E373DC.3060306@gmail.com> In-Reply-To: <55DDE9B8.4080903@freebsd.org> References: <55DDE9B8.4080903@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/26/2015 9:00 PM, Sean Bruno wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > We've been diagnosing what appeared to be out of order processing in > the network stack this week only to find out that the network card > driver was shoveling bits to us out of order (em). > > This *seems* to be due to a design choice where the driver is allowed > to assert a "soft interrupt" to the h/w device while real interrupts > are disabled. This allows a fake "em_msix_rx" to be started *while* > "em_handle_que" is running from the taskqueue. We've isolated and > worked around this by setting our processing_limit in the driver to > - -1. This means that *most* packet processing is now handled in the > MSI-X handler instead of being deferred. Some periodic interference > is still detectable via em_local_timer() which causes one of these > "fake" interrupt assertions in the normal, card is *not* hung case. > > Both functions use identical code for a start. Both end up down > inside of em_rxeof() to process packets. Both drop the RX lock prior > to handing the data up the network stack. > > This means that the em_handle_que running from the taskqueue will be > preempted. Dtrace confirms that this allows out of order processing > to occur at times and generates a lot of resets. > > The reason I'm bringing this up on -arch and not on -net is that this > is a common design pattern in some of the Ethernet drivers. We've > done preliminary tests on a patch that moves *all* processing of RX > packets to the rx_task taskqueue, which means that em_handle_que is > now the only path to get packets processed. > > <stable10 diff> > https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff > > My sense is that this is a slightly "better" method to handle the > packets but removes some immediacy from packet processing since all > processing is deferred. However, all packet processing is now > serialized per queue, which I think is the proper implementation. > > Am I smoking "le dope" here or is this the way forward? > > Which versions of the driver have this problem? -- Best regards Hooman Fazaeli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55E373DC.3060306>