Date: Wed, 26 Aug 2015 09:36:06 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Sean Bruno <sbruno@freebsd.org> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: Network card interrupt handling Message-ID: <CAFOYbcnTw-B3jQjwNDGjuORcRHqu9tSgpvYY4vuiAUOpwB-KjA@mail.gmail.com> In-Reply-To: <55DDE9B8.4080903@freebsd.org> References: <55DDE9B8.4080903@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I recall actually trying something like this once myself Sean, but if memory serves the performance was poor enough that I decided against pursuing it. Still, maybe it deserves further investigation. Jack On Wed, Aug 26, 2015 at 9:30 AM, Sean Bruno <sbruno@freebsd.org> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > We've been diagnosing what appeared to be out of order processing in > the network stack this week only to find out that the network card > driver was shoveling bits to us out of order (em). > > This *seems* to be due to a design choice where the driver is allowed > to assert a "soft interrupt" to the h/w device while real interrupts > are disabled. This allows a fake "em_msix_rx" to be started *while* > "em_handle_que" is running from the taskqueue. We've isolated and > worked around this by setting our processing_limit in the driver to > - -1. This means that *most* packet processing is now handled in the > MSI-X handler instead of being deferred. Some periodic interference > is still detectable via em_local_timer() which causes one of these > "fake" interrupt assertions in the normal, card is *not* hung case. > > Both functions use identical code for a start. Both end up down > inside of em_rxeof() to process packets. Both drop the RX lock prior > to handing the data up the network stack. > > This means that the em_handle_que running from the taskqueue will be > preempted. Dtrace confirms that this allows out of order processing > to occur at times and generates a lot of resets. > > The reason I'm bringing this up on -arch and not on -net is that this > is a common design pattern in some of the Ethernet drivers. We've > done preliminary tests on a patch that moves *all* processing of RX > packets to the rx_task taskqueue, which means that em_handle_que is > now the only path to get packets processed. > > <stable10 diff> > https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff > > My sense is that this is a slightly "better" method to handle the > packets but removes some immediacy from packet processing since all > processing is deferred. However, all packet processing is now > serialized per queue, which I think is the proper implementation. > > Am I smoking "le dope" here or is this the way forward? > > sean > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQF8BAEBCgBmBQJV3em1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx > MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5klyYH+wX22JSRYkKMeCJGVSi1dJiM > fcd+DWZVhru2qyUNEfhBSoGEgi7HzXqaBwddy7GK2IRtbKeRlF/oebsII941SIsz > t2f35MoZunw0rWObIEw4WxxkXAajeATDjx87wozVmsZZ40JbmgZ0jKIGXiNie3Is > 04pkXiIOElWqjlLtFlkITUUrOeKsN7kKbwaZAHYeFRdbUgsnxsh7fRvsFucOCgyr > CONHBGWEwu/g50YUruR+rPOHFAA1HD3dQuIoHcTjQx/uX4l5bw+8CFzzMjpw6X9d > G+boH6l1ZZ6U3uZCXOSmkPiXka7Ix8rdbUyrUrJTJrGEB7+U7rF2lSSq8cX+4pk= > =UibL > -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbcnTw-B3jQjwNDGjuORcRHqu9tSgpvYY4vuiAUOpwB-KjA>