Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 26 Aug 2015 09:30:48 -0700
From:      Sean Bruno <sbruno@freebsd.org>
To:        freebsd-arch@freebsd.org
Subject:   Network card interrupt handling
Message-ID:  <55DDE9B8.4080903@freebsd.org>

next in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

We've been diagnosing what appeared to be out of order processing in
the network stack this week only to find out that the network card
driver was shoveling bits to us out of order (em).

This *seems* to be due to a design choice where the driver is allowed
to assert a "soft interrupt" to the h/w device while real interrupts
are disabled.  This allows a fake "em_msix_rx" to be started *while*
"em_handle_que" is running from the taskqueue.  We've isolated and
worked around this by setting our processing_limit in the driver to
- -1.  This means that *most* packet processing is now handled in the
MSI-X handler instead of being deferred.  Some periodic interference
is still detectable via em_local_timer() which causes one of these
"fake" interrupt assertions in the normal, card is *not* hung case.

Both functions use identical code for a start.  Both end up down
inside of em_rxeof() to process packets.  Both drop the RX lock prior
to handing the data up the network stack.

This means that the em_handle_que running from the taskqueue will be
preempted.  Dtrace confirms that this allows out of order processing
to occur at times and generates a lot of resets.

The reason I'm bringing this up on -arch and not on -net is that this
is a common design pattern in some of the Ethernet drivers.  We've
done preliminary tests on a patch that moves *all* processing of RX
packets to the rx_task taskqueue, which means that em_handle_que is
now the only path to get packets processed.

<stable10 diff>
https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff

My sense is that this is a slightly "better" method to handle the
packets but removes some immediacy from packet processing since all
processing is deferred.  However, all packet processing is now
serialized per queue, which I think is the proper implementation.

Am I smoking "le dope" here or is this the way forward?

sean
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQF8BAEBCgBmBQJV3em1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx
MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5klyYH+wX22JSRYkKMeCJGVSi1dJiM
fcd+DWZVhru2qyUNEfhBSoGEgi7HzXqaBwddy7GK2IRtbKeRlF/oebsII941SIsz
t2f35MoZunw0rWObIEw4WxxkXAajeATDjx87wozVmsZZ40JbmgZ0jKIGXiNie3Is
04pkXiIOElWqjlLtFlkITUUrOeKsN7kKbwaZAHYeFRdbUgsnxsh7fRvsFucOCgyr
CONHBGWEwu/g50YUruR+rPOHFAA1HD3dQuIoHcTjQx/uX4l5bw+8CFzzMjpw6X9d
G+boH6l1ZZ6U3uZCXOSmkPiXka7Ix8rdbUyrUrJTJrGEB7+U7rF2lSSq8cX+4pk=
=UibL
-----END PGP SIGNATURE-----



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55DDE9B8.4080903>