From owner-freebsd-arch@freebsd.org Wed Aug 26 16:30:51 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB31C9C32AF for ; Wed, 26 Aug 2015 16:30:51 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D224C809 for ; Wed, 26 Aug 2015 16:30:51 +0000 (UTC) (envelope-from sbruno@freebsd.org) Received: from [192.168.200.200] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 490AB193655 for ; Wed, 26 Aug 2015 16:30:49 +0000 (UTC) To: freebsd-arch@freebsd.org From: Sean Bruno Subject: Network card interrupt handling X-Enigmail-Draft-Status: N1210 Message-ID: <55DDE9B8.4080903@freebsd.org> Date: Wed, 26 Aug 2015 09:30:48 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 16:30:52 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 We've been diagnosing what appeared to be out of order processing in the network stack this week only to find out that the network card driver was shoveling bits to us out of order (em). This *seems* to be due to a design choice where the driver is allowed to assert a "soft interrupt" to the h/w device while real interrupts are disabled. This allows a fake "em_msix_rx" to be started *while* "em_handle_que" is running from the taskqueue. We've isolated and worked around this by setting our processing_limit in the driver to - -1. This means that *most* packet processing is now handled in the MSI-X handler instead of being deferred. Some periodic interference is still detectable via em_local_timer() which causes one of these "fake" interrupt assertions in the normal, card is *not* hung case. Both functions use identical code for a start. Both end up down inside of em_rxeof() to process packets. Both drop the RX lock prior to handing the data up the network stack. This means that the em_handle_que running from the taskqueue will be preempted. Dtrace confirms that this allows out of order processing to occur at times and generates a lot of resets. The reason I'm bringing this up on -arch and not on -net is that this is a common design pattern in some of the Ethernet drivers. We've done preliminary tests on a patch that moves *all* processing of RX packets to the rx_task taskqueue, which means that em_handle_que is now the only path to get packets processed. https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff My sense is that this is a slightly "better" method to handle the packets but removes some immediacy from packet processing since all processing is deferred. However, all packet processing is now serialized per queue, which I think is the proper implementation. Am I smoking "le dope" here or is this the way forward? sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJV3em1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5klyYH+wX22JSRYkKMeCJGVSi1dJiM fcd+DWZVhru2qyUNEfhBSoGEgi7HzXqaBwddy7GK2IRtbKeRlF/oebsII941SIsz t2f35MoZunw0rWObIEw4WxxkXAajeATDjx87wozVmsZZ40JbmgZ0jKIGXiNie3Is 04pkXiIOElWqjlLtFlkITUUrOeKsN7kKbwaZAHYeFRdbUgsnxsh7fRvsFucOCgyr CONHBGWEwu/g50YUruR+rPOHFAA1HD3dQuIoHcTjQx/uX4l5bw+8CFzzMjpw6X9d G+boH6l1ZZ6U3uZCXOSmkPiXka7Ix8rdbUyrUrJTJrGEB7+U7rF2lSSq8cX+4pk= =UibL -----END PGP SIGNATURE-----