From owner-freebsd-arch@freebsd.org Sun Aug 30 21:21:40 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 030719C67C7 for ; Sun, 30 Aug 2015 21:21:40 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91F8CCF2; Sun, 30 Aug 2015 21:21:39 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: by wicne3 with SMTP id ne3so58419725wic.0; Sun, 30 Aug 2015 14:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=6JHzySLlG4jGQX+Shkwu2ewVXTwSpj3uohSYUu6htAQ=; b=kkD1H4mhHmxn5fhkvErKgXIgfM846EaIfmkpOyRB1rWagFbu/CT+DQoqkPOB2dvSbF tR41UD6hyhPlIkwIbIM2Uu0NVVkRHl9ec1n3GX4CPCKN6b7Ur0z99MgoPGSsVBBBMTNV 9tZfEiMiBp+FoavBf16bI6ASwYRmqm0+W5minri6uBD3agr+2zxtc4jZwFHNDti/Yu3/ A5JbaAQTnyZdLqrJ2MqTkHgS4Wve1Q6YTDTRF1sMZQaSZy89EW631u1ytM23wGgGgPYS TDlxY9xRbJN95c8xPJLzjutnR0Uoa53r/Lpi6khZIZow5cErSx1a9uvfxPw72FvOW89s HYzQ== X-Received: by 10.180.189.108 with SMTP id gh12mr5132512wic.53.1440969697901; Sun, 30 Aug 2015 14:21:37 -0700 (PDT) Received: from [192.168.2.30] ([2.190.161.53]) by smtp.googlemail.com with ESMTPSA id bq7sm18994000wjc.31.2015.08.30.14.21.35 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 30 Aug 2015 14:21:37 -0700 (PDT) Message-ID: <55E373DC.3060306@gmail.com> Date: Mon, 31 Aug 2015 01:51:32 +0430 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3 MIME-Version: 1.0 To: Sean Bruno CC: freebsd-arch@freebsd.org Subject: Re: Network card interrupt handling References: <55DDE9B8.4080903@freebsd.org> In-Reply-To: <55DDE9B8.4080903@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Aug 2015 21:21:40 -0000 On 8/26/2015 9:00 PM, Sean Bruno wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > We've been diagnosing what appeared to be out of order processing in > the network stack this week only to find out that the network card > driver was shoveling bits to us out of order (em). > > This *seems* to be due to a design choice where the driver is allowed > to assert a "soft interrupt" to the h/w device while real interrupts > are disabled. This allows a fake "em_msix_rx" to be started *while* > "em_handle_que" is running from the taskqueue. We've isolated and > worked around this by setting our processing_limit in the driver to > - -1. This means that *most* packet processing is now handled in the > MSI-X handler instead of being deferred. Some periodic interference > is still detectable via em_local_timer() which causes one of these > "fake" interrupt assertions in the normal, card is *not* hung case. > > Both functions use identical code for a start. Both end up down > inside of em_rxeof() to process packets. Both drop the RX lock prior > to handing the data up the network stack. > > This means that the em_handle_que running from the taskqueue will be > preempted. Dtrace confirms that this allows out of order processing > to occur at times and generates a lot of resets. > > The reason I'm bringing this up on -arch and not on -net is that this > is a common design pattern in some of the Ethernet drivers. We've > done preliminary tests on a patch that moves *all* processing of RX > packets to the rx_task taskqueue, which means that em_handle_que is > now the only path to get packets processed. > > > https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff > > My sense is that this is a slightly "better" method to handle the > packets but removes some immediacy from packet processing since all > processing is deferred. However, all packet processing is now > serialized per queue, which I think is the proper implementation. > > Am I smoking "le dope" here or is this the way forward? > > Which versions of the driver have this problem? -- Best regards Hooman Fazaeli