Date: Tue, 12 Mar 2013 17:20:01 GMT From: "Charbon, Julien" <jcharbon@verisign.com> To: freebsd-net@FreeBSD.org Subject: Re: kern/176446: [netinet] [patch] Concurrency in ixgbe driving out-of-order packet process and spurious RST Message-ID: <201303121720.r2CHK1vp038249@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/176446; it has been noted by GNATS.
From: "Charbon, Julien" <jcharbon@verisign.com>
To: Cc: bug-followup@freebsd.org,
"De La Gueronniere, Marc" <mdelagueronniere@verisign.com>
Subject: Re: kern/176446: [netinet] [patch] Concurrency in ixgbe driving out-of-order
packet process and spurious RST
Date: Tue, 12 Mar 2013 14:59:11 +0100
This is a multi-part message in MIME format.
--------------050609010000050001050509
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
On 3/7/13 11:11 AM, Charbon, Julien wrote:
> On 2/28/13 8:10 PM, Charbon, Julien wrote:
>> On 2/28/13 4:57 PM, John Baldwin wrote:
>>> Can you try the fixes from http://svnweb.freebsd.org/base?view=revision&revision=240968?
>>
>> Actually, Marc (I CC'ed him) did find the r240968 fix for concurrency
>> between ixgbe_msix_que() and ixgbe_handle_que(), and made a backport for
>> release-8.3.0 (see patch [1] below). However, the issue was still
>> reproducible, then Marc found another place for concurrency from
>> ixgbe_local_timer() and fix it (see patch [2]). But it was still not
>> enough, and he found a last place for concurrency due to
>> ixgbe_rearm_queues() call (see patch [3]). We all these patches
>> applied, we were not able to reproduce this issue.
>
> Just for the record: As expected this issue is reproducible on
> 9.1-RELEASE:
Just for the record: As expected this issue is reproducible also on
10.0-CURRENT:
# uname -a
FreeBSD atlas 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r248173M: Tue Mar 12
07:52:58 UTC 2013
jcharbon@atlas:/usr/obj/app/jcharbon/head/sys/GENERIC amd64
1. Enable TCP debug log:
# sysctl net.inet.tcp.log_debug=1
net.inet.tcp.log_debug: 1 -> 1
2. Load a TCP service with numerous small requests/responses:
3. Look in /var/log/debug.log for the pattern:
Mar 12 10:31:22 atlas kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080; syncache_socket: in_pcbconnect failed with error 48
Mar 12 10:31:22 atlas-dl360-4 kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080 tcpflags 0x10<ACK>; tcp_input: Listen socket:
Socket allocation failed due to limits or memory shortage, sending RST
Mar 12 10:31:22 atlas-dl360-4 kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST
without matching syncache entry (possibly syncookie only), segment ignored
Joined the patch we use to fix this issue in 10-CURRENT.
--
Julien
--------------050609010000050001050509
Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0";
name="ixgbe.c.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="ixgbe.c.patch"
Index: sys/dev/ixgbe/ixgbe.c
===================================================================
--- sys/dev/ixgbe/ixgbe.c (revision 248173)
+++ sys/dev/ixgbe/ixgbe.c (working copy)
@@ -2038,14 +2038,14 @@
(paused == 0))
++hung;
else if (txr->queue_status == IXGBE_QUEUE_WORKING)
- taskqueue_enqueue(que->tq, &que->que_task);
+ taskqueue_enqueue(que->tq, &txr->txq_task);
}
/* Only truely watchdog if all queues show hung */
if (hung == adapter->num_queues)
goto watchdog;
out:
- ixgbe_rearm_queues(adapter, adapter->que_mask);
+ // ixgbe_rearm_queues(adapter, adapter->que_mask);
callout_reset(&adapter->timer, hz, ixgbe_local_timer, adapter);
return;
@@ -4575,7 +4575,7 @@
** Schedule another interrupt if so.
*/
if ((staterr & IXGBE_RXD_STAT_DD) != 0) {
- ixgbe_rearm_queues(adapter, (u64)(1 << que->msix));
+ // ixgbe_rearm_queues(adapter, (u64)(1 << que->msix));
return (TRUE);
}
--------------050609010000050001050509--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201303121720.r2CHK1vp038249>