From owner-freebsd-net@freebsd.org Fri Feb 28 14:51:11 2020
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 53E71264689
for <freebsd-net@mailman.nyi.freebsd.org>;
Fri, 28 Feb 2020 14:51:11 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 48TXXr49F1z4HnC
for <freebsd-net@freebsd.org>; Fri, 28 Feb 2020 14:51:08 +0000 (UTC)
(envelope-from slw@zxy.spb.ru)
Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD))
(envelope-from <slw@zxy.spb.ru>)
id 1j7gyl-000LAa-Bi; Fri, 28 Feb 2020 17:51:03 +0300
Date: Fri, 28 Feb 2020 17:51:03 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Hans Petter Selasky <hps@selasky.org>
Cc: Nick Rogers <ncrogers@gmail.com>,
"freebsd-net@freebsd.org" <freebsd-net@freebsd.org>
Subject: Re: Intel ix staled under heavy load
Message-ID: <20200228145103.GQ8012@zxy.spb.ru>
References: <20200114145443.GG38096@zxy.spb.ru>
<1deada88-beae-9747-d505-5fe142dd5c37@selasky.org>
<20200114150740.GH38096@zxy.spb.ru>
<bd8a2f59-ae39-01d6-c828-ebc3c3e29eee@selasky.org>
<CAKOb=YbCUhZKMfciP2JXvr8S4KSm3j+owTwrQtc=Y-qCFKEGcw@mail.gmail.com>
<a7b3a93d-12ea-803e-5cbf-dbd1874644b3@selasky.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <a7b3a93d-12ea-803e-5cbf-dbd1874644b3@selasky.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
X-Rspamd-Queue-Id: 48TXXr49F1z4HnC
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when
checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru
X-Spamd-Result: default: False [0.57 / 15.00]; ARC_NA(0.00)[];
TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_HAS_DN(0.00)[];
RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[];
IP_SCORE(0.08)[asn: 5495(0.38), country: RU(0.01)];
MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[];
DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[];
NEURAL_SPAM_MEDIUM(0.05)[0.047,0];
NEURAL_HAM_LONG(-0.46)[-0.457,0];
TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[];
FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[];
MIME_TRACE(0.00)[0:+];
ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU];
FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[];
RCVD_COUNT_TWO(0.00)[2]
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 28 Feb 2020 14:51:11 -0000
On Wed, Jan 15, 2020 at 09:41:06AM +0100, Hans Petter Selasky wrote:
> On 2020-01-14 19:26, Nick Rogers wrote:
> > On Tue, Jan 14, 2020 at 10:09 AM Hans Petter Selasky <hps@selasky.org>
> > wrote:
> >
> >> On 2020-01-14 16:07, Slawa Olhovchenkov wrote:
> >>>> this is a known issue in iflib.
> >>> Unresolved?
> >>
> >> See mail I sent off-list.
> >>
> >
> > I would be interested to know if this is resolved or not as well.
> >
>
> No, not yet.
>
> Slawa, can you dump the iflib sysctl's when the card is in the stalled
> state?
I am see individual stalled RX queue.
For this case:
a) rxd_tail == rxd_head (in normal case w/o traffic rxd_tail == rxd_head-1)
b) rxq_fl0.cidx == rxd_tail (in normal case w/o traffic rxq_fl0.cidx == rxd_head)
For example1:
# sysctl dev.ix | grep -e rxd_ -e rxq_fl0.cidx
dev.ix.1.queue3.rxd_tail: 1693
dev.ix.1.queue3.rxd_head: 1694
dev.ix.1.queue2.rxd_tail: 1917
dev.ix.1.queue2.rxd_head: 1918
dev.ix.1.queue1.rxd_tail: 1628
dev.ix.1.queue1.rxd_head: 1629
dev.ix.1.queue0.rxd_tail: 161
dev.ix.1.queue0.rxd_head: 162
dev.ix.1.iflib.rxq3.rxq_fl0.cidx: 1694
dev.ix.1.iflib.rxq2.rxq_fl0.cidx: 1918
dev.ix.1.iflib.rxq1.rxq_fl0.cidx: 1629
dev.ix.1.iflib.rxq0.rxq_fl0.cidx: 162
dev.ix.0.queue3.rxd_tail: 1141
dev.ix.0.queue3.rxd_head: 1142
dev.ix.0.queue2.rxd_tail: 1598
dev.ix.0.queue2.rxd_head: 1598
^^^^^^^^^^^
dev.ix.0.queue1.rxd_tail: 720
dev.ix.0.queue1.rxd_head: 721
dev.ix.0.queue0.rxd_tail: 1356
dev.ix.0.queue0.rxd_head: 1356
^^^^^^^
dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 1142
dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 1598
dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 721
dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 1356
For example2:
# sysctl dev.ix | grep -e rxd_ -e rxq_fl0.cidx
dev.ix.0.queue3.rxd_tail: 205
dev.ix.0.queue3.rxd_head: 240
dev.ix.0.queue2.rxd_tail: 1566
dev.ix.0.queue2.rxd_head: 1566
^^^^^^^^^^^^^^
dev.ix.0.queue1.rxd_tail: 1371
dev.ix.0.queue1.rxd_head: 1456
dev.ix.0.queue0.rxd_tail: 1120
dev.ix.0.queue0.rxd_head: 1120
^^^^^^^^^^
dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 708
dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 1566
dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 1764
dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 1120