From owner-freebsd-net@freebsd.org Tue Jan 14 14:54:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA39D1ED5DB for ; Tue, 14 Jan 2020 14:54:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47xtlw5GxYz4FF4 for ; Tue, 14 Jan 2020 14:54:52 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1irNad-000MAj-T8 for freebsd-net@freebsd.org; Tue, 14 Jan 2020 17:54:43 +0300 Date: Tue, 14 Jan 2020 17:54:43 +0300 From: Slawa Olhovchenkov To: freebsd-net@freebsd.org Subject: Intel ix staled under heavy load Message-ID: <20200114145443.GG38096@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 47xtlw5GxYz4FF4 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [0.88 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.37)[-0.374,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_LONG(0.26)[0.261,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.09)[asn: 5495(0.43), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2020 14:54:53 -0000 I am see strange behavior of Intel ix card (82599ES 10-Gigabit SFI/SFP+ Network Connection). Under heavy load in NETMAP mode NIC is lost input traffic, i.e. all packets counted as `dev.ix.1.mac_stats.rx_missed_packets`, netstat show high traffic rate: # netstat -nbI ix1 1 input ix1 output packets errs idrops bytes packets errs bytes colls 14204463 0 0 965903486 2 0 414 0 14204467 0 0 965903756 0 0 0 0 14204478 0 0 965904512 2 0 193 0 14204470 0 0 965903962 0 0 0 0 my program don't get any input packets, all input netmap rings staled (cur/head/tail): ix1#0 640/640/640 ix1#1 52/52/52 ix1#2 1012/1012/1012 ix1#3 631/631/631 Hardware rings also staled: dev.ix.1.queue3.rxd_tail: 631 dev.ix.1.queue3.rxd_head: 631 dev.ix.1.queue2.rxd_tail: 1011 dev.ix.1.queue2.rxd_head: 1012 dev.ix.1.queue1.rxd_tail: 51 dev.ix.1.queue1.rxd_head: 52 dev.ix.1.queue0.rxd_tail: 639 dev.ix.1.queue0.rxd_head: 640 iflib rings also staled: dev.ix.1.iflib.rxq3.rxq_fl0.credits: 0 dev.ix.1.iflib.rxq3.rxq_fl0.cidx: 631 dev.ix.1.iflib.rxq3.rxq_fl0.pidx: 0 dev.ix.1.iflib.rxq2.rxq_fl0.credits: 0 dev.ix.1.iflib.rxq2.rxq_fl0.cidx: 1012 dev.ix.1.iflib.rxq2.rxq_fl0.pidx: 0 dev.ix.1.iflib.rxq1.rxq_fl0.credits: 0 dev.ix.1.iflib.rxq1.rxq_fl0.cidx: 52 dev.ix.1.iflib.rxq1.rxq_fl0.pidx: 0 dev.ix.1.iflib.rxq0.rxq_fl0.credits: 0 dev.ix.1.iflib.rxq0.rxq_fl0.cidx: 640 dev.ix.1.iflib.rxq0.rxq_fl0.pidx: 0 Interraupt's mostly don't generated: # sysctl dev.ix.1 | grep irq ; sleep 10 ; sysctl dev.ix.1 | grep irq dev.ix.1.queue3.irqs: 4375622959 dev.ix.1.queue2.irqs: 4282320604 dev.ix.1.queue1.irqs: 4196905785 dev.ix.1.queue0.irqs: 4195682690 dev.ix.1.link_irq: 314 dev.ix.1.queue3.irqs: 4375622959 dev.ix.1.queue2.irqs: 4282320610 dev.ix.1.queue1.irqs: 4196905785 dev.ix.1.queue0.irqs: 4195682690 What is problem? How to resolve this?