Date: Tue, 05 Jan 2021 22:25:02 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 252453] iflib doesn't initialize multi-queue netmap properly Message-ID: <bug-252453-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252453 Bug ID: 252453 Summary: iflib doesn't initialize multi-queue netmap properly Product: Base System Version: 12.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: brpoole@vt.edu Hello, I am testing a variety of network cards configured for multiple netmap queu= es on FreeBSD-12.2. I was debugging a panic in iflib's netmap_fl_refill() which always occurs when running pkt-gen in receive mode with multiple threads and the ice driver. That led me to discover what I think is a logic error with netmap usage under iflib. I believe iflib doesn't properly track the initial ringid. Then when multiple threads or processes each connect to a different queue on the same interface using NR_REG_ONE_NIC, rings are overwritten and re-configured. I observed this behavior by looking at the output of netmap_reset() which is called by iflib_netmap_rxq_init() which in turn is called by iflib_init_locked(). Each time a ring is configured, the function steps thr= ough all configured rings starting at 0. For comparison, I looked at the cxgbe driver, which does not use iflib. It has a first_nm_rxq variable which is a= dded to the start of all loops. Below is my procedure, followed by the output from the failing ice and igb drivers using iflib, followed by the output of the working cxgbe driver. I added CONFIG_NETMAP_DEBUG to netmap_kern.h to enable the netmap_debug varia= ble. Then I rebuilt/rebooted and at runtime set sysctl dev.netmap.debug=3D1 to e= nable NM_DEBUG_ON as well as sysctl dev.netmap.verbose=3D1. # recv is a simple netmap receiver, this opens just queue 9 ./recv netmap:ice0-9 [1921] netmap_interp_ringid ice0: tx [9,10) rx [9,10) id 9 [1994] netmap_krings_get ice0: grabbing tx [9, 10) rx [9, 10) [4121] netmap_reset ice0 TX9 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset ice0 RX9 hwofs 0 -> 0, hwtail 0 -> 0 [1083] netmap_mmap_single cdev 0xfffff80107432c00 foff 0 size 343121= 920 objp 0xfffffe0125f36968 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff801707580b0 size 343121920 p= rot 3 foff 0 [3400] netmap_poll device ice0 events 0x1 # Now start second process connecting to queue 10, also resets queue 9 ./recv netmap:ice0-10 [1921] netmap_interp_ringid ice0: tx [10,11) rx [10,11) id 10 [1994] netmap_krings_get ice0: grabbing tx [10, 11) rx [10, 11) [4121] netmap_reset ice0 TX9 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset ice0 TX10 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset ice0 RX9 hwofs 0 -> 0, hwtail 0 -> 0 [4121] netmap_reset ice0 RX10 hwofs 0 -> 0, hwtail 0 -> 0 [3400] netmap_poll device ice0 events 0x1 [3400] netmap_poll device ice0 events 0x1 [3400] netmap_poll device ice0 events 0x1 [3400] netmap_poll device ice0 events 0x1 [1083] netmap_mmap_single cdev 0xfffff80107432c00 foff 0 size 343121= 920 objp 0xfffffe0125f4f968 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff801707580c0 size 343121920 p= rot 3 foff 0 [3400] netmap_poll device ice0 events 0x1 # On to the igb driver ./recv netmap:igb4-2 [1921] netmap_interp_ringid igb4: tx [2,3) rx [2,3) id 2 [1994] netmap_krings_get igb4: grabbing tx [2, 3) rx [2, 3) [4121] netmap_reset igb4 TX2 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset igb4 RX2 hwofs 0 -> 0, hwtail 0 -> 0 [1083] netmap_mmap_single cdev 0xfffff80104b30400 foff 0 size 343019= 520 objp 0xfffffe010006d9a8 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff80104f827a0 size 343019520 p= rot 3 foff 0 [3400] netmap_poll device igb4 events 0x1 # Now start second process, again this resets queues 2 and 3 ./recv netmap:igb4-3 [1921] netmap_interp_ringid igb4: tx [3,4) rx [3,4) id 3 [1994] netmap_krings_get igb4: grabbing tx [3, 4) rx [3, 4) [4121] netmap_reset igb4 TX2 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset igb4 TX3 hwofs 0 -> 0, hwtail 1023 -> 1023 [4121] netmap_reset igb4 RX2 hwofs 0 -> 0, hwtail 0 -> 0 [3400] netmap_poll device igb4 events 0x1 [4121] netmap_reset igb4 RX3 hwofs 0 -> 0, hwtail 0 -> 0 [3400] netmap_poll device igb4 events 0x1 [1083] netmap_mmap_single cdev 0xfffff80104b30400 foff 0 size 343019= 520 objp 0xfffffe011720e9a8 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff80100087a00 size 343019520 p= rot 3 foff 0 [3400] netmap_poll device igb4 events 0x1 # As a sanity check, here's the output from cxgbe - only one queue is modif= ied ./recv netmap:cc0-9 [1921] netmap_interp_ringid cc0: tx [9,10) rx [9,10) id 9 [1994] netmap_krings_get cc0: grabbing tx [9, 10) rx [9, 10) [4121] netmap_reset cc0 RX9 hwofs 0 -> 0, hwtail 0 -> 0 [4121] netmap_reset cc0 TX9 hwofs 0 -> 0, hwtail 1022 -> 1022 [1083] netmap_mmap_single cdev 0xfffff80107432c00 foff 0 size 343121= 920 objp 0xfffffe0125f4a968 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff8016ea92f40 size 343121920 p= rot 3 foff 0 [3400] netmap_poll device cc0 events 0x1 ./recv netmap:cc0-10 [1921] netmap_interp_ringid cc0: tx [10,11) rx [10,11) id 10 [1994] netmap_krings_get cc0: grabbing tx [10, 11) rx [10, 11) [4121] netmap_reset cc0 RX10 hwofs 0 -> 0, hwtail 0 -> 0 [4121] netmap_reset cc0 TX10 hwofs 0 -> 0, hwtail 1022 -> 1022 [1083] netmap_mmap_single cdev 0xfffff80107432c00 foff 0 size 343121= 920 objp 0xfffffe0120623968 prot 3 [ 991] netmap_dev_pager_ctor handle 0xfffff8016e9a9d20 size 343121920 p= rot 3 foff 0 [3400] netmap_poll device cc0 events 0x1 --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-252453-227>