From owner-freebsd-net@freebsd.org Tue Jan 29 11:25:26 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9170914B8AE2 for ; Tue, 29 Jan 2019 11:25:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2BACC81AD0 for ; Tue, 29 Jan 2019 11:25:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E3B7614B8AD3; Tue, 29 Jan 2019 11:25:25 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C128814B8ACF for ; Tue, 29 Jan 2019 11:25:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5F7ED81AC5 for ; Tue, 29 Jan 2019 11:25:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 72303116CF for ; Tue, 29 Jan 2019 11:25:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x0TBPO4l024100 for ; Tue, 29 Jan 2019 11:25:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x0TBPOSq024099 for net@FreeBSD.org; Tue, 29 Jan 2019 11:25:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 230465] ixl: not working in netmap mode Date: Tue, 29 Jan 2019 11:25:24 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: halfling@halfling.com.br X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2019 11:25:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230465 --- Comment #20 from Charles Goncalves --- (In reply to Vincenzo Maffione from comment #19) Oh yes now I remember it's about 5 months. I had set this to max_queues =3D= 1 (in /boot/loader.conf) to test with pkt-gen, because with max_queues =3D 8 doen= s't work. Output of pkt-gen with hw.ixl.max_queues=3D1: # pkt-gen -i ixl0 -f tx 945.763330 main [2593] interface is ixl0 945.763376 main [2727] running on 1 cpus (have 8) 945.763602 extract_ip_range [468] range is 10.0.0.1:1234 to 10.0.0.1:1234 945.763619 extract_ip_range [468] range is 10.1.0.1:1234 to 10.1.0.1:1234 945.907652 main [2822] mapped 294020KB at 0x801600000 Sending on netmap:ixl0: 1 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) 945.907746 main [2919] Sending 512 packets every 0.000000000 s 945.907791 start_threads [2274] Wait 2 secs for phy reset 947.974562 start_threads [2276] Ready... 947.974737 sender_body [1464] start, fd 3 main_fd 3 947.989037 sender_body [1546] drop copy 949.001809 main_thread [2364] 4.285 Mpps (4.401 Mpkts 2.112 Gbps in 1027068 usec) 343.53 avg_batch 0 min_space 950.016811 main_thread [2364] 4.265 Mpps (4.329 Mpkts 2.078 Gbps in 1015002 usec) 341.05 avg_batch 99999 min_space 951.053574 main_thread [2364] 4.212 Mpps (4.367 Mpkts 2.096 Gbps in 1036763 usec) 341.39 avg_batch 99999 min_space 952.054597 main_thread [2364] 4.262 Mpps (4.266 Mpkts 2.048 Gbps in 1001023 usec) 341.66 avg_batch 99999 min_space Now with hw.ixl.max_queues=3D8: # pkt-gen -i ixl0 -f tx 734.500918 main [2593] interface is ixl0 734.500963 main [2727] running on 1 cpus (have 8) 734.501188 extract_ip_range [468] range is 10.0.0.1:1234 to 10.0.0.1:1234 734.501205 extract_ip_range [468] range is 10.1.0.1:1234 to 10.1.0.1:1234 734.651421 main [2822] mapped 294020KB at 0x801600000 Sending on netmap:ixl0: 8 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) 734.651514 main [2919] Sending 512 packets every 0.000000000 s 734.651558 start_threads [2274] Wait 2 secs for phy reset 736.666615 start_threads [2276] Ready... 736.666799 sender_body [1464] start, fd 3 main_fd 3 737.506822 sender_body [1538] poll error on 3 ring 0-7 737.677616 main_thread [2364] 8.103 Kpps (8.191 Kpkts 3.932 Mbps in 1010813 usec) 511.94 avg_batch 0 min_space 737.856747 main_thread [2391] ouch, thread 0 exited with error Sent 8191 packets 491460 bytes 16 events 60 bytes each in -1548760736.67 seconds. Speed: -0.000 pps Bandwidth: -0.003 bps (raw -0.004 bps). Average batch: 51= 1.94 pkts Then with max_queues=3D8 I have errors in my /var/log/messages: Jan 29 09:21:29 rt1 kernel: 888.923727 [1637] nm_txsync_prologue ixl0 TX1: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n || kring->nr_hwtail >=3D n' h 0 c 0 t -1678351873 rh 512 rc 512 rt -1678351873= hc 512 ht -1678351873 Jan 29 09:21:29 rt1 kernel: 888.981468 [1758] netmap_ring_reinit cal= led for ixl0 TX1 Jan 29 09:21:29 rt1 kernel: 888.998372 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.013966 [1787] netmap_ring_reinit ixl0 TX1 reinit, cur 0 -> 512 tail -1678351873 -> -1678351873 Jan 29 09:21:29 rt1 kernel: 889.041778 [1637] nm_txsync_prologue ixl0 TX2: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n || kring->nr_hwtail >=3D n' h 0 c 0 t -1676512769 rh 512 rc 512 rt -1676512769= hc 512 ht -1676512769 Jan 29 09:21:29 rt1 kernel: 889.099769 [1758] netmap_ring_reinit cal= led for ixl0 TX2 Jan 29 09:21:29 rt1 kernel: 889.116664 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.132258 [1787] netmap_ring_reinit ixl0 TX2 reinit, cur 0 -> 512 tail -1676512769 -> -1676512769 Jan 29 09:21:29 rt1 kernel: 889.160084 [1637] nm_txsync_prologue ixl0 TX3: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n || kring->nr_hwtail >=3D n' h 0 c 0 t -1674673665 rh 512 rc 512 rt -1674673665= hc 512 ht -1674673665 Jan 29 09:21:29 rt1 kernel: 889.218053 [1758] netmap_ring_reinit cal= led for ixl0 TX3 Jan 29 09:21:29 rt1 kernel: 889.234957 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.250560 [1787] netmap_ring_reinit ixl0 TX3 reinit, cur 0 -> 512 tail -1674673665 -> -1674673665 Jan 29 09:21:29 rt1 kernel: 889.278372 [1637] nm_txsync_prologue ixl0 TX4: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n || kring->nr_hwtail >=3D n' h 0 c 0 t -1672834561 rh 512 rc 512 rt -1672834561= hc 512 ht -1672834561 Jan 29 09:21:29 rt1 kernel: 889.336348 [1758] netmap_ring_reinit cal= led for ixl0 TX4 Jan 29 09:21:29 rt1 kernel: 889.353258 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.368847 [1787] netmap_ring_reinit ixl0 TX4 reinit, cur 0 -> 512 tail -1672834561 -> -1672834561 Jan 29 09:21:29 rt1 kernel: 889.396679 [1637] nm_txsync_prologue ixl0 TX5: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n || kring->nr_hwtail >=3D n' h 0 c 0 t -1670995457 rh 512 rc 512 rt -1670995457= hc 512 ht -1670995457 Jan 29 09:21:29 rt1 kernel: 889.454648 [1758] netmap_ring_reinit cal= led for ixl0 TX5 Jan 29 09:21:29 rt1 kernel: 889.471553 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.487148 [1787] netmap_ring_reinit ixl0 TX5 reinit, cur 0 -> 512 tail -1670995457 -> -1670995457 Jan 29 09:21:29 rt1 kernel: 889.514967 [1637] nm_txsync_prologue ixl0 TX6: fail 'kring->nr_hwcur >=3D n || kring->rhead >=3D n || kring->rtail >= =3D n ||=20 Jan 29 09:21:29 rt1 kernel: kring->nr_hwtail >=3D n' h 0 c 0 t -1669156353 = rh 512 rc 512 rt -1669156353 hc 512 ht -1669156353 Jan 29 09:21:29 rt1 kernel: 889.572947 [1758] netmap_ring_reinit cal= led for ixl0 TX6 Jan 29 09:21:29 rt1 kernel: 889.589850 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.605444 [1787] netmap_ring_reinit ixl0 TX6 reinit, cur 0 -> 512 tail -1669156353 -> -1669156353 Jan 29 09:21:29 rt1 kernel: 889.633261 [1758] netmap_ring_reinit cal= led for ixl0 TX7 Jan 29 09:21:29 rt1 kernel: 889.650170 [1783] netmap_ring_reinit tot= al 1 errors Jan 29 09:21:29 rt1 kernel: 889.665760 [1787] netmap_ring_reinit ixl0 TX7 reinit, cur 0 -> 512 tail -1667321345 -> -1667321345 I also test this in my netmap application and with these tests I think that when a application try to write in txring > 0 like TX1, TX2 or so this error occurs --=20 You are receiving this mail because: You are on the CC list for the bug.=