From owner-freebsd-net@freebsd.org Sun Jan 17 21:00:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27DA2A86D89 for ; Sun, 17 Jan 2016 21:00:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F2E810A2 for ; Sun, 17 Jan 2016 21:00:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0HL01XW084776 for ; Sun, 17 Jan 2016 21:00:02 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201601172100.u0HL01XW084776@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-net@FreeBSD.org Subject: Problem reports for freebsd-net@FreeBSD.org that need special attention Date: Sun, 17 Jan 2016 21:00:02 +0000 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2016 21:00:03 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 203422 | mpd/ppoe not working with re(4) with revision 285 New | 203175 | Daily kernel crashes in tcp_twclose
Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C521A85765 for ; Sun, 17 Jan 2016 21:18:57 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E62681EC9 for ; Sun, 17 Jan 2016 21:18:56 +0000 (UTC) (envelope-from jilles@stack.nl) Received: by mailman.ysv.freebsd.org (Postfix) id E3554A85764; Sun, 17 Jan 2016 21:18:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C91A9A85763 for ; Sun, 17 Jan 2016 21:18:56 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 60F471EC8 for ; Sun, 17 Jan 2016 21:18:56 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id C2C98B806B; Sun, 17 Jan 2016 22:18:53 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id AD77C28494; Sun, 17 Jan 2016 22:18:53 +0100 (CET) Date: Sun, 17 Jan 2016 22:18:53 +0100 From: Jilles Tjoelker To: Konstantin Belousov Cc: Boris Astardzhiev , net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160117211853.GA37847@stack.nl> References: <20160108172323.W1815@besplex.bde.org> <20160108075815.3243.qmail@f5-external.bushwire.net> <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160116202534.GK3942@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2016 21:18:57 -0000 On Sat, Jan 16, 2016 at 10:25:34PM +0200, Konstantin Belousov wrote: > On Sat, Jan 16, 2016 at 09:56:57PM +0200, Konstantin Belousov wrote: > > After thinking some more, I believe I managed to construct a possible > > way to implement this, in libc, with some libthr extensions. Basically, > > the idea is to have function pthread_cancel_is_pending_np(), which > > would return the state of pending cancel. For some time I thought that > > this cannot work, because cancellation could happen between call to > > the cancel_is_pending() and next recvmmsg(). But, libc has a privilege > > of having access to the syscalls without libthr interposing, just > > call __sys_recvmmsg(), which would give EINTR on the cancel attempt. > > This is an implementation detail, but we can rely on it in implementation. > > In other words, the structure of the code would be like this > > for (i = 0; i < vlen; i++) { > > if (pthread_cancel_is_pending_np()) > > goto out; > Right after writing the text and hitting send, I realized that the > pthread_cancel_is_pending_np() is not needed at all. You get EINTR > from __sys_recvmsg() on the cancel attempt, so everything would just > work without the function. > The crusial part is to use __sys_recvmsg instead of interposable > _recvmsg(). This will typically work (if the cancellation occurs while blocked inside __sys_recvmsg()) but has the usual problem of relying on [EINTR]: lost wakeups. This is certainly less bad than using the interposable recvmsg(), though, which would discard the already received data. As a slight modification, the first recvmsg could use the interposable version, since there is no pending data at that point. This avoids needing to call pthread_testcancel() manually. The regular cancellation code closes this race window using the undocumented thr_wake() system call, on the thread itself, in the signal handler for the cancellation signal. This causes the next attempt to sleep(9) to fail with [EINTR]. (On another note, it appears to be possible for user code (cleanup handlers and pthread_key_create() destructors) to be called with thr_wake() still active, if the cancellation signal handler is called immediately after the cancellation point system call returns.) The race in recvmmsg could be removed using this mechanism but it requires either a separate version of recvmmsg in libthr or a new interface in libthr. I imagine the new interface as a new cancellation type which causes cancellation point functions such as recvmsg() to fail with a new errno when cancelled while leaving cancellation pending. This seems conceptually possible but adds some code to the common path for cancellation points. A new version of pthread_testcancel() with a return value would be needed. -- Jilles Tjoelker From owner-freebsd-net@freebsd.org Sun Jan 17 21:38:13 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58C5FA85DE6 for ; Sun, 17 Jan 2016 21:38:13 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3E7C31770 for ; Sun, 17 Jan 2016 21:38:13 +0000 (UTC) (envelope-from jilles@stack.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 3DC2EA85DE5; Sun, 17 Jan 2016 21:38:13 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22989A85DDF for ; Sun, 17 Jan 2016 21:38:13 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AC402176F for ; Sun, 17 Jan 2016 21:38:12 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 6AFA1B805A; Sun, 17 Jan 2016 22:38:10 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 563A328494; Sun, 17 Jan 2016 22:38:10 +0100 (CET) Date: Sun, 17 Jan 2016 22:38:10 +0100 From: Jilles Tjoelker To: Konstantin Belousov Cc: Boris Astardzhiev , net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160117213810.GA38279@stack.nl> References: <20160108075815.3243.qmail@f5-external.bushwire.net> <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160117211853.GA37847@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2016 21:38:13 -0000 On Sun, Jan 17, 2016 at 10:18:53PM +0100, Jilles Tjoelker wrote: > On Sat, Jan 16, 2016 at 10:25:34PM +0200, Konstantin Belousov wrote: > > On Sat, Jan 16, 2016 at 09:56:57PM +0200, Konstantin Belousov wrote: > > > After thinking some more, I believe I managed to construct a possible > > > way to implement this, in libc, with some libthr extensions. Basically, > > > the idea is to have function pthread_cancel_is_pending_np(), which > > > would return the state of pending cancel. For some time I thought that > > > this cannot work, because cancellation could happen between call to > > > the cancel_is_pending() and next recvmmsg(). But, libc has a privilege > > > of having access to the syscalls without libthr interposing, just > > > call __sys_recvmmsg(), which would give EINTR on the cancel attempt. > > > This is an implementation detail, but we can rely on it in implementation. > > > In other words, the structure of the code would be like this > > > for (i = 0; i < vlen; i++) { > > > if (pthread_cancel_is_pending_np()) > > > goto out; > > Right after writing the text and hitting send, I realized that the > > pthread_cancel_is_pending_np() is not needed at all. You get EINTR > > from __sys_recvmsg() on the cancel attempt, so everything would just > > work without the function. > > The crusial part is to use __sys_recvmsg instead of interposable > > _recvmsg(). > This will typically work (if the cancellation occurs while blocked > inside __sys_recvmsg()) but has the usual problem of relying on [EINTR]: > lost wakeups. This is certainly less bad than using the interposable > recvmsg(), though, which would discard the already received data. > As a slight modification, the first recvmsg could use the interposable > version, since there is no pending data at that point. This avoids > needing to call pthread_testcancel() manually. > The regular cancellation code closes this race window using the > undocumented thr_wake() system call, on the thread itself, in the signal > handler for the cancellation signal. This causes the next attempt to > sleep(9) to fail with [EINTR]. (On another note, it appears to be > possible for user code (cleanup handlers and pthread_key_create() > destructors) to be called with thr_wake() still active, if the > cancellation signal handler is called immediately after the cancellation > point system call returns.) > The race in recvmmsg could be removed using this mechanism but it > requires either a separate version of recvmmsg in libthr or a new > interface in libthr. I imagine the new interface as a new cancellation > type which causes cancellation point functions such as recvmsg() to fail > with a new errno when cancelled while leaving cancellation pending. This > seems conceptually possible but adds some code to the common path for > cancellation points. A new version of pthread_testcancel() with a return > value would be needed. I realized that the above may be interpreted as requiring cancellation to be completely fixed before recvmmsg/sendmmsg are "done". That is not my intention. Given that cancellation is not commonly used in applications, I think an approach based on __sys_recvmsg() is good enough. -- Jilles Tjoelker From owner-freebsd-net@freebsd.org Sun Jan 17 23:53:25 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FF23A8677E for ; Sun, 17 Jan 2016 23:53:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3FFAE1DF2 for ; Sun, 17 Jan 2016 23:53:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0HNrP2O055565 for ; Sun, 17 Jan 2016 23:53:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 201694] 10.2-BETA2 crashing when killing VIMAGE/VNET jails Date: Sun, 17 Jan 2016 23:53:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: crash, needs-qa, vimage X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@otoh.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2016 23:53:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D201694 --- Comment #5 from Paul Armstrong --- I'm still testing various combinations, but ALTQ seems to be the main culpr= it here. Once I remove the following from the kernel config, the crashes stop: ALTQ ALTQ_CBQ ALTQ_RED ALTQ_RIO ALTQ_HFSC ALTQ_PRIQ Once I've refined it a bit more, I'll provide a Virtual Box image. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Jan 18 01:30:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A745FA869A4 for ; Mon, 18 Jan 2016 01:30:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97F2114E7 for ; Mon, 18 Jan 2016 01:30:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0I1URvd043771 for ; Mon, 18 Jan 2016 01:30:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 201694] 10.2-BETA2 crashing when killing VIMAGE/VNET jails Date: Mon, 18 Jan 2016 01:30:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: crash, needs-qa, vimage X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@otoh.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 01:30:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D201694 --- Comment #6 from Paul Armstrong --- Just adding ALTQ to a 10-STABLE system causes crash on jail start. 10-STABLE OVA (appliance export as OVA2 from Virtual Box 5.0.10): https://s3.amazonaws.com/local-ami-us-east-1/freebsd/FreeBSDJailTest.ova jail config is /var/tmp/jail.conf (to avoid crash cycles). 10 STABLE sources are in /usr/src/10, kernel in /usr/src/10/sys/amd64/conf/JAILTEST login: jailtest password: jailtest jailtest is in wheel To cause crash: jail -f /var/tmp/jail.conf -c jail1 If ALTQ is removed from the kernel, it seems to do fine, as soon as it's th= ere, there's multiple ways to crash. I can also confirm that 10.2-p7 (on bare metal) no longer crashes once ALTQ= is removed. Am building an 11-CURRENT VM to see if I can replicate it there or not. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Jan 18 01:49:21 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26946A86071 for ; Mon, 18 Jan 2016 01:49:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 172591CB6 for ; Mon, 18 Jan 2016 01:49:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0I1nKKL000887 for ; Mon, 18 Jan 2016 01:49:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 201694] 10.2-BETA2 crashing when killing VIMAGE/VNET jails Date: Mon, 18 Jan 2016 01:49:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: crash, needs-qa, vimage X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 01:49:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D201694 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Version|10.2-RELEASE |10.2-STABLE --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Jan 18 04:48:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBF51A86A55 for ; Mon, 18 Jan 2016 04:48:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A0DAD1BFC for ; Mon, 18 Jan 2016 04:48:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 9D765A86A52; Mon, 18 Jan 2016 04:48:32 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8428CA86A50; Mon, 18 Jan 2016 04:48:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 283401BFA; Mon, 18 Jan 2016 04:48:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0I4mQTo015893 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 18 Jan 2016 06:48:27 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0I4mQTo015893 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0I4mQTg015892; Mon, 18 Jan 2016 06:48:26 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 18 Jan 2016 06:48:26 +0200 From: Konstantin Belousov To: Jilles Tjoelker Cc: Boris Astardzhiev , net@freebsd.org, threads@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160118044826.GS3942@kib.kiev.ua> References: <20160108075815.3243.qmail@f5-external.bushwire.net> <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160117211853.GA37847@stack.nl> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 04:48:32 -0000 Added threads@ where this discussion is more appropriate. On Sun, Jan 17, 2016 at 10:18:53PM +0100, Jilles Tjoelker wrote: > This will typically work (if the cancellation occurs while blocked > inside __sys_recvmsg()) but has the usual problem of relying on [EINTR]: > lost wakeups. This is certainly less bad than using the interposable > recvmsg(), though, which would discard the already received data. > > As a slight modification, the first recvmsg could use the interposable > version, since there is no pending data at that point. This avoids > needing to call pthread_testcancel() manually. > > The regular cancellation code closes this race window using the > undocumented thr_wake() system call, on the thread itself, in the signal > handler for the cancellation signal. This causes the next attempt to > sleep(9) to fail with [EINTR]. (On another note, it appears to be > possible for user code (cleanup handlers and pthread_key_create() > destructors) to be called with thr_wake() still active, if the > cancellation signal handler is called immediately after the cancellation > point system call returns.) > > The race in recvmmsg could be removed using this mechanism but it > requires either a separate version of recvmmsg in libthr or a new > interface in libthr. I imagine the new interface as a new cancellation > type which causes cancellation point functions such as recvmsg() to fail > with a new errno when cancelled while leaving cancellation pending. This > seems conceptually possible but adds some code to the common path for > cancellation points. A new version of pthread_testcancel() with a return > value would be needed. Yes, I should have remembered about TDP_WAKEUP. In fact, this discussion and recvmmsg() skeleton structured my understanding of the better split between libc and libthr. I start thinking that libthr should not interpose most of the syscalls. Instead, libthr should interpose (wrappers around) thr_cancel_enter*() and thr_cancel_leave(), the body of the cancellable syscalls should live in libc. The benefits are removal of double implementation of cancellable syscalls, more compact interface between libc and libthr, and potentially we can grow an _np extension which would allow user libraries to correctly implement composable cancellable functions (as needed for sendmmsg). From owner-freebsd-net@freebsd.org Mon Jan 18 10:37:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A536CA86EED for ; Mon, 18 Jan 2016 10:37:03 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 82A661FAB for ; Mon, 18 Jan 2016 10:37:03 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 81D9BA86EE9; Mon, 18 Jan 2016 10:37:03 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67613A86EE8; Mon, 18 Jan 2016 10:37:03 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC4CC1FAA; Mon, 18 Jan 2016 10:37:02 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id b14so115681986wmb.1; Mon, 18 Jan 2016 02:37:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xpxqdNf9r5dQJyHyblaHo48LH//gjFPotsBieLUcaQ4=; b=PVhFvDxwUq0W2TcGKoheMWZerfTVvhAsGiNgkxy5rSwNufzQ34lkyKYrpsI+9QTzQ7 prEGX/2IHl5Or0xPRIFeSYM8qMb9dC8VQTp1ZWrmWqsxBUFO67cSbZQPNnLs3GsqUjdS TunyNKuNboYuVS4OlaGgfMCnybsWyJt+j57XDN1Efz+dmqQFYKGBPc6ZkKMVuRDFozqr Tn5njjroQ/CdmejtmL3p5EGVH9HzkJ9Q5OrROt6LhUt+3xaavL7qyayV3Fmbb9lV5axq BlOBtxHmk1Y9OdU7sVd80mzxKsVJHaMuMmequpZ/HV7TyhdwgaAOadvFcgrrYddUTQla +2oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=xpxqdNf9r5dQJyHyblaHo48LH//gjFPotsBieLUcaQ4=; b=dM9TAB6yeJwpc8Y0QQOkUbDo5oBDR2/YIEVnMT0ZB0LyNkwyW/xhquETHeY6AkciXN 7mjVuxFktbLV/6SV/9TuV+uYAs4Mm9Sp9XbDQzg5tFV82yUDqyQ17GkLFgHzFwS6Mas1 ooANXP/u4FMEbqD1GxuqhLPGwPVpvQtX1JhbzPXF1AJeWe40gS0U7PYtDd8TOSYQbhes yXm/UOny+kHGKA+u9Knq+EicDBUdlBG3aH/fjpfv3f31xptBrtR6C0dbz+1ckkkOI2Vz NK/m/NMEQGn453zESv2XIbkoLab09dzoaTswQinIKfJBEyQgCVT2NyLvzSpPuBuQm2+x FjdA== X-Gm-Message-State: AG10YOTtId8OPWJtB4C9UbNIL2PajQ14stC1OinmPIX3crYxfv4Q71z2ZdCmIgIs6U6C9UelIFofPfhAAi6M8g== MIME-Version: 1.0 X-Received: by 10.28.1.23 with SMTP id 23mr11759270wmb.37.1453113421506; Mon, 18 Jan 2016 02:37:01 -0800 (PST) Received: by 10.28.136.148 with HTTP; Mon, 18 Jan 2016 02:37:01 -0800 (PST) In-Reply-To: <20160118044826.GS3942@kib.kiev.ua> References: <20160108075815.3243.qmail@f5-external.bushwire.net> <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> Date: Mon, 18 Jan 2016 12:37:01 +0200 Message-ID: Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? From: Boris Astardzhiev To: Konstantin Belousov Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 10:37:03 -0000 Hello, Sorry for the delay of my reply. As far as I understand pthread_testcancel() is not necessary in the recvmmsg syscall since cancellation is not quite common among apps. But if there is cancellation attempts as long as I use __sys_recvmsg() instead of the interposing approach on a cancel attempt recvmmsg() will return EINTR which will get me out? Secondly, I guess it's better to use __sys_sendmmsg() similarly instead of the insterposing table regarding sendmmsg(). Lastly, regarding the manpage - should I extend send/recv(2) for the new calls or create new manpage files? Best regards, Boris Astardzhiev On Mon, Jan 18, 2016 at 6:48 AM, Konstantin Belousov wrote: > Added threads@ where this discussion is more appropriate. > > On Sun, Jan 17, 2016 at 10:18:53PM +0100, Jilles Tjoelker wrote: > > This will typically work (if the cancellation occurs while blocked > > inside __sys_recvmsg()) but has the usual problem of relying on [EINTR]: > > lost wakeups. This is certainly less bad than using the interposable > > recvmsg(), though, which would discard the already received data. > > > > As a slight modification, the first recvmsg could use the interposable > > version, since there is no pending data at that point. This avoids > > needing to call pthread_testcancel() manually. > > > > The regular cancellation code closes this race window using the > > undocumented thr_wake() system call, on the thread itself, in the signal > > handler for the cancellation signal. This causes the next attempt to > > sleep(9) to fail with [EINTR]. (On another note, it appears to be > > possible for user code (cleanup handlers and pthread_key_create() > > destructors) to be called with thr_wake() still active, if the > > cancellation signal handler is called immediately after the cancellation > > point system call returns.) > > > > The race in recvmmsg could be removed using this mechanism but it > > requires either a separate version of recvmmsg in libthr or a new > > interface in libthr. I imagine the new interface as a new cancellation > > type which causes cancellation point functions such as recvmsg() to fail > > with a new errno when cancelled while leaving cancellation pending. This > > seems conceptually possible but adds some code to the common path for > > cancellation points. A new version of pthread_testcancel() with a return > > value would be needed. > > Yes, I should have remembered about TDP_WAKEUP. > > In fact, this discussion and recvmmsg() skeleton structured my > understanding of the better split between libc and libthr. I start > thinking that libthr should not interpose most of the syscalls. Instead, > libthr should interpose (wrappers around) thr_cancel_enter*() and > thr_cancel_leave(), the body of the cancellable syscalls should live in > libc. > > The benefits are removal of double implementation of cancellable syscalls, > more compact interface between libc and libthr, and potentially we can > grow an _np extension which would allow user libraries to correctly > implement composable cancellable functions (as needed for sendmmsg). > From owner-freebsd-net@freebsd.org Mon Jan 18 14:08:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F8C1A878FB for ; Mon, 18 Jan 2016 14:08:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 469C5176E for ; Mon, 18 Jan 2016 14:08:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 413D5A878F8; Mon, 18 Jan 2016 14:08:17 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40946A878F6; Mon, 18 Jan 2016 14:08:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2198176C; Mon, 18 Jan 2016 14:08:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0IE8CrR054635 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 18 Jan 2016 16:08:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0IE8CrR054635 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0IE8Boc054634; Mon, 18 Jan 2016 16:08:11 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 18 Jan 2016 16:08:11 +0200 From: Konstantin Belousov To: Boris Astardzhiev Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160118140811.GW3942@kib.kiev.ua> References: <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 14:08:17 -0000 On Mon, Jan 18, 2016 at 12:37:01PM +0200, Boris Astardzhiev wrote: > Hello, > > Sorry for the delay of my reply. As far as I understand pthread_testcancel() > is not necessary in the recvmmsg syscall since cancellation is not quite > common > among apps. But if there is cancellation attempts as long as I use > __sys_recvmsg() instead > of the interposing approach on a cancel attempt recvmmsg() will return > EINTR which will get > me out? Yes. The corner case is the cancellation attempt (SIGCANCEL == SIGTHR) coming while the thread is executing code around the syscall. > > Secondly, I guess it's better to use __sys_sendmmsg() similarly instead of > the > insterposing table regarding sendmmsg(). Sure, sendmmsg and recvmmsg are same. > > Lastly, regarding the manpage - should I extend send/recv(2) for the new > calls or > create new manpage files? IMO it is more logical to extend the existing page than write a new one. From owner-freebsd-net@freebsd.org Mon Jan 18 16:20:04 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A3ABA86D5B for ; Mon, 18 Jan 2016 16:20:04 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 243701A27 for ; Mon, 18 Jan 2016 16:20:03 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 050F567ACF for ; Mon, 18 Jan 2016 17:13:36 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id 9sGqKNdlG-Ya for ; Mon, 18 Jan 2016 17:13:35 +0100 (CET) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 59E6667ACE for ; Mon, 18 Jan 2016 17:13:35 +0100 (CET) Received: from bsdlo.incore (bsdlo.incore [192.168.0.84]) by mail.local.incore (Postfix) with ESMTP id 4691350898 for ; Mon, 18 Jan 2016 17:13:35 +0100 (CET) Message-ID: <569D0F2F.8060508@incore.de> Date: Mon, 18 Jan 2016 17:13:35 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: pf not seeing inbound packets coming from IPSec on epair interface Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 16:20:04 -0000 in the situation IPSec --> epair0a --> epair0b pf does not see inbound packets on the interface epair0b, because the epair driver does not clear the flag PACKET_TAG_IPSEC_IN_DONE when he transfers a packet from epair0a to epair0b. The following patch for FreeBSD 10 works for me and is adapted from lists.freebsd.org/pipermail/freebsd-net/2012-January/031161.html: --- if_epair.c.1st 2015-03-13 12:06:49.000000000 +0100 +++ if_epair.c 2016-01-18 17:07:14.911942000 +0100 @@ -469,6 +469,7 @@ struct ifnet *oifp; int error, len; short mflags; + struct m_tag *mtag; DPRINTF("ifp=%p m=%p\n", ifp, m); sc = ifp->if_softc; @@ -510,6 +511,11 @@ mflags = m->m_flags; DPRINTF("packet %s -> %s\n", ifp->if_xname, oifp->if_xname); + /* Delete an existing ipsec tag */ + mtag = m_tag_find(m, PACKET_TAG_IPSEC_IN_DONE, NULL); + if (mtag != NULL) + m_tag_delete(m, mtag); + #ifdef ALTQ /* Support ALTQ via the clasic if_start() path. */ IF_LOCK(&ifp->if_snd); Maybe some more internel kernel information from a packet should be cleared by the epair driver when he transfers a packet from epair0a ro epair0b. -- Dr. Andreas Longwitz Data Service GmbH Beethovenstr. 2A 23617 Stockelsdorf Amtsgericht Lübeck, HRB 318 BS Geschäftsführer: Wilfried Paepcke, Dr. Andreas Longwitz, Josef Flatau From owner-freebsd-net@freebsd.org Mon Jan 18 16:27:45 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94855A8705F for ; Mon, 18 Jan 2016 16:27:45 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 574471E96 for ; Mon, 18 Jan 2016 16:27:44 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id CAC6325D38A4; Mon, 18 Jan 2016 16:27:35 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 16773C76FF6; Mon, 18 Jan 2016 16:27:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id ij0_nFnB_CLs; Mon, 18 Jan 2016 16:27:33 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 63C9BC76FEC; Mon, 18 Jan 2016 16:27:33 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: pf not seeing inbound packets coming from IPSec on epair interface From: "Bjoern A. Zeeb" In-Reply-To: <569D0F2F.8060508@incore.de> Date: Mon, 18 Jan 2016 16:27:31 +0000 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5ADF2343-7643-41ED-B2AE-8A94A3478B95@lists.zabbadoz.net> References: <569D0F2F.8060508@incore.de> To: Andreas Longwitz X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 16:27:45 -0000 > On 18 Jan 2016, at 16:13 , Andreas Longwitz = wrote: >=20 > in the situation > IPSec --> epair0a --> epair0b > pf does not see inbound packets on the interface epair0b, because the > epair driver does not clear the flag PACKET_TAG_IPSEC_IN_DONE when he > transfers a packet from epair0a to epair0b. The following patch for > FreeBSD 10 works for me and is adapted from > lists.freebsd.org/pipermail/freebsd-net/2012-January/031161.html: Where does epair get the packet from? A physical interface bridged to = epair? If anything should clear that; I guess it=E2=80=99s the bridge = interface? Hmm, but then if you are using epairs to cross between network stacks, = you are changing boundries, indeed, so if you=E2=80=99d run ipsec on a = single epair between two VNETs, that might be interesting as well? I guess we=E2=80=99ll need to find a couple of these places (epair, = bridge, netgraph, =E2=80=A6) and make sure we strip all of the tags IFF = we change the VNET? /bz From owner-freebsd-net@freebsd.org Mon Jan 18 23:03:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66E5CA8778A for ; Mon, 18 Jan 2016 23:03:39 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2FC8013C2 for ; Mon, 18 Jan 2016 23:03:39 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 752BE67C18; Tue, 19 Jan 2016 00:03:36 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id gSgOUIM-PNrh; Tue, 19 Jan 2016 00:02:59 +0100 (CET) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id DEA3067BE9; Tue, 19 Jan 2016 00:02:59 +0100 (CET) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id 5854750895; Tue, 19 Jan 2016 00:02:59 +0100 (CET) Message-ID: <569D6F22.1000405@incore.de> Date: Tue, 19 Jan 2016 00:02:58 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: "Bjoern A. Zeeb" CC: freebsd-net@freebsd.org Subject: Re: pf not seeing inbound packets coming from IPSec on epair interface References: <569D0F2F.8060508@incore.de> <5ADF2343-7643-41ED-B2AE-8A94A3478B95@lists.zabbadoz.net> In-Reply-To: <5ADF2343-7643-41ED-B2AE-8A94A3478B95@lists.zabbadoz.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2016 23:03:39 -0000 Hi, thanks for answer. >> in the situation >> IPSec --> epair0a --> epair0b --> em1 >> pf does not see inbound packets on the interface epair0b, because the >> epair driver does not clear the flag PACKET_TAG_IPSEC_IN_DONE when he >> transfers a packet from epair0a to epair0b. The following patch for >> FreeBSD 10 works for me and is adapted from >> lists.freebsd.org/pipermail/freebsd-net/2012-January/031161.html: > > Where does epair get the packet from? A physical interface bridged to epair? I use epair on a firewall machine FW running to serve VPN's for IPhones (IPSec with XAuth). Every user gets an IP (racoon + freeradius) to use in his tunnel, the tunnel IP of FW is fix. Different user groups must connect to different mail server in my network. FW has two hardware interfaces em0 (internet) and em1 (intranet), no jails, no bridges. I use the rdr command of pf on interface epair0b to redirect the user to the correct mailserver before the packets leaves my FW on interface em1 (with nat and a pass rule using reply-to ( epair0b $ip_epair0a). I am not aware of another method to rewrite the destination address of an IPSec incoming packet on the same machine, therefore the use of epair. > Hmm, but then if you are using epairs to cross between network stacks, you are > changing boundries, indeed, so if you’d run ipsec on a single epair between two > VNETs, that might be interesting as well? I think epair should behave identical to em2 + em3 with a crossover cable, but I do not have enough network interfaces. > I guess we’ll need to find a couple of these places (epair, bridge, netgraph, …) > and make sure we strip all of the tags IFF we change the VNET? I think so. One example is mentioned in lists.freebsd.org/pipermail/freebsd-net/2012-January/031161.html for Clients using a VPN with L2TP over IPSec (racoon + mpd5). Dr. Andreas Longwitz Data Service GmbH Beethovenstr. 2A 23617 Stockelsdorf Amtsgericht Lübeck, HRB 318 BS Geschäftsführer: Wilfried Paepcke, Dr. Andreas Longwitz, Josef Flatau From owner-freebsd-net@freebsd.org Tue Jan 19 07:09:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D3D3A87A87 for ; Tue, 19 Jan 2016 07:09:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5CF19144B for ; Tue, 19 Jan 2016 07:09:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0J79oVR005529 for ; Tue, 19 Jan 2016 07:09:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 201694] 10.2-BETA2 crashing when killing VIMAGE/VNET jails Date: Tue, 19 Jan 2016 07:09:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: crash, needs-qa, vimage X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@otoh.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 07:09:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D201694 --- Comment #7 from Paul Armstrong --- I can also cause a crash in 11-CURRENT 20160113-r293801 I have uploaded another VM to: https://s3.amazonaws.com/local-ami-us-east-1/freebsd/FreeBSDJailTest-curren= t.ova user/pass: jailtest/jailtest To reproduce: jail -f /var/tmp/jail.conf -c jail1 service pf onestart jail -f /var/tmp/jail.conf -rc jail1 The image has some crash dumps in /var/crash and /var/log/messages also has stack traces. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Jan 19 07:12:13 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFC27A87C2A for ; Tue, 19 Jan 2016 07:12:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B03781831 for ; Tue, 19 Jan 2016 07:12:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0J7CDlu014897 for ; Tue, 19 Jan 2016 07:12:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 201694] 10.2-BETA2 crashing when killing VIMAGE/VNET jails Date: Tue, 19 Jan 2016 07:12:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: crash, needs-qa, vimage X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@otoh.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 07:12:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D201694 --- Comment #8 from Paul Armstrong --- Note that 11-CURRENT is slightly better in that it requires pf unloaded bef= ore jail start and then restart after loading it (as opposed to crashing on jail restart if pf is loaded at all). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Jan 19 09:04:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53CBDA87D60 for ; Tue, 19 Jan 2016 09:04:02 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 10F261EAD for ; Tue, 19 Jan 2016 09:04:01 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 14D122001CF for ; Tue, 19 Jan 2016 10:03:53 +0100 (CET) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 49D12405882; Tue, 19 Jan 2016 10:03:51 +0100 (CET) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id ADF0B405881; Tue, 19 Jan 2016 10:03:50 +0100 (CET) Received: from arc.aei.uni-hannover.de ([10.117.15.110]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016011910034232-108105 ; Tue, 19 Jan 2016 10:03:42 +0100 Date: Tue, 19 Jan 2016 10:03:42 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: freebsd-net@freebsd.org Cc: "carsten.aulbert@aei.mpg.de" Subject: ix hangs (almost) Message-Id: <20160119100342.7c458ee5406ba10362237cd0@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 19/01/2016 10:03:42, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 19/01/2016 10:03:52, Serialize complete at 19/01/2016 10:03:52 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.1.19.85116 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1800_1899 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, ECARD_WORD 0, NO_URI_HTTPS 0, __ANY_URI 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __TO_MALFORMED_2 0, __TO_NO_NAME 0, __URI_NO_PATH 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 09:04:02 -0000 Hi all, Half a year ago we had reported here about issues with ix-interfaces when transporting data over nfs. We only got about 50MByte/s back then, and were not able to fix this issue. Instead, we went for rsyncd transport that gave us about 200MByte/s (still not the full bandwidth, but definitely better than nfs). Now we recognize strange "hangs" in these sessions: We do nightly backups on this link, and it works fine for days, sometimes even weeks. Then suddenly the whole network traffic slows down almost to a full halt. Write speeds (both rsyncd and nfs) go down to a few kByte/s. I see no error messages, no bad packets, nothing that would indicate a reason for this. The issue goes away (for some time) when doing ifconfig ix0 down/up, but comes back after some days. Any suggestions on how to debug this (when it happens next) are highly appreciated. If it looks like this is interface/driver-related, I'll also take suggestions on what different 10GE (copper) card to try instead. FreeBSD crest 10.2-RELEASE-p7 FreeBSD 10.2-RELEASE-p7 #0: Mon Nov 2 14:19:39 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 ix0@pci0:5:0:0: class=0x020000 card=0x00028086 chip=0x15288086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller 10-Gigabit X540-AT2' class = network subclass = ethernet ix0: flags=8843 metric 0 mtu 1500 options=8404bb ether a0:36:9f:23:3d:b6 inet 192.168.6.2 netmask 0xffffff00 broadcast 192.168.6.255 nd6 options=29 media: Ethernet autoselect (10Gbase-T ) status: active cu Gerrit From owner-freebsd-net@freebsd.org Tue Jan 19 11:58:30 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E26E4A87471 for ; Tue, 19 Jan 2016 11:58:30 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BE3711634 for ; Tue, 19 Jan 2016 11:58:30 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id BB274A8746E; Tue, 19 Jan 2016 11:58:30 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F9F5A8746B; Tue, 19 Jan 2016 11:58:30 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 232771633; Tue, 19 Jan 2016 11:58:30 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id 123so88281065wmz.0; Tue, 19 Jan 2016 03:58:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cjJvvthu0PsGR0vHaheXs36H0B2fWNcn0AiSP3C/TCc=; b=DjJmWbIUkELuUzOk0e1exGOj9LXCKZmfFOcqSx8abE0sibrXej/Y14e8gOkhdxYAqY boI7/tcO/3s2XAU6ubask5Aym5z5J7zkSplutn6k9lZbI/et/1nq6oiv/MkZNFhPriaI HJCcBzKoCWfDTgh/cTuwXsroPTkGD0DD5Pyd4CkqycPjOrgMdEMFMb21u3fMGuVK/1Ws Y5jZWFBITXQedzOwR1EDvuzp3ioWMBEoNvV8o3/eEVNO+EZztGPwMCYfKI/UqdFQfNd/ 7vvQBvXrd1D8s/q7J2fXdmiMVGzO/HrIQiAQr2R00vCq34d5bNw4/QKFYSWMPdj42DtP p5gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=cjJvvthu0PsGR0vHaheXs36H0B2fWNcn0AiSP3C/TCc=; b=EH6xrZsMATTJ8HxPtAvBWNbxMX+ZwPBfhybT6VZ3bAkD9SZvw9cgbk6D1f3lZmCVF9 dpPxTtSftGxdNzW9m0reQjSnR4yzE0ehjMrmtrePJyN5RKh9VkGdUgyKXFZG7JzbC8SA +1S3H1N74ec/GMx4IniCxf6qMZ/clb1JBzh5p9nbvNLIRKt0WR19LM+lWq4J2DcluPdH f/GeieW0xR8VABjqI022wHamTFT3+y7Z7g0ekEySre3bGRb9MA0psZJPzvhxTlZvcHt5 xT5OsiyT/Hx5CtT8fjlUP5eR2cdilMOoBXTz0XNmcFhTW4NaPM/UY2fb1z5RZDyHGm7M y0JQ== X-Gm-Message-State: AG10YORzGC1ywMcduBdWWy2JxY/xyaaZj7zzOgXpIEeUSZUBXGoLhV0ZE5Es7Q3EjeJlR79+BxY0IsraniL3tg== MIME-Version: 1.0 X-Received: by 10.28.63.22 with SMTP id m22mr17456074wma.59.1453204708288; Tue, 19 Jan 2016 03:58:28 -0800 (PST) Received: by 10.28.136.148 with HTTP; Tue, 19 Jan 2016 03:58:27 -0800 (PST) In-Reply-To: <20160118140811.GW3942@kib.kiev.ua> References: <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> Date: Tue, 19 Jan 2016 13:58:27 +0200 Message-ID: Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? From: Boris Astardzhiev To: Konstantin Belousov Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Content-Type: multipart/mixed; boundary=001a114b3ebc6c912e0529ae96fb X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 11:58:31 -0000 --001a114b3ebc6c912e0529ae96fb Content-Type: text/plain; charset=UTF-8 Hello, I removed the pthread_testcancel() calls and cut the interposing stuff from my patch as suggested. I extended the send/recv(2) manpages regarding the mm calls. Comments and suggestions? Best regards, Boris Astardzhiev On Mon, Jan 18, 2016 at 4:08 PM, Konstantin Belousov wrote: > On Mon, Jan 18, 2016 at 12:37:01PM +0200, Boris Astardzhiev wrote: > > Hello, > > > > Sorry for the delay of my reply. As far as I understand > pthread_testcancel() > > is not necessary in the recvmmsg syscall since cancellation is not quite > > common > > among apps. But if there is cancellation attempts as long as I use > > __sys_recvmsg() instead > > of the interposing approach on a cancel attempt recvmmsg() will return > > EINTR which will get > > me out? > Yes. > > The corner case is the cancellation attempt (SIGCANCEL == SIGTHR) coming > while the thread is executing code around the syscall. > > > > > Secondly, I guess it's better to use __sys_sendmmsg() similarly instead > of > > the > > insterposing table regarding sendmmsg(). > Sure, sendmmsg and recvmmsg are same. > > > > > Lastly, regarding the manpage - should I extend send/recv(2) for the new > > calls or > > create new manpage files? > IMO it is more logical to extend the existing page than write a new one. > --001a114b3ebc6c912e0529ae96fb Content-Type: text/plain; charset=US-ASCII; name="sendrecvmmsg-libconly3.diff" Content-Disposition: attachment; filename="sendrecvmmsg-libconly3.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ijlch3460 ZGlmZiAtLWdpdCBhL2xpYi9saWJjL2luY2x1ZGUvbmFtZXNwYWNlLmggYi9saWIvbGliYy9pbmNs dWRlL25hbWVzcGFjZS5oCmluZGV4IDczOWQ3YjEuLmM5NTgyOWUgMTAwNjQ0Ci0tLSBhL2xpYi9s aWJjL2luY2x1ZGUvbmFtZXNwYWNlLmgKKysrIGIvbGliL2xpYmMvaW5jbHVkZS9uYW1lc3BhY2Uu aApAQCAtMjA4LDYgKzIwOCw3IEBACiAjZGVmaW5lCQlyZWFkdgkJCQlfcmVhZHYKICNkZWZpbmUJ CXJlY3Zmcm9tCQkJX3JlY3Zmcm9tCiAjZGVmaW5lCQlyZWN2bXNnCQkJCV9yZWN2bXNnCisjZGVm aW5lCQlyZWN2bW1zZwkJCV9yZWN2bW1zZwogI2RlZmluZQkJc2VsZWN0CQkJCV9zZWxlY3QKICNk ZWZpbmUJCXNlbV9jbG9zZQkJCV9zZW1fY2xvc2UKICNkZWZpbmUJCXNlbV9kZXN0cm95CQkJX3Nl bV9kZXN0cm95CkBAIC0yMjAsNiArMjIxLDcgQEAKICNkZWZpbmUJCXNlbV91bmxpbmsJCQlfc2Vt X3VubGluawogI2RlZmluZQkJc2VtX3dhaXQJCQlfc2VtX3dhaXQKICNkZWZpbmUJCXNlbmRtc2cJ CQkJX3NlbmRtc2cKKyNkZWZpbmUJCXNlbmRtbXNnCQkJX3NlbmRtbXNnCiAjZGVmaW5lCQlzZW5k dG8JCQkJX3NlbmR0bwogI2RlZmluZQkJc2V0c29ja29wdAkJCV9zZXRzb2Nrb3B0CiAvKiNkZWZp bmUJCXNpZ2FjdGlvbgkJCV9zaWdhY3Rpb24qLwpkaWZmIC0tZ2l0IGEvbGliL2xpYmMvaW5jbHVk ZS91bi1uYW1lc3BhY2UuaCBiL2xpYi9saWJjL2luY2x1ZGUvdW4tbmFtZXNwYWNlLmgKaW5kZXgg ZjMxZmE3YS4uMDIzMzM0OCAxMDA2NDQKLS0tIGEvbGliL2xpYmMvaW5jbHVkZS91bi1uYW1lc3Bh Y2UuaAorKysgYi9saWIvbGliYy9pbmNsdWRlL3VuLW5hbWVzcGFjZS5oCkBAIC0xODksNiArMTg5 LDcgQEAKICN1bmRlZgkJcmVhZHYKICN1bmRlZgkJcmVjdmZyb20KICN1bmRlZgkJcmVjdm1zZwor I3VuZGVmCQlyZWN2bW1zZwogI3VuZGVmCQlzZWxlY3QKICN1bmRlZgkJc2VtX2Nsb3NlCiAjdW5k ZWYJCXNlbV9kZXN0cm95CkBAIC0yMDEsNiArMjAyLDcgQEAKICN1bmRlZgkJc2VtX3VubGluawog I3VuZGVmCQlzZW1fd2FpdAogI3VuZGVmCQlzZW5kbXNnCisjdW5kZWYJCXNlbmRtbXNnCiAjdW5k ZWYJCXNlbmR0bwogI3VuZGVmCQlzZXRzb2Nrb3B0CiAjdW5kZWYJCXNpZ2FjdGlvbgpkaWZmIC0t Z2l0IGEvbGliL2xpYmMvc3lzL01ha2VmaWxlLmluYyBiL2xpYi9saWJjL3N5cy9NYWtlZmlsZS5p bmMKaW5kZXggZTRmZTFiMi4uNWY4YjY5OSAxMDA2NDQKLS0tIGEvbGliL2xpYmMvc3lzL01ha2Vm aWxlLmluYworKysgYi9saWIvbGliYy9zeXMvTWFrZWZpbGUuaW5jCkBAIC0yOCw2ICsyOCw4IEBA IFNSQ1MrPSBmdXRpbWVucy5jIHV0aW1lbnNhdC5jCiBOT0FTTSs9IGZ1dGltZW5zLm8gdXRpbWVu c2F0Lm8KIFBTRVVETys9IF9mdXRpbWVucy5vIF91dGltZW5zYXQubwogCitTUkNTKz0gcmVjdm1t c2cuYyBzZW5kbW1zZy5jCisKIElOVEVSUE9TRUQgPSBcCiAJYWNjZXB0IFwKIAlhY2NlcHQ0IFwK ZGlmZiAtLWdpdCBhL2xpYi9saWJjL3N5cy9TeW1ib2wubWFwIGIvbGliL2xpYmMvc3lzL1N5bWJv bC5tYXAKaW5kZXggN2IzMjU3Yy4uNzI0ZTFiNCAxMDA2NDQKLS0tIGEvbGliL2xpYmMvc3lzL1N5 bWJvbC5tYXAKKysrIGIvbGliL2xpYmMvc3lzL1N5bWJvbC5tYXAKQEAgLTM5OSw2ICszOTksOCBA QCBGQlNEXzEuNCB7CiAJdXRpbWVuc2F0OwogCW51bWFfc2V0YWZmaW5pdHk7CiAJbnVtYV9nZXRh ZmZpbml0eTsKKwlzZW5kbW1zZzsKKwlyZWN2bW1zZzsKIH07CiAKIEZCU0Rwcml2YXRlXzEuMCB7 CkBAIC0xMDUxLDQgKzEwNTMsNiBAQCBGQlNEcHJpdmF0ZV8xLjAgewogCWdzc2Rfc3lzY2FsbDsK IAlfX2xpYmNfaW50ZXJwb3Npbmdfc2xvdDsKIAlfX2xpYmNfc2lnd2FpdDsKKwlfc2VuZG1tc2c7 CisJX3JlY3ZtbXNnOwogfTsKZGlmZiAtLWdpdCBhL2xpYi9saWJjL3N5cy9yZWN2LjIgYi9saWIv bGliYy9zeXMvcmVjdi4yCmluZGV4IDMyNmU3ZmYuLjgxYTAyMDEgMTAwNjQ0Ci0tLSBhL2xpYi9s aWJjL3N5cy9yZWN2LjIKKysrIGIvbGliL2xpYmMvc3lzL3JlY3YuMgpAQCAtMzQsOCArMzQsOSBA QAogLlNoIE5BTUUKIC5ObSByZWN2ICwKIC5ObSByZWN2ZnJvbSAsCi0uTm0gcmVjdm1zZwotLk5k IHJlY2VpdmUgYSBtZXNzYWdlIGZyb20gYSBzb2NrZXQKKy5ObSByZWN2bXNnICwKKy5ObSByZWN2 bW1zZworLk5kIHJlY2VpdmUgbWVzc2FnZShzKSBmcm9tIGEgc29ja2V0CiAuU2ggTElCUkFSWQog LkxiIGxpYmMKIC5TaCBTWU5PUFNJUwpAQCAtNDcsMTEgKzQ4LDE1IEBACiAuRm4gcmVjdmZyb20g ImludCBzIiAidm9pZCAqYnVmIiAic2l6ZV90IGxlbiIgImludCBmbGFncyIgInN0cnVjdCBzb2Nr YWRkciAqIHJlc3RyaWN0IGZyb20iICJzb2NrbGVuX3QgKiByZXN0cmljdCBmcm9tbGVuIgogLkZ0 IHNzaXplX3QKIC5GbiByZWN2bXNnICJpbnQgcyIgInN0cnVjdCBtc2doZHIgKm1zZyIgImludCBm bGFncyIKKy5GdCBpbnQKKy5GbiByZWN2bW1zZyAiaW50IHMiICJzdHJ1Y3QgbW1zZ2hkciAqbXNn dmVjIiAiaW50IHZsZW4iICJpbnQgZmxhZ3MiCiAuU2ggREVTQ1JJUFRJT04KIFRoZQogLkZuIHJl Y3Zmcm9tCiBhbmQKIC5GbiByZWN2bXNnCithbmQKKy5GbiByZWN2bW1zZwogc3lzdGVtIGNhbGxz CiBhcmUgdXNlZCB0byByZWNlaXZlIG1lc3NhZ2VzIGZyb20gYSBzb2NrZXQsCiBhbmQgbWF5IGJl IHVzZWQgdG8gcmVjZWl2ZSBkYXRhIG9uIGEgc29ja2V0IHdoZXRoZXIgb3Igbm90CkBAIC04NCw4 ICs4OSwyNSBAQCBudWxsIHBvaW50ZXIgcGFzc2VkIGFzIGl0cwogLkZhIGZyb20KIGFyZ3VtZW50 LgogLlBwCi1BbGwgdGhyZWUgcm91dGluZXMgcmV0dXJuIHRoZSBsZW5ndGggb2YgdGhlIG1lc3Nh Z2Ugb24gc3VjY2Vzc2Z1bAotY29tcGxldGlvbi4KK1RoZQorLkZuIHJlY3ZtbXNnCitmdW5jdGlv biBpcyB1c2VkIHRvIHJlY2VpdmUgbXVsdGlwbGUKK21lc3NhZ2VzIGF0IGEgY2FsbC4KK1RoZWly IG51bWJlcgoraXMgc3VwcGxpZWQgYnkKKy5GYSB2bGVuCisobGltaXRlZCB0byAxMDI0KS4KK1Ro ZSBtZXNzYWdlcyBhcmUgcGxhY2VkIGluIHRoZQorLkZhIG1zZ3ZlYwordmVjdG9yIGFmdGVyIHJl Y2VwdGlvbi4KK1RoZSBzaXplIG9mIGVhY2ggcmVjZWl2ZWQgbWVzc2FnZSBpcyBwbGFjZWQgaW4g dGhlCisuRmEgbXNnX2xlbgorZmllbGQgb2YgZWFjaCBlbGVtZW50IG9mIHRoZSB2ZWN0b3IuCisu UHAKK1RoZSBmaXJzdCB0aHJlZSByb3V0aW5lcyByZXR1cm4gdGhlIGxlbmd0aCBvZiB0aGUgbWVz c2FnZSBvbiBzdWNjZXNzZnVsCitjb21wbGV0aW9uIHdoZXJlYXMKKy5GbiByZWN2bW1zZworcmV0 dXJucyB0aGUgbnVtYmVyIG9mIHJlY2VpdmVkIG1lc3NhZ2VzLgogSWYgYSBtZXNzYWdlIGlzIHRv byBsb25nIHRvIGZpdCBpbiB0aGUgc3VwcGxpZWQgYnVmZmVyLAogZXhjZXNzIGJ5dGVzIG1heSBi ZSBkaXNjYXJkZWQgZGVwZW5kaW5nIG9uIHRoZSB0eXBlIG9mIHNvY2tldAogdGhlIG1lc3NhZ2Ug aXMgcmVjZWl2ZWQgZnJvbSAoc2VlCkBAIC0xMDAsNyArMTIyLDkgQEAgaW4gd2hpY2ggY2FzZSB0 aGUgdmFsdWUKIC5WYSBlcnJubwogaXMgc2V0IHRvCiAuRXIgRUFHQUlOIC4KLVRoZSByZWNlaXZl IGNhbGxzIG5vcm1hbGx5IHJldHVybiBhbnkgZGF0YSBhdmFpbGFibGUsCitUaGUgcmVjZWl2ZSBj YWxscyBleGNlcHQKKy5GbiByZWN2bW1zZworbm9ybWFsbHkgcmV0dXJuIGFueSBkYXRhIGF2YWls YWJsZSwKIHVwIHRvIHRoZSByZXF1ZXN0ZWQgYW1vdW50LAogcmF0aGVyIHRoYW4gd2FpdGluZyBm b3IgcmVjZWlwdCBvZiB0aGUgZnVsbCBhbW91bnQgcmVxdWVzdGVkOwogdGhpcyBiZWhhdmlvciBp cyBhZmZlY3RlZCBieSB0aGUgc29ja2V0LWxldmVsIG9wdGlvbnMKQEAgLTI5MCw5ICszMTQsMzQg QEAgY29udHJvbCBkYXRhIHdlcmUgZGlzY2FyZGVkIGR1ZSB0byBsYWNrIG9mIHNwYWNlIGluIHRo ZSBidWZmZXIKIGZvciBhbmNpbGxhcnkgZGF0YS4KIC5EdiBNU0dfT09CCiBpcyByZXR1cm5lZCB0 byBpbmRpY2F0ZSB0aGF0IGV4cGVkaXRlZCBvciBvdXQtb2YtYmFuZCBkYXRhIHdlcmUgcmVjZWl2 ZWQuCisuUHAKK1RoZQorLkZuIHJlY3ZtbXNnCitzeXN0ZW0gY2FsbCB1c2VzIHRoZQorLkZhIG1t c2doZHIKK3N0cnVjdHVyZS4gSXRzIGZvcm0gaXMgYXMgZm9sbG93cywgYXMgZGVmaW5lZCBpbgor LkluIHN5cy9zb2NrZXQuaCA6CisuQmQgLWxpdGVyYWwKK3N0cnVjdCBtbXNnaGRyIHsKKwlzdHJ1 Y3QgbXNnaGRyCSBtc2dfaGRyOwkvKiBtZXNzYWdlIGhlYWRlciAqLworCXVuc2lnbmVkIGludAkg bXNnX2xlbjsJLyogbWVzc2FnZSBsZW5ndGggKi8KK307CisuRWQKKy5QcAorRm9yCisuRmEgbXNn X2hkcgorc2VlIGFib3ZlLiBPbiBkYXRhIHJlY2VwdGlvbiB0aGUKKy5GYSBtc2dfbGVuCitmaWVs ZCBpcyB1cGRhdGVkIHRvIHRoZSBsZW5ndGggb2YgdGhlIHJlY2VpdmVkIG1lc3NhZ2UuIE9uCitk YXRhIHRyYW5zbWlzc2lvbiBpdCBpcyB1cGRhdGVkIHRvIHRoZSBudW1iZXIgb2YKK2NoYXJhY3Rl cnMgc2VudC4KIC5TaCBSRVRVUk4gVkFMVUVTCi1UaGVzZSBjYWxscyByZXR1cm4gdGhlIG51bWJl ciBvZiBieXRlcyByZWNlaXZlZCwgb3IgLTEKLWlmIGFuIGVycm9yIG9jY3VycmVkLgorVGhlc2Ug Y2FsbHMgZXhjZXB0CisuRm4gcmVjdm1tc2cKK3JldHVybiB0aGUgbnVtYmVyIG9mIGJ5dGVzIHJl Y2VpdmVkLgorLkZuIHJlY3ZtbXNnCityZXR1cm5zIHRoZSBudW1iZXIgb2YgbWVzc2FnZXMgcmVj ZWl2ZWQuCitBIHZhbHVlIG9mIC0xIGlzIHJldHVybmVkIGlmIGFuIGVycm9yIG9jY3VycmVkLgog LlNoIEVSUk9SUwogVGhlIGNhbGxzIGZhaWwgaWY6CiAuQmwgLXRhZyAtd2lkdGggRXIKZGlmZiAt LWdpdCBhL2xpYi9saWJjL3N5cy9yZWN2bW1zZy5jIGIvbGliL2xpYmMvc3lzL3JlY3ZtbXNnLmMK bmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMC4uNmI1MTU4YQotLS0gL2Rldi9udWxs CisrKyBiL2xpYi9saWJjL3N5cy9yZWN2bW1zZy5jCkBAIC0wLDAgKzEsNzIgQEAKKy8qCisgKiBD b3B5cmlnaHQgKGMpIDIwMTYgQm9yaXMgQXN0YXJkemhpZXYsIFNtYXJ0Y29tLUJ1bGdhcmlhIEFE CisgKiBBbGwgcmlnaHRzIHJlc2VydmVkLgorICoKKyAqIFJlZGlzdHJpYnV0aW9uIGFuZCB1c2Ug aW4gc291cmNlIGFuZCBiaW5hcnkgZm9ybXMsIHdpdGggb3Igd2l0aG91dAorICogbW9kaWZpY2F0 aW9uLCBhcmUgcGVybWl0dGVkIHByb3ZpZGVkIHRoYXQgdGhlIGZvbGxvd2luZyBjb25kaXRpb25z CisgKiBhcmUgbWV0OgorICogMS4gUmVkaXN0cmlidXRpb25zIG9mIHNvdXJjZSBjb2RlIG11c3Qg cmV0YWluIHRoZSBhYm92ZSBjb3B5cmlnaHQKKyAqICAgIG5vdGljZShzKSwgdGhpcyBsaXN0IG9m IGNvbmRpdGlvbnMgYW5kIHRoZSBmb2xsb3dpbmcgZGlzY2xhaW1lciBhcworICogICAgdGhlIGZp cnN0IGxpbmVzIG9mIHRoaXMgZmlsZSB1bm1vZGlmaWVkIG90aGVyIHRoYW4gdGhlIHBvc3NpYmxl CisgKiAgICBhZGRpdGlvbiBvZiBvbmUgb3IgbW9yZSBjb3B5cmlnaHQgbm90aWNlcy4KKyAqIDIu IFJlZGlzdHJpYnV0aW9ucyBpbiBiaW5hcnkgZm9ybSBtdXN0IHJlcHJvZHVjZSB0aGUgYWJvdmUg Y29weXJpZ2h0CisgKiAgICBub3RpY2UocyksIHRoaXMgbGlzdCBvZiBjb25kaXRpb25zIGFuZCB0 aGUgZm9sbG93aW5nIGRpc2NsYWltZXIgaW4KKyAqICAgIHRoZSBkb2N1bWVudGF0aW9uIGFuZC9v ciBvdGhlciBtYXRlcmlhbHMgcHJvdmlkZWQgd2l0aCB0aGUKKyAqICAgIGRpc3RyaWJ1dGlvbi4K KyAqCisgKiBUSElTIFNPRlRXQVJFIElTIFBST1ZJREVEIEJZIFRIRSBDT1BZUklHSFQgSE9MREVS KFMpIGBgQVMgSVMnJyBBTkQgQU5ZCisgKiBFWFBSRVNTIE9SIElNUExJRUQgV0FSUkFOVElFUywg SU5DTFVESU5HLCBCVVQgTk9UIExJTUlURUQgVE8sIFRIRQorICogSU1QTElFRCBXQVJSQU5USUVT IE9GIE1FUkNIQU5UQUJJTElUWSBBTkQgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSCisgKiBQVVJQ T1NFIEFSRSBESVNDTEFJTUVELiAgSU4gTk8gRVZFTlQgU0hBTEwgVEhFIENPUFlSSUdIVCBIT0xE RVIoUykgQkUKKyAqIExJQUJMRSBGT1IgQU5ZIERJUkVDVCwgSU5ESVJFQ1QsIElOQ0lERU5UQUws IFNQRUNJQUwsIEVYRU1QTEFSWSwgT1IKKyAqIENPTlNFUVVFTlRJQUwgREFNQUdFUyAoSU5DTFVE SU5HLCBCVVQgTk9UIExJTUlURUQgVE8sIFBST0NVUkVNRU5UIE9GCisgKiBTVUJTVElUVVRFIEdP T0RTIE9SIFNFUlZJQ0VTOyBMT1NTIE9GIFVTRSwgREFUQSwgT1IgUFJPRklUUzsgT1IKKyAqIEJV U0lORVNTIElOVEVSUlVQVElPTikgSE9XRVZFUiBDQVVTRUQgQU5EIE9OIEFOWSBUSEVPUlkgT0Yg TElBQklMSVRZLAorICogV0hFVEhFUiBJTiBDT05UUkFDVCwgU1RSSUNUIExJQUJJTElUWSwgT1Ig VE9SVCAoSU5DTFVESU5HIE5FR0xJR0VOQ0UKKyAqIE9SIE9USEVSV0lTRSkgQVJJU0lORyBJTiBB TlkgV0FZIE9VVCBPRiBUSEUgVVNFIE9GIFRISVMgU09GVFdBUkUsCisgKiBFVkVOIElGIEFEVklT RUQgT0YgVEhFIFBPU1NJQklMSVRZIE9GIFNVQ0ggREFNQUdFLgorICovCisKKyNpbmNsdWRlIDxz eXMvY2RlZnMuaD4KK19fRkJTRElEKCIkRnJlZUJTRCQiKTsKKworI2luY2x1ZGUgPGVycm5vLmg+ CisjaW5jbHVkZSA8c3lzL3R5cGVzLmg+CisjaW5jbHVkZSA8c3lzL3N5c2NhbGwuaD4KKyNpbmNs dWRlIDxzeXMvc29ja2V0Lmg+CisjaW5jbHVkZSA8cHRocmVhZC5oPgorI2luY2x1ZGUgImxpYmNf cHJpdmF0ZS5oIgorCisjZGVmaW5lIFZMRU5fTUFYIDEwMjQKKworaW50CityZWN2bW1zZyhpbnQg cywgc3RydWN0IG1tc2doZHIgKm1zZ3ZlYywgdW5zaWduZWQgaW50IHZsZW4sIGludCBmbGFncykK K3sKKwlpbnQgaSwgcmV0LCByY3ZkOworCisJaWYgKHZsZW4gPiBWTEVOX01BWCkKKwkJdmxlbiA9 IFZMRU5fTUFYOworCisJcmN2ZCA9IDA7CisJZm9yIChpID0gMDsgaSA8IHZsZW47IGkrKykgewor CQllcnJubyA9IDA7CisJCXJldCA9IF9fc3lzX3JlY3Ztc2cocywgJm1zZ3ZlY1tpXS5tc2dfaGRy LCBmbGFncyk7CisJCWlmIChyZXQgPCAwIHx8IGVycm5vICE9IDApIHsKKwkJCWlmIChyY3ZkICE9 IDApIHsKKwkJCQkvKiBXZSd2ZSByZWNlaXZlZCBtZXNzYWdlcy4gTGV0IGNhbGxlciBrbm93LiAq LworCQkJCWVycm5vID0gMDsKKwkJCQlyZXR1cm4gKHJjdmQpOworCQkJfQorCQkJcmV0dXJuICgt MSk7CisJCX0KKworCQkvKiBTYXZlIHJlY2VpdmVkIGJ5dGVzICovCisJCW1zZ3ZlY1tpXS5tc2df bGVuID0gcmV0OworCisJCXJjdmQrKzsKKwl9CisKKwlyZXR1cm4gKHJjdmQpOworfQorCisjdW5k ZWYgVkxFTl9NQVgKZGlmZiAtLWdpdCBhL2xpYi9saWJjL3N5cy9zZW5kLjIgYi9saWIvbGliYy9z eXMvc2VuZC4yCmluZGV4IDhmYTJjNjQuLjZjMTRhNjAgMTAwNjQ0Ci0tLSBhL2xpYi9saWJjL3N5 cy9zZW5kLjIKKysrIGIvbGliL2xpYmMvc3lzL3NlbmQuMgpAQCAtMzQsOCArMzQsOSBAQAogLlNo IE5BTUUKIC5ObSBzZW5kICwKIC5ObSBzZW5kdG8gLAotLk5tIHNlbmRtc2cKLS5OZCBzZW5kIGEg bWVzc2FnZSBmcm9tIGEgc29ja2V0CisuTm0gc2VuZG1zZyAsCisuTm0gc2VuZG1tc2cKKy5OZCBz ZW5kIG1lc3NhZ2UocykgZnJvbSBhIHNvY2tldAogLlNoIExJQlJBUlkKIC5MYiBsaWJjCiAuU2gg U1lOT1BTSVMKQEAgLTQ3LDYgKzQ4LDggQEAKIC5GbiBzZW5kdG8gImludCBzIiAiY29uc3Qgdm9p ZCAqbXNnIiAic2l6ZV90IGxlbiIgImludCBmbGFncyIgImNvbnN0IHN0cnVjdCBzb2NrYWRkciAq dG8iICJzb2NrbGVuX3QgdG9sZW4iCiAuRnQgc3NpemVfdAogLkZuIHNlbmRtc2cgImludCBzIiAi Y29uc3Qgc3RydWN0IG1zZ2hkciAqbXNnIiAiaW50IGZsYWdzIgorLkZ0IGludAorLkZuIHNlbmRt bXNnICJpbnQgcyIgInN0cnVjdCBtbXNnaGRyICptc2d2ZWMiICJpbnQgdmxlbiIgImludCBmbGFn cyIKIC5TaCBERVNDUklQVElPTgogVGhlCiAuRm4gc2VuZApAQCAtNTUsOCArNTgsMTAgQEAgYW5k CiAuRm4gc2VuZHRvCiBhbmQKIC5GbiBzZW5kbXNnCithbmQKKy5GbiBzZW5kbW1zZwogc3lzdGVt IGNhbGxzCi1hcmUgdXNlZCB0byB0cmFuc21pdCBhIG1lc3NhZ2UgdG8gYW5vdGhlciBzb2NrZXQu CithcmUgdXNlZCB0byB0cmFuc21pdCBvbmUgb3IgbXVsdGlwbGUgbWVzc2FnZXMgKHdpdGggdGhl IGxhdHRlciBjYWxsKSB0byBhbm90aGVyIHNvY2tldC4KIFRoZQogLkZuIHNlbmQKIGZ1bmN0aW9u CkBAIC02Niw2ICs3MSw4IEBAIHN0YXRlLCB3aGlsZQogLkZuIHNlbmR0bwogYW5kCiAuRm4gc2Vu ZG1zZworYW5kCisuRm4gc2VuZG1tc2cKIG1heSBiZSB1c2VkIGF0IGFueSB0aW1lLgogLlBwCiBU aGUgYWRkcmVzcyBvZiB0aGUgdGFyZ2V0IGlzIGdpdmVuIGJ5CkBAIC04MSw2ICs4OCwxOCBAQCB1 bmRlcmx5aW5nIHByb3RvY29sLCB0aGUgZXJyb3IKIGlzIHJldHVybmVkLCBhbmQKIHRoZSBtZXNz YWdlIGlzIG5vdCB0cmFuc21pdHRlZC4KIC5QcAorVGhlCisuRm4gc2VuZG1tc2cKK2Z1bmN0aW9u IHNlbmRzIG11bHRpcGxlIG1lc3NhZ2VzIGF0IGEgY2FsbC4KK1RoZXkgYXJlIGdpdmVuIGJ5IHRo ZQorLkZhIG1zZ3ZlYwordmVjdG9yIGFsb25nIHdpdGgKKy5GYSB2bGVuCitzcGVjaWZ5aW5nIGl0 cyBzaXplIChsaW1pdGVkIHRvIDEwMjQpLiBUaGUgbnVtYmVyIG9mCitjaGFyYWN0ZXJzIHNlbnQg cGVyIGVhY2ggbWVzc2FnZSBpcyBwbGFjZWQgaW4gdGhlCisuRmEgbXNnX2xlbgorZmllbGQgb2Yg ZWFjaCBlbGVtZW50IG9mIHRoZSB2ZWN0b3IgYWZ0ZXIgdHJhbnNtaXNzaW9uLgorLlBwCiBObyBp bmRpY2F0aW9uIG9mIGZhaWx1cmUgdG8gZGVsaXZlciBpcyBpbXBsaWNpdCBpbiBhCiAuRm4gc2Vu ZCAuCiBMb2NhbGx5IGRldGVjdGVkIGVycm9ycyBhcmUgaW5kaWNhdGVkIGJ5IGEgcmV0dXJuIHZh bHVlIG9mIC0xLgpAQCAtMTM4LDEwICsxNTcsMTYgQEAgU2VlCiAuWHIgcmVjdiAyCiBmb3IgYSBk ZXNjcmlwdGlvbiBvZiB0aGUKIC5GYSBtc2doZHIKK3N0cnVjdHVyZSBhbmQgdGhlCisuRmEgbW1z Z2hkcgogc3RydWN0dXJlLgogLlNoIFJFVFVSTiBWQUxVRVMKLVRoZSBjYWxsIHJldHVybnMgdGhl IG51bWJlciBvZiBjaGFyYWN0ZXJzIHNlbnQsIG9yIC0xCi1pZiBhbiBlcnJvciBvY2N1cnJlZC4K K0FsbCBjYWxscyBleGNlcHQKKy5GbiBzZW5kbW1zZworcmV0dXJuIHRoZSBudW1iZXIgb2YgY2hh cmFjdGVycyBzZW50LiBUaGUKKy5GbiBzZW5kbW1zZworY2FsbCByZXR1cm5zIHRoZSBudW1iZXIg b2YgbWVzc2FnZXMgc2VudC4KK0lmIGFuIGVycm9yIG9jY3VycmVkIGEgdmFsdWUgb2YgLTEgaXMg cmV0dXJuZWQuCiAuU2ggRVJST1JTCiBUaGUKIC5GbiBzZW5kCkBAIC0xNDksNiArMTc0LDggQEAg ZnVuY3Rpb24gYW5kCiAuRm4gc2VuZHRvCiBhbmQKIC5GbiBzZW5kbXNnCithbmQKKy5GbiBzZW5k bW1zZwogc3lzdGVtIGNhbGxzCiBmYWlsIGlmOgogLkJsIC10YWcgLXdpZHRoIEVyCmRpZmYgLS1n aXQgYS9saWIvbGliYy9zeXMvc2VuZG1tc2cuYyBiL2xpYi9saWJjL3N5cy9zZW5kbW1zZy5jCm5l dyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAuLjNiOTBiOTAKLS0tIC9kZXYvbnVsbAor KysgYi9saWIvbGliYy9zeXMvc2VuZG1tc2cuYwpAQCAtMCwwICsxLDcyIEBACisvKgorICogQ29w eXJpZ2h0IChjKSAyMDE2IEJvcmlzIEFzdGFyZHpoaWV2LCBTbWFydGNvbS1CdWxnYXJpYSBBRAor ICogQWxsIHJpZ2h0cyByZXNlcnZlZC4KKyAqCisgKiBSZWRpc3RyaWJ1dGlvbiBhbmQgdXNlIGlu IHNvdXJjZSBhbmQgYmluYXJ5IGZvcm1zLCB3aXRoIG9yIHdpdGhvdXQKKyAqIG1vZGlmaWNhdGlv biwgYXJlIHBlcm1pdHRlZCBwcm92aWRlZCB0aGF0IHRoZSBmb2xsb3dpbmcgY29uZGl0aW9ucwor ICogYXJlIG1ldDoKKyAqIDEuIFJlZGlzdHJpYnV0aW9ucyBvZiBzb3VyY2UgY29kZSBtdXN0IHJl dGFpbiB0aGUgYWJvdmUgY29weXJpZ2h0CisgKiAgICBub3RpY2UocyksIHRoaXMgbGlzdCBvZiBj b25kaXRpb25zIGFuZCB0aGUgZm9sbG93aW5nIGRpc2NsYWltZXIgYXMKKyAqICAgIHRoZSBmaXJz dCBsaW5lcyBvZiB0aGlzIGZpbGUgdW5tb2RpZmllZCBvdGhlciB0aGFuIHRoZSBwb3NzaWJsZQor ICogICAgYWRkaXRpb24gb2Ygb25lIG9yIG1vcmUgY29weXJpZ2h0IG5vdGljZXMuCisgKiAyLiBS ZWRpc3RyaWJ1dGlvbnMgaW4gYmluYXJ5IGZvcm0gbXVzdCByZXByb2R1Y2UgdGhlIGFib3ZlIGNv cHlyaWdodAorICogICAgbm90aWNlKHMpLCB0aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhl IGZvbGxvd2luZyBkaXNjbGFpbWVyIGluCisgKiAgICB0aGUgZG9jdW1lbnRhdGlvbiBhbmQvb3Ig b3RoZXIgbWF0ZXJpYWxzIHByb3ZpZGVkIHdpdGggdGhlCisgKiAgICBkaXN0cmlidXRpb24uCisg KgorICogVEhJUyBTT0ZUV0FSRSBJUyBQUk9WSURFRCBCWSBUSEUgQ09QWVJJR0hUIEhPTERFUihT KSBgYEFTIElTJycgQU5EIEFOWQorICogRVhQUkVTUyBPUiBJTVBMSUVEIFdBUlJBTlRJRVMsIElO Q0xVRElORywgQlVUIE5PVCBMSU1JVEVEIFRPLCBUSEUKKyAqIElNUExJRUQgV0FSUkFOVElFUyBP RiBNRVJDSEFOVEFCSUxJVFkgQU5EIEZJVE5FU1MgRk9SIEEgUEFSVElDVUxBUgorICogUFVSUE9T RSBBUkUgRElTQ0xBSU1FRC4gIElOIE5PIEVWRU5UIFNIQUxMIFRIRSBDT1BZUklHSFQgSE9MREVS KFMpIEJFCisgKiBMSUFCTEUgRk9SIEFOWSBESVJFQ1QsIElORElSRUNULCBJTkNJREVOVEFMLCBT UEVDSUFMLCBFWEVNUExBUlksIE9SCisgKiBDT05TRVFVRU5USUFMIERBTUFHRVMgKElOQ0xVRElO RywgQlVUIE5PVCBMSU1JVEVEIFRPLCBQUk9DVVJFTUVOVCBPRgorICogU1VCU1RJVFVURSBHT09E UyBPUiBTRVJWSUNFUzsgTE9TUyBPRiBVU0UsIERBVEEsIE9SIFBST0ZJVFM7IE9SCisgKiBCVVNJ TkVTUyBJTlRFUlJVUFRJT04pIEhPV0VWRVIgQ0FVU0VEIEFORCBPTiBBTlkgVEhFT1JZIE9GIExJ QUJJTElUWSwKKyAqIFdIRVRIRVIgSU4gQ09OVFJBQ1QsIFNUUklDVCBMSUFCSUxJVFksIE9SIFRP UlQgKElOQ0xVRElORyBORUdMSUdFTkNFCisgKiBPUiBPVEhFUldJU0UpIEFSSVNJTkcgSU4gQU5Z IFdBWSBPVVQgT0YgVEhFIFVTRSBPRiBUSElTIFNPRlRXQVJFLAorICogRVZFTiBJRiBBRFZJU0VE IE9GIFRIRSBQT1NTSUJJTElUWSBPRiBTVUNIIERBTUFHRS4KKyAqLworCisjaW5jbHVkZSA8c3lz L2NkZWZzLmg+CitfX0ZCU0RJRCgiJEZyZWVCU0QkIik7CisKKyNpbmNsdWRlIDxlcnJuby5oPgor I2luY2x1ZGUgPHN5cy90eXBlcy5oPgorI2luY2x1ZGUgPHN5cy9zeXNjYWxsLmg+CisjaW5jbHVk ZSA8c3lzL3NvY2tldC5oPgorI2luY2x1ZGUgPHB0aHJlYWQuaD4KKyNpbmNsdWRlICJsaWJjX3By aXZhdGUuaCIKKworI2RlZmluZSBWTEVOX01BWCAxMDI0CisKK2ludAorc2VuZG1tc2coaW50IHMs IHN0cnVjdCBtbXNnaGRyICptc2d2ZWMsIHVuc2lnbmVkIGludCB2bGVuLCBpbnQgZmxhZ3MpCit7 CisJaW50IGksIHJldCwgc2VudDsKKworCWlmICh2bGVuID4gVkxFTl9NQVgpCisJCXZsZW4gPSBW TEVOX01BWDsKKworCXNlbnQgPSAwOworCWZvciAoaSA9IDA7IGkgPCB2bGVuOyBpKyspIHsKKwkJ ZXJybm8gPSAwOworCQlyZXQgPSBfX3N5c19zZW5kbXNnKHMsICZtc2d2ZWNbaV0ubXNnX2hkciwg ZmxhZ3MpOworCQlpZiAocmV0IDwgMCB8fCBlcnJubyAhPSAwKSB7CisJCQlpZiAoc2VudCAhPSAw KSB7CisJCQkJLyogV2UgaGF2ZSBzZW50IG1lc3NhZ2VzLiBMZXQgY2FsbGVyIGtub3cuICovCisJ CQkJZXJybm8gPSAwOworCQkJCXJldHVybiAoc2VudCk7CisJCQl9CisJCQlyZXR1cm4gKC0xKTsK KwkJfQorCisJCS8qIFNhdmUgc2VudCBieXRlcyAqLworCQltc2d2ZWNbaV0ubXNnX2xlbiA9IHJl dDsKKworCQlzZW50Kys7CisJfQorCisJcmV0dXJuIChzZW50KTsKK30KKworI3VuZGVmIFZMRU5f TUFYCmRpZmYgLS1naXQgYS9zeXMvc3lzL3NvY2tldC5oIGIvc3lzL3N5cy9zb2NrZXQuaAppbmRl eCAxOGUyZGUxLi5mNzBkY2VkIDEwMDY0NAotLS0gYS9zeXMvc3lzL3NvY2tldC5oCisrKyBiL3N5 cy9zeXMvc29ja2V0LmgKQEAgLTU5NSw2ICs1OTUsMTggQEAgc3RydWN0IHNmX2hkdHIgewogI2Vu ZGlmIC8qIF9LRVJORUwgKi8KICNlbmRpZiAvKiBfX0JTRF9WSVNJQkxFICovCiAKKyNpZm5kZWYg X0tFUk5FTAorI2lmZGVmIF9fQlNEX1ZJU0lCTEUKKy8qCisgKiBTZW5kL3JlY3ZtbXNnIHNwZWNp ZmljIHN0cnVjdHVyZShzKQorICovCitzdHJ1Y3QgbW1zZ2hkciB7CisJc3RydWN0IG1zZ2hkcglt c2dfaGRyOwkJLyogbWVzc2FnZSBoZWFkZXIgKi8KKwl1bnNpZ25lZCBpbnQJbXNnX2xlbjsJCS8q IG1lc3NhZ2UgbGVuZ3RoICovCit9OworI2VuZGlmIC8qIF9fQlNEX1ZJU0lCTEUgKi8KKyNlbmRp ZiAvKiAhX0tFUk5FTCAqLworCiAjaWZuZGVmCV9LRVJORUwKIAogI2luY2x1ZGUgPHN5cy9jZGVm cy5oPgpAQCAtNjE1LDExICs2MjcsMTcgQEAgaW50CWxpc3RlbihpbnQsIGludCk7CiBzc2l6ZV90 CXJlY3YoaW50LCB2b2lkICosIHNpemVfdCwgaW50KTsKIHNzaXplX3QJcmVjdmZyb20oaW50LCB2 b2lkICosIHNpemVfdCwgaW50LCBzdHJ1Y3Qgc29ja2FkZHIgKiBfX3Jlc3RyaWN0LCBzb2NrbGVu X3QgKiBfX3Jlc3RyaWN0KTsKIHNzaXplX3QJcmVjdm1zZyhpbnQsIHN0cnVjdCBtc2doZHIgKiwg aW50KTsKKyNpZiBfX0JTRF9WSVNJQkxFCitpbnQJcmVjdm1tc2coaW50LCBzdHJ1Y3QgbW1zZ2hk ciAqLCB1bnNpZ25lZCBpbnQsIGludCk7CisjZW5kaWYKIHNzaXplX3QJc2VuZChpbnQsIGNvbnN0 IHZvaWQgKiwgc2l6ZV90LCBpbnQpOwogc3NpemVfdAlzZW5kdG8oaW50LCBjb25zdCB2b2lkICos CiAJICAgIHNpemVfdCwgaW50LCBjb25zdCBzdHJ1Y3Qgc29ja2FkZHIgKiwgc29ja2xlbl90KTsK IHNzaXplX3QJc2VuZG1zZyhpbnQsIGNvbnN0IHN0cnVjdCBtc2doZHIgKiwgaW50KTsKICNpZiBf X0JTRF9WSVNJQkxFCitpbnQJc2VuZG1tc2coaW50LCBzdHJ1Y3QgbW1zZ2hkciAqLCB1bnNpZ25l ZCBpbnQsIGludCk7CisjZW5kaWYKKyNpZiBfX0JTRF9WSVNJQkxFCiBpbnQJc2VuZGZpbGUoaW50 LCBpbnQsIG9mZl90LCBzaXplX3QsIHN0cnVjdCBzZl9oZHRyICosIG9mZl90ICosIGludCk7CiBp bnQJc2V0ZmliKGludCk7CiAjZW5kaWYK --001a114b3ebc6c912e0529ae96fb-- From owner-freebsd-net@freebsd.org Tue Jan 19 22:00:53 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83416A895B8 for ; Tue, 19 Jan 2016 22:00:53 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6FCFD1F91 for ; Tue, 19 Jan 2016 22:00:53 +0000 (UTC) (envelope-from jilles@stack.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 6B7D3A895B7; Tue, 19 Jan 2016 22:00:53 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6AE6EA895B5; Tue, 19 Jan 2016 22:00:53 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 364741F8E; Tue, 19 Jan 2016 22:00:53 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 9B1AFB8059; Tue, 19 Jan 2016 23:00:49 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 86B1828494; Tue, 19 Jan 2016 23:00:49 +0100 (CET) Date: Tue, 19 Jan 2016 23:00:49 +0100 From: Jilles Tjoelker To: Boris Astardzhiev Cc: Konstantin Belousov , net@freebsd.org, threads@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160119220049.GA56408@stack.nl> References: <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 22:00:53 -0000 On Tue, Jan 19, 2016 at 01:58:27PM +0200, Boris Astardzhiev wrote: > I removed the pthread_testcancel() calls and cut the interposing > stuff from my patch as suggested. I extended the send/recv(2) manpages > regarding > the mm calls. Comments and suggestions? > [snip] > diff --git a/lib/libc/sys/Symbol.map b/lib/libc/sys/Symbol.map > index 7b3257c..724e1b4 100644 > --- a/lib/libc/sys/Symbol.map > +++ b/lib/libc/sys/Symbol.map > @@ -399,6 +399,8 @@ FBSD_1.4 { > utimensat; > numa_setaffinity; > numa_getaffinity; > + sendmmsg; > + recvmmsg; > }; OK. > > FBSDprivate_1.0 { > @@ -1051,4 +1053,6 @@ FBSDprivate_1.0 { > gssd_syscall; > __libc_interposing_slot; > __libc_sigwait; > + _sendmmsg; > + _recvmmsg; > }; The _ versions need not be exported. Not exporting reduces code size and improves performance. > diff --git a/lib/libc/sys/recv.2 b/lib/libc/sys/recv.2 > index 326e7ff..81a0201 100644 > --- a/lib/libc/sys/recv.2 > +++ b/lib/libc/sys/recv.2 > [snip] I think the recv.2 and send.2 man pages are long enough as they are, and separate recvmmsg.3 and sendmmsg.3 pages will be clearer. This is also because recvmmsg/sendmmsg can be ignored when performance is good enough without them. This differs from what Konstantin thinks. > diff --git a/lib/libc/sys/recvmmsg.c b/lib/libc/sys/recvmmsg.c > new file mode 100644 > index 0000000..6b5158a > --- /dev/null > +++ b/lib/libc/sys/recvmmsg.c > @@ -0,0 +1,72 @@ > [snip] > +int > +recvmmsg(int s, struct mmsghdr *msgvec, unsigned int vlen, int flags) > +{ The Linux version has an additional parameter struct timespec *timeout (but only for recvmmsg, not for sendmmsg). Note that implementing this in a Linux-compatible manner has low overhead, since Linux only checks it between packets and never interrupts a wait because of this timeout (source: http://man7.org/linux/man-pages/man2/recvmmsg.2.html ). > [snip] -- Jilles Tjoelker From owner-freebsd-net@freebsd.org Tue Jan 19 22:16:09 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B45D4A89ABD for ; Tue, 19 Jan 2016 22:16:09 +0000 (UTC) (envelope-from c2h@romeo.emu.st) Received: from f5.bushwire.net (f5.bushwire.net [IPv6:2607:fc50:1000:5b00::2]) by mx1.freebsd.org (Postfix) with ESMTP id 9B13F19D3 for ; Tue, 19 Jan 2016 22:16:09 +0000 (UTC) (envelope-from c2h@romeo.emu.st) Received: by f5.bushwire.net (Postfix, from userid 1001) id 640E4AC909; Tue, 19 Jan 2016 14:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/simple; d=emu.st; s=2015; t=1453241768; bh=j7iNw7TIHYvCedcHKRMeft4lRsU=; h=Comments:Received:Date:Message-ID:From:To:Subject:References: MIME-Version:Content-Type:Content-Disposition:In-Reply-To; b=mkEN7nvjIZG1yXu6nSR5G8DlpOoSdIcVCMRSsJwe/fYPbw87dePX1p8Vaqo3cN7sz 5Q8rqaF/oCosHoIKAqeLdEaURO7qxZQwFujBQ3C4LZwa7uDvmNS8o0LraFJKYLBdNO seYGB4OPpScTdaxGCvCk7LhS7WUk/YYkLZGug6wk=ug6wk= Comments: QMDA 0.3 Received: (qmail 20609 invoked by uid 1001); 19 Jan 2016 22:16:08 -0000 Date: 19 Jan 2016 22:16:08 +0000 Message-ID: <20160119221608.20608.qmail@f5-external.bushwire.net> From: "Mark Delany" To: freebsd-net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? References: <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160119220049.GA56408@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20160119220049.GA56408@stack.nl> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 22:16:09 -0000 On 19Jan16, Jilles Tjoelker allegedly wrote: > I think the recv.2 and send.2 man pages are long enough as they are, and > separate recvmmsg.3 and sendmmsg.3 pages will be clearer. This is also > because recvmmsg/sendmmsg can be ignored when performance is good enough > without them. This differs from what Konstantin thinks. If they are to be made separate man pages can I suggest that the recv/send(2) manpages be changes to at least make early reference to the *mmsg() calls? Purely as marketing. My perception is that awareness of the *mmsg() calls is rather limited. Mark. From owner-freebsd-net@freebsd.org Tue Jan 19 23:21:12 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCC5A88431 for ; Tue, 19 Jan 2016 23:21:12 +0000 (UTC) (envelope-from Vedant.Mathur@netapp.com) Received: from mx144.netapp.com (mx144.netapp.com [216.240.21.25]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B7D5E18AF for ; Tue, 19 Jan 2016 23:21:11 +0000 (UTC) (envelope-from Vedant.Mathur@netapp.com) X-IronPort-AV: E=Sophos;i="5.22,319,1449561600"; d="scan'208,217";a="93168100" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx144-out.netapp.com with ESMTP; 19 Jan 2016 15:20:16 -0800 Received: from HIOEXCMBX04-PRD.hq.netapp.com (10.122.105.37) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1130.7; Tue, 19 Jan 2016 15:20:16 -0800 Received: from HIOEXCMBX04-PRD.hq.netapp.com ([::1]) by hioexcmbx04-prd.hq.netapp.com ([fe80::5d07:58:69ea:1a63%21]) with mapi id 15.00.1130.005; Tue, 19 Jan 2016 15:20:16 -0800 From: "Mathur, Vedant" To: "freebsd-net@freebsd.org" Subject: Performing a socket accept in listen sockets' upcall - Yes or No? Thread-Topic: Performing a socket accept in listen sockets' upcall - Yes or No? Thread-Index: AQHRUw/0G8ZgJLe6xkWgTa5dcyoGxA== Date: Tue, 19 Jan 2016 23:20:15 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 23:21:12 -0000 Hi all, I am working with FreeBSD in an environment where majority of my applicatio= ns run in the kernel and are heavily relying on the socket upcall mechanism= . All of these kernel applications use the socket upcall mechanism for sock= et I/O. These applications also use the upcall mechanism to accept incoming TCP con= nections. How the applications do so is by registering a receive upcall on= the head socket and waiting for the upcall. When the stack makes the recei= ve upcall on the listen socket, most of the applications defer the actual a= ccept to a different thread using a queueing mechanism and return from the = upcall. However, I have some applications which instead of deferring the socket acc= ept operation to a different thread want to perform the socket accept opera= tion inline. In other words the application wants to accept the child socke= t in the context of the listen sockets' upcall. My questions are regarding = this behavior and are as follows: A) First and foremost, does the FreeBSD socket upcall framework even suppor= t such kind of an accept operation? Does this conceptually break FreeBSD so= cket accept and upcall semantics? Is the upcall mechanism to be ONLY used f= or socket I/O? B) If the accept operation is supported in the context of the head sockets'= upcall then, how do we handle the use after free cases where issuing a hea= d sockets upcall can potentially make the application accept and then close= the child socket which we access after the upcall returns. To be more spec= ific, in FreeBSD 11.x the syncache_socket() code calls sonewconn() and then= connects the child socket to the peer. Later it calls soisconnected() whic= h places the child socket in head's so_comp queue and makes an upcall which= can potentially free the child socket. Once syncache_socket() returns, the= call stack eventually returns until tcp_input() for TCP sockets. tcp_input= () now calls tcp_do_segment() which accesses the child socket and its inpcb= and tcpcb which can potentially be freed memory. If A) is possible then, I can see that commit r261242 modified the code su= ch that we deferred placing the child socket on the so_comp queue until the= connect was complete. I believe this change is still susceptible to the us= e after free case described above. If we agree then what are the potential= solutions? Thanks! Vedant Mathur From owner-freebsd-net@freebsd.org Tue Jan 19 23:37:07 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76DE0A88D06 for ; Tue, 19 Jan 2016 23:37:07 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 417AB195A for ; Tue, 19 Jan 2016 23:37:07 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.9/8.14.9) with ESMTP id u0JNb57f015196 for ; Tue, 19 Jan 2016 18:37:05 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.9/8.14.4/Submit) id u0JNb5lh015193; Tue, 19 Jan 2016 18:37:05 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <22174.51361.2022.984959@hergotha.csail.mit.edu> Date: Tue, 19 Jan 2016 18:37:05 -0500 From: Garrett Wollman To: freebsd-net@freebsd.org Subject: Another issue with ixl(4) with netmap X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 19 Jan 2016 18:37:05 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hergotha.csail.mit.edu X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 23:37:07 -0000 I noticed that a large number -- but by no means all -- of the packets captured using libpcap on a netmap'ified ixl(4) interface show up as truncated -- usually by exactly four bytes. They show up in tcpdump like this: 18:10:05.348735 IP truncated-ip - 4 bytes missing! 128.30.xxx.xxx.443 > yyy.yyy.yyy.yyy.54711: Flags [P.], seq 3151716053:3151716112, ack 3109956633, win 31, length 59 18:10:05.348735 IP truncated-ip - 4 bytes missing! zzz.zzz.zzz.zzz.17376 > 128.30.aaa.aaa.80: Flags [.], ack 1771224170, win 12775, options [nop,nop,sack 1 [|tcp]> 18:10:05.348735 IP truncated-ip - 4 bytes missing! zzz.zzz.zzz.zzz.17376 > 128.30.aaa.aaa.80: Flags [.], ack 1, win 12775, options [nop,nop,sack 1 [|tcp]> This does not occur when the same interface is used with BPF (but libpcap can't keep up with the traffic when using BPF). Has anyone see this before, and if so, what was the solution? (This is on 10.2 with the ixl(4) updates merged from 10-stable.) -GAWollman From owner-freebsd-net@freebsd.org Tue Jan 19 23:47:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33659A8906A for ; Tue, 19 Jan 2016 23:47:58 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B260B1D60; Tue, 19 Jan 2016 23:47:57 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x231.google.com with SMTP id oh2so373399088lbb.3; Tue, 19 Jan 2016 15:47:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=BLpI8YUgoiUTbXQ2wac+1gMxQhkGfacfysvAvmugUFQ=; b=jYvEYGSgKv0AksbIykuxl465ZDKOpnLcx2/ENqMstbBWfepLXUYFU9pqhYjF3nvUhC IhCt9VgdM6KP70yaJpebIKRCaayGBjYvhyJBrQWcTH5V+3hiHFHLJv8TALZUrwvD+6Se UNII2uH1+ldeellK8fNKdFHS3S3IVw906u9IupUFI8uIUdrwyfAcPDGi/5lU61DME5kz FKd+A8tEixIBq0GI+996ea9Z+D16JfhzMys2xErxCAOVpZxUZUUuLHFxHRg/lO3pSDUt /acprn9VQj7uJ4wv2s+3nXjF29/Y88tEkYMuxzr1yNdgHZPvKANcuA+NkOnGB9vzTpBG 0sfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=BLpI8YUgoiUTbXQ2wac+1gMxQhkGfacfysvAvmugUFQ=; b=Ly2T2/hjwMPHjr/na5yE7YdlV2rxKXiQKWUnfgRuYjccUzQQieptAHyr5AdENL17M1 TZb1i/wVbjKBSMY+0A1Vpj6imLuELRp0Fvl+vEu1VNmfVV2uZLXl+2+CNWw/jGqTK37U q8KCWmnKNA6kjz06bjEfmsH1m5p/3b3DlFzcZmu5BJAkCH4oGux9goCS6S3GHvdMzb0l 7g5Wu1h+K3jZ/2N9U+z8lX5NasW/36HU6Euv7o0822v8lm86Nt1d1V/SkMRot944KZe2 q8LVfdEeiCiTZT63yBT8GtS+NWOaGTcRniGOyToIHszjhj8qdl7PtVbD4KegpXYHDHgm mWBw== X-Gm-Message-State: AG10YORf/vCMOXpVdZznVz+udd0n11U53HtwiGYMadjsnM7021cl25mntboQobsKxv7DLBKSTvE8x4il8FZVEA== MIME-Version: 1.0 X-Received: by 10.112.155.201 with SMTP id vy9mr4163649lbb.54.1453247275610; Tue, 19 Jan 2016 15:47:55 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Tue, 19 Jan 2016 15:47:55 -0800 (PST) In-Reply-To: <22174.51361.2022.984959@hergotha.csail.mit.edu> References: <22174.51361.2022.984959@hergotha.csail.mit.edu> Date: Wed, 20 Jan 2016 00:47:55 +0100 X-Google-Sender-Auth: YLgjw5_af5vkVDWytqPLr9L_Lls Message-ID: Subject: Re: Another issue with ixl(4) with netmap From: Luigi Rizzo To: Garrett Wollman , Giuseppe Lettieri Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2016 23:47:58 -0000 Interesting report, thanks. My colleague Giuseppe can have a look at this. While not sure about the actual issue I can tell you that generally we try to include checksums in drivers that support netmap, because PCIe bus transactions are faster for transfers that are multiple of 64 bytes and aligned to those boundaries. Keeping the CRC (and doing the stripping in software) makes the requirement for min-sized packets, which are a common test case. The symptoms you see suggest a mismatch on who is doing the CRC stripping, which seems to occur twice; what is strange is that in case i would expect the opposite error, i.e. disable stripping in the hardware and forget to enable it in the software. cheers luigi On Wed, Jan 20, 2016 at 12:37 AM, Garrett Wollman wrote: > I noticed that a large number -- but by no means all -- of the packets > captured using libpcap on a netmap'ified ixl(4) interface show up as > truncated -- usually by exactly four bytes. They show up in tcpdump > like this: > > 18:10:05.348735 IP truncated-ip - 4 bytes missing! 128.30.xxx.xxx.443 > yyy.yyy.yyy.yyy.54711: Flags [P.], seq 3151716053:3151716112, ack 3109956633, win 31, length 59 > 18:10:05.348735 IP truncated-ip - 4 bytes missing! zzz.zzz.zzz.zzz.17376 > 128.30.aaa.aaa.80: Flags [.], ack 1771224170, win 12775, options [nop,nop,sack 1 [|tcp]> > 18:10:05.348735 IP truncated-ip - 4 bytes missing! zzz.zzz.zzz.zzz.17376 > 128.30.aaa.aaa.80: Flags [.], ack 1, win 12775, options [nop,nop,sack 1 [|tcp]> > > This does not occur when the same interface is used with BPF (but > libpcap can't keep up with the traffic when using BPF). Has anyone > see this before, and if so, what was the solution? (This is on 10.2 > with the ixl(4) updates merged from 10-stable.) > > -GAWollman > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Wed Jan 20 07:32:01 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EE32A8944E for ; Wed, 20 Jan 2016 07:32:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 55B5E15BA for ; Wed, 20 Jan 2016 07:32:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 54FEAA8944C; Wed, 20 Jan 2016 07:32:01 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 547ADA8944B; Wed, 20 Jan 2016 07:32:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C6D6C15B5; Wed, 20 Jan 2016 07:32:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0K7VtTp043835 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 20 Jan 2016 09:31:55 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0K7VtTp043835 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0K7VtE8043834; Wed, 20 Jan 2016 09:31:55 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 20 Jan 2016 09:31:55 +0200 From: Konstantin Belousov To: Boris Astardzhiev Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160120073154.GB3942@kib.kiev.ua> References: <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 07:32:01 -0000 On Tue, Jan 19, 2016 at 01:58:27PM +0200, Boris Astardzhiev wrote: > +int > +recvmmsg(int s, struct mmsghdr *msgvec, unsigned int vlen, int flags) > +{ > + int i, ret, rcvd; Shouldn't i and rcvd be unsigned as well ? Shouldn't return value also be unsigned ? > + > + if (vlen > VLEN_MAX) > + vlen = VLEN_MAX; Why is this restriction needed ? > + > + rcvd = 0; > + for (i = 0; i < vlen; i++) { > + errno = 0; > + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); > + if (ret < 0 || errno != 0) { I do not see why do you need to clear errno before, and then do this test. Just check ret == -1, in which case errno was set from the immediate syscall. > + if (rcvd != 0) { > + /* We've received messages. Let caller know. */ > + errno = 0; This cleaning is not needed as well. For successfull functions returns, errno value is undefined. > + return (rcvd); > + } > + return (-1); > + } > + > + /* Save received bytes */ > + msgvec[i].msg_len = ret; > + Extra empty line. > + rcvd++; > + } > + > + return (rcvd); > +} From owner-freebsd-net@freebsd.org Wed Jan 20 07:50:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A556AA89CDD for ; Wed, 20 Jan 2016 07:50:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 961441247 for ; Wed, 20 Jan 2016 07:50:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0K7o3Bp080252 for ; Wed, 20 Jan 2016 07:50:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206368] [PATCH] kevent doesn't notify EV_ENABLE-ed events Date: Wed, 20 Jan 2016 07:50:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: hiren@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 07:50:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206368 Hiren Panchasara changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org CC| |hiren@FreeBSD.org --- Comment #2 from Hiren Panchasara --- jmg@, can you please take a look to see if this indeed is a regression caus= ed by r274560? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Wed Jan 20 08:29:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 864F3A8896C for ; Wed, 20 Jan 2016 08:29:50 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 614321E3A for ; Wed, 20 Jan 2016 08:29:50 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5E953A88969; Wed, 20 Jan 2016 08:29:50 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 440DEA88968; Wed, 20 Jan 2016 08:29:50 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C4A841E38; Wed, 20 Jan 2016 08:29:49 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id l65so168946761wmf.1; Wed, 20 Jan 2016 00:29:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+vTDR6o5a+4Vpe7UStvzcqXSbAADfj4yw89x0DnoWgA=; b=sGAOiMjRI6vo85cKo2Kj5Lhhy6oKMdSGRZM30DGLX5wifwlwCrJ6/BxhePBuQApvNz R8lgSFMVj0+C5I6kmhN8m/1PnWcOWiwk92BZfudeSWnLyursnXFSh0iVLUQmhXbCmEGt HPpIGjt+TJHkBb9E8YB1kYWB6jFqK9XcnwtUBmdjCLYAV9DVUgFuUt7+zcDkTAYj5v5h +7wbw+BjLfWFAfKdxOucv8Pm8si6VdDhOtX+OcdC1uZ51GDwhjbWel0slMBqPkfSoclh Fxf+TBBfr9xQu8EJjsCNDOSy97Tyw9wizwqOgMtTTcU/X37FzGTkuhj8Mq8Krm1s023A fkEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+vTDR6o5a+4Vpe7UStvzcqXSbAADfj4yw89x0DnoWgA=; b=LCkaoWd21uG2ICHyMwxn0NPtIuAjNwu22BkymWa65x9ccmwyVJNReki+CwNnY9RgkG 0GdBMdRoZ7s4UdQFEjdWvAssx4ikjF3DnliZE2Z6tF1KbW1CcOA3t2gP5DKJ+PZcaj2u Yla6oWNJusaCqndqaWFwDfscoPPZytBBsCUDK1dwI8tpKJZE2t46UCm/n22NdR/U6D6/ 6/KpkVxlxJRxegoThdr+4hUWIK8W75IVOjFHcI0GJp3lLAgwWhU7JgU/5+7kZIBPD55F 4zEvusKV2erak1AtrdlYOIgQw/OHXdWALOVttq5qqemw5tQ2UUNRCeMJ64VOoYw7/ntF GkWA== X-Gm-Message-State: AG10YORiKsUqT7/oQ1+Brt8KV+D4XOAoqsDWEPX2P8dZQCyEFYjH9O7q9A3TZ09nfuFVNFpX29UsfXZxGgzwPw== MIME-Version: 1.0 X-Received: by 10.28.16.8 with SMTP id 8mr2643914wmq.77.1453278587459; Wed, 20 Jan 2016 00:29:47 -0800 (PST) Received: by 10.28.136.148 with HTTP; Wed, 20 Jan 2016 00:29:47 -0800 (PST) In-Reply-To: <20160120073154.GB3942@kib.kiev.ua> References: <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160120073154.GB3942@kib.kiev.ua> Date: Wed, 20 Jan 2016 10:29:47 +0200 Message-ID: Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? From: Boris Astardzhiev To: Konstantin Belousov Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 08:29:50 -0000 jt>> jt>> FBSDprivate_1.0 { jt>> @@ -1051,4 +1053,6 @@ FBSDprivate_1.0 { jt>> gssd_syscall; jt>> __libc_interposing_slot; jt>> __libc_sigwait; jt>> + _sendmmsg; jt>> + _recvmmsg; jt>> }; jt> jt>The _ versions need not be exported. Not exporting reduces code size and jt>improves performance. I'll fix it. jt>> diff --git a/lib/libc/sys/recv.2 b/lib/libc/sys/recv.2 jt>> index 326e7ff..81a0201 100644 jt>> --- a/lib/libc/sys/recv.2 jt>> +++ b/lib/libc/sys/recv.2 jt>> [snip] jt> jt>I think the recv.2 and send.2 man pages are long enough as they are, and jt>separate recvmmsg.3 and sendmmsg.3 pages will be clearer. This is also jt>because recvmmsg/sendmmsg can be ignored when performance is good enough jt>without them. This differs from what Konstantin thinks. md>If they are to be made separate man pages can I suggest that the md>recv/send(2) manpages be changes to at least make early reference to md>the *mmsg() calls? md> md>Purely as marketing. My perception is that awareness of the *mmsg() md>calls is rather limited. Let me know the final decision then - whether in the existing manpages or in new files. jt>The Linux version has an additional parameter struct timespec *timeout jt>(but only for recvmmsg, not for sendmmsg). Note that implementing this jt>in a Linux-compatible manner has low overhead, since Linux only checks jt>it between packets and never interrupts a wait because of this timeout jt>(source: http://man7.org/linux/man-pages/man2/recvmmsg.2.html ). That's right. Shall I try to implement the timeout part or leave it the way it is now? kb>Shouldn't i and rcvd be unsigned as well ? Shouldn't return value kb>also be unsigned ? I think i and rcvd should be unsigned whereas ret should not - after all if an error occurred we get -1. kb>> + kb>> + if (vlen > VLEN_MAX) kb>> + vlen = VLEN_MAX; kb>Why is this restriction needed ? Not needed. I'll remove it. kb>> + kb>> + rcvd = 0; kb>> + for (i = 0; i < vlen; i++) { kb>> + errno = 0; kb>> + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); kb>> + if (ret < 0 || errno != 0) { kb>I do not see why do you need to clear errno before, and then do this test. kb>Just check ret == -1, in which case errno was set from the immediate syscall. kb> kb>> + if (rcvd != 0) { kb>> + /* We've received messages. Let caller know. */ kb>> + errno = 0; kb>This cleaning is not needed as well. For successfull functions returns, kb>errno value is undefined. Wouldn't I confuse apps if they check errno in the follow case - I want to receive two messages. The first __sys_recvmsg succeeds and then for the second __sys_recvmsg fails. Thus errno will be != 0 and I'm telling the app that I have received one message by returning 1 but errno will be != 0. Is this correct? Regards, Boris Astardzhiev On Wed, Jan 20, 2016 at 9:31 AM, Konstantin Belousov wrote: > On Tue, Jan 19, 2016 at 01:58:27PM +0200, Boris Astardzhiev wrote: > > +int > > +recvmmsg(int s, struct mmsghdr *msgvec, unsigned int vlen, int flags) > > +{ > > + int i, ret, rcvd; > Shouldn't i and rcvd be unsigned as well ? Shouldn't return value > also be unsigned ? > > + > > + if (vlen > VLEN_MAX) > > + vlen = VLEN_MAX; > Why is this restriction needed ? > > > + > > + rcvd = 0; > > + for (i = 0; i < vlen; i++) { > > + errno = 0; > > + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); > > + if (ret < 0 || errno != 0) { > I do not see why do you need to clear errno before, and then do this test. > Just check ret == -1, in which case errno was set from the immediate > syscall. > > > + if (rcvd != 0) { > > + /* We've received messages. Let caller > know. */ > > + errno = 0; > This cleaning is not needed as well. For successfull functions returns, > errno value is undefined. > > > + return (rcvd); > > + } > > + return (-1); > > + } > > + > > + /* Save received bytes */ > > + msgvec[i].msg_len = ret; > > + > Extra empty line. > > + rcvd++; > > + } > > + > > + return (rcvd); > > +} > From owner-freebsd-net@freebsd.org Wed Jan 20 10:30:56 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAA7AA89682 for ; Wed, 20 Jan 2016 10:30:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 94C381D94 for ; Wed, 20 Jan 2016 10:30:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 91AF6A8967F; Wed, 20 Jan 2016 10:30:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9121CA8967D; Wed, 20 Jan 2016 10:30:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 58AEE1D92; Wed, 20 Jan 2016 10:30:55 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id CF54910471B6; Wed, 20 Jan 2016 21:30:45 +1100 (AEDT) Date: Wed, 20 Jan 2016 21:30:44 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Boris Astardzhiev , threads@freebsd.org, Jilles Tjoelker , net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? In-Reply-To: <20160120073154.GB3942@kib.kiev.ua> Message-ID: <20160120205544.Q2305@besplex.bde.org> References: <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160120073154.GB3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=R4L+YolX c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=FZoMsACIJPzh2gu83OkA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 10:30:56 -0000 On Wed, 20 Jan 2016, Konstantin Belousov wrote: > On Tue, Jan 19, 2016 at 01:58:27PM +0200, Boris Astardzhiev wrote: >> +int >> +recvmmsg(int s, struct mmsghdr *msgvec, unsigned int vlen, int flags) >> +{ >> + int i, ret, rcvd; > Shouldn't i and rcvd be unsigned as well ? Shouldn't return value > also be unsigned ? None of the above. Plain recvmsg() returns ssize_t and its len arg has type size_t. That is excessively typedefed and excessively large with 64-bit ssize_t, but it is silly for the multiple-message variant to use smaller types. Otherwise, all the integer types should be int. recvmsg() is still declared in syscalls.master as returning int, but it seems to be correct internallly. It is like read() and returns ssize_t in td_retval[0]. No wrong prototypes seem to be generated from the wrong retun types in syscalls.master. >> + rcvd = 0; >> + for (i = 0; i < vlen; i++) { >> + errno = 0; >> + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); >> + if (ret < 0 || errno != 0) { > I do not see why do you need to clear errno before, and then do this test. > Just check ret == -1, in which case errno was set from the immediate syscall. The errno method (and not checking ret at all) is best if for syscalls that return -1 for a non-error. It is not needed here. Bruce From owner-freebsd-net@freebsd.org Wed Jan 20 10:36:56 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCA5AA89866 for ; Wed, 20 Jan 2016 10:36:56 +0000 (UTC) (envelope-from chagin.dmitry@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B915A10B7 for ; Wed, 20 Jan 2016 10:36:56 +0000 (UTC) (envelope-from chagin.dmitry@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B83FAA89865; Wed, 20 Jan 2016 10:36:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F1D2A89860; Wed, 20 Jan 2016 10:36:56 +0000 (UTC) (envelope-from chagin.dmitry@gmail.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6537A10B6; Wed, 20 Jan 2016 10:36:56 +0000 (UTC) (envelope-from chagin.dmitry@gmail.com) Received: by mail-ig0-x22e.google.com with SMTP id z14so98573846igp.1; Wed, 20 Jan 2016 02:36:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=WyZEfFBDMONX4ohADRokt0i8vFA21mKNiz5jHtizpwU=; b=0ym1B6tTPv+vMlj/PBvOa0KmUiPJmBx+qP2/jAlNgFp797oyDFEwgKKsNnZuVqP9UO p1YzYAiMq9PEixz07Z44qL140wnpsuZqazcJWbtzQxp+G6CRFlsu+s1N9fit43gf/+1d pwtTh6tBQ6NfQt2C6LlvZ3SpYxxlm1GVcH4Q3I+FCDZ4/H/HbWkpVAerC8LxPyx4yMFq kSiOkUbU886tWc7rbCYqiBnTQocruIWrGl30KKSucTlI2Nhy6wvmnbAsxZ2h1fW8J7Ks uPiuiBCzmUSvMYg1sUI4ecDlE0GG3r0A0XYxnK2Hi4fm6WFLavqZ1eBv++foir16y0IY DWoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=WyZEfFBDMONX4ohADRokt0i8vFA21mKNiz5jHtizpwU=; b=cIhf9rAEJcv+je7Fampm1R4qdf7Vzw+Z9T1KU4QPkcdM31Y8oQnMpWgtWR9Of/aiI3 1KGteMFB/8TpvtyqwhGaad761D0NnV8nWYEwsX1VlvDNg3tbIk3XrfxV1E8tQeQM4F+I hglD/+YPchgsbbftMisLpw1fLuSq3C7jvpMQDO5KX++YwQo1ZcyTHQR1qRU1HGOUA6S1 teRzoyl1RvTP2lODjHLXTxQF6wMSq+VtU2XPwLe+QuT4DER6Y+4xjbQ8/26QDQ4qTqvf R4laVKBql0juHlK5LKtxi0ZjdHjoN80UPAiCHbGngvlfYvRI+f5xSiOmRGYKrvKSNxvC OmGA== X-Gm-Message-State: AG10YOS0Ti5kB5pxEzI9elX1pZsxsxBFD3q4b5bH8nIaHCBd2lMBtYLXxdzJep2IRmI9n+aekBTsoHWG5Ym5eg== MIME-Version: 1.0 X-Received: by 10.50.21.10 with SMTP id r10mr2928653ige.93.1453286215854; Wed, 20 Jan 2016 02:36:55 -0800 (PST) Sender: chagin.dmitry@gmail.com Received: by 10.79.82.130 with HTTP; Wed, 20 Jan 2016 02:36:55 -0800 (PST) In-Reply-To: References: <20160108204606.G2420@besplex.bde.org> <20160113080349.GC72455@kib.kiev.ua> <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> Date: Wed, 20 Jan 2016 13:36:55 +0300 X-Google-Sender-Auth: EbHlJ9b9lzgCZi_O4I6WYJ0WkL0 Message-ID: Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? From: Dmitry Chagin To: Boris Astardzhiev Cc: Konstantin Belousov , threads@freebsd.org, net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 10:36:57 -0000 2016-01-19 14:58 GMT+03:00 Boris Astardzhiev : > Hello, > > I removed the pthread_testcancel() calls and cut the interposing > stuff from my patch as suggested. I extended the send/recv(2) manpages > regarding > the mm calls. Comments and suggestions? > > btw, what is the reason to not make this functions as a system call? seems that calling [xxxx]msg syscalls in a loop will be a considerable overhead From owner-freebsd-net@freebsd.org Wed Jan 20 15:08:57 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B5DDA88B99 for ; Wed, 20 Jan 2016 15:08:57 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: from mail-oi0-x22a.google.com (mail-oi0-x22a.google.com [IPv6:2607:f8b0:4003:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03CBF1C8A for ; Wed, 20 Jan 2016 15:08:57 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: by mail-oi0-x22a.google.com with SMTP id k206so6903257oia.1 for ; Wed, 20 Jan 2016 07:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=cvi1P5aEnbE1JmN/h1kPX/qDTe+M/0jyNaxtGI0IXjA=; b=j+7ceJhYfqKXz/gAmvMJOIcPSjyo/1pvL4RyNEi4oPyTmec/O5Ea92iIUEdTf8I4Pb 94GEv8kgGoOSAoMZNE8uIXm02OdSgqvEqRcBAao3GEVxubrcSFNo93eI4gV8Ka8DpVGw +ssKPFG+89V6hSPpluw30zq+lzo+h/GPCDQmDsttjEuL0wwgCFtsTnY6gA6KvAfItxfw V1oJDja6iFRZDfU5S/ZUpOuAcHO8a+K6noNW7cBW++OJviYQqqo5Hwu6SXIFqv5j/jWy O8aqOP5tBzoaT3t1UVAeRhaHe4Ilwnydrs2KRVqOuAQRO0T41hzCot9wYQriaPcunCHP LwqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=cvi1P5aEnbE1JmN/h1kPX/qDTe+M/0jyNaxtGI0IXjA=; b=hXoCBoB56E0bo6bfjpWsyuWmutS0g85eJBdgzUF670Ngha7uY31bAc2QFsWaUoltfs rzXh481hSYJvYAB/J+tyjrvQiZFAiPGmJTyb7nEMNXFZHF533cRUp/56QHAKGF8XOXWr JgJ+fCLROjSjVZ7ypWqL2sncIVJOM7AWw4D+Xd0mIplp+t/UDBUVjcwUNkFhxUJ7pSCb A+CbRuQuqWZ4Si6OTRqRumOdToDhjXam4DuTN9hb5udxo3XAvU6lwJjB5Jmfva5ZCVMH qRAWzjn0nJcoDvaVYzvxx2/6y1VbZ3VlgPwcht1BVSvev8Nxkit4uHCMOs8R4mnetq4O Vr2A== X-Gm-Message-State: ALoCoQlDJvVND8hW4qkOZCN5GVKuv8moGzBmAwt32+YuhnIpI+zdFAXHNebCxnFd5ZcVk4K3QN/EkI7j1PyJy9/bd4Rlwl0S+w== MIME-Version: 1.0 X-Received: by 10.202.199.82 with SMTP id x79mr27317866oif.139.1453302535757; Wed, 20 Jan 2016 07:08:55 -0800 (PST) Received: by 10.182.88.138 with HTTP; Wed, 20 Jan 2016 07:08:55 -0800 (PST) Date: Wed, 20 Jan 2016 13:08:55 -0200 Message-ID: Subject: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Eduardo Meyer To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 15:08:57 -0000 Hello all, I have some doubts regarding netmap design direct queue usage. If open netmap:ix0 I am opening all 0-7 queues. Are those queues FIFO among themselves? I mean first packeds will be available on netmap:ix0-0 and if this queue fills up the next packets will be on netmap:ix0-1, and via netmap:ix0 I have all queues from 0 to 7, is this understanding correct? Or something else happens, like, only 1 queue is used if I open netmap:ix0? A later question, if I open netmap:ix0-0 and nothing else is it supposed to work? On my tests I can see that "pkt-gen -f tx -i ix0-0" will work, but "pkt-gen -f rx -i ix0-0" will not. I can transmit but cant receive on a given queue. Why is that? And how could I make something like this, work (code change required?): bridge -i netmap:ix0-0 -i netmap:ix1-0 Or should it already work? Mr Pavel Odiltsov, the author from fastnetmon mentioned he can run on Linux and it works: "kipfw netmap:eth0-n netmap:eth1-n" But this or the above bridge example won't work on FreeBSD. Is that any different? (I did not try on Linux). I could also notice performance differences I would like to understand, if I run: pkt-gen -i ix0 -f tx -s 192.168.0.2 -d 192.168.0.1 I have 14.8Mpps (like rate). If I run: pkt-gen -i ix0-1 -f tx -s 192.168.0.2 -d 192.168.0.1 I can have only 11Mpps. In fact I have 11Mpps if I run on ix0-2, ix0-3, ... ix0-7. I can understand if I run all queues I can have better pps rates than only one single queue, sure, however if I run: pkt-gen -i ix0-0 -f tx -s 192.168.0.2 -d 192.168.0.1 I also have 14.8Mpps. So yeah, ix0 or ix0-0 both give me 14.8Mpps while any other queue give me 11Mpps. How should I understand this? I am asking this because I want to hack (for learning) into the bridge code to make it multithreaded, and I want to have a thread opening each one of the 8 available queues allocated on each one of my 8 CPUs. However both opening ix0-1 and ix1-1 on source code or as a parameter to the bridge application, I can't have it working. I also looked on "pkt-gen -p 8 -c 8" and although debug shows I have 8 threads on 8 CPU with 8 queues I still only see 1 thread working, all other threads go IDLE. I expected to see at least 2 threads working, say, for ix0-0 and ix0-1 to fill line rate. Thank you in advance. -- =========== Eduardo Meyer From owner-freebsd-net@freebsd.org Wed Jan 20 15:59:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64638A8A118 for ; Wed, 20 Jan 2016 15:59:28 +0000 (UTC) (envelope-from teleric-lists@outlook.com) Received: from BAY004-OMC3S15.hotmail.com (bay004-omc3s15.hotmail.com [65.54.190.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44E381975 for ; Wed, 20 Jan 2016 15:59:28 +0000 (UTC) (envelope-from teleric-lists@outlook.com) Received: from na01-by2-obe.outbound.protection.outlook.com ([65.54.190.189]) by BAY004-OMC3S15.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Wed, 20 Jan 2016 07:58:22 -0800 Received: from SN1PR18MB0509.namprd18.prod.outlook.com (10.163.226.16) by SN1PR18MB0510.namprd18.prod.outlook.com (10.163.226.17) with Microsoft SMTP Server (TLS) id 15.1.365.19; Wed, 20 Jan 2016 15:58:18 +0000 Received: from SN1PR18MB0509.namprd18.prod.outlook.com ([10.163.226.16]) by SN1PR18MB0509.namprd18.prod.outlook.com ([10.163.226.16]) with mapi id 15.01.0365.024; Wed, 20 Jan 2016 15:58:18 +0000 From: Teleric Team To: "freebsd-net@freebsd.org" Subject: Chelsio cxl and ncxl interface, whats the difference? Thread-Topic: Chelsio cxl and ncxl interface, whats the difference? Thread-Index: AQHRU5sp44Z5iOLtx06HbGf33LZONQ== Date: Wed, 20 Jan 2016 15:58:18 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=outlook.com; x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [t3W1ltnmJU+wnZlAyhJua028gnxVkZ++] x-microsoft-exchange-diagnostics: 1; SN1PR18MB0510; 23:yh2kaIlbikij5xfWSYNG8L5EdXT2pUNNnYeOMvgDuSnyCN/jq0ISJNub+9IK1Udx9U0sJrGsSUUhK4InkPxTZlj+krIl4RVBL1b5M8yrlzGb3Y/g9x2XoNwsQJEusF/1wM+f9hLYyAOoVxXCKgQz+lT7bU8eadCeC0+Kyo/PfFe05DMXN4EUPbAQS8rX+tItg9Se+1nkMH7kOmaGd9LzrA==; 5:MgMfz9UQR6VNqW8gegYDjUFXxY/dqiVI+vCsWA+KAC3vHOCu6otrzi+VxpBWfK0llwwYGKxNuDYuQcvwd4MYwFvUr+yeKb/kH4B/1zfHQXNPTJCiC6GKrPvovcrnETQFbSTDK1GecZYprt4hy+RC0Q==; 24:LnIwxJIjWwT7XRdEdhxuisEcipS6T5iZ11in+sh/Ve//lkSIbLjF8ggS7a6P/GqQ8kl9X1WUKuZ7cAePioXhyAfzZutJ101awP1BblEa/Qs= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR18MB0510; x-ms-office365-filtering-correlation-id: 65ac0192-8492-4356-9b20-08d321b283b1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015012)(82015046); SRVR:SN1PR18MB0510; BCL:0; PCL:0; RULEID:; SRVR:SN1PR18MB0510; x-forefront-prvs: 0827D7ACB9 x-forefront-antispam-report: SFV:NSPM; SFS:(7070004)(98900002); DIR:OUT; SFP:1901; SCL:1; SRVR:SN1PR18MB0510; H:SN1PR18MB0509.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jan 2016 15:58:18.5784 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR18MB0510 X-OriginalArrivalTime: 20 Jan 2016 15:58:22.0191 (UTC) FILETIME=[635387F0:01D1539B] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 15:59:28 -0000 I got a Chelsio T5 520-SO with two ports and I get 2 interfaces for it port= , cxl and ncxl (cxl0 ncxl0 cxl1 ncxl1). Man page mentions cxl is for T5, wh= at about ncxl? Should I get both or is something wrong? Which one should I = use? (is there any difference?). Thank you. From owner-freebsd-net@freebsd.org Wed Jan 20 16:48:16 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF8D3A89144 for ; Wed, 20 Jan 2016 16:48:16 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x236.google.com (mail-pa0-x236.google.com [IPv6:2607:f8b0:400e:c03::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0C1A1814 for ; Wed, 20 Jan 2016 16:48:16 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x236.google.com with SMTP id yy13so7222030pab.3 for ; Wed, 20 Jan 2016 08:48:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=uinjxlgZDDrHgmTfzJmH9sM6fXySjRoeNpDjLAMzSmI=; b=XqMDZyIt5u3aew8V1TMUZ7nw1TVMDzYImDBeG3jYXWGGqpkwOT923z/AkgOpPOfi1g /yohO/07hlorPcjnr2t+ZK8vONJo7f0cVbygeTP/v+z+955Q/PySalFKs3LPB4wHl7+c IPjXHU0biMep/5cdX4EWqmkEjniijZ0YqCtW/RiFLd8Lz+llcj+f0NwZ7FW/fe/wkFUU gk6bDMenC8QNfRRMNnFait4B+pYRZILlUu55/tx8HunMxLxK3Nu2/KLJFfAyp4U2u9Xx xnh+w2X4Yb9k3y910flz6J8m5UGIjryVMqOBt3RXbVOefzlB0SbCfR2hrGxyb70EbaV2 P/fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=uinjxlgZDDrHgmTfzJmH9sM6fXySjRoeNpDjLAMzSmI=; b=XOR9UJ2Qow+ZhQWOBvREZVLaXLBpWkrDtVmn5LBePf9VlW+prxZ5vw/kqT7r/PTM2R SFFFqWFTF1mb+K6aw2kxuAvYjuReb/GqALHG4fk2X03WccOdPcPxUAhBGMDuKAwISZh9 nP7BmjzuNjDMzG58waVwhe8sWOKmV6P9IYziJ0z1exp4J1IpGJ7SvwmWrQQUNac0vZ+I GeWstKkmaku+2mC17oja9AufgtcO3t2vpCVpAjtsfrMh2yVuDVsNWJDuL4MPEi9zbydX gZpA6EJhNLGoJ64rSfQie1PlDOjSPGHQIAQHRMQO9kcaI6/Bnb5hNzOOCJ7DoSULPjY6 HxSw== X-Gm-Message-State: ALoCoQlQ67/2oeHdk8nIyZT47db0UFqZBvIBRkvW4y+DVqptjIMB6Y2w6SM6k2CubUNTGab0A4csqkqwWQ0/OY78RwVnjwdzOA== X-Received: by 10.66.148.99 with SMTP id tr3mr53964122pab.19.1453308496360; Wed, 20 Jan 2016 08:48:16 -0800 (PST) Received: from ox ([2601:641:c001:8a00:acad:4200:4cdf:3770]) by smtp.gmail.com with ESMTPSA id ah10sm50312162pad.23.2016.01.20.08.48.14 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 20 Jan 2016 08:48:14 -0800 (PST) Date: Wed, 20 Jan 2016 08:48:05 -0800 From: Navdeep Parhar To: Teleric Team Cc: "freebsd-net@freebsd.org" Subject: Re: Chelsio cxl and ncxl interface, whats the difference? Message-ID: <20160120164805.GA21735@ox> Mail-Followup-To: Teleric Team , "freebsd-net@freebsd.org" References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 16:48:17 -0000 On Wed, Jan 20, 2016 at 03:58:18PM +0000, Teleric Team wrote: > I got a Chelsio T5 520-SO with two ports and I get 2 interfaces for it > port, cxl and ncxl (cxl0 ncxl0 cxl1 ncxl1). Man page mentions cxl is > for T5, what about ncxl? Should I get both or is something wrong? > Which one should I use? (is there any difference?). You should use the cxl interfaces. The 'n' interfaces are for netmap use and show up only if you have netmap support compiled into the kernel (which is the default for HEAD but not any stable/release branch). There is work in progress to move the 'n' interfaces to their own module so that they'll show up only if you load that extra module to get native netmap support for cxl/cxgbe. Regards, Navdeep From owner-freebsd-net@freebsd.org Wed Jan 20 18:46:43 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAB74A8A28B for ; Wed, 20 Jan 2016 18:46:42 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4C8F19FA for ; Wed, 20 Jan 2016 18:46:42 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: by mail-io0-x236.google.com with SMTP id g73so30056256ioe.3 for ; Wed, 20 Jan 2016 10:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vUx5L7ZidlXG7AAgjvdfrRMvmi+7Y+dWjBQDN6ESZOU=; b=kHIKpUNr9qV+AZeiX7BUxu2pZLwODBciTm2izX95/DkKWSleJXKRHCKvI53C7s5Pao z1I5zUo0OFAPJRmYaK9h5jFFUZjvY/AoEy9PC/ojVo/3Qc6tsaSdzbGHN85c6N+yCnGG Pnlx0abme2s9Tgo4bSXYTxzChi68Vu7rQxQ8WGye2TmKpUKguW8wwnt7gcC4BtdlhhwX ZtndTrr7UxNqFKE4CVq3TMFZOAGetdn9vw5DcpSbpt7NHOFApN0JjSvqx9/wQS2Rlflk 7h5qx3/Ck6mXk5ON6HH9pwbB7BLXAztbFkkOSC82pUPQVJ90s5onRlOYzqjaiYEBfE/O 8a4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vUx5L7ZidlXG7AAgjvdfrRMvmi+7Y+dWjBQDN6ESZOU=; b=Z1bh7yva7sBik2iBc+HLcYa6H/EuLOxTTGE2qlLU6FbL40opRMq8C+uOd4MM7hrYCN QTtqKFFvquCxAML/QeRwXp2KrgS0XuN3n/qjgqJRsN6cUxx2wxxwWUXMX+moB7pfgzrB pttxTFMMPAUb53I+h+YSIsZU5fCdU5rsIh6smaFD5dIkAHelMr2wdOvnY6wU1daewjyw GyscJU+FQsA93hMZ5gXhgNoyrCCGLSkassO4/XM00MQrn2osCAnPx4fsgsxBGBwFAKIr An0fxiVRGvVvDDuc5RC9VgpAM3Ib+yYkiSKXiPArhomct1S3Nw/4shSGwsYrLcitNuy4 WvtA== X-Gm-Message-State: ALoCoQkH8prU0DJjl6eqlv/EYn9NvDkCd9YnQvnubKvHwk+b0nnd0h/30B5yhfa3BCSomuCLQfZ21wNlm2+je0dt6BGPq9vHyQ== MIME-Version: 1.0 X-Received: by 10.107.16.95 with SMTP id y92mr37995071ioi.194.1453315602114; Wed, 20 Jan 2016 10:46:42 -0800 (PST) Received: by 10.79.36.65 with HTTP; Wed, 20 Jan 2016 10:46:42 -0800 (PST) In-Reply-To: References: Date: Wed, 20 Jan 2016 21:46:42 +0300 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Pavel Odintsov To: Eduardo Meyer Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 18:46:43 -0000 Hi Yes, this approach working really well on Linux. But I have never tried to do same on FreeBSD. I'm using similar approach in dastnetmon abd read data from the network card in X threads where each thread assigned to physical queue. So for Linux you should use my custom (based on Intel's drivers from Sourceforge with netmap patches) vbecause vanilla drivers haven't support for multi queue mode. On Wednesday, 20 January 2016, Eduardo Meyer wrote: > Hello all, > > I have some doubts regarding netmap design direct queue usage. > > If open netmap:ix0 I am opening all 0-7 queues. Are those queues FIFO among > themselves? I mean first packeds will be available on netmap:ix0-0 and if > this queue fills up the next packets will be on netmap:ix0-1, and via > netmap:ix0 I have all queues from 0 to 7, is this understanding correct? > > Or something else happens, like, only 1 queue is used if I open netmap:ix0? > > A later question, if I open netmap:ix0-0 and nothing else is it supposed to > work? > > On my tests I can see that "pkt-gen -f tx -i ix0-0" will work, but "pkt-gen > -f rx -i ix0-0" will not. I can transmit but cant receive on a given queue. > Why is that? And how could I make something like this, work (code change > required?): > > bridge -i netmap:ix0-0 -i netmap:ix1-0 > > Or should it already work? > > Mr Pavel Odiltsov, the author from fastnetmon mentioned he can run on Linux > and it works: > > "kipfw netmap:eth0-n netmap:eth1-n" > > But this or the above bridge example won't work on FreeBSD. Is that any > different? (I did not try on Linux). > > I could also notice performance differences I would like to understand, if > I run: > > pkt-gen -i ix0 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I have 14.8Mpps (like rate). > > If I run: > > pkt-gen -i ix0-1 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I can have only 11Mpps. In fact I have 11Mpps if I run on ix0-2, ix0-3, ... > ix0-7. > > I can understand if I run all queues I can have better pps rates than only > one single queue, sure, however if I run: > > pkt-gen -i ix0-0 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I also have 14.8Mpps. So yeah, ix0 or ix0-0 both give me 14.8Mpps while any > other queue give me 11Mpps. How should I understand this? > > I am asking this because I want to hack (for learning) into the bridge code > to make it multithreaded, and I want to have a thread opening each one of > the 8 available queues allocated on each one of my 8 CPUs. > > However both opening ix0-1 and ix1-1 on source code or as a parameter to > the bridge application, I can't have it working. > > I also looked on "pkt-gen -p 8 -c 8" and although debug shows I have 8 > threads on 8 CPU with 8 queues I still only see 1 thread working, all other > threads go IDLE. I expected to see at least 2 threads working, say, for > ix0-0 and ix0-1 to fill line rate. > > Thank you in advance. > > -- > =========== > Eduardo Meyer > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > -- Sincerely yours, Pavel Odintsov From owner-freebsd-net@freebsd.org Wed Jan 20 20:34:45 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F12AEA89E51 for ; Wed, 20 Jan 2016 20:34:45 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC76E163C for ; Wed, 20 Jan 2016 20:34:45 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22e.google.com with SMTP id h5so19802575igh.0 for ; Wed, 20 Jan 2016 12:34:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dB/l9aq/M0HFo9AOxXYYkvGyTLYwFVnQphhHaW5vOKg=; b=SWfAGSUbMvX32LqfjIhDULmFPBwhOo/gYPwsnvrMBhIZxjTvAinZLn5KQwuhX7PaqF xeD/S5M5TR6uV+7Jbhwvr6oCqVZzJKZKAOkHcw/S3eJR9EAJKZ/PrUt1vrdz39HbyTwG BF8cKehWm0+3cu+fkkRFr8P+BfUZ5FiW+qjrMh4/Rza0lKqtnPPHTzR7mPs+J1DPloR8 4SG+nDLfw5EHtmM36aUzoS7Yv+0uPkihVoskj6OjmpvtVsfaiYaFV+yTH/XvSID8NoNw NwhC95Ub2WYnPjmLEJEQ7+96GW39LNhrlg8nXCdwcFB+TlgWvYNIoZX/sw8KQVud1s+a fAQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=dB/l9aq/M0HFo9AOxXYYkvGyTLYwFVnQphhHaW5vOKg=; b=JtOg9YF6VyQkFuHXW0EHf+OGMTMl7cOoAp4JoysukK5hCA1LByIF8t9kQixudgYyag GSgzpiF4KASXanCx7C2/VQqhV3vLEQZnDDIu3A6nRGT+ZrI+3Q1OOED/D/cI77jR4Tx9 kv4jDjMgYNZT94qPI6qry+8hrcwjDNJs5sP8RKLVam+VR81SS1x7Vh04Sxwl00fu70wA zZNLkPstEczfjFOeZ5JPLVxobC3feJX8EWkQTd83cUCJLZ7xJN7BR/NEOtUMSdzyOLie SxP4VtEYv7r+Re4pmMtCXKWg2kKTnSrjp/LaO9sNjpTL1QG5RL1CnscUmqssDbiBl1sh 9mBw== X-Gm-Message-State: AG10YOTR1ymnl7pKKfe2uJIVrtKqc2s3VfRD8Pvjg3yszOfXVh/ISR3stL/lozI1806v1cqkWh9CPdDQGbvrzA== MIME-Version: 1.0 X-Received: by 10.50.150.36 with SMTP id uf4mr5240775igb.61.1453322085152; Wed, 20 Jan 2016 12:34:45 -0800 (PST) Received: by 10.36.121.16 with HTTP; Wed, 20 Jan 2016 12:34:45 -0800 (PST) In-Reply-To: References: Date: Wed, 20 Jan 2016 12:34:45 -0800 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Adrian Chadd To: Pavel Odintsov Cc: Eduardo Meyer , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 20:34:46 -0000 Ok, so, I mostly did this already: https://github.com/erikarn/netmap-tools/ it has a multi-threaded, multi-queue bridge + ipv4 decap for testing. -a From owner-freebsd-net@freebsd.org Wed Jan 20 22:18:11 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F11F7A89B84 for ; Wed, 20 Jan 2016 22:18:10 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD8A11162 for ; Wed, 20 Jan 2016 22:18:10 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.14.7/8.14.7) with ESMTP id u0KLx8Y0016239 for ; Wed, 20 Jan 2016 15:59:10 -0600 (CST) (envelope-from mgrooms@shrew.net) Received: from [10.16.32.30] (72-48-144-84.static.grandenetworks.net [72.48.144.84]) by mail.shrew.net (Postfix) with ESMTPSA id 2400018C780 for ; Wed, 20 Jan 2016 15:58:58 -0600 (CST) To: freebsd-net@freebsd.org From: Matthew Grooms Subject: pf state disappearing Message-ID: <56A003B8.9090104@shrew.net> Date: Wed, 20 Jan 2016 16:01:28 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx2.shrew.net [10.24.10.11]); Wed, 20 Jan 2016 15:59:10 -0600 (CST) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 22:18:11 -0000 All, I have a curious problem with a lightly loaded pair of pf firewall running on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from the state table for no good reason that I can see. The entry limit is set to 100000 and I never see the system go over about 70000 entries, so we shouldn't be hitting the configured limit ... # pfctl -sm states hard limit 100000 src-nodes hard limit 100000 frags hard limit 50000 table-entries hard limit 200000 # pfctl -si Status: Enabled for 78 days 14:24:18 Debug: Urgent State Table Total Rate current entries 67829 searches 113412118733 16700.2/s inserts 386313496 56.9/s removals 386245667 56.9/s Counters match 441731678 65.0/s bad-offset 0 0.0/s fragment 1090 0.0/s short 220 0.0/s normalize 761 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 4366487 0.6/s proto-cksum 0 0.0/s state-mismatch 50334 0.0/s state-insert 10 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s This problem is easy to reproduce by establishing an SSH connection to the firewall itself, letting it sit for a while and then examining the state table. After a connection is made, I can see the entry with an established:established state ... # pfctl -ss | grep X.X.X.X | grep 63446 all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED If I let the SSH session sit for a while and then try to type into the terminal on the client end, the connection stalls and produces a network error message. When I look at the pf state table again, the state entry for the connection is no longer visible. However, the ssh process is still running and I still see the TCP connection established in the output of netstat ... # netstat -na | grep 63446 tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED When I observe the packet flow in TCP dump when a connection stalls, packets being sent from the client are visible on the physical interface but are shown as blocked on the pflog0 interface. All this points to a state table entry being evicted from the state table for a healthy TCP connection, but I have no idea why. Is there a secondary resource limit I could be hitting that would cause the state entry to be removed? Maybe there was a bug has been fixed recently that would cause this behavior? I'd be very grateful for any input that would help me track down or resolve this problem. Thanks in advance, -Matthew From owner-freebsd-net@freebsd.org Thu Jan 21 05:43:45 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB878A8A1BA for ; Thu, 21 Jan 2016 05:43:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC64314C0 for ; Thu, 21 Jan 2016 05:43:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0L5hjhq017669 for ; Thu, 21 Jan 2016 05:43:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206456] pkt-gen unable work with three interfaces simultaneously in my freebsd system Date: Thu, 21 Jan 2016 05:43:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 05:43:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206456 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org --- Comment #1 from Mark Linimon --- Assign to net@, although it is not clear to me if this is a problem with the igb drivers, or the port. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Jan 21 09:35:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B79CA8920A for ; Thu, 21 Jan 2016 09:35:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 218ED1D41 for ; Thu, 21 Jan 2016 09:35:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1E963A89205; Thu, 21 Jan 2016 09:35:17 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04132A89202; Thu, 21 Jan 2016 09:35:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2AB21D40; Thu, 21 Jan 2016 09:35:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0L9Z99s097441 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 21 Jan 2016 11:35:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0L9Z99s097441 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0L9Z9Kr097440; Thu, 21 Jan 2016 11:35:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 21 Jan 2016 11:35:09 +0200 From: Konstantin Belousov To: Boris Astardzhiev Cc: Jilles Tjoelker , net@freebsd.org, threads@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? Message-ID: <20160121093509.GK3942@kib.kiev.ua> References: <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160120073154.GB3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 09:35:17 -0000 On Wed, Jan 20, 2016 at 10:29:47AM +0200, Boris Astardzhiev wrote: > Let me know the final decision then - whether in the existing manpages or > in new files. Decide it yourself, it is your patch. If you are fine with writing new man page, I do not object. > > jt>The Linux version has an additional parameter struct timespec *timeout > jt>(but only for recvmmsg, not for sendmmsg). Note that implementing this > jt>in a Linux-compatible manner has low overhead, since Linux only checks > jt>it between packets and never interrupts a wait because of this timeout > jt>(source: http://man7.org/linux/man-pages/man2/recvmmsg.2.html ). > > That's right. Shall I try to implement the timeout part or leave > it the way it is now? I do not see any sense in making the functions with signature or semantic different from Linux version. Right now, the goal of including the patch is compatibility. > > kb>Shouldn't i and rcvd be unsigned as well ? Shouldn't return value > kb>also be unsigned ? > I think i and rcvd should be unsigned whereas ret should not - after all > if an error occurred we get -1. I looked at the real signatures and man pages for the Linux functions, finally. There is indeed the timeout arg, the MSG_WAITFORONE flag for recvmmsg(3), and Linux uses ints for what would be naturally size_t. > kb>> + > kb>> + rcvd = 0; > kb>> + for (i = 0; i < vlen; i++) { > kb>> + errno = 0; > kb>> + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); > kb>> + if (ret < 0 || errno != 0) { > kb>I do not see why do you need to clear errno before, and then do this > test. > kb>Just check ret == -1, in which case errno was set from the immediate > syscall. > kb> > kb>> + if (rcvd != 0) { > kb>> + /* We've received messages. Let caller > know. */ > kb>> + errno = 0; > kb>This cleaning is not needed as well. For successfull functions returns, > kb>errno value is undefined. > > Wouldn't I confuse apps if they check errno in the follow case - I want to > receive two messages. The first __sys_recvmsg succeeds and then for the > second __sys_recvmsg fails. Thus errno will be != 0 and I'm telling the app > that I have received one message by returning 1 but errno will be != 0. > Is this correct? errno value is only defined after the function explicitely returned error. Apps which test for errno without testing for error are wrong. From owner-freebsd-net@freebsd.org Thu Jan 21 12:10:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F03BA8A50B for ; Thu, 21 Jan 2016 12:10:58 +0000 (UTC) (envelope-from bat@netadmin.hu) Received: from ns.netadmin.hu (ns.netadmin.hu [195.228.254.7]) by mx1.freebsd.org (Postfix) with SMTP id 783C019C6 for ; Thu, 21 Jan 2016 12:10:56 +0000 (UTC) (envelope-from bat@netadmin.hu) Received: (qmail 23082 invoked from network); 21 Jan 2016 12:10:54 -0000 Received: from localhost (HELO ?192.168.216.180?) (batppp@netadmin.hu@127.0.0.1) by ns.netadmin.hu with ESMTPA; 21 Jan 2016 12:10:54 -0000 From: Attila Balogh Subject: RB1100 network detection Date: Thu, 21 Jan 2016 13:10:24 +0100 Message-Id: <71D01AD4-CC62-4CA0-938B-743ED420D18F@netadmin.hu> Cc: ppc@freebsd.org To: freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 12:10:58 -0000 Hi Guys, i am trying to get FreeBSD running on Mikrotik RB1100. the box is pretty much straightforward, has a P2020E/P2010 SOC with 3 GE = controllers. two of the controllers has an AR8327 on them, the third one is connected = to an AR8035 rgmii phy. there are two PCIe GE controllers (AR8131). after some poking the box booted 10.3 prerelease via bootp, but failed = to find the phy and to enumerate the switches. but the switches are my least concern, first = get the box running freebsd properly. so the boot ended up in a kernel panic: transfer started ................................................... = transfer ok, time=3D2.09s setting up elf image... OK jumping to kernel code GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.3-PRERELEASE #0: Tue Jan 19 20:23:09 CET 2016 bat@ppcbuilder:/usr/obj/powerpc.powerpc/usr/src/sys/MPC85XX powerpc gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. cpu0: Freescale e500v2 core revision 5.1, 1066.65 MHz cpu0: Features 84000000 cpu0: HID0 80804000 real memory =3D 2080358400 (1983 MB) avail memory =3D 2040160256 (1945 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0: dev=3D0 (BSP) cpu1: dev=3D1 random device not loaded; using insecure entropy random: initialized ofwbus0: on nexus0 simplebus0: mem 0xe0000000-0xe00fffff = on ofwbus0 openpic0: mem 0x40000-0x7ffff on = simplebus0 uart0: <16550 or compatible> mem 0x4500-0x45ff irq 42 on simplebus0 uart0: console (115339,n,8,1) tsec0: mem 0x26000-0x26fff = irq 31,32,33 on simplebus0 mii_attach: invalid phyloc 33575 tsec0: attaching PHYs failed tsec0: could not be configured device_attach: tsec0 attach returned 6 tsec0: mem 0x24000-0x24fff = irq 29,30,34 on simplebus0 mii_attach: invalid phyloc 33575 tsec0: attaching PHYs failed tsec0: could not be configured device_attach: tsec0 attach returned 6 tsec0: mem 0x25000-0x25fff = irq 35,36,40 on simplebus0 miibus0: on tsec0 ukphy0: PHY 1 on miibus0 ukphy0: OUI 0x00c82e, model 0x0003, rev. 3 ukphy0: no media present ifmedia_set: no match for 0x0/0xfffffff panic: ifmedia_set cpuid =3D 0 KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at kdb_enter+0x60: addi r0, r0, 0x0 i did some changes to atphy and miidevs to get past the detection (just = added the model number). miibus0: on tsec2 atphy0: PHY 1 on miibus0 atphy0: OUI 0x00c82e, model 0x0003, rev. 3 atphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto tsec0: attaching PHYs OK! tsec0: Ethernet address: d4:ca:6d:33:57:90 tsec0: so far we cool tsec0: tx int ok tsec0: rx int ok tsec0: err int ok so at least i got rid of panic, properly retrieved mac address, but the = box does not see link. i=E2=80=99ll try to look into this and asked Pyun = who wrote the atphy part for help and advices. since bootp is wired to this ethernet controller, it does not want to = seek further. so i disabled it, and added alc to the kernel config. = although pci is also in the config, pci bus is not detected and enumerated, so no chance to = detect the pcie GEs. i need some kind of network support to able to connect to NFS to have a = root fs, so either way (pcie ge or rgmii on tsec) is a good fix for me. i tried to insert ramdisk into the kernel with some minimal tools to = look around. it gets to mountroot but fails to mount the ufs on md0 with = error 22. i am crosscompiling the stuff on 10.2-release on i386, and created the = disk image using the usual manual way. if someone is able to give a hint why this latter could fail, i=E2=80=99d = be very happy! kind regards, Attila block diagram: = http://www.cloudrouterswitches.com/images/Ethernet-Routers/RB1100AHx2-Bloc= k-Diagram.jpg = the kernel config: # Custom kernel for Freescale MPC85XX development boards like the CDS = etc. # # $FreeBSD: stable/10/sys/powerpc/conf/MPC85XX 266331 2014-05-17 = 17:34:37Z ian $ # cpu BOOKE cpu BOOKE_E500 ident MPC85XX machine powerpc powerpc makeoptions DEBUG=3D"-Wa,-me500 -g" makeoptions NO_MODULES=3Dyes options FPU_EMU options _KPOSIX_PRIORITY_SCHEDULING options ALT_BREAK_TO_DEBUGGER options BREAK_TO_DEBUGGER options BOOTP options BOOTP_NFSROOT options BOOTP_NFSV3 options BOOTP_WIRED_TO=3Dtsec0 options CD9660 options COMPAT_43 options DDB #options DEADLKRES options DEVICE_POLLING #options DIAGNOSTIC options FDT makeoptions FDT_DTS_FILE=3Dmpc8572ds.dts #makeoptions FDT_DTS_FILE=3Dmpc8555cds.dts options FFS options GDB options GEOM_PART_GPT options INET #options INET6 #options INVARIANTS #options INVARIANT_SUPPORT options KDB options KTRACE options MD_ROOT options MPC85XX options MSDOSFS options NFS_ROOT options NFSCL options NFSLOCKD options PROCFS options PSEUDOFS options SCHED_ULE options CAPABILITIES options CAPABILITY_MODE options SMP options SYSVMSG options SYSVSEM options SYSVSHM #options WITNESS #options WITNESS_SKIPSPIN device ata device bpf device cfi device crypto device cryptodev device da device ds1553 #device em device alc device ether #device fxp device iic device iicbus #device isa device loop device md device miibus device pass device pci #device quicc device random #device rl device scbus device scc #device sec device tsec device tun device uart options USB_DEBUG # enable debug msgs #device uhci #device umass #device usb #device vlan device nand device gpio #options ROOTDEVNAME=3D\"ufs:md0\" #options MD_ROOT_SIZE=3D8192 #makeoptions MFS_IMAGE=3D/test/root.img device mmc device mmcsd device sdhci device etherswitch device miiproxy device arswitch= From owner-freebsd-net@freebsd.org Thu Jan 21 13:27:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F15BDA8B76F for ; Thu, 21 Jan 2016 13:27:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DA7BA137D for ; Thu, 21 Jan 2016 13:27:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id D84AEA8B76D; Thu, 21 Jan 2016 13:27:27 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6C41A8B76C; Thu, 21 Jan 2016 13:27:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 989F1137C; Thu, 21 Jan 2016 13:27:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id C742B1044CE3; Fri, 22 Jan 2016 00:27:14 +1100 (AEDT) Date: Fri, 22 Jan 2016 00:27:14 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov cc: Boris Astardzhiev , threads@freebsd.org, Jilles Tjoelker , net@freebsd.org Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? In-Reply-To: <20160121093509.GK3942@kib.kiev.ua> Message-ID: <20160121233040.E1864@besplex.bde.org> References: <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160120073154.GB3942@kib.kiev.ua> <20160121093509.GK3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=cK4dyQqN c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=-W6hXtnpK6wFgIz_TycA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 13:27:28 -0000 On Thu, 21 Jan 2016, Konstantin Belousov wrote: > On Wed, Jan 20, 2016 at 10:29:47AM +0200, Boris Astardzhiev wrote: >> kb>Shouldn't i and rcvd be unsigned as well ? Shouldn't return value >> kb>also be unsigned ? >> I think i and rcvd should be unsigned whereas ret should not - after all >> if an error occurred we get -1. > I looked at the real signatures and man pages for the Linux functions, > finally. There is indeed the timeout arg, the MSG_WAITFORONE flag for > recvmmsg(3), and Linux uses ints for what would be naturally size_t. It knows better than POSIX, so has restored the v7 types :-). Probably not intentionally. FreeBSD-1 used the v7 types for recv* in FreeBSD-1, but this was changed in 4.4BSD-Lite1. >> kb>> + rcvd = 0; >> kb>> + for (i = 0; i < vlen; i++) { >> kb>> + errno = 0; >> kb>> + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); >> kb>> + if (ret < 0 || errno != 0) { >> kb>I do not see why do you need to clear errno before, and then do this >> test. >> kb>Just check ret == -1, in which case errno was set from the immediate >> syscall. >> kb> >> kb>> + if (rcvd != 0) { >> kb>> + /* We've received messages. Let caller >> know. */ >> kb>> + errno = 0; >> kb>This cleaning is not needed as well. For successfull functions returns, >> kb>errno value is undefined. >> >> Wouldn't I confuse apps if they check errno in the follow case - I want to >> receive two messages. The first __sys_recvmsg succeeds and then for the >> second __sys_recvmsg fails. Thus errno will be != 0 and I'm telling the app >> that I have received one message by returning 1 but errno will be != 0. >> Is this correct? > > errno value is only defined after the function explicitely returned error. > Apps which test for errno without testing for error are wrong. Mostly such tests find wrong implementations. Only low quality implmentations set error for non-errors. I checked some details: - both C99 and POSIX.1-2001 forbid setting errno to 0 in library functions - C99 and draft C11 explicitly say that errno may be clobbered for non-errors by functions that are not documented to set errno. It intends to also say that errno may not be clobbered for non-errors by functions are documented to set errno, but actually says nothing about this case (except in a footnote, it says "Thus [the usual method of setting errno before the call and checking it after ``should'' be used to check for errors]." Footnotes are not part of the standard, so this also says nothing. But it gives the intent. - C99 doesn't say anything specific about this for strtol. But strtol needs errno to not be set on non-error for its API. This gives the intent of the documentation for errno in the standard itself. - POSIX.1-2001 says that applications "should" only be examined when it is indicated to be valid by a function's return value. This is wrong for all functions where errno is part of the API (like strtol), and it would be inconsistent with C99 and C11 if these standards said what they intend to say, for many more functions. First, all of the C99 functions that are documented to set errno. Then, if POSIX follows the spirit of the intent of C99, then just about all POSIX functions must not clobber errno for non-errors, since (unlike in C99) just about all functions in POSIX are documented to set errno. - POSIX.1-2001 apparently agrees with me that C99 doesn't say what it intends to say. It explicitly says that strtol shall not change "t setting of" errno on success, and marks this as an extension of C99. - I couldn't find even a buggy generic clause in POSIX.1-2001 saying what C99 intends to say even for the C99 parts of POSIX. If there were such a clause, then strtol wouldn't need a special extension. This means that some other C99 functions need special clauses, and there are likely to be bugs in this since some don't really need them except to conform with C99's intentions. - not many C99 functions are specified to set errno. The main ones are the strtol family, math functions (most can set ERANGE or EDOM), and wide character functions (many can set EILSEQ). POSIX.1-2001 seems to be missing a special clause for the first wide character function that I looked at -- fputwc(). For math functions, it has special statements ad nauseum (about 20 lines per function for duplicated for 20-50 functions). Bruce From owner-freebsd-net@freebsd.org Thu Jan 21 17:04:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C565A8CDAF for ; Thu, 21 Jan 2016 17:04:28 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C795A13CE for ; Thu, 21 Jan 2016 17:04:27 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id o124so30226100oia.3 for ; Thu, 21 Jan 2016 09:04:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kCIiy9Y9OBLldJAslUowB1x6CtdBhXP5KQsscw35dLM=; b=gyi4lq6iNa3+yStm4iXYVsAEft/6TUfDhe2YsRXZUA1rBfvSE76dzC2FnW4niHrSfv b3jMbcl2IgpoUF/QzqCjUQjQy6mli6VNGDLZU0HiuV9aOR39pX28Z9b2sGS6wq09ToWD WLjz5Y9LeKaw5Vf3CqOjX5KD8HwBeSRb7KGEkPXQqJfIPkROUsTg7kR9nq98HE8dQVtk HXMFtTl4hsbddDrgvBBwhN1B4YpdzelAt0xc1zd+fqgGpzqyWsMVx5aO8UVSwa3aBo+9 FlJG8hK1JHwmF7EgP3p6NJuMXCB0bLGK1e9jCPGp9lxr9y5+5VrL26SKmdoA4NqEzT8M 8xkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=kCIiy9Y9OBLldJAslUowB1x6CtdBhXP5KQsscw35dLM=; b=fWaVQi6Jbytt/Ytcw5WA8EZpuyi9IbKtCsLY9VEsOWi1nhEBAdamu0IjegjcMcNsDw BfyX1zngO2+D471tq0EIcv6qK0dwKdUmUgC2llpzR0jGMn/quFDiYLtCeTHNscdGhTSL Q86IuMUgkrl7YZJhIcxBKsa/0qV79jsIxbS+LOlbfzlbukcA8HDuxeYp9wpf/9Yfvk/4 kIvnCi/1A+A378EdLNBaxqVNq0RZKdphQ6K+NGJHIYlRQK7IZrrrJofEWUXOg82fJBcF LG7edtGK3HPhTQdwYIVikZBR8Lq2MtA1TYscba+wUYlaA2CgLRcTp8wct2xN0JJitFcr mP+g== X-Gm-Message-State: ALoCoQlfz29+TaRM16ot4+/1a20sNLAXT78G59D8RdYZcFtMcylVf9+Y2Q5IyS25E95U2ElxxiLWH7QBBL6Lx7lxCm56qc57Bg== MIME-Version: 1.0 X-Received: by 10.202.217.5 with SMTP id q5mr29845360oig.29.1453395867152; Thu, 21 Jan 2016 09:04:27 -0800 (PST) Received: by 10.202.242.132 with HTTP; Thu, 21 Jan 2016 09:04:27 -0800 (PST) In-Reply-To: <56A003B8.9090104@shrew.net> References: <56A003B8.9090104@shrew.net> Date: Thu, 21 Jan 2016 09:04:27 -0800 Message-ID: Subject: Re: pf state disappearing From: Nick Rogers To: Matthew Grooms Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 17:04:28 -0000 On Wed, Jan 20, 2016 at 2:01 PM, Matthew Grooms wrote: > All, > > I have a curious problem with a lightly loaded pair of pf firewall running > on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from > the state table for no good reason that I can see. The entry limit is set > to 100000 and I never see the system go over about 70000 entries, so we > shouldn't be hitting the configured limit ... > In my experience if you hit the state limit, new connections/states are dropped and existing states are unaffected. > > # pfctl -sm > states hard limit 100000 > src-nodes hard limit 100000 > frags hard limit 50000 > table-entries hard limit 200000 > > # pfctl -si > Status: Enabled for 78 days 14:24:18 Debug: Urgent > > State Table Total Rate > current entries 67829 > searches 113412118733 16700.2/s > inserts 386313496 56.9/s > removals 386245667 56.9/s > Counters > match 441731678 65.0/s > bad-offset 0 0.0/s > fragment 1090 0.0/s > short 220 0.0/s > normalize 761 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 4366487 0.6/s > proto-cksum 0 0.0/s > state-mismatch 50334 0.0/s > state-insert 10 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > This problem is easy to reproduce by establishing an SSH connection to the > firewall itself, letting it sit for a while and then examining the state > table. After a connection is made, I can see the entry with an > established:established state ... > > # pfctl -ss | grep X.X.X.X | grep 63446 > all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED > > If I let the SSH session sit for a while and then try to type into the > terminal on the client end, the connection stalls and produces a network > error message. When I look at the pf state table again, the state entry for > the connection is no longer visible. However, the ssh process is still > running and I still see the TCP connection established in the output of > netstat ... > > # netstat -na | grep 63446 > tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED > > When I observe the packet flow in TCP dump when a connection stalls, > packets being sent from the client are visible on the physical interface > but are shown as blocked on the pflog0 interface. > Does this happen with non-SSH connections? It sounds like your SSH client/server interaction is not performing a keep-alive frequently enough to keep the PF state established. If no packets are sent over the connection (state) for some time, then PF will timeout (remove) the state. At this point your SSH client still believes it has a successful connection, so it tries to send packets when you resume typing, but they are blocked by your PF rules which likely specify "flags S/SA keep state", either explicitly or implicitly (it is the filter rule default), which means block packets that don't match an existing state or are not part of the initial SYN handshake of the TCP connection. Look at your settings in pf.conf for "timeout tcp.established", which affects how long before an idle ESTABLISHED state will timeout. Also look into ClientAliveInterval in sshd configuration, which I believe is 0 (disabled) by default, which means it will let the client timeout without sending a keep-alive. If you don't want PF to force timeout an idle SSH connection, then ideally ClientAliveInterval is less than or equal (i.e., more-frequent) to PF's tcp.established timeout value. > All this points to a state table entry being evicted from the state table > for a healthy TCP connection, but I have no idea why. Is there a secondary > resource limit I could be hitting that would cause the state entry to be > removed? Maybe there was a bug has been fixed recently that would cause > this behavior? I'd be very grateful for any input that would help me track > down or resolve this problem. > > Thanks in advance, > > -Matthew > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Jan 21 19:45:12 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B3EAA8C926 for ; Thu, 21 Jan 2016 19:45:12 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEB771E57 for ; Thu, 21 Jan 2016 19:45:11 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.14.7/8.14.7) with ESMTP id u0LJgO5n096209 for ; Thu, 21 Jan 2016 13:42:24 -0600 (CST) (envelope-from mgrooms@shrew.net) Received: from [10.16.32.30] (72-48-144-84.static.grandenetworks.net [72.48.144.84]) by mail.shrew.net (Postfix) with ESMTPSA id 4776018C688 for ; Thu, 21 Jan 2016 13:42:19 -0600 (CST) From: Matthew Grooms Subject: Re: pf state disappearing [ adaptive timeout bug ] To: freebsd-net@freebsd.org References: <56A003B8.9090104@shrew.net> Message-ID: <56A13531.8090209@shrew.net> Date: Thu, 21 Jan 2016 13:44:49 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx1.shrew.net [10.24.10.10]); Thu, 21 Jan 2016 13:42:24 -0600 (CST) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 19:45:12 -0000 On 1/21/2016 11:04 AM, Nick Rogers wrote: > On Wed, Jan 20, 2016 at 2:01 PM, Matthew Grooms wrote: > >> All, >> >> I have a curious problem with a lightly loaded pair of pf firewall running >> on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from >> the state table for no good reason that I can see. The entry limit is set >> to 100000 and I never see the system go over about 70000 entries, so we >> shouldn't be hitting the configured limit ... >> > In my experience if you hit the state limit, new connections/states are > dropped and existing states are unaffected. Aha! You shook something out of the dusty depths of my slow brain :) I believe that what you say is true as long as adaptive timeouts are disabled, which by default they are not ... Timeout values can be reduced adaptively as the number of state ta- ble entries grows. adaptive.start When the number of state entries exceeds this value, adaptive scaling begins. All timeout values are scaled linearly with factor (adaptive.end - number of states) / (adaptive.end - adaptive.start). adaptive.end When reaching this number of state entries, all timeout val- ues become zero, effectively purging all state entries imme- diately. This value is used to define the scale factor, it should not actually be reached (set a lower state limit, see below). Adaptive timeouts are enabled by default, with an adaptive.start value equal to 60% of the state limit, and an adaptive.end value equal to 120% of the state limit. They can be disabled by setting both adaptive.start and adaptive.end to 0. >> # pfctl -sm >> states hard limit 100000 >> src-nodes hard limit 100000 >> frags hard limit 50000 >> table-entries hard limit 200000 >> >> # pfctl -si >> Status: Enabled for 78 days 14:24:18 Debug: Urgent >> >> State Table Total Rate >> current entries 67829 >> searches 113412118733 16700.2/s >> inserts 386313496 56.9/s >> removals 386245667 56.9/s >> Counters >> match 441731678 65.0/s >> bad-offset 0 0.0/s >> fragment 1090 0.0/s >> short 220 0.0/s >> normalize 761 0.0/s >> memory 0 0.0/s >> bad-timestamp 0 0.0/s >> congestion 0 0.0/s >> ip-option 4366487 0.6/s >> proto-cksum 0 0.0/s >> state-mismatch 50334 0.0/s >> state-insert 10 0.0/s >> state-limit 0 0.0/s >> src-limit 0 0.0/s >> synproxy 0 0.0/s >> >> This problem is easy to reproduce by establishing an SSH connection to the >> firewall itself, letting it sit for a while and then examining the state >> table. After a connection is made, I can see the entry with an >> established:established state ... >> >> # pfctl -ss | grep X.X.X.X | grep 63446 >> all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED >> >> If I let the SSH session sit for a while and then try to type into the >> terminal on the client end, the connection stalls and produces a network >> error message. When I look at the pf state table again, the state entry for >> the connection is no longer visible. However, the ssh process is still >> running and I still see the TCP connection established in the output of >> netstat ... >> >> # netstat -na | grep 63446 >> tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED >> >> When I observe the packet flow in TCP dump when a connection stalls, >> packets being sent from the client are visible on the physical interface >> but are shown as blocked on the pflog0 interface. >> > Does this happen with non-SSH connections? It sounds like your SSH > client/server interaction is not performing a keep-alive frequently enough > to keep the PF state established. If no packets are sent over the > connection (state) for some time, then PF will timeout (remove) the state. > At this point your SSH client still believes it has a successful > connection, so it tries to send packets when you resume typing, but they > are blocked by your PF rules which likely specify "flags S/SA keep state", > either explicitly or implicitly (it is the filter rule default), which > means block packets that don't match an existing state or are not part of > the initial SYN handshake of the TCP connection. It happened with UDP SIP and log running HTTP sessions that sit idle as well. The SSH connection was just the easiest to test. Besides that, the default TCP timeout value for established connections is quite high at 86400s. An established TCP connection should be able to sit for a full day with no traffic before the related state table entry gets evicted. > Look at your settings in pf.conf for "timeout tcp.established", which > affects how long before an idle ESTABLISHED state will timeout. Also look > into ClientAliveInterval in sshd configuration, which I believe is 0 > (disabled) by default, which means it will let the client timeout without > sending a keep-alive. If you don't want PF to force timeout an idle SSH > connection, then ideally ClientAliveInterval is less than or equal (i.e., > more-frequent) to PF's tcp.established timeout value. Thanks for the suggestion! I completely forgot about the adaptive timeout options until I double checked the settings based on you reply :) My values are set to default for TCP and extended a bit for UDP. The adaptive.start value was calculated at 60k for the 100k state limit. That in particular looked way too relevant to be a coincidence. After increasing the value to 90k, my total state count started increasing and leveled out around 75k. It's always hovered around 65k up until now, so 10k sate entries were being discarded on a regular basis ... # pfctl -si Status: Enabled for 0 days 02:25:41 Debug: Urgent State Table Total Rate current entries 77759 searches 483831701 55352.0/s inserts 825821 94.5/s removals 748060 85.6/s Counters match 27118754 3102.5/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 6655 0.8/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -st tcp.first 120s tcp.opening 30s tcp.established 86400s tcp.closing 900s tcp.finwait 45s tcp.closed 90s tcp.tsdiff 30s udp.first 600s udp.single 600s udp.multiple 900s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start 90000 states adaptive.end 120000 states src.track 0s I think there may be a problem with the code that calculates adaptive timeout values that is making it way too aggressive. If by default it's supposed to decrease linearly between %60 and %120 of the state table max, I shouldn't be loosing TCP connections that are only idle for a few minutes when the sate table is < %70 full. Unfortunately that appears to be the case. At most this should have decreased the 86400s timeout by %17 to 72000s for established TCP connections. I've tested this for a few hours now and all my idle SSH sessions have been rock solid. If anyone else is scratching their head over a problem like this, I would suggest disabling the adaptive timeout feature or increasing it to a much higher value. Maybe one of the pf maintainers can chime in and shed some light on why this is happening. If not, I'm going to file a bug report as this certainly feels like one. Thanks again, -Matthew From owner-freebsd-net@freebsd.org Thu Jan 21 23:41:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8222A8A130; Thu, 21 Jan 2016 23:41:23 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward8j.cmail.yandex.net (forward8j.cmail.yandex.net [IPv6:2a02:6b8:0:1630::183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 60789174D; Thu, 21 Jan 2016 23:41:22 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from web14j.yandex.ru (web14j.yandex.ru [IPv6:2a02:6b8:0:1619::314]) by forward8j.cmail.yandex.net (Yandex) with ESMTP id CB95421BBD; Fri, 22 Jan 2016 02:41:07 +0300 (MSK) Received: from 127.0.0.1 (localhost [127.0.0.1]) by web14j.yandex.ru (Yandex) with ESMTP id E81CF28A23C8; Fri, 22 Jan 2016 02:41:06 +0300 (MSK) Received: by web14j.yandex.ru with HTTP; Fri, 22 Jan 2016 02:41:03 +0300 From: Alexander V. Chernikov Envelope-From: melifaro@ipfw.ru To: FreeBSD Net , freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org Subject: projects/routing announcement/status MIME-Version: 1.0 Message-Id: <6151261453419663@web14j.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Fri, 22 Jan 2016 02:41:03 +0300 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 23:41:24 -0000 I would like to introduce routing rework which started as projects/routing SVN branch. It has been around for quite a long time, some of the code has made its way to HEAD, but there hasn't been any public announcements. So, what is projects/routing about? First, it is about bringing more scalability by solving most annoying problems on packet output path. To be more specific, it eliminates 2 out of 4 locks, converts other 2 to rmlock(9) and adds infrastructure to reduce locking to single rmlock for certain traffic types. With these changes, OS is able to forward 12MPPS on 16-core box for both IPv4/IPv6 which is 6-10 times better than stock HEAD. Second, it eases hacking by avoiding direct access to route/lltable internals and providing higher level API instead. Third, it is about bringing advanced features like route multipath, and even more speed by adding modular lookup API permitting to use different route lookup algorithms based on server role. Description with graphs and links is available at: http://wiki.freebsd.org/ProjectsRoutingProposal Used API is described in http://wiki.freebsd.org/ProjectsRoutingProposal/API Current status is available at http://wiki.freebsd.org/ProjectsRoutingProposal/ConversionStatus It is probably much more convenient to read project details on wiki, however I’ll try to summarise the most important things here (wiki readers can skip till the end). Typical packet processing (forwarding for router, or output for web server) path consists of: doing routing lookup (radix read rwlock + routing entry (rte) mutex lock) (optionally) interface address (ifa) atomic refcount acquire/release doing link level entry (lle, llentry) lookup (afdata read rwlock + llentry read (or write) lock) Most annoying one is the rtentry mutex. The only goal of this mutex is to provide rtentry refcounting so consumer code can use it without the risk of rtentrry being deleted. We solve this by saving all needed data into on-stack optimised structure instead of refcounting. Additionally, we are trying to pre-calculate the data we need to pass by using special next-hop structures instead of route entries. Several different (in terms of returned info and relative overhead) functions for retrieving routing data are provided. Most of the consumers have already been switched to the new KPI. Actual output/forward path are not converted yet. It should be noted, that since individual rtentries are not returned, it is not possible to do per-ifa output packet accouting (can be observed in netstat -s). Route table lock is switched to ipfw-like dual-locking mode (read rmlock() for data path, rwlock for config changes, route export, etc..). The reasons of having rwlock are to 1) provide serialization for things in control plane not directly used for data path and 2) avoid acquiring contested/sleeping locks for rmlock. See projects/routing r287078 for an example. Lltable entry locks were eliminated in r291853, r292155. Lltable lock is also planned to be converted to dual-locking model, with the similar reasoning. However, instead of (ab)using AFDATA lock, it needs to be converted to per-lltable set of locks. Open problems: SCTP/Flowtable references rtentries directly. It is not possible to convert ip[6]_output() path without dealing with that. Brief merge plan: Discuss/merge new routing KPI for data path Discuss/merge lltable dual-lock (WIP) Discuss/merge explicit nexthop changes Discuss/merge IPv4/IPv6 output path (along with converted sctp/flowtable) Discuss/merge route table dual-lock Current outstanding reviews (I encourage you to take a look at these) D5009 (IPv4 fast forwarding conversion) D5010 (IPv6 forwarding conversion) D4794 (Deal with per-ifa output counters) D4962 (new LLE lookup functions, no sockaddrs in lltable data path) D4751 (move all lltable code to separate files) From owner-freebsd-net@freebsd.org Fri Jan 22 01:02:37 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22ED0A8C634 for ; Fri, 22 Jan 2016 01:02:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12A6615C3 for ; Fri, 22 Jan 2016 01:02:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0M12ah4075374 for ; Fri, 22 Jan 2016 01:02:36 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206478] Setting laggproto fails on 10.2 Date: Fri, 22 Jan 2016 01:02:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 01:02:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206478 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |regression Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jan 22 08:35:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C323A8D1A5 for ; Fri, 22 Jan 2016 08:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DCE012C9 for ; Fri, 22 Jan 2016 08:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0M8ZR00079939 for ; Fri, 22 Jan 2016 08:35:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 196252] [patch] show tcp hostcache usage in netstat -s -p tcp Date: Fri, 22 Jan 2016 08:35:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: araujo@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: araujo@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 08:35:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D196252 Marcelo Araujo changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-net@FreeBSD.org |araujo@FreeBSD.org CC| |araujo@FreeBSD.org --- Comment #1 from Marcelo Araujo --- I'm taking it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jan 22 09:15:20 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA4E4A8A3F0 for ; Fri, 22 Jan 2016 09:15:20 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B53A51C98 for ; Fri, 22 Jan 2016 09:15:20 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B123CA8A3ED; Fri, 22 Jan 2016 09:15:20 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9698DA8A3EB; Fri, 22 Jan 2016 09:15:20 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06F7A1C96; Fri, 22 Jan 2016 09:15:20 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id u188so10684798wmu.1; Fri, 22 Jan 2016 01:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=auR0WeQdkc1zTo965Qk2tdsr9vGRrHhKpeD3uXFHyNg=; b=rf/AYq6b4exhWxh7Tgs9TweWXRVlKc1BAfi4iBd8qHfhfdBZ7wHfW61U2HWW8Qjvca RJz7Y5+RHReXXOh8eA747cAzupR75Je+51Uzo5AqQ8OMDHrK4zUa5pJvlo8g5E/Uykck +nJIeMvJpvBgTGhiWltdR2QvbMttPGK6P2BPirPtl83LUXrth2akvPuhup4FKs68bZ3N O3MGXfJZv+FSlZX5dQvI7v6t/SheF01N8T70ChYckdOKDYEh4Xgd6Gbtf0LPKQqr0QaX e2qlkNizTucnf2E2bY6+/fR6R7UangP7Bz6t/ozkIDDV+0kJmSpq09m5B2ISZxhnm0Up 6ebQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=auR0WeQdkc1zTo965Qk2tdsr9vGRrHhKpeD3uXFHyNg=; b=A9yZcRoHu2AP1ajNU7/0iLdGPb9B5GVWS1HNcD0ziCmxJdvvbe/usmPyaOkHrVY/6n O/FB8d1q0eAqhnl4c0A8+hr8ins/a6f5uMrTpmYpqNZsKsen6ZJ2ndJDAnqVaXaJd3Ht d3/9ViysVm4NK93MKsRE0j8OiEqv1bX3x7vXv1VEYgc81Pxg3NYotNqwD/HwsoNGFig+ C9NrnBKX4RCV+rwfM5l5Z58SDdyPHwXOhgT0VKsqRIuJF3dq6VU3qJDZ2Tlt/s/BbB+T eJqj05xldPpkbK16MDmJP3Uw043MSNWnuKlG+DIuY9wWXgs9E7+Tko7QtCnudVZJcjA0 Uhzg== X-Gm-Message-State: AG10YOQxTYx+B9QkRN+tNzqfVUbk3zELDxhs1PImF7DJFhyaHeVHMATWFNJURKiGiWqWilwXvkpszlzvI2Wu5w== MIME-Version: 1.0 X-Received: by 10.28.216.134 with SMTP id p128mr2156488wmg.59.1453454118400; Fri, 22 Jan 2016 01:15:18 -0800 (PST) Received: by 10.28.136.148 with HTTP; Fri, 22 Jan 2016 01:15:18 -0800 (PST) In-Reply-To: <20160121233040.E1864@besplex.bde.org> References: <20160116195657.GJ3942@kib.kiev.ua> <20160116202534.GK3942@kib.kiev.ua> <20160117211853.GA37847@stack.nl> <20160118044826.GS3942@kib.kiev.ua> <20160118140811.GW3942@kib.kiev.ua> <20160120073154.GB3942@kib.kiev.ua> <20160121093509.GK3942@kib.kiev.ua> <20160121233040.E1864@besplex.bde.org> Date: Fri, 22 Jan 2016 11:15:18 +0200 Message-ID: Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls? From: Boris Astardzhiev To: Bruce Evans Cc: Konstantin Belousov , threads@freebsd.org, Jilles Tjoelker , net@freebsd.org Content-Type: multipart/mixed; boundary=001a114715826cd8660529e8a863 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 09:15:21 -0000 --001a114715826cd8660529e8a863 Content-Type: text/plain; charset=UTF-8 be>None of the above. Plain recvmsg() returns ssize_t and its len arg has be>type size_t. That is excessively typedefed and excessively large with be>64-bit ssize_t, but it is silly for the multiple-message variant to use be>smaller types. be> be>Otherwise, all the integer types should be int. It seems logical. I'll convert the ret easily to ssize_t and the vector length to size_t. Now it differs from the Linux prototype but I guess it's okay. be>The errno method (and not checking ret at all) is best if for syscalls that be>return -1 for a non-error. It is not needed here. Fixing it. kb> I do not see any sense in making the functions with signature or semantic kb> different from Linux version. Right now, the goal of including the patch kb> is compatibility. Regarding recvmmsg() - I tried to implement MSG_WAITFORONE and the timeout stuff using pselect(2) due to the timespec structure. I could have used ppoll and I'm not sure which of these two is more appropriate or maybe there's another approach? Now it has timeout just as in the Linux prototype. Comments are welcomed. See patch. Best regards, Boris Astardzhiev On Thu, Jan 21, 2016 at 3:27 PM, Bruce Evans wrote: > On Thu, 21 Jan 2016, Konstantin Belousov wrote: > > On Wed, Jan 20, 2016 at 10:29:47AM +0200, Boris Astardzhiev wrote: >> > > kb>Shouldn't i and rcvd be unsigned as well ? Shouldn't return value >>> kb>also be unsigned ? >>> I think i and rcvd should be unsigned whereas ret should not - after all >>> if an error occurred we get -1. >>> >> I looked at the real signatures and man pages for the Linux functions, >> finally. There is indeed the timeout arg, the MSG_WAITFORONE flag for >> recvmmsg(3), and Linux uses ints for what would be naturally size_t. >> > > It knows better than POSIX, so has restored the v7 types :-). Probably > not intentionally. > > FreeBSD-1 used the v7 types for recv* in FreeBSD-1, but this was changed > in 4.4BSD-Lite1. > > kb>> + rcvd = 0; >>> kb>> + for (i = 0; i < vlen; i++) { >>> kb>> + errno = 0; >>> kb>> + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); >>> kb>> + if (ret < 0 || errno != 0) { >>> kb>I do not see why do you need to clear errno before, and then do this >>> test. >>> kb>Just check ret == -1, in which case errno was set from the immediate >>> syscall. >>> kb> >>> kb>> + if (rcvd != 0) { >>> kb>> + /* We've received messages. Let caller >>> know. */ >>> kb>> + errno = 0; >>> kb>This cleaning is not needed as well. For successfull functions >>> returns, >>> kb>errno value is undefined. >>> >>> Wouldn't I confuse apps if they check errno in the follow case - I want >>> to >>> receive two messages. The first __sys_recvmsg succeeds and then for the >>> second __sys_recvmsg fails. Thus errno will be != 0 and I'm telling the >>> app >>> that I have received one message by returning 1 but errno will be != 0. >>> Is this correct? >>> >> >> errno value is only defined after the function explicitely returned error. >> Apps which test for errno without testing for error are wrong. >> > > Mostly such tests find wrong implementations. Only low quality > implmentations set error for non-errors. > > I checked some details: > > - both C99 and POSIX.1-2001 forbid setting errno to 0 in library functions > > - C99 and draft C11 explicitly say that errno may be clobbered for > non-errors by functions that are not documented to set errno. It > intends to also say that errno may not be clobbered for non-errors by > functions are documented to set errno, but actually says nothing > about this case (except in a footnote, it says "Thus [the usual > method of setting errno before the call and checking it after > ``should'' be used to check for errors]." Footnotes are not part > of the standard, so this also says nothing. But it gives the intent. > > - C99 doesn't say anything specific about this for strtol. But strtol > needs errno to not be set on non-error for its API. This gives the > intent of the documentation for errno in the standard itself. > > - POSIX.1-2001 says that applications "should" only be examined when it is > indicated to be valid by a function's return value. This is wrong for > all functions where errno is part of the API (like strtol), and it > would be inconsistent with C99 and C11 if these standards said what they > intend to say, for many more functions. First, all of the C99 functions > that are documented to set errno. Then, if POSIX follows the spirit of > the intent of C99, then just about all POSIX functions must not clobber > errno for non-errors, since (unlike in C99) just about all functions in > POSIX are documented to set errno. > > - POSIX.1-2001 apparently agrees with me that C99 doesn't say what it > intends to say. It explicitly says that strtol shall not change "t > setting of" errno on success, and marks this as an extension of C99. > > - I couldn't find even a buggy generic clause in POSIX.1-2001 saying > what C99 intends to say even for the C99 parts of POSIX. If there > were such a clause, then strtol wouldn't need a special extension. > This means that some other C99 functions need special clauses, and > there are likely to be bugs in this since some don't really need > them except to conform with C99's intentions. > > - not many C99 functions are specified to set errno. The main ones > are the strtol family, math functions (most can set ERANGE or EDOM), > and wide character functions (many can set EILSEQ). POSIX.1-2001 > seems to be missing a special clause for the first wide character > function that I looked at -- fputwc(). For math functions, it has > special statements ad nauseum (about 20 lines per function for > duplicated for 20-50 functions). > > Bruce > --001a114715826cd8660529e8a863 Content-Type: text/plain; charset=US-ASCII; name="sendrecvmmsg-libconly4.diff" Content-Disposition: attachment; filename="sendrecvmmsg-libconly4.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ijpgx4140 ZGlmZiAtLWdpdCBhL2xpYi9saWJjL2luY2x1ZGUvbmFtZXNwYWNlLmggYi9saWIvbGliYy9pbmNs dWRlL25hbWVzcGFjZS5oCmluZGV4IDczOWQ3YjEuLmM5NTgyOWUgMTAwNjQ0Ci0tLSBhL2xpYi9s aWJjL2luY2x1ZGUvbmFtZXNwYWNlLmgKKysrIGIvbGliL2xpYmMvaW5jbHVkZS9uYW1lc3BhY2Uu aApAQCAtMjA4LDYgKzIwOCw3IEBACiAjZGVmaW5lCQlyZWFkdgkJCQlfcmVhZHYKICNkZWZpbmUJ CXJlY3Zmcm9tCQkJX3JlY3Zmcm9tCiAjZGVmaW5lCQlyZWN2bXNnCQkJCV9yZWN2bXNnCisjZGVm aW5lCQlyZWN2bW1zZwkJCV9yZWN2bW1zZwogI2RlZmluZQkJc2VsZWN0CQkJCV9zZWxlY3QKICNk ZWZpbmUJCXNlbV9jbG9zZQkJCV9zZW1fY2xvc2UKICNkZWZpbmUJCXNlbV9kZXN0cm95CQkJX3Nl bV9kZXN0cm95CkBAIC0yMjAsNiArMjIxLDcgQEAKICNkZWZpbmUJCXNlbV91bmxpbmsJCQlfc2Vt X3VubGluawogI2RlZmluZQkJc2VtX3dhaXQJCQlfc2VtX3dhaXQKICNkZWZpbmUJCXNlbmRtc2cJ CQkJX3NlbmRtc2cKKyNkZWZpbmUJCXNlbmRtbXNnCQkJX3NlbmRtbXNnCiAjZGVmaW5lCQlzZW5k dG8JCQkJX3NlbmR0bwogI2RlZmluZQkJc2V0c29ja29wdAkJCV9zZXRzb2Nrb3B0CiAvKiNkZWZp bmUJCXNpZ2FjdGlvbgkJCV9zaWdhY3Rpb24qLwpkaWZmIC0tZ2l0IGEvbGliL2xpYmMvaW5jbHVk ZS91bi1uYW1lc3BhY2UuaCBiL2xpYi9saWJjL2luY2x1ZGUvdW4tbmFtZXNwYWNlLmgKaW5kZXgg ZjMxZmE3YS4uMDIzMzM0OCAxMDA2NDQKLS0tIGEvbGliL2xpYmMvaW5jbHVkZS91bi1uYW1lc3Bh Y2UuaAorKysgYi9saWIvbGliYy9pbmNsdWRlL3VuLW5hbWVzcGFjZS5oCkBAIC0xODksNiArMTg5 LDcgQEAKICN1bmRlZgkJcmVhZHYKICN1bmRlZgkJcmVjdmZyb20KICN1bmRlZgkJcmVjdm1zZwor I3VuZGVmCQlyZWN2bW1zZwogI3VuZGVmCQlzZWxlY3QKICN1bmRlZgkJc2VtX2Nsb3NlCiAjdW5k ZWYJCXNlbV9kZXN0cm95CkBAIC0yMDEsNiArMjAyLDcgQEAKICN1bmRlZgkJc2VtX3VubGluawog I3VuZGVmCQlzZW1fd2FpdAogI3VuZGVmCQlzZW5kbXNnCisjdW5kZWYJCXNlbmRtbXNnCiAjdW5k ZWYJCXNlbmR0bwogI3VuZGVmCQlzZXRzb2Nrb3B0CiAjdW5kZWYJCXNpZ2FjdGlvbgpkaWZmIC0t Z2l0IGEvbGliL2xpYmMvc3lzL01ha2VmaWxlLmluYyBiL2xpYi9saWJjL3N5cy9NYWtlZmlsZS5p bmMKaW5kZXggZTRmZTFiMi4uNWY4YjY5OSAxMDA2NDQKLS0tIGEvbGliL2xpYmMvc3lzL01ha2Vm aWxlLmluYworKysgYi9saWIvbGliYy9zeXMvTWFrZWZpbGUuaW5jCkBAIC0yOCw2ICsyOCw4IEBA IFNSQ1MrPSBmdXRpbWVucy5jIHV0aW1lbnNhdC5jCiBOT0FTTSs9IGZ1dGltZW5zLm8gdXRpbWVu c2F0Lm8KIFBTRVVETys9IF9mdXRpbWVucy5vIF91dGltZW5zYXQubwogCitTUkNTKz0gcmVjdm1t c2cuYyBzZW5kbW1zZy5jCisKIElOVEVSUE9TRUQgPSBcCiAJYWNjZXB0IFwKIAlhY2NlcHQ0IFwK ZGlmZiAtLWdpdCBhL2xpYi9saWJjL3N5cy9TeW1ib2wubWFwIGIvbGliL2xpYmMvc3lzL1N5bWJv bC5tYXAKaW5kZXggN2IzMjU3Yy4uZGMyZWQwZSAxMDA2NDQKLS0tIGEvbGliL2xpYmMvc3lzL1N5 bWJvbC5tYXAKKysrIGIvbGliL2xpYmMvc3lzL1N5bWJvbC5tYXAKQEAgLTM5OSw2ICszOTksOCBA QCBGQlNEXzEuNCB7CiAJdXRpbWVuc2F0OwogCW51bWFfc2V0YWZmaW5pdHk7CiAJbnVtYV9nZXRh ZmZpbml0eTsKKwlzZW5kbW1zZzsKKwlyZWN2bW1zZzsKIH07CiAKIEZCU0Rwcml2YXRlXzEuMCB7 CmRpZmYgLS1naXQgYS9saWIvbGliYy9zeXMvcmVjdi4yIGIvbGliL2xpYmMvc3lzL3JlY3YuMgpp bmRleCAzMjZlN2ZmLi5mZDJiMmExIDEwMDY0NAotLS0gYS9saWIvbGliYy9zeXMvcmVjdi4yCisr KyBiL2xpYi9saWJjL3N5cy9yZWN2LjIKQEAgLTM0LDggKzM0LDkgQEAKIC5TaCBOQU1FCiAuTm0g cmVjdiAsCiAuTm0gcmVjdmZyb20gLAotLk5tIHJlY3Ztc2cKLS5OZCByZWNlaXZlIGEgbWVzc2Fn ZSBmcm9tIGEgc29ja2V0CisuTm0gcmVjdm1zZyAsCisuTm0gcmVjdm1tc2cKKy5OZCByZWNlaXZl IG1lc3NhZ2UocykgZnJvbSBhIHNvY2tldAogLlNoIExJQlJBUlkKIC5MYiBsaWJjCiAuU2ggU1lO T1BTSVMKQEAgLTQ3LDExICs0OCwxNSBAQAogLkZuIHJlY3Zmcm9tICJpbnQgcyIgInZvaWQgKmJ1 ZiIgInNpemVfdCBsZW4iICJpbnQgZmxhZ3MiICJzdHJ1Y3Qgc29ja2FkZHIgKiByZXN0cmljdCBm cm9tIiAic29ja2xlbl90ICogcmVzdHJpY3QgZnJvbWxlbiIKIC5GdCBzc2l6ZV90CiAuRm4gcmVj dm1zZyAiaW50IHMiICJzdHJ1Y3QgbXNnaGRyICptc2ciICJpbnQgZmxhZ3MiCisuRnQgc3NpemVf dAorLkZuIHJlY3ZtbXNnICJpbnQgcyIgInN0cnVjdCBtbXNnaGRyICptc2d2ZWMiICJzaXplX3Qg dmxlbiIgImludCBmbGFncyIgImNvbnN0IHN0cnVjdCB0aW1lc3BlYyAqdGltZW91dCIKIC5TaCBE RVNDUklQVElPTgogVGhlCiAuRm4gcmVjdmZyb20KIGFuZAogLkZuIHJlY3Ztc2cKK2FuZAorLkZu IHJlY3ZtbXNnCiBzeXN0ZW0gY2FsbHMKIGFyZSB1c2VkIHRvIHJlY2VpdmUgbWVzc2FnZXMgZnJv bSBhIHNvY2tldCwKIGFuZCBtYXkgYmUgdXNlZCB0byByZWNlaXZlIGRhdGEgb24gYSBzb2NrZXQg d2hldGhlciBvciBub3QKQEAgLTg0LDggKzg5LDMwIEBAIG51bGwgcG9pbnRlciBwYXNzZWQgYXMg aXRzCiAuRmEgZnJvbQogYXJndW1lbnQuCiAuUHAKLUFsbCB0aHJlZSByb3V0aW5lcyByZXR1cm4g dGhlIGxlbmd0aCBvZiB0aGUgbWVzc2FnZSBvbiBzdWNjZXNzZnVsCi1jb21wbGV0aW9uLgorVGhl CisuRm4gcmVjdm1tc2cKK2Z1bmN0aW9uIGlzIHVzZWQgdG8gcmVjZWl2ZSBtdWx0aXBsZQorbWVz c2FnZXMgYXQgYSBjYWxsLgorVGhlaXIgbnVtYmVyCitpcyBzdXBwbGllZCBieQorLkZhIHZsZW4g LgorVGhlIG1lc3NhZ2VzIGFyZSBwbGFjZWQgaW4gdGhlCisuRmEgbXNndmVjCit2ZWN0b3IgYWZ0 ZXIgcmVjZXB0aW9uLgorVGhlIHNpemUgb2YgZWFjaCByZWNlaXZlZCBtZXNzYWdlIGlzIHBsYWNl ZCBpbiB0aGUKKy5GYSBtc2dfbGVuCitmaWVsZCBvZiBlYWNoIGVsZW1lbnQgb2YgdGhlIHZlY3Rv ci4KK0lmCisuRmEgdGltZW91dAoraXMgTlVMTCB0aGUgY2FsbCB3aWxsIG5vcm1hbGx5IGJsb2Nr LiBPdGhlcndpc2UgaXQgd2lsbCB3YWl0IGZvciBkYXRhCitmb3IgdGhlIHNwZWNpZmllZCBhbW91 bnQgb2YgdGltZS4gSWYgdGhlIHRpbWVvdXQgZXhwaXJlcyBhbmQgdGhlcmUgaXMKK25vIGRhdGEg cmVjZWl2ZWQgYSB2YWx1ZSBvZiAwIGlzIHJldHVybmVkLiBwc2VsZWN0KDIpIGlzIHVzZWQgZm9y IHRoZQoraW1wbGVtZW50YXRpb24gb2YgdGhlIHRpbWVvdXQgbWVjaGFuaXNtLgorLlBwCitUaGUg Zmlyc3QgdGhyZWUgcm91dGluZXMgcmV0dXJuIHRoZSBsZW5ndGggb2YgdGhlIG1lc3NhZ2Ugb24g c3VjY2Vzc2Z1bAorY29tcGxldGlvbiB3aGVyZWFzCisuRm4gcmVjdm1tc2cKK3JldHVybnMgdGhl IG51bWJlciBvZiByZWNlaXZlZCBtZXNzYWdlcy4KIElmIGEgbWVzc2FnZSBpcyB0b28gbG9uZyB0 byBmaXQgaW4gdGhlIHN1cHBsaWVkIGJ1ZmZlciwKIGV4Y2VzcyBieXRlcyBtYXkgYmUgZGlzY2Fy ZGVkIGRlcGVuZGluZyBvbiB0aGUgdHlwZSBvZiBzb2NrZXQKIHRoZSBtZXNzYWdlIGlzIHJlY2Vp dmVkIGZyb20gKHNlZQpAQCAtMTAwLDcgKzEyNyw5IEBAIGluIHdoaWNoIGNhc2UgdGhlIHZhbHVl CiAuVmEgZXJybm8KIGlzIHNldCB0bwogLkVyIEVBR0FJTiAuCi1UaGUgcmVjZWl2ZSBjYWxscyBu b3JtYWxseSByZXR1cm4gYW55IGRhdGEgYXZhaWxhYmxlLAorVGhlIHJlY2VpdmUgY2FsbHMgZXhj ZXB0CisuRm4gcmVjdm1tc2cKK25vcm1hbGx5IHJldHVybiBhbnkgZGF0YSBhdmFpbGFibGUsCiB1 cCB0byB0aGUgcmVxdWVzdGVkIGFtb3VudCwKIHJhdGhlciB0aGFuIHdhaXRpbmcgZm9yIHJlY2Vp cHQgb2YgdGhlIGZ1bGwgYW1vdW50IHJlcXVlc3RlZDsKIHRoaXMgYmVoYXZpb3IgaXMgYWZmZWN0 ZWQgYnkgdGhlIHNvY2tldC1sZXZlbCBvcHRpb25zCkBAIC0xMjcsNiArMTU2LDkgQEAgb25lIG9y IG1vcmUgb2YgdGhlIHZhbHVlczoKIC5JdCBEdiBNU0dfV0FJVEFMTCBUYSB3YWl0IGZvciBmdWxs IHJlcXVlc3Qgb3IgZXJyb3IKIC5JdCBEdiBNU0dfRE9OVFdBSVQgVGEgZG8gbm90IGJsb2NrCiAu SXQgRHYgTVNHX0NNU0dfQ0xPRVhFQyBUYSBzZXQgcmVjZWl2ZWQgZmRzIGNsb3NlLW9uLWV4ZWMK Ky5JdCBEdiBNU0dfV0FJVEZPUk9ORSBUYSBkbyBub3QgYmxvY2sgYWZ0ZXIgcmVjZWl2aW5nIHRo ZSBmaXJzdCBtZXNzYWdlCisocmVsZXZhbnQgb25seSBmb3IKKy5GbiByZWN2bW1zZyApCiAuRWwK IC5QcAogVGhlCkBAIC0xNTgsNiArMTkwLDExIEBAIGlzIHNldCB0bwogVGhpcyBmbGFnIGlzIG5v dCBhdmFpbGFibGUgaW4gc3RyaWN0CiAuVG4gQU5TSQogb3IgQzk5IGNvbXBpbGF0aW9uIG1vZGUu CitUaGUKKy5EdiBNU0dfV0FJVEZPUk9ORQorZmxhZyBzZXRzIE1TR19ET05UV0FJVCBhZnRlciB0 aGUgZmlyc3QgbWVzc2FnZSBoYXMgYmVlbiByZWNlaXZlZC4gVGhpcyBmbGFnCitpcyBvbmx5IHJl bGV2YW50IGZvcgorLkZuIHJlY3ZtbXNnIC4KIC5QcAogVGhlCiAuRm4gcmVjdm1zZwpAQCAtMjkw LDkgKzMyNywzNCBAQCBjb250cm9sIGRhdGEgd2VyZSBkaXNjYXJkZWQgZHVlIHRvIGxhY2sgb2Yg c3BhY2UgaW4gdGhlIGJ1ZmZlcgogZm9yIGFuY2lsbGFyeSBkYXRhLgogLkR2IE1TR19PT0IKIGlz IHJldHVybmVkIHRvIGluZGljYXRlIHRoYXQgZXhwZWRpdGVkIG9yIG91dC1vZi1iYW5kIGRhdGEg d2VyZSByZWNlaXZlZC4KKy5QcAorVGhlCisuRm4gcmVjdm1tc2cKK3N5c3RlbSBjYWxsIHVzZXMg dGhlCisuRmEgbW1zZ2hkcgorc3RydWN0dXJlLiBJdHMgZm9ybSBpcyBhcyBmb2xsb3dzLCBhcyBk ZWZpbmVkIGluCisuSW4gc3lzL3NvY2tldC5oIDoKKy5CZCAtbGl0ZXJhbAorc3RydWN0IG1tc2do ZHIgeworCXN0cnVjdCBtc2doZHIJIG1zZ19oZHI7CS8qIG1lc3NhZ2UgaGVhZGVyICovCisJdW5z aWduZWQgaW50CSBtc2dfbGVuOwkvKiBtZXNzYWdlIGxlbmd0aCAqLworfTsKKy5FZAorLlBwCitG b3IKKy5GYSBtc2dfaGRyCitzZWUgYWJvdmUuIE9uIGRhdGEgcmVjZXB0aW9uIHRoZQorLkZhIG1z Z19sZW4KK2ZpZWxkIGlzIHVwZGF0ZWQgdG8gdGhlIGxlbmd0aCBvZiB0aGUgcmVjZWl2ZWQgbWVz c2FnZS4gT24KK2RhdGEgdHJhbnNtaXNzaW9uIGl0IGlzIHVwZGF0ZWQgdG8gdGhlIG51bWJlciBv ZgorY2hhcmFjdGVycyBzZW50LgogLlNoIFJFVFVSTiBWQUxVRVMKLVRoZXNlIGNhbGxzIHJldHVy biB0aGUgbnVtYmVyIG9mIGJ5dGVzIHJlY2VpdmVkLCBvciAtMQotaWYgYW4gZXJyb3Igb2NjdXJy ZWQuCitUaGVzZSBjYWxscyBleGNlcHQKKy5GbiByZWN2bW1zZworcmV0dXJuIHRoZSBudW1iZXIg b2YgYnl0ZXMgcmVjZWl2ZWQuCisuRm4gcmVjdm1tc2cKK3JldHVybnMgdGhlIG51bWJlciBvZiBt ZXNzYWdlcyByZWNlaXZlZC4KK0EgdmFsdWUgb2YgLTEgaXMgcmV0dXJuZWQgaWYgYW4gZXJyb3Ig b2NjdXJyZWQuCiAuU2ggRVJST1JTCiBUaGUgY2FsbHMgZmFpbCBpZjoKIC5CbCAtdGFnIC13aWR0 aCBFcgpkaWZmIC0tZ2l0IGEvbGliL2xpYmMvc3lzL3JlY3ZtbXNnLmMgYi9saWIvbGliYy9zeXMv cmVjdm1tc2cuYwpuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwLi4xOWE5MzdiCi0t LSAvZGV2L251bGwKKysrIGIvbGliL2xpYmMvc3lzL3JlY3ZtbXNnLmMKQEAgLTAsMCArMSw5NiBA QAorLyoKKyAqIENvcHlyaWdodCAoYykgMjAxNiBCb3JpcyBBc3RhcmR6aGlldiwgU21hcnRjb20t QnVsZ2FyaWEgQUQKKyAqIEFsbCByaWdodHMgcmVzZXJ2ZWQuCisgKgorICogUmVkaXN0cmlidXRp b24gYW5kIHVzZSBpbiBzb3VyY2UgYW5kIGJpbmFyeSBmb3Jtcywgd2l0aCBvciB3aXRob3V0Cisg KiBtb2RpZmljYXRpb24sIGFyZSBwZXJtaXR0ZWQgcHJvdmlkZWQgdGhhdCB0aGUgZm9sbG93aW5n IGNvbmRpdGlvbnMKKyAqIGFyZSBtZXQ6CisgKiAxLiBSZWRpc3RyaWJ1dGlvbnMgb2Ygc291cmNl IGNvZGUgbXVzdCByZXRhaW4gdGhlIGFib3ZlIGNvcHlyaWdodAorICogICAgbm90aWNlKHMpLCB0 aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhlIGZvbGxvd2luZyBkaXNjbGFpbWVyIGFzCisg KiAgICB0aGUgZmlyc3QgbGluZXMgb2YgdGhpcyBmaWxlIHVubW9kaWZpZWQgb3RoZXIgdGhhbiB0 aGUgcG9zc2libGUKKyAqICAgIGFkZGl0aW9uIG9mIG9uZSBvciBtb3JlIGNvcHlyaWdodCBub3Rp Y2VzLgorICogMi4gUmVkaXN0cmlidXRpb25zIGluIGJpbmFyeSBmb3JtIG11c3QgcmVwcm9kdWNl IHRoZSBhYm92ZSBjb3B5cmlnaHQKKyAqICAgIG5vdGljZShzKSwgdGhpcyBsaXN0IG9mIGNvbmRp dGlvbnMgYW5kIHRoZSBmb2xsb3dpbmcgZGlzY2xhaW1lciBpbgorICogICAgdGhlIGRvY3VtZW50 YXRpb24gYW5kL29yIG90aGVyIG1hdGVyaWFscyBwcm92aWRlZCB3aXRoIHRoZQorICogICAgZGlz dHJpYnV0aW9uLgorICoKKyAqIFRISVMgU09GVFdBUkUgSVMgUFJPVklERUQgQlkgVEhFIENPUFlS SUdIVCBIT0xERVIoUykgYGBBUyBJUycnIEFORCBBTlkKKyAqIEVYUFJFU1MgT1IgSU1QTElFRCBX QVJSQU5USUVTLCBJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgVEhFCisgKiBJTVBMSUVE IFdBUlJBTlRJRVMgT0YgTUVSQ0hBTlRBQklMSVRZIEFORCBGSVRORVNTIEZPUiBBIFBBUlRJQ1VM QVIKKyAqIFBVUlBPU0UgQVJFIERJU0NMQUlNRUQuICBJTiBOTyBFVkVOVCBTSEFMTCBUSEUgQ09Q WVJJR0hUIEhPTERFUihTKSBCRQorICogTElBQkxFIEZPUiBBTlkgRElSRUNULCBJTkRJUkVDVCwg SU5DSURFTlRBTCwgU1BFQ0lBTCwgRVhFTVBMQVJZLCBPUgorICogQ09OU0VRVUVOVElBTCBEQU1B R0VTIChJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgUFJPQ1VSRU1FTlQgT0YKKyAqIFNV QlNUSVRVVEUgR09PRFMgT1IgU0VSVklDRVM7IExPU1MgT0YgVVNFLCBEQVRBLCBPUiBQUk9GSVRT OyBPUgorICogQlVTSU5FU1MgSU5URVJSVVBUSU9OKSBIT1dFVkVSIENBVVNFRCBBTkQgT04gQU5Z IFRIRU9SWSBPRiBMSUFCSUxJVFksCisgKiBXSEVUSEVSIElOIENPTlRSQUNULCBTVFJJQ1QgTElB QklMSVRZLCBPUiBUT1JUIChJTkNMVURJTkcgTkVHTElHRU5DRQorICogT1IgT1RIRVJXSVNFKSBB UklTSU5HIElOIEFOWSBXQVkgT1VUIE9GIFRIRSBVU0UgT0YgVEhJUyBTT0ZUV0FSRSwKKyAqIEVW RU4gSUYgQURWSVNFRCBPRiBUSEUgUE9TU0lCSUxJVFkgT0YgU1VDSCBEQU1BR0UuCisgKi8KKwor I2luY2x1ZGUgPHN5cy9jZGVmcy5oPgorX19GQlNESUQoIiRGcmVlQlNEJCIpOworCisjaW5jbHVk ZSA8ZXJybm8uaD4KKyNpbmNsdWRlIDxzeXMvdHlwZXMuaD4KKyNpbmNsdWRlIDxzeXMvc3lzY2Fs bC5oPgorI2luY2x1ZGUgPHN5cy9zb2NrZXQuaD4KKyNpbmNsdWRlIDxzeXMvc2VsZWN0Lmg+Cisj aW5jbHVkZSA8cHRocmVhZC5oPgorI2luY2x1ZGUgImxpYmNfcHJpdmF0ZS5oIgorCisjZGVmaW5l IENNVFIocywgdGltZW91dCkJCQkJCQlcCisJZG8gewkJCQkJCQkJXAorCQlmZF9zZXQgZmRzOwkJ CQkJCVwKKwkJaW50IHJlczsJCQkJCQlcCisJCQkJCQkJCQlcCisJCUZEX1pFUk8oJmZkcyk7CQkJ CQkJXAorCQlGRF9TRVQoKHMpLCAmZmRzKTsJCQkJCVwKKwkJcmVzID0gX19zeXNfcHNlbGVjdCgo cykrMSwgJmZkcywgTlVMTCwgTlVMTCwgKHRpbWVvdXQpLCBOVUxMKTtcCisJCWlmIChyZXMgPT0g LTEgfHwgcmVzID09IDApCQkJCVwKKwkJCXJldHVybiAocmVzKTsJCQkJCVwKKwkJaWYgKCFGRF9J U1NFVCgocyksICZmZHMpKQkJCQlcCisJCQlyZXR1cm4gKC0xKTsJCQkJCVwKKwl9IHdoaWxlICgw KTsKKworc3NpemVfdAorcmVjdm1tc2coaW50IHMsIHN0cnVjdCBtbXNnaGRyICptc2d2ZWMsIHNp emVfdCB2bGVuLCBpbnQgZmxhZ3MsCisgICAgY29uc3Qgc3RydWN0IHRpbWVzcGVjICp0aW1lb3V0 KQoreworCXNpemVfdCBpLCByY3ZkOworCXNzaXplX3QgcmV0OworCisJaWYgKHRpbWVvdXQgIT0g TlVMTCkKKwkJQ01UUihzLCB0aW1lb3V0KTsKKworCXJldCA9IF9fc3lzX3JlY3Ztc2cocywgJm1z Z3ZlY1swXS5tc2dfaGRyLCBmbGFncyk7CisJaWYgKHJldCA9PSAtMSkKKwkJcmV0dXJuIChyZXQp OworCisJLyogQ2hlY2sgaW5pdGlhbGx5IGZvciB0aGUgcHJlc2VuY2Ugb2YgTVNHX1dBSVRGT1JP TkUuCisJICogVHVybiBvbiBNU0dfRE9OVFdBSVQgaWYgc2V0LiAqLworCWlmIChmbGFncyAmIE1T R19XQUlURk9ST05FKSB7CisJCWZsYWdzIHw9IE1TR19ET05UV0FJVDsKKwkJLyogVGhlIGtlcm5l bCBkb2Vzbid0IG5lZWQgdG8ga25vdyBhYm91dCB0aGlzIGZsYWcuICovCisJCWZsYWdzICY9IH5N U0dfV0FJVEZPUk9ORTsKKwl9CisKKwlyY3ZkID0gMTsKKwlmb3IgKGkgPSByY3ZkOyBpIDwgdmxl bjsgaSsrKSB7CisJCXJldCA9IF9fc3lzX3JlY3Ztc2cocywgJm1zZ3ZlY1tpXS5tc2dfaGRyLCBm bGFncyk7CisJCWlmIChyZXQgPT0gLTEpIHsKKwkJCWlmIChyY3ZkICE9IDApIHsKKwkJCQkvKiBX ZSd2ZSByZWNlaXZlZCBtZXNzYWdlcy4gTGV0IGNhbGxlciBrbm93LiAqLworCQkJCXJldHVybiAo cmN2ZCk7CisJCQl9CisJCQlyZXR1cm4gKHJldCk7CisJCX0KKworCQkvKiBTYXZlIHJlY2VpdmVk IGJ5dGVzICovCisJCW1zZ3ZlY1tpXS5tc2dfbGVuID0gcmV0OworCQlyY3ZkKys7CisJfQorCisJ cmV0dXJuIChyY3ZkKTsKK30KKworI3VuZGVmIENNVFIKZGlmZiAtLWdpdCBhL2xpYi9saWJjL3N5 cy9zZW5kLjIgYi9saWIvbGliYy9zeXMvc2VuZC4yCmluZGV4IDhmYTJjNjQuLjMzZmE1OGQgMTAw NjQ0Ci0tLSBhL2xpYi9saWJjL3N5cy9zZW5kLjIKKysrIGIvbGliL2xpYmMvc3lzL3NlbmQuMgpA QCAtMzQsOCArMzQsOSBAQAogLlNoIE5BTUUKIC5ObSBzZW5kICwKIC5ObSBzZW5kdG8gLAotLk5t IHNlbmRtc2cKLS5OZCBzZW5kIGEgbWVzc2FnZSBmcm9tIGEgc29ja2V0CisuTm0gc2VuZG1zZyAs CisuTm0gc2VuZG1tc2cKKy5OZCBzZW5kIG1lc3NhZ2UocykgZnJvbSBhIHNvY2tldAogLlNoIExJ QlJBUlkKIC5MYiBsaWJjCiAuU2ggU1lOT1BTSVMKQEAgLTQ3LDYgKzQ4LDggQEAKIC5GbiBzZW5k dG8gImludCBzIiAiY29uc3Qgdm9pZCAqbXNnIiAic2l6ZV90IGxlbiIgImludCBmbGFncyIgImNv bnN0IHN0cnVjdCBzb2NrYWRkciAqdG8iICJzb2NrbGVuX3QgdG9sZW4iCiAuRnQgc3NpemVfdAog LkZuIHNlbmRtc2cgImludCBzIiAiY29uc3Qgc3RydWN0IG1zZ2hkciAqbXNnIiAiaW50IGZsYWdz IgorLkZ0IHNzaXplX3QKKy5GbiBzZW5kbW1zZyAiaW50IHMiICJzdHJ1Y3QgbW1zZ2hkciAqbXNn dmVjIiAic2l6ZV90IHZsZW4iICJpbnQgZmxhZ3MiCiAuU2ggREVTQ1JJUFRJT04KIFRoZQogLkZu IHNlbmQKQEAgLTU1LDggKzU4LDEwIEBAIGFuZAogLkZuIHNlbmR0bwogYW5kCiAuRm4gc2VuZG1z ZworYW5kCisuRm4gc2VuZG1tc2cKIHN5c3RlbSBjYWxscwotYXJlIHVzZWQgdG8gdHJhbnNtaXQg YSBtZXNzYWdlIHRvIGFub3RoZXIgc29ja2V0LgorYXJlIHVzZWQgdG8gdHJhbnNtaXQgb25lIG9y IG11bHRpcGxlIG1lc3NhZ2VzICh3aXRoIHRoZSBsYXR0ZXIgY2FsbCkgdG8gYW5vdGhlciBzb2Nr ZXQuCiBUaGUKIC5GbiBzZW5kCiBmdW5jdGlvbgpAQCAtNjYsNiArNzEsOCBAQCBzdGF0ZSwgd2hp bGUKIC5GbiBzZW5kdG8KIGFuZAogLkZuIHNlbmRtc2cKK2FuZAorLkZuIHNlbmRtbXNnCiBtYXkg YmUgdXNlZCBhdCBhbnkgdGltZS4KIC5QcAogVGhlIGFkZHJlc3Mgb2YgdGhlIHRhcmdldCBpcyBn aXZlbiBieQpAQCAtODEsNiArODgsMTggQEAgdW5kZXJseWluZyBwcm90b2NvbCwgdGhlIGVycm9y CiBpcyByZXR1cm5lZCwgYW5kCiB0aGUgbWVzc2FnZSBpcyBub3QgdHJhbnNtaXR0ZWQuCiAuUHAK K1RoZQorLkZuIHNlbmRtbXNnCitmdW5jdGlvbiBzZW5kcyBtdWx0aXBsZSBtZXNzYWdlcyBhdCBh IGNhbGwuCitUaGV5IGFyZSBnaXZlbiBieSB0aGUKKy5GYSBtc2d2ZWMKK3ZlY3RvciBhbG9uZyB3 aXRoCisuRmEgdmxlbgorc3BlY2lmeWluZyBpdHMgc2l6ZS4gVGhlIG51bWJlciBvZgorY2hhcmFj dGVycyBzZW50IHBlciBlYWNoIG1lc3NhZ2UgaXMgcGxhY2VkIGluIHRoZQorLkZhIG1zZ19sZW4K K2ZpZWxkIG9mIGVhY2ggZWxlbWVudCBvZiB0aGUgdmVjdG9yIGFmdGVyIHRyYW5zbWlzc2lvbi4K Ky5QcAogTm8gaW5kaWNhdGlvbiBvZiBmYWlsdXJlIHRvIGRlbGl2ZXIgaXMgaW1wbGljaXQgaW4g YQogLkZuIHNlbmQgLgogTG9jYWxseSBkZXRlY3RlZCBlcnJvcnMgYXJlIGluZGljYXRlZCBieSBh IHJldHVybiB2YWx1ZSBvZiAtMS4KQEAgLTEzOCwxMCArMTU3LDE2IEBAIFNlZQogLlhyIHJlY3Yg MgogZm9yIGEgZGVzY3JpcHRpb24gb2YgdGhlCiAuRmEgbXNnaGRyCitzdHJ1Y3R1cmUgYW5kIHRo ZQorLkZhIG1tc2doZHIKIHN0cnVjdHVyZS4KIC5TaCBSRVRVUk4gVkFMVUVTCi1UaGUgY2FsbCBy ZXR1cm5zIHRoZSBudW1iZXIgb2YgY2hhcmFjdGVycyBzZW50LCBvciAtMQotaWYgYW4gZXJyb3Ig b2NjdXJyZWQuCitBbGwgY2FsbHMgZXhjZXB0CisuRm4gc2VuZG1tc2cKK3JldHVybiB0aGUgbnVt YmVyIG9mIGNoYXJhY3RlcnMgc2VudC4gVGhlCisuRm4gc2VuZG1tc2cKK2NhbGwgcmV0dXJucyB0 aGUgbnVtYmVyIG9mIG1lc3NhZ2VzIHNlbnQuCitJZiBhbiBlcnJvciBvY2N1cnJlZCBhIHZhbHVl IG9mIC0xIGlzIHJldHVybmVkLgogLlNoIEVSUk9SUwogVGhlCiAuRm4gc2VuZApAQCAtMTQ5LDYg KzE3NCw4IEBAIGZ1bmN0aW9uIGFuZAogLkZuIHNlbmR0bwogYW5kCiAuRm4gc2VuZG1zZworYW5k CisuRm4gc2VuZG1tc2cKIHN5c3RlbSBjYWxscwogZmFpbCBpZjoKIC5CbCAtdGFnIC13aWR0aCBF cgpkaWZmIC0tZ2l0IGEvbGliL2xpYmMvc3lzL3NlbmRtbXNnLmMgYi9saWIvbGliYy9zeXMvc2Vu ZG1tc2cuYwpuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwLi5jZWYzNWEzCi0tLSAv ZGV2L251bGwKKysrIGIvbGliL2xpYmMvc3lzL3NlbmRtbXNnLmMKQEAgLTAsMCArMSw2MyBAQAor LyoKKyAqIENvcHlyaWdodCAoYykgMjAxNiBCb3JpcyBBc3RhcmR6aGlldiwgU21hcnRjb20tQnVs Z2FyaWEgQUQKKyAqIEFsbCByaWdodHMgcmVzZXJ2ZWQuCisgKgorICogUmVkaXN0cmlidXRpb24g YW5kIHVzZSBpbiBzb3VyY2UgYW5kIGJpbmFyeSBmb3Jtcywgd2l0aCBvciB3aXRob3V0CisgKiBt b2RpZmljYXRpb24sIGFyZSBwZXJtaXR0ZWQgcHJvdmlkZWQgdGhhdCB0aGUgZm9sbG93aW5nIGNv bmRpdGlvbnMKKyAqIGFyZSBtZXQ6CisgKiAxLiBSZWRpc3RyaWJ1dGlvbnMgb2Ygc291cmNlIGNv ZGUgbXVzdCByZXRhaW4gdGhlIGFib3ZlIGNvcHlyaWdodAorICogICAgbm90aWNlKHMpLCB0aGlz IGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhlIGZvbGxvd2luZyBkaXNjbGFpbWVyIGFzCisgKiAg ICB0aGUgZmlyc3QgbGluZXMgb2YgdGhpcyBmaWxlIHVubW9kaWZpZWQgb3RoZXIgdGhhbiB0aGUg cG9zc2libGUKKyAqICAgIGFkZGl0aW9uIG9mIG9uZSBvciBtb3JlIGNvcHlyaWdodCBub3RpY2Vz LgorICogMi4gUmVkaXN0cmlidXRpb25zIGluIGJpbmFyeSBmb3JtIG11c3QgcmVwcm9kdWNlIHRo ZSBhYm92ZSBjb3B5cmlnaHQKKyAqICAgIG5vdGljZShzKSwgdGhpcyBsaXN0IG9mIGNvbmRpdGlv bnMgYW5kIHRoZSBmb2xsb3dpbmcgZGlzY2xhaW1lciBpbgorICogICAgdGhlIGRvY3VtZW50YXRp b24gYW5kL29yIG90aGVyIG1hdGVyaWFscyBwcm92aWRlZCB3aXRoIHRoZQorICogICAgZGlzdHJp YnV0aW9uLgorICoKKyAqIFRISVMgU09GVFdBUkUgSVMgUFJPVklERUQgQlkgVEhFIENPUFlSSUdI VCBIT0xERVIoUykgYGBBUyBJUycnIEFORCBBTlkKKyAqIEVYUFJFU1MgT1IgSU1QTElFRCBXQVJS QU5USUVTLCBJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgVEhFCisgKiBJTVBMSUVEIFdB UlJBTlRJRVMgT0YgTUVSQ0hBTlRBQklMSVRZIEFORCBGSVRORVNTIEZPUiBBIFBBUlRJQ1VMQVIK KyAqIFBVUlBPU0UgQVJFIERJU0NMQUlNRUQuICBJTiBOTyBFVkVOVCBTSEFMTCBUSEUgQ09QWVJJ R0hUIEhPTERFUihTKSBCRQorICogTElBQkxFIEZPUiBBTlkgRElSRUNULCBJTkRJUkVDVCwgSU5D SURFTlRBTCwgU1BFQ0lBTCwgRVhFTVBMQVJZLCBPUgorICogQ09OU0VRVUVOVElBTCBEQU1BR0VT IChJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgUFJPQ1VSRU1FTlQgT0YKKyAqIFNVQlNU SVRVVEUgR09PRFMgT1IgU0VSVklDRVM7IExPU1MgT0YgVVNFLCBEQVRBLCBPUiBQUk9GSVRTOyBP UgorICogQlVTSU5FU1MgSU5URVJSVVBUSU9OKSBIT1dFVkVSIENBVVNFRCBBTkQgT04gQU5ZIFRI RU9SWSBPRiBMSUFCSUxJVFksCisgKiBXSEVUSEVSIElOIENPTlRSQUNULCBTVFJJQ1QgTElBQklM SVRZLCBPUiBUT1JUIChJTkNMVURJTkcgTkVHTElHRU5DRQorICogT1IgT1RIRVJXSVNFKSBBUklT SU5HIElOIEFOWSBXQVkgT1VUIE9GIFRIRSBVU0UgT0YgVEhJUyBTT0ZUV0FSRSwKKyAqIEVWRU4g SUYgQURWSVNFRCBPRiBUSEUgUE9TU0lCSUxJVFkgT0YgU1VDSCBEQU1BR0UuCisgKi8KKworI2lu Y2x1ZGUgPHN5cy9jZGVmcy5oPgorX19GQlNESUQoIiRGcmVlQlNEJCIpOworCisjaW5jbHVkZSA8 ZXJybm8uaD4KKyNpbmNsdWRlIDxzeXMvdHlwZXMuaD4KKyNpbmNsdWRlIDxzeXMvc3lzY2FsbC5o PgorI2luY2x1ZGUgPHN5cy9zb2NrZXQuaD4KKyNpbmNsdWRlIDxwdGhyZWFkLmg+CisjaW5jbHVk ZSAibGliY19wcml2YXRlLmgiCisKK3NzaXplX3QKK3NlbmRtbXNnKGludCBzLCBzdHJ1Y3QgbW1z Z2hkciAqbXNndmVjLCBzaXplX3QgdmxlbiwgaW50IGZsYWdzKQoreworCXNpemVfdCBpLCBzZW50 OworCXNzaXplX3QgcmV0OworCisJc2VudCA9IDA7CisJZm9yIChpID0gMDsgaSA8IHZsZW47IGkr KykgeworCQlyZXQgPSBfX3N5c19zZW5kbXNnKHMsICZtc2d2ZWNbaV0ubXNnX2hkciwgZmxhZ3Mp OworCQlpZiAocmV0ID09IC0xKSB7CisJCQlpZiAoc2VudCAhPSAwKSB7CisJCQkJLyogV2UgaGF2 ZSBzZW50IG1lc3NhZ2VzLiBMZXQgY2FsbGVyIGtub3cuICovCisJCQkJcmV0dXJuIChzZW50KTsK KwkJCX0KKwkJCXJldHVybiAocmV0KTsKKwkJfQorCisJCS8qIFNhdmUgc2VudCBieXRlcyAqLwor CQltc2d2ZWNbaV0ubXNnX2xlbiA9IHJldDsKKwkJc2VudCsrOworCX0KKworCXJldHVybiAoc2Vu dCk7Cit9CmRpZmYgLS1naXQgYS9zeXMvc3lzL3NvY2tldC5oIGIvc3lzL3N5cy9zb2NrZXQuaApp bmRleCAxOGUyZGUxLi5kOTVmMjllIDEwMDY0NAotLS0gYS9zeXMvc3lzL3NvY2tldC5oCisrKyBi L3N5cy9zeXMvc29ja2V0LmgKQEAgLTQzNSw2ICs0MzUsMTEgQEAgc3RydWN0IG1zZ2hkciB7CiAj aWZkZWYgX0tFUk5FTAogI2RlZmluZQlNU0dfU09DQUxMQkNLICAgMHgxMDAwMAkJLyogZm9yIHVz ZSBieSBzb2NrZXQgY2FsbGJhY2tzIC0gc29yZWNlaXZlIChUQ1ApICovCiAjZW5kaWYKKyNpZm5k ZWYgX0tFUk5FTAorI2lmZGVmIF9fQlNEX1ZJU0lCTEUKKyNkZWZpbmUgTVNHX1dBSVRGT1JPTkUJ MHgxMDAwMDAJLyogdXNlZCBpbiByZWN2bW1zZygpICovCisjZW5kaWYgLyogX19CU0RfVklTSUJM RSAqLworI2VuZGlmIC8qICFfS0VSTkVMICovCiAKIC8qCiAgKiBIZWFkZXIgZm9yIGFuY2lsbGFy eSBkYXRhIG9iamVjdHMgaW4gbXNnX2NvbnRyb2wgYnVmZmVyLgpAQCAtNTk1LDYgKzYwMCwxOCBA QCBzdHJ1Y3Qgc2ZfaGR0ciB7CiAjZW5kaWYgLyogX0tFUk5FTCAqLwogI2VuZGlmIC8qIF9fQlNE X1ZJU0lCTEUgKi8KIAorI2lmbmRlZiBfS0VSTkVMCisjaWZkZWYgX19CU0RfVklTSUJMRQorLyoK KyAqIFNlbmQvcmVjdm1tc2cgc3BlY2lmaWMgc3RydWN0dXJlKHMpCisgKi8KK3N0cnVjdCBtbXNn aGRyIHsKKwlzdHJ1Y3QgbXNnaGRyCW1zZ19oZHI7CQkvKiBtZXNzYWdlIGhlYWRlciAqLworCXVu c2lnbmVkIGludAltc2dfbGVuOwkJLyogbWVzc2FnZSBsZW5ndGggKi8KK307CisjZW5kaWYgLyog X19CU0RfVklTSUJMRSAqLworI2VuZGlmIC8qICFfS0VSTkVMICovCisKICNpZm5kZWYJX0tFUk5F TAogCiAjaW5jbHVkZSA8c3lzL2NkZWZzLmg+CkBAIC02MTUsMTEgKzYzMiwxOSBAQCBpbnQJbGlz dGVuKGludCwgaW50KTsKIHNzaXplX3QJcmVjdihpbnQsIHZvaWQgKiwgc2l6ZV90LCBpbnQpOwog c3NpemVfdAlyZWN2ZnJvbShpbnQsIHZvaWQgKiwgc2l6ZV90LCBpbnQsIHN0cnVjdCBzb2NrYWRk ciAqIF9fcmVzdHJpY3QsIHNvY2tsZW5fdCAqIF9fcmVzdHJpY3QpOwogc3NpemVfdAlyZWN2bXNn KGludCwgc3RydWN0IG1zZ2hkciAqLCBpbnQpOworI2lmIF9fQlNEX1ZJU0lCTEUKK3N0cnVjdCB0 aW1lc3BlYzsKK3NzaXplX3QJcmVjdm1tc2coaW50LCBzdHJ1Y3QgbW1zZ2hkciAqLCBzaXplX3Qs IGludCwKKyAgICBjb25zdCBzdHJ1Y3QgdGltZXNwZWMgKik7CisjZW5kaWYKIHNzaXplX3QJc2Vu ZChpbnQsIGNvbnN0IHZvaWQgKiwgc2l6ZV90LCBpbnQpOwogc3NpemVfdAlzZW5kdG8oaW50LCBj b25zdCB2b2lkICosCiAJICAgIHNpemVfdCwgaW50LCBjb25zdCBzdHJ1Y3Qgc29ja2FkZHIgKiwg c29ja2xlbl90KTsKIHNzaXplX3QJc2VuZG1zZyhpbnQsIGNvbnN0IHN0cnVjdCBtc2doZHIgKiwg aW50KTsKICNpZiBfX0JTRF9WSVNJQkxFCitzc2l6ZV90CXNlbmRtbXNnKGludCwgc3RydWN0IG1t c2doZHIgKiwgc2l6ZV90LCBpbnQpOworI2VuZGlmCisjaWYgX19CU0RfVklTSUJMRQogaW50CXNl bmRmaWxlKGludCwgaW50LCBvZmZfdCwgc2l6ZV90LCBzdHJ1Y3Qgc2ZfaGR0ciAqLCBvZmZfdCAq LCBpbnQpOwogaW50CXNldGZpYihpbnQpOwogI2VuZGlmCg== --001a114715826cd8660529e8a863-- From owner-freebsd-net@freebsd.org Fri Jan 22 17:04:42 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E458DA8D02B for ; Fri, 22 Jan 2016 17:04:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BDA2A1381 for ; Fri, 22 Jan 2016 17:04:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0MH4gnv024905 for ; Fri, 22 Jan 2016 17:04:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206478] Setting laggproto fails on 10.2 Date: Fri, 22 Jan 2016 17:04:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: lakshmi.n@msystechnologies.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 17:04:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206478 LN changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lakshmi.n@msystechnologies. | |com --- Comment #2 from LN --- > Based on our cursory code reading, below looks like the culprit in functi= on > lagg_ioctl(). > Please let us know if the below patch works for you, (taken with head). > > Thanks, > LN > > Index: sys/net/if_lagg.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/net/if_lagg.c (revision 292047) > +++ sys/net/if_lagg.c (working copy) > @@ -1,2219 +1,2219 @@ > > break; > case SIOCSLAGG: > error =3D priv_check(td, PRIV_NET_LAGG); > if (error) > break; > - if (ra->ra_proto < 1 || ra->ra_proto >=3D LAGG_PROTO_MAX) { > + if (ra->ra_proto >=3D LAGG_PROTO_MAX) { > error =3D EPROTONOSUPPORT; > break; > } > > LAGG_WLOCK(sc); > lagg_proto_detach(sc); > LAGG_UNLOCK_ASSERT(sc); > lagg_proto_attach(sc, ra->ra_proto); > break; > case SIOCGLAGGOPTS: > --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jan 22 19:26:09 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C516EA8E58F for ; Fri, 22 Jan 2016 19:26:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B64E110A1 for ; Fri, 22 Jan 2016 19:26:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0MJQ9gg027878 for ; Fri, 22 Jan 2016 19:26:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206478] Setting laggproto fails on 10.2 Date: Fri, 22 Jan 2016 19:26:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: erin.clark.ix@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 19:26:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206478 --- Comment #3 from Erin Clark --- (In reply to LN from comment #2) That portion of code appears different in 10.2-STABLE, is this perhaps the = part that should be changed in that? if (proto->ti_proto =3D=3D LAGG_PROTO_NONE) { error =3D EPROTONOSUPPORT; break; } --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jan 22 21:35:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50C0DA8EF4B for ; Fri, 22 Jan 2016 21:35:02 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E99510CD for ; Fri, 22 Jan 2016 21:35:02 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: by mail-oi0-x22f.google.com with SMTP id w75so55473996oie.0 for ; Fri, 22 Jan 2016 13:35:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yIhhU8dQSo7pNZAqdXLrLUqa6Wh7Bgs9NUGguKQsdSM=; b=xCV4F3X7rcf0r1obSzqO4a+hu7O8TGDEJlqZAW/2XlRzmENZibqkaj7zdirO3+UgWj QN7p+qkx1tENLXU0NgV+74FdDaTx/Lk9fPPQO0VQ2neoMFsyGy0E5JLXXE52yvQ9kg92 bFTxZGQWtjq26UsvhlAtRsg84fEtjOKiPbZnMh3RpFvRRXFPpcTjsmslyegFROPh8MEK r6VjNY3H6wo0Q8R6n1dnpcA4WZhgch7KfG4O3PXKJ0lAm6aYfKSWdo1shrTce++P0bxl WfnncpouQCD84FOPa0iN1hG2oAhEMa5h/kf300RTSgZG9wDf7zoU0LjL1E6E/E1G3Cx6 V50g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=yIhhU8dQSo7pNZAqdXLrLUqa6Wh7Bgs9NUGguKQsdSM=; b=eVYC3kG8buXG1MDKG20H2t6tpBRhJ6dV0gQRvF9JlTG1ZQMxYxFON2HfQ7/O0obuzj 9oI9oNgjkpqYhPUg8Th6p9/ky+NHSegf3fIwp1UutfX2q/pBGA/IGGo9dfBsdZ/Ksifb mo9avA0BmwlgDIscCMBuY3uVHimTRuqCGFpXqDIFG+BQZZ4zkB5KNjQeAQKJO5jbSM0/ Nf7Qa96CB5bXWCQqLg9+5qoRlVtnFSJver0WVE4p6ECW0YaZKNed7ZB31gnqbZTdfexy ykylUtwHT+N4gTrL3Qq2NTQGDW5VmoXkyb/YmClsxmUjVEiIeYo0hoj0XlyUgkXWhenD MFHg== X-Gm-Message-State: AG10YOSw9tJedhDrEPONMUJdzFnuExZW7PaW0R49dFUTzaaTjchQwHOpCCUctAoWcWshOk1WAkBnwU6pvBVLLA== MIME-Version: 1.0 X-Received: by 10.202.180.66 with SMTP id d63mr4102405oif.137.1453498501116; Fri, 22 Jan 2016 13:35:01 -0800 (PST) Received: by 10.202.242.132 with HTTP; Fri, 22 Jan 2016 13:35:01 -0800 (PST) In-Reply-To: <56A13531.8090209@shrew.net> References: <56A003B8.9090104@shrew.net> <56A13531.8090209@shrew.net> Date: Fri, 22 Jan 2016 13:35:01 -0800 Message-ID: Subject: Re: pf state disappearing [ adaptive timeout bug ] From: Nick Rogers To: Matthew Grooms Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 21:35:02 -0000 On Thu, Jan 21, 2016 at 11:44 AM, Matthew Grooms wrote: > On 1/21/2016 11:04 AM, Nick Rogers wrote: > >> On Wed, Jan 20, 2016 at 2:01 PM, Matthew Grooms >> wrote: >> >> All, >>> >>> I have a curious problem with a lightly loaded pair of pf firewall >>> running >>> on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from >>> the state table for no good reason that I can see. The entry limit is set >>> to 100000 and I never see the system go over about 70000 entries, so we >>> shouldn't be hitting the configured limit ... >>> >>> In my experience if you hit the state limit, new connections/states are >> dropped and existing states are unaffected. >> > > Aha! You shook something out of the dusty depths of my slow brain :) I > believe that what you say is true as long as adaptive timeouts are > disabled, which by default they are not ... > > Timeout values can be reduced adaptively as the number of state > ta- > ble entries grows. > > adaptive.start > When the number of state entries exceeds this value, > adaptive > scaling begins. All timeout values are scaled linearly > with > factor (adaptive.end - number of states) / (adaptive.end - > adaptive.start). > adaptive.end > When reaching this number of state entries, all timeout > val- > ues become zero, effectively purging all state entries > imme- > diately. This value is used to define the scale factor, > it > should not actually be reached (set a lower state limit, > see > below). > > Adaptive timeouts are enabled by default, with an adaptive.start > value equal to 60% of the state limit, and an adaptive.end value > equal to 120% of the state limit. They can be disabled by > setting > both adaptive.start and adaptive.end to 0. > >> # pfctl -sm >>> states hard limit 100000 >>> src-nodes hard limit 100000 >>> frags hard limit 50000 >>> table-entries hard limit 200000 >>> >>> # pfctl -si >>> Status: Enabled for 78 days 14:24:18 Debug: Urgent >>> >>> State Table Total Rate >>> current entries 67829 >>> searches 113412118733 16700.2/s >>> inserts 386313496 56.9/s >>> removals 386245667 56.9/s >>> Counters >>> match 441731678 65.0/s >>> bad-offset 0 0.0/s >>> fragment 1090 0.0/s >>> short 220 0.0/s >>> normalize 761 0.0/s >>> memory 0 0.0/s >>> bad-timestamp 0 0.0/s >>> congestion 0 0.0/s >>> ip-option 4366487 0.6/s >>> proto-cksum 0 0.0/s >>> state-mismatch 50334 0.0/s >>> state-insert 10 0.0/s >>> state-limit 0 0.0/s >>> src-limit 0 0.0/s >>> synproxy 0 0.0/s >>> >>> This problem is easy to reproduce by establishing an SSH connection to >>> the >>> firewall itself, letting it sit for a while and then examining the state >>> table. After a connection is made, I can see the entry with an >>> established:established state ... >>> >>> # pfctl -ss | grep X.X.X.X | grep 63446 >>> all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED >>> >>> If I let the SSH session sit for a while and then try to type into the >>> terminal on the client end, the connection stalls and produces a network >>> error message. When I look at the pf state table again, the state entry >>> for >>> the connection is no longer visible. However, the ssh process is still >>> running and I still see the TCP connection established in the output of >>> netstat ... >>> >>> # netstat -na | grep 63446 >>> tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED >>> >>> When I observe the packet flow in TCP dump when a connection stalls, >>> packets being sent from the client are visible on the physical interface >>> but are shown as blocked on the pflog0 interface. >>> >>> Does this happen with non-SSH connections? It sounds like your SSH >> client/server interaction is not performing a keep-alive frequently enough >> to keep the PF state established. If no packets are sent over the >> connection (state) for some time, then PF will timeout (remove) the state. >> At this point your SSH client still believes it has a successful >> connection, so it tries to send packets when you resume typing, but they >> are blocked by your PF rules which likely specify "flags S/SA keep state", >> either explicitly or implicitly (it is the filter rule default), which >> means block packets that don't match an existing state or are not part of >> the initial SYN handshake of the TCP connection. >> > > It happened with UDP SIP and log running HTTP sessions that sit idle as > well. The SSH connection was just the easiest to test. Besides that, the > default TCP timeout value for established connections is quite high at > 86400s. An established TCP connection should be able to sit for a full day > with no traffic before the related state table entry gets evicted. > > Look at your settings in pf.conf for "timeout tcp.established", which >> affects how long before an idle ESTABLISHED state will timeout. Also look >> into ClientAliveInterval in sshd configuration, which I believe is 0 >> (disabled) by default, which means it will let the client timeout without >> sending a keep-alive. If you don't want PF to force timeout an idle SSH >> connection, then ideally ClientAliveInterval is less than or equal (i.e., >> more-frequent) to PF's tcp.established timeout value. >> > > Thanks for the suggestion! I completely forgot about the adaptive timeout > options until I double checked the settings based on you reply :) My values > are set to default for TCP and extended a bit for UDP. The adaptive.start > value was calculated at 60k for the 100k state limit. That in particular > looked way too relevant to be a coincidence. After increasing the value to > 90k, my total state count started increasing and leveled out around 75k. > It's always hovered around 65k up until now, so 10k sate entries were being > discarded on a regular basis ... > Glad I could help somehow. I forgot about adaptive timeout as well and I've been using a much shorter tcp.established timeout than the default for years, so I'm familiar with the "disappearing states" problem. > # pfctl -si > Status: Enabled for 0 days 02:25:41 Debug: Urgent > > State Table Total Rate > current entries 77759 > searches 483831701 55352.0/s > inserts 825821 94.5/s > removals 748060 85.6/s > Counters > match 27118754 3102.5/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 0 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 6655 0.8/s > proto-cksum 0 0.0/s > state-mismatch 0 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -st > tcp.first 120s > tcp.opening 30s > tcp.established 86400s > tcp.closing 900s > tcp.finwait 45s > tcp.closed 90s > tcp.tsdiff 30s > udp.first 600s > udp.single 600s > udp.multiple 900s > icmp.first 20s > icmp.error 10s > other.first 60s > other.single 30s > other.multiple 60s > frag 30s > interval 10s > adaptive.start 90000 states > adaptive.end 120000 states > src.track 0s > > I think there may be a problem with the code that calculates adaptive > timeout values that is making it way too aggressive. If by default it's > supposed to decrease linearly between %60 and %120 of the state table max, > I shouldn't be loosing TCP connections that are only idle for a few minutes > when the sate table is < %70 full. Unfortunately that appears to be the > case. At most this should have decreased the 86400s timeout by %17 to > 72000s for established TCP connections. > That doesn't make sense to me either. Even if the math is off by a factor of 10 the state should live for about 24 minutes. > I've tested this for a few hours now and all my idle SSH sessions have > been rock solid. If anyone else is scratching their head over a problem > like this, I would suggest disabling the adaptive timeout feature or > increasing it to a much higher value. Maybe one of the pf maintainers can > chime in and shed some light on why this is happening. If not, I'm going to > file a bug report as this certainly feels like one. > Did you go with making adaptive timeout less aggressive or disable it entirely? I would think that if adaptive timeout is really that broken more people would notice this problem, especially myself since I have many servers running a very short tcp.established timeout, but the fact that you are noticing this kind of weirdness has me concerned about how the adaptive setting is affecting my environment. > Thanks again, > > -Matthew > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Fri Jan 22 22:02:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E0D6A8DBA6 for ; Fri, 22 Jan 2016 22:02:19 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED804158C for ; Fri, 22 Jan 2016 22:02:18 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.14.7/8.14.7) with ESMTP id u0MLxgiu030641 for ; Fri, 22 Jan 2016 15:59:42 -0600 (CST) (envelope-from mgrooms@shrew.net) Received: from [10.16.32.30] (72-48-144-84.static.grandenetworks.net [72.48.144.84]) by mail.shrew.net (Postfix) with ESMTPSA id 0702318C732 for ; Fri, 22 Jan 2016 15:59:31 -0600 (CST) Subject: Re: pf state disappearing [ adaptive timeout bug ] To: freebsd-net@freebsd.org References: <56A003B8.9090104@shrew.net> <56A13531.8090209@shrew.net> From: Matthew Grooms Message-ID: <56A2A6DA.1040304@shrew.net> Date: Fri, 22 Jan 2016 16:02:02 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx2.shrew.net [10.24.10.11]); Fri, 22 Jan 2016 15:59:42 -0600 (CST) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 22:02:19 -0000 On 1/22/2016 3:35 PM, Nick Rogers wrote: > On Thu, Jan 21, 2016 at 11:44 AM, Matthew Grooms wrote: > >> # pfctl -si >> Status: Enabled for 0 days 02:25:41 Debug: Urgent >> >> State Table Total Rate >> current entries 77759 >> searches 483831701 55352.0/s >> inserts 825821 94.5/s >> removals 748060 85.6/s >> Counters >> match 27118754 3102.5/s >> bad-offset 0 0.0/s >> fragment 0 0.0/s >> short 0 0.0/s >> normalize 0 0.0/s >> memory 0 0.0/s >> bad-timestamp 0 0.0/s >> congestion 0 0.0/s >> ip-option 6655 0.8/s >> proto-cksum 0 0.0/s >> state-mismatch 0 0.0/s >> state-insert 0 0.0/s >> state-limit 0 0.0/s >> src-limit 0 0.0/s >> synproxy 0 0.0/s >> >> # pfctl -st >> tcp.first 120s >> tcp.opening 30s >> tcp.established 86400s >> tcp.closing 900s >> tcp.finwait 45s >> tcp.closed 90s >> tcp.tsdiff 30s >> udp.first 600s >> udp.single 600s >> udp.multiple 900s >> icmp.first 20s >> icmp.error 10s >> other.first 60s >> other.single 30s >> other.multiple 60s >> frag 30s >> interval 10s >> adaptive.start 90000 states >> adaptive.end 120000 states >> src.track 0s >> >> I think there may be a problem with the code that calculates adaptive >> timeout values that is making it way too aggressive. If by default it's >> supposed to decrease linearly between %60 and %120 of the state table max, >> I shouldn't be loosing TCP connections that are only idle for a few minutes >> when the sate table is < %70 full. Unfortunately that appears to be the >> case. At most this should have decreased the 86400s timeout by %17 to >> 72000s for established TCP connections. > That doesn't make sense to me either. Even if the math is off by a factor > of 10 the state should live for about 24 minutes. > >> I've tested this for a few hours now and all my idle SSH sessions have >> been rock solid. If anyone else is scratching their head over a problem >> like this, I would suggest disabling the adaptive timeout feature or >> increasing it to a much higher value. Maybe one of the pf maintainers can >> chime in and shed some light on why this is happening. If not, I'm going to >> file a bug report as this certainly feels like one. >> > Did you go with making adaptive timeout less aggressive or disable it > entirely? I would think that if adaptive timeout is really that broken more > people would notice this problem, especially myself since I have many > servers running a very short tcp.established timeout, but the fact that you > are noticing this kind of weirdness has me concerned about how the adaptive > setting is affecting my environment. I increased the value to 90K for the 10K limit. Yes, it's concerning. Today I setup a test environment at about 1/10th the connections to see if I could reproduce the issue on a smaller scale, but had no luck. I'm trying to find a cmd line test program that will generate enough tcp connections so I can reproduce it on a similar scale to my production environment. So far I haven't found anything that will do the trick. I may end up rolling my own. I'll reply back to the list if I can find a way to reproduce this. Thanks again, -Matthew From owner-freebsd-net@freebsd.org Sat Jan 23 06:12:08 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F40E3A8E450 for ; Sat, 23 Jan 2016 06:12:07 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DA26B100A for ; Sat, 23 Jan 2016 06:12:07 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) by smtp5.hushmail.com (Postfix) with SMTP id 4A26D201CB for ; Sat, 23 Jan 2016 05:34:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=YPxBlJZu+2/3mLZJKHuUnB5l2gQwW5jsMGq2wtGU/6g=; b=YvU21+qyRfd27fzpwcF/j6uDh+e9tu8nlHsc7ZToljXVQrXuFGeTfdxjS44GohDORmc7VhaFvRafUsXAbSd8nllDTXJg2BZsVSk+Lqquv5fyW7kEAPEj1KSkCFlLj/uqtwpvXBYflwB4z3l5aExV8YFbQCobWp0n/TIJlO5riCJUQqMDiJ8dYJZ+pMo3jdm2YpXpu7nxg/rBfF2bGA5hwRqHHxDKPDc48tKOo4l0TvN9aliqhlKxXaaYBV3HTiHt2RP/vKNnIaJbMV+vuDT7iy94x5gE53NzBbwFm1OdzqjrvojsK3Fe19QItOc1uuspkQww0juCH9e2IL/vsPJ2Eg== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp5.hushmail.com (Postfix) with ESMTP for ; Sat, 23 Jan 2016 05:34:28 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 2091EA0121; Sat, 23 Jan 2016 05:34:28 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 03:34:27 -0200 To: freebsd-net@freebsd.org Subject: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123053428.2091EA0121@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 06:12:08 -0000 hello, I am testing a chelsio t520-so-cr connected to a Intel card with ix(4) driver, I can get the ncxl0 interface to transmit at 14Mpps to another chelsio or to a Intel card. However I can only get 800Kpps-1Mpps for RX tests from both chelsio or Intel. I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. I tested it untuned first and later I have applied tuning recommendations I found on BSDRP[1] website. Results still ranging from 800Kpps to 1Mpps for RX. Tests are done w/ with pkt-gen in netmap mode on ncxl interface with both IP address and MAC address source/dest. I have tested ix-ix and I can confirm 14Mpps for both RX and TX directions. I have tested with two different chelsio T520 and both have the very same results. What particular loader/sysctl or ifconfig options I should investigate? I also tested disabling txcsum, rxcsum and TSO. Results are different but still on the much lower mentioned 800K-1M pps rate. thank you [1]http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr From owner-freebsd-net@freebsd.org Sat Jan 23 09:31:16 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A2A3A8CA77 for ; Sat, 23 Jan 2016 09:31:16 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F3C921558 for ; Sat, 23 Jan 2016 09:31:15 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: by mail-ig0-x230.google.com with SMTP id ik10so6738127igb.1 for ; Sat, 23 Jan 2016 01:31:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7gLhiR9Piojg+ti/7rXrso1jM5gMx2zp9Yf70h8GoAk=; b=UzYjoYCCDv+hlet26qs7AfwC69YVKaMe/W8vVvBa29kO6iQpiwQDHNcremEeW/B6+4 No89z7F+uhviVtJuCf+3l3gW/z+kznYVLcN+gyTuQ/IKZ7Mb9RWF+/3rQSY1rFO9Eexe lQjTBpAccA1NFqAXyFO/u8raoyjeDe4sBDdSUJCO8YAPcNR5ToBNO2AhulrfpqaMw4W0 IudQM/WqcqA24CMS1/IfHeTKsxD4QDl2lxEC4hKbzzfsmGEfVhT5oDTwkRS70UY7H4U0 FjthCIxsd9u6rGxz8DH8XCrYdnSbDB9aaet319cPmq7RDYoXQ5CJkNqqlY7Nnuxwd5gR m8gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=7gLhiR9Piojg+ti/7rXrso1jM5gMx2zp9Yf70h8GoAk=; b=ObAkdwSxAk1c/Uxt1GEQ8A9aMSHv/tyckEfCJJ9dlMaPJJ02vawzYnNnvGp5bnOcLY N8+DJqrwChE+asUdOFj2amwppg58pcO9T9jFweZLl0GDL84Mo4fQTWdwesyF9qHuzH4F TzXZnQcrGKCYnU036W+eyzPKMi+738nREvulv8UQgyIzuT9Lp+E8C4KJUmObedLyEi78 vuM5bLX4T2/rlINvgXZ+eyHZyhTDvE8SGi/FR2DIF0rZdxdYONzuP2C8UwBhZYbzmPL0 RSs8kNCfFIn2CmmJfuHVkMJ3ib3Jgv5lnYat+xN9H0fYUYESMVHxcH7XywK/bFo4OuBh TMCg== X-Gm-Message-State: AG10YOTa7aQiFpRqloY48Bmzo7vhr9WS1nOXfLry4wYWGqRlcOpCa/geVaZd3cYfIOFTIW+YvozdAj9pUv0/1w== MIME-Version: 1.0 X-Received: by 10.50.79.231 with SMTP id m7mr7958771igx.15.1453541475340; Sat, 23 Jan 2016 01:31:15 -0800 (PST) Received: by 10.79.36.65 with HTTP; Sat, 23 Jan 2016 01:31:15 -0800 (PST) In-Reply-To: References: Date: Sat, 23 Jan 2016 12:31:15 +0300 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Pavel Odintsov To: Adrian Chadd Cc: Eduardo Meyer , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 09:31:16 -0000 Hi! Great job! Do you have performance estimations? On Wednesday, 20 January 2016, Adrian Chadd wrote: > Ok, so, I mostly did this already: > > https://github.com/erikarn/netmap-tools/ > > it has a multi-threaded, multi-queue bridge + ipv4 decap for testing. > > > > -a > -- Sincerely yours, Pavel Odintsov From owner-freebsd-net@freebsd.org Sat Jan 23 15:29:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83E34A8EF85 for ; Sat, 23 Jan 2016 15:29:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 506E9107D for ; Sat, 23 Jan 2016 15:29:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22d.google.com with SMTP id h5so8741759igh.0 for ; Sat, 23 Jan 2016 07:29:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=e9q/kmiigzoL2qh80eM7ryMeNg4JocjTqzBY5gJj/nA=; b=KBdt/fUx3vnPtU1GZiMu6FTokoGA5VICqKt7a/y+J5GdUlcWztKma6W0rG4ZRs0CpR AI11daU9vAz8Aw+JUsVfKHu8zSyWgKjt/0GQFkrytBUaEo9N4LVmYcvWUbHlNLJibKLp 1GXPtfNDWidQE4TAfgZev/Hp5JKQC5K0jd7dDFpC0QRYkV7SpCNsxTygk3rnk8wGheIH YlsZBgwMgv7rI5BcbcBofrz/npXzTsXkrdDeDupPSZ+tLv9BhOnrlLNqJv7N0jCG+dkd EFlGHBOtJwCqxKzs0TdJPh+wMiyZm4mBj7ZXe38qr/awlaTsETASRcB8t4QMs0D+Rq1a Wh1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=e9q/kmiigzoL2qh80eM7ryMeNg4JocjTqzBY5gJj/nA=; b=cybeQidHole5Png0lWuKmY+UZ450uc2qAhZtJ0sQaKJORtP+jxivrs0MtuS89P+Bn+ AvtR5+dIv9y09FHJMWFMox9D+TbXr91hRt6Gf1miyz7BY8kvSRvXw6l+f2GQrbLZS8BM QqJRf5cgM2N4jCsKa+UOoa62xzp4nZ0UjX+0aKEejlcsT8tZmyE8wcvFFJTtf111a7e6 rtbW/Ayxk3brEN9DLaYXqjA7z7kA7zbRYVYbKfrqs5bKypYZUnGWXemBoq51Xk4RAZW/ HBIn861QceoEp/g8jUjnENTou/bmAc6CbSPXmft6xBmh4beSL6CMVcQc84BelRYzWrTw wApQ== X-Gm-Message-State: AG10YORnA/Ewf67Awl5euTC6NXk/x1cgVmD9dows86jVXEfTWfXIjIGc6E6/Miv1MidfGaYmkWL07Es6bM2CcQ== MIME-Version: 1.0 X-Received: by 10.50.150.36 with SMTP id uf4mr8539149igb.61.1453562978728; Sat, 23 Jan 2016 07:29:38 -0800 (PST) Received: by 10.36.121.16 with HTTP; Sat, 23 Jan 2016 07:29:38 -0800 (PST) In-Reply-To: <20160123053428.2091EA0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> Date: Sat, 23 Jan 2016 07:29:38 -0800 Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Adrian Chadd To: Marcus Cenzatti Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 15:29:39 -0000 What are you doing for RX? More netmap? Or the normal stack? -a On 22 January 2016 at 21:34, Marcus Cenzatti wrote: > hello, > > I am testing a chelsio t520-so-cr connected to a Intel card with ix(4) driver, I can get the ncxl0 interface to transmit at 14Mpps to another chelsio or to a Intel card. However I can only get 800Kpps-1Mpps for RX tests from both chelsio or Intel. > > I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. > > I tested it untuned first and later I have applied tuning recommendations I found on BSDRP[1] website. Results still ranging from 800Kpps to 1Mpps for RX. > > Tests are done w/ with pkt-gen in netmap mode on ncxl interface with both IP address and MAC address source/dest. > > I have tested ix-ix and I can confirm 14Mpps for both RX and TX directions. I have tested with two different chelsio T520 and both have the very same results. > > What particular loader/sysctl or ifconfig options I should investigate? > > I also tested disabling txcsum, rxcsum and TSO. Results are different but still on the much lower mentioned 800K-1M pps rate. > > thank you > > [1]http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Sat Jan 23 15:31:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67BDFA8E083 for ; Sat, 23 Jan 2016 15:31:23 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3633512C6 for ; Sat, 23 Jan 2016 15:31:23 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22e.google.com with SMTP id ik10so9733359igb.1 for ; Sat, 23 Jan 2016 07:31:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZkBA5PljA42QcjLgtuRKCrdPutpAqCZKrFFE/OfRUhY=; b=yfGAuwBFk5WpitpuOjQFzhnRZoocttP37cdWFquIXG2HRrFbdHQzXcbTp5ttfnI9Ok ioRbumrPfp2gsR8LSeb/btt1zP41nuH+/1SrI02hOF6QiHWosgabH1dZ1ge4Pk/saNka OF9LBdkE8NqPW74nvb/clEwHlCSTvaQWdDjqQLy+s82tX+4OxXbPBIySDUrdPzuGiM10 UEiOYJMf2cHqrd5aHEwz4DxxT/9SMGM5XolvFVLgu7EIUNoPR3ZtOLc44mMb1WeC+0dq OFGUfsTDWLb/X7XpL+viOh4x6aDuzQD9PAi+cfs9tNKc34Dmh8hNbsbLTM53A77TZbzi 4vug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZkBA5PljA42QcjLgtuRKCrdPutpAqCZKrFFE/OfRUhY=; b=XqK1XyLDodn0NoSxsjXGWBnSDRb0Ke2O/stJdGXJEc0G5ZgnSYBhC5OxeJQ6I6PlgZ sTOj20g3XOi/755v3f8abehhhWG4KvcV6oXJ2wrZdSL2i5njYV2wY7tPfyS3/DdBOAwk bGvlnUu6Uv3lUHFzooTt0HgY4kqGLAxDpklbXFT7PCxBlphc4HMt6RmWa3Xos9Ry9sDs Q35CE3ajEEIbKambNYHcmvebFDj8zsVDaBGr18jEU3UXKHfBZmiimWd4zfrGqbCbPdOJ 8fy1zvUKKXfQeQ5RHUr/wu63AJiSEyIgK9BTaSpLMK7ly/kcWM6E0WB7KOdArIqs25Qk apPg== X-Gm-Message-State: AG10YORViX8OMwLjjSMsle4JpKGgwv5VL0V/Mt5gtgGqQMOkEtI9wagPUj7LmNqZ2bR07w4LLRm12hro4Ajl8Q== MIME-Version: 1.0 X-Received: by 10.50.178.178 with SMTP id cz18mr9348474igc.37.1453563082688; Sat, 23 Jan 2016 07:31:22 -0800 (PST) Received: by 10.36.121.16 with HTTP; Sat, 23 Jan 2016 07:31:22 -0800 (PST) In-Reply-To: References: Date: Sat, 23 Jan 2016 07:31:22 -0800 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Adrian Chadd To: Pavel Odintsov Cc: Eduardo Meyer , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 15:31:23 -0000 For random src/dst ports and IPs and on the chelsio t5 40gig hardware, I was getting what, uhm, 40mil tx pps and around 25ish mil rx pps? The chelsio rx path really wants to be coalescing rx buffers, which the netmap API currently doesn't support. I've no idea if luigi has plans to add that. So, it has the hilarious side effect of "adding more RX queues" translates to "drops in RX performance." :( Thanks, -a On 23 January 2016 at 01:31, Pavel Odintsov wrote: > Hi! > > Great job! Do you have performance estimations? > > > On Wednesday, 20 January 2016, Adrian Chadd wrote: >> >> Ok, so, I mostly did this already: >> >> https://github.com/erikarn/netmap-tools/ >> >> it has a multi-threaded, multi-queue bridge + ipv4 decap for testing. >> >> >> >> -a > > > > -- > Sincerely yours, Pavel Odintsov From owner-freebsd-net@freebsd.org Sat Jan 23 15:40:59 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211A2A8E3EC for ; Sat, 23 Jan 2016 15:40:59 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E70BE194E for ; Sat, 23 Jan 2016 15:40:58 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x229.google.com with SMTP id ho8so57167704pac.2 for ; Sat, 23 Jan 2016 07:40:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=qIqrKnp0o/lMZdW9DqC22OH3Ot6VXK9iQCyRE93gkMA=; b=uYAdIjPd62HBSg1o1uGVw/nJTbI6VTAhCowFfSVKi9TOot60wVUq7zM584/cYixenI 1bBu+xIHfyWXGJ66D+bFvUmoufUcnER+7wKHfNNGQUFZ1mVEYe2AkNzYMxYqF4q3lrXh wIBe9fMMjfb5ZcBvW4saLBDhCV5apgGYeaoV6Oak8P9PegUHaQrl/7x/JNTzwij2uVT+ AQbIdoHS5pZCq/BkYEHL8SY2YFKeityMEuiWtlcuSbNrW9lZpGlHvuonkLfUqxIdr81w pllvw5ddPoAUu9M/6Lxeyqybuiai6m81+XyOhnayfFz0/br0fZWi7/Xr4ClyM7+us89Z jtSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=qIqrKnp0o/lMZdW9DqC22OH3Ot6VXK9iQCyRE93gkMA=; b=bynJPWPf4WqvGg8hNPCiEwl2q+wCHbhyYVlIpgyZ7LZhIOJ4w6PKn6PlHD3YbV/CWO TDEeFUB4G9pa8Xw/N3BSRu8GV61eveCFK9mUu7StZjQIiaGbiwziFVRYG9Hw6pAh/tUd CiuAftuhXiqecj98Pi8tfVKp+4O2pxmDPpntvOCFAUzJXg+NQt+x8anktk5ACX6A9Dae 3fdfsM6BSPR9PQjQK/yGkiKd7b7tUb1JIH4sOObdAAm/1e2l865hQzwzNbH76yMVygdN 1hvWOG4mfzWlJs7HYU+338e4RLLG66XDuuDyIYrRWaouPlB6NQmUsFu1Ea+hXyrOhYb6 Khxw== X-Gm-Message-State: AG10YOTecjkbpl6Yn0AOI4C38mrwqqvKaT5LjM/SIObcteoktpSobZq6mEkt0HMVk50xkA== X-Received: by 10.67.1.164 with SMTP id bh4mr12859513pad.118.1453563658591; Sat, 23 Jan 2016 07:40:58 -0800 (PST) Received: from ox ([2601:641:c001:8a00:591d:d471:ff4a:b8bf]) by smtp.gmail.com with ESMTPSA id qj8sm17160300pac.40.2016.01.23.07.40.56 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 23 Jan 2016 07:40:57 -0800 (PST) Date: Sat, 23 Jan 2016 07:40:52 -0800 From: Navdeep Parhar To: Marcus Cenzatti Cc: freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123154052.GA4574@ox> Mail-Followup-To: Marcus Cenzatti , freebsd-net@freebsd.org References: <20160123053428.2091EA0121@smtp.hushmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160123053428.2091EA0121@smtp.hushmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 15:40:59 -0000 On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: > hello, > > I am testing a chelsio t520-so-cr connected to a Intel card with ix(4) > driver, I can get the ncxl0 interface to transmit at 14Mpps to another > chelsio or to a Intel card. However I can only get 800Kpps-1Mpps for > RX tests from both chelsio or Intel. > > I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. > > I tested it untuned first and later I have applied tuning > recommendations I found on BSDRP[1] website. Results still ranging > from 800Kpps to 1Mpps for RX. > > Tests are done w/ with pkt-gen in netmap mode on ncxl interface with > both IP address and MAC address source/dest. The ncxl interfaces have their own MAC addresses. Make sure the sender uses the MAC of the receiver's ncxl interface as the destination MAC. (netmap's pkt-gen -f tx transmits L2 broadcasts by default). Check for PAUSE frames coming out of the receiver (sysctl dev.cxl | grep tx_pause). If it's receiving frames on netmap interface the tx_pause counter should not move. Regards, Navdeep > > I have tested ix-ix and I can confirm 14Mpps for both RX and TX > directions. I have tested with two different chelsio T520 and both > have the very same results. > > What particular loader/sysctl or ifconfig options I should > investigate? > > I also tested disabling txcsum, rxcsum and TSO. Results are different > but still on the much lower mentioned 800K-1M pps rate. > > thank you > > [1]http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Sat Jan 23 15:52:12 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11FC2A8E92D for ; Sat, 23 Jan 2016 15:52:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 01BD910DC for ; Sat, 23 Jan 2016 15:52:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NFqBk7086261 for ; Sat, 23 Jan 2016 15:52:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206528] Emulex LPe 16002 FC HBA Not Recognized by oce(4) driver Date: Sat, 23 Jan 2016 15:52:11 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: feature, needs-patch, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? X-Bugzilla-Changed-Fields: cc bug_severity keywords flagtypes.name short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 15:52:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206528 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- CC|freebsd-amd64@FreeBSD.org |freebsd-net@FreeBSD.org Severity|Affects Only Me |Affects Some People Keywords| |feature, needs-patch, | |needs-qa Flags| |mfc-stable10? Summary|Emulex LPe 16002 FC HBA Not |Emulex LPe 16002 FC HBA Not |Recognized by oce driver |Recognized by oce(4) driver --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 17:13:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FDC8A8EAAE for ; Sat, 23 Jan 2016 17:13:02 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 568BF1293 for ; Sat, 23 Jan 2016 17:13:01 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) by smtp3.hushmail.com (Postfix) with SMTP id 37272E0264 for ; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=hVKRM2Erm5dv9jzviRkO4K6UzdLVheBEupEpBAPByUc=; b=gNIeGVsMQLQbbnoFFTt28F/9Z/T9wg5gOoocpDW73vlHqq5+6DEmfcL3tk3KDFzjVwlMumPjT0Ysea3/Y+p8YoquUedXG2jdlykzvUI+1nFew8hq2QxeriEJ6C2UPPdwCglxmJK9QoOfbMVWYCAOv/HMH3/9LBwacwsZamE8311Turm6tw0mkhCq12B6H1Y2UbTXYEvpQdSDoPadRP2yKdPfxmYCCmALZ4pkGwQpL4vz01odMwmDMq9u6Mxcip2aCon7gkCZT00xLSFbhn5H2Lx9/FAftstV7L5SzRQZ2SO+eUGpuXDfLjbykT6UM0vrbvaSvujF0I38w9d4SneNaQ== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp3.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 0F448A0121; Sat, 23 Jan 2016 17:13:00 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 15:12:59 -0200 To: "Navdeep Parhar" Cc: freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: <20160123154052.GA4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123171300.0F448A0121@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:13:02 -0000 On 1/23/2016 at 1:40 PM, "Navdeep Parhar" wrote: > >On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >> hello, >> >> I am testing a chelsio t520-so-cr connected to a Intel card with >ix(4) >> driver, I can get the ncxl0 interface to transmit at 14Mpps to >another >> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps >for >> RX tests from both chelsio or Intel. >> >> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >> >> I tested it untuned first and later I have applied tuning >> recommendations I found on BSDRP[1] website. Results still >ranging >> from 800Kpps to 1Mpps for RX. >> >> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >with >> both IP address and MAC address source/dest. > >The ncxl interfaces have their own MAC addresses. Make sure the >sender >uses the MAC of the receiver's ncxl interface as the destination >MAC. >(netmap's pkt-gen -f tx transmits L2 broadcasts by default). > >Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >| grep >tx_pause). If it's receiving frames on netmap interface the >tx_pause >counter should not move. > >Regards, >Navdeep > hello, yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests: ===> BOX #1 CHELSIO chelsio# ifconfig -v ncxl0 ncxl0: flags=8843 metric 0 mtu 1500 ether 00:07:43:33:8d:c1 inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 nd6 options=29 media: Ethernet 10Gbase-SR status: active chelsio# ifconfig -v cxl0 cxl0: flags=8843 metric 0 mtu 1500 options=ec00bb ether 00:07:43:33:8d:c0 nd6 options=29 media: Ethernet 10Gbase-SR status: active plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28 module temperature: 42.79 C Voltage: 3.23 Volts RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba 311.132189 main [1715] interface is ncxl0 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 311.191286 main [1910] mapped 334980KB at 0x801c00000 Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus. 311.191371 main [1996] Wait 2 secs for phy reset 313.230423 main [1998] Ready... 313.230570 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1 313.247998 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3 314.249172 receiver_body [1254] waiting for initial packets, poll returns 0 0 (...) when the other box starts transmitting: (...) 380.492236 main_thread [1512] 0 pps (0 pkts in 1006897 usec) 381.326159 receiver_body [1254] waiting for initial packets, poll returns 0 0 381.493141 main_thread [1512] 80110 pps (80137 pkts in 1000906 usec) 382.494145 main_thread [1512] 801585 pps (801888 pkts in 1001004 usec) 383.494979 main_thread [1512] 801381 pps (801632 pkts in 1000833 usec) 384.497148 main_thread [1512] 801490 pps (802144 pkts in 1002169 usec) 385.497650 main_thread [1512] 801416 pps (801568 pkts in 1000503 usec) 386.502147 main_thread [1512] 801110 pps (802464 pkts in 1004497 usec) 387.502853 main_thread [1512] 800844 pps (801056 pkts in 1000705 usec) 388.503651 main_thread [1512] 801519 pps (801760 pkts in 1000799 usec) 389.504654 main_thread [1512] 801394 pps (801696 pkts in 1001002 usec) 184.339250 sender_body [1179] pending tx tail 1433 head 51 on ring 0 ===> BOX #2 Intel: on the other box: intel# ifconfig ix0 ix0: flags=28943 metric 0 mtu 1500 options=8407b8 ether 00:07:e9:44:d2:ba inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255 nd6 options=29 media: Ethernet Other instance 8 (10Gbase-SR ) status: active intel# ./pkt-gen -i ix0 -f tx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba 267.767848 main [1715] interface is ix0 267.767990 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 267.768006 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 267.872796 main [1910] mapped 334980KB at 0x801c00000 Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus. 00 -> 00 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) 267.872867 main [1994] Sending 512 packets every 0.000000000 s 267.872872 main [1996] Wait 2 secs for phy reset 269.878594 main [1998] Ready... 269.878716 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1 269.896132 sender_body [1073] start, fd 4 main_fd 3 269.930589 sender_body [1142] drop copy 270.907474 main_thread [1512] 14483889 pps (14648527 pkts in 1011367 usec) 271.910479 main_thread [1512] 14882743 pps (14927466 pkts in 1003005 usec) 272.910975 main_thread [1512] 14879303 pps (14886683 pkts in 1000496 usec) 273.912467 main_thread [1512] 14879189 pps (14901389 pkts in 1001492 usec) 274.912970 main_thread [1512] 14881027 pps (14888527 pkts in 1000504 usec) ===> BOX #1 Chelsio, Other Terminal chelsio# while true ;do sysctl dev.cxl | grep tx_pause ; sleep 1 ; done dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 dev.cxl.1.stats.tx_pause: 0 dev.cxl.0.stats.tx_pause: 0 (...) counters for tx_pause keep unchanged. tried with and without rxcsum/txcsu/tso4 what else should i try? From owner-freebsd-net@freebsd.org Sat Jan 23 17:24:57 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8AD60A8EF9A for ; Sat, 23 Jan 2016 17:24:57 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp2.hushmail.com (smtp2a.hushmail.com [65.39.178.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6BD7C1A54 for ; Sat, 23 Jan 2016 17:24:57 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp2.hushmail.com (smtp2a.hushmail.com [65.39.178.237]) by smtp2.hushmail.com (Postfix) with SMTP id F13ABA01B8 for ; Sat, 23 Jan 2016 17:24:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=jDFLBUlaSIl+ZT/wy2NlJfc8/FDM6Z2scTwwR9eTlq8=; b=CbYniPH2mzQwnCTTHUIVc3GZiFGqqscjTgbg6qf8HnWg2zw74HeqepsQpyHO2CKsIMJC2DqKNAmVKVS2dDtk1AvCu29E2zh4sZHp4UyxaDt5vgoBUJ8eQRQX7PEiqKGOzf1a81xI61mCSC6I3gkzIiG3be30gaggZMI+jBJXDGM6UCN92gkl2xI/IuvuOmIO+1+bWdQxiukh22ZITohOVVbQLPnFT/ejqCLeoPZFoRCYeRuHFp9MOeR1+Rn38FKmk+s0d2Nkz+WIWmxZYTWiwaTRNlwfJ4dCFI9FZ6c3xp+4F0AKI0nWbzbNRZbYfyEdcGM3hppwDgsLro/R0bB3aQ== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp2.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 17:24:55 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id C1A25A0121; Sat, 23 Jan 2016 17:24:55 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 15:24:55 -0200 To: "Adrian Chadd" , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: References: <20160123053428.2091EA0121@smtp.hushmail.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123172455.C1A25A0121@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:24:57 -0000 On 1/23/2016 at 1:29 PM, "Adrian Chadd" wrote: > >What are you doing for RX? More netmap? Or the normal stack? yes, netmap w/ pkt-gen -f rx, I just sent a transcript for a testing session in my previous e-mail From owner-freebsd-net@freebsd.org Sat Jan 23 17:35:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 437A4A8E349 for ; Sat, 23 Jan 2016 17:35:39 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAD4C1EDE for ; Sat, 23 Jan 2016 17:35:38 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x22a.google.com with SMTP id oh2so55910012lbb.3 for ; Sat, 23 Jan 2016 09:35:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=bkiZls4LqG8CK2b7nQoYaGkGoqFTLXFOusicrZEykoQ=; b=UBd8CddEVrPpc88ch45NZ9wZqyxIE/1T3zLVn1YzGJrdfVyvuU80n2KeQ05Ek4I5mM bw3U6h0Yj4rkaC9e1j+YaoJVckAPvypICttS8f5WsWAjl+SfSI0QFji/f9jW3UHCLenZ bM/3whudqEyxaNpQqX0DvoPw6hZuqEzX5yANoOa5es6kkR2HHKLA3tW8JMMKafQaSr2A khH8iN6jAv3L4E/BF6reM4sNmLR89vGN7nHAPHh+1B2zlBmee9hV2gSjc4Y18h7GftpP CLTNicmNnoY7vZ6i6DkPsTyJfC6mE0y4cEfzAAQFQPwLAT8bfRSxUyzRs2eb3gjG2kBc GeZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=bkiZls4LqG8CK2b7nQoYaGkGoqFTLXFOusicrZEykoQ=; b=hXS1ctTzA64SXL12hqY8UKtHWAXIFh5qH7tiV4B448zCX2LqPBSjXQaBiiZs6aL8ib 5xE6X3Vs8DVurZLvb4LTU+/rGKquIybksc3Ba1Ks8ZiOkTFhOIid/rxuxoI7m3qsfcAj 2iZx1vHKayIGysg+bEbytAeHyd+xFTd3Ex05sibSVjLyoBSzPXp2KMsp1+0jRmqQf8kH ZDs4YJcuHTMoGlxXe752nQXzO5jGvnIs2/JOaPC4jtg8BzGbs1oBBUCiMFWmgdDPxGRU q6iCVRdQ2fAZX+Jd4trC/gSMR34hAfQc/boV1YWjhFy/uUjh3e5f1YtrBg7V9dDKHhsp oCrA== X-Gm-Message-State: AG10YOQtfHzdPgAnXAM7owRoWU9boR9rlmoDVZCRMG/INVbgQmiTn3gW8i9H+XmPF1T2uPojupSxWsUP5NBIGw== MIME-Version: 1.0 X-Received: by 10.112.13.99 with SMTP id g3mr3284924lbc.86.1453570535803; Sat, 23 Jan 2016 09:35:35 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 09:35:35 -0800 (PST) In-Reply-To: <20160123171300.0F448A0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> Date: Sat, 23 Jan 2016 09:35:35 -0800 X-Google-Sender-Auth: 8YPAKAHihL7_9uVxYBQ7VbG9JJo Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: Navdeep Parhar , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:35:39 -0000 On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti wrote: > > > On 1/23/2016 at 1:40 PM, "Navdeep Parhar" wrote: >> >>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>> hello, >>> >>> I am testing a chelsio t520-so-cr connected to a Intel card with >>ix(4) >>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>another >>> chelsio or to a Intel card. However I can only get 800Kpps-1Mpps >>for >>> RX tests from both chelsio or Intel. >>> >>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>> >>> I tested it untuned first and later I have applied tuning >>> recommendations I found on BSDRP[1] website. Results still >>ranging >>> from 800Kpps to 1Mpps for RX. >>> >>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>with >>> both IP address and MAC address source/dest. >> >>The ncxl interfaces have their own MAC addresses. Make sure the >>sender >>uses the MAC of the receiver's ncxl interface as the destination >>MAC. >>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >> >>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>| grep >>tx_pause). If it's receiving frames on netmap interface the >>tx_pause >>counter should not move. >> >>Regards, >>Navdeep >> > > hello, > > yes, MAC addresses are correct, I did the tests again and tx_pause won't move, here is the full transcript for the tests: > > ===> BOX #1 CHELSIO > > chelsio# ifconfig -v ncxl0 > ncxl0: flags=8843 metric 0 mtu 1500 > ether 00:07:43:33:8d:c1 > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 > nd6 options=29 > media: Ethernet 10Gbase-SR > status: active > > chelsio# ifconfig -v cxl0 > cxl0: flags=8843 metric 0 mtu 1500 > options=ec00bb > ether 00:07:43:33:8d:c0 > nd6 options=29 > media: Ethernet 10Gbase-SR > status: active > plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K DATE: 2011-06-28 > module temperature: 42.79 C Voltage: 3.23 Volts > RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) > > chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba > 311.132189 main [1715] interface is ncxl0 > 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 > 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 wait, the lower case -s and -d are for IP addresses, you need to use -S and -D for the MAC addresses. This way you are sending broadcasts, which likely means that the chelsio is replicating the packets to both the netmap and the regular port and the latter (which perhaps comes first) is likely dropping packets. cheers luigi From owner-freebsd-net@freebsd.org Sat Jan 23 17:41:47 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C594A8E67A for ; Sat, 23 Jan 2016 17:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F16D812A9 for ; Sat, 23 Jan 2016 17:41:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NHfkDP069102 for ; Sat, 23 Jan 2016 17:41:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206478] Setting laggproto fails on 10.2 Date: Sat, 23 Jan 2016 17:41:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: araujo@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: araujo@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:41:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206478 Marcelo Araujo changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |araujo@FreeBSD.org Assignee|freebsd-net@FreeBSD.org |araujo@FreeBSD.org --- Comment #4 from Marcelo Araujo --- I'm gonna take it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 17:48:41 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9247A8E919 for ; Sat, 23 Jan 2016 17:48:41 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E74315BC for ; Sat, 23 Jan 2016 17:48:40 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) by smtp3.hushmail.com (Postfix) with SMTP id 62C22E0418 for ; Sat, 23 Jan 2016 17:48:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=ScmR4XcRoL3wcuDmX34hD2FaWjS+//mFwbTtyLtCZS0=; b=D1J7uizHcECBNuwsQ77AdrQ2Z99BiKYwP/9+DrevF5xJ2uOvwSSNu6ZuqM62fCh6nl/FKFqds0F/udV/umylmGhhwus9Co/X58ovSWOp/AdDaDzSgRC/ooHktSxXr1LkA4S1fV8i6x0qa/DMpwz/+KcWJaTU0tkAv9VDPwZ796y5r2W7seF7g1M2ZjW5seGO4XZmTrGOVg9BSuDv26/fm0Hi5YhpoMwzuSRrgNc/bc9a4xfar/b6lhva2mvqQaDnlNMOXMqpP0tR+dcRe0LyOy4qkDLpmCcb7Dyf08sB9HGXRyZ6qfJNaYCiWKJ5JmU3D38Oqa3BJXGnjwEKYCS0ew== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp3.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 17:48:40 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 32B1DA0121; Sat, 23 Jan 2016 17:48:40 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 15:48:39 -0200 To: "Luigi Rizzo" Cc: "Navdeep Parhar" , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123174840.32B1DA0121@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 17:48:41 -0000 On 1/23/2016 at 3:35 PM, "Luigi Rizzo" wrote: > >On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti > wrote: >> >> >> On 1/23/2016 at 1:40 PM, "Navdeep Parhar" >wrote: >>> >>>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>>> hello, >>>> >>>> I am testing a chelsio t520-so-cr connected to a Intel card >with >>>ix(4) >>>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>>another >>>> chelsio or to a Intel card. However I can only get 800Kpps- >1Mpps >>>for >>>> RX tests from both chelsio or Intel. >>>> >>>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>>> >>>> I tested it untuned first and later I have applied tuning >>>> recommendations I found on BSDRP[1] website. Results still >>>ranging >>>> from 800Kpps to 1Mpps for RX. >>>> >>>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>>with >>>> both IP address and MAC address source/dest. >>> >>>The ncxl interfaces have their own MAC addresses. Make sure the >>>sender >>>uses the MAC of the receiver's ncxl interface as the destination >>>MAC. >>>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >>> >>>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>>| grep >>>tx_pause). If it's receiving frames on netmap interface the >>>tx_pause >>>counter should not move. >>> >>>Regards, >>>Navdeep >>> >> >> hello, >> >> yes, MAC addresses are correct, I did the tests again and >tx_pause won't move, here is the full transcript for the tests: >> >> ===> BOX #1 CHELSIO >> >> chelsio# ifconfig -v ncxl0 >> ncxl0: flags=8843 metric >0 mtu 1500 >> ether 00:07:43:33:8d:c1 >> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >> nd6 options=29 >> media: Ethernet 10Gbase-SR >> status: active >> >> chelsio# ifconfig -v cxl0 >> cxl0: flags=8843 metric >0 mtu 1500 >> >options=ec00bb_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> >> ether 00:07:43:33:8d:c0 >> nd6 options=29 >> media: Ethernet 10Gbase-SR >> status: active >> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K >DATE: 2011-06-28 >> module temperature: 42.79 C Voltage: 3.23 Volts >> RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) >> >> chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s >00:07:e9:44:d2:ba >> 311.132189 main [1715] interface is ncxl0 >> 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to >0.0.0.0:90 >> 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 > > >wait, the lower case -s and -d are for IP addresses, >you need to use -S and -D for the MAC addresses. >This way you are sending broadcasts, which likely >means that the chelsio is replicating the packets >to both the netmap and the regular port and the >latter (which perhaps comes first) is likely >dropping packets. > >cheers >luigi woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. curiously, I have always used -s/-d with IP addresses on ix-ix testing this is why I never noticed the case, since ix always received 14Mpps, but you probably explained it since ix has one single deviceport per wire, hence the different behavior performance stills very low when compared to TX and to what is expected thank you for noticing the case From owner-freebsd-net@freebsd.org Sat Jan 23 18:00:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93F79A8F0F6 for ; Sat, 23 Jan 2016 18:00:50 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22a.google.com (mail-lf0-x22a.google.com [IPv6:2a00:1450:4010:c07::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09ECC1D98 for ; Sat, 23 Jan 2016 18:00:50 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x22a.google.com with SMTP id c192so64013444lfe.2 for ; Sat, 23 Jan 2016 10:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=8wI8braWB3xNC9XcjSNGFL+db4thxYNlR5fZRJGxvEc=; b=chlwGFn2bAtbM9FbGJDx6epGxlQG9W6hptKLtWLb/btfRu+arKKR72Zhm4iSydOyj9 vbH8T+hLYqTBLwYZNeiF+Ujh5wYgmSRVzj4D2rK0nyrmJdtZabs5q1cpmHtdaYiSfJiw H3QpAu18rn1fZrwjo2/3QGZafqVndPmpbntlNcCTdRvEneXQpUOq4Ux2+cGbC4QpENIY UX554unETODrievLS6pXZ83/vniHj1vkZh0tSx8fNdLzX4ekq2KZ40TOHbtaKVEUcPxD A0ts8pS7EiZYFbSbDLYkdd8gj//yNqVj4jUkKKupHiqCUBFO2uNXHpQ5YAfqGRt6gPIk cW2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8wI8braWB3xNC9XcjSNGFL+db4thxYNlR5fZRJGxvEc=; b=QJTxWG9qV763DWRbjO+w7Ko9FopA0ZlsUuO9wUbZdA0T53WEPCtn+FrlC2ufcoadbJ eGf65+wzZEBgGcZidEfTftbU5BZe09AHHBAYReE6mjbvbGn8zbyzy2sG11vuIMs7eZ+b 8PBI8SquBPdu6bYuBvb4lJhLqEXl3iM5VgDpgt4L1cMcKp9vuek1WIlrD3rFUMcQKNSy WU2WKi+WU6LtbT/wbrohXSCgZdO8RageIPElQZQJaj80d8QNk+1OYSL/v/gXX8SN+mqQ 1uOYczlU8N1mCFnmln3fy6dT7YQ0A0z5iOVeVsjUfKTpxo5I+IiKNcJYhE8h7EZYsE4d 4nnw== X-Gm-Message-State: AG10YOTA5mwaHdb93aQ8gA1x6zgapzrJmj8d8E3i5wRx4aNkXjWl0apMyOvapwG+FT6KSGIQJMAN62R8/hLgGg== MIME-Version: 1.0 X-Received: by 10.25.158.136 with SMTP id h130mr3382298lfe.136.1453572048078; Sat, 23 Jan 2016 10:00:48 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 10:00:48 -0800 (PST) In-Reply-To: <20160123174840.32B1DA0121@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> Date: Sat, 23 Jan 2016 10:00:48 -0800 X-Google-Sender-Auth: jQ5WIH6dSN45Lgg4HXYp0yDk2YY Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: "freebsd-net@freebsd.org" , Navdeep Parhar Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:00:50 -0000 On Sat, Jan 23, 2016 at 9:48 AM, Marcus Cenzatti wrote: > > > On 1/23/2016 at 3:35 PM, "Luigi Rizzo" wrote: >> >>On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti >> wrote: >>> >>> >>> On 1/23/2016 at 1:40 PM, "Navdeep Parhar" >>wrote: >>>> >>>>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti wrote: >>>>> hello, >>>>> >>>>> I am testing a chelsio t520-so-cr connected to a Intel card >>with >>>>ix(4) >>>>> driver, I can get the ncxl0 interface to transmit at 14Mpps to >>>>another >>>>> chelsio or to a Intel card. However I can only get 800Kpps- >>1Mpps >>>>for >>>>> RX tests from both chelsio or Intel. >>>>> >>>>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>>>> >>>>> I tested it untuned first and later I have applied tuning >>>>> recommendations I found on BSDRP[1] website. Results still >>>>ranging >>>>> from 800Kpps to 1Mpps for RX. >>>>> >>>>> Tests are done w/ with pkt-gen in netmap mode on ncxl interface >>>>with >>>>> both IP address and MAC address source/dest. >>>> >>>>The ncxl interfaces have their own MAC addresses. Make sure the >>>>sender >>>>uses the MAC of the receiver's ncxl interface as the destination >>>>MAC. >>>>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >>>> >>>>Check for PAUSE frames coming out of the receiver (sysctl dev.cxl >>>>| grep >>>>tx_pause). If it's receiving frames on netmap interface the >>>>tx_pause >>>>counter should not move. >>>> >>>>Regards, >>>>Navdeep >>>> >>> >>> hello, >>> >>> yes, MAC addresses are correct, I did the tests again and >>tx_pause won't move, here is the full transcript for the tests: >>> >>> ===> BOX #1 CHELSIO >>> >>> chelsio# ifconfig -v ncxl0 >>> ncxl0: flags=8843 metric >>0 mtu 1500 >>> ether 00:07:43:33:8d:c1 >>> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >>> nd6 options=29 >>> media: Ethernet 10Gbase-SR >>> status: active >>> >>> chelsio# ifconfig -v cxl0 >>> cxl0: flags=8843 metric >>0 mtu 1500 >>> >>options=ec00bb>_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> >>> ether 00:07:43:33:8d:c0 >>> nd6 options=29 >>> media: Ethernet 10Gbase-SR >>> status: active >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >>> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K >>DATE: 2011-06-28 >>> module temperature: 42.79 C Voltage: 3.23 Volts >>> RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) >>> >>> chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s >>00:07:e9:44:d2:ba >>> 311.132189 main [1715] interface is ncxl0 >>> 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to >>0.0.0.0:90 >>> 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 >> >> >>wait, the lower case -s and -d are for IP addresses, >>you need to use -S and -D for the MAC addresses. >>This way you are sending broadcasts, which likely >>means that the chelsio is replicating the packets >>to both the netmap and the regular port and the >>latter (which perhaps comes first) is likely >>dropping packets. >> >>cheers >>luigi > > woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > curiously, I have always used -s/-d with IP addresses on ix-ix testing this is why I never noticed the case, since ix always received 14Mpps, but you probably explained it since ix has one single deviceport per wire, hence the different behavior > > performance stills very low when compared to TX and to what is expected ok so next we can try and see what else is going on. please check the following: a) are you connected through a switch ? if so, try to send out some packets through the ncxl0 port (using pkt-gen and its native MAC address) so the switch can learn the address and does not need to replicate traffic on all ports (which generally is done at a limited rate). b) see if using different packet sizes (say 256, 512, 1024, 1500 passed as the -l option to pkt-gen) affects the rx rate. If the rate does not change (except for 1500 bytes) it may be a problem with interrupt moderation c) use progressively increasing packet rates on the sender, using -R xxxx (start at 500000 packets per second, and then go up until the receiver cannot sustain the tx rate. d) use a smaller batch size on the receiver (-b XXX, use values such as 2, 4, 8, 16...) and see if things improve. Smaller batch sizes make pkt-gen check the NIC more often thus overcoming possible problems with interrupt moderation. Let us know the outcome. Depending on what you see there are several possible explanations. cheers luigi From owner-freebsd-net@freebsd.org Sat Jan 23 18:38:40 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1BFDA8FB86 for ; Sat, 23 Jan 2016 18:38:40 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pf0-x22b.google.com (mail-pf0-x22b.google.com [IPv6:2607:f8b0:400e:c00::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78FF31368 for ; Sat, 23 Jan 2016 18:38:40 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pf0-x22b.google.com with SMTP id n128so59390199pfn.3 for ; Sat, 23 Jan 2016 10:38:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=wVMZzmeFkn/WaEPW4kK/xg5Y/UvjIqXVNF35QCo2GEA=; b=WQr6hd1oWuBl8j9kKgIO7Qw/avlbWKOOflXQuY9suD74P7plVo4EucF2NPbwlh/yMv GtRUZOt2DYIrZxvotTLFF9c/lWXZyaYPWgZjfgzs3Dx5d98Dbru2AP7mA+j8ytIqDze5 +PGRVbk2jsRqr3+ZVjENQ0H9qZh6BULwuhuOcgG60ZCAM8qYyXSb5brRcCBUS5b7LtD4 zWLAypVOBBkDTz5ND+PFvAq/bI/+ekrlFnbcIJOjSG8fzdfekol3i2yoDxh79EbDTNLU yxJfPjiWRY3V7hBV7wgRCFqhCIDr+ApdwGW8E2/5YeNE5QKeutugxmLuvUGXIwsMWloT 53ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=wVMZzmeFkn/WaEPW4kK/xg5Y/UvjIqXVNF35QCo2GEA=; b=eGGeKqjEeeXOujx7rSPuG56nEZ1ohIRCHkPYKuSw+SSTkrLromeZDASW7zHgE2f3XV qZI2l2izmIUOL/hKnSr8mXwt6cAIfTWTUXXC8GIVdsrZxeTT6ygkhlPYx9u7jJAbnTkB 5fjnkSSx4Sju2e+g/+bE34jKyBe4yLUW6WtUYFScZ0zLkV4WmdJoOAgG6VElvKQ4W/CM cu7eSbuKeawALDbdyhdBdFw03tklRskmrZ5L7dSmqpKV2uGAdjU0+RanXC+EoUgTJt3j 3ZQLalWusTAu3R/d/xyT4+beoOWuyPlMFwO+CmIJm1oVa8pOdrvlOaPtafH4d1qb5xPe /pqw== X-Gm-Message-State: AG10YOT0TjNL39l/zheeR8qmDm2d9HLQvScHsBgANR4g3ROZoWeJTKKcIieB5IjFwZ4v5w== X-Received: by 10.98.13.195 with SMTP id 64mr13731917pfn.164.1453574320120; Sat, 23 Jan 2016 10:38:40 -0800 (PST) Received: from ox ([2601:641:c001:8a00:591d:d471:ff4a:b8bf]) by smtp.gmail.com with ESMTPSA id i76sm17855992pfj.68.2016.01.23.10.38.38 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 23 Jan 2016 10:38:38 -0800 (PST) Date: Sat, 23 Jan 2016 10:38:36 -0800 From: Navdeep Parhar To: Marcus Cenzatti Cc: Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123183836.GB4574@ox> Mail-Followup-To: Marcus Cenzatti , Luigi Rizzo , freebsd-net@freebsd.org References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160123174840.32B1DA0121@smtp.hushmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:38:40 -0000 On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote: ... > > woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. Run "netstat -hdw1 -i cxl" on the receiver during your test. Do you see errs and/or idrops incrementing? The input "packets" counter should match what the transmitter is claiming to transmit at. Also check the output of this: # sysctl -n dev.t5nex.0.misc.tp_err_stats It is ok if you see tnlCongDrops, but any of the Errs counter going up is not good -- it means the incoming frames had errors. Do you know if the transmitter will pad up so as not to put runts on the wire? If not then you might want to bump up the size of the frame explicitly (there's some pkt-gen knob for this). Regards, Navdeep > > curiously, I have always used -s/-d with IP addresses on ix-ix testing this is > why I never noticed the case, since ix always received 14Mpps, but you > probably explained it since ix has one single deviceport per wire, hence the > different behavior > > performance stills very low when compared to TX and to what is expected > > thank you for noticing the case > > > From owner-freebsd-net@freebsd.org Sat Jan 23 18:54:55 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70ABDA8FF99 for ; Sat, 23 Jan 2016 18:54:55 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3DCA91AA8 for ; Sat, 23 Jan 2016 18:54:54 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) by smtp3.hushmail.com (Postfix) with SMTP id 57600E046A for ; Sat, 23 Jan 2016 18:54:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=OnP01EcKQ96h0GVj83+XkXDkr4RWi+QQ5wqDZIKMS9U=; b=YPfNOcqefU1dAZ1DUbW+ktREi/Wu2Ed9QFjsTlIsOAKlWTPVUmadSTuhXClaCPLXTqzkATAsTumy/jHmHmXZp13m3RJ5H50/8uQEYphC0WMZE+DkmiyP8sBdCR2VjPpjdjTNCilxkJsCgbdvXg6fCGMwjBhXmDRkR4zb3C/qO6X9d2QsPKumZeHjp8P8xwmXg2MB1JxkoAIXTsvuuUJHKIfh4tphI+uhibJz2/PTaHnCaf0Fjd6aGFVw5cFd149YWB0Esev7YyNXQcAN5ntGT8OUsxm29sUMwo7IqRBpfsaHfOzrbKYFxZRuFSnirS3IUDGhaB3EOf4bgxPh78NKSQ== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp3.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 18:54:53 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 0FA7CA0128; Sat, 23 Jan 2016 18:54:53 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 16:54:52 -0200 To: "Navdeep Parhar" Cc: "Luigi Rizzo" , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: <20160123183836.GB4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123185453.0FA7CA0128@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:54:55 -0000 On 1/23/2016 at 4:38 PM, "Navdeep Parhar" wrote: > >On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote: >... >> >> woops, my bad, yes probably we had some drop, with -S and -D now >I get 1.2Mpps. > >Run "netstat -hdw1 -i cxl" on the receiver during your test. >Do you >see errs and/or idrops incrementing? The input "packets" counter >should >match what the transmitter is claiming to transmit at. > >Also check the output of this: ># sysctl -n dev.t5nex.0.misc.tp_err_stats >It is ok if you see tnlCongDrops, but any of the Errs counter >going up >is not good -- it means the incoming frames had errors. > >Do you know if the transmitter will pad up so as not to put runts >on the >wire? If not then you might want to bump up the size of the frame >explicitly (there's some pkt-gen knob for this). > >Regards, >Navdeep > the size frames I sent based on luigi's suggestions show some particular behaviour as you see it? here is the output for netstat when I pkt-gen -f tx un-throttled (14Mpps): input (Total) output packets errs idrops bytes packets errs bytes colls drops 900k 0 0 55M 3 0 550 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 9 0 2.4K 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 900k 0 0 55M 3 0 422 0 0 as for the error stats: # sysctl -n dev.t5nex.0.misc.tp_err_stats channel 0 channel 1 channel 2 channel 3 macInErrs: 292496 0 0 0 hdrInErrs: 0 0 0 0 tcpInErrs: 0 0 0 0 tcp6InErrs: 0 0 0 0 tnlCongDrops: 30750953 0 0 0 tnlTxDrops: 0 0 0 0 ofldVlanDrops: 0 0 0 0 ofldChanDrops: 0 0 0 0 ofldNoNeigh: 0 ofldCongDefer: 0 From owner-freebsd-net@freebsd.org Sat Jan 23 18:55:15 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A935DA8FFC8 for ; Sat, 23 Jan 2016 18:55:15 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FD7B1B48 for ; Sat, 23 Jan 2016 18:55:15 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x229.google.com with SMTP id x4so56992059lbm.0 for ; Sat, 23 Jan 2016 10:55:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=yJZxU9s2GkCiP8wP5veDNoNZRJrUpRhJImD8FFvtC28=; b=bim0ZnY8AXyTqU9bIt93wGd8YSHpFgn89ZVYaIjdwiu937KyDlDavACJflH2hwh5Px tt52oVPbDqCK34kZ/ZP9q76K3g8Xx9/1YaR4vJ7k2/muoMLYaAyufy0nKerXu1y3Xc/U MxmYSbNuQQXngaCBpaRpHL9VqNw8tNohsob+1NkQWo9utC7dxlQ6lPO1Ae/+GNpKsm6x EUKjVwyChndioZBwBKbYAqB5t/BPjkpZ8hGQRenlJJcB6DarXG4kLW+j34w4fnAn2f50 Dc5F71PiPoU5A5RyrF0VVF48FhMyafNWdSQT0u5FKIoqhggSUrT+IKX+gQ/L9iFGmF47 cgdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=yJZxU9s2GkCiP8wP5veDNoNZRJrUpRhJImD8FFvtC28=; b=COPNEoKWoU/r1EFeHrV+tCiwLPUf4o7lNe3F8Dk+mpj3Xx4PHpHrDYl/tX73kHKoC3 EDmKUxoiNlwEVI7H7fkpiM/D9d5hxt0Xj4+qFbyQosaxfHC2EsapB7cDaLMFq9BFfwl6 djbbpiez+IfB7/AYmFMKSgNAa5a/nnM3rohLSKeu1f1nO5uICaCFOyaGYM7jVp7mn9+e Vo5EZ2TKAju4AZk5HAZSG7KxuupwP17tHtHeK1gQSiUyNEzkew4QNwLkpLgiCb66SpA8 u2BZJhQORy1PSHAUmJni2FRcPfC/y6HQO0wU3J7CW8IRyX3qbCfstx4j9F+zpqrd9zHJ Cxyg== X-Gm-Message-State: AG10YOTyBSwF17BLMsXvwiGRllc2a71I5b7V3/wyJwHs+n1XjjSo4oW4/2KhyCTbxdefovqfHwJaDRdoPxs0ZQ== MIME-Version: 1.0 X-Received: by 10.112.170.101 with SMTP id al5mr3451625lbc.92.1453575312893; Sat, 23 Jan 2016 10:55:12 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 10:55:12 -0800 (PST) In-Reply-To: <20160123183447.4B150A0126@smtp.hushmail.com> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183447.4B150A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 10:55:12 -0800 X-Google-Sender-Auth: 65a90C_j9f0Ejqc3ZHa9_d9X_Gk Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:55:15 -0000 On Sat, Jan 23, 2016 at 10:34 AM, Marcus Cenzatti wrote: >>> woops, my bad, yes probably we had some drop, with -S and -D now >>I get 1.2Mpps. >>> >>> curiously, I have always used -s/-d with IP addresses on ix-ix >>testing this is why I never noticed the case, since ix always >>received 14Mpps, but you probably explained it since ix has one >>single deviceport per wire, hence the different behavior >>> >>> performance stills very low when compared to TX and to what is >>expected >> >>ok so next we can try and see what else is going on. >>please check the following: >>a) are you connected through a switch ? if so, try to send >> out some packets through the ncxl0 port (using pkt-gen >> and its native MAC address) so the switch can learn the >> address and does not need to replicate traffic on all >> ports (which generally is done at a limited rate). >>b) see if using different packet sizes (say 256, 512, 1024, 1500 >> passed as the -l option to pkt-gen) affects the rx rate. >> If the rate does not change (except for 1500 bytes) >> it may be a problem with interrupt moderation >> >>c) use progressively increasing packet rates on the sender, >> using -R xxxx (start at 500000 packets per second, >> and then go up until the receiver cannot sustain the >> tx rate. >> >>d) use a smaller batch size on the receiver (-b XXX, use >> values such as 2, 4, 8, 16...) and see if things improve. >> Smaller batch sizes make pkt-gen check the NIC more often >> thus overcoming possible problems with interrupt moderation. >> >>Let us know the outcome. Depending on what you see there >>are several possible explanations. >> > > Ok, revisiting the summary > - TX host = Intel ix (host 1) > - RX host = Chelsio T520 (host 2) > - Simple topology host1==host2 directly connected intel port 0 (ix0) w/ chelsio port 0 (ncxl0). > > Tests results: > > => Batch 1 packet len > > TX at 256 bytes = 4.46Mpps/TX and 889Kpps/RX > TX at 256 bytes = 2.33Mpps/TX and 888Kpps/RX, 9.3Gbps on TX side according to pkt-gen > TX at 1024 bytes = 1.19Mpps/TX and 889Kpps/RX, 9.3Bps on TX > TX at 1500 bytes = 816Kpps/TX and 816Kpps/RX, 9.8Gbps on TX > > => Batch 2 rates > > -R 500000 / TX Speed: 499.99 Kpps Bandwidth: 240.00 Mbps (raw 336.00 Mbps) / RX 499Kpps > -R 700000 / TX Speed: 699.96 Kpps Bandwidth: 335.98 Mbps (raw 470.38 Mbps) / RX 699Kpps > -R 900000 / TX Speed: 899.98 Kpps Bandwidth: 431.99 Mbps (raw 604.78 Mbps) / RX 888Kpps > > reached the same limits on batch #1. > > => Batch 3 RX batch sizes, default pkt-gen packet len and fixed 900000 rate > > -r 2 / TX 899.98Kpps / RX 672Kpps > -r 4 / TX 899.98Kpps / RX 713Kpps > -r 8 / TX 899.98Kpps / RX 889Kpps > -r 16 / TX 899.98Kpps / RX 889Kpps > > Results make sense for rates bellow the max, but did not improve... only degraded. I think you are still sending to the broadcast addresss (otherwise i'd expect at least the 1.2 Mpps you were seeing before), and if this is the case the tests are not significant because you are hitting the in-nic duplication. Can you please re-run them sending to the MAC address of ncxl ? cheers luigi From owner-freebsd-net@freebsd.org Sat Jan 23 18:56:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30B9CA8E07B for ; Sat, 23 Jan 2016 18:56:27 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02BE51C7E for ; Sat, 23 Jan 2016 18:56:27 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x234.google.com with SMTP id uo6so59938590pac.1 for ; Sat, 23 Jan 2016 10:56:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=B0suCoHijZWCF07iMB+xFzlSPJIywOn/nPzFHqJBPT4=; b=o04PFlu8tkQb65+LEzLqk55CpZj5t9bZutfwoSR8h92esSK+PAwjFqy4NrmtRtZm+v hpn3Bg+Ias2Sz4psRsj4FdDVrHAiRmNlA7oC/CB8Et6njgKFcfDrIgDpgbVNYDL+OvTR VHFio2tBhZfCBLbQkWHcd8+iIWuf+eEYLjZTA0z+DnToSBV1zhkKtEmJbCcQfxCcpD3O hB5ArBl6NI3PrGUr46GcWCqtaSpdW85W1nhnWKbki+Eu4idORBIOzVxt7o56yNNlkgjo 5tSKW5rdHmYqKo+ujQoQFrIySCY8E4g9RIIefZ5u8Kp+d5QwnI6QA7mmDaqD2zJ0yfG6 F+Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=B0suCoHijZWCF07iMB+xFzlSPJIywOn/nPzFHqJBPT4=; b=CCE479GkFNCaVVC1MbBOdam7ANbrBQVoP6Tk04/QXc53ORPrpV8uI1s11s2oQ76BAu Z7Ip5KA3JJ1ZI7f+S5OpBYFWDDkjCoIbbWYFypEvyHhpyQVsnT7yxw7e/477rr6DVFxP Ld27Mat2m9pMsbE3oRg2xtk8nE3uRApm4FsYDEGLJSaNclcb99Y97WFO0YiZyXH8PH5R pVkSj9DhBh0Sj3o+eew1qczWi3hvIrDjrX6Jc9vyHk7y9kt5JfM+2DV9UYMzJf9Dt2qG GnmBuEI7C13217EhuIofgZLr0suDsID494cLU2CYrXkyTkF8ForvVM+sg8AEI7IZ4D1z nlog== X-Gm-Message-State: AG10YOQOdnpzqDXdBcJhJcj/zGo6N9gnrU+1o7Mu1u1lIZ3CikOlQSx4wWoLYxee7sK2dA== X-Received: by 10.66.101.3 with SMTP id fc3mr13885877pab.2.1453575386727; Sat, 23 Jan 2016 10:56:26 -0800 (PST) Received: from ox ([2601:641:c001:8a00:591d:d471:ff4a:b8bf]) by smtp.gmail.com with ESMTPSA id 2sm17982640pfr.22.2016.01.23.10.56.24 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 23 Jan 2016 10:56:25 -0800 (PST) Date: Sat, 23 Jan 2016 10:56:23 -0800 From: Navdeep Parhar To: Marcus Cenzatti Cc: freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123185623.GC4574@ox> Mail-Followup-To: Marcus Cenzatti , freebsd-net@freebsd.org References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160123171300.0F448A0121@smtp.hushmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 18:56:27 -0000 On Sat, Jan 23, 2016 at 03:12:59PM -0200, Marcus Cenzatti wrote: ... > intel# ./pkt-gen -i ix0 -f tx -d 00:07:43:33:8d:c1 -s 00:07:e9:44:d2:ba > 267.767848 main [1715] interface is ix0 > 267.767990 extract_ip_range [291] range is 0.0.0.0:90 to 0.0.0.0:90 > 267.768006 extract_ip_range [291] range is 0.0.0.0:7 to 0.0.0.0:7 Does this mean the packets are being transmitted with source and destination IP all 0? Try to provide a more reasonable IP address range. The T5 receiver might be throwing away "obviously" bad frames. (See my other email for how to check if it's dropping frames) > 267.872796 main [1910] mapped 334980KB at 0x801c00000 > Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus. > 00 -> 00 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) This you've fixed already. L2 broadcasts will get replicated by the receiver and will be delivered to both the cxl and the ncxl interface. The ncxl interface is set to drop on congestion but the cxl interface is set to emit PAUSE on congestion. cxl plugs into the stack, which is slow at pps workloads, and so L2 broadcasts will result in PAUSE out of the port and will slow down the transmitter. Regards, Navdeep From owner-freebsd-net@freebsd.org Sat Jan 23 19:12:31 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5326CA8E5C2 for ; Sat, 23 Jan 2016 19:12:31 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x230.google.com (mail-lf0-x230.google.com [IPv6:2a00:1450:4010:c07::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCE5314D1 for ; Sat, 23 Jan 2016 19:12:30 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x230.google.com with SMTP id 17so64605286lfz.1 for ; Sat, 23 Jan 2016 11:12:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=nvuBrkx1fLNWXhdJwgwJls+frjLQG5XOMxPjbN2TvFI=; b=dOQi+/vhNd8W4xvZ2zVFFttw9DozrO8mnp+MgrKuu+GZs+VGT/MRJyCpGMUR5OUHHC R48VSuxx5Sm56/G0NzC3/f7XiOHQRomZqHWqXvuFwktEyGPb9RAgb4Ggwp8cScZjW81n KSl6r4wnZioiiZ0nd1aH3ZtzLwnLYtvEEbhq6F695+thUA883OF6pVyAX2rdjpPsi2vV Y0GFT5z5iSE0RrCULAHj7hy0o2bsWOa8aSsSV2Y1hL2o4QYVRQLLR4en13F6MkHdsJTv earJScNM+Q2hHP8+X4G73ZTnW868HvhuJksHkLdck4BChs352bUK2M+NIMKy17O9nOpk gqkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=nvuBrkx1fLNWXhdJwgwJls+frjLQG5XOMxPjbN2TvFI=; b=ltyId2n9gsfFqXROA93t8BhSdFxRLoYNGXHDclIMwNVC2Y2pW9cicfz5E9nsCy93DF Kiq+vVbsY5QGUS3Yb7+FUBMbPcK+rG2a9+LcE7woNn+LfTninVhY48PTlVfPVNchvgrE 685KfSjLpkjnnV6wvKNgVGj2uVHF+iJF16kXoEqZd2ait0UPEmeKCrpRfCOEpNrQxzty 1Ka9DikfMhOv4N0tPfvJimVw4btN5apBC/QInrY+MtXlkkBX1ik37YXRBC4Xa2oCTPUO blo5Jof5pMSVHOum2OZZBGT/fEJ7m8EW0alMp+bcXp74XOIbI6uSatz9oSd29sIFA4LS SyXw== X-Gm-Message-State: AG10YOTHC5A0NaiZBdWBEpAYQe8/HrZ1CqilC01YveVw0XxfDooozOFbJyIuM2m5ByICfiXtFldmADYfNqBV+A== MIME-Version: 1.0 X-Received: by 10.25.143.17 with SMTP id r17mr3541937lfd.37.1453576348879; Sat, 23 Jan 2016 11:12:28 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 11:12:28 -0800 (PST) In-Reply-To: <20160123183836.GB4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> Date: Sat, 23 Jan 2016 11:12:28 -0800 X-Google-Sender-Auth: Cfk9s2Kijid6nFrLiI6IEjHa33w Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti , Luigi Rizzo , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:12:31 -0000 On Sat, Jan 23, 2016 at 10:38 AM, Navdeep Parhar wrote: > On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote: > ... >> >> woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > Run "netstat -hdw1 -i cxl" on the receiver during your test. Navdeep, does this give any info on the ncxl port rather than the cxl port connected to the host stack ? ... > Do you know if the transmitter will pad up so as not to put runts on the > wire? If not then you might want to bump up the size of the frame > explicitly (there's some pkt-gen knob for this). > ix/ixl do automatic padding, and in any case pkt-gen by default generates valid packet sizes (and so it does with the variable-size tests I suggested). Is there any parameter that controls interrupt moderation ? In any case we need to know the numbers when sending to the ncxl MAC address as opposed to broadcast. I suspects one of these problems: - the interrupt moderation interval is too long thus limiting the rate to one ring/interval. Unlikely though, even with 1k slots, the measured 1.2 Mpps corresponds to almost 1ms which is too long - the receiver cannot cope with the input load and somehow takes a long time to recover from congestion. If this is the case, running the sender at a lower rate might reach a peak throughput > 1.2 Mpps when the receiver can still keep up, and then drop to the low rate under congestion. - and of course bus errors, when the device is connected on a PCIe slot with only 1-2 data lanes. This actually happens a lot, physical connector sizes do not reflect the number of active PCIe lanes. cheers luigi From owner-freebsd-net@freebsd.org Sat Jan 23 19:13:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95719A8E628 for ; Sat, 23 Jan 2016 19:13:22 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp10.hushmail.com (smtp10a.hushmail.com [65.39.178.239]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A18915C4 for ; Sat, 23 Jan 2016 19:13:22 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp10.hushmail.com (smtp10a.hushmail.com [65.39.178.239]) by smtp10.hushmail.com (Postfix) with SMTP id 797C7C016E for ; Sat, 23 Jan 2016 18:34:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=JG3YHQPgU0FROptS9PXJTZ5qwK0Mdd4t7v1Zl9eEI18=; b=PZA8XqY0iJnC2L2HXBDxQWv+YpwdTVMIH+usxgdG9I6NN1smf69+XplIb7wHU8nD4IKf4caFhVu7/SEEAKnQyFHIbETiNIfa+2d1DY9fHdgdeqs58VwbDrlru1vL2SNxv/gzCevQSHQ7yyU/tm3caZ4HzQOJsldF4lKwUZzS2mfzizC//ng5PP991PLNFMdtO1xNTvmUmZ2LVfN3N9DLJGSQ0P/YoWdmVU1Cz9zeq3zMrt6xx648PcLktJfhJ3sHol5wOJlGnlJdFsCCnCrqqFZYPCWuYsZBwpLmC4qHR10Fh59OHMnro5gs23rSzz0ADe7KajyVoLMBTUevrMZCxg== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp10.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 18:34:47 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 4B150A0126; Sat, 23 Jan 2016 18:34:47 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 16:34:46 -0200 To: "Luigi Rizzo" , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: "Marcus Cenzatti" In-Reply-To: References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123183447.4B150A0126@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:13:22 -0000 On 1/23/2016 at 4:00 PM, "Luigi Rizzo" wrote: > >On Sat, Jan 23, 2016 at 9:48 AM, Marcus Cenzatti > wrote: >> >> >> On 1/23/2016 at 3:35 PM, "Luigi Rizzo" >wrote: >>> >>>On Sat, Jan 23, 2016 at 9:12 AM, Marcus Cenzatti >>> wrote: >>>> >>>> >>>> On 1/23/2016 at 1:40 PM, "Navdeep Parhar" >>>wrote: >>>>> >>>>>On Sat, Jan 23, 2016 at 03:34:27AM -0200, Marcus Cenzatti >wrote: >>>>>> hello, >>>>>> >>>>>> I am testing a chelsio t520-so-cr connected to a Intel card >>>with >>>>>ix(4) >>>>>> driver, I can get the ncxl0 interface to transmit at 14Mpps >to >>>>>another >>>>>> chelsio or to a Intel card. However I can only get 800Kpps- >>>1Mpps >>>>>for >>>>>> RX tests from both chelsio or Intel. >>>>>> >>>>>> I have test with both FreeBSD 11 and FreeBSD 10.3-PRERELEASE. >>>>>> >>>>>> I tested it untuned first and later I have applied tuning >>>>>> recommendations I found on BSDRP[1] website. Results still >>>>>ranging >>>>>> from 800Kpps to 1Mpps for RX. >>>>>> >>>>>> Tests are done w/ with pkt-gen in netmap mode on ncxl >interface >>>>>with >>>>>> both IP address and MAC address source/dest. >>>>> >>>>>The ncxl interfaces have their own MAC addresses. Make sure >the >>>>>sender >>>>>uses the MAC of the receiver's ncxl interface as the >destination >>>>>MAC. >>>>>(netmap's pkt-gen -f tx transmits L2 broadcasts by default). >>>>> >>>>>Check for PAUSE frames coming out of the receiver (sysctl >dev.cxl >>>>>| grep >>>>>tx_pause). If it's receiving frames on netmap interface the >>>>>tx_pause >>>>>counter should not move. >>>>> >>>>>Regards, >>>>>Navdeep >>>>> >>>> >>>> hello, >>>> >>>> yes, MAC addresses are correct, I did the tests again and >>>tx_pause won't move, here is the full transcript for the tests: >>>> >>>> ===> BOX #1 CHELSIO >>>> >>>> chelsio# ifconfig -v ncxl0 >>>> ncxl0: flags=8843 >metric >>>0 mtu 1500 >>>> ether 00:07:43:33:8d:c1 >>>> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >>>> nd6 options=29 >>>> media: Ethernet 10Gbase-SR >>>> status: active >>>> >>>> chelsio# ifconfig -v cxl0 >>>> cxl0: flags=8843 metric >>>0 mtu 1500 >>>> >>>options=ec00bbAN >>>_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> >>>> ether 00:07:43:33:8d:c0 >>>> nd6 options=29 >>>> media: Ethernet 10Gbase-SR >>>> status: active >>>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >>>> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AL1073K >>>DATE: 2011-06-28 >>>> module temperature: 42.79 C Voltage: 3.23 Volts >>>> RX: 0.53 mW (-2.74 dBm) TX: 0.48 mW (-3.12 dBm) >>>> >>>> chelsio# ./pkt-gen -i ncxl0 -f rx -d 00:07:43:33:8d:c1 -s >>>00:07:e9:44:d2:ba >>>> 311.132189 main [1715] interface is ncxl0 >>>> 311.132447 extract_ip_range [291] range is 0.0.0.0:90 to >>>0.0.0.0:90 >>>> 311.132472 extract_ip_range [291] range is 0.0.0.0:7 to >0.0.0.0:7 >>> >>> >>>wait, the lower case -s and -d are for IP addresses, >>>you need to use -S and -D for the MAC addresses. >>>This way you are sending broadcasts, which likely >>>means that the chelsio is replicating the packets >>>to both the netmap and the regular port and the >>>latter (which perhaps comes first) is likely >>>dropping packets. >>> >>>cheers >>>luigi >> >> woops, my bad, yes probably we had some drop, with -S and -D now >I get 1.2Mpps. >> >> curiously, I have always used -s/-d with IP addresses on ix-ix >testing this is why I never noticed the case, since ix always >received 14Mpps, but you probably explained it since ix has one >single deviceport per wire, hence the different behavior >> >> performance stills very low when compared to TX and to what is >expected > >ok so next we can try and see what else is going on. >please check the following: >a) are you connected through a switch ? if so, try to send > out some packets through the ncxl0 port (using pkt-gen > and its native MAC address) so the switch can learn the > address and does not need to replicate traffic on all > ports (which generally is done at a limited rate). >b) see if using different packet sizes (say 256, 512, 1024, 1500 > passed as the -l option to pkt-gen) affects the rx rate. > If the rate does not change (except for 1500 bytes) > it may be a problem with interrupt moderation > >c) use progressively increasing packet rates on the sender, > using -R xxxx (start at 500000 packets per second, > and then go up until the receiver cannot sustain the > tx rate. > >d) use a smaller batch size on the receiver (-b XXX, use > values such as 2, 4, 8, 16...) and see if things improve. > Smaller batch sizes make pkt-gen check the NIC more often > thus overcoming possible problems with interrupt moderation. > >Let us know the outcome. Depending on what you see there >are several possible explanations. > Ok, revisiting the summary - TX host = Intel ix (host 1) - RX host = Chelsio T520 (host 2) - Simple topology host1==host2 directly connected intel port 0 (ix0) w/ chelsio port 0 (ncxl0). Tests results: => Batch 1 packet len TX at 256 bytes = 4.46Mpps/TX and 889Kpps/RX TX at 256 bytes = 2.33Mpps/TX and 888Kpps/RX, 9.3Gbps on TX side according to pkt-gen TX at 1024 bytes = 1.19Mpps/TX and 889Kpps/RX, 9.3Bps on TX TX at 1500 bytes = 816Kpps/TX and 816Kpps/RX, 9.8Gbps on TX => Batch 2 rates -R 500000 / TX Speed: 499.99 Kpps Bandwidth: 240.00 Mbps (raw 336.00 Mbps) / RX 499Kpps -R 700000 / TX Speed: 699.96 Kpps Bandwidth: 335.98 Mbps (raw 470.38 Mbps) / RX 699Kpps -R 900000 / TX Speed: 899.98 Kpps Bandwidth: 431.99 Mbps (raw 604.78 Mbps) / RX 888Kpps reached the same limits on batch #1. => Batch 3 RX batch sizes, default pkt-gen packet len and fixed 900000 rate -r 2 / TX 899.98Kpps / RX 672Kpps -r 4 / TX 899.98Kpps / RX 713Kpps -r 8 / TX 899.98Kpps / RX 889Kpps -r 16 / TX 899.98Kpps / RX 889Kpps Results make sense for rates bellow the max, but did not improve... only degraded. From owner-freebsd-net@freebsd.org Sat Jan 23 19:16:47 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB20AA8E79F for ; Sat, 23 Jan 2016 19:16:47 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x22d.google.com (mail-pa0-x22d.google.com [IPv6:2607:f8b0:400e:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B18A19A9 for ; Sat, 23 Jan 2016 19:16:47 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x22d.google.com with SMTP id cy9so58940625pac.0 for ; Sat, 23 Jan 2016 11:16:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=2BINt3TXegoUrwc8OMcBvzz11flKcMtV3y1NP6Q6uYY=; b=RWpCVuFqy/Z05+jLLz0+7McwWZJADzm9lyxTRqrRHsIVOXmwlZdNCbQEtyjQ3EO1Xz 1Hznk54rMgqnfF9V1+vNe+/Fg75ucPzdnKPL5wG5lypKylFe0qlNrhnAg7Jmzdb08yE2 IbLWtni1EgrL6vOwWajZa5/JRpQCsz0tvH+/g8Yll+q/8bDP6SWOjC8y4PFDkFV6djGd pwCaC1XX60eIJd1kQXRzOQJkV3YEGhqHPuStvje5t0zKHaeovVGKCyCpIKYNzov7kr6f roOQJVjwhMnkgeFfPyoZgIydjrmR3ayk1w5z9/VC9ZOB3ZBofQMUXv312+Ux4vtTTCvh 71OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=2BINt3TXegoUrwc8OMcBvzz11flKcMtV3y1NP6Q6uYY=; b=Xk/zbEYv6tESUf85Y+ueqNAK102ugxgvXgYRGlrlxcNdYh/ktKvXrX426T+eM8KA33 ssuzHgn58PQSKFkwYK/x7VuE+pVEhlI8hhzOcxQ9L0uVo27RVCy3dggxfPQSxVp5Cat6 NbxCG75hCFsiufHUGSnshMQN1+iZN/TR69GBSHHJLUHvVjixGyLooP0420yMIlWqiLGM N9DEpOxrAZye9MQB9AlGOXYjEL3FfIXkjmTPlGch5/OxspafrdTwL2hB4sqfe0U2IEXC IzC6YzHarNdT5+KcpcO+Xl9WlVfbcQBLZyLmzrUeuAkWMJeHEpQ+AMcFAv3VKyq/nIJQ 9gVg== X-Gm-Message-State: AG10YOQAfcL80ZBgpJPec+QZhKxLdKTMYnYDEWM+2fMAaxl4blOfDwrnM7D7c506vBmhVw== X-Received: by 10.66.140.79 with SMTP id re15mr13618659pab.127.1453576607292; Sat, 23 Jan 2016 11:16:47 -0800 (PST) Received: from ox ([2601:641:c001:8a00:591d:d471:ff4a:b8bf]) by smtp.gmail.com with ESMTPSA id cl3sm18065060pad.11.2016.01.23.11.16.45 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 23 Jan 2016 11:16:46 -0800 (PST) Date: Sat, 23 Jan 2016 11:16:43 -0800 From: Navdeep Parhar To: Marcus Cenzatti Cc: Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123191643.GD4574@ox> Mail-Followup-To: Marcus Cenzatti , Luigi Rizzo , freebsd-net@freebsd.org References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> <20160123185453.0FA7CA0128@smtp.hushmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160123185453.0FA7CA0128@smtp.hushmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:16:47 -0000 On Sat, Jan 23, 2016 at 04:54:52PM -0200, Marcus Cenzatti wrote: ... > here is the output for netstat when I pkt-gen -f tx un-throttled (14Mpps): > > input (Total) output > packets errs idrops bytes packets errs bytes colls drops > 900k 0 0 55M 3 0 550 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 9 0 2.4K 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 > 900k 0 0 55M 3 0 422 0 0 This means that the chip really is getting ~900K packets per second from the wire. If it was receiving 14.8M and delivering 900K you'd have seen 14.8M in packets and 13.9M (14.8 - 900K) in idrops. Regards, Navdeep From owner-freebsd-net@freebsd.org Sat Jan 23 19:17:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A168FA8E805 for ; Sat, 23 Jan 2016 19:17:14 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp1.hushmail.com (smtp1a.hushmail.com [65.39.178.236]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "smtp.hushmail.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 85EB71A56 for ; Sat, 23 Jan 2016 19:17:14 +0000 (UTC) (envelope-from cenzatti@hush.com) Received: from smtp1.hushmail.com (smtp1a.hushmail.com [65.39.178.236]) by smtp1.hushmail.com (Postfix) with SMTP id 14F5340297 for ; Sat, 23 Jan 2016 18:43:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=hush.com; h=date:to:subject:from; s=hush; bh=nAfv2av4gUYI09ULdiJAiY5Bbtn+vPFXBwyrnm5KEjY=; b=lBz1NXai+xJr0+gDVEfkL8SV2nS6CNDmo3k4JdKbuSlH73A5pNN1J7uh455tZQw5FlliMvUAXx0OogKXrTsncgicAsGwOXQ51H9FrW/svmumQMl0hcPoebmoyurJGYf1n4vrOxEjoBvqlwldto2/R7l8j5MoA75cHu7qmuWyndw+aKC2puoh30ZP3TFABRO6FzbOuuexONed4a5INBWjnV+7sA0pH2XmpudkISwHwzXB2d6e3dSRM43TD+UdXmX0PJfbs5yzRN143sz34qVw/0N9+fk2MPE369UDcVEBMI5Zy9jlNi8pr+Iwauja7Sm9mFiI6GM0giqW4pc+fEYdOQ== Received: from smtp.hushmail.com (w5.hushmail.com [65.39.178.80]) by smtp1.hushmail.com (Postfix) with ESMTP; Sat, 23 Jan 2016 18:43:20 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id CA903A0126; Sat, 23 Jan 2016 18:43:20 +0000 (UTC) MIME-Version: 1.0 Date: Sat, 23 Jan 2016 16:43:20 -0200 To: "Adrian Chadd" , "Pavel Odintsov" Cc: freebsd-net@freebsd.org, "Eduardo Meyer" Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: "Marcus Cenzatti" In-Reply-To: References: Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20160123184320.CA903A0126@smtp.hushmail.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:17:14 -0000 On 1/23/2016 at 1:31 PM, "Adrian Chadd" wrote: > >For random src/dst ports and IPs and on the chelsio t5 40gig >hardware, >I was getting what, uhm, 40mil tx pps and around 25ish mil rx pps? > >The chelsio rx path really wants to be coalescing rx buffers, which >the netmap API currently doesn't support. I've no idea if luigi has >plans to add that. So, it has the hilarious side effect of "adding >more RX queues" translates to "drops in RX performance." :( > >Thanks, hello, I am sorry, are you saying intel and chelsio distribute RX packet load differently? If I am not mistaken intel will distributed traffic among queues based on ip addresses flow/hashes/whatever, does chelsio make it per packet or somethig other? how does this behavior you noticed could affect single queue (or applications on netmap:if0 and not netmap:if0-n) on chelsio? thanks From owner-freebsd-net@freebsd.org Sat Jan 23 19:27:40 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8F78A8ED37 for ; Sat, 23 Jan 2016 19:27:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81B8E1FE3 for ; Sat, 23 Jan 2016 19:27:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22b.google.com with SMTP id t15so12145745igr.0 for ; Sat, 23 Jan 2016 11:27:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=p6A/f3gjiiwGDJg3So/yNxa/HLxwLCSZBU3U034E1O8=; b=LLRLZq0hXeCCcQiL9J8iiR1oqKJBZWIG9cHn8tcoiSJmvncic1yymN5eiTg3qzcxVz lQE5O/gOseYdN3N2829ChtU1KmpwhXe26co0EUJ5fDtgPchHQLe3zqIPPzYQXMJbI0jl X7NeRkbyHIS8WEmSXms6yBD0tudKrw1QpKoolzJr2/57DV9kj/+JF9pE7TEcOpGdf+3k pmcDgyBXHb44kukg8R4fCpEdkY1ofZJyHQvRCPP8w9mGaOZJTGvTVOlrQia6L8NA5IHz DVzk9KcnCMfwejMx/rcf67nd/NOg8cujANMCUKf6yeEbNze/shgY0F5v6F7E3aJN9y1Y n5uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=p6A/f3gjiiwGDJg3So/yNxa/HLxwLCSZBU3U034E1O8=; b=ewhOwH3g9Kr1XhROk2WdHK+gojhOIHJkKr5ONGUiMXtWvHyPCNpnrvT76gUuHtuDsi zfLVpaZxCNL+WZffIoHTh/TTifyASmkGsFtyIUlCqTb55Vz//4XmNWcGTCz9MuWLyUCQ VjsaIzW90c3mtJaC1ONQN0rISqydF9f0W/1Y5oXwp5gTbu+WE6A8NC6i9mWZY9YcLNc+ LKm7kGFDswR7gxrFy3HHYDe3uGb4wbymfWmDG2uKG++MeRefIG9BQq7bOLf1xnaKo6Er mltpAXP51ptKDHslzd375gBZpTUx9Aw8ptCYyayLEsAR9c5nRMpARPtPliUYHhJkpvr9 PZTA== X-Gm-Message-State: AG10YOQzHrxxvmbnps4wXhN/uE8Zr32jjmZMCSPNGlU0oMaSbAeioT6zxqawbRy/CZ9waVkO4TqA/3F5rq83dQ== MIME-Version: 1.0 X-Received: by 10.50.150.36 with SMTP id uf4mr9272791igb.61.1453577259832; Sat, 23 Jan 2016 11:27:39 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.36.121.16 with HTTP; Sat, 23 Jan 2016 11:27:39 -0800 (PST) In-Reply-To: <20160123184320.CA903A0126@smtp.hushmail.com> References: <20160123184320.CA903A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 11:27:39 -0800 X-Google-Sender-Auth: vAwOVImR1GQZbSp2ljc1VHIGELk Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Adrian Chadd To: Marcus Cenzatti Cc: Pavel Odintsov , FreeBSD Net , Eduardo Meyer Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:27:40 -0000 ok, so it's .. a little more complicated than that. The chelsio hardware (thanks jim!) and intel hardware (thanks sean/limelight!) do support various kinds of traffic hashing into different queues. The common subset of behaviour is the microsoft RSS requirement spec. You can hash on v4, v6 headers as well as v4+(tcp,udp) ports and v6+(tcp,udp) ports. It depends on the traffic and the RSS config. Now, each NIC and driver has different defaults. This meant that yes, each NIC and driver would distribute traffic differently, leading to some pretty amusing differences in workload behaviour that was purely due to how the driver defaults worked. This is one of the motivations behind me pushing the freebsd rss stuff along - I wanted the RX queue config code and the RSS hashing config code to be explicitly done in the driver to match what the system configuration is so that we /have/ that code. Otherwise it's arbitrary - some drivers hash on just L3 headers, some hash on L3/L4 headers, some hash fragments, some may not; some drivers populate the RSS hash id in the mbuf flowid (cxgbe); some just put the queue id in there (intel drivers.) So when you use "RSS" in -HEAD, the NICs that support it hopefully obey the config in sys/net/rss_config.c - which is to hash on TCP ports for v4/v6 traffic, and just the L3 v4/v6 headers for everything else. The NICS support hashing on UDP, but the challenge is that fragments hash differently to non-fragments, so you have to re-constitute the fragments back into a normal packet and rehash that in software. Now, for TCP fragments are infrequent, but for UDP they're quite frequent in some workloads. So, I defaulted UDP RSS off and just let UDP be hashed to the L3 address. This means that if you use RSS enabled, and you're using a supported NIC (igb, ixgbe, cxgbe, ixl is currently broken, and I think the mellanox driver does RSS now?) then it'll distribute like this: * TCP traffic: will hash based on L3 (src,dst) and L4 (port); * UDP traffic, will hash on L3 (src, dst) only; * other (eg ICMP, GRE, etc) - hash on L3 (src, dst) only; * non-IP traffic is currently hashed however the NIC does it. This isn't currently expressed via RSS. If you use RSS on a non-supported NIC, then it'll re-hash the packet in software and distribute work to multiple netisr queues as appropriate. Now, my eventual (!) plan is to expose the RSS queue/key/hash config per NIC rather than global, and then users can select what they want for things like netmap, and let the kernel TCP/UDP stack code rehash things as appropriate. But it's all spare time project for me at the moment, and right now I'm debugging the ixl RSS code and preparing some UDP stack changes to cope with uhm, "behaviour." Ok, so with all of that - if you don't use RSS in HEAD, then the traffic distribution will depend purely upon the driver defaults. intel and chelsio should be hashing on TCP/UDP headers for those packets, and just straight L3 only for others. There's no shared RSS key - sometimes it's random (older intel drivers), sometimes it's fixed (cxgbe) so you can end up with reboot-to-reboot variations with intel drivers. If you're doing straight iperf testing, with one or a small number of connections, it's quite likely you'll end up with only a small subset of the RX queues being used. If you want to see the NIC really distribute things, you should use pkt-gen with the random port/IP options that I added in -HEAD about a year ago now. Next - how well does it scale across multiple queues. So I noticed on the cxgbe hardware that adding more RX queues actually caused the aggregate RX throughput to drop by a few million pps. After Navdeep/I poked at it, we and the chelsio hardware people concluded that because netmap is doing one-buffer-is-one-packet on receive, the RX DMA engine may be not keeping up feeding it descriptors and we really should be supporting RX buffer batching. It doesn't show up at 10g line rate, only when you're trying to hit the NIC theoretical max on 40g. Yeah, we hit the NIC theoretical max on 40g with one RX queue, but then we couldn't do any work - that's why I was trying to farm it out to multiple queues via hardware. Finally - some drivers had some very .. silly defaults for interrupt handling. The chelsio driver was generating a lot of notifications in netmap mode, which navdeep heavily tuned and fixed whilst we were digging into 40g behaviour. The intel interrupt moderation code assumes you're not using netmap and the netmap code doesn't increment the right counters, so AIM is just plainly broken and you end up with crazy high varying interrupt rates which really slow things down. (there's also NUMA related things at high pps rates, but I'm going to ignore this for now.) I hope this helps. -adrian From owner-freebsd-net@freebsd.org Sat Jan 23 19:38:12 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58EEDA8F04E for ; Sat, 23 Jan 2016 19:38:12 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19F4918DC for ; Sat, 23 Jan 2016 19:38:12 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22a.google.com with SMTP id z14so11924446igp.0 for ; Sat, 23 Jan 2016 11:38:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hhG8uEnElMrnkHsSEj2vrOdUNqJeK5Dzg+dWmZkLPJ8=; b=vtwu/r6OcRimPIUTu5ku2l2i0E95qpuRl+8Mc+QKgLI445ZF6zr0d4Z0OoxDaXprjZ UqUUj/bHmRvkfltx8A6Ph8OeLanfW+s36AF2XvXUbTFrdrH3wULOTER2ffnAWGj29mKX 9MUwolqkNvRLvJ5+JwLuQTv/FvczkV9xsCzR6kiLkY/zlLMCpprCAalHNKyWuRJuUDsJ aujBnzAqqT+E+MGth9uMmmOflqqteCIB8LTII1PU/IaaiODeyArlXZ3G/UUEDq9RAoTX 1/HTTTjmFaj0k01s0HFVoZriKG9CyTAXhoVE+LUXOyKvwDUu1LJnRZRC3cSv0Ht/J2Pl NAFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hhG8uEnElMrnkHsSEj2vrOdUNqJeK5Dzg+dWmZkLPJ8=; b=OeiOpndyC5E7zh9XCrNT7IxmyWwP8ZF/cn980f4U/TlSS+AqObgfxRPrgTlJ4khL31 XA0lc6NQVbKNz0xjbILP+MQSzCp12KKn05ZHcx5qJN2GUfGylmbUhVFrWr74n/THXuHo AMKHxaw7Fg6Gjvrxl2giH69bSP1JnKMIaA0zXNHfsCwDoouXQ1OmbAwprvhhe0XYwSne 1XlzY1W8mYzr/1W7NH7UhgkkIyN8yOBuSs9L/vChXyRNQmpAlFmcYZxKtVns4wG+SArQ rytvCwsXgwgSdh+wVrhj7I8HblP0NxovynybsOyO0ySr/xD50LDssZLlGIsmJN9CTIAt 2iVQ== X-Gm-Message-State: AG10YOQVHa/YIUpidct/m4uaAZGCa0Bkj+EkvISGki62nWtxH7jEqNCus5JbHgOB7ZPYr6no9R2V12HiHcBPbQ== MIME-Version: 1.0 X-Received: by 10.50.122.100 with SMTP id lr4mr10168415igb.37.1453577891466; Sat, 23 Jan 2016 11:38:11 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.36.121.16 with HTTP; Sat, 23 Jan 2016 11:38:11 -0800 (PST) In-Reply-To: References: <20160123184320.CA903A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 11:38:11 -0800 X-Google-Sender-Auth: tqZxnVyO3-4v-xHOus7GS9z41BI Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Adrian Chadd To: Marcus Cenzatti Cc: Pavel Odintsov , FreeBSD Net , Eduardo Meyer Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:38:12 -0000 Oh and one other thing - on the cxgbe hardware, the netmap interfaces (ncxl) have a different MAC. things like broadcast traffic is duplicated to cxlX AND ncxlX. So, if you're only using netmap and you're testing promisc/bridging, you should bring /down/ the cxlX interface and leave ncxlX up - otherwise yeah, the cxbge MAC will duplicate packets in hardware, which halves the RX bandwidth available on the NIC /and/ chews CPU in FreeBSD as the normal ethernet input path drops all of those packets. (I think the same thing holds with the virtual devices via SR-IOV on ixgbe hardware, btw..) -a From owner-freebsd-net@freebsd.org Sat Jan 23 19:49:30 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F7F2A8F2B4 for ; Sat, 23 Jan 2016 19:49:30 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C92721D3F for ; Sat, 23 Jan 2016 19:49:29 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x22a.google.com with SMTP id x4so57319413lbm.0 for ; Sat, 23 Jan 2016 11:49:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=1AjhG5zupXGU1muDv3i35hW2x+32d3T43R98voRNUUY=; b=BLz+7Ud7DWmm49soljZKMGr1l0+aZimsgE7idireUHPVNdiGj3bY+a/BdQdedZyb08 PT/3VOS2nQk3+0XzhMZh5uP9xeBQhqo5Rr9K7ETud+scpAcenBHpA1F4WSbSsubJjoEG pwmtZYWFAX4kxpNKQE53scmryKssN780xeyv6rc0EwVCnVZf5n5/lyRoJMURtsv6PPMN mV8/thaVfTha/4xDwwfflLLasj2FIQk0pvCMtVsAGEhtzHQMnbtmeM79+9fzQNN3rIFe Xa0kpUAArVETwsl1zynUNOr+60MfGJKetjXBiRd3v1ZYUBVSaNR/JQPm312gDeIwODog IIfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=1AjhG5zupXGU1muDv3i35hW2x+32d3T43R98voRNUUY=; b=Qt2bJZROMd7G3L/HeOsEqdjTvozw9r1M0pNTUjpRr0cpF41r2Z/ZReXNmUL+7lJhEc NTdzrPVJpDhUpD9qdCJATbpUJe03jk/nWyYFpFbd8Ae6p8h5C0mAmN0CajfraRn/knZg Ylzityvf3Dzf1m058W8x+9ZSZXoVacWV0qQ+hpPibczfJsltggV/HKOLaycuSkJbXYqX nEwpga069fQFScxWBIgXFgFKHHro4lEpua8UyLnraOWf2wBeLroKoGGKcx2k0OWbUeOJ o03G+9wBc8Lr2NEjHzUC9p8hUwGVt/wY3Z9PgLZHRV91M9PQt1VJob+dP1iEPOHNpOJ+ ijwA== X-Gm-Message-State: AG10YOQb5oj6mOEQxhVyEuEkioeo8mcgGmqmDUST4l8W0sPU2hmXkr80tU4k2hOa6MwNtlKM6jNpuI4lvItDeA== MIME-Version: 1.0 X-Received: by 10.112.13.165 with SMTP id i5mr2855230lbc.79.1453578567958; Sat, 23 Jan 2016 11:49:27 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 11:49:27 -0800 (PST) In-Reply-To: <20160123184320.CA903A0126@smtp.hushmail.com> References: <20160123184320.CA903A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 11:49:27 -0800 X-Google-Sender-Auth: Gx6P2QTEsvhZKf6d7UguoK_J2wk Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Luigi Rizzo To: Marcus Cenzatti Cc: Adrian Chadd , Pavel Odintsov , "freebsd-net@freebsd.org" , Eduardo Meyer Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:49:30 -0000 On Sat, Jan 23, 2016 at 10:43 AM, Marcus Cenzatti wrote: > > > On 1/23/2016 at 1:31 PM, "Adrian Chadd" wrote: >> >>For random src/dst ports and IPs and on the chelsio t5 40gig >>hardware, >>I was getting what, uhm, 40mil tx pps and around 25ish mil rx pps? >> >>The chelsio rx path really wants to be coalescing rx buffers, which >>the netmap API currently doesn't support. I've no idea if luigi has >>plans to add that. So, it has the hilarious side effect of "adding >>more RX queues" translates to "drops in RX performance." :( >> >>Thanks, > > hello, > > I am sorry, are you saying intel and chelsio distribute RX packet load differently? If I am not mistaken intel will distributed traffic among queues based on ip addresses flow/hashes/whatever, does chelsio make it per packet or somethig other? > I think there are several orthogonal issues here: - traffic distribution has been discussed by Adrian so please look at the email he just sent; - when you use netmap on a single queue ie netmap:ix0-X the software side is as efficient as it can, as it needs to check the status of a single queue on poll() or ioctl(..RXSYNC..). On the contrary, when you access netmap:if0 (i.e. all queues on a single file descriptor) every system call has to check all the queues so you are better off with a smaller number of queues. - on the hardware side, distributing traffic to multiple RX queues has also a cost that increases with the number of queues, as the NIC needs to update the ring pointers and fetch buffers for multiple queues, and you can easily run out of PCIe bandwidth for these transactions. This affects all NICs. Some (ix ?) have parameters to configure how often to update the rings and fetch descriptors, mitigating the problem. Some (ixl) don't. My opinion is that you should use multiple queues only if you want to rely on hw-based traffic steering, and/or your workload is bottlenecked by the CPU rather than bus I/O bandwidth. Even so, use as few queues as possible. Sometimes people use multiple queues to increase the number of receive buffers and tolerate more latency in the software side, but this really depends on the traffic distribution, so in the worst case you are still dealing with a single ring. Often you are better off using a single hw queue and have a process read from it using netmap and demultiplex to different netmap pipes (zero copy). That reduces bus transactions. Another option which I am experimenting these days is forget about individual packets once you are off the wire, and connect the various processes in your pipeline with a stream (TCP or similar) where packets and descriptors are back to back. CPUs and OSes are very efficient in dealing with streams of data. cheers luigi Another motivation would be to have more Often you are better off d CPU limited. on multiqueue is that you should use it only if your workload > how does this behavior you noticed could affect single queue (or applications on netmap:if0 and not netmap:if0-n) on chelsio? > > thanks > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Sat Jan 23 20:22:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84422A8FB2A for ; Sat, 23 Jan 2016 20:22:14 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: from mail-oi0-x236.google.com (mail-oi0-x236.google.com [IPv6:2607:f8b0:4003:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 42AFE1D04 for ; Sat, 23 Jan 2016 20:22:14 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: by mail-oi0-x236.google.com with SMTP id w75so66883698oie.0 for ; Sat, 23 Jan 2016 12:22:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gy8MQS87rKNhR/HqSgPWHELU3CuiAtS8iVvouOxUZrs=; b=HgD/JAHNiK5O8yCe+ct54NXjhyqI3CHXBGLfJTwuek6pRA8T2M/th6UQgLFeWhUswv M21k6ZohOM6p59bkbDEk2zXjmLAqNFRFOVgVUHbzPt078ehUNdo7vk0IX9SMv/PJYP8r nRDuVGimL07KOHOo4yCa81kEoodv+e67qpeRVNk4UTuPqmVk37QjIlgEc10i3VoMaojU xrqinl79tm2dDE3oKKL4yRgO8jcnvkw+3BwAp0MJG9AttuGtHeQpQVvIT8TIdwcQCF0W RNVdE61eg5gigb33vEeNI1hyxQgAH1KweFu8nI6rwRxf/nwhzgtCqEwWXygp3oJ9jy6x 3nNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=gy8MQS87rKNhR/HqSgPWHELU3CuiAtS8iVvouOxUZrs=; b=mXYq/9llFBexuy5dae9GR9dt8xuTBx9CNriuFM4dFLoky/u5leBqayz6nPpbwbNGA7 a0QhdswQNUXgSV8OP+iQKICR2drhFVbIege/PeVb7+EBa81N05fmWNfTHRiIP9orv+Ef wyfBxFMcx5hpL9X6XMLGmuVpPnT4YRMqThL7e9hAVM/lQG1n29EPlspRJf73y48lNOKx IanKv+UhgHcmN793ipVuhRENxSc5mV5BAfbPNM7xcBWvz+IPgue4AQ1p+9HCNbIEIAJj dY76r46xKjjB850MRzGZ6pWw1SlH1uioRQk9h5UbpTjLy5zNgPINPUUacklhGTffJUNh S5Zg== X-Gm-Message-State: AG10YOQf6Mek54n0Ui8dr/fQxCQYZuKsRbW+8kWRhKziHKjoFXeHNrsQFvu3da1/YBsh2h/aaoCPP8xocZDddQ== MIME-Version: 1.0 X-Received: by 10.202.73.67 with SMTP id w64mr7147014oia.84.1453580533503; Sat, 23 Jan 2016 12:22:13 -0800 (PST) Received: by 10.182.88.138 with HTTP; Sat, 23 Jan 2016 12:22:13 -0800 (PST) In-Reply-To: References: <20160123184320.CA903A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 18:22:13 -0200 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Eduardo Meyer To: Luigi Rizzo Cc: Marcus Cenzatti , Adrian Chadd , Pavel Odintsov , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 20:22:14 -0000 On Sat, Jan 23, 2016 at 5:49 PM, Luigi Rizzo wrote: > On Sat, Jan 23, 2016 at 10:43 AM, Marcus Cenzatti > wrote: > > > > > > On 1/23/2016 at 1:31 PM, "Adrian Chadd" wrote: > >> > >>For random src/dst ports and IPs and on the chelsio t5 40gig > >>hardware, > >>I was getting what, uhm, 40mil tx pps and around 25ish mil rx pps? > >> > >>The chelsio rx path really wants to be coalescing rx buffers, which > >>the netmap API currently doesn't support. I've no idea if luigi has > >>plans to add that. So, it has the hilarious side effect of "adding > >>more RX queues" translates to "drops in RX performance." :( > >> > >>Thanks, > > > > hello, > > > > I am sorry, are you saying intel and chelsio distribute RX packet load > differently? If I am not mistaken intel will distributed traffic among > queues based on ip addresses flow/hashes/whatever, does chelsio make it per > packet or somethig other? > > > > I think there are several orthogonal issues here: > - traffic distribution has been discussed by Adrian > so please look at the email he just sent; > > - when you use netmap on a single queue ie netmap:ix0-X > the software side is as efficient as it can, as it needs > to check the status of a single queue on poll() or ioctl(..RXSYNC..). > On the contrary, when you access netmap:if0 (i.e. all > queues on a single file descriptor) every system call > has to check all the queues so you are better off with > a smaller number of queues. > > - on the hardware side, distributing traffic to multiple RX queues > has also a cost that increases with the number of queues, as the > NIC needs to update the ring pointers and fetch buffers for > multiple queues, and you can easily run out of PCIe bandwidth for > these transactions. This affects all NICs. > Some (ix ?) have parameters to configure how often to update the rings > and fetch descriptors, mitigating the problem. Some (ixl) don't. > > My opinion is that you should use multiple queues only if you want > to rely on hw-based traffic steering, and/or your workload is > bottlenecked by the CPU rather than bus I/O bandwidth. Even so, > use as few queues as possible. > > Sometimes people use multiple queues to increase the number of > receive buffers and tolerate more latency in the software side, but > this really depends on the traffic distribution, so in the worst case > you are still dealing with a single ring. > > Often you are better off using a single hw queue and have a > process read from it using netmap and demultiplex to different > netmap pipes (zero copy). That reduces bus transactions. > > Another option which I am experimenting these days is forget about > individual packets once you are off the wire, and connect the various > processes in your pipeline with a stream (TCP or similar) where packets > and descriptors are back to back. CPUs and OSes are very efficient in > dealing with streams of data. > > cheers > luigi > > > > Another motivation would be to have more > > Often you are better off d > CPU limited. on multiqueue is that you should use it only if your workload > Thanks for the explanation. What i was trying to achieve is more performance using more than one CPU to actually bridge at line rate, since I have several cores (16 cores) but low CPU clock, and growing up horizontally is easier and cheaper than getting faster CPUs. I though that using 2 queues with two bridge instances or two threads (adrian's bridge) and using that on one of the other 15 idle cores would just allow me to grow from 9Mpps to 14Mpps. It looks like it's not that simple. If I was a developer making a multithreaded netmap application to increase pps rates, is there any other/better strategy than using multiple queues? Should I distribute the load along the application reading and writing to one single Q or something better? I mean, a multithreaded bridge to check how many packets we have in the queue and distributing a constant number of packets to each thread, is it possible/efficient? If I don't have the constant number of packets I should be able to see TAIL via netmap, right? Thank you all for all the details on how things actually work. -- =========== Eduardo Meyer pessoal: dudu.meyer@gmail.com profissional: ddm.farmaciap@saude.gov.br From owner-freebsd-net@freebsd.org Sat Jan 23 21:18:21 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 398CDA8EE9F for ; Sat, 23 Jan 2016 21:18:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x232.google.com (mail-pa0-x232.google.com [IPv6:2607:f8b0:400e:c03::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 081451C53 for ; Sat, 23 Jan 2016 21:18:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pa0-x232.google.com with SMTP id uo6so60968885pac.1 for ; Sat, 23 Jan 2016 13:18:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=5HdHv3E1sNJEMRk9gunI7u32y5uSkiUso9hcxyUO3/8=; b=wbqNcBgPw+z0SqAEIItF+hcpvUOtq26L2cH6FjALmGvxl1cFhOxn2l2+Ec+LJu3qcn 7Xa/0sTMKZOVFcXoCpTlmSthhYCtS/JjN1FOb02rwjAJtvV2CZs2/uxG1umrQXx8Ab3X LoS1YroSoXHM23CTnTohC50tgfKVWwAhQmRtSXklc44P0JOIiwn98s4A93lFxZ6C1QSt /haDKFDJmKeHbQqNVnBKyjabP30EWpsXzIT1K4o/p4WRh+MJ8JgfKBdZOMOmq1hsZGgj JsRfSrvJr3UbKXtRKZfStipaNobn9cGeyKYXE5m7WgfXlwyVUUkIYMMhVYZQBTsIBrl6 4+Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=5HdHv3E1sNJEMRk9gunI7u32y5uSkiUso9hcxyUO3/8=; b=EeO3mL2Szm+krKWhPlzMG4Rx8lpopsr8TYpFW2GH2szV6wd8SAl1J5ok6SdhvcG5as 9uVbsjRlNHmZbkIMYYsxoJnhTj7Qe+GrVZCAOUKi76dTskjwyQdz2z0FXPxwhr8KZKYi b2jjwg070HqoFRXr0y+q9FizHAq0jNF8N86yJbJDF8RBthgxDYdmWZJkE7Z26KdlUN8U P0Q53M4941C5RbD+LxYfArJVTUEfSa8j8AW6lD0Is1m3emgaOHDg3caQgU/zBp563lPO AspxjrZYJf30LuVywxb4Igy75t3wdji28ybNrzj0RetUw3hbq/bQQmGsOohNNsqKrhis RW5A== X-Gm-Message-State: AG10YOQLnlbFSteT+FgXodYDcMXD2mEY7cVTeWTE2nKJzyxRwlh/VWNcFRL0SS4HKtmEIw== X-Received: by 10.66.255.39 with SMTP id an7mr14476626pad.101.1453583900714; Sat, 23 Jan 2016 13:18:20 -0800 (PST) Received: from ox ([2601:641:c001:8a00:591d:d471:ff4a:b8bf]) by smtp.gmail.com with ESMTPSA id 11sm18269423pfq.87.2016.01.23.13.18.18 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 23 Jan 2016 13:18:19 -0800 (PST) Date: Sat, 23 Jan 2016 13:18:16 -0800 From: Navdeep Parhar To: Luigi Rizzo Cc: Marcus Cenzatti , "freebsd-net@freebsd.org" Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <20160123211816.GE4574@ox> Mail-Followup-To: Luigi Rizzo , Marcus Cenzatti , "freebsd-net@freebsd.org" References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 21:18:21 -0000 On Sat, Jan 23, 2016 at 11:12:28AM -0800, Luigi Rizzo wrote: > On Sat, Jan 23, 2016 at 10:38 AM, Navdeep Parhar wrote: > > On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote: > > ... > >> > >> woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > > > Run "netstat -hdw1 -i cxl" on the receiver during your test. > > Navdeep, does this give any info on the ncxl port rather > than the cxl port connected to the host stack ? You're right, it should have been "netstat -hdw 1 -I ncxl". In these kinds of experiments it might even be best to run two netstats in parallel on cxl and ncxl. > > ... > > Do you know if the transmitter will pad up so as not to put runts on the > > wire? If not then you might want to bump up the size of the frame > > explicitly (there's some pkt-gen knob for this). > > > > ix/ixl do automatic padding, and in any case pkt-gen > by default generates valid packet sizes (and so it does > with the variable-size tests I suggested). > > Is there any parameter that controls interrupt moderation ? > > In any case we need to know the numbers when sending to the > ncxl MAC address as opposed to broadcast. > > I suspects one of these problems: > > - the interrupt moderation interval is too long thus limiting > the rate to one ring/interval. Unlikely though, even > with 1k slots, the measured 1.2 Mpps corresponds to almost > 1ms which is too long > > - the receiver cannot cope with the input load and somehow > takes a long time to recover from congestion. If this is > the case, running the sender at a lower rate might reach > a peak throughput > 1.2 Mpps when the receiver can still > keep up, and then drop to the low rate under congestion. > > - and of course bus errors, when the device is connected on > a PCIe slot with only 1-2 data lanes. > This actually happens a lot, physical connector sizes > do not reflect the number of active PCIe lanes. There are no drops or PAUSE or any sign of backpressure. The netstat counters show 900K incoming and 0 drops/errors, which means 900K packets on the wire for the port and all were delivered to the driver successfully. The mismatch in the transmitter's counter and the incoming counter can only be explained by a) Frames whose DMAC address didn't match the local interface's MAC. This can be tested by switching cxl0 and ncxl0 to promisc mode to see if that opens the flood gates. b) Frames mangled badly enough to be discarded. But these should show as an error or drop in at least one of these: sysctl dev.cxl..stats sysctl -n dev.t5nex.0.misc.tp_err_stats netstat -hd -I cxl0 netstat -hd -I ncxl0 The only broken counter in cxgbe that I know of is rx_runts and we've already verified that the transmitter isn't generating runts. Regards, Navdeep From owner-freebsd-net@freebsd.org Sat Jan 23 22:14:07 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AF1CA8D04E for ; Sat, 23 Jan 2016 22:14:07 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: from mail-lf0-x230.google.com (mail-lf0-x230.google.com [IPv6:2a00:1450:4010:c07::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 999D71022 for ; Sat, 23 Jan 2016 22:14:06 +0000 (UTC) (envelope-from woodsb02@gmail.com) Received: by mail-lf0-x230.google.com with SMTP id h129so66346312lfh.3 for ; Sat, 23 Jan 2016 14:14:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=geYqlU5b9MOvRGDHaM9uhLwoj5hWRZEVgVJxPrcl6aY=; b=HbQqsxSDz7IccJ2AdLo9vX1K33HHQPz3AD9oq76QCHLZp4jwinDp7hkT9IUnYoTE8d 44VJHO4by2K5+EkV3Vw5vZGYQiLv2eOPoBWpXxWxDf9mB2/vg92tHcd6rg45JIIL85u0 nAUNv9mvVgvhDf2tMrAHwUSA0ZjNXOqEk3TOfsOuhYhNMg1RuHzWk4lYM8x8fddHADeE nzd+zysQwJweXeUffSvvGdZ4ceYGTo24wkShujuLp+i7vID0aV29J4nP8NKph/y0S6R+ vjkb7pBBUI5bARcfANdSw2BzmTu+DS22RZHIxhch3Bxy/19ZE9MJ85J3GeW26tvQ+esF TE4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=geYqlU5b9MOvRGDHaM9uhLwoj5hWRZEVgVJxPrcl6aY=; b=Y1iDwBpjshHzjV9E2eN7jOrtr1YCfL+yfiLpgBrqWnTM3mxs1nRAQbzhm8OJqQbgyl OgDzvSRYtMCzx0pxRLkgt6POmEVIsAy6m+Vioc0JywnnYqr7p2YcUn38TgoMEUtpgj2u h4nLF8SQ8CJb4zgzWT3LjZYJxhzwJJ/SaLXmQ/m1aSvQ+tYTDFSL7A+O2jBSjw0LvQHx rUD3TqR3dtFGmOJkpzkPO8rqrkJVGeZgR08ijt1+TrdtdsvN/8TqB40U/5wqk2ap92Rp xVEFDCAN8dQiB6NZmgRB2MGaZ2syPJcs5EinpZtDHp8hSSpLan604pfJHIkQHtdrjAXe CxRg== X-Gm-Message-State: AG10YOSy5VDhoTql5Zv2wtGRZ5pYLkBbpvrQ9RbtB8TgAtIWem3PRuEVaM4CaalY7xy6q8Q72cnS2+34wPi6CQ== MIME-Version: 1.0 X-Received: by 10.25.29.147 with SMTP id d141mr3149747lfd.26.1453587244790; Sat, 23 Jan 2016 14:14:04 -0800 (PST) Received: by 10.25.89.10 with HTTP; Sat, 23 Jan 2016 14:14:04 -0800 (PST) Date: Sat, 23 Jan 2016 23:14:04 +0100 Message-ID: Subject: Multicast routing on FreeBSD 11 current From: Ben Woods To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 22:14:07 -0000 Hey everyone, I am trying to set up multicast routing on FreeBSD 11 current, so I can pass IPTV from my ISP to my set top box via my FreeBSD router. I intend to use net/igmpproxy as per this forum post: https://lafibre.info/remplacer-livebox/remplacer-sa-livebox-par-un-routeur-pfsense/ I am running the GENERIC kernel, except with VIMAGE enabled and SCTP disabled. When I try to load the kernel module, I am getting an error: % sudo kldload -v ip_mroute kldload: an error occurred while loading the module. Please check dmesg(8) for more details. % dmesg linker_load_file: Unsupported file type Any ideas what could be causing this error? Thanks for your help. Regards, Ben -- From: Benjamin Woods woodsb02@gmail.com From owner-freebsd-net@freebsd.org Sat Jan 23 22:34:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08A47A8D99B for ; Sat, 23 Jan 2016 22:34:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EC5EB1D43 for ; Sat, 23 Jan 2016 22:34:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NMYGwk016072 for ; Sat, 23 Jan 2016 22:34:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendit "KPI" (in reality sendmsg(2); maybe sendto(2)) will fail with EINVAL if mp->msg_control != NULL and mp->msg_controllen is < sizeof(struct cmsghdr); is not documented in send(2) Date: Sat, 23 Jan 2016 22:34:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to bug_severity Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 22:34:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 NGie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Severity|Affects Only Me |Affects Some People --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 22:49:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09BFEA8DED5 for ; Sat, 23 Jan 2016 22:49:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED8B61602 for ; Sat, 23 Jan 2016 22:49:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NMnR9W040877 for ; Sat, 23 Jan 2016 22:49:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendit "KPI" (in reality sendmsg(2); maybe sendto(2)) will fail with EINVAL if mp->msg_control != NULL and mp->msg_controllen is < sizeof(struct cmsghdr); is not documented in send(2) Date: Sat, 23 Jan 2016 22:49:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 22:49:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 --- Comment #1 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Sat Jan 23 22:49:14 UTC 2016 New revision: 294646 URL: https://svnweb.freebsd.org/changeset/base/294646 Log: Don't run the t_cmsg_len testcase on 64-bit architectures It always fails when trying to send through the sendit(9) private KPI in = the kernel due to a size mismatch between the msghdr and data being sent [*], which suspiciously seems like it's related to sizeof pointers instead of scalar= s, or something of that ilk MFC after: 1 week PR: 206543, 206544 [*] Sponsored by: EMC / Isilon Storage Division Changes: _U head/ head/tools/regression/sockets/unix_cmsg/unix_cmsg.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 22:53:36 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC405A8E27D for ; Sat, 23 Jan 2016 22:53:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CB4AC1C33 for ; Sat, 23 Jan 2016 22:53:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NMra66052384 for ; Sat, 23 Jan 2016 22:53:36 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendit "KPI" (in reality sendmsg(2); maybe sendto(2)) will fail with EINVAL if mp->msg_control != NULL and mp->msg_controllen is < sizeof(struct cmsghdr); is not documented in send(2) Date: Sat, 23 Jan 2016 22:53:36 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Flags: mfc-stable9? mfc-stable10? X-Bugzilla-Changed-Fields: cc assigned_to flagtypes.name bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 22:53:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd-net@FreeBSD.org Assignee|freebsd-net@FreeBSD.org |ngie@FreeBSD.org Flags| |mfc-stable9?, mfc-stable10? Status|New |In Progress --- Comment #2 from Kubilay Kocak --- Reporter is Committer, assign accordingly. @Ngie, please set mfc-* flags to + once they are committed in those branche= s. Don't forget to include the relevant PR: line :) --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 22:55:51 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B2F6A8E363 for ; Sat, 23 Jan 2016 22:55:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 797871D0B for ; Sat, 23 Jan 2016 22:55:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NMtpN3055360 for ; Sat, 23 Jan 2016 22:55:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendit "KPI" (in reality sendmsg(2); maybe sendto(2)) will fail with EINVAL if mp->msg_control != NULL and mp->msg_controllen is < sizeof(struct cmsghdr); is not documented in send(2) Date: Sat, 23 Jan 2016 22:55:51 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable9? mfc-stable10? X-Bugzilla-Changed-Fields: assigned_to bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 22:55:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 NGie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|ngie@FreeBSD.org |freebsd-net@FreeBSD.org Status|In Progress |Open --- Comment #3 from NGie Cooper --- (In reply to Kubilay Kocak from comment #2) Hi koobs@! The change I proposed isn't going to fix the missing documentation. I can a= dd it later (will remain CCed on the bug), but I want to start a quick discuss= ion first with appropriate parties. Reassigning to freebsd-net and removing "In Progress" :). --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 23:32:13 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9BE8A8EEFC for ; Sat, 23 Jan 2016 23:32:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8DC91D34 for ; Sat, 23 Jan 2016 23:32:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NNWDhe064217 for ; Sat, 23 Jan 2016 23:32:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendit "KPI" (in reality sendmsg(2); maybe sendto(2)) will fail with EINVAL if mp->msg_control != NULL and mp->msg_controllen is < sizeof(struct cmsghdr); is not documented in send(2) Date: Sat, 23 Jan 2016 23:32:14 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable9? mfc-stable10? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 23:32:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 --- Comment #4 from Kubilay Kocak --- (In reply to NGie Cooper from comment #3) Ah thanks. Could you then update (terse'ify) the summary to describe either: * A summary of the 'issue', OR * A summary of the action/fix/change that is needed That way we can annotate/classify the issue accordingly. Am I understanding correctly that r294646 referenced in comment 1 was to temporarily disable a test due to the issue described here, until its resol= ved, by a change (patch) that will be added here in due course? --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 23:34:56 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31BBAA8EFA4 for ; Sat, 23 Jan 2016 23:34:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 21C471E29 for ; Sat, 23 Jan 2016 23:34:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NNYtrt069741 for ; Sat, 23 Jan 2016 23:34:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendmsg(2) (maybe sendto(2) as well) will fail with EINVAL; isn't documented in manpage Date: Sat, 23 Jan 2016 23:34:56 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable9? mfc-stable10? X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 23:34:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 NGie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|sendit "KPI" (in reality |sendmsg(2) (maybe sendto(2) |sendmsg(2); maybe |as well) will fail with |sendto(2)) will fail with |EINVAL; isn't documented in |EINVAL if mp->msg_control |manpage |!=3D NULL and | |mp->msg_controllen is < | |sizeof(struct cmsghdr); is | |not documented in send(2) | --- Comment #5 from NGie Cooper --- Bug 206543 is tracking the test issue. I've updated the description to be m= ore terse :). --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sat Jan 23 23:35:36 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DA2BA8EFFD for ; Sat, 23 Jan 2016 23:35:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1D241EC8 for ; Sat, 23 Jan 2016 23:35:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u0NNZZVM070835 for ; Sat, 23 Jan 2016 23:35:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206544] sendmsg(2) (sendto(2) too?) can fail with EINVAL; isn't documented in manpage Date: Sat, 23 Jan 2016 23:35:36 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable9? mfc-stable10? X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 23:35:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206544 NGie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|sendmsg(2) (maybe sendto(2) |sendmsg(2) (sendto(2) too?) |as well) will fail with |can fail with EINVAL; isn't |EINVAL; isn't documented in |documented in manpage |manpage | --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.=