From owner-freebsd-stable@freebsd.org Wed Feb 10 13:53:22 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9B4EAA441E for ; Wed, 10 Feb 2016 13:53:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D632BF63 for ; Wed, 10 Feb 2016 13:53:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id D22FDAA441D; Wed, 10 Feb 2016 13:53:22 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1D02AA441C for ; Wed, 10 Feb 2016 13:53:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93135F62 for ; Wed, 10 Feb 2016 13:53:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aTVCo-0004on-8S; Wed, 10 Feb 2016 16:53:18 +0300 Date: Wed, 10 Feb 2016 16:53:18 +0300 From: Slawa Olhovchenkov To: Giuseppe Lettieri Cc: Luigi Rizzo , Adrian Chadd , "stable@freebsd.org" Subject: Re: 82576 + NETMAP + VLAN Message-ID: <20160210135318.GL68298@zxy.spb.ru> References: <20160202204446.GQ88527@zxy.spb.ru> <20160204130029.GC88527@zxy.spb.ru> <20160208173935.GK68298@zxy.spb.ru> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56BB3C20.600@iet.unipi.it> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2016 13:53:23 -0000 On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: > Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: > > Can you look also on second issue? > > > > PS: What need from me? May be open PR? > > May you provide some example code that triggers the issue? This is about 700 lines of code (not very clear), may be I can describe it? First: this is thread code. I think it impotant point. All sync with netmap go throw kevent() interface. Main thread got information about two NIC (determine number of rings) and spawn 1 balancer thread and NCPU worker thread. All thread pinned to cores. Balancer thread open all rings on both NIC separate (i.e. open 6+6 NETMAP fds for hardware rings) and allocate on first NIC and open NCPU*2 master NETMAP pipes. All fds register in kevent for read event as EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx); and for future write event as EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISABLE, 0, 0, 0); after write occur next event will be added: EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0); Each worker thread open two slave NETMAP pipes in same maner as above. Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe. Worker got packet from inside slave pipe and zero-copy to outside slave pipe. Balancer got packet from outside master pipe and dispatch (zero-copy) to outside hardware ring. And vise versa. After program start I have deadlock instantly (with WITNESS enabled kernel). W/o WITNESS deadloc occur only under load. W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets. NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread.