From owner-freebsd-stable@freebsd.org Thu Feb 11 09:06:07 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65DE4AA5258 for ; Thu, 11 Feb 2016 09:06:07 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4A8591DB1 for ; Thu, 11 Feb 2016 09:06:07 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: by mailman.ysv.freebsd.org (Postfix) id 4A168AA5257; Thu, 11 Feb 2016 09:06:07 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49B0AAA5256 for ; Thu, 11 Feb 2016 09:06:07 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from smtp.unipi.it (smtp1.unipi.it [131.114.21.19]) by mx1.freebsd.org (Postfix) with ESMTP id 123121DB0 for ; Thu, 11 Feb 2016 09:06:06 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from localhost (localhost [127.0.0.1]) by smtp.unipi.it (Postfix) with ESMTP id 1AD1F42073; Thu, 11 Feb 2016 10:06:04 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at unipi.it Received: from [10.216.1.203] (prova.iet.unipi.it [131.114.58.86]) (Authenticated User) by smtp.unipi.it (Postfix) with ESMTPSA id 557C64207C; Thu, 11 Feb 2016 10:06:03 +0100 (CET) Subject: Re: 82576 + NETMAP + VLAN To: Slawa Olhovchenkov References: <20160202204446.GQ88527@zxy.spb.ru> <20160204130029.GC88527@zxy.spb.ru> <20160208173935.GK68298@zxy.spb.ru> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it> <20160210135318.GL68298@zxy.spb.ru> Cc: Luigi Rizzo , Adrian Chadd , "stable@freebsd.org" From: Giuseppe Lettieri Message-ID: <56BC505F.7080309@iet.unipi.it> Date: Thu, 11 Feb 2016 10:11:59 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160210135318.GL68298@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Feb 2016 09:06:07 -0000 Il 10/02/2016 14:53, Slawa Olhovchenkov ha scritto: > On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: > >> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: >>> Can you look also on second issue? >>> >>> PS: What need from me? May be open PR? >> >> May you provide some example code that triggers the issue? > > This is about 700 lines of code (not very clear), may be I can describe it? I just need some code to trigger the problem locally. Don't worry about the clarity and the line count, unless you cannot share the code for other reasons. Cheers, Giuseppe > First: this is thread code. I think it impotant point. > All sync with netmap go throw kevent() interface. > > Main thread got information about two NIC (determine number of rings) > and spawn 1 balancer thread and NCPU worker thread. > All thread pinned to cores. > > Balancer thread open all rings on both NIC separate (i.e. open 6+6 > NETMAP fds for hardware rings) and allocate on first NIC and open > NCPU*2 master NETMAP pipes. > All fds register in kevent for read event as > > EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx); > > and for future write event as > > EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISABLE, 0, 0, 0); > > after write occur next event will be added: > > EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0); > > Each worker thread open two slave NETMAP pipes in same maner as above. > > Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe. > Worker got packet from inside slave pipe and zero-copy to outside slave pipe. > Balancer got packet from outside master pipe and dispatch (zero-copy) to outside hardware ring. > And vise versa. > > After program start I have deadlock instantly (with WITNESS enabled kernel). > W/o WITNESS deadloc occur only under load. > > W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets. > NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread. > -- Dr. Ing. Giuseppe Lettieri Dipartimento di Ingegneria della Informazione Universita' di Pisa Largo Lucio Lazzarino 1, 56122 Pisa - Italy Ph. : (+39) 050-2217.649 (direct) .599 (switch) Fax : (+39) 050-2217.600 e-mail: g.lettieri@iet.unipi.it