Date: Thu, 11 Feb 2016 10:11:59 +0100 From: Giuseppe Lettieri <g.lettieri@iet.unipi.it> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, Adrian Chadd <adrian.chadd@gmail.com>, "stable@freebsd.org" <stable@freebsd.org> Subject: Re: 82576 + NETMAP + VLAN Message-ID: <56BC505F.7080309@iet.unipi.it> In-Reply-To: <20160210135318.GL68298@zxy.spb.ru> References: <CA%2BhQ2%2Bg0ggpS%2BE5nOpON66efs7cwsed=NvaKa=mzsg6FycGhiQ@mail.gmail.com> <20160202204446.GQ88527@zxy.spb.ru> <20160204130029.GC88527@zxy.spb.ru> <CAJ-Vmok%2B7Vt4ww4iWkQY505eapxVQF4MBtnb%2BwGg-TNSmJTLGw@mail.gmail.com> <CAJ-VmomzvoZZZPUveTZUJ5zAhHQkJ5M9%2B7gfN8gGSGp05JpOWw@mail.gmail.com> <20160208173935.GK68298@zxy.spb.ru> <CA%2BhQ2%2BiD3X9wR8exw2p-9G8pPNHCQtLdMdJJXU78PDrQaWBH7w@mail.gmail.com> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it> <20160210135318.GL68298@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Il 10/02/2016 14:53, Slawa Olhovchenkov ha scritto: > On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: > >> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: >>> Can you look also on second issue? >>> >>> PS: What need from me? May be open PR? >> >> May you provide some example code that triggers the issue? > > This is about 700 lines of code (not very clear), may be I can describe it? I just need some code to trigger the problem locally. Don't worry about the clarity and the line count, unless you cannot share the code for other reasons. Cheers, Giuseppe > First: this is thread code. I think it impotant point. > All sync with netmap go throw kevent() interface. > > Main thread got information about two NIC (determine number of rings) > and spawn 1 balancer thread and NCPU worker thread. > All thread pinned to cores. > > Balancer thread open all rings on both NIC separate (i.e. open 6+6 > NETMAP fds for hardware rings) and allocate on first NIC and open > NCPU*2 master NETMAP pipes. > All fds register in kevent for read event as > > EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx); > > and for future write event as > > EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISABLE, 0, 0, 0); > > after write occur next event will be added: > > EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0); > > Each worker thread open two slave NETMAP pipes in same maner as above. > > Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe. > Worker got packet from inside slave pipe and zero-copy to outside slave pipe. > Balancer got packet from outside master pipe and dispatch (zero-copy) to outside hardware ring. > And vise versa. > > After program start I have deadlock instantly (with WITNESS enabled kernel). > W/o WITNESS deadloc occur only under load. > > W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets. > NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread. > -- Dr. Ing. Giuseppe Lettieri Dipartimento di Ingegneria della Informazione Universita' di Pisa Largo Lucio Lazzarino 1, 56122 Pisa - Italy Ph. : (+39) 050-2217.649 (direct) .599 (switch) Fax : (+39) 050-2217.600 e-mail: g.lettieri@iet.unipi.it
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?56BC505F.7080309>