Date: Wed, 10 Feb 2016 16:53:18 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Giuseppe Lettieri <g.lettieri@iet.unipi.it> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, Adrian Chadd <adrian.chadd@gmail.com>, "stable@freebsd.org" <stable@freebsd.org> Subject: Re: 82576 + NETMAP + VLAN Message-ID: <20160210135318.GL68298@zxy.spb.ru> In-Reply-To: <56BB3C20.600@iet.unipi.it> References: <CA%2BhQ2%2Bg0ggpS%2BE5nOpON66efs7cwsed=NvaKa=mzsg6FycGhiQ@mail.gmail.com> <20160202204446.GQ88527@zxy.spb.ru> <20160204130029.GC88527@zxy.spb.ru> <CAJ-Vmok%2B7Vt4ww4iWkQY505eapxVQF4MBtnb%2BwGg-TNSmJTLGw@mail.gmail.com> <CAJ-VmomzvoZZZPUveTZUJ5zAhHQkJ5M9%2B7gfN8gGSGp05JpOWw@mail.gmail.com> <20160208173935.GK68298@zxy.spb.ru> <CA%2BhQ2%2BiD3X9wR8exw2p-9G8pPNHCQtLdMdJJXU78PDrQaWBH7w@mail.gmail.com> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it>
index | next in thread | previous in thread | raw e-mail
On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: > Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: > > Can you look also on second issue? > > > > PS: What need from me? May be open PR? > > May you provide some example code that triggers the issue? This is about 700 lines of code (not very clear), may be I can describe it? First: this is thread code. I think it impotant point. All sync with netmap go throw kevent() interface. Main thread got information about two NIC (determine number of rings) and spawn 1 balancer thread and NCPU worker thread. All thread pinned to cores. Balancer thread open all rings on both NIC separate (i.e. open 6+6 NETMAP fds for hardware rings) and allocate on first NIC and open NCPU*2 master NETMAP pipes. All fds register in kevent for read event as EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx); and for future write event as EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISABLE, 0, 0, 0); after write occur next event will be added: EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0); Each worker thread open two slave NETMAP pipes in same maner as above. Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe. Worker got packet from inside slave pipe and zero-copy to outside slave pipe. Balancer got packet from outside master pipe and dispatch (zero-copy) to outside hardware ring. And vise versa. After program start I have deadlock instantly (with WITNESS enabled kernel). W/o WITNESS deadloc occur only under load. W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets. NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160210135318.GL68298>