From owner-freebsd-stable@freebsd.org Mon Feb 15 16:05:06 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C1ABAA945B for ; Mon, 15 Feb 2016 16:05:06 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 68ECF1FA7 for ; Mon, 15 Feb 2016 16:05:06 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: by mailman.ysv.freebsd.org (Postfix) id 64665AA945A; Mon, 15 Feb 2016 16:05:06 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A1CEAA9459 for ; Mon, 15 Feb 2016 16:05:06 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from smtp.unipi.it (smtp2.unipi.it [131.114.21.21]) by mx1.freebsd.org (Postfix) with ESMTP id D40231FA4 for ; Mon, 15 Feb 2016 16:05:05 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from localhost (localhost [127.0.0.1]) by smtp.unipi.it (Postfix) with ESMTP id 0033040902; Mon, 15 Feb 2016 16:56:42 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at unipi.it Received: from [10.216.1.203] (prova.iet.unipi.it [131.114.58.86]) (Authenticated User) by smtp.unipi.it (Postfix) with ESMTPSA id CD4B9405AD; Mon, 15 Feb 2016 16:56:40 +0100 (CET) Subject: Re: 82576 + NETMAP + VLAN To: Slawa Olhovchenkov References: <20160208173935.GK68298@zxy.spb.ru> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it> <20160210135318.GL68298@zxy.spb.ru> <56BC505F.7080309@iet.unipi.it> <20160211133428.GM68298@zxy.spb.ru> <56C1EA66.807@iet.unipi.it> <20160215151318.GQ68298@zxy.spb.ru> Cc: Luigi Rizzo , Adrian Chadd , "stable@freebsd.org" From: Giuseppe Lettieri Message-ID: <56C1F69C.5010004@iet.unipi.it> Date: Mon, 15 Feb 2016 17:02:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160215151318.GQ68298@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2016 16:05:06 -0000 Il 15/02/2016 16:13, Slawa Olhovchenkov ha scritto: > On Mon, Feb 15, 2016 at 04:10:30PM +0100, Giuseppe Lettieri wrote: > >> Hi Slawa, >> >> I think WITNESS is seeing a false positive, since those two are always >> different mutexes. >> >> The actual deadlock you experience should be caused by something else. I > > Are you sure? When deadlock occur I am see threads waiting on nm_kn_lock. The deadlock I mentioned still involves nm_kn_locks, sorry if I was not clear about that. I am just saying that we never try to take the same lock that we already holding. Nonetheless, there are indeed problems in the path that WITNESS has seen. The problem is that pipes have to notify the other end while called by kevent. kevent holds the nm_kn_lock on the TX src ring and the notification takes the nm_kn_lock on the RX dst ring. > >> have not been able to reproduce it locally (I have not tried that hard, >> to be honest). I am pretty sure that there is a lock inversion - one >> that may cause real deadlocks - when you use netmap pipes+kqueue and you >> don't pass NETMAP_NO_TX_POLL at NIOCREGIF time. The attached patch >> should solve this particular problem, but there may be others. May you >> please try it? > > Try it with or w/o WITNESS? I am trying to see if the actual deadlock disappears, so disable WITNESS if it slows down the system and masks the real deadlock. Otherwise, leave it on. Cheers, Giuseppe > >> Cheers, >> Giuseppe >> >> Il 11/02/2016 14:34, Slawa Olhovchenkov ha scritto: >>> On Thu, Feb 11, 2016 at 10:11:59AM +0100, Giuseppe Lettieri wrote: >>> >>>> Il 10/02/2016 14:53, Slawa Olhovchenkov ha scritto: >>>>> On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: >>>>> >>>>>> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: >>>>>>> Can you look also on second issue? >>>>>>> >>>>>>> PS: What need from me? May be open PR? >>>>>> >>>>>> May you provide some example code that triggers the issue? >>>>> >>>>> This is about 700 lines of code (not very clear), may be I can describe it? >>>> >>>> I just need some code to trigger the problem locally. Don't worry about >>>> the clarity and the line count, unless you cannot share the code for >>>> other reasons. >>> >>> I am attach source. >>> run as "prog if1 if2" >>> Got `acquiring duplicate lock of same type: "nm_kn_lock"` immediatly >>> after start. >>> Dead locking may be occur immediatly after start or may be need >>> traffic flooding. >>> >> >> >> -- >> Dr. Ing. Giuseppe Lettieri >> Dipartimento di Ingegneria della Informazione >> Universita' di Pisa >> Largo Lucio Lazzarino 1, 56122 Pisa - Italy >> Ph. : (+39) 050-2217.649 (direct) .599 (switch) >> Fax : (+39) 050-2217.600 >> e-mail: g.lettieri@iet.unipi.it > >> Index: dev/netmap/netmap.c >> =================================================================== >> --- dev/netmap/netmap.c (revision 287671) >> +++ dev/netmap/netmap.c (working copy) >> @@ -2378,7 +2378,7 @@ >> * XXX should also check cur != hwcur on the tx rings. >> * Fortunately, normal tx mode has np_txpoll set. >> */ >> - if (priv->np_txpoll || want_tx) { >> + if ((priv->np_txpoll && !is_kevent) || want_tx) { >> /* >> * The first round checks if anyone is ready, if not >> * do a selrecord and another round to handle races. > -- Dr. Ing. Giuseppe Lettieri Dipartimento di Ingegneria della Informazione Universita' di Pisa Largo Lucio Lazzarino 1, 56122 Pisa - Italy Ph. : (+39) 050-2217.649 (direct) .599 (switch) Fax : (+39) 050-2217.600 e-mail: g.lettieri@iet.unipi.it