From owner-svn-src-all@freebsd.org Thu May 2 13:14:43 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08F1415958B4; Thu, 2 May 2019 13:14:43 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A4678E8B9; Thu, 2 May 2019 13:14:42 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-ua1-f50.google.com (mail-ua1-f50.google.com [209.85.222.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 5A270BFCF; Thu, 2 May 2019 13:14:42 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-ua1-f50.google.com with SMTP id 90so675674uav.10; Thu, 02 May 2019 06:14:42 -0700 (PDT) X-Gm-Message-State: APjAAAWqLK3b4/qudZ4nIA5/wG66yt9K0zBUAy9r+RQ3z2GrDMowI/cd 5QxeAkmjMB8KHSRIpXe3lANlk0EbDxjt3cd/s40= X-Google-Smtp-Source: APXvYqzVsd1mY5MSF6qa3S4O4NhhWNA+oEWoqzdb1jD3wB3MshrD1VqMZ11EcG8btVJudzYI14sK0PL++1g0wg8EhHE= X-Received: by 2002:ab0:413:: with SMTP id 19mr1702257uav.87.1556802881748; Thu, 02 May 2019 06:14:41 -0700 (PDT) MIME-Version: 1.0 References: <20190502111106.pfosaq73kgo6g33j@Air-de-Roger.citrite.net> <201905021158.x42BwduG006865@slippy.cwsent.com> <20190502125334.ly3putfkfnxvbhqv@Air-de-Roger.citrite.net> In-Reply-To: <20190502125334.ly3putfkfnxvbhqv@Air-de-Roger.citrite.net> From: Kyle Evans Date: Thu, 2 May 2019 08:14:08 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: svn commit: r346670 - head/sys/net To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Cy Schubert , src-committers , svn-src-all@freebsd.org, svn-src-head@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 9A4678E8B9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 May 2019 13:14:43 -0000 On Thu, May 2, 2019 at 7:54 AM Roger Pau Monn=C3=A9 = wrote: > > On Thu, May 02, 2019 at 04:58:39AM -0700, Cy Schubert wrote: > > In message <20190502111106.pfosaq73kgo6g33j@Air-de-Roger.citrite.net>, > > Roger Pa > > u =3D?utf-8?B?TW9ubsOp?=3D writes: > > > On Thu, Apr 25, 2019 at 12:44:08PM +0000, Kyle Evans wrote: > > > Apr 26 16:23:57.662653 panic: mtx_lock() of spin mutex (null) @ /usr/= home/oss > > > test/build.135317.build-amd64-freebsd/freebsd/sys/kern/subr_bus.c:620 > > > Apr 26 16:23:57.674650 cpuid =3D 2 > > > Apr 26 16:23:57.686653 time =3D 1 > > > Apr 26 16:23:57.686720 KDB: stack backtrace: > > > Apr 26 16:23:57.686797 db_trace_self_wrapper() at db_trace_self_wrapp= er+0x2b/ > > > frame 0xfffffe003abe8710 > > > Apr 26 16:23:57.686879 vpanic() at vpanic+0x19d/frame 0xfffffe003abe8= 760 > > > Apr 26 16:23:57.698637 panic() at panic+0x43/frame 0xfffffe003abe87c0 > > > Apr 26 16:23:57.698700 __mtx_lock_flags() at __mtx_lock_flags+0x145/f= rame 0xf > > > ffffe003abe8810 > > > Apr 26 16:23:57.710640 devctl_queue_data_f() at devctl_queue_data_f+0= x6a/fram > > > e 0xfffffe003abe8840 > > > Apr 26 16:23:57.722625 g_dev_taste() at g_dev_taste+0x463/frame 0xfff= ffe003ab > > > e8a00 > > > Apr 26 16:23:57.722690 g_load_class() at g_load_class+0x1bc/frame 0xf= ffffe003 > > > abe8a30 > > > Apr 26 16:23:57.734638 g_run_events() at g_run_events+0x197/frame 0xf= ffffe003 > > > abe8a70 > > > Apr 26 16:23:57.734704 fork_exit() at fork_exit+0x84/frame 0xfffffe00= 3abe8ab0 > > > Apr 26 16:23:57.746655 fork_trampoline() at fork_trampoline+0xe/frame= 0xfffff > > > e003abe8ab0 > > > Apr 26 16:23:57.746721 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --= - > > > Apr 26 16:23:57.758797 KDB: enter: panic > > > Apr 26 16:23:57.758913 [ thread pid 13 tid 100029 ] > > > Apr 26 16:23:57.758943 Stopped at kdb_enter+0x3b: movq $0,kdb= _why > > > Apr 26 16:23:57.770557 db> > > > > > > The automatic bisector has pointed as this commit as the culprit, you > > > can see the full bisection at: > > > > > > https://lists.xenproject.org/archives/html/xen-devel/2019-04/msg02061= .html > > > > > > And an example of a failed test at: > > > > > > https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg00104= .html > > > http://logs.test-lab.xenproject.org/osstest/logs/135458/ > > > > > > Thanks, Roger. > > > > > > > > > > It made a strange connection to this commit. The panic has geom written > > all over it. > > I agree it's a strange connection, but the results from the bisection > are quite clear, the previous commit which is 070cf1ede1850d8c > (r346664) works fine and d61e108233bfdb3 (r346670) this commit > fails. > > The bisection looks reliable as there are no skipped revisions or > spurious failures. > This panic seems to make sense, generally if I read things right, but I'm not immediately sure how my commit triggered it. The mutex in question is initialized in devinit, invoked by the root_bus_mod at SI_SUB_DRIVERS + SI_ORDER_FIRST [0]. geom classes are also declared at SI_SUB_DRIVERS + SI_ORDER_FIRST [1], which takes us on a trip through g_modevent -> g_init. g_init creates the g_event thread [2] -> g_event_procbody -> g_run_events -> ... -> (boom). I guess the timing/ordering of these things normally works out so that devinit gets the mutex ready before the g_event thread does any loading/tasting, but not here. [0] https://svnweb.freebsd.org/base/head/sys/kern/subr_bus.c?view=3Dmarkup#= l5127 [1] https://svnweb.freebsd.org/base/head/sys/geom/geom.h?view=3Dmarkup#l414 [2] https://svnweb.freebsd.org/base/head/sys/geom/geom_kern.c?view=3Dmarkup= #l149