From owner-freebsd-current@freebsd.org Thu Aug 31 21:51:16 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C58F9E06793 for ; Thu, 31 Aug 2017 21:51:16 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9AECE693D9 for ; Thu, 31 Aug 2017 21:51:16 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-oi0-x233.google.com with SMTP id t75so7751632oie.3 for ; Thu, 31 Aug 2017 14:51:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=5Gxt1bRKaPS2J4Pqxp24QRaU+/dq5bN39mxhbGaUYgQ=; b=imaHIlAbhJELIioSRAnDbFNZHnLon73PNyDzCxqnz8G7Trr51WjIpBXHPMxSwP821H pa/8MgNPhYUAdyTHZTdWp8XjHX7ncCB6ihCh2SSjbcf6ACeNSRAjqpBzRXaW7bcsrl6T F7HwliFCmHlUsOMEiZnp7cD7aCO8B/Q00p5hw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=5Gxt1bRKaPS2J4Pqxp24QRaU+/dq5bN39mxhbGaUYgQ=; b=cb1fzVOMKdapsaJ3J3XakhVBjAwEwnWyET4m6VUI0OqohhJ/RCx2zvQ8451N3JtAaM Y9hebxZGgcdj0jvpAWIHD8k4kv+a38b7RD19QyLUcMDvwa16Xt6UoRv7tRTkpyxniA78 agJOwcKryF5ScJVyu8wPqyseXpkXGnLxCyDHJ8V5G2ztonWL46Cy9uVXhSmSNgdfMvCt BDZ/A7GwJPCK/ZRD6ny9JIWBHKbNIYu/FX2P4MXN2oEJNOmtHLGW8pc0hkIur9ilsWKM QoTjCgaRwUIP2aZlZovW2q+YwSspagP231Tn4GMEAJgGk9OgPNEGFuvLrAIIXZTj6qQr eAHA== X-Gm-Message-State: AHYfb5iQlr0kdh1UFqnB3jhiuup+hvD5YVhHMtrXn5ObChl9HeTeoUuK aICia6T5BxmOgfhr/tYr/hRruA0sWyva X-Google-Smtp-Source: ADKCNb5CxYk3ibJ5dORgjvOMMFqjK1M1P47ZizDfiT4VzP3hRC8YJgQh/OEl041MbSD8peI1+EbImF6IsLP0109uIfc= X-Received: by 10.202.87.78 with SMTP id l75mr7607455oib.137.1504216275754; Thu, 31 Aug 2017 14:51:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.57.165 with HTTP; Thu, 31 Aug 2017 14:51:15 -0700 (PDT) In-Reply-To: <2607331.p0nfSuSmN8@ralph.baldwin.cx> References: <2607331.p0nfSuSmN8@ralph.baldwin.cx> From: Kevin Bowling Date: Thu, 31 Aug 2017 14:51:15 -0700 Message-ID: Subject: Re: !EARLY_AP_STARTUP and -CURRENT To: John Baldwin Cc: freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2017 21:51:16 -0000 panic: mutex sched lock 0 not owned at /d0/kev/freebsd/sys/kern/sched_ule.c:2379 On Thu, Aug 31, 2017 at 7:38 AM, John Baldwin wrote: > On Wednesday, August 30, 2017 04:54:07 PM Kevin Bowling wrote: >> I'm dealing with a shit sandwich right now where the mps(4) or cam_da >> reorders drives on a few thousand legacy MBR machines I have (and I >> can't easily install glabel ATM), and !EARLY_AP_STARTUP seems to have >> regressed. I'd like to be able to run w/o EARLY_AP_STARTUP right >> quick so I can take a more leisurely approach to fixing mps(4) boot >> probe correctly (freebsd-scsi@ has that thread). >> >> With WITNESS and !EARLY_AP_STARTUP I hit an assert in sched_setpreempt >> in kern/sched_ule.c 100% of the time. Here are a couple invocations, >> with oddness around a different CPU holding the curthread lock but >> somehow a different AP is runnable in the function: > > Do you have the panic messages? > >> Tracing pid 11 tid 100020 td 0xfffff80128cd1560 >> kdb_enter() at kdb_enter+0x3b/frame 0xfffffe3e653dcc10 >> vpanic() at vpanic+0x1b9/frame 0xfffffe3e653dcc90 >> panic() at panic+0x43/frame 0xfffffe3e653dccf0 >> __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe3e653dcd00 >> sched_add() at sched_add+0x152/frame 0xfffffe3e653dcd40 >> intr_event_schedule_thread() at intr_event_schedule_thread+0xca/frame >> 0xfffffe3e653dcd80 >> swi_sched() at swi_sched+0x6c/frame 0xfffffe3e653dcdc0 >> softclock_call_cc() at softclock_call_cc+0x155/frame 0xfffffe3e653dce70 >> callout_process() at callout_process+0x1f9/frame 0xfffffe3e653dcef0 >> handleevents() at handleevents+0x1a4/frame 0xfffffe3e653dcf30 >> cpu_initclocks_ap() at cpu_initclocks_ap+0xc8/frame 0xfffffe3e653dcf60 >> init_secondary_tail() at init_secondary_tail+0x1e3/frame 0xfffffe3e653dcf90 >> init_secondary() at init_secondary+0x2b3/frame 0xfffffe3e653dcff0 >> >> >> db> show thread 0xfffff80128cd1560 >> Thread 100020 at 0xfffff80128cd1560: >> proc (pid 11): 0xfffff80128cb5000 >> name: idle: cpu17 >> stack: 0xfffffe3e5cd88000-0xfffffe3e5cd8bfff >> flags: 0x40024 pflags: 0x200000 >> state: CAN RUN >> priority: 255 >> container lock: sched lock 0 (0xffffffff81c39800) >> db> show lock 0xffffffff81c39800 >> class: spin mutex >> name: sched lock 0 >> flags: {SPIN, RECURSE} >> state: {OWNED} >> owner: 0xfffff80128cca000 (tid 100017, pid 11, "idle: cpu14") >> >> >> db> bt >> Tracing pid 11 tid 100021 td 0xfffff80128cd2000 >> kdb_enter() at kdb_enter+0x3b/frame 0xfffffe3e655e4c10 >> vpanic() at vpanic+0x1b9/frame 0xfffffe3e655e4c90 >> panic() at panic+0x43/frame 0xfffffe3e655e4cf0 >> __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe3e655e4d00 >> sched_add() at sched_add+0x152/frame 0xfffffe3e655e4d40 >> intr_event_schedule_thread() at intr_event_schedule_thread+0xca/frame >> 0xfffffe3e655e4d80 >> swi_sched() at swi_sched+0x6c/frame 0xfffffe3e655e4dc0 >> softclock_call_cc() at softclock_call_cc+0x155/frame 0xfffffe3e655e4e70 >> callout_process() at callout_process+0x1f9/frame 0xfffffe3e655e4ef0 >> handleevents() at handleevents+0x1a4/frame 0xfffffe3e655e4f30 >> cpu_initclocks_ap() at cpu_initclocks_ap+0xc8/frame 0xfffffe3e655e4f60 >> init_secondary_tail() at init_secondary_tail+0x1e3/frame 0xfffffe3e655e4f90 >> init_secondary() at init_secondary+0x2b3/frame 0xfffffe3e655e4ff0 >> db> show thread 0xfffff80128cd2000 >> Thread 100021 at 0xfffff80128cd2000: >> proc (pid 11): 0xfffff80128cb6000 >> name: idle: cpu18 >> stack: 0xfffffe3e5cf17000-0xfffffe3e5cf1afff >> flags: 0x40024 pflags: 0x200000 >> state: CAN RUN >> priority: 255 >> container lock: sched lock 0 (0xffffffff81c39800) >> db> show lock 0xffffffff81c39800 >> class: spin mutex >> name: sched lock 0 >> flags: {SPIN, RECURSE} >> state: {OWNED} >> owner: 0xfffff80128cdb560 (tid 100028, pid 11, "idle: cpu25") >> >> Regards, >> Kevin > > > -- > John Baldwin