From owner-freebsd-arch@FreeBSD.ORG Thu Nov 13 09:52:52 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8E023483; Thu, 13 Nov 2014 09:52:52 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A2C5EB0; Thu, 13 Nov 2014 09:52:52 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id x13so16643931wgg.19 for ; Thu, 13 Nov 2014 01:52:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=zghHcdYlYyirQMi3dXPjo6qAxQk+1a+RzJMmRvDHgu0=; b=bT8fiR1h6BlU7AvmRXtyCt0uuLeCSt1rYn6njCIgkFaACLj1ud2CFuNHvIRwRMGvBY dfi2RU+MRMz0348f+iYTlQRRHGiej5UDAgwv7LZGAViUnTIvMmGpcKGlvVmNcazElAPP f/VgXOcoKQTcAoUa2qJEdk/Qj2O6/It0tW9yNaI0wkOS8kzzm94hjZm8inbyWGUNW8m4 bzes5GeE4td2dgnQHoylUTHXKcGF8WhFiFuNiOLBqv2WGvrEwLK5dei79+d+7Pbp73UU RzoUbcGUiAU6VvSEVc3C1DD7W3tnnnd+4Kl3KrifafwKg3KDRG5hk4Rd6onfiYkwRuVS 0o6A== MIME-Version: 1.0 X-Received: by 10.194.85.83 with SMTP id f19mr2429878wjz.20.1415872370440; Thu, 13 Nov 2014 01:52:50 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Thu, 13 Nov 2014 01:52:50 -0800 (PST) In-Reply-To: <54647D1E.9010904@freebsd.org> References: <20141112212613.21037929@kan> <546472DA.3080006@freebsd.org> <5464764E.9080308@freebsd.org> <54647D1E.9010904@freebsd.org> Date: Thu, 13 Nov 2014 01:52:50 -0800 X-Google-Sender-Auth: MXuuXbVAppHEhEieRbE4bsCdYJQ Message-ID: Subject: Re: Questions about locking; turnstiles and sleeping threads From: Adrian Chadd To: Alfred Perlstein Content-Type: text/plain; charset=UTF-8 Cc: Hans Petter Selasky , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 09:52:52 -0000 Hm, the more I dig into this, the more I realise it's not a 1:45am question to ask. Specifically, callout_stop_safe() takes 'safe', which says "are we waiting around for this callout to finish if it started". Ie, callout_drain() is callout_stop_safe(c, 1) ; callout_stop() is callout_stop_safe(c, 0). If safe is 1, then it'll potentially put the current thread to sleep in order to wait for it to synchronise with the callout that's running. It's sleeping with cc_lock which is the per-callwheel lock and it's doing that with whatever other locks are held. That's the situation which is tripping things up. The manpage says that no locks should be held that the callout may block on, which isn't the case here at all - I'm trying to grab a lock in another thread that the caller _into_ the callout subsystem holds. The manpage doesn't mention anything about this. Sniffle. -adrian On 13 November 2014 01:42, Alfred Perlstein wrote: > OK that makes more sense. > > I've cc'd Hans for the usb issue. > > > On 11/13/14, 1:38 AM, Adrian Chadd wrote: >> >> It looks like the initial firings are because the check I put in >> didn't check to see if it's MPSAFE. >> >> eg: >> >> ip6_input -> tcp6_input -> tcp_input -> tcp_do_segment -> >> tcp_timer_active -> callout_stop_safe; called with tcpinp held. >> closefp() -> closef() -> fdrop -> soclose() -> sofree() -> >> tcp_usr_detach() -> tcp_discardcb() -> callout_stop_safe() with the >> tcpinp and tcp locks held. >> ioctl -> sys_ioctl-> devfs_ioctl_f -> acpi_ackSleepState -> >> callout_stop_safe; with ACPI global lock held; >> suspend path -> hdaa_suspend -> callout_stop_safe() with HDA driver mutex >> held >> >> So we can't just put the simple witness check from _sleep() in >> _callout_stop_safe(), it looks like it's mis-firing on MPSAFE callouts >> (which the tcp timers all are) and that won't go via the sleepq. >> It looks like the acpi callout is also mpsafe, as well as the HDA jack >> callout. >> >> However, I did pick up this: >> >> detach path -> usbd_transfer_drain() -> usbd_transfer_stop() -> >> ehci_device_intr_close() -> usbd_transfer_done() -> >> callout_stop_safe() with USB HUB mutex held >> >> The usbd_transfer_done() callout is initialised with a mutex, but the >> witness code should've detected it wasn't callout->c_lock and thus >> warned. >> >> >> >> -adrian >> >> On 13 November 2014 01:13, Alfred Perlstein wrote: >>> >>> On 11/13/14, 1:09 AM, Adrian Chadd wrote: >>>> >>>> On 13 November 2014 00:59, Alfred Perlstein wrote: >>>>> >>>>> On 11/12/14, 11:25 PM, Adrian Chadd wrote: >>>>>> >>>>>> On 12 November 2014 19:48, Adrian Chadd wrote: >>>>>>> >>>>>>> kan pointed out that we should likely do a WITNESS_WARN() in the >>>>>>> relevant spots in the callout code so we catch these before it >>>>>>> happens. >>>>>>> >>>>>>> I'll see what we can add to -HEAD to be pro-active about it. >>>>>> >>>>>> Amusingly, I tried adding it and it made my laptop turn to soup very >>>>>> quickly - among other things, the TCP timer callouts are all setup >>>>>> with non sleep locks held. >>>>>> >>>>> Howso? You only have to worry about callout_drain(), most other >>>>> callout >>>>> functions should be safe-ish.... >>>> >>>> yeah, except for all the places where they drain the timer once >>>> something happens so it doesn't fire. >>>> >>>> :) >>> >>> >>> What is the backtrace...? >>> >>> >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >> >