From owner-freebsd-drivers@FreeBSD.ORG Sat May 31 17:57:38 2008 Return-Path: Delivered-To: freebsd-drivers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FF00106564A for ; Sat, 31 May 2008 17:57:38 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from tom.kiev.farlep.net (tom.kiev.farlep.net [213.130.24.4]) by mx1.freebsd.org (Postfix) with ESMTP id E68C98FC14 for ; Sat, 31 May 2008 17:57:37 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from ilya.kiev.farlep.net ([62.221.47.37]:1476 helo=[10.0.0.3]) by tom.kiev.farlep.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1K2VKg-0003bk-0S; Sat, 31 May 2008 20:57:34 +0300 Message-ID: <48419182.1030200@po4ta.com> Date: Sat, 31 May 2008 20:57:22 +0300 From: Ilya Bobir User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Alexander Popov References: <987362.36986.qm@web51404.mail.re2.yahoo.com> In-Reply-To: <987362.36986.qm@web51404.mail.re2.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Farlep-Data: Cc: freebsd-drivers@FreeBSD.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 May 2008 17:57:38 -0000 Alexander Popov wrote: > Hi, Warner, > > Thanks, I appreciate your help. > Yes, indeed I am writing a USB device driver. I have done some tests, > and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this: > > void driver_open() > { > mtx_lock(&sc->mtx) > // open USB pipes to device, etc. > mtx_unlock(&sc->mtx) > } > > void driver_write(...) > { > mtx_lock(&sc->mtx) > // write to device (I do not sleep here!) > mtx_unlock(&sc->mtx) > } > > then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock". The only spot in the source were I was able to find a message like this is in src/sys/kern/subr_turnstile.c:propagate_priority(struct thread *td) (http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/subr_turnstile.c?rev=1.173;content-type=text%2Fplain search for "non-sleepable"). IIUC, this message means that a thread that is sleeping on a lock also holds a spin lock. Another thread (current thread) is trying to acquire this spin lock but finds out that the current owner of the spin lock is sleeping. As sleeping while holding a spin lock is prohibited as it may cause a deadlock the function panics. Note that the misbehaving thread's id is present in the message. Chapter 11 of the "FreeBSD Developers' Handbook" explains how you can get a thread stack trace from a kernel dump or a running kernel. But you probably already know it. > When I change default mutex to spin type, it works well, and I was not able to crash it. As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked. > It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes... > I think you would get a lock order reversal message or some other kind of message if you are locking Giant after some other lock. "Sleeping thread (tid %d, pid %d) owns a non-sleepable lock\n" is not about Giant. > However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet... > [...] It probably means that the mutex is already locked. You can drop into a debugger and try do debug it as described in "11.9 Debugging Deadlocks" in the "FreeBSD Developers' Handbook". P.S. Note, that sleeping and preemption are different. A thread is sleeping when it is waiting on a sleepable lock. While waiting on a non-sleepable lock or in some other cases a thread may be preempted but it is not marked as sleeping.