From owner-freebsd-drivers@FreeBSD.ORG Tue May 27 09:59:15 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89495106566B for ; Tue, 27 May 2008 09:59:15 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: from web51410.mail.re2.yahoo.com (web51410.mail.re2.yahoo.com [206.190.38.189]) by mx1.freebsd.org (Postfix) with SMTP id 4943D8FC15 for ; Tue, 27 May 2008 09:59:14 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: (qmail 32982 invoked by uid 60001); 27 May 2008 09:32:34 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=i4RqX/YtWYa3RjRIwL7Ue3VKgbzvd3rde6STsA4MDKZR9WEfKgq7xF6s5PT+u+VQGDoX4jkaEE/rVwuuEWwjae5SvY7QU5etDhr1DJ0zDQWIjVGBHwkZzsgVIL3MnGkO52UWGLrGGvF707E4RZJPV74dGdwolsjF8cG0QzUpl88=; Received: from [193.173.39.45] by web51410.mail.re2.yahoo.com via HTTP; Tue, 27 May 2008 02:32:34 PDT X-Mailer: YahooMailRC/975.41 YahooMailWebService/0.7.199 Date: Tue, 27 May 2008 02:32:34 -0700 (PDT) From: Alexander Popov To: freebsd-drivers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <730729.32628.qm@web51410.mail.re2.yahoo.com> Subject: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2008 09:59:15 -0000 Hello all, I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done. Previously, i.e. in 4.4 BSD, the following statements were true: 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode; 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver). This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever. Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE. It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: int driver_read(...) { mtx_lock(&sc->mtx); if (no data is available) => tsleep(..) mtx_unlock(&sc->mtx); } First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel... The bottom line is: 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this); 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed; 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed. Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver? Of course, I might be missing something here, please help. Thanks in advance, Alexander.