From owner-freebsd-drivers@FreeBSD.ORG Sun May 25 23:21:25 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32BA8106566B for ; Sun, 25 May 2008 23:21:25 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from plato.miralink.com (mail.miralink.com [70.103.185.20]) by mx1.freebsd.org (Postfix) with ESMTP id 0508F8FC0A for ; Sun, 25 May 2008 23:21:24 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by plato.miralink.com (Postfix) with ESMTP id 53FD61A9334 for ; Sun, 25 May 2008 16:20:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at X-Spam-Flag: NO X-Spam-Score: -4.399 X-Spam-Level: X-Spam-Status: No, score=-4.399 tagged_above=-10 required=6.6 tests=[ALL_TRUSTED=-1.8, BAYES_00=-2.599] Received: from plato.miralink.com ([127.0.0.1]) by localhost (plato.miralink.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KBe13dsJoQ9l for ; Sun, 25 May 2008 16:20:29 -0700 (PDT) Received: from [10.47.1.18] (vpn.office.miralink.com [10.0.0.5]) by plato.miralink.com (Postfix) with ESMTP id E6E2D1A9331 for ; Sun, 25 May 2008 16:20:28 -0700 (PDT) Message-ID: <4839F473.6070109@miralink.com> Date: Sun, 25 May 2008 16:21:23 -0700 From: Sean Bruno User-Agent: Thunderbird 2.0.0.14 (X11/20080501) MIME-Version: 1.0 To: freebsd-drivers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: ATA patch for RELENG_6 ... a patch looking for a good home X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 May 2008 23:21:25 -0000 I got this patch a while ago and I don't see it appearing in RELENG_6 yet. Can someone "sheperd" this along or point out why it's not acceptable? This patch was generated by a failure to boot correctly off of a compact flash IDE module from Transcend. Index: dev/ata/ata-chipset.c =================================================================== --- dev/ata/ata-chipset.c (.../FreeBSD_RELENG_6_13APR07/src/sys) (revision 5436) +++ dev/ata/ata-chipset.c (.../miralink.FreeBSD.6/src/sys) (revision 5436) @@ -2059,7 +2059,8 @@ atadev->mode = ATA_SA150; } else { - mode = ata_limit_mode(dev, mode, ATA_UDMA5); + /*mode = ata_limit_mode(dev, mode, ATA_UDMA5);*/ + mode = ata_check_80pin(dev, ATA_UDMA5); if (!ata_controlcmd(dev, ATA_SETFEATURES, ATA_SF_SETXFER, 0, mode)) atadev->mode = mode; } -- Sean Bruno MiraLink Corporation 6015 NE 80th Ave, Ste 100 Portland, OR 97218 Phone 503-621-5143 Fax 503-621-5199 From owner-freebsd-drivers@FreeBSD.ORG Tue May 27 09:59:15 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89495106566B for ; Tue, 27 May 2008 09:59:15 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: from web51410.mail.re2.yahoo.com (web51410.mail.re2.yahoo.com [206.190.38.189]) by mx1.freebsd.org (Postfix) with SMTP id 4943D8FC15 for ; Tue, 27 May 2008 09:59:14 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: (qmail 32982 invoked by uid 60001); 27 May 2008 09:32:34 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=i4RqX/YtWYa3RjRIwL7Ue3VKgbzvd3rde6STsA4MDKZR9WEfKgq7xF6s5PT+u+VQGDoX4jkaEE/rVwuuEWwjae5SvY7QU5etDhr1DJ0zDQWIjVGBHwkZzsgVIL3MnGkO52UWGLrGGvF707E4RZJPV74dGdwolsjF8cG0QzUpl88=; Received: from [193.173.39.45] by web51410.mail.re2.yahoo.com via HTTP; Tue, 27 May 2008 02:32:34 PDT X-Mailer: YahooMailRC/975.41 YahooMailWebService/0.7.199 Date: Tue, 27 May 2008 02:32:34 -0700 (PDT) From: Alexander Popov To: freebsd-drivers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <730729.32628.qm@web51410.mail.re2.yahoo.com> Subject: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2008 09:59:15 -0000 Hello all, I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done. Previously, i.e. in 4.4 BSD, the following statements were true: 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode; 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver). This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever. Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE. It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: int driver_read(...) { mtx_lock(&sc->mtx); if (no data is available) => tsleep(..) mtx_unlock(&sc->mtx); } First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel... The bottom line is: 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this); 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed; 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed. Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver? Of course, I might be missing something here, please help. Thanks in advance, Alexander. From owner-freebsd-drivers@FreeBSD.ORG Tue May 27 23:21:25 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C0F5106564A for ; Tue, 27 May 2008 23:21:25 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from tom.kiev.farlep.net (tom.kiev.farlep.net [213.130.24.4]) by mx1.freebsd.org (Postfix) with ESMTP id EEBD58FC1E for ; Tue, 27 May 2008 23:21:24 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from ilya.kiev.farlep.net ([62.221.47.37]:2101 helo=[10.0.0.3]) by tom.kiev.farlep.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1K189B-00010i-4B; Wed, 28 May 2008 02:00:01 +0300 Message-ID: <483C926D.50104@po4ta.com> Date: Wed, 28 May 2008 01:59:57 +0300 From: Ilya Bobir User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Alexander Popov References: <730729.32628.qm@web51410.mail.re2.yahoo.com> In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Farlep-Data: Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2008 23:21:25 -0000 Alexander Popov wrote: > Hello all, > > [...] > > It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: > int driver_read(...) > { > mtx_lock(&sc->mtx); > > if (no data is available) => tsleep(..) > > mtx_unlock(&sc->mtx); > } > > First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). > [...] > Do you really need to hold the mutex while sleeping? Maybe you need to mtx_sleep instead? From owner-freebsd-drivers@FreeBSD.ORG Tue May 27 23:24:22 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F5F8106566C for ; Tue, 27 May 2008 23:24:22 +0000 (UTC) (envelope-from Benjamin.Close@clearchain.com) Received: from ipmail05.adl2.internode.on.net (ipmail05.adl2.internode.on.net [203.16.214.145]) by mx1.freebsd.org (Postfix) with ESMTP id CF6B28FC16 for ; Tue, 27 May 2008 23:24:21 +0000 (UTC) (envelope-from Benjamin.Close@clearchain.com) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlAFAJwuPEh5LZFh/2dsb2JhbACBVa03 X-IronPort-AV: E=Sophos;i="4.27,550,1204464600"; d="scan'208";a="123768894" Received: from ppp121-45-145-97.lns11.adl6.internode.on.net (HELO mail.clearchain.com) ([121.45.145.97]) by ipmail05.adl2.internode.on.net with ESMTP; 28 May 2008 08:39:04 +0930 Received: from [192.168.155.54] (taurus.internal.clearchain.com [192.168.155.54]) (authenticated bits=0) by mail.clearchain.com (8.14.2/8.14.2) with ESMTP id m4RN92NS025203 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 28 May 2008 08:39:03 +0930 (CST) (envelope-from Benjamin.Close@clearchain.com) Message-ID: <483C9493.3070607@clearchain.com> Date: Wed, 28 May 2008 08:39:07 +0930 From: Benjamin Close User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Alexander Popov References: <730729.32628.qm@web51410.mail.re2.yahoo.com> In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on pegasus.clearchain.com X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (mail.clearchain.com [192.168.154.1]); Wed, 28 May 2008 08:39:03 +0930 (CST) Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2008 23:24:22 -0000 Alexander Popov wrote: > Hello all, > > I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done. > > Previously, i.e. in 4.4 BSD, the following statements were true: > > 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode; > 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver). > > This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever. > > Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE. > > It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: > int driver_read(...) > { > mtx_lock(&sc->mtx); > > if (no data is available) => tsleep(..) > > mtx_unlock(&sc->mtx); > } > > First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). > > Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel... > > The bottom line is: > 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this); > 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed; > 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed. > > Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver? > > Of course, I might be missing something here, please help. > > Thanks in advance, > > Hi Alexander, As you've noticed, sleeping using tlseep whilst holding a mutex can cause you grief due to preemption. This is why mtx_sleep was introduced. It drops the lock before the sleep and reaquires it before continuing. Hence you can then be preempted without issues. Under 7 and above, man 9 locking is a very useful page. In general drivers tend to use mutexes or in some cases rwlocks. Be aware though, you should not sleep in an interrupt handler. Cheers, Benjamin From owner-freebsd-drivers@FreeBSD.ORG Wed May 28 12:25:06 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2040106564A for ; Wed, 28 May 2008 12:25:06 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: from web51403.mail.re2.yahoo.com (web51403.mail.re2.yahoo.com [206.190.38.182]) by mx1.freebsd.org (Postfix) with SMTP id 625088FC16 for ; Wed, 28 May 2008 12:25:06 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: (qmail 42191 invoked by uid 60001); 28 May 2008 12:25:04 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Message-ID; b=prOn3BVOpHQv3vWPiqPjnX9/sdyN91P8hogKhz4k16hern+pWK2dOZcgGn1SxUfO/mSIHMMcVrhXhDe+EB8ZlUE4X6dGRndr9euSNlJwB0jaViQXyggN9Uq6HuwA1FvJO84LHGyV2wLnlN9z0fRzla7DlZYecHHoHZ2TI9zx2ms=; Received: from [193.173.39.45] by web51403.mail.re2.yahoo.com via HTTP; Wed, 28 May 2008 05:25:04 PDT X-Mailer: YahooMailRC/975.41 YahooMailWebService/0.7.199 Date: Wed, 28 May 2008 05:25:04 -0700 (PDT) From: Alexander Popov To: Benjamin Close MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <600706.33129.qm@web51403.mail.re2.yahoo.com> Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2008 12:25:07 -0000 Hi, Benjamin, also Ilya, Thanks for your quick responses. mtx_sleep() would definitely help. Does it mean that user process executing in kernel space is guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then I would get a sleeping thread with non-sleepable mutex anyway... or have I been in application development for too long? :-) Regards, Alexander. ----- Original Message ---- From: Benjamin Close To: Alexander Popov Cc: freebsd-drivers@freebsd.org Sent: Wednesday, May 28, 2008 1:09:07 AM Subject: Re: Synchronization in drivers (after SMP improvements) Alexander Popov wrote: > Hello all, > > I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done. > > Previously, i.e. in 4.4 BSD, the following statements were true: > > 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode; > 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver). > > This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever. > > Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE. > > It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: > int driver_read(...) > { > mtx_lock(&sc->mtx); > > if (no data is available) => tsleep(..) > > mtx_unlock(&sc->mtx); > } > > First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). > > Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel... > > The bottom line is: > 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this); > 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed; > 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed. > > Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver? > > Of course, I might be missing something here, please help. > > Thanks in advance, > > Hi Alexander, As you've noticed, sleeping using tlseep whilst holding a mutex can cause you grief due to preemption. This is why mtx_sleep was introduced. It drops the lock before the sleep and reaquires it before continuing. Hence you can then be preempted without issues. Under 7 and above, man 9 locking is a very useful page. In general drivers tend to use mutexes or in some cases rwlocks. Be aware though, you should not sleep in an interrupt handler. Cheers, Benjamin From owner-freebsd-drivers@FreeBSD.ORG Wed May 28 13:49:34 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67296106566C for ; Wed, 28 May 2008 13:49:34 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from tom.kiev.farlep.net (tom.kiev.farlep.net [213.130.24.4]) by mx1.freebsd.org (Postfix) with ESMTP id 0DDBE8FC1A for ; Wed, 28 May 2008 13:49:33 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from ilya.kiev.farlep.net ([62.221.47.37]:3971 helo=[10.0.0.3]) by tom.kiev.farlep.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1K1M1f-000EDj-HR; Wed, 28 May 2008 16:49:12 +0300 Message-ID: <483D62D0.6070800@po4ta.com> Date: Wed, 28 May 2008 16:49:04 +0300 From: Ilya Bobir User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Alexander Popov References: <600706.33129.qm@web51403.mail.re2.yahoo.com> In-Reply-To: <600706.33129.qm@web51403.mail.re2.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Farlep-Data: Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2008 13:49:34 -0000 Alexander Popov wrote: > Hi, Benjamin, also Ilya, > > > Thanks for your quick responses. mtx_sleep() would > definitely help. Does it mean that user process executing in kernel space is > guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then > I would get a sleeping thread with > non-sleepable mutex anyway... or have I been in application > development for too long? :-) > Regards, > > Alexander. > > According to locking(9) (" Context mode table." at the very bottom) the only type of mutex you can use in your interrupt handler is a spin mutex. And you are trying to synchronize with an interrupt handler. So, the sc->mtx ought to be a spin mutex. While a spin mutex is held all interrupts on the current CPU are blocked or deferred (mutex(9), DESCRIPTION fifth paragraph) and the thread holding the mutex will not be preempted. And you should use msleep_spin(). From owner-freebsd-drivers@FreeBSD.ORG Wed May 28 14:19:23 2008 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76DC6106564A for ; Wed, 28 May 2008 14:19:23 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 05D448FC12 for ; Wed, 28 May 2008 14:19:22 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m4SEGGr8027601; Wed, 28 May 2008 08:16:16 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 28 May 2008 08:17:46 -0600 (MDT) Message-Id: <20080528.081746.1683967403.imp@bsdimp.com> To: ilya@po4ta.com From: "M. Warner Losh" In-Reply-To: <483D62D0.6070800@po4ta.com> References: <600706.33129.qm@web51403.mail.re2.yahoo.com> <483D62D0.6070800@po4ta.com> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2008 14:19:23 -0000 In message: <483D62D0.6070800@po4ta.com> Ilya Bobir writes: : Alexander Popov wrote: : > Hi, Benjamin, also Ilya, : > : > : > Thanks for your quick responses. mtx_sleep() would : > definitely help. Does it mean that user process executing in kernel space is : > guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then : > I would get a sleeping thread with : > non-sleepable mutex anyway... or have I been in application : > development for too long? :-) : > Regards, : > : > Alexander. : > : > : : According to locking(9) (" Context mode table." at the very bottom) the : only type of mutex you can use in your interrupt handler is a spin : mutex. And you are trying to synchronize with an interrupt handler. : So, the sc->mtx ought to be a spin mutex. While a spin mutex is held : all interrupts on the current CPU are blocked or deferred (mutex(9), : DESCRIPTION fifth paragraph) and the thread holding the mutex will not : be preempted. That's only true of 'fast' interrupt handlers. Normal Ithread interrupt hanlders, mutexes are fine since you aren't really in an interrupt context. If this is indeed a usb device driver, then the context for a callback routine for an interrupt endpoint really is a taskqueue... Warner From owner-freebsd-drivers@FreeBSD.ORG Wed May 28 14:53:21 2008 Return-Path: Delivered-To: freebsd-drivers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A31601065670 for ; Wed, 28 May 2008 14:53:21 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 4CFE88FC0C for ; Wed, 28 May 2008 14:53:21 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m4SEnLBu028132; Wed, 28 May 2008 08:49:21 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 28 May 2008 08:50:51 -0600 (MDT) Message-Id: <20080528.085051.-1253012495.imp@bsdimp.com> To: aopopov@yahoo.com From: "M. Warner Losh" In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com> References: <730729.32628.qm@web51410.mail.re2.yahoo.com> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-drivers@FreeBSD.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2008 14:53:21 -0000 In message: <730729.32628.qm@web51410.mail.re2.yahoo.com> Alexander Popov writes: : I'm currently writing a FreeBSD USB driver and have a few questions : about synchronization between bottom and top half of the : driver. By USB driver, I'm assuming you mean "a driver for a usb device that's plugged into the system" as opposed to "a driver for a usb host adapter." : This is the strategy that is followed by most of the (USB) drivers : currently. However, after the SMP improvements have been : implemented, the splxxx family of functions has been successfully : stubbed out, so now these functions do absolutely nothing, which : (correct me if am wrong) leaves most of the existing USB drivers : without any synchronization with their bottom halves whatsoever. Right now, the USB stack is Giant locked, so this stuff works. : Next, in the scope of SMP project, interrupt handlers have been : replaced with (lightweight?) threads, which makes it possible to use : synchronization primitives in the interrupt handlers to synchronize : them with driver's top half. However, this does not apply to USB : stack, which, as I understand, still uses Giant lock. So, from this : perspective, I can not make interrupts in my USB driver MPSAFE. Well, you could, but it is really hard to do since you have to understand the current USB code and be able to acquire Giant at the right times, etc. So theoretically possible, but I don't know of anybody that's succeed in doing so. : It has been suggested everywhere that mutexes shall be used to : protect top and bottom halves of the driver. So I've tried. The : following is the pseudo-code that I've used: : int driver_read(...) : { : mtx_lock(&sc->mtx); : : if (no data is available) => tsleep(..) : : mtx_unlock(&sc->mtx); : } As others have pointed out, the fundamental problem here is that you are using tsleep while holding a mutex. This is not allowed because it leads to bad things happening. Not always, but sometimes as you've discovered. : First it worked very well, but under relatively heavy load I started : getting kernel panic, all the time related to one error: a thread : holding a non-sleepable lock. Which means that user process has been : suspended somewhere during execution in the read() function : (probably awaiting data, but not necessary) and its thread has been : holding mutex that I've used for synchronization, but that it : apparently not allowed (look in man page on mutex). The correct solution here is to just use msleep. : Then I found that in SMP project an additional flag was added to : mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping : threads in kernel that hold mutexes. But, apparently, this flag has : been recently removed from kernel... This isn't what you want... : Question is: what is now a synchronization model for modern kernels, : i.e. FreeBSD 7.0? How shall I properly implement synchronization in : my driver? I think you can use msleep. Alternatively, you could use condvars for synchronization. And I'd make the if () statement a while () statement above to guard against races. Warner From owner-freebsd-drivers@FreeBSD.ORG Fri May 30 15:34:09 2008 Return-Path: Delivered-To: freebsd-drivers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49D011065676 for ; Fri, 30 May 2008 15:34:09 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: from web51404.mail.re2.yahoo.com (web51404.mail.re2.yahoo.com [206.190.38.183]) by mx1.freebsd.org (Postfix) with SMTP id 09DA28FC18 for ; Fri, 30 May 2008 15:34:08 +0000 (UTC) (envelope-from aopopov@yahoo.com) Received: (qmail 38471 invoked by uid 60001); 30 May 2008 15:34:08 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Message-ID; b=zFW7rDBEsmqgg3k4WCAAFiqEbCaWlH86sHoXfwZN0UqBUP+NWur+HfaGUNUDgShMFlQ5S0Zaqa/KkLkhLZhKXNVO+C7SnRuZZttM/QWW7NdWIKyuxKoFAaP+SBlQNz3nf9tHxDA+DEBSB64DcAvhqJGXto46u9GxFNPuRRk+Uy0=; Received: from [193.173.39.45] by web51404.mail.re2.yahoo.com via HTTP; Fri, 30 May 2008 08:34:07 PDT X-Mailer: YahooMailRC/975.41 YahooMailWebService/0.7.199 Date: Fri, 30 May 2008 08:34:07 -0700 (PDT) From: Alexander Popov To: "M. Warner Losh" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <987362.36986.qm@web51404.mail.re2.yahoo.com> Cc: freebsd-drivers@FreeBSD.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 May 2008 15:34:09 -0000 Hi, Warner, Thanks, I appreciate your help. Yes, indeed I am writing a USB device driver. I have done some tests, and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this: void driver_open() { mtx_lock(&sc->mtx) // open USB pipes to device, etc. mtx_unlock(&sc->mtx) } void driver_write(...) { mtx_lock(&sc->mtx) // write to device (I do not sleep here!) mtx_unlock(&sc->mtx) } then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock". When I change default mutex to spin type, it works well, and I was not able to crash it. As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked. It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes... However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet... Regards, Alexander. ----- Original Message ---- From: M. Warner Losh To: aopopov@yahoo.com Cc: freebsd-drivers@FreeBSD.org Sent: Wednesday, May 28, 2008 4:50:51 PM Subject: Re: Synchronization in drivers (after SMP improvements) In message: <730729.32628.qm@web51410.mail.re2.yahoo.com> Alexander Popov writes: : I'm currently writing a FreeBSD USB driver and have a few questions : about synchronization between bottom and top half of the : driver. By USB driver, I'm assuming you mean "a driver for a usb device that's plugged into the system" as opposed to "a driver for a usb host adapter." : This is the strategy that is followed by most of the (USB) drivers : currently. However, after the SMP improvements have been : implemented, the splxxx family of functions has been successfully : stubbed out, so now these functions do absolutely nothing, which : (correct me if am wrong) leaves most of the existing USB drivers : without any synchronization with their bottom halves whatsoever. Right now, the USB stack is Giant locked, so this stuff works. : Next, in the scope of SMP project, interrupt handlers have been : replaced with (lightweight?) threads, which makes it possible to use : synchronization primitives in the interrupt handlers to synchronize : them with driver's top half. However, this does not apply to USB : stack, which, as I understand, still uses Giant lock. So, from this : perspective, I can not make interrupts in my USB driver MPSAFE. Well, you could, but it is really hard to do since you have to understand the current USB code and be able to acquire Giant at the right times, etc. So theoretically possible, but I don't know of anybody that's succeed in doing so. : It has been suggested everywhere that mutexes shall be used to : protect top and bottom halves of the driver. So I've tried. The : following is the pseudo-code that I've used: : int driver_read(...) : { : mtx_lock(&sc->mtx); : : if (no data is available) => tsleep(..) : : mtx_unlock(&sc->mtx); : } As others have pointed out, the fundamental problem here is that you are using tsleep while holding a mutex. This is not allowed because it leads to bad things happening. Not always, but sometimes as you've discovered. : First it worked very well, but under relatively heavy load I started : getting kernel panic, all the time related to one error: a thread : holding a non-sleepable lock. Which means that user process has been : suspended somewhere during execution in the read() function : (probably awaiting data, but not necessary) and its thread has been : holding mutex that I've used for synchronization, but that it : apparently not allowed (look in man page on mutex). The correct solution here is to just use msleep. : Then I found that in SMP project an additional flag was added to : mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping : threads in kernel that hold mutexes. But, apparently, this flag has : been recently removed from kernel... This isn't what you want... : Question is: what is now a synchronization model for modern kernels, : i.e. FreeBSD 7.0? How shall I properly implement synchronization in : my driver? I think you can use msleep. Alternatively, you could use condvars for synchronization. And I'd make the if () statement a while () statement above to guard against races. Warner From owner-freebsd-drivers@FreeBSD.ORG Sat May 31 17:57:38 2008 Return-Path: Delivered-To: freebsd-drivers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FF00106564A for ; Sat, 31 May 2008 17:57:38 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from tom.kiev.farlep.net (tom.kiev.farlep.net [213.130.24.4]) by mx1.freebsd.org (Postfix) with ESMTP id E68C98FC14 for ; Sat, 31 May 2008 17:57:37 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from ilya.kiev.farlep.net ([62.221.47.37]:1476 helo=[10.0.0.3]) by tom.kiev.farlep.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1K2VKg-0003bk-0S; Sat, 31 May 2008 20:57:34 +0300 Message-ID: <48419182.1030200@po4ta.com> Date: Sat, 31 May 2008 20:57:22 +0300 From: Ilya Bobir User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Alexander Popov References: <987362.36986.qm@web51404.mail.re2.yahoo.com> In-Reply-To: <987362.36986.qm@web51404.mail.re2.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Farlep-Data: Cc: freebsd-drivers@FreeBSD.org Subject: Re: Synchronization in drivers (after SMP improvements) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 May 2008 17:57:38 -0000 Alexander Popov wrote: > Hi, Warner, > > Thanks, I appreciate your help. > Yes, indeed I am writing a USB device driver. I have done some tests, > and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this: > > void driver_open() > { > mtx_lock(&sc->mtx) > // open USB pipes to device, etc. > mtx_unlock(&sc->mtx) > } > > void driver_write(...) > { > mtx_lock(&sc->mtx) > // write to device (I do not sleep here!) > mtx_unlock(&sc->mtx) > } > > then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock". The only spot in the source were I was able to find a message like this is in src/sys/kern/subr_turnstile.c:propagate_priority(struct thread *td) (http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/subr_turnstile.c?rev=1.173;content-type=text%2Fplain search for "non-sleepable"). IIUC, this message means that a thread that is sleeping on a lock also holds a spin lock. Another thread (current thread) is trying to acquire this spin lock but finds out that the current owner of the spin lock is sleeping. As sleeping while holding a spin lock is prohibited as it may cause a deadlock the function panics. Note that the misbehaving thread's id is present in the message. Chapter 11 of the "FreeBSD Developers' Handbook" explains how you can get a thread stack trace from a kernel dump or a running kernel. But you probably already know it. > When I change default mutex to spin type, it works well, and I was not able to crash it. As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked. > It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes... > I think you would get a lock order reversal message or some other kind of message if you are locking Giant after some other lock. "Sleeping thread (tid %d, pid %d) owns a non-sleepable lock\n" is not about Giant. > However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet... > [...] It probably means that the mutex is already locked. You can drop into a debugger and try do debug it as described in "11.9 Debugging Deadlocks" in the "FreeBSD Developers' Handbook". P.S. Note, that sleeping and preemption are different. A thread is sleeping when it is waiting on a sleepable lock. While waiting on a non-sleepable lock or in some other cases a thread may be preempted but it is not marked as sleeping.