From owner-freebsd-arch@FreeBSD.ORG Mon Aug 30 18:02:24 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9E8BE16A4D4; Mon, 30 Aug 2004 18:02:24 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id E580043D4C; Mon, 30 Aug 2004 18:02:23 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.11/8.12.11) with ESMTP id i7UI1IP8053881; Mon, 30 Aug 2004 12:01:18 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Mon, 30 Aug 2004 12:01:24 -0600 (MDT) Message-Id: <20040830.120124.28086427.imp@bsdimp.com> To: scottl@freebsd.org From: "M. Warner Losh" In-Reply-To: <4133682D.3000403@freebsd.org> References: <20040830.102606.130865377.imp@bsdimp.com> <20040830.111428.56562495.imp@bsdimp.com> <4133682D.3000403@freebsd.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: sah@softcardsystems.com cc: freebsd-arch@freebsd.org Subject: Re: splxxx level? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Aug 2004 18:02:24 -0000 In message: <4133682D.3000403@freebsd.org> Scott Long writes: : M. Warner Losh wrote: : : > In message: <20040830.102606.130865377.imp@bsdimp.com> : > "M. Warner Losh" writes: : > : In message: <41334C3B.4070101@freebsd.org> : > : Scott Long writes: : > : : Sam wrote: : > : : : > : : > Hello - : > : : > : > : : > I'm almost to testing on my AoE driver for 4.x and have : > : : > a question about interrupt priority levels. : > : : > : > : : > There are currently three entry points into the driver: : > : : > : > : : > a) strategy routine : > : : > b) network frame reception routine : > : : > c) timer rexmit routine : > : : > : > : : > Any of the three can diddle with the device structure : > : : > and thusly I need to ensure they're not running simultaneously. : > : : > For example, the network reception can cause a buf to be completed : > : : > and the rexmit timer can cause a buf to be failed. : > : : > : > : : > So, what kind of contexts are the callout, strategy, and : > : : > network soft interrupt called in? Which splxxx will give : > : : > one of them exclusive access to whatever they need? : > : : > : > : : > Just as a reality check -- I am thinking about this correct, right? : > : : > : > : : > Cheers, : > : : > : > : : > Sam : > : : > : > : : : > : : With 4.x, only one CPU can be in the kernel at a time. You won't have : > : : to worry about multiple processes trying to get into strategy at the : > : : same time and whatnot. However, you can be preempted by your interrupt : > : : handler or by a timeout or by a software interrupt like the netisr. I : > : : don't remember if your driver is for a specific piece of hardware or if : > : : it's a generic layer that sits in between the network interface and the : > : : block layer. If it's for dedicated hardware then you'll need to define : > : : a interrupt type in bus_setup_intr() and use that type for the spl : > : : (i.e. INTR_TYPE_NET translates to splnet(), INTR_TYPE_BIO translates to : > : : splbio(), etc). : > : : : > : : The safe way to go is to protect all of your critical code sections with : > : : the appropriate spl level regardless. spls are very cheap and can be : > : : set/reset from an interrupt context so there is little penalty in using : > : : them liberally at first and then narrowing them down later. Just make : > : : sure that you don't leak an spl references, and don't hold an spl for so : > : : long that it creates priority inversions. Since the only interrupts and : > : : timeouts that you'll likely be dealing with are network related, : > : : splnet() is probably the right one to use. : > : : > : splimp() is what you want to use, not splnet(). Yes, this is : > : confusing, but it appears to be what all the other network drivers : > : use. None of them are using splnet() that I could find. splimp() is : > : also used by the mbuf routines to protect mbuf operations. : > : : > : splnet() is a list of the software interrupts that are network : > : drivers. : > : : > : splimp() is splnet() plus the hardware interrupts, so is more : > : appropriate to block things called from the driver. Especially one : > : that's described as having timeouts. If it is a network driver, you : > : might consider using the timeout functionality in the net stack as : > : opposed to the callout functions. This makes it possible to have : > : almost the entire driver w/o doing any spls (most of the network : > : drivers in 4 don't do spl at all, except for entry points that are : > : outside the scope of the network/interrupt entry points, eg : > : device_suspend). : > : > Ah, just saw 'AoE' in the reply. : > : > This likely means that you are writing a network stack that's glued : > into the system with the strategy routine. Inside the strategy : > routine, splbio() is likely what you need to hold while dealing with : > the struct buf that's passed into that routine. I'm not sure what : > spl level you need to be at to call into the network code. I'm : > thinking it is splimp(), but I'm not 100% sure about that. Stevens : > will likely be a good resource for 4.x. : > : > You may have to define a software interrupt to pass the packets to the : > network code to make the spls work out correctly. : > : > Warner : : No, splbio() is not explicitely needed in this case. You'll be : dealing with a bioq that might need protection, but you have total : control over that. There is nothing else in the block layer that : will interrupt you, not like the netisr in the network stack or the : camisr in the CAM layer. Why does da use splbio in its strategy routine then: static void dastrategy(struct buf *bp) { ... /* * Mask interrupts so that the pack cannot be invalidated until * after we are in the queue. Otherwise, we might not properly * clean up one of the buffers. */ s = splbio(); ...(error cases) /* * Place it in the queue of disk activities for this disk */ bufqdisksort(&softc->buf_queue, bp); splx(s); /* * Schedule ourselves for performing the work. */ xpt_schedule(periph, /* XXX priority */1); return; ... (error cases) } Is that due to the ISR routine from the cam layer? Or is the softc->buf_queue the 'bioq' that you were talking about? Warner