From owner-freebsd-arch@FreeBSD.ORG Sat May 28 21:32:28 2005 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 301E116A41F for ; Sat, 28 May 2005 21:32:28 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7C3EF43D1F for ; Sat, 28 May 2005 21:32:22 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j4SLXxcI057066; Sat, 28 May 2005 15:33:59 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4298E316.9020303@samsco.org> Date: Sat, 28 May 2005 15:31:02 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Daniel Eriksson References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: freebsd-arch@freebsd.org Subject: Re: MPSAFE CAM? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 May 2005 21:32:28 -0000 Daniel Eriksson wrote: > I'm sorry to hijack your thread Robert, but I just wanted to hear if there > is any progress in making CAM (and the individual card drivers) MPSAFE? > > FreeBSD is mainly used for servers, where SCSI is pretty common. To me it > seems like locking down CAM would be beneficial to a large number of FreeBSD > users. > > I don't have the knowledge or time to do any real work on this myself, but I > should be able to help test patches. > > /Daniel Eriksson > > CAM was originally designed to make _very_ efficient use of SPL synchronization. Simply dropping locks in place of spl's has been tried, and the result is horrible performance. It's going to take a significant rewrite of much of the XPT code to make it efficiently locked. I started on this about a year ago but got sidetracked. It's still very high on my priority list, and I hope that I can get to it this fall. In the mean time, I put in some minimal locking so that xpt_done() can be called from drivers without Giant being held. This means that drivers can lock their interrupt service thread and declare themselves MPSAFE. The top half of the driver will still run with Giant held when it's called from CAM, but the bottom half can run Giant-free. I modified the esp(4) driver a few months ago to demonstrate this, and the results seem to be ok. esp definitely benefits from it since its scsi state machine is in the driver, so 4-5 interrupts get generated for each I/O. For more modern SCSI chips like ahc/ahd and mpt, the benefits might not be as much. If anyone wants to help tackle this, I'd be glad to help them get started. BSD/OS 5.(mumble) did a minimal locking job on CAM that probably works OK for systems with a single SCSI port, but likely gets very inefficient once you go beyond that. They also rewrote the SCSI probe state machine, which in our CAM is a source for quite a bit of lock recursion. The work I did last year focused on addressing this. Scott