From owner-freebsd-scsi@FreeBSD.ORG Sun Nov 4 09:10:43 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E270DEC for ; Sun, 4 Nov 2012 09:10:43 +0000 (UTC) (envelope-from jacks.1785@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 343108FC18 for ; Sun, 4 Nov 2012 09:10:42 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id k10so8787009iea.13 for ; Sun, 04 Nov 2012 01:10:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=u4ebzBAIsU3ZCSt0QHct2r3P9s1lKVnWZq+Iy8Cpy/I=; b=vqj7pw5llaHTp1mYmeyAZiD+NbEokbmC5e+F3AvC8dgJPlaYogIjrCBAy0j9bQDN+a OOHSkZ48AxSY0ZEyAMhTZVnrYNJ5yNjtb7XvS/jHRLwFXU3F/9Kum55ZDrXhbI7BlB5R +uWU2ydLR1dXPXqRYXFYQrGK8L5nwsuyMKR0qgj+KFUqxvVHlb8/hPjcQQf1LgEtEimy 4E7NTnkZJHig8IlhJzZuxHzZzPYF5pHoEfpsC3le+vYrwRkVM0t/8l31iVV/ni9hRmO6 V0N+ngUO5aa4/2A3O13qeAG5AWrw96hKtbBmclQk6T6Kz/qoME34imwVbw51068hEqPM bW1g== MIME-Version: 1.0 Received: by 10.50.46.134 with SMTP id v6mr6430453igm.55.1352020242449; Sun, 04 Nov 2012 01:10:42 -0800 (PST) Received: by 10.64.11.166 with HTTP; Sun, 4 Nov 2012 01:10:42 -0800 (PST) Date: Sun, 4 Nov 2012 14:40:42 +0530 Message-ID: Subject: cam probe sequence From: Jack To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 09:10:43 -0000 Hello all, I'm trying to understand the probe sequence of a scsi device, say a scsi hard drive( with SPI/SAS/FC interface). I'm particularly interested in probe sequence of a device which is attached to the system _after_ FreeBSD has booted completely. (ie post-boot probing.) While going through the source of cam subsystem, a few questions arose in my understanding. ( The relevant files I found are cam/cam_xpt.c, cam/cam_periph.c and cam/scsi/scsi_xpt.c or cam/ata/ata_xpt.c in case of (S)ATA drives. ) As far as I could got till now, it seems that struct cam_et and struct cam_ed are _always_ created even _before_ the probing of device commences. But how does cam sub-system get to know that a device got attached to the system, so that it creates struct cam_et and struct cam_ed in advance, and _then_ probes the device. Is it something like a event notfication at hardware level that the HBA driver passes to cam sub-system which notifies cam sub-system that a device has recently been attached. If it is true, then does all SPI/SAS/FC/(S)ATA HBAs support such event notification -- something like a standard thing in SPI/SAS/FC/(S)ATA world. If it is not so, then does userland have to intiate a scan after device is attached, so that during scanning the cam-subsystem get to know that Oh! there is a device we are meeting first time. In other words, _until_ userland instructs cam subsystem to scan all the buses by XPT_SCAN_BUS ccb through CAMIOCOMMAND ioctl of /dev/xpt0, the struct cam_et and struct cam_ed will not get created. Is it so? But in this case also, does cam-subsystem creates struct cam_et and struct cam_ed before probing the device?. In case of SPI/SAS/FC hard drives the probe code is in cam/scsi/scsi_xpt.c While reading this source file it also seems to me that a struct cam_periph is also created as soon as struct cam_et and struct cam_ed are created. This also happens before probing of device commences. Is it the pass device that is created even before probing begins. Then after probing and determining the type of device - e.g. disk, cd-rom, etc, then the appropriate peripheral device(da, cd, etc. ) is created for the probed device. Is it so? It would be really helpful if someone could clear these things, specially from the instant the device -say a hard disk; is attached, to the instant the pass device and da device are created and probing of this hard disk commences or finishes. Regards -- Jack From owner-freebsd-scsi@FreeBSD.ORG Sun Nov 4 13:47:59 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 272053EA; Sun, 4 Nov 2012 13:47:59 +0000 (UTC) (envelope-from omerfsen@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id AFF2C8FC0C; Sun, 4 Nov 2012 13:47:58 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so4187126qcs.13 for ; Sun, 04 Nov 2012 05:47:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Xczt6TKpXKyHzQbKv4WJdzWz9pdoXziZKQIhlxBJdtk=; b=eZGSC2+pbBONdFDpotF48JgtPbMqCJu5Q1vN2EbMLusrRbhet95DXBuzab6OQgj1hD dgHurp3OoEhqh6lJgyJl6R/HyZrd8PDZr6lOHZ4EJOzCdXiLFR+4X8QCM7JS0LuNUwSQ M5UFlCKRjbMqU69dpxrj6m1FGInCeMWE0GTMq6xfHbGOUGPD7B0jljtlFCbTjcvaO5C1 71rse7Br6rJWvCHY307h7ukyPIuhsPRRo4i6CnjtNJsZyEzGbcZcQ5iUroYhtI/qNEXC K6VOtDb4uq+F3MsxH0LArjXSudq8ZRp/RJDnruFnmCvn6CYJAZ2DjFMrsHkjdE3cCHXo L4EA== MIME-Version: 1.0 Received: by 10.224.186.17 with SMTP id cq17mr10577166qab.83.1352036872495; Sun, 04 Nov 2012 05:47:52 -0800 (PST) Received: by 10.229.157.143 with HTTP; Sun, 4 Nov 2012 05:47:52 -0800 (PST) In-Reply-To: <20121102141056.GA64665@neutralgood.org> References: <20121102141056.GA64665@neutralgood.org> Date: Sun, 4 Nov 2012 15:47:52 +0200 Message-ID: Subject: Re: Dell H710 and H310 Raid Controller From: Omer Faruk SEN To: kpneal@pobox.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-scsi@freebsd.org, FreeBSD X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 13:47:59 -0000 Hi, Just tried 9.1-RC3 with R720 which has H710p( the only difference with H710 is 1 gb cache instead of 512mb ). It has recognized both H710p raid as mfid0 and also network cards are recognized as bgeX (*BCM5720)* but network cards times out (watchdog timeout) I think it is about http://forums.freebsd.org/showthread.php?t=31769&page=2 It seems right now only way to go with Rx20 Server models is to use Intel cards (dell provides i350 chipset network interfaces as alternative) PS: I really need your comments on R420 with H710 or R320 with H310 raid controllers Regads. On Fri, Nov 2, 2012 at 4:10 PM, wrote: > On Thu, Nov 01, 2012 at 07:46:45PM +0200, Omer Faruk SEN wrote: > > Hi, > > > > Can anyone in this list verify that both RAID controllers are supported > on > > FreeBSD 8.3 or 9.1 > > Negative on 8.3. I'm running a post-8.3-release 8.3-STABLE compiled after > the new mfi driver went in. > > > H710 has LSISAS2208 dual-core PowerPC ROC > > H310 has LSISAS2008. > > > > I am planning to use these controllers on R420 and R320 Dell Servers. I > > would also like to get comments on these two platfoms and if there are > any > > issues on FreeBSD 9.1 (I know it is RC2 right now) > > I've got an H710 in an R620. It works fine for me. > > Of course, my system is lightly loaded for the most part. The past couple > of days I've had one client pounding on it via netatalk. It held up well > considering ZFS performing really badly on a 96% full pool when writing > a lot of data. But that's an outlier. Usually my system is lightly loaded. > > Make sure you turn off all power savings controls in the BIOS. > -- > Kevin P. Neal http://www.pobox.com/~kpn/ > On the community of supercomputer fans: > "But what we lack in size we make up for in eccentricity." > from Steve Gombosi, comp.sys.super, 31 Jul 2000 11:22:43 -0600 > From owner-freebsd-scsi@FreeBSD.ORG Sun Nov 4 20:03:50 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B75FC54; Sun, 4 Nov 2012 20:03:50 +0000 (UTC) (envelope-from seanbru@yahoo-inc.com) Received: from mrout3.yahoo.com (mrout3.yahoo.com [216.145.54.173]) by mx1.freebsd.org (Postfix) with ESMTP id 23FB48FC12; Sun, 4 Nov 2012 20:03:49 +0000 (UTC) Received: from [127.0.0.1] (proxy7.corp.yahoo.com [216.145.48.98]) by mrout3.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id qA4JrTDq022460; Sun, 4 Nov 2012 11:53:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1352058810; bh=5wdbct559tQqik23WpnNd53T3a1kgzyQEESM9pywt/w=; h=Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type: Date:Message-ID:Mime-Version; b=kAw51x4084CFdxwRGCJVRv/ZumWsVsSXhNyoDZV36vl0tMsfmnpLNJsYLIOIGxVtu hgxHfbBJ+amtlQcUNZmiFO4PPtSMKJHDV1nlO6ilrxSvJwBai/ItaGJ+RALqGVtynZ nAe2d0ggiyWw1syg2RgKJ4iQ2MqUsMpyPGwOlzU4= Subject: Re: Dell H710 and H310 Raid Controller From: Sean Bruno To: Omer Faruk SEN In-Reply-To: References: <20121102141056.GA64665@neutralgood.org> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-A34weRjvt+tN2KMKR+q9" Date: Sun, 04 Nov 2012 11:53:29 -0800 Message-ID: <1352058809.4628.1.camel@powernoodle.corp.yahoo.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 058810000 Cc: "freebsd-scsi@freebsd.org" , FreeBSD X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 20:03:50 -0000 --=-A34weRjvt+tN2KMKR+q9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, 2012-11-04 at 05:47 -0800, Omer Faruk SEN wrote: > It seems right now only way to go with Rx20 Server models is to use > Intel > cards (dell provides i350 chipset network interfaces as alternative)=20 The Broadcom 5720 support is in current right now. It will not be in 9.1, but will be available in stable/9 soon-ish. Sean --=-A34weRjvt+tN2KMKR+q9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQEcBAABAgAGBQJQlse5AAoJEBkJRdwI6BaHF44H/RG2E75/TaWwBUXT7/VTMaq6 hw3dwBIKnTq8laWcCZafS1EC8TFIPE3N8gzATRptlbPMbfwCDLurbD0gj7/XRDdS nRXndi55AHpfUzgCqUzt7xtgGeZiRbCoRqZsrvQjETloYhT2vG8OyMkpwkKXlHbU vzdd/7RSXPqZ7fGkAqKoDEHKzC+MWZlW+ak6hCRRzRNquqcZKmxgF+A8kU2NW/3i Lt9xlw5v/wH/R4kIIkA8t/hQx78HmJulZoJNdNZbtYq2Wtt5vcgicWwuYr7RaXq+ C3u7/NjqURN7g778ZKyfwyOU4erF9SM5UGKaW81pchHPUgCWZw6+DObLCsRuReA= =YnPA -----END PGP SIGNATURE----- --=-A34weRjvt+tN2KMKR+q9-- From owner-freebsd-scsi@FreeBSD.ORG Mon Nov 5 11:06:38 2012 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C7E65CFA for ; Mon, 5 Nov 2012 11:06:38 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id A1E328FC19 for ; Mon, 5 Nov 2012 11:06:38 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qA5B6cFT001284 for ; Mon, 5 Nov 2012 11:06:38 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qA5B6clY001282 for freebsd-scsi@FreeBSD.org; Mon, 5 Nov 2012 11:06:38 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 5 Nov 2012 11:06:38 GMT Message-Id: <201211051106.qA5B6clY001282@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 11:06:38 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/172864 scsi [mfi] mfip does not notice new drives o kern/172575 scsi [mfi] ioctl CAMGETPASSTHRU fails with mfi driver o kern/171650 scsi [da] da(4) driver does not recognize end of cciss (Sma o kern/169976 scsi [cam] [patch] make scsi_da use sysctl values where app o kern/169835 scsi [patch] remove some unused variables from scsi_da prob o kern/169801 scsi [cam] [patc] make changes to delete_method in scsi_da o kern/169403 scsi [cam] [patch] CAM layer, I/O starvation, no fairness o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free o kern/163713 scsi [aic7xxx] [patch] Add Adaptec29329LPE to aic79xx_pci.c o kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o kern/161809 scsi [cam] [patch] set kern.cam.boot_delay via build option o kern/159412 scsi [ciss] 7.3 RELEASE: ciss0 ADAPTER HEARTBEAT FAILED err o kern/157770 scsi [iscsi] [panic] iscsi_initiator panic o kern/154432 scsi [xpt] run_interrupt_driven_hooks: still waiting after o kern/153514 scsi [cam] [panic] CAM related panic o kern/153361 scsi [ciss] Smart Array 5300 boot/detect drive problem o kern/152250 scsi [ciss] [patch] Kernel panic when hw.ciss.expose_hidden o kern/151564 scsi [ciss] ciss(4) should increase CISS_MAX_LOGICAL to 10 o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c s kern/149927 scsi [cam] hard drive not stopped before removing power dur o kern/148083 scsi [aac] Strange device reporting o kern/147704 scsi [mpt] sys/dev/mpt: new chip revision, partially unsupp o kern/146287 scsi [ciss] ciss(4) cannot see more than one SmartArray con o kern/145768 scsi [mpt] can't perform I/O on SAS based SAN disk in freeb o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/144301 scsi [ciss] [hang] HP proliant server locks when using ciss o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/132250 scsi [ciss] ciss driver does not support more then 15 drive o kern/132206 scsi [mpt] system panics on boot when mirroring and 2nd dri o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 o kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/128452 scsi [sa] [panic] Accessing SCSI tape drive randomly crashe o kern/128245 scsi [scsi] "inquiry data fails comparison at DV1 step" [re o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up o kern/127717 scsi [ata] [patch] [request] - support write cache toggling o kern/123674 scsi [ahc] ahc driver dumping o kern/123520 scsi [ahd] unable to boot from net while using ahd o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc o kern/120487 scsi [sg] scsi_sg incompatible with scanners o kern/120247 scsi [mpt] FreeBSD 6.3 and LSI Logic 1030 = only 3.300MB/s o kern/114597 scsi [sym] System hangs at SCSI bus reset with dual HBAs o kern/110847 scsi [ahd] Tyan U320 onboard problem with more than 3 disks o kern/99954 scsi [ahc] reading from DVD failes on 6.x [regression] o kern/92798 scsi [ahc] SCSI problem with timeouts o kern/90282 scsi [sym] SCSI bus resets cause loss of ch device o kern/76178 scsi [ahd] Problem with ahd and large SCSI Raid system o kern/74627 scsi [ahc] [hang] Adaptec 2940U2W Can't boot 5.3 s kern/61165 scsi [panic] kernel page fault after calling cam_send_ccb o kern/60641 scsi [sym] Sporadic SCSI bus resets with 53C810 under load o kern/60598 scsi wire down of scsi devices conflicts with config s kern/57398 scsi [mly] Current fails to install on mly(4) based RAID di o kern/52638 scsi [panic] SCSI U320 on SMP server won't run faster than o kern/44587 scsi dev/dpt/dpt.h is missing defines required for DPT_HAND o kern/39388 scsi ncr/sym drivers fail with 53c810 and more than 256MB m o kern/35234 scsi World access to /dev/pass? (for scanner) requires acce 57 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Nov 5 16:55:16 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 28714B04; Mon, 5 Nov 2012 16:55:16 +0000 (UTC) (envelope-from prvs=1656497edf=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 7414B8FC0C; Mon, 5 Nov 2012 16:55:14 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000960905.msg; Mon, 05 Nov 2012 16:55:12 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 05 Nov 2012 16:55:12 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1656497edf=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> From: "Steven Hartland" To: , References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Date: Mon, 5 Nov 2012 16:55:11 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 16:55:16 -0000 I've managed to get the machine to reproduce this fairly regularly now. Without a debug kernel it still results in a panic, just at a later stage or so I believe, the none debug panic messages is "command not in queue". In each none debug panic I've seen the cm_flags indicates the command being dequeued is on the busy queue and not on the expected free or ready queue which is being processed at the time. The triggering issue seems to be the adapter reset code run from mfi_timeout. I've had a good look but can't see how a cm could be in a queue yet have its cm_flags set to that of a different queue as all manipulation seems to be being done via the "mfi_ ## name" macros which all correctly maintain the queue / cm_flags relationship. At this point I believe it could be a thread being interrupted by a timeout part way the processing of a queue request hence queue and cm_flags being out of sync. Any pointers on how to debug this issue further / fix it would be most appreciated. Regards Steve ----- Original Message ----- From: "Steven Hartland" > Testing a new machine which is based on 8.3-RELEASE with the mfi > driver from 8-STABLE and just got a panic. > > > The below is translation of the hand copied from console:- > mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > mfisyspd5: hard error cmd=write 90827650-90827905 > mfi0: I/O error, status= 46 scsi_status= 240 > mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > mfisyspd5: hard error cmd=write 90827394-90827649 > mfi0: I/O error, status= 46 scsi_status= 240 > mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > mfisyspd5: hard error cmd=write 90827138-90827393 > mfi0: I/O error, status= 46 scsi_status= 240 > mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > mfisyspd5: hard error cmd=write 90826882-90827137 > mfi0: I/O error, status= 2 scsi_status= 2 > mfi0: sense error 112, sense_key 6, asc 41, ascq 0 > mfisyspd4: hard error cmd=write 90830466-90830721 > mfi0: I/O error, status= 2 scsi_status= 2 > mfi0: sense error 112, sense_key 6, asc 41, ascq 0 > mfisyspd5: hard error cmd=write 90830722-90830977 > mfi0: Adapter RESET condition detected > mfi0: First state FW reset initiated... > mfi0: ADP_RESET_TBOLT: HostDiag=a0 > mfi0: first state of reset complete, second state initiated... > mfi0: Second state FW reset initiated... > panic: _mtx_lock_sleep: recursed on non-recusive mutex MFI I/O lock @ /usr/src/sys/dev/mfi/mfi_tbolt:346 > > cpuid = 6 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x178 > _mtx_lock_sleep() at _mtx_lock_sleep+0x152 > _mtx_lock_flags() at _mtx_lock_flags+0x80 > mfi_tbolt_init_MFI_queue() at mfi_tbolt_init_MFI_queue+0x72 > mfi_timeout() at mfi_timeout+0x27 > softclock() at softclock+0x2aa > intr_event_execute_handlers() at intr_event_execute_handlers+0x66 > ithread_loop() at ithread_loop+0xb2 > fork_exit() at fork_exit+0x135 > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff80005ccd00, rbp = 0 --- > KDB: enter panic > [thread pid 12 tid 100020 ] > Stopperd at kdb_enter+0x3b: movq $0,0x51cb32(%rip) > db> > > So questions:- > 1. What are the "hard error" errors? The machine was testing IO > with dd but due to the panic I cant tell if that was the cause. > 2. Looking at the code this seems like the reset was tripped by > firmware bug, is that the case? > 3. Is the fix the panic a simple one we cat test? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Mon Nov 5 21:29:14 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 163855CD; Mon, 5 Nov 2012 21:29:14 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 973628FC08; Mon, 5 Nov 2012 21:29:13 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 05 Nov 2012 13:30:30 -0800 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id qA5LTCS6019319; Mon, 5 Nov 2012 13:29:12 -0800 (PST) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id qA5LTBl2019318; Mon, 5 Nov 2012 13:29:11 -0800 (PST) (envelope-from ambrisko) Date: Mon, 5 Nov 2012 13:29:11 -0800 From: Doug Ambrisko To: Steven Hartland Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Message-ID: <20121105212911.GA17904@ambrisko.com> References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="kXdP64Ggrk/fb43R" Content-Disposition: inline In-Reply-To: <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> User-Agent: Mutt/1.4.2.3i Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 21:29:14 -0000 --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Nov 05, 2012 at 04:55:11PM -0000, Steven Hartland wrote: | I've managed to get the machine to reproduce this fairly regularly | now. | | Without a debug kernel it still results in a panic, just at a later | stage or so I believe, the none debug panic messages is "command not | in queue". | | In each none debug panic I've seen the cm_flags indicates the | command being dequeued is on the busy queue and not on the expected | free or ready queue which is being processed at the time. | | The triggering issue seems to be the adapter reset code run from | mfi_timeout. | | I've had a good look but can't see how a cm could be in a queue yet | have its cm_flags set to that of a different queue as all manipulation | seems to be being done via the "mfi_ ## name" macros which | all correctly maintain the queue / cm_flags relationship. | | At this point I believe it could be a thread being interrupted by | a timeout part way the processing of a queue request hence queue | and cm_flags being out of sync. | | Any pointers on how to debug this issue further / fix it would be most | appreciated. | | Regards | Steve | | ----- Original Message ----- | From: "Steven Hartland" | >Testing a new machine which is based on 8.3-RELEASE with the mfi | >driver from 8-STABLE and just got a panic. | > | > | >The below is translation of the hand copied from console:- | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 | >mfisyspd5: hard error cmd=write 90827650-90827905 | >mfi0: I/O error, status= 46 scsi_status= 240 | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 | >mfisyspd5: hard error cmd=write 90827394-90827649 | >mfi0: I/O error, status= 46 scsi_status= 240 | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 | >mfisyspd5: hard error cmd=write 90827138-90827393 | >mfi0: I/O error, status= 46 scsi_status= 240 | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 | >mfisyspd5: hard error cmd=write 90826882-90827137 | >mfi0: I/O error, status= 2 scsi_status= 2 | >mfi0: sense error 112, sense_key 6, asc 41, ascq 0 | >mfisyspd4: hard error cmd=write 90830466-90830721 | >mfi0: I/O error, status= 2 scsi_status= 2 | >mfi0: sense error 112, sense_key 6, asc 41, ascq 0 | >mfisyspd5: hard error cmd=write 90830722-90830977 | >mfi0: Adapter RESET condition detected | >mfi0: First state FW reset initiated... | >mfi0: ADP_RESET_TBOLT: HostDiag=a0 | >mfi0: first state of reset complete, second state initiated... | >mfi0: Second state FW reset initiated... | >panic: _mtx_lock_sleep: recursed on non-recusive mutex MFI I/O lock @ | >/usr/src/sys/dev/mfi/mfi_tbolt:346 | > | >cpuid = 6 | >KDB: stack backtrace: | >db_trace_self_wrapper() at db_trace_self_wrapper+0x2a | >kdb_backtrace() at kdb_backtrace+0x37 | >panic() at panic+0x178 | >_mtx_lock_sleep() at _mtx_lock_sleep+0x152 | >_mtx_lock_flags() at _mtx_lock_flags+0x80 | >mfi_tbolt_init_MFI_queue() at mfi_tbolt_init_MFI_queue+0x72 | >mfi_timeout() at mfi_timeout+0x27 | >softclock() at softclock+0x2aa | >intr_event_execute_handlers() at intr_event_execute_handlers+0x66 | >ithread_loop() at ithread_loop+0xb2 | >fork_exit() at fork_exit+0x135 | >fork_trampoline() at fork_trampoline+0xe | >--- trap 0, rip = 0, rsp = 0xffffff80005ccd00, rbp = 0 --- | >KDB: enter panic | >[thread pid 12 tid 100020 ] | >Stopperd at kdb_enter+0x3b: movq $0,0x51cb32(%rip) | >db> | > | >So questions:- | >1. What are the "hard error" errors? The machine was testing IO | >with dd but due to the panic I cant tell if that was the cause. | >2. Looking at the code this seems like the reset was tripped by | >firmware bug, is that the case? | >3. Is the fix the panic a simple one we cat test? As soon as I get caught up on email, I'll be checking in some updates that fix a bunch of issues. I don't know if it will fix this but it could help. At any case it would get us to a common base line to look at debugging this. The patch is attached and follows. It is relative to -head. It should be easy to head and this to stable. Index: mfi_tbolt.c =================================================================== --- mfi_tbolt.c (revision 242617) +++ mfi_tbolt.c (working copy) @@ -69,13 +69,10 @@ mfi_build_mpt_pass_thru(struct mfi_softc *sc, struct mfi_command *mfi_cmd); union mfi_mpi2_request_descriptor *mfi_build_and_issue_cmd(struct mfi_softc *sc, struct mfi_command *mfi_cmd); -int mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd); void mfi_tbolt_build_ldio(struct mfi_softc *sc, struct mfi_command *mfi_cmd, struct mfi_cmd_tbolt *cmd); static int mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command *mfi_cmd, pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd); -static int mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command - *mfi_cmd, uint8_t *cdb); void map_tbolt_cmd_status(struct mfi_command *mfi_cmd, uint8_t status, uint8_t ext_status); @@ -502,6 +499,7 @@ + i * MEGASAS_MAX_SZ_CHAIN_FRAME); cmd->sg_frame_phys_addr = sc->sg_frame_busaddr + i * MEGASAS_MAX_SZ_CHAIN_FRAME; + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; TAILQ_INSERT_TAIL(&(sc->mfi_cmd_tbolt_tqh), cmd, next); } @@ -608,6 +606,8 @@ } } +int outstanding = 0; + /* * mfi_tbolt_return_cmd - Return a cmd to free command pool * @instance: Adapter soft state @@ -618,7 +618,9 @@ { mtx_assert(&sc->mfi_io_lock, MA_OWNED); + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; TAILQ_INSERT_TAIL(&sc->mfi_cmd_tbolt_tqh, cmd, next); + outstanding--; } void @@ -667,16 +669,27 @@ extStatus = cmd_mfi->cm_frame->dcmd.header.scsi_status; map_tbolt_cmd_status(cmd_mfi, status, extStatus); - /* remove command from busy queue if not polled */ - TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { - if (cmd_mfi_check == cmd_mfi) { - mfi_remove_busy(cmd_mfi); - break; + if (cmd_mfi->cm_flags & MFI_CMD_SCSI && + (cmd_mfi->cm_flags & MFI_CMD_POLLED) != 0) { + /* polled LD/SYSPD IO command */ + cmd_mfi->cm_error = 0; + cmd_mfi->cm_frame->header.cmd_status = 0; + mfi_tbolt_return_cmd(sc, cmd_tbolt); + } else { + + /* remove command from busy queue if not polled */ + TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { + if (cmd_mfi_check == cmd_mfi) { + mfi_remove_busy(cmd_mfi); + break; + } } + + /* complete the command */ + cmd_mfi->cm_error = 0; + mfi_complete(sc, cmd_mfi); + mfi_tbolt_return_cmd(sc, cmd_tbolt); } - cmd_mfi->cm_error = 0; - mfi_complete(sc, cmd_mfi); - mfi_tbolt_return_cmd(sc, cmd_tbolt); sc->last_reply_idx++; if (sc->last_reply_idx >= sc->mfi_max_fw_cmds) { @@ -746,6 +759,7 @@ p = sc->request_desc_pool + sizeof(union mfi_mpi2_request_descriptor) * index; memset(p, 0, sizeof(union mfi_mpi2_request_descriptor)); + outstanding++; return (union mfi_mpi2_request_descriptor *)p; } @@ -811,13 +825,13 @@ MFI_FRAME_DIR_READ) io_info.isRead = 1; - io_request->RaidContext.timeoutValue - = MFI_FUSION_FP_DEFAULT_TIMEOUT; - io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; - io_request->DevHandle = device_id; - cmd->request_desc->header.RequestFlags - = (MFI_REQ_DESCRIPT_FLAGS_LD_IO - << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); + io_request->RaidContext.timeoutValue + = MFI_FUSION_FP_DEFAULT_TIMEOUT; + io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; + io_request->DevHandle = device_id; + cmd->request_desc->header.RequestFlags + = (MFI_REQ_DESCRIPT_FLAGS_LD_IO + << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); if ((io_request->IoFlags == 6) && (io_info.numBlocks == 0)) io_request->RaidContext.RegLockLength = 0x100; io_request->DataLength = mfi_cmd->cm_frame->io.header.data_len @@ -825,41 +839,37 @@ } int -mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd) -{ - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ - || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - return 1; - else - return 0; -} - -int mfi_tbolt_build_io(struct mfi_softc *sc, struct mfi_command *mfi_cmd, struct mfi_cmd_tbolt *cmd) { - uint32_t device_id; + struct mfi_mpi2_request_raid_scsi_io *io_request; uint32_t sge_count; - uint8_t cdb[32], cdb_len; + uint8_t cdb_len; + int readop; + u_int64_t lba; - memset(cdb, 0, 32); - struct mfi_mpi2_request_raid_scsi_io *io_request = cmd->io_request; - - device_id = mfi_cmd->cm_frame->header.target_id; - - /* Have to build CDB here for TB as BSD don't have a scsi layer */ - if ((cdb_len = mfi_tbolt_build_cdb(sc, mfi_cmd, cdb)) == 1) + io_request = cmd->io_request; + if (!(mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ + || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE)) return 1; - /* Just the CDB length,rest of the Flags are zero */ - io_request->IoFlags = cdb_len; - memcpy(io_request->CDB.CDB32, cdb, 32); + mfi_tbolt_build_ldio(sc, mfi_cmd, cmd); - if (mfi_tbolt_is_ldio(mfi_cmd)) - mfi_tbolt_build_ldio(sc, mfi_cmd , cmd); + /* Convert to SCSI command CDB */ + bzero(io_request->CDB.CDB32, sizeof(io_request->CDB.CDB32)); + if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) + readop = 0; else - return 1; + readop = 1; + lba = mfi_cmd->cm_frame->io.lba_hi; + lba = (lba << 32) + mfi_cmd->cm_frame->io.lba_lo; + cdb_len = mfi_build_cdb(readop, 0, lba, + mfi_cmd->cm_frame->io.header.data_len, io_request->CDB.CDB32); + + /* Just the CDB length, rest of the Flags are zero */ + io_request->IoFlags = cdb_len; + /* * Construct SGL */ @@ -883,85 +893,13 @@ io_request->SenseBufferLowAddress = mfi_cmd->cm_sense_busaddr; io_request->SenseBufferLength = MFI_SENSE_LEN; + io_request->RaidContext.Status = MFI_STAT_INVALID_STATUS; + io_request->RaidContext.exStatus = MFI_STAT_INVALID_STATUS; + return 0; } -static int -mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command *mfi_cmd, - uint8_t *cdb) -{ - uint32_t lba_lo, lba_hi, num_lba; - uint8_t cdb_len; - if (mfi_cmd == NULL || cdb == NULL) - return 1; - num_lba = mfi_cmd->cm_frame->io.header.data_len; - lba_lo = mfi_cmd->cm_frame->io.lba_lo; - lba_hi = mfi_cmd->cm_frame->io.lba_hi; - - if (lba_hi == 0 && (num_lba <= 0xFF) && (lba_lo <= 0x1FFFFF)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 6 or Write 6 */ - cdb[0] = (uint8_t) (0x0A); - else - cdb[0] = (uint8_t) (0x08); - - cdb[4] = (uint8_t) num_lba; - cdb[3] = (uint8_t) (lba_lo & 0xFF); - cdb[2] = (uint8_t) (lba_lo >> 8); - cdb[1] = (uint8_t) ((lba_lo >> 16) & 0x1F); - cdb_len = 6; - } - else if (lba_hi == 0 && (num_lba <= 0xFFFF) && (lba_lo <= 0xFFFFFFFF)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 10 or Write 10 */ - cdb[0] = (uint8_t) (0x2A); - else - cdb[0] = (uint8_t) (0x28); - cdb[8] = (uint8_t) (num_lba & 0xFF); - cdb[7] = (uint8_t) (num_lba >> 8); - cdb[5] = (uint8_t) (lba_lo & 0xFF); - cdb[4] = (uint8_t) (lba_lo >> 8); - cdb[3] = (uint8_t) (lba_lo >> 16); - cdb[2] = (uint8_t) (lba_lo >> 24); - cdb_len = 10; - } else if ((num_lba > 0xFFFF) && (lba_hi == 0)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 12 or Write 12 */ - cdb[0] = (uint8_t) (0xAA); - else - cdb[0] = (uint8_t) (0xA8); - cdb[9] = (uint8_t) (num_lba & 0xFF); - cdb[8] = (uint8_t) (num_lba >> 8); - cdb[7] = (uint8_t) (num_lba >> 16); - cdb[6] = (uint8_t) (num_lba >> 24); - cdb[5] = (uint8_t) (lba_lo & 0xFF); - cdb[4] = (uint8_t) (lba_lo >> 8); - cdb[3] = (uint8_t) (lba_lo >> 16); - cdb[2] = (uint8_t) (lba_lo >> 24); - cdb_len = 12; - } else { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - cdb[0] = (uint8_t) (0x8A); - else - cdb[0] = (uint8_t) (0x88); - cdb[13] = (uint8_t) (num_lba & 0xFF); - cdb[12] = (uint8_t) (num_lba >> 8); - cdb[11] = (uint8_t) (num_lba >> 16); - cdb[10] = (uint8_t) (num_lba >> 24); - cdb[9] = (uint8_t) (lba_lo & 0xFF); - cdb[8] = (uint8_t) (lba_lo >> 8); - cdb[7] = (uint8_t) (lba_lo >> 16); - cdb[6] = (uint8_t) (lba_lo >> 24); - cdb[5] = (uint8_t) (lba_hi & 0xFF); - cdb[4] = (uint8_t) (lba_hi >> 8); - cdb[3] = (uint8_t) (lba_hi >> 16); - cdb[2] = (uint8_t) (lba_hi >> 24); - cdb_len = 16; - } - return cdb_len; -} - static int mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command *mfi_cmd, pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd) @@ -1100,8 +1038,7 @@ if ((cm->cm_flags & MFI_CMD_POLLED) == 0) { cm->cm_timestamp = time_uptime; mfi_enqueue_busy(cm); - } - else { /* still get interrupts for it */ + } else { /* still get interrupts for it */ hdr->cmd_status = MFI_STAT_INVALID_STATUS; hdr->flags |= MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; } @@ -1118,31 +1055,49 @@ } else device_printf(sc->mfi_dev, "DJA NA XXX SYSPDIO\n"); - } - else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || + } else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || hdr->cmd == MFI_CMD_LD_READ || hdr->cmd == MFI_CMD_LD_WRITE) { + cm->cm_flags |= MFI_CMD_SCSI; if ((req_desc = mfi_build_and_issue_cmd(sc, cm)) == NULL) { device_printf(sc->mfi_dev, "LDIO Failed \n"); return 1; } - } else - if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { + } else if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { device_printf(sc->mfi_dev, "Mapping from MFI to MPT " "Failed\n"); return 1; - } + } + + if (cm->cm_flags & MFI_CMD_SCSI) { + /* + * LD IO needs to be posted since it doesn't get + * acknowledged via a status update so have the + * controller reply via mfi_tbolt_complete_cmd. + */ + hdr->flags &= ~MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; + } + MFI_WRITE4(sc, MFI_ILQP, (req_desc->words & 0xFFFFFFFF)); MFI_WRITE4(sc, MFI_IHQP, (req_desc->words >>0x20)); if ((cm->cm_flags & MFI_CMD_POLLED) == 0) return 0; + if (cm->cm_flags & MFI_CMD_SCSI) { + /* check reply queue */ + mfi_tbolt_complete_cmd(sc); + } + /* This is a polled command, so busy-wait for it to complete. */ while (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { DELAY(1000); tm -= 1; if (tm <= 0) - break; + break; + if (cm->cm_flags & MFI_CMD_SCSI) { + /* check reply queue */ + mfi_tbolt_complete_cmd(sc); + } } if (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { @@ -1375,7 +1330,7 @@ free(ld_sync, M_MFIBUF); goto out; } - + context = cmd->cm_frame->header.context; bzero(cmd->cm_frame, sizeof(union mfi_frame)); cmd->cm_frame->header.context = context; Index: mfi_disk.c =================================================================== --- mfi_disk.c (revision 242617) +++ mfi_disk.c (working copy) @@ -93,6 +93,7 @@ { struct mfi_disk *sc; struct mfi_ld_info *ld_info; + struct mfi_disk_pending *ld_pend; uint64_t sectors; uint32_t secsize; char *state; @@ -111,6 +112,13 @@ secsize = MFI_SECTOR_LEN; mtx_lock(&sc->ld_controller->mfi_io_lock); TAILQ_INSERT_TAIL(&sc->ld_controller->mfi_ld_tqh, sc, ld_link); + TAILQ_FOREACH(ld_pend, &sc->ld_controller->mfi_ld_pend_tqh, + ld_link) { + TAILQ_REMOVE(&sc->ld_controller->mfi_ld_pend_tqh, + ld_pend, ld_link); + free(ld_pend, M_MFIBUF); + break; + } mtx_unlock(&sc->ld_controller->mfi_io_lock); switch (ld_info->ld_config.params.state) { @@ -131,16 +139,16 @@ break; } - if ( strlen(ld_info->ld_config.properties.name) == 0 ) { - device_printf(dev, - "%juMB (%ju sectors) RAID volume (no label) is %s\n", - sectors / (1024 * 1024 / secsize), sectors, state); - } else { - device_printf(dev, - "%juMB (%ju sectors) RAID volume '%s' is %s\n", - sectors / (1024 * 1024 / secsize), sectors, - ld_info->ld_config.properties.name, state); - } + if ( strlen(ld_info->ld_config.properties.name) == 0 ) { + device_printf(dev, + "%juMB (%ju sectors) RAID volume (no label) is %s\n", + sectors / (1024 * 1024 / secsize), sectors, state); + } else { + device_printf(dev, + "%juMB (%ju sectors) RAID volume '%s' is %s\n", + sectors / (1024 * 1024 / secsize), sectors, + ld_info->ld_config.properties.name, state); + } sc->ld_disk = disk_alloc(); sc->ld_disk->d_drv1 = sc; Index: mfivar.h =================================================================== --- mfivar.h (revision 242617) +++ mfivar.h (working copy) @@ -106,6 +106,7 @@ #define MFI_ON_MFIQ_READY (1<<6) #define MFI_ON_MFIQ_BUSY (1<<7) #define MFI_ON_MFIQ_MASK ((1<<5)|(1<<6)|(1<<7)) +#define MFI_CMD_SCSI (1<<8) uint8_t retry_for_fw_reset; void (* cm_complete)(struct mfi_command *cm); void *cm_private; @@ -126,6 +127,11 @@ #define MFI_DISK_FLAGS_DISABLED 0x02 }; +struct mfi_disk_pending { + TAILQ_ENTRY(mfi_disk_pending) ld_link; + int ld_id; +}; + struct mfi_system_pd { TAILQ_ENTRY(mfi_system_pd) pd_link; device_t pd_dev; @@ -137,6 +143,11 @@ int pd_flags; }; +struct mfi_system_pending { + TAILQ_ENTRY(mfi_system_pending) pd_link; + int pd_id; +}; + struct mfi_evt_queue_elm { TAILQ_ENTRY(mfi_evt_queue_elm) link; struct mfi_evt_detail detail; @@ -285,6 +296,8 @@ TAILQ_HEAD(,mfi_disk) mfi_ld_tqh; TAILQ_HEAD(,mfi_system_pd) mfi_syspd_tqh; + TAILQ_HEAD(,mfi_disk_pending) mfi_ld_pend_tqh; + TAILQ_HEAD(,mfi_system_pending) mfi_syspd_pend_tqh; eventhandler_tag mfi_eh; struct cdev *mfi_cdev; @@ -421,7 +434,8 @@ extern void mfi_tbolt_sync_map_info(struct mfi_softc *sc); extern void mfi_handle_map_sync(void *context, int pending); extern int mfi_dcmd_command(struct mfi_softc *, struct mfi_command **, - uint32_t, void **, size_t); + uint32_t, void **, size_t); +extern int mfi_build_cdb(int, uint8_t, u_int64_t, u_int32_t, uint8_t *); #define MFIQ_ADD(sc, qname) \ do { \ Index: mfi.c =================================================================== --- mfi.c (revision 242617) +++ mfi.c (working copy) @@ -106,11 +106,9 @@ static struct mfi_command * mfi_bio_command(struct mfi_softc *); static void mfi_bio_complete(struct mfi_command *); static struct mfi_command *mfi_build_ldio(struct mfi_softc *,struct bio*); -static int mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, - uint64_t lba, uint8_t byte2, int readop); static struct mfi_command *mfi_build_syspdio(struct mfi_softc *,struct bio*); static int mfi_send_frame(struct mfi_softc *, struct mfi_command *); -static int mfi_abort(struct mfi_softc *, struct mfi_command *); +static int mfi_abort(struct mfi_softc *, struct mfi_command **); static int mfi_linux_ioctl_int(struct cdev *, u_long, caddr_t, int, struct thread *); static void mfi_timeout(void *); static int mfi_user_command(struct mfi_softc *, @@ -376,6 +374,8 @@ sx_init(&sc->mfi_config_lock, "MFI config"); TAILQ_INIT(&sc->mfi_ld_tqh); TAILQ_INIT(&sc->mfi_syspd_tqh); + TAILQ_INIT(&sc->mfi_ld_pend_tqh); + TAILQ_INIT(&sc->mfi_syspd_pend_tqh); TAILQ_INIT(&sc->mfi_evt_queue); TASK_INIT(&sc->mfi_evt_task, 0, mfi_handle_evt, sc); TASK_INIT(&sc->mfi_map_sync_task, 0, mfi_handle_map_sync, sc); @@ -1281,6 +1281,17 @@ struct mfi_command *cm; int error; + + if (sc->mfi_aen_cm) + sc->cm_aen_abort = 1; + if (sc->mfi_aen_cm != NULL) + mfi_abort(sc, &sc->mfi_aen_cm); + + if (sc->mfi_map_sync_cm) + sc->cm_map_abort = 1; + if (sc->mfi_map_sync_cm != NULL) + mfi_abort(sc, &sc->mfi_map_sync_cm); + mtx_lock(&sc->mfi_io_lock); error = mfi_dcmd_command(sc, &cm, MFI_DCMD_CTRL_SHUTDOWN, NULL, 0); if (error) { @@ -1288,12 +1299,6 @@ return (error); } - if (sc->mfi_aen_cm != NULL) - mfi_abort(sc, sc->mfi_aen_cm); - - if (sc->mfi_map_sync_cm != NULL) - mfi_abort(sc, sc->mfi_map_sync_cm); - dcmd = &cm->cm_frame->dcmd; dcmd->header.flags = MFI_FRAME_DIR_NONE; cm->cm_flags = MFI_CMD_POLLED; @@ -1315,6 +1320,7 @@ struct mfi_command *cm = NULL; struct mfi_pd_list *pdlist = NULL; struct mfi_system_pd *syspd, *tmp; + struct mfi_system_pending *syspd_pend; int error, i, found; sx_assert(&sc->mfi_config_lock, SA_XLOCKED); @@ -1355,6 +1361,10 @@ if (syspd->pd_id == pdlist->addr[i].device_id) found = 1; } + TAILQ_FOREACH(syspd_pend, &sc->mfi_syspd_pend_tqh, pd_link) { + if (syspd_pend->pd_id == pdlist->addr[i].device_id) + found = 1; + } if (found == 0) mfi_add_sys_pd(sc, pdlist->addr[i].device_id); } @@ -1390,6 +1400,7 @@ struct mfi_command *cm = NULL; struct mfi_ld_list *list = NULL; struct mfi_disk *ld; + struct mfi_disk_pending *ld_pend; int error, i; sx_assert(&sc->mfi_config_lock, SA_XLOCKED); @@ -1418,6 +1429,10 @@ if (ld->ld_id == list->ld_list[i].ld.v.target_id) goto skip_add; } + TAILQ_FOREACH(ld_pend, &sc->mfi_ld_pend_tqh, ld_link) { + if (ld_pend->ld_id == list->ld_list[i].ld.v.target_id) + goto skip_add; + } mfi_add_ld(sc, list->ld_list[i].ld.v.target_id); skip_add:; } @@ -1620,9 +1635,7 @@ < current_aen.members.evt_class) current_aen.members.evt_class = prior_aen.members.evt_class; - mtx_lock(&sc->mfi_io_lock); - mfi_abort(sc, sc->mfi_aen_cm); - mtx_unlock(&sc->mfi_io_lock); + mfi_abort(sc, &sc->mfi_aen_cm); } } @@ -1814,10 +1827,17 @@ struct mfi_command *cm; struct mfi_dcmd_frame *dcmd = NULL; struct mfi_ld_info *ld_info = NULL; + struct mfi_disk_pending *ld_pend; int error; mtx_assert(&sc->mfi_io_lock, MA_OWNED); + ld_pend = malloc(sizeof(*ld_pend), M_MFIBUF, M_NOWAIT | M_ZERO); + if (ld_pend != NULL) { + ld_pend->ld_id = id; + TAILQ_INSERT_TAIL(&sc->mfi_ld_pend_tqh, ld_pend, ld_link); + } + error = mfi_dcmd_command(sc, &cm, MFI_DCMD_LD_GET_INFO, (void **)&ld_info, sizeof(*ld_info)); if (error) { @@ -1858,11 +1878,13 @@ hdr = &cm->cm_frame->header; ld_info = cm->cm_private; - if (hdr->cmd_status != MFI_STAT_OK) { + if (sc->cm_map_abort || hdr->cmd_status != MFI_STAT_OK) { free(ld_info, M_MFIBUF); + wakeup(&sc->mfi_map_sync_cm); mfi_release_command(cm); return; } + wakeup(&sc->mfi_map_sync_cm); mfi_release_command(cm); mtx_unlock(&sc->mfi_io_lock); @@ -1887,10 +1909,17 @@ struct mfi_command *cm; struct mfi_dcmd_frame *dcmd = NULL; struct mfi_pd_info *pd_info = NULL; + struct mfi_system_pending *syspd_pend; int error; mtx_assert(&sc->mfi_io_lock, MA_OWNED); + syspd_pend = malloc(sizeof(*syspd_pend), M_MFIBUF, M_NOWAIT | M_ZERO); + if (syspd_pend != NULL) { + syspd_pend->pd_id = id; + TAILQ_INSERT_TAIL(&sc->mfi_syspd_pend_tqh, syspd_pend, pd_link); + } + error = mfi_dcmd_command(sc, &cm, MFI_DCMD_PD_GET_INFO, (void **)&pd_info, sizeof(*pd_info)); if (error) { @@ -1985,9 +2014,12 @@ return cm; } -static int -mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, - uint64_t lba, uint8_t byte2, int readop) +/* + * mostly copied from cam/scsi/scsi_all.c:scsi_read_write + */ + +int +mfi_build_cdb(int readop, uint8_t byte2, u_int64_t lba, u_int32_t block_count, uint8_t *cdb) { int cdb_len; @@ -1997,7 +2029,7 @@ /* We can fit in a 6 byte cdb */ struct scsi_rw_6 *scsi_cmd; - scsi_cmd = (struct scsi_rw_6 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_6 *)cdb; scsi_cmd->opcode = readop ? READ_6 : WRITE_6; scsi_ulto3b(lba, scsi_cmd->addr); scsi_cmd->length = block_count & 0xff; @@ -2007,7 +2039,7 @@ /* Need a 10 byte CDB */ struct scsi_rw_10 *scsi_cmd; - scsi_cmd = (struct scsi_rw_10 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_10 *)cdb; scsi_cmd->opcode = readop ? READ_10 : WRITE_10; scsi_cmd->byte2 = byte2; scsi_ulto4b(lba, scsi_cmd->addr); @@ -2020,7 +2052,7 @@ /* Block count is too big for 10 byte CDB use a 12 byte CDB */ struct scsi_rw_12 *scsi_cmd; - scsi_cmd = (struct scsi_rw_12 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_12 *)cdb; scsi_cmd->opcode = readop ? READ_12 : WRITE_12; scsi_cmd->byte2 = byte2; scsi_ulto4b(lba, scsi_cmd->addr); @@ -2035,7 +2067,7 @@ */ struct scsi_rw_16 *scsi_cmd; - scsi_cmd = (struct scsi_rw_16 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_16 *)cdb; scsi_cmd->opcode = readop ? READ_16 : WRITE_16; scsi_cmd->byte2 = byte2; scsi_u64to8b(lba, scsi_cmd->addr); @@ -2053,15 +2085,15 @@ { struct mfi_command *cm; struct mfi_pass_frame *pass; - int flags = 0; + uint32_t context = 0; + int flags = 0, blkcount = 0, readop; uint8_t cdb_len; - uint32_t block_count, context = 0; if ((cm = mfi_dequeue_free(sc)) == NULL) return (NULL); /* Zero out the MFI frame */ - context = cm->cm_frame->header.context; + context = cm->cm_frame->header.context; bzero(cm->cm_frame, sizeof(union mfi_frame)); cm->cm_frame->header.context = context; pass = &cm->cm_frame->pass; @@ -2070,22 +2102,24 @@ switch (bio->bio_cmd & 0x03) { case BIO_READ: flags = MFI_CMD_DATAIN; + readop = 1; break; case BIO_WRITE: flags = MFI_CMD_DATAOUT; + readop = 0; break; default: /* TODO: what about BIO_DELETE??? */ - panic("Unsupported bio command"); + panic("Unsupported bio command %x\n", bio->bio_cmd); } /* Cheat with the sector length to avoid a non-constant division */ - block_count = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; + blkcount = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; /* Fill the LBA and Transfer length in CDB */ - cdb_len = mfi_build_syspd_cdb(pass, block_count, bio->bio_pblkno, 0, - flags == MFI_CMD_DATAIN); - + cdb_len = mfi_build_cdb(readop, 0, bio->bio_pblkno, blkcount, + pass->cdb); pass->header.target_id = (uintptr_t)bio->bio_driver1; + pass->header.lun_id = 0; pass->header.timeout = 0; pass->header.flags = 0; pass->header.scsi_status = 0; @@ -2132,7 +2166,7 @@ break; default: /* TODO: what about BIO_DELETE??? */ - panic("Unsupported bio command"); + panic("Unsupported bio command %x\n", bio->bio_cmd); } /* Cheat with the sector length to avoid a non-constant division */ @@ -2422,15 +2456,14 @@ } static int -mfi_abort(struct mfi_softc *sc, struct mfi_command *cm_abort) +mfi_abort(struct mfi_softc *sc, struct mfi_command **cm_abort) { struct mfi_command *cm; struct mfi_abort_frame *abort; int i = 0; uint32_t context = 0; - mtx_assert(&sc->mfi_io_lock, MA_OWNED); - + mtx_lock(&sc->mfi_io_lock); if ((cm = mfi_dequeue_free(sc)) == NULL) { return (EBUSY); } @@ -2444,29 +2477,27 @@ abort->header.cmd = MFI_CMD_ABORT; abort->header.flags = 0; abort->header.scsi_status = 0; - abort->abort_context = cm_abort->cm_frame->header.context; - abort->abort_mfi_addr_lo = (uint32_t)cm_abort->cm_frame_busaddr; + abort->abort_context = (*cm_abort)->cm_frame->header.context; + abort->abort_mfi_addr_lo = (uint32_t)(*cm_abort)->cm_frame_busaddr; abort->abort_mfi_addr_hi = - (uint32_t)((uint64_t)cm_abort->cm_frame_busaddr >> 32); + (uint32_t)((uint64_t)(*cm_abort)->cm_frame_busaddr >> 32); cm->cm_data = NULL; cm->cm_flags = MFI_CMD_POLLED; - if (sc->mfi_aen_cm) - sc->cm_aen_abort = 1; - if (sc->mfi_map_sync_cm) - sc->cm_map_abort = 1; mfi_mapcmd(sc, cm); mfi_release_command(cm); - while (i < 5 && sc->mfi_aen_cm != NULL) { - msleep(&sc->mfi_aen_cm, &sc->mfi_io_lock, 0, "mfiabort", + mtx_unlock(&sc->mfi_io_lock); + while (i < 5 && *cm_abort != NULL) { + tsleep(cm_abort, 0, "mfiabort", 5 * hz); i++; } - while (i < 5 && sc->mfi_map_sync_cm != NULL) { - msleep(&sc->mfi_map_sync_cm, &sc->mfi_io_lock, 0, "mfiabort", - 5 * hz); - i++; + if (*cm_abort != NULL) { + /* Force a complete if command didn't abort */ + mtx_lock(&sc->mfi_io_lock); + (*cm_abort)->cm_complete(*cm_abort); + mtx_unlock(&sc->mfi_io_lock); } return (0); @@ -2522,7 +2553,7 @@ { struct mfi_command *cm; struct mfi_pass_frame *pass; - int error; + int error, readop, cdb_len; uint32_t blkcount; if ((cm = mfi_dequeue_free(sc)) == NULL) @@ -2531,21 +2562,24 @@ pass = &cm->cm_frame->pass; bzero(pass->cdb, 16); pass->header.cmd = MFI_CMD_PD_SCSI_IO; + + readop = 0; blkcount = (len + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; + cdb_len = mfi_build_cdb(readop, 0, lba, blkcount, pass->cdb); pass->header.target_id = id; pass->header.timeout = 0; pass->header.flags = 0; pass->header.scsi_status = 0; pass->header.sense_len = MFI_SENSE_LEN; pass->header.data_len = len; - pass->header.cdb_len = mfi_build_syspd_cdb(pass, blkcount, lba, 0, 0); + pass->header.cdb_len = cdb_len; pass->sense_addr_lo = (uint32_t)cm->cm_sense_busaddr; pass->sense_addr_hi = (uint32_t)((uint64_t)cm->cm_sense_busaddr >> 32); cm->cm_data = virt; cm->cm_len = len; cm->cm_sg = &pass->sgl; cm->cm_total_frame_size = MFI_PASS_FRAME_SIZE; - cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT; + cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT | MFI_CMD_SCSI; error = mfi_mapcmd(sc, cm); bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap, @@ -2745,16 +2779,24 @@ } } -static int mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) +static int +mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) { - struct mfi_config_data *conf_data=(struct mfi_config_data *)cm->cm_data; + struct mfi_config_data *conf_data; struct mfi_command *ld_cm = NULL; struct mfi_ld_info *ld_info = NULL; + struct mfi_ld_config *ld; + char *p; int error = 0; - if ((cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) && - (conf_data->ld[0].params.isSSCD == 1)) { - error = 1; + conf_data = (struct mfi_config_data *)cm->cm_data; + + if (cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) { + p = (char *)conf_data->array; + p += conf_data->array_size * conf_data->array_count; + ld = (struct mfi_ld_config *)p; + if (ld->params.isSSCD == 1) + error = 1; } else if (cm->cm_frame->dcmd.opcode == MFI_DCMD_LD_DELETE) { error = mfi_dcmd_command (sc, &ld_cm, MFI_DCMD_LD_GET_INFO, (void **)&ld_info, sizeof(*ld_info)); Index: mfi_cam.c =================================================================== --- mfi_cam.c (revision 242617) +++ mfi_cam.c (working copy) @@ -79,6 +79,11 @@ static struct mfi_command * mfip_start(void *); static void mfip_done(struct mfi_command *cm); +static int mfi_allow_disks = 0; +TUNABLE_INT("hw.mfi.allow_cam_disk_passthrough", &mfi_allow_disks); +SYSCTL_INT(_hw_mfi, OID_AUTO, allow_cam_disk_passthrough, CTLFLAG_RD, + &mfi_allow_disks, 0, "event message locale"); + static devclass_t mfip_devclass; static device_method_t mfip_methods[] = { DEVMETHOD(device_probe, mfip_probe), @@ -349,7 +354,8 @@ command = csio->cdb_io.cdb_bytes[0]; if (command == INQUIRY) { device = csio->data_ptr[0] & 0x1f; - if ((device == T_DIRECT) || (device == T_PROCESSOR)) + if ((!mfi_allow_disks && device == T_DIRECT) || + (device == T_PROCESSOR)) csio->data_ptr[0] = (csio->data_ptr[0] & 0xe0) | T_NODEVICE; } Index: mfi_syspd.c =================================================================== --- mfi_syspd.c (revision 242617) +++ mfi_syspd.c (working copy) @@ -89,7 +89,6 @@ static int mfi_syspd_probe(device_t dev) { - return (0); } @@ -98,12 +97,12 @@ { struct mfi_system_pd *sc; struct mfi_pd_info *pd_info; + struct mfi_system_pending *syspd_pend; uint64_t sectors; uint32_t secsize; sc = device_get_softc(dev); pd_info = device_get_ivars(dev); - sc->pd_dev = dev; sc->pd_id = pd_info->ref.v.device_id; sc->pd_unit = device_get_unit(dev); @@ -115,6 +114,13 @@ secsize = MFI_SECTOR_LEN; mtx_lock(&sc->pd_controller->mfi_io_lock); TAILQ_INSERT_TAIL(&sc->pd_controller->mfi_syspd_tqh, sc, pd_link); + TAILQ_FOREACH(syspd_pend, &sc->pd_controller->mfi_syspd_pend_tqh, + pd_link) { + TAILQ_REMOVE(&sc->pd_controller->mfi_syspd_pend_tqh, + syspd_pend, pd_link); + free(syspd_pend, M_MFIBUF); + break; + } mtx_unlock(&sc->pd_controller->mfi_io_lock); device_printf(dev, "%juMB (%ju sectors) SYSPD volume\n", sectors / (1024 * 1024 / secsize), sectors); @@ -139,6 +145,7 @@ disk_create(sc->pd_disk, DISK_VERSION); device_printf(dev, " SYSPD volume attached\n"); + return (0); } Thanks, Doug A. --kXdP64Ggrk/fb43R Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="mfi.patch" Index: mfi_tbolt.c =================================================================== --- mfi_tbolt.c (revision 242617) +++ mfi_tbolt.c (working copy) @@ -69,13 +69,10 @@ mfi_build_mpt_pass_thru(struct mfi_softc *sc, struct mfi_command *mfi_cmd); union mfi_mpi2_request_descriptor *mfi_build_and_issue_cmd(struct mfi_softc *sc, struct mfi_command *mfi_cmd); -int mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd); void mfi_tbolt_build_ldio(struct mfi_softc *sc, struct mfi_command *mfi_cmd, struct mfi_cmd_tbolt *cmd); static int mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command *mfi_cmd, pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd); -static int mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command - *mfi_cmd, uint8_t *cdb); void map_tbolt_cmd_status(struct mfi_command *mfi_cmd, uint8_t status, uint8_t ext_status); @@ -502,6 +499,7 @@ + i * MEGASAS_MAX_SZ_CHAIN_FRAME); cmd->sg_frame_phys_addr = sc->sg_frame_busaddr + i * MEGASAS_MAX_SZ_CHAIN_FRAME; + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; TAILQ_INSERT_TAIL(&(sc->mfi_cmd_tbolt_tqh), cmd, next); } @@ -608,6 +606,8 @@ } } +int outstanding = 0; + /* * mfi_tbolt_return_cmd - Return a cmd to free command pool * @instance: Adapter soft state @@ -618,7 +618,9 @@ { mtx_assert(&sc->mfi_io_lock, MA_OWNED); + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; TAILQ_INSERT_TAIL(&sc->mfi_cmd_tbolt_tqh, cmd, next); + outstanding--; } void @@ -667,16 +669,27 @@ extStatus = cmd_mfi->cm_frame->dcmd.header.scsi_status; map_tbolt_cmd_status(cmd_mfi, status, extStatus); - /* remove command from busy queue if not polled */ - TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { - if (cmd_mfi_check == cmd_mfi) { - mfi_remove_busy(cmd_mfi); - break; + if (cmd_mfi->cm_flags & MFI_CMD_SCSI && + (cmd_mfi->cm_flags & MFI_CMD_POLLED) != 0) { + /* polled LD/SYSPD IO command */ + cmd_mfi->cm_error = 0; + cmd_mfi->cm_frame->header.cmd_status = 0; + mfi_tbolt_return_cmd(sc, cmd_tbolt); + } else { + + /* remove command from busy queue if not polled */ + TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { + if (cmd_mfi_check == cmd_mfi) { + mfi_remove_busy(cmd_mfi); + break; + } } + + /* complete the command */ + cmd_mfi->cm_error = 0; + mfi_complete(sc, cmd_mfi); + mfi_tbolt_return_cmd(sc, cmd_tbolt); } - cmd_mfi->cm_error = 0; - mfi_complete(sc, cmd_mfi); - mfi_tbolt_return_cmd(sc, cmd_tbolt); sc->last_reply_idx++; if (sc->last_reply_idx >= sc->mfi_max_fw_cmds) { @@ -746,6 +759,7 @@ p = sc->request_desc_pool + sizeof(union mfi_mpi2_request_descriptor) * index; memset(p, 0, sizeof(union mfi_mpi2_request_descriptor)); + outstanding++; return (union mfi_mpi2_request_descriptor *)p; } @@ -811,13 +825,13 @@ MFI_FRAME_DIR_READ) io_info.isRead = 1; - io_request->RaidContext.timeoutValue - = MFI_FUSION_FP_DEFAULT_TIMEOUT; - io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; - io_request->DevHandle = device_id; - cmd->request_desc->header.RequestFlags - = (MFI_REQ_DESCRIPT_FLAGS_LD_IO - << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); + io_request->RaidContext.timeoutValue + = MFI_FUSION_FP_DEFAULT_TIMEOUT; + io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; + io_request->DevHandle = device_id; + cmd->request_desc->header.RequestFlags + = (MFI_REQ_DESCRIPT_FLAGS_LD_IO + << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); if ((io_request->IoFlags == 6) && (io_info.numBlocks == 0)) io_request->RaidContext.RegLockLength = 0x100; io_request->DataLength = mfi_cmd->cm_frame->io.header.data_len @@ -825,41 +839,37 @@ } int -mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd) -{ - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ - || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - return 1; - else - return 0; -} - -int mfi_tbolt_build_io(struct mfi_softc *sc, struct mfi_command *mfi_cmd, struct mfi_cmd_tbolt *cmd) { - uint32_t device_id; + struct mfi_mpi2_request_raid_scsi_io *io_request; uint32_t sge_count; - uint8_t cdb[32], cdb_len; + uint8_t cdb_len; + int readop; + u_int64_t lba; - memset(cdb, 0, 32); - struct mfi_mpi2_request_raid_scsi_io *io_request = cmd->io_request; - - device_id = mfi_cmd->cm_frame->header.target_id; - - /* Have to build CDB here for TB as BSD don't have a scsi layer */ - if ((cdb_len = mfi_tbolt_build_cdb(sc, mfi_cmd, cdb)) == 1) + io_request = cmd->io_request; + if (!(mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ + || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE)) return 1; - /* Just the CDB length,rest of the Flags are zero */ - io_request->IoFlags = cdb_len; - memcpy(io_request->CDB.CDB32, cdb, 32); + mfi_tbolt_build_ldio(sc, mfi_cmd, cmd); - if (mfi_tbolt_is_ldio(mfi_cmd)) - mfi_tbolt_build_ldio(sc, mfi_cmd , cmd); + /* Convert to SCSI command CDB */ + bzero(io_request->CDB.CDB32, sizeof(io_request->CDB.CDB32)); + if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) + readop = 0; else - return 1; + readop = 1; + lba = mfi_cmd->cm_frame->io.lba_hi; + lba = (lba << 32) + mfi_cmd->cm_frame->io.lba_lo; + cdb_len = mfi_build_cdb(readop, 0, lba, + mfi_cmd->cm_frame->io.header.data_len, io_request->CDB.CDB32); + + /* Just the CDB length, rest of the Flags are zero */ + io_request->IoFlags = cdb_len; + /* * Construct SGL */ @@ -883,85 +893,13 @@ io_request->SenseBufferLowAddress = mfi_cmd->cm_sense_busaddr; io_request->SenseBufferLength = MFI_SENSE_LEN; + io_request->RaidContext.Status = MFI_STAT_INVALID_STATUS; + io_request->RaidContext.exStatus = MFI_STAT_INVALID_STATUS; + return 0; } -static int -mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command *mfi_cmd, - uint8_t *cdb) -{ - uint32_t lba_lo, lba_hi, num_lba; - uint8_t cdb_len; - if (mfi_cmd == NULL || cdb == NULL) - return 1; - num_lba = mfi_cmd->cm_frame->io.header.data_len; - lba_lo = mfi_cmd->cm_frame->io.lba_lo; - lba_hi = mfi_cmd->cm_frame->io.lba_hi; - - if (lba_hi == 0 && (num_lba <= 0xFF) && (lba_lo <= 0x1FFFFF)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 6 or Write 6 */ - cdb[0] = (uint8_t) (0x0A); - else - cdb[0] = (uint8_t) (0x08); - - cdb[4] = (uint8_t) num_lba; - cdb[3] = (uint8_t) (lba_lo & 0xFF); - cdb[2] = (uint8_t) (lba_lo >> 8); - cdb[1] = (uint8_t) ((lba_lo >> 16) & 0x1F); - cdb_len = 6; - } - else if (lba_hi == 0 && (num_lba <= 0xFFFF) && (lba_lo <= 0xFFFFFFFF)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 10 or Write 10 */ - cdb[0] = (uint8_t) (0x2A); - else - cdb[0] = (uint8_t) (0x28); - cdb[8] = (uint8_t) (num_lba & 0xFF); - cdb[7] = (uint8_t) (num_lba >> 8); - cdb[5] = (uint8_t) (lba_lo & 0xFF); - cdb[4] = (uint8_t) (lba_lo >> 8); - cdb[3] = (uint8_t) (lba_lo >> 16); - cdb[2] = (uint8_t) (lba_lo >> 24); - cdb_len = 10; - } else if ((num_lba > 0xFFFF) && (lba_hi == 0)) { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - /* Read 12 or Write 12 */ - cdb[0] = (uint8_t) (0xAA); - else - cdb[0] = (uint8_t) (0xA8); - cdb[9] = (uint8_t) (num_lba & 0xFF); - cdb[8] = (uint8_t) (num_lba >> 8); - cdb[7] = (uint8_t) (num_lba >> 16); - cdb[6] = (uint8_t) (num_lba >> 24); - cdb[5] = (uint8_t) (lba_lo & 0xFF); - cdb[4] = (uint8_t) (lba_lo >> 8); - cdb[3] = (uint8_t) (lba_lo >> 16); - cdb[2] = (uint8_t) (lba_lo >> 24); - cdb_len = 12; - } else { - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) - cdb[0] = (uint8_t) (0x8A); - else - cdb[0] = (uint8_t) (0x88); - cdb[13] = (uint8_t) (num_lba & 0xFF); - cdb[12] = (uint8_t) (num_lba >> 8); - cdb[11] = (uint8_t) (num_lba >> 16); - cdb[10] = (uint8_t) (num_lba >> 24); - cdb[9] = (uint8_t) (lba_lo & 0xFF); - cdb[8] = (uint8_t) (lba_lo >> 8); - cdb[7] = (uint8_t) (lba_lo >> 16); - cdb[6] = (uint8_t) (lba_lo >> 24); - cdb[5] = (uint8_t) (lba_hi & 0xFF); - cdb[4] = (uint8_t) (lba_hi >> 8); - cdb[3] = (uint8_t) (lba_hi >> 16); - cdb[2] = (uint8_t) (lba_hi >> 24); - cdb_len = 16; - } - return cdb_len; -} - static int mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command *mfi_cmd, pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd) @@ -1100,8 +1038,7 @@ if ((cm->cm_flags & MFI_CMD_POLLED) == 0) { cm->cm_timestamp = time_uptime; mfi_enqueue_busy(cm); - } - else { /* still get interrupts for it */ + } else { /* still get interrupts for it */ hdr->cmd_status = MFI_STAT_INVALID_STATUS; hdr->flags |= MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; } @@ -1118,31 +1055,49 @@ } else device_printf(sc->mfi_dev, "DJA NA XXX SYSPDIO\n"); - } - else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || + } else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || hdr->cmd == MFI_CMD_LD_READ || hdr->cmd == MFI_CMD_LD_WRITE) { + cm->cm_flags |= MFI_CMD_SCSI; if ((req_desc = mfi_build_and_issue_cmd(sc, cm)) == NULL) { device_printf(sc->mfi_dev, "LDIO Failed \n"); return 1; } - } else - if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { + } else if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { device_printf(sc->mfi_dev, "Mapping from MFI to MPT " "Failed\n"); return 1; - } + } + + if (cm->cm_flags & MFI_CMD_SCSI) { + /* + * LD IO needs to be posted since it doesn't get + * acknowledged via a status update so have the + * controller reply via mfi_tbolt_complete_cmd. + */ + hdr->flags &= ~MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; + } + MFI_WRITE4(sc, MFI_ILQP, (req_desc->words & 0xFFFFFFFF)); MFI_WRITE4(sc, MFI_IHQP, (req_desc->words >>0x20)); if ((cm->cm_flags & MFI_CMD_POLLED) == 0) return 0; + if (cm->cm_flags & MFI_CMD_SCSI) { + /* check reply queue */ + mfi_tbolt_complete_cmd(sc); + } + /* This is a polled command, so busy-wait for it to complete. */ while (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { DELAY(1000); tm -= 1; if (tm <= 0) - break; + break; + if (cm->cm_flags & MFI_CMD_SCSI) { + /* check reply queue */ + mfi_tbolt_complete_cmd(sc); + } } if (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { @@ -1375,7 +1330,7 @@ free(ld_sync, M_MFIBUF); goto out; } - + context = cmd->cm_frame->header.context; bzero(cmd->cm_frame, sizeof(union mfi_frame)); cmd->cm_frame->header.context = context; Index: mfi_disk.c =================================================================== --- mfi_disk.c (revision 242617) +++ mfi_disk.c (working copy) @@ -93,6 +93,7 @@ { struct mfi_disk *sc; struct mfi_ld_info *ld_info; + struct mfi_disk_pending *ld_pend; uint64_t sectors; uint32_t secsize; char *state; @@ -111,6 +112,13 @@ secsize = MFI_SECTOR_LEN; mtx_lock(&sc->ld_controller->mfi_io_lock); TAILQ_INSERT_TAIL(&sc->ld_controller->mfi_ld_tqh, sc, ld_link); + TAILQ_FOREACH(ld_pend, &sc->ld_controller->mfi_ld_pend_tqh, + ld_link) { + TAILQ_REMOVE(&sc->ld_controller->mfi_ld_pend_tqh, + ld_pend, ld_link); + free(ld_pend, M_MFIBUF); + break; + } mtx_unlock(&sc->ld_controller->mfi_io_lock); switch (ld_info->ld_config.params.state) { @@ -131,16 +139,16 @@ break; } - if ( strlen(ld_info->ld_config.properties.name) == 0 ) { - device_printf(dev, - "%juMB (%ju sectors) RAID volume (no label) is %s\n", - sectors / (1024 * 1024 / secsize), sectors, state); - } else { - device_printf(dev, - "%juMB (%ju sectors) RAID volume '%s' is %s\n", - sectors / (1024 * 1024 / secsize), sectors, - ld_info->ld_config.properties.name, state); - } + if ( strlen(ld_info->ld_config.properties.name) == 0 ) { + device_printf(dev, + "%juMB (%ju sectors) RAID volume (no label) is %s\n", + sectors / (1024 * 1024 / secsize), sectors, state); + } else { + device_printf(dev, + "%juMB (%ju sectors) RAID volume '%s' is %s\n", + sectors / (1024 * 1024 / secsize), sectors, + ld_info->ld_config.properties.name, state); + } sc->ld_disk = disk_alloc(); sc->ld_disk->d_drv1 = sc; Index: mfivar.h =================================================================== --- mfivar.h (revision 242617) +++ mfivar.h (working copy) @@ -106,6 +106,7 @@ #define MFI_ON_MFIQ_READY (1<<6) #define MFI_ON_MFIQ_BUSY (1<<7) #define MFI_ON_MFIQ_MASK ((1<<5)|(1<<6)|(1<<7)) +#define MFI_CMD_SCSI (1<<8) uint8_t retry_for_fw_reset; void (* cm_complete)(struct mfi_command *cm); void *cm_private; @@ -126,6 +127,11 @@ #define MFI_DISK_FLAGS_DISABLED 0x02 }; +struct mfi_disk_pending { + TAILQ_ENTRY(mfi_disk_pending) ld_link; + int ld_id; +}; + struct mfi_system_pd { TAILQ_ENTRY(mfi_system_pd) pd_link; device_t pd_dev; @@ -137,6 +143,11 @@ int pd_flags; }; +struct mfi_system_pending { + TAILQ_ENTRY(mfi_system_pending) pd_link; + int pd_id; +}; + struct mfi_evt_queue_elm { TAILQ_ENTRY(mfi_evt_queue_elm) link; struct mfi_evt_detail detail; @@ -285,6 +296,8 @@ TAILQ_HEAD(,mfi_disk) mfi_ld_tqh; TAILQ_HEAD(,mfi_system_pd) mfi_syspd_tqh; + TAILQ_HEAD(,mfi_disk_pending) mfi_ld_pend_tqh; + TAILQ_HEAD(,mfi_system_pending) mfi_syspd_pend_tqh; eventhandler_tag mfi_eh; struct cdev *mfi_cdev; @@ -421,7 +434,8 @@ extern void mfi_tbolt_sync_map_info(struct mfi_softc *sc); extern void mfi_handle_map_sync(void *context, int pending); extern int mfi_dcmd_command(struct mfi_softc *, struct mfi_command **, - uint32_t, void **, size_t); + uint32_t, void **, size_t); +extern int mfi_build_cdb(int, uint8_t, u_int64_t, u_int32_t, uint8_t *); #define MFIQ_ADD(sc, qname) \ do { \ Index: mfi.c =================================================================== --- mfi.c (revision 242617) +++ mfi.c (working copy) @@ -106,11 +106,9 @@ static struct mfi_command * mfi_bio_command(struct mfi_softc *); static void mfi_bio_complete(struct mfi_command *); static struct mfi_command *mfi_build_ldio(struct mfi_softc *,struct bio*); -static int mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, - uint64_t lba, uint8_t byte2, int readop); static struct mfi_command *mfi_build_syspdio(struct mfi_softc *,struct bio*); static int mfi_send_frame(struct mfi_softc *, struct mfi_command *); -static int mfi_abort(struct mfi_softc *, struct mfi_command *); +static int mfi_abort(struct mfi_softc *, struct mfi_command **); static int mfi_linux_ioctl_int(struct cdev *, u_long, caddr_t, int, struct thread *); static void mfi_timeout(void *); static int mfi_user_command(struct mfi_softc *, @@ -376,6 +374,8 @@ sx_init(&sc->mfi_config_lock, "MFI config"); TAILQ_INIT(&sc->mfi_ld_tqh); TAILQ_INIT(&sc->mfi_syspd_tqh); + TAILQ_INIT(&sc->mfi_ld_pend_tqh); + TAILQ_INIT(&sc->mfi_syspd_pend_tqh); TAILQ_INIT(&sc->mfi_evt_queue); TASK_INIT(&sc->mfi_evt_task, 0, mfi_handle_evt, sc); TASK_INIT(&sc->mfi_map_sync_task, 0, mfi_handle_map_sync, sc); @@ -1281,6 +1281,17 @@ struct mfi_command *cm; int error; + + if (sc->mfi_aen_cm) + sc->cm_aen_abort = 1; + if (sc->mfi_aen_cm != NULL) + mfi_abort(sc, &sc->mfi_aen_cm); + + if (sc->mfi_map_sync_cm) + sc->cm_map_abort = 1; + if (sc->mfi_map_sync_cm != NULL) + mfi_abort(sc, &sc->mfi_map_sync_cm); + mtx_lock(&sc->mfi_io_lock); error = mfi_dcmd_command(sc, &cm, MFI_DCMD_CTRL_SHUTDOWN, NULL, 0); if (error) { @@ -1288,12 +1299,6 @@ return (error); } - if (sc->mfi_aen_cm != NULL) - mfi_abort(sc, sc->mfi_aen_cm); - - if (sc->mfi_map_sync_cm != NULL) - mfi_abort(sc, sc->mfi_map_sync_cm); - dcmd = &cm->cm_frame->dcmd; dcmd->header.flags = MFI_FRAME_DIR_NONE; cm->cm_flags = MFI_CMD_POLLED; @@ -1315,6 +1320,7 @@ struct mfi_command *cm = NULL; struct mfi_pd_list *pdlist = NULL; struct mfi_system_pd *syspd, *tmp; + struct mfi_system_pending *syspd_pend; int error, i, found; sx_assert(&sc->mfi_config_lock, SA_XLOCKED); @@ -1355,6 +1361,10 @@ if (syspd->pd_id == pdlist->addr[i].device_id) found = 1; } + TAILQ_FOREACH(syspd_pend, &sc->mfi_syspd_pend_tqh, pd_link) { + if (syspd_pend->pd_id == pdlist->addr[i].device_id) + found = 1; + } if (found == 0) mfi_add_sys_pd(sc, pdlist->addr[i].device_id); } @@ -1390,6 +1400,7 @@ struct mfi_command *cm = NULL; struct mfi_ld_list *list = NULL; struct mfi_disk *ld; + struct mfi_disk_pending *ld_pend; int error, i; sx_assert(&sc->mfi_config_lock, SA_XLOCKED); @@ -1418,6 +1429,10 @@ if (ld->ld_id == list->ld_list[i].ld.v.target_id) goto skip_add; } + TAILQ_FOREACH(ld_pend, &sc->mfi_ld_pend_tqh, ld_link) { + if (ld_pend->ld_id == list->ld_list[i].ld.v.target_id) + goto skip_add; + } mfi_add_ld(sc, list->ld_list[i].ld.v.target_id); skip_add:; } @@ -1620,9 +1635,7 @@ < current_aen.members.evt_class) current_aen.members.evt_class = prior_aen.members.evt_class; - mtx_lock(&sc->mfi_io_lock); - mfi_abort(sc, sc->mfi_aen_cm); - mtx_unlock(&sc->mfi_io_lock); + mfi_abort(sc, &sc->mfi_aen_cm); } } @@ -1814,10 +1827,17 @@ struct mfi_command *cm; struct mfi_dcmd_frame *dcmd = NULL; struct mfi_ld_info *ld_info = NULL; + struct mfi_disk_pending *ld_pend; int error; mtx_assert(&sc->mfi_io_lock, MA_OWNED); + ld_pend = malloc(sizeof(*ld_pend), M_MFIBUF, M_NOWAIT | M_ZERO); + if (ld_pend != NULL) { + ld_pend->ld_id = id; + TAILQ_INSERT_TAIL(&sc->mfi_ld_pend_tqh, ld_pend, ld_link); + } + error = mfi_dcmd_command(sc, &cm, MFI_DCMD_LD_GET_INFO, (void **)&ld_info, sizeof(*ld_info)); if (error) { @@ -1858,11 +1878,13 @@ hdr = &cm->cm_frame->header; ld_info = cm->cm_private; - if (hdr->cmd_status != MFI_STAT_OK) { + if (sc->cm_map_abort || hdr->cmd_status != MFI_STAT_OK) { free(ld_info, M_MFIBUF); + wakeup(&sc->mfi_map_sync_cm); mfi_release_command(cm); return; } + wakeup(&sc->mfi_map_sync_cm); mfi_release_command(cm); mtx_unlock(&sc->mfi_io_lock); @@ -1887,10 +1909,17 @@ struct mfi_command *cm; struct mfi_dcmd_frame *dcmd = NULL; struct mfi_pd_info *pd_info = NULL; + struct mfi_system_pending *syspd_pend; int error; mtx_assert(&sc->mfi_io_lock, MA_OWNED); + syspd_pend = malloc(sizeof(*syspd_pend), M_MFIBUF, M_NOWAIT | M_ZERO); + if (syspd_pend != NULL) { + syspd_pend->pd_id = id; + TAILQ_INSERT_TAIL(&sc->mfi_syspd_pend_tqh, syspd_pend, pd_link); + } + error = mfi_dcmd_command(sc, &cm, MFI_DCMD_PD_GET_INFO, (void **)&pd_info, sizeof(*pd_info)); if (error) { @@ -1985,9 +2014,12 @@ return cm; } -static int -mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, - uint64_t lba, uint8_t byte2, int readop) +/* + * mostly copied from cam/scsi/scsi_all.c:scsi_read_write + */ + +int +mfi_build_cdb(int readop, uint8_t byte2, u_int64_t lba, u_int32_t block_count, uint8_t *cdb) { int cdb_len; @@ -1997,7 +2029,7 @@ /* We can fit in a 6 byte cdb */ struct scsi_rw_6 *scsi_cmd; - scsi_cmd = (struct scsi_rw_6 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_6 *)cdb; scsi_cmd->opcode = readop ? READ_6 : WRITE_6; scsi_ulto3b(lba, scsi_cmd->addr); scsi_cmd->length = block_count & 0xff; @@ -2007,7 +2039,7 @@ /* Need a 10 byte CDB */ struct scsi_rw_10 *scsi_cmd; - scsi_cmd = (struct scsi_rw_10 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_10 *)cdb; scsi_cmd->opcode = readop ? READ_10 : WRITE_10; scsi_cmd->byte2 = byte2; scsi_ulto4b(lba, scsi_cmd->addr); @@ -2020,7 +2052,7 @@ /* Block count is too big for 10 byte CDB use a 12 byte CDB */ struct scsi_rw_12 *scsi_cmd; - scsi_cmd = (struct scsi_rw_12 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_12 *)cdb; scsi_cmd->opcode = readop ? READ_12 : WRITE_12; scsi_cmd->byte2 = byte2; scsi_ulto4b(lba, scsi_cmd->addr); @@ -2035,7 +2067,7 @@ */ struct scsi_rw_16 *scsi_cmd; - scsi_cmd = (struct scsi_rw_16 *)&pass->cdb; + scsi_cmd = (struct scsi_rw_16 *)cdb; scsi_cmd->opcode = readop ? READ_16 : WRITE_16; scsi_cmd->byte2 = byte2; scsi_u64to8b(lba, scsi_cmd->addr); @@ -2053,15 +2085,15 @@ { struct mfi_command *cm; struct mfi_pass_frame *pass; - int flags = 0; + uint32_t context = 0; + int flags = 0, blkcount = 0, readop; uint8_t cdb_len; - uint32_t block_count, context = 0; if ((cm = mfi_dequeue_free(sc)) == NULL) return (NULL); /* Zero out the MFI frame */ - context = cm->cm_frame->header.context; + context = cm->cm_frame->header.context; bzero(cm->cm_frame, sizeof(union mfi_frame)); cm->cm_frame->header.context = context; pass = &cm->cm_frame->pass; @@ -2070,22 +2102,24 @@ switch (bio->bio_cmd & 0x03) { case BIO_READ: flags = MFI_CMD_DATAIN; + readop = 1; break; case BIO_WRITE: flags = MFI_CMD_DATAOUT; + readop = 0; break; default: /* TODO: what about BIO_DELETE??? */ - panic("Unsupported bio command"); + panic("Unsupported bio command %x\n", bio->bio_cmd); } /* Cheat with the sector length to avoid a non-constant division */ - block_count = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; + blkcount = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; /* Fill the LBA and Transfer length in CDB */ - cdb_len = mfi_build_syspd_cdb(pass, block_count, bio->bio_pblkno, 0, - flags == MFI_CMD_DATAIN); - + cdb_len = mfi_build_cdb(readop, 0, bio->bio_pblkno, blkcount, + pass->cdb); pass->header.target_id = (uintptr_t)bio->bio_driver1; + pass->header.lun_id = 0; pass->header.timeout = 0; pass->header.flags = 0; pass->header.scsi_status = 0; @@ -2132,7 +2166,7 @@ break; default: /* TODO: what about BIO_DELETE??? */ - panic("Unsupported bio command"); + panic("Unsupported bio command %x\n", bio->bio_cmd); } /* Cheat with the sector length to avoid a non-constant division */ @@ -2422,15 +2456,14 @@ } static int -mfi_abort(struct mfi_softc *sc, struct mfi_command *cm_abort) +mfi_abort(struct mfi_softc *sc, struct mfi_command **cm_abort) { struct mfi_command *cm; struct mfi_abort_frame *abort; int i = 0; uint32_t context = 0; - mtx_assert(&sc->mfi_io_lock, MA_OWNED); - + mtx_lock(&sc->mfi_io_lock); if ((cm = mfi_dequeue_free(sc)) == NULL) { return (EBUSY); } @@ -2444,29 +2477,27 @@ abort->header.cmd = MFI_CMD_ABORT; abort->header.flags = 0; abort->header.scsi_status = 0; - abort->abort_context = cm_abort->cm_frame->header.context; - abort->abort_mfi_addr_lo = (uint32_t)cm_abort->cm_frame_busaddr; + abort->abort_context = (*cm_abort)->cm_frame->header.context; + abort->abort_mfi_addr_lo = (uint32_t)(*cm_abort)->cm_frame_busaddr; abort->abort_mfi_addr_hi = - (uint32_t)((uint64_t)cm_abort->cm_frame_busaddr >> 32); + (uint32_t)((uint64_t)(*cm_abort)->cm_frame_busaddr >> 32); cm->cm_data = NULL; cm->cm_flags = MFI_CMD_POLLED; - if (sc->mfi_aen_cm) - sc->cm_aen_abort = 1; - if (sc->mfi_map_sync_cm) - sc->cm_map_abort = 1; mfi_mapcmd(sc, cm); mfi_release_command(cm); - while (i < 5 && sc->mfi_aen_cm != NULL) { - msleep(&sc->mfi_aen_cm, &sc->mfi_io_lock, 0, "mfiabort", + mtx_unlock(&sc->mfi_io_lock); + while (i < 5 && *cm_abort != NULL) { + tsleep(cm_abort, 0, "mfiabort", 5 * hz); i++; } - while (i < 5 && sc->mfi_map_sync_cm != NULL) { - msleep(&sc->mfi_map_sync_cm, &sc->mfi_io_lock, 0, "mfiabort", - 5 * hz); - i++; + if (*cm_abort != NULL) { + /* Force a complete if command didn't abort */ + mtx_lock(&sc->mfi_io_lock); + (*cm_abort)->cm_complete(*cm_abort); + mtx_unlock(&sc->mfi_io_lock); } return (0); @@ -2522,7 +2553,7 @@ { struct mfi_command *cm; struct mfi_pass_frame *pass; - int error; + int error, readop, cdb_len; uint32_t blkcount; if ((cm = mfi_dequeue_free(sc)) == NULL) @@ -2531,21 +2562,24 @@ pass = &cm->cm_frame->pass; bzero(pass->cdb, 16); pass->header.cmd = MFI_CMD_PD_SCSI_IO; + + readop = 0; blkcount = (len + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; + cdb_len = mfi_build_cdb(readop, 0, lba, blkcount, pass->cdb); pass->header.target_id = id; pass->header.timeout = 0; pass->header.flags = 0; pass->header.scsi_status = 0; pass->header.sense_len = MFI_SENSE_LEN; pass->header.data_len = len; - pass->header.cdb_len = mfi_build_syspd_cdb(pass, blkcount, lba, 0, 0); + pass->header.cdb_len = cdb_len; pass->sense_addr_lo = (uint32_t)cm->cm_sense_busaddr; pass->sense_addr_hi = (uint32_t)((uint64_t)cm->cm_sense_busaddr >> 32); cm->cm_data = virt; cm->cm_len = len; cm->cm_sg = &pass->sgl; cm->cm_total_frame_size = MFI_PASS_FRAME_SIZE; - cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT; + cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT | MFI_CMD_SCSI; error = mfi_mapcmd(sc, cm); bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap, @@ -2745,16 +2779,24 @@ } } -static int mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) +static int +mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) { - struct mfi_config_data *conf_data=(struct mfi_config_data *)cm->cm_data; + struct mfi_config_data *conf_data; struct mfi_command *ld_cm = NULL; struct mfi_ld_info *ld_info = NULL; + struct mfi_ld_config *ld; + char *p; int error = 0; - if ((cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) && - (conf_data->ld[0].params.isSSCD == 1)) { - error = 1; + conf_data = (struct mfi_config_data *)cm->cm_data; + + if (cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) { + p = (char *)conf_data->array; + p += conf_data->array_size * conf_data->array_count; + ld = (struct mfi_ld_config *)p; + if (ld->params.isSSCD == 1) + error = 1; } else if (cm->cm_frame->dcmd.opcode == MFI_DCMD_LD_DELETE) { error = mfi_dcmd_command (sc, &ld_cm, MFI_DCMD_LD_GET_INFO, (void **)&ld_info, sizeof(*ld_info)); Index: mfi_cam.c =================================================================== --- mfi_cam.c (revision 242617) +++ mfi_cam.c (working copy) @@ -79,6 +79,11 @@ static struct mfi_command * mfip_start(void *); static void mfip_done(struct mfi_command *cm); +static int mfi_allow_disks = 0; +TUNABLE_INT("hw.mfi.allow_cam_disk_passthrough", &mfi_allow_disks); +SYSCTL_INT(_hw_mfi, OID_AUTO, allow_cam_disk_passthrough, CTLFLAG_RD, + &mfi_allow_disks, 0, "event message locale"); + static devclass_t mfip_devclass; static device_method_t mfip_methods[] = { DEVMETHOD(device_probe, mfip_probe), @@ -349,7 +354,8 @@ command = csio->cdb_io.cdb_bytes[0]; if (command == INQUIRY) { device = csio->data_ptr[0] & 0x1f; - if ((device == T_DIRECT) || (device == T_PROCESSOR)) + if ((!mfi_allow_disks && device == T_DIRECT) || + (device == T_PROCESSOR)) csio->data_ptr[0] = (csio->data_ptr[0] & 0xe0) | T_NODEVICE; } Index: mfi_syspd.c =================================================================== --- mfi_syspd.c (revision 242617) +++ mfi_syspd.c (working copy) @@ -89,7 +89,6 @@ static int mfi_syspd_probe(device_t dev) { - return (0); } @@ -98,12 +97,12 @@ { struct mfi_system_pd *sc; struct mfi_pd_info *pd_info; + struct mfi_system_pending *syspd_pend; uint64_t sectors; uint32_t secsize; sc = device_get_softc(dev); pd_info = device_get_ivars(dev); - sc->pd_dev = dev; sc->pd_id = pd_info->ref.v.device_id; sc->pd_unit = device_get_unit(dev); @@ -115,6 +114,13 @@ secsize = MFI_SECTOR_LEN; mtx_lock(&sc->pd_controller->mfi_io_lock); TAILQ_INSERT_TAIL(&sc->pd_controller->mfi_syspd_tqh, sc, pd_link); + TAILQ_FOREACH(syspd_pend, &sc->pd_controller->mfi_syspd_pend_tqh, + pd_link) { + TAILQ_REMOVE(&sc->pd_controller->mfi_syspd_pend_tqh, + syspd_pend, pd_link); + free(syspd_pend, M_MFIBUF); + break; + } mtx_unlock(&sc->pd_controller->mfi_io_lock); device_printf(dev, "%juMB (%ju sectors) SYSPD volume\n", sectors / (1024 * 1024 / secsize), sectors); @@ -139,6 +145,7 @@ disk_create(sc->pd_disk, DISK_VERSION); device_printf(dev, " SYSPD volume attached\n"); + return (0); } --kXdP64Ggrk/fb43R-- From owner-freebsd-scsi@FreeBSD.ORG Tue Nov 6 00:09:49 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46AF19B8; Tue, 6 Nov 2012 00:09:49 +0000 (UTC) (envelope-from prvs=1657173cfa=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id D86F48FC15; Tue, 6 Nov 2012 00:09:47 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000965135.msg; Tue, 06 Nov 2012 00:09:46 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 06 Nov 2012 00:09:46 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1657173cfa=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> From: "Steven Hartland" To: "Doug Ambrisko" References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Date: Tue, 6 Nov 2012 00:09:42 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2012 00:09:49 -0000 Thanks Doug, actually just finished another test run with some more debugging in and I believe I've found the reason for the non-recusive lock and at least some of the queuing issues. The non-recursive lock is due to the mfi_tbolt_reset calling mfi_process_fw_state_chg_isr with mfi_io_lock held which in turn calls mfi_tbolt_init_MFI_queue which tries to acquire mfi_io_lock hence the problem. mfi-lock.txt attached I believe fixes this as well as what appears to be an invalid call to mtx_unlock(&sc->mfi_io_lock) in mfi_attach which never acquires the lock as far as can see, possibly a cut and paste error. The invalid queue problems seem to stem from the error cases of the calls to mfi_mapcmd, some of which call mfi_release_command which blindly sets cm_flags = 0 and then enqueues it on the free queue. Now depending on the flow of mfi_mapcmd and where the error occurs the command may or may not have been put on the busy queue which is going to cause problems. Going to investigate this further but that's what my current theory is. Your patch seems quite extensive, so if could you give me brief run down on the changes that would be most appreciated. FYI, I'm aware that the cause of my underlying issues are some hardware issues (likely cable or backplane related) but it does mean I'm in the position to test these usually rare error cases, so wanting the make the most of it before we get the hardware swapped out. Regards Steve ----- Original Message ----- From: "Doug Ambrisko" To: "Steven Hartland" Cc: ; Sent: Monday, November 05, 2012 9:29 PM Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock > On Mon, Nov 05, 2012 at 04:55:11PM -0000, Steven Hartland wrote: > | I've managed to get the machine to reproduce this fairly regularly > | now. > | > | Without a debug kernel it still results in a panic, just at a later > | stage or so I believe, the none debug panic messages is "command not > | in queue". > | > | In each none debug panic I've seen the cm_flags indicates the > | command being dequeued is on the busy queue and not on the expected > | free or ready queue which is being processed at the time. > | > | The triggering issue seems to be the adapter reset code run from > | mfi_timeout. > | > | I've had a good look but can't see how a cm could be in a queue yet > | have its cm_flags set to that of a different queue as all manipulation > | seems to be being done via the "mfi_ ## name" macros which > | all correctly maintain the queue / cm_flags relationship. > | > | At this point I believe it could be a thread being interrupted by > | a timeout part way the processing of a queue request hence queue > | and cm_flags being out of sync. > | > | Any pointers on how to debug this issue further / fix it would be most > | appreciated. > | > | Regards > | Steve > | > | ----- Original Message ----- > | From: "Steven Hartland" > | >Testing a new machine which is based on 8.3-RELEASE with the mfi > | >driver from 8-STABLE and just got a panic. > | > > | > > | >The below is translation of the hand copied from console:- > | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > | >mfisyspd5: hard error cmd=write 90827650-90827905 > | >mfi0: I/O error, status= 46 scsi_status= 240 > | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > | >mfisyspd5: hard error cmd=write 90827394-90827649 > | >mfi0: I/O error, status= 46 scsi_status= 240 > | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > | >mfisyspd5: hard error cmd=write 90827138-90827393 > | >mfi0: I/O error, status= 46 scsi_status= 240 > | >mfi0: sense error 0, sense_key 0, asc 0, ascq 0 > | >mfisyspd5: hard error cmd=write 90826882-90827137 > | >mfi0: I/O error, status= 2 scsi_status= 2 > | >mfi0: sense error 112, sense_key 6, asc 41, ascq 0 > | >mfisyspd4: hard error cmd=write 90830466-90830721 > | >mfi0: I/O error, status= 2 scsi_status= 2 > | >mfi0: sense error 112, sense_key 6, asc 41, ascq 0 > | >mfisyspd5: hard error cmd=write 90830722-90830977 > | >mfi0: Adapter RESET condition detected > | >mfi0: First state FW reset initiated... > | >mfi0: ADP_RESET_TBOLT: HostDiag=a0 > | >mfi0: first state of reset complete, second state initiated... > | >mfi0: Second state FW reset initiated... > | >panic: _mtx_lock_sleep: recursed on non-recusive mutex MFI I/O lock @ > | >/usr/src/sys/dev/mfi/mfi_tbolt:346 > | > > | >cpuid = 6 > | >KDB: stack backtrace: > | >db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > | >kdb_backtrace() at kdb_backtrace+0x37 > | >panic() at panic+0x178 > | >_mtx_lock_sleep() at _mtx_lock_sleep+0x152 > | >_mtx_lock_flags() at _mtx_lock_flags+0x80 > | >mfi_tbolt_init_MFI_queue() at mfi_tbolt_init_MFI_queue+0x72 > | >mfi_timeout() at mfi_timeout+0x27 > | >softclock() at softclock+0x2aa > | >intr_event_execute_handlers() at intr_event_execute_handlers+0x66 > | >ithread_loop() at ithread_loop+0xb2 > | >fork_exit() at fork_exit+0x135 > | >fork_trampoline() at fork_trampoline+0xe > | >--- trap 0, rip = 0, rsp = 0xffffff80005ccd00, rbp = 0 --- > | >KDB: enter panic > | >[thread pid 12 tid 100020 ] > | >Stopperd at kdb_enter+0x3b: movq $0,0x51cb32(%rip) > | >db> > | > > | >So questions:- > | >1. What are the "hard error" errors? The machine was testing IO > | >with dd but due to the panic I cant tell if that was the cause. > | >2. Looking at the code this seems like the reset was tripped by > | >firmware bug, is that the case? > | >3. Is the fix the panic a simple one we cat test? > > As soon as I get caught up on email, I'll be checking in some updates > that fix a bunch of issues. I don't know if it will fix this but > it could help. At any case it would get us to a common base line to > look at debugging this. The patch is attached and follows. It is > relative to -head. It should be easy to head and this to stable. > > Index: mfi_tbolt.c > =================================================================== > --- mfi_tbolt.c (revision 242617) > +++ mfi_tbolt.c (working copy) > @@ -69,13 +69,10 @@ > mfi_build_mpt_pass_thru(struct mfi_softc *sc, struct mfi_command *mfi_cmd); > union mfi_mpi2_request_descriptor *mfi_build_and_issue_cmd(struct mfi_softc > *sc, struct mfi_command *mfi_cmd); > -int mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd); > void mfi_tbolt_build_ldio(struct mfi_softc *sc, struct mfi_command *mfi_cmd, > struct mfi_cmd_tbolt *cmd); > static int mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command > *mfi_cmd, pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd); > -static int mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command > - *mfi_cmd, uint8_t *cdb); > void > map_tbolt_cmd_status(struct mfi_command *mfi_cmd, uint8_t status, > uint8_t ext_status); > @@ -502,6 +499,7 @@ > + i * MEGASAS_MAX_SZ_CHAIN_FRAME); > cmd->sg_frame_phys_addr = sc->sg_frame_busaddr + i > * MEGASAS_MAX_SZ_CHAIN_FRAME; > + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; > > TAILQ_INSERT_TAIL(&(sc->mfi_cmd_tbolt_tqh), cmd, next); > } > @@ -608,6 +606,8 @@ > } > } > > +int outstanding = 0; > + > /* > * mfi_tbolt_return_cmd - Return a cmd to free command pool > * @instance: Adapter soft state > @@ -618,7 +618,9 @@ > { > mtx_assert(&sc->mfi_io_lock, MA_OWNED); > > + cmd->sync_cmd_idx = sc->mfi_max_fw_cmds; > TAILQ_INSERT_TAIL(&sc->mfi_cmd_tbolt_tqh, cmd, next); > + outstanding--; > } > > void > @@ -667,16 +669,27 @@ > extStatus = cmd_mfi->cm_frame->dcmd.header.scsi_status; > map_tbolt_cmd_status(cmd_mfi, status, extStatus); > > - /* remove command from busy queue if not polled */ > - TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { > - if (cmd_mfi_check == cmd_mfi) { > - mfi_remove_busy(cmd_mfi); > - break; > + if (cmd_mfi->cm_flags & MFI_CMD_SCSI && > + (cmd_mfi->cm_flags & MFI_CMD_POLLED) != 0) { > + /* polled LD/SYSPD IO command */ > + cmd_mfi->cm_error = 0; > + cmd_mfi->cm_frame->header.cmd_status = 0; > + mfi_tbolt_return_cmd(sc, cmd_tbolt); > + } else { > + > + /* remove command from busy queue if not polled */ > + TAILQ_FOREACH(cmd_mfi_check, &sc->mfi_busy, cm_link) { > + if (cmd_mfi_check == cmd_mfi) { > + mfi_remove_busy(cmd_mfi); > + break; > + } > } > + > + /* complete the command */ > + cmd_mfi->cm_error = 0; > + mfi_complete(sc, cmd_mfi); > + mfi_tbolt_return_cmd(sc, cmd_tbolt); > } > - cmd_mfi->cm_error = 0; > - mfi_complete(sc, cmd_mfi); > - mfi_tbolt_return_cmd(sc, cmd_tbolt); > > sc->last_reply_idx++; > if (sc->last_reply_idx >= sc->mfi_max_fw_cmds) { > @@ -746,6 +759,7 @@ > p = sc->request_desc_pool + sizeof(union mfi_mpi2_request_descriptor) > * index; > memset(p, 0, sizeof(union mfi_mpi2_request_descriptor)); > + outstanding++; > return (union mfi_mpi2_request_descriptor *)p; > } > > @@ -811,13 +825,13 @@ > MFI_FRAME_DIR_READ) > io_info.isRead = 1; > > - io_request->RaidContext.timeoutValue > - = MFI_FUSION_FP_DEFAULT_TIMEOUT; > - io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; > - io_request->DevHandle = device_id; > - cmd->request_desc->header.RequestFlags > - = (MFI_REQ_DESCRIPT_FLAGS_LD_IO > - << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); > + io_request->RaidContext.timeoutValue > + = MFI_FUSION_FP_DEFAULT_TIMEOUT; > + io_request->Function = MPI2_FUNCTION_LD_IO_REQUEST; > + io_request->DevHandle = device_id; > + cmd->request_desc->header.RequestFlags > + = (MFI_REQ_DESCRIPT_FLAGS_LD_IO > + << MFI_REQ_DESCRIPT_FLAGS_TYPE_SHIFT); > if ((io_request->IoFlags == 6) && (io_info.numBlocks == 0)) > io_request->RaidContext.RegLockLength = 0x100; > io_request->DataLength = mfi_cmd->cm_frame->io.header.data_len > @@ -825,41 +839,37 @@ > } > > int > -mfi_tbolt_is_ldio(struct mfi_command *mfi_cmd) > -{ > - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ > - || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > - return 1; > - else > - return 0; > -} > - > -int > mfi_tbolt_build_io(struct mfi_softc *sc, struct mfi_command *mfi_cmd, > struct mfi_cmd_tbolt *cmd) > { > - uint32_t device_id; > + struct mfi_mpi2_request_raid_scsi_io *io_request; > uint32_t sge_count; > - uint8_t cdb[32], cdb_len; > + uint8_t cdb_len; > + int readop; > + u_int64_t lba; > > - memset(cdb, 0, 32); > - struct mfi_mpi2_request_raid_scsi_io *io_request = cmd->io_request; > - > - device_id = mfi_cmd->cm_frame->header.target_id; > - > - /* Have to build CDB here for TB as BSD don't have a scsi layer */ > - if ((cdb_len = mfi_tbolt_build_cdb(sc, mfi_cmd, cdb)) == 1) > + io_request = cmd->io_request; > + if (!(mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_READ > + || mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE)) > return 1; > > - /* Just the CDB length,rest of the Flags are zero */ > - io_request->IoFlags = cdb_len; > - memcpy(io_request->CDB.CDB32, cdb, 32); > + mfi_tbolt_build_ldio(sc, mfi_cmd, cmd); > > - if (mfi_tbolt_is_ldio(mfi_cmd)) > - mfi_tbolt_build_ldio(sc, mfi_cmd , cmd); > + /* Convert to SCSI command CDB */ > + bzero(io_request->CDB.CDB32, sizeof(io_request->CDB.CDB32)); > + if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > + readop = 0; > else > - return 1; > + readop = 1; > > + lba = mfi_cmd->cm_frame->io.lba_hi; > + lba = (lba << 32) + mfi_cmd->cm_frame->io.lba_lo; > + cdb_len = mfi_build_cdb(readop, 0, lba, > + mfi_cmd->cm_frame->io.header.data_len, io_request->CDB.CDB32); > + > + /* Just the CDB length, rest of the Flags are zero */ > + io_request->IoFlags = cdb_len; > + > /* > * Construct SGL > */ > @@ -883,85 +893,13 @@ > > io_request->SenseBufferLowAddress = mfi_cmd->cm_sense_busaddr; > io_request->SenseBufferLength = MFI_SENSE_LEN; > + io_request->RaidContext.Status = MFI_STAT_INVALID_STATUS; > + io_request->RaidContext.exStatus = MFI_STAT_INVALID_STATUS; > + > return 0; > } > > -static int > -mfi_tbolt_build_cdb(struct mfi_softc *sc, struct mfi_command *mfi_cmd, > - uint8_t *cdb) > -{ > - uint32_t lba_lo, lba_hi, num_lba; > - uint8_t cdb_len; > > - if (mfi_cmd == NULL || cdb == NULL) > - return 1; > - num_lba = mfi_cmd->cm_frame->io.header.data_len; > - lba_lo = mfi_cmd->cm_frame->io.lba_lo; > - lba_hi = mfi_cmd->cm_frame->io.lba_hi; > - > - if (lba_hi == 0 && (num_lba <= 0xFF) && (lba_lo <= 0x1FFFFF)) { > - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > - /* Read 6 or Write 6 */ > - cdb[0] = (uint8_t) (0x0A); > - else > - cdb[0] = (uint8_t) (0x08); > - > - cdb[4] = (uint8_t) num_lba; > - cdb[3] = (uint8_t) (lba_lo & 0xFF); > - cdb[2] = (uint8_t) (lba_lo >> 8); > - cdb[1] = (uint8_t) ((lba_lo >> 16) & 0x1F); > - cdb_len = 6; > - } > - else if (lba_hi == 0 && (num_lba <= 0xFFFF) && (lba_lo <= 0xFFFFFFFF)) { > - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > - /* Read 10 or Write 10 */ > - cdb[0] = (uint8_t) (0x2A); > - else > - cdb[0] = (uint8_t) (0x28); > - cdb[8] = (uint8_t) (num_lba & 0xFF); > - cdb[7] = (uint8_t) (num_lba >> 8); > - cdb[5] = (uint8_t) (lba_lo & 0xFF); > - cdb[4] = (uint8_t) (lba_lo >> 8); > - cdb[3] = (uint8_t) (lba_lo >> 16); > - cdb[2] = (uint8_t) (lba_lo >> 24); > - cdb_len = 10; > - } else if ((num_lba > 0xFFFF) && (lba_hi == 0)) { > - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > - /* Read 12 or Write 12 */ > - cdb[0] = (uint8_t) (0xAA); > - else > - cdb[0] = (uint8_t) (0xA8); > - cdb[9] = (uint8_t) (num_lba & 0xFF); > - cdb[8] = (uint8_t) (num_lba >> 8); > - cdb[7] = (uint8_t) (num_lba >> 16); > - cdb[6] = (uint8_t) (num_lba >> 24); > - cdb[5] = (uint8_t) (lba_lo & 0xFF); > - cdb[4] = (uint8_t) (lba_lo >> 8); > - cdb[3] = (uint8_t) (lba_lo >> 16); > - cdb[2] = (uint8_t) (lba_lo >> 24); > - cdb_len = 12; > - } else { > - if (mfi_cmd->cm_frame->header.cmd == MFI_CMD_LD_WRITE) > - cdb[0] = (uint8_t) (0x8A); > - else > - cdb[0] = (uint8_t) (0x88); > - cdb[13] = (uint8_t) (num_lba & 0xFF); > - cdb[12] = (uint8_t) (num_lba >> 8); > - cdb[11] = (uint8_t) (num_lba >> 16); > - cdb[10] = (uint8_t) (num_lba >> 24); > - cdb[9] = (uint8_t) (lba_lo & 0xFF); > - cdb[8] = (uint8_t) (lba_lo >> 8); > - cdb[7] = (uint8_t) (lba_lo >> 16); > - cdb[6] = (uint8_t) (lba_lo >> 24); > - cdb[5] = (uint8_t) (lba_hi & 0xFF); > - cdb[4] = (uint8_t) (lba_hi >> 8); > - cdb[3] = (uint8_t) (lba_hi >> 16); > - cdb[2] = (uint8_t) (lba_hi >> 24); > - cdb_len = 16; > - } > - return cdb_len; > -} > - > static int > mfi_tbolt_make_sgl(struct mfi_softc *sc, struct mfi_command *mfi_cmd, > pMpi25IeeeSgeChain64_t sgl_ptr, struct mfi_cmd_tbolt *cmd) > @@ -1100,8 +1038,7 @@ > if ((cm->cm_flags & MFI_CMD_POLLED) == 0) { > cm->cm_timestamp = time_uptime; > mfi_enqueue_busy(cm); > - } > - else { /* still get interrupts for it */ > + } else { /* still get interrupts for it */ > hdr->cmd_status = MFI_STAT_INVALID_STATUS; > hdr->flags |= MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; > } > @@ -1118,31 +1055,49 @@ > } > else > device_printf(sc->mfi_dev, "DJA NA XXX SYSPDIO\n"); > - } > - else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || > + } else if (hdr->cmd == MFI_CMD_LD_SCSI_IO || > hdr->cmd == MFI_CMD_LD_READ || hdr->cmd == MFI_CMD_LD_WRITE) { > + cm->cm_flags |= MFI_CMD_SCSI; > if ((req_desc = mfi_build_and_issue_cmd(sc, cm)) == NULL) { > device_printf(sc->mfi_dev, "LDIO Failed \n"); > return 1; > } > - } else > - if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { > + } else if ((req_desc = mfi_tbolt_build_mpt_cmd(sc, cm)) == NULL) { > device_printf(sc->mfi_dev, "Mapping from MFI to MPT " > "Failed\n"); > return 1; > - } > + } > + > + if (cm->cm_flags & MFI_CMD_SCSI) { > + /* > + * LD IO needs to be posted since it doesn't get > + * acknowledged via a status update so have the > + * controller reply via mfi_tbolt_complete_cmd. > + */ > + hdr->flags &= ~MFI_FRAME_DONT_POST_IN_REPLY_QUEUE; > + } > + > MFI_WRITE4(sc, MFI_ILQP, (req_desc->words & 0xFFFFFFFF)); > MFI_WRITE4(sc, MFI_IHQP, (req_desc->words >>0x20)); > > if ((cm->cm_flags & MFI_CMD_POLLED) == 0) > return 0; > > + if (cm->cm_flags & MFI_CMD_SCSI) { > + /* check reply queue */ > + mfi_tbolt_complete_cmd(sc); > + } > + > /* This is a polled command, so busy-wait for it to complete. */ > while (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { > DELAY(1000); > tm -= 1; > if (tm <= 0) > - break; > + break; > + if (cm->cm_flags & MFI_CMD_SCSI) { > + /* check reply queue */ > + mfi_tbolt_complete_cmd(sc); > + } > } > > if (hdr->cmd_status == MFI_STAT_INVALID_STATUS) { > @@ -1375,7 +1330,7 @@ > free(ld_sync, M_MFIBUF); > goto out; > } > - > + > context = cmd->cm_frame->header.context; > bzero(cmd->cm_frame, sizeof(union mfi_frame)); > cmd->cm_frame->header.context = context; > Index: mfi_disk.c > =================================================================== > --- mfi_disk.c (revision 242617) > +++ mfi_disk.c (working copy) > @@ -93,6 +93,7 @@ > { > struct mfi_disk *sc; > struct mfi_ld_info *ld_info; > + struct mfi_disk_pending *ld_pend; > uint64_t sectors; > uint32_t secsize; > char *state; > @@ -111,6 +112,13 @@ > secsize = MFI_SECTOR_LEN; > mtx_lock(&sc->ld_controller->mfi_io_lock); > TAILQ_INSERT_TAIL(&sc->ld_controller->mfi_ld_tqh, sc, ld_link); > + TAILQ_FOREACH(ld_pend, &sc->ld_controller->mfi_ld_pend_tqh, > + ld_link) { > + TAILQ_REMOVE(&sc->ld_controller->mfi_ld_pend_tqh, > + ld_pend, ld_link); > + free(ld_pend, M_MFIBUF); > + break; > + } > mtx_unlock(&sc->ld_controller->mfi_io_lock); > > switch (ld_info->ld_config.params.state) { > @@ -131,16 +139,16 @@ > break; > } > > - if ( strlen(ld_info->ld_config.properties.name) == 0 ) { > - device_printf(dev, > - "%juMB (%ju sectors) RAID volume (no label) is %s\n", > - sectors / (1024 * 1024 / secsize), sectors, state); > - } else { > - device_printf(dev, > - "%juMB (%ju sectors) RAID volume '%s' is %s\n", > - sectors / (1024 * 1024 / secsize), sectors, > - ld_info->ld_config.properties.name, state); > - } > + if ( strlen(ld_info->ld_config.properties.name) == 0 ) { > + device_printf(dev, > + "%juMB (%ju sectors) RAID volume (no label) is %s\n", > + sectors / (1024 * 1024 / secsize), sectors, state); > + } else { > + device_printf(dev, > + "%juMB (%ju sectors) RAID volume '%s' is %s\n", > + sectors / (1024 * 1024 / secsize), sectors, > + ld_info->ld_config.properties.name, state); > + } > > sc->ld_disk = disk_alloc(); > sc->ld_disk->d_drv1 = sc; > Index: mfivar.h > =================================================================== > --- mfivar.h (revision 242617) > +++ mfivar.h (working copy) > @@ -106,6 +106,7 @@ > #define MFI_ON_MFIQ_READY (1<<6) > #define MFI_ON_MFIQ_BUSY (1<<7) > #define MFI_ON_MFIQ_MASK ((1<<5)|(1<<6)|(1<<7)) > +#define MFI_CMD_SCSI (1<<8) > uint8_t retry_for_fw_reset; > void (* cm_complete)(struct mfi_command *cm); > void *cm_private; > @@ -126,6 +127,11 @@ > #define MFI_DISK_FLAGS_DISABLED 0x02 > }; > > +struct mfi_disk_pending { > + TAILQ_ENTRY(mfi_disk_pending) ld_link; > + int ld_id; > +}; > + > struct mfi_system_pd { > TAILQ_ENTRY(mfi_system_pd) pd_link; > device_t pd_dev; > @@ -137,6 +143,11 @@ > int pd_flags; > }; > > +struct mfi_system_pending { > + TAILQ_ENTRY(mfi_system_pending) pd_link; > + int pd_id; > +}; > + > struct mfi_evt_queue_elm { > TAILQ_ENTRY(mfi_evt_queue_elm) link; > struct mfi_evt_detail detail; > @@ -285,6 +296,8 @@ > > TAILQ_HEAD(,mfi_disk) mfi_ld_tqh; > TAILQ_HEAD(,mfi_system_pd) mfi_syspd_tqh; > + TAILQ_HEAD(,mfi_disk_pending) mfi_ld_pend_tqh; > + TAILQ_HEAD(,mfi_system_pending) mfi_syspd_pend_tqh; > eventhandler_tag mfi_eh; > struct cdev *mfi_cdev; > > @@ -421,7 +434,8 @@ > extern void mfi_tbolt_sync_map_info(struct mfi_softc *sc); > extern void mfi_handle_map_sync(void *context, int pending); > extern int mfi_dcmd_command(struct mfi_softc *, struct mfi_command **, > - uint32_t, void **, size_t); > + uint32_t, void **, size_t); > +extern int mfi_build_cdb(int, uint8_t, u_int64_t, u_int32_t, uint8_t *); > > #define MFIQ_ADD(sc, qname) \ > do { \ > Index: mfi.c > =================================================================== > --- mfi.c (revision 242617) > +++ mfi.c (working copy) > @@ -106,11 +106,9 @@ > static struct mfi_command * mfi_bio_command(struct mfi_softc *); > static void mfi_bio_complete(struct mfi_command *); > static struct mfi_command *mfi_build_ldio(struct mfi_softc *,struct bio*); > -static int mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, > - uint64_t lba, uint8_t byte2, int readop); > static struct mfi_command *mfi_build_syspdio(struct mfi_softc *,struct bio*); > static int mfi_send_frame(struct mfi_softc *, struct mfi_command *); > -static int mfi_abort(struct mfi_softc *, struct mfi_command *); > +static int mfi_abort(struct mfi_softc *, struct mfi_command **); > static int mfi_linux_ioctl_int(struct cdev *, u_long, caddr_t, int, struct thread *); > static void mfi_timeout(void *); > static int mfi_user_command(struct mfi_softc *, > @@ -376,6 +374,8 @@ > sx_init(&sc->mfi_config_lock, "MFI config"); > TAILQ_INIT(&sc->mfi_ld_tqh); > TAILQ_INIT(&sc->mfi_syspd_tqh); > + TAILQ_INIT(&sc->mfi_ld_pend_tqh); > + TAILQ_INIT(&sc->mfi_syspd_pend_tqh); > TAILQ_INIT(&sc->mfi_evt_queue); > TASK_INIT(&sc->mfi_evt_task, 0, mfi_handle_evt, sc); > TASK_INIT(&sc->mfi_map_sync_task, 0, mfi_handle_map_sync, sc); > @@ -1281,6 +1281,17 @@ > struct mfi_command *cm; > int error; > > + > + if (sc->mfi_aen_cm) > + sc->cm_aen_abort = 1; > + if (sc->mfi_aen_cm != NULL) > + mfi_abort(sc, &sc->mfi_aen_cm); > + > + if (sc->mfi_map_sync_cm) > + sc->cm_map_abort = 1; > + if (sc->mfi_map_sync_cm != NULL) > + mfi_abort(sc, &sc->mfi_map_sync_cm); > + > mtx_lock(&sc->mfi_io_lock); > error = mfi_dcmd_command(sc, &cm, MFI_DCMD_CTRL_SHUTDOWN, NULL, 0); > if (error) { > @@ -1288,12 +1299,6 @@ > return (error); > } > > - if (sc->mfi_aen_cm != NULL) > - mfi_abort(sc, sc->mfi_aen_cm); > - > - if (sc->mfi_map_sync_cm != NULL) > - mfi_abort(sc, sc->mfi_map_sync_cm); > - > dcmd = &cm->cm_frame->dcmd; > dcmd->header.flags = MFI_FRAME_DIR_NONE; > cm->cm_flags = MFI_CMD_POLLED; > @@ -1315,6 +1320,7 @@ > struct mfi_command *cm = NULL; > struct mfi_pd_list *pdlist = NULL; > struct mfi_system_pd *syspd, *tmp; > + struct mfi_system_pending *syspd_pend; > int error, i, found; > > sx_assert(&sc->mfi_config_lock, SA_XLOCKED); > @@ -1355,6 +1361,10 @@ > if (syspd->pd_id == pdlist->addr[i].device_id) > found = 1; > } > + TAILQ_FOREACH(syspd_pend, &sc->mfi_syspd_pend_tqh, pd_link) { > + if (syspd_pend->pd_id == pdlist->addr[i].device_id) > + found = 1; > + } > if (found == 0) > mfi_add_sys_pd(sc, pdlist->addr[i].device_id); > } > @@ -1390,6 +1400,7 @@ > struct mfi_command *cm = NULL; > struct mfi_ld_list *list = NULL; > struct mfi_disk *ld; > + struct mfi_disk_pending *ld_pend; > int error, i; > > sx_assert(&sc->mfi_config_lock, SA_XLOCKED); > @@ -1418,6 +1429,10 @@ > if (ld->ld_id == list->ld_list[i].ld.v.target_id) > goto skip_add; > } > + TAILQ_FOREACH(ld_pend, &sc->mfi_ld_pend_tqh, ld_link) { > + if (ld_pend->ld_id == list->ld_list[i].ld.v.target_id) > + goto skip_add; > + } > mfi_add_ld(sc, list->ld_list[i].ld.v.target_id); > skip_add:; > } > @@ -1620,9 +1635,7 @@ > < current_aen.members.evt_class) > current_aen.members.evt_class = > prior_aen.members.evt_class; > - mtx_lock(&sc->mfi_io_lock); > - mfi_abort(sc, sc->mfi_aen_cm); > - mtx_unlock(&sc->mfi_io_lock); > + mfi_abort(sc, &sc->mfi_aen_cm); > } > } > > @@ -1814,10 +1827,17 @@ > struct mfi_command *cm; > struct mfi_dcmd_frame *dcmd = NULL; > struct mfi_ld_info *ld_info = NULL; > + struct mfi_disk_pending *ld_pend; > int error; > > mtx_assert(&sc->mfi_io_lock, MA_OWNED); > > + ld_pend = malloc(sizeof(*ld_pend), M_MFIBUF, M_NOWAIT | M_ZERO); > + if (ld_pend != NULL) { > + ld_pend->ld_id = id; > + TAILQ_INSERT_TAIL(&sc->mfi_ld_pend_tqh, ld_pend, ld_link); > + } > + > error = mfi_dcmd_command(sc, &cm, MFI_DCMD_LD_GET_INFO, > (void **)&ld_info, sizeof(*ld_info)); > if (error) { > @@ -1858,11 +1878,13 @@ > hdr = &cm->cm_frame->header; > ld_info = cm->cm_private; > > - if (hdr->cmd_status != MFI_STAT_OK) { > + if (sc->cm_map_abort || hdr->cmd_status != MFI_STAT_OK) { > free(ld_info, M_MFIBUF); > + wakeup(&sc->mfi_map_sync_cm); > mfi_release_command(cm); > return; > } > + wakeup(&sc->mfi_map_sync_cm); > mfi_release_command(cm); > > mtx_unlock(&sc->mfi_io_lock); > @@ -1887,10 +1909,17 @@ > struct mfi_command *cm; > struct mfi_dcmd_frame *dcmd = NULL; > struct mfi_pd_info *pd_info = NULL; > + struct mfi_system_pending *syspd_pend; > int error; > > mtx_assert(&sc->mfi_io_lock, MA_OWNED); > > + syspd_pend = malloc(sizeof(*syspd_pend), M_MFIBUF, M_NOWAIT | M_ZERO); > + if (syspd_pend != NULL) { > + syspd_pend->pd_id = id; > + TAILQ_INSERT_TAIL(&sc->mfi_syspd_pend_tqh, syspd_pend, pd_link); > + } > + > error = mfi_dcmd_command(sc, &cm, MFI_DCMD_PD_GET_INFO, > (void **)&pd_info, sizeof(*pd_info)); > if (error) { > @@ -1985,9 +2014,12 @@ > return cm; > } > > -static int > -mfi_build_syspd_cdb(struct mfi_pass_frame *pass, uint32_t block_count, > - uint64_t lba, uint8_t byte2, int readop) > +/* > + * mostly copied from cam/scsi/scsi_all.c:scsi_read_write > + */ > + > +int > +mfi_build_cdb(int readop, uint8_t byte2, u_int64_t lba, u_int32_t block_count, uint8_t *cdb) > { > int cdb_len; > > @@ -1997,7 +2029,7 @@ > /* We can fit in a 6 byte cdb */ > struct scsi_rw_6 *scsi_cmd; > > - scsi_cmd = (struct scsi_rw_6 *)&pass->cdb; > + scsi_cmd = (struct scsi_rw_6 *)cdb; > scsi_cmd->opcode = readop ? READ_6 : WRITE_6; > scsi_ulto3b(lba, scsi_cmd->addr); > scsi_cmd->length = block_count & 0xff; > @@ -2007,7 +2039,7 @@ > /* Need a 10 byte CDB */ > struct scsi_rw_10 *scsi_cmd; > > - scsi_cmd = (struct scsi_rw_10 *)&pass->cdb; > + scsi_cmd = (struct scsi_rw_10 *)cdb; > scsi_cmd->opcode = readop ? READ_10 : WRITE_10; > scsi_cmd->byte2 = byte2; > scsi_ulto4b(lba, scsi_cmd->addr); > @@ -2020,7 +2052,7 @@ > /* Block count is too big for 10 byte CDB use a 12 byte CDB */ > struct scsi_rw_12 *scsi_cmd; > > - scsi_cmd = (struct scsi_rw_12 *)&pass->cdb; > + scsi_cmd = (struct scsi_rw_12 *)cdb; > scsi_cmd->opcode = readop ? READ_12 : WRITE_12; > scsi_cmd->byte2 = byte2; > scsi_ulto4b(lba, scsi_cmd->addr); > @@ -2035,7 +2067,7 @@ > */ > struct scsi_rw_16 *scsi_cmd; > > - scsi_cmd = (struct scsi_rw_16 *)&pass->cdb; > + scsi_cmd = (struct scsi_rw_16 *)cdb; > scsi_cmd->opcode = readop ? READ_16 : WRITE_16; > scsi_cmd->byte2 = byte2; > scsi_u64to8b(lba, scsi_cmd->addr); > @@ -2053,15 +2085,15 @@ > { > struct mfi_command *cm; > struct mfi_pass_frame *pass; > - int flags = 0; > + uint32_t context = 0; > + int flags = 0, blkcount = 0, readop; > uint8_t cdb_len; > - uint32_t block_count, context = 0; > > if ((cm = mfi_dequeue_free(sc)) == NULL) > return (NULL); > > /* Zero out the MFI frame */ > - context = cm->cm_frame->header.context; > + context = cm->cm_frame->header.context; > bzero(cm->cm_frame, sizeof(union mfi_frame)); > cm->cm_frame->header.context = context; > pass = &cm->cm_frame->pass; > @@ -2070,22 +2102,24 @@ > switch (bio->bio_cmd & 0x03) { > case BIO_READ: > flags = MFI_CMD_DATAIN; > + readop = 1; > break; > case BIO_WRITE: > flags = MFI_CMD_DATAOUT; > + readop = 0; > break; > default: > /* TODO: what about BIO_DELETE??? */ > - panic("Unsupported bio command"); > + panic("Unsupported bio command %x\n", bio->bio_cmd); > } > > /* Cheat with the sector length to avoid a non-constant division */ > - block_count = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; > + blkcount = (bio->bio_bcount + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; > /* Fill the LBA and Transfer length in CDB */ > - cdb_len = mfi_build_syspd_cdb(pass, block_count, bio->bio_pblkno, 0, > - flags == MFI_CMD_DATAIN); > - > + cdb_len = mfi_build_cdb(readop, 0, bio->bio_pblkno, blkcount, > + pass->cdb); > pass->header.target_id = (uintptr_t)bio->bio_driver1; > + pass->header.lun_id = 0; > pass->header.timeout = 0; > pass->header.flags = 0; > pass->header.scsi_status = 0; > @@ -2132,7 +2166,7 @@ > break; > default: > /* TODO: what about BIO_DELETE??? */ > - panic("Unsupported bio command"); > + panic("Unsupported bio command %x\n", bio->bio_cmd); > } > > /* Cheat with the sector length to avoid a non-constant division */ > @@ -2422,15 +2456,14 @@ > } > > static int > -mfi_abort(struct mfi_softc *sc, struct mfi_command *cm_abort) > +mfi_abort(struct mfi_softc *sc, struct mfi_command **cm_abort) > { > struct mfi_command *cm; > struct mfi_abort_frame *abort; > int i = 0; > uint32_t context = 0; > > - mtx_assert(&sc->mfi_io_lock, MA_OWNED); > - > + mtx_lock(&sc->mfi_io_lock); > if ((cm = mfi_dequeue_free(sc)) == NULL) { > return (EBUSY); > } > @@ -2444,29 +2477,27 @@ > abort->header.cmd = MFI_CMD_ABORT; > abort->header.flags = 0; > abort->header.scsi_status = 0; > - abort->abort_context = cm_abort->cm_frame->header.context; > - abort->abort_mfi_addr_lo = (uint32_t)cm_abort->cm_frame_busaddr; > + abort->abort_context = (*cm_abort)->cm_frame->header.context; > + abort->abort_mfi_addr_lo = (uint32_t)(*cm_abort)->cm_frame_busaddr; > abort->abort_mfi_addr_hi = > - (uint32_t)((uint64_t)cm_abort->cm_frame_busaddr >> 32); > + (uint32_t)((uint64_t)(*cm_abort)->cm_frame_busaddr >> 32); > cm->cm_data = NULL; > cm->cm_flags = MFI_CMD_POLLED; > > - if (sc->mfi_aen_cm) > - sc->cm_aen_abort = 1; > - if (sc->mfi_map_sync_cm) > - sc->cm_map_abort = 1; > mfi_mapcmd(sc, cm); > mfi_release_command(cm); > > - while (i < 5 && sc->mfi_aen_cm != NULL) { > - msleep(&sc->mfi_aen_cm, &sc->mfi_io_lock, 0, "mfiabort", > + mtx_unlock(&sc->mfi_io_lock); > + while (i < 5 && *cm_abort != NULL) { > + tsleep(cm_abort, 0, "mfiabort", > 5 * hz); > i++; > } > - while (i < 5 && sc->mfi_map_sync_cm != NULL) { > - msleep(&sc->mfi_map_sync_cm, &sc->mfi_io_lock, 0, "mfiabort", > - 5 * hz); > - i++; > + if (*cm_abort != NULL) { > + /* Force a complete if command didn't abort */ > + mtx_lock(&sc->mfi_io_lock); > + (*cm_abort)->cm_complete(*cm_abort); > + mtx_unlock(&sc->mfi_io_lock); > } > > return (0); > @@ -2522,7 +2553,7 @@ > { > struct mfi_command *cm; > struct mfi_pass_frame *pass; > - int error; > + int error, readop, cdb_len; > uint32_t blkcount; > > if ((cm = mfi_dequeue_free(sc)) == NULL) > @@ -2531,21 +2562,24 @@ > pass = &cm->cm_frame->pass; > bzero(pass->cdb, 16); > pass->header.cmd = MFI_CMD_PD_SCSI_IO; > + > + readop = 0; > blkcount = (len + MFI_SECTOR_LEN - 1) / MFI_SECTOR_LEN; > + cdb_len = mfi_build_cdb(readop, 0, lba, blkcount, pass->cdb); > pass->header.target_id = id; > pass->header.timeout = 0; > pass->header.flags = 0; > pass->header.scsi_status = 0; > pass->header.sense_len = MFI_SENSE_LEN; > pass->header.data_len = len; > - pass->header.cdb_len = mfi_build_syspd_cdb(pass, blkcount, lba, 0, 0); > + pass->header.cdb_len = cdb_len; > pass->sense_addr_lo = (uint32_t)cm->cm_sense_busaddr; > pass->sense_addr_hi = (uint32_t)((uint64_t)cm->cm_sense_busaddr >> 32); > cm->cm_data = virt; > cm->cm_len = len; > cm->cm_sg = &pass->sgl; > cm->cm_total_frame_size = MFI_PASS_FRAME_SIZE; > - cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT; > + cm->cm_flags = MFI_CMD_POLLED | MFI_CMD_DATAOUT | MFI_CMD_SCSI; > > error = mfi_mapcmd(sc, cm); > bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap, > @@ -2745,16 +2779,24 @@ > } > } > > -static int mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) > +static int > +mfi_check_for_sscd(struct mfi_softc *sc, struct mfi_command *cm) > { > - struct mfi_config_data *conf_data=(struct mfi_config_data *)cm->cm_data; > + struct mfi_config_data *conf_data; > struct mfi_command *ld_cm = NULL; > struct mfi_ld_info *ld_info = NULL; > + struct mfi_ld_config *ld; > + char *p; > int error = 0; > > - if ((cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) && > - (conf_data->ld[0].params.isSSCD == 1)) { > - error = 1; > + conf_data = (struct mfi_config_data *)cm->cm_data; > + > + if (cm->cm_frame->dcmd.opcode == MFI_DCMD_CFG_ADD) { > + p = (char *)conf_data->array; > + p += conf_data->array_size * conf_data->array_count; > + ld = (struct mfi_ld_config *)p; > + if (ld->params.isSSCD == 1) > + error = 1; > } else if (cm->cm_frame->dcmd.opcode == MFI_DCMD_LD_DELETE) { > error = mfi_dcmd_command (sc, &ld_cm, MFI_DCMD_LD_GET_INFO, > (void **)&ld_info, sizeof(*ld_info)); > Index: mfi_cam.c > =================================================================== > --- mfi_cam.c (revision 242617) > +++ mfi_cam.c (working copy) > @@ -79,6 +79,11 @@ > static struct mfi_command * mfip_start(void *); > static void mfip_done(struct mfi_command *cm); > > +static int mfi_allow_disks = 0; > +TUNABLE_INT("hw.mfi.allow_cam_disk_passthrough", &mfi_allow_disks); > +SYSCTL_INT(_hw_mfi, OID_AUTO, allow_cam_disk_passthrough, CTLFLAG_RD, > + &mfi_allow_disks, 0, "event message locale"); > + > static devclass_t mfip_devclass; > static device_method_t mfip_methods[] = { > DEVMETHOD(device_probe, mfip_probe), > @@ -349,7 +354,8 @@ > command = csio->cdb_io.cdb_bytes[0]; > if (command == INQUIRY) { > device = csio->data_ptr[0] & 0x1f; > - if ((device == T_DIRECT) || (device == T_PROCESSOR)) > + if ((!mfi_allow_disks && device == T_DIRECT) || > + (device == T_PROCESSOR)) > csio->data_ptr[0] = > (csio->data_ptr[0] & 0xe0) | T_NODEVICE; > } > Index: mfi_syspd.c > =================================================================== > --- mfi_syspd.c (revision 242617) > +++ mfi_syspd.c (working copy) > @@ -89,7 +89,6 @@ > static int > mfi_syspd_probe(device_t dev) > { > - > return (0); > } > > @@ -98,12 +97,12 @@ > { > struct mfi_system_pd *sc; > struct mfi_pd_info *pd_info; > + struct mfi_system_pending *syspd_pend; > uint64_t sectors; > uint32_t secsize; > > sc = device_get_softc(dev); > pd_info = device_get_ivars(dev); > - > sc->pd_dev = dev; > sc->pd_id = pd_info->ref.v.device_id; > sc->pd_unit = device_get_unit(dev); > @@ -115,6 +114,13 @@ > secsize = MFI_SECTOR_LEN; > mtx_lock(&sc->pd_controller->mfi_io_lock); > TAILQ_INSERT_TAIL(&sc->pd_controller->mfi_syspd_tqh, sc, pd_link); > + TAILQ_FOREACH(syspd_pend, &sc->pd_controller->mfi_syspd_pend_tqh, > + pd_link) { > + TAILQ_REMOVE(&sc->pd_controller->mfi_syspd_pend_tqh, > + syspd_pend, pd_link); > + free(syspd_pend, M_MFIBUF); > + break; > + } > mtx_unlock(&sc->pd_controller->mfi_io_lock); > device_printf(dev, "%juMB (%ju sectors) SYSPD volume\n", > sectors / (1024 * 1024 / secsize), sectors); > @@ -139,6 +145,7 @@ > disk_create(sc->pd_disk, DISK_VERSION); > > device_printf(dev, " SYSPD volume attached\n"); > + > return (0); > } > > > Thanks, > > Doug A. > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Tue Nov 6 18:01:54 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8550D753; Tue, 6 Nov 2012 18:01:54 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 559B48FC0C; Tue, 6 Nov 2012 18:01:53 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 06 Nov 2012 10:03:11 -0800 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id qA6I1rHx042710; Tue, 6 Nov 2012 10:01:53 -0800 (PST) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id qA6I1qaB042708; Tue, 6 Nov 2012 10:01:52 -0800 (PST) (envelope-from ambrisko) Date: Tue, 6 Nov 2012 10:01:52 -0800 From: Doug Ambrisko To: Steven Hartland Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Message-ID: <20121106180152.GA40422@ambrisko.com> References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> User-Agent: Mutt/1.4.2.3i Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2012 18:01:54 -0000 On Tue, Nov 06, 2012 at 12:09:42AM -0000, Steven Hartland wrote: | Thanks Doug, actually just finished another test run with some more | debugging in and I believe I've found the reason for the non-recusive | lock and at least some of the queuing issues. | | The non-recursive lock is due to the mfi_tbolt_reset calling | mfi_process_fw_state_chg_isr with mfi_io_lock held which in turn calls | mfi_tbolt_init_MFI_queue which tries to acquire mfi_io_lock hence | the problem. | | mfi-lock.txt attached I believe fixes this as well as what appears | to be an invalid call to mtx_unlock(&sc->mfi_io_lock) in mfi_attach | which never acquires the lock as far as can see, possibly a cut and | paste error. I don't seem to see the attachment. | The invalid queue problems seem to stem from the error cases of | the calls to mfi_mapcmd, some of which call mfi_release_command which | blindly sets cm_flags = 0 and then enqueues it on the free queue. Now | depending on the flow of mfi_mapcmd and where the error occurs the | command may or may not have been put on the busy queue which is going | to cause problems. | | Going to investigate this further but that's what my current theory is. | | Your patch seems quite extensive, so if could you give me brief run | down on the changes that would be most appreciated. I'll being doing that in the commit message which should happen today. | FYI, I'm aware that the cause of my underlying issues are some | hardware issues (likely cable or backplane related) but it does mean | I'm in the position to test these usually rare error cases, so wanting | the make the most of it before we get the hardware swapped out. That would be good. It makes it easier to debug things when it shows the problem. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Tue Nov 6 18:26:19 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1EF7BB5C; Tue, 6 Nov 2012 18:26:19 +0000 (UTC) (envelope-from prvs=1657173cfa=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 7BC478FC08; Tue, 6 Nov 2012 18:26:18 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000972147.msg; Tue, 06 Nov 2012 18:26:11 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 06 Nov 2012 18:26:11 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1657173cfa=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <6B5B65F4FC854EB8BBC701500096602E@multiplay.co.uk> From: "Steven Hartland" To: "Doug Ambrisko" References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> <20121106180152.GA40422@ambrisko.com> Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Date: Tue, 6 Nov 2012 18:26:13 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2012 18:26:19 -0000 ----- Original Message ----- From: "Doug Ambrisko" > On Tue, Nov 06, 2012 at 12:09:42AM -0000, Steven Hartland wrote: > | Thanks Doug, actually just finished another test run with some more > | debugging in and I believe I've found the reason for the non-recusive > | lock and at least some of the queuing issues. > | > | The non-recursive lock is due to the mfi_tbolt_reset calling > | mfi_process_fw_state_chg_isr with mfi_io_lock held which in turn calls > | mfi_tbolt_init_MFI_queue which tries to acquire mfi_io_lock hence > | the problem. > | > | mfi-lock.txt attached I believe fixes this as well as what appears > | to be an invalid call to mtx_unlock(&sc->mfi_io_lock) in mfi_attach > | which never acquires the lock as far as can see, possibly a cut and > | paste error. > > I don't seem to see the attachment. Yer seems like some mail fail by me there, but I've had some more locking panics during todays tests anyway, requiring additional fixes. Will update and post when I'm happy with it. > | The invalid queue problems seem to stem from the error cases of > | the calls to mfi_mapcmd, some of which call mfi_release_command which > | blindly sets cm_flags = 0 and then enqueues it on the free queue. Now > | depending on the flow of mfi_mapcmd and where the error occurs the > | command may or may not have been put on the busy queue which is going > | to cause problems. > | > | Going to investigate this further but that's what my current theory is. I think I've pretty much nailed the queuing issues, there's quite a few it seems caused by inconsitent handling of calls to mfi_mapcmd as suspected. My current outstanding issue is that after adapter reset, commands are left in the queue causing constant timeouts. Hopefully this should be relatively easy to track down and fix too. > | Your patch seems quite extensive, so if could you give me brief run > | down on the changes that would be most appreciated. > > I'll being doing that in the commit message which should happen today. Cool look forward to it :) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Wed Nov 7 09:04:37 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C476C94 for ; Wed, 7 Nov 2012 09:04:37 +0000 (UTC) (envelope-from linuxfreaker@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5465C8FC0C for ; Wed, 7 Nov 2012 09:04:37 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id k10so2619243iea.13 for ; Wed, 07 Nov 2012 01:04:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=6HeX1CdqwKudOA4Z3I0Yf9mleH+4YtuZB9DCHENu9EU=; b=CP91q9wywFNwbhR1OiEkGo76AVjCW94fpe/LjAIEfZzXjVsxG4ZllILcTQ6EwddEGW jGwuKD1MQE1ktyatE02Vkyodyb+nosn5snTey+MgJGOtUXycmB0OOWxOahwpZ3wmP0cB 0Ot6EY7tEf6TjHFNcrhXyNHn2INO+Idlgg7mC+pVePxvYc2wdDZMDtbAoUx/qzje0471 frO5obPzw7rmYcQB/j8xxfEWam2Ko5wvpCYfatNkb1xn9qiazH5+oKDcnm9dpP4jq6FV dhiUsl7RvasDQumA0ITKd0rhUe9Q+HlTKK6M3yQsPwyAmqc1JC9Ngga4HSNTaB709kBx hYHA== MIME-Version: 1.0 Received: by 10.42.68.203 with SMTP id y11mr3496051ici.26.1352279077100; Wed, 07 Nov 2012 01:04:37 -0800 (PST) Received: by 10.64.106.39 with HTTP; Wed, 7 Nov 2012 01:04:37 -0800 (PST) Date: Wed, 7 Nov 2012 01:04:37 -0800 Message-ID: Subject: From: linux freaker To: bharat707@gmail.com, araina2@sapient.com, pgap_gtr@bsnl.co.in, ahmed.omar.ali@gmail.com, freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2012 09:04:37 -0000 http://www.kivivorm.ee/test/wordpress/wp-content/themes/redline/ugoogle.html From owner-freebsd-scsi@FreeBSD.ORG Wed Nov 7 14:05:31 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDB0C4EC for ; Wed, 7 Nov 2012 14:05:31 +0000 (UTC) (envelope-from jacks.1785@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id B303C8FC14 for ; Wed, 7 Nov 2012 14:05:30 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id k10so1471028iag.13 for ; Wed, 07 Nov 2012 06:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=UOqpmwB7kMqCjE02LsIrv0KVD/AvGjivVAQ9M3nYhCE=; b=lz8Mf3coJCNbgBE0w2dcMHhGmIR6qaCWKSzZdYBFAPFpg/kWPcF3Z2jcg+FA4H6zAc e+6898iKcFdJ4clPuCWuitzHvw1T951W897vJm74Us4AtTi6KThfWDxlTrJ3B8k97V3i 8HfmjIL8sPQc1z5/bec3hICmSDuA0tNY4IGudHXrWnyLmLSJ36b0MVV41MbMTVgvP1iZ PjLw27N6YT0/UEmcCxi6JCz3ckWmJ5/Uxq3vRewvgiPWmJk4wuku8uHZj02ukUWO6fG8 fLTAuhsEK3Htm1z7g9tYnezYMibxXaXpx6Qq6xMCqu39XM8dHjZKC4zwX8GeXLBXqg41 BDUw== MIME-Version: 1.0 Received: by 10.42.80.207 with SMTP id w15mr4046844ick.40.1352297130335; Wed, 07 Nov 2012 06:05:30 -0800 (PST) Received: by 10.64.11.166 with HTTP; Wed, 7 Nov 2012 06:05:29 -0800 (PST) In-Reply-To: References: Date: Wed, 7 Nov 2012 19:35:29 +0530 Message-ID: Subject: Re: cam probe sequence From: Jack To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2012 14:05:32 -0000 Hi, On Sun, Nov 4, 2012 at 2:40 PM, Jack wrote: > Hello all, > > I'm trying to understand the probe sequence of a scsi device, say > a scsi hard drive( with SPI/SAS/FC interface). > > I'm particularly interested in probe sequence of a device which is > attached to the system _after_ FreeBSD has booted completely. > (ie post-boot probing.) > > While going through the source of cam subsystem, a few > questions arose in my understanding. > > ( The relevant files I found are cam/cam_xpt.c, cam/cam_periph.c > and cam/scsi/scsi_xpt.c or cam/ata/ata_xpt.c in case of (S)ATA > drives. ) > > As far as I could got till now, it seems that struct cam_et and > struct cam_ed are _always_ created even _before_ the probing of > device commences. > > But how does cam sub-system get to know that a device got > attached to the system, so that it creates struct cam_et and > struct cam_ed in advance, and _then_ probes the device. > > Is it something like a event notfication at hardware level that > the HBA driver passes to cam sub-system which notifies cam > sub-system that a device has recently been attached. > > If it is true, then does all SPI/SAS/FC/(S)ATA HBAs support > such event notification -- something like a standard thing > in SPI/SAS/FC/(S)ATA world. > > If it is not so, then does userland have to intiate a scan after > device is attached, so that during scanning the cam-subsystem > get to know that Oh! there is a device we are meeting first time. > In other words, _until_ userland instructs cam subsystem to scan > all the buses by XPT_SCAN_BUS ccb through CAMIOCOMMAND > ioctl of /dev/xpt0, the struct cam_et and struct cam_ed will not > get created. Is it so? > But in this case also, does cam-subsystem creates > struct cam_et and struct cam_ed before probing the device?. > > In case of SPI/SAS/FC hard drives the probe code is in > cam/scsi/scsi_xpt.c While reading this source file it > also seems to me that a struct cam_periph is also created > as soon as struct cam_et and struct cam_ed are created. > This also happens before probing of device commences. > Is it the pass device that is created even before probing begins. > Then after probing and determining the type of device - e.g. disk, > cd-rom, etc, then the appropriate peripheral device(da, cd, etc. ) is > created for the probed device. Is it so? > > > It would be really helpful if someone could clear these things, > specially from the instant the device -say a hard disk; is attached, > to the instant the pass device and da device are created and > probing of this hard disk commences or finishes. > > > > Regards > -- > Jack Going through sources of cam, I still can't figure out the the invocation of probeschedule() and proberegister(). Actually I am trying to figure out the relation of probe device with xpt layer. a) What exactly is the purpose of "probe" device. It is not shown in /dev , so I guess it is for internal use only. Is there only one instance of "probe" device for each instance of xpt ? or, for each instance of struct cam_ed, a probe device is created, or I am missing something. b) Can anyone point me as where and how is probeschedule() or proberegister() got invoked from xptioctl(). ? (For the case of CAMIOCOMMAND and XPT_SCAN_LUN ccb.) I only know that this causes scsi_scan_lun() to be invoked, but then how is probeschedule() or proberegister() got invoked from it. c )It seems to me that there is no way for cam to automatically detect the presence/attachement of a device on some bus. Ofcourse, usb mass and firewire are exceptions, where the device removal event is triggered at hardware level. In case of SPI/SAS/FC/(S)ATA, this is not the case. So, the only way to determine the presence of recently attached device is to instruct cam to rescan the bus/lun. And this instruction is CAMIOCOMMAND ioctl with XPT_SCAN_xxx ccbs. of pass driver. But then why is xpt_rescan() ? I'm not sure what exactly is the purpose of xpt_rescan() . I guess it is intended to be used from within the kernel, as from within the kernel, xptioctl() can't be invoked. Any clarifications or pointers would be really appreciated. Thanks -- Jack From owner-freebsd-scsi@FreeBSD.ORG Thu Nov 8 00:33:15 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D67A6C5D for ; Thu, 8 Nov 2012 00:33:15 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id AB8D88FC1B for ; Thu, 8 Nov 2012 00:33:15 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 07 Nov 2012 16:34:33 -0800 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id qA80XE1j074728; Wed, 7 Nov 2012 16:33:14 -0800 (PST) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id qA80XEik074727; Wed, 7 Nov 2012 16:33:14 -0800 (PST) (envelope-from ambrisko) Date: Wed, 7 Nov 2012 16:33:14 -0800 From: Doug Ambrisko To: Ryan Stone Subject: Re: kern/172864: [mfi] mfip does not notice new drives Message-ID: <20121108003314.GA72834@ambrisko.com> References: <201210190250.q9J2o1cv043937@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 00:33:15 -0000 On Thu, Oct 25, 2012 at 05:18:01PM -0400, Ryan Stone wrote: | On Thu, Oct 18, 2012 at 10:50 PM, Mark Johnston wrote: | > The following reply was made to PR kern/172864; it has been noted by GNATS. | > | > From: Mark Johnston | > To: bug-followup@FreeBSD.org, kpneal@pobox.com | > Cc: | > Subject: Re: kern/172864: [mfi] mfip does not notice new drives | > Date: Thu, 18 Oct 2012 22:48:22 -0400 | > | > This is because mfi(4) currently doesn't notify CAM when it receives a | > drive insertion/removal event from the hardware. I fixed this in the | > tree at work; I'll post it to freebsd-scsi when I have access to it | > again, probably in a couple of months. | > | > FWIW, running "camcontrol rescan all" should have also caused the | > passthrough devices to appear, so that's another workaround. | | I believe that this is the patch that Mark refers to: | http://people.freebsd.org/~rstone/patches/mfi_hotplug.diff This has been checked in now. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Thu Nov 8 00:35:20 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7B5ECCA3; Thu, 8 Nov 2012 00:35:20 +0000 (UTC) (envelope-from prvs=1659aa6059=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 770088FC12; Thu, 8 Nov 2012 00:35:18 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000985388.msg; Thu, 08 Nov 2012 00:35:17 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 08 Nov 2012 00:35:17 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1659aa6059=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <0B4E8AFF9DA04C6EBD2496A8B58F1D67@multiplay.co.uk> From: "Steven Hartland" To: "Doug Ambrisko" References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> <20121106180152.GA40422@ambrisko.com> <6B5B65F4FC854EB8BBC701500096602E@multiplay.co.uk> Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Date: Thu, 8 Nov 2012 00:35:22 -0000 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0112_01CDBD48.EFF94C40" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 00:35:20 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0112_01CDBD48.EFF94C40 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit ----- Original Message ----- From: "Steven Hartland" >> On Tue, Nov 06, 2012 at 12:09:42AM -0000, Steven Hartland wrote: >> | Thanks Doug, actually just finished another test run with some more >> | debugging in and I believe I've found the reason for the non-recusive >> | lock and at least some of the queuing issues. >> | >> | The non-recursive lock is due to the mfi_tbolt_reset calling >> | mfi_process_fw_state_chg_isr with mfi_io_lock held which in turn calls >> | mfi_tbolt_init_MFI_queue which tries to acquire mfi_io_lock hence >> | the problem. >> | >> | mfi-lock.txt attached I believe fixes this as well as what appears >> | to be an invalid call to mtx_unlock(&sc->mfi_io_lock) in mfi_attach >> | which never acquires the lock as far as can see, possibly a cut and >> | paste error. >> >> I don't seem to see the attachment. > > Yer seems like some mail fail by me there, but I've had some more locking > panics during todays tests anyway, requiring additional fixes. Will update > and post when I'm happy with it. OK two patches attached == zz-mfi-lock.patch == Fixes mfi panic on recused on non-recusive mutex MFI I/O lock Removes a mtx_unlock call for mfi_io_lock which is never aquired == zz-mfi-queue.patch == Fixes queuing issues where mfi_release_command blindly sets the cm_flags = 0 without first removing the command from the relavent queue. This was causing panics in the queue functions which check to ensure a command is not on another queue. Also fixed some cases where the error from mfi_mapcmd was lost and where the command was never released / dequeued in error cases. Ensure that all failures to mfi_mapcmd are logged Fixed possible null pointer exception in mfi_aen_setup if mfi_get_log_state failed. Fixed mfi_parse_entries & mfi_aen_setup not returning possible errors Corrected MFI_DUMP_CMDS calls with invalid vars SC vs sc Commands which have timed out now set cm_error to ETIMEDOUT and call mfi_complete which prevents them getting stuck in the busy queue forever. Fixed possible use of NULL pointer in mfi_tbolt_get_cmd Changed output formats to be more easily recognisable when debugging. A few style (9) fixes e.g. braced single line conditions and double blank lines ---------- I've just had another panic, trace below, but it doesn't seem to be related to my changes so I'd appreciate your feedback on them as they are for now. While the lock patch fixes the problems I've seen, its not clear to me why mfi_tbolt_reset is acquiring the lock and hence requiring mfi_process_fw_state_chg_isr to jump through hoops to ensure locking around queue manipulation is done correctly. Given what its doing (resetting the entire adapter) I wouldn't be surprised if it should really be acquiring the config lock. Other things I've noticed / questions * Should mfi_abort sleep even if its call to mfi_mapcmd fails? * Should mfi_get_controller_info really ignore the error from mfi_mapcmd? * Do these controllers not support none 512 byte requests? Currently all syspd requests are done assuming 512 byte sectors which the disk may not be. This will both reduce performance or potentially break totally if the firmware isn't translating it under the surface correctly. Anyway the new panic manually transcribed is:- panic: Bad linx elm 0xffffff0069b0fc0 next->prev != elm ... mfi_tbolt_get_cmd() mfi_build_mpt_pass_thru() mfi_tbolt_build_mpt_cmd() mfi_tbolt_send_frame() bus_dmamap_load() mfi_mapcmd() mfi_startio() mfi_syspd_strategy() g_disk_start() g_io_schedule_down() g_down_proc_body() fork_exit() fork_trampoline() Looks like mfi_cmd_tbolt_tqh has become corrupt some how, but as far as I can tell all manip is done using the TAILQ macros and under mfi_io_lock so its not obvious to me at this time why this is, any ideas? There was an obvious error in mfi_tbolt_get_cmd which is now fixed in the queue patch, where cmd can be used even if queue was empty and TAILQ_FIRST returned NULL, but I can't see this causing this panic. This is running with a debug kernel with:- options WITNESS options INVARIANTS options INVARIANT_SUPPORT options DDB options GDB options PRINTF_BUFR_SIZE=2048 options MFI_DEBUG Unfortunately I've only got this hardware till Friday unfortunately so any ideas would be most appreciated so I can get testing done before then. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. ------=_NextPart_000_0112_01CDBD48.EFF94C40 Content-Type: application/octet-stream; name="zz-mfi-lock.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zz-mfi-lock.patch" Fixes mfi panic on recused on non-recusive mutex MFI I/O lock=0A= =0A= Removes a mtx_unlock call for mfi_io_lock which is never aquired=0A= --- sys/dev/mfi/mfi.c.orig 2012-11-07 14:40:33.960774577 +0000=0A= +++ sys/dev/mfi/mfi.c 2012-11-07 14:50:28.267789676 +0000=0A= @@ -728,10 +728,8 @@=0A= "hook\n");=0A= return (EINVAL);=0A= }=0A= - if ((error =3D mfi_aen_setup(sc, 0), 0) !=3D 0) {=0A= - mtx_unlock(&sc->mfi_io_lock);=0A= + if ((error =3D mfi_aen_setup(sc, 0), 0) !=3D 0)=0A= return (error);=0A= - }=0A= =0A= /*=0A= * Register a shutdown handler.=0A= --- sys/dev/mfi/mfi_tbolt.c.orig 2012-11-07 12:21:56.249116533 +0000=0A= +++ sys/dev/mfi/mfi_tbolt.c 2012-11-07 14:50:28.268789748 +0000=0A= @@ -1194,6 +1194,7 @@=0A= sc->hw_crit_error=3D 1;=0A= return ;=0A= }=0A= + mtx_unlock(&sc->mfi_io_lock);=0A= if ((error =3D mfi_tbolt_init_MFI_queue(sc)) !=3D 0)=0A= return;=0A= =0A= @@ -1225,7 +1226,9 @@=0A= /*=0A= * Initiate AEN (Asynchronous Event Notification)=0A= */=0A= + mtx_unlock(&sc->mfi_io_lock);=0A= mfi_aen_setup(sc, sc->last_seq_num);=0A= + mtx_lock(&sc->mfi_io_lock);=0A= sc->issuepend_done =3D 1;=0A= device_printf(sc->mfi_dev, "second stage of reset "=0A= "complete, FW is ready now.\n");=0A= @@ -1237,7 +1240,6 @@=0A= device_printf(sc->mfi_dev, "mfi_process_fw_state_chg_isr "=0A= "called with unhandled value:%d\n", sc->adpreset);=0A= }=0A= - mtx_unlock(&sc->mfi_io_lock);=0A= }=0A= =0A= /*=0A= ------=_NextPart_000_0112_01CDBD48.EFF94C40 Content-Type: application/octet-stream; name="zz-mfi-queue.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zz-mfi-queue.patch" Fixes queuing issues where mfi_release_command blindly sets the cm_flags = =3D 0=0A= without first removing the command from the relavent queue.=0A= =0A= This was causing panics in the queue functions which check to ensure a = command=0A= is not on another queue.=0A= =0A= Also fixed some cases where the error from mfi_mapcmd was lost and where = the=0A= command was never released / dequeued in error cases.=0A= =0A= Ensure that all failures to mfi_mapcmd are logged=0A= =0A= Fixed possible null pointer exception in mfi_aen_setup if = mfi_get_log_state=0A= failed.=0A= =0A= Fixed mfi_parse_entries & mfi_aen_setup not returning possible errors=0A= =0A= Corrected MFI_DUMP_CMDS calls with invalid vars SC vs sc=0A= =0A= Commands which have timed out now set cm_error to ETIMEDOUT and call=0A= mfi_complete which prevents them getting stuck in the busy queue forever.=0A= =0A= Fixed possible use of NULL pointer in mfi_tbolt_get_cmd=0A= =0A= Changed output formats to be more easily recognisable when debugging.=0A= =0A= A few style (9) fixes e.g. braced single line conditions and double = blank lines=0A= --- sys/dev/mfi/mfi.c.orig 2012-11-07 14:50:28.267789676 +0000=0A= +++ sys/dev/mfi/mfi.c 2012-11-07 14:55:16.768525352 +0000=0A= @@ -837,6 +837,23 @@=0A= cm->cm_sg->sg32[0].addr =3D 0;=0A= }=0A= =0A= + /*=0A= + * Command may be on other queues e.g. busy queue depending on the=0A= + * flow of a previous call to mfi_mapcmd, so ensure its dequeued=0A= + * properly=0A= + */=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cm);=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_READY) !=3D 0)=0A= + mfi_remove_ready(cm);=0A= +=0A= + /* We're not expecting it to be on any other queue but check */=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_MASK) !=3D 0) {=0A= + printf("command %p is still on another queue, flags =3D %#x\n",=0A= + cm, cm->cm_flags);=0A= + panic("command is still on a queue");=0A= + }=0A= +=0A= hdr_data =3D (uint32_t *)cm->cm_frame;=0A= hdr_data[0] =3D 0; /* cmd, sense_len, cmd_status, scsi_status */=0A= hdr_data[1] =3D 0; /* target_id, lun_id, cdb_len, sg_count */=0A= @@ -950,15 +967,12 @@=0A= cm->cm_data =3D NULL;=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= =0A= - if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= device_printf(sc->mfi_dev, "failed to send init command\n");=0A= - mtx_unlock(&sc->mfi_io_lock);=0A= - return (error);=0A= - }=0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= =0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= static int=0A= @@ -1046,27 +1060,26 @@=0A= class_locale.members.evt_class =3D mfi_event_class;=0A= =0A= if (seq_start =3D=3D 0) {=0A= - error =3D mfi_get_log_state(sc, &log_state);=0A= + if((error =3D mfi_get_log_state(sc, &log_state)) !=3D 0)=0A= + goto out;=0A= sc->mfi_boot_seq_num =3D log_state->boot_seq_num;=0A= - if (error) {=0A= - if (log_state)=0A= - free(log_state, M_MFIBUF);=0A= - return (error);=0A= - }=0A= =0A= /*=0A= * Walk through any events that fired since the last=0A= * shutdown.=0A= */=0A= - mfi_parse_entries(sc, log_state->shutdown_seq_num,=0A= - log_state->newest_seq_num);=0A= + if((error =3D mfi_parse_entries(sc, log_state->shutdown_seq_num,=0A= + log_state->newest_seq_num)) !=3D 0)=0A= + goto out;=0A= seq =3D log_state->newest_seq_num;=0A= } else=0A= seq =3D seq_start;=0A= - mfi_aen_register(sc, seq, class_locale.word);=0A= - free(log_state, M_MFIBUF);=0A= + error =3D mfi_aen_register(sc, seq, class_locale.word);=0A= +out:=0A= + if (log_state)=0A= + free(log_state, M_MFIBUF);=0A= =0A= - return 0;=0A= + return (error);=0A= }=0A= =0A= int=0A= @@ -1076,7 +1089,6 @@=0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= cm->cm_complete =3D NULL;=0A= =0A= -=0A= /*=0A= * MegaCli can issue a DCMD of 0. In this case do nothing=0A= * and return 0 to it as status=0A= @@ -1310,9 +1322,8 @@=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= cm->cm_data =3D NULL;=0A= =0A= - if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= device_printf(sc->mfi_dev, "Failed to shutdown controller\n");=0A= - }=0A= =0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -1796,6 +1807,7 @@=0A= mtx_lock(&sc->mfi_io_lock);=0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= + error =3D EIO;=0A= break;=0A= }=0A= mtx_lock(&sc->mfi_io_lock);=0A= @@ -1824,7 +1836,7 @@=0A= }=0A= =0A= free(el, M_MFIBUF);=0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= static int=0A= @@ -1941,11 +1953,12 @@=0A= dcmd->mbox[0]=3Did;=0A= dcmd->header.scsi_status =3D 0;=0A= dcmd->header.pad0 =3D 0;=0A= - if (mfi_mapcmd(sc, cm) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= device_printf(sc->mfi_dev,=0A= "Failed to get physical drive info %d\n", id);=0A= free(pd_info, M_MFIBUF);=0A= - return (0);=0A= + mfi_release_command(cm);=0A= + return (error);=0A= }=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTREAD);=0A= @@ -2211,11 +2224,14 @@=0A= if ((hdr->cmd_status !=3D MFI_STAT_OK) || (hdr->scsi_status !=3D 0)) {=0A= bio->bio_flags |=3D BIO_ERROR;=0A= bio->bio_error =3D EIO;=0A= - device_printf(sc->mfi_dev, "I/O error, status=3D %d "=0A= - "scsi_status=3D %d\n", hdr->cmd_status, hdr->scsi_status);=0A= + device_printf(sc->mfi_dev, "I/O error, status=3D%#x "=0A= + "scsi_status=3D%#x\n", hdr->cmd_status, hdr->scsi_status);=0A= mfi_print_sense(cm->cm_sc, cm->cm_sense);=0A= } else if (cm->cm_error !=3D 0) {=0A= bio->bio_flags |=3D BIO_ERROR;=0A= + bio->bio_error =3D cm->cm_error;=0A= + device_printf(sc->mfi_dev, "I/O error, error=3D%#x\n",=0A= + cm->cm_error);=0A= }=0A= =0A= mfi_release_command(cm);=0A= @@ -2251,6 +2267,9 @@=0A= =0A= /* Send the command to the controller */=0A= if (mfi_mapcmd(sc, cm) !=3D 0) {=0A= + device_printf(sc->mfi_dev, "Failed to startio\n");=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cm);=0A= mfi_requeue_ready(cm);=0A= break;=0A= }=0A= @@ -2374,7 +2393,7 @@=0A= cm->cm_extra_frames =3D (cm->cm_total_frame_size - 1) / MFI_FRAME_SIZE;=0A= =0A= if (sc->MFA_enabled)=0A= - mfi_tbolt_send_frame(sc, cm);=0A= + mfi_tbolt_send_frame(sc, cm);=0A= else=0A= mfi_send_frame(sc, cm);=0A= =0A= @@ -2466,7 +2485,7 @@=0A= {=0A= struct mfi_command *cm;=0A= struct mfi_abort_frame *abort;=0A= - int i =3D 0;=0A= + int i =3D 0, error;=0A= uint32_t context =3D 0;=0A= =0A= mtx_lock(&sc->mfi_io_lock);=0A= @@ -2490,7 +2509,8 @@=0A= cm->cm_data =3D NULL;=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= =0A= - mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed to abort command\n");=0A= mfi_release_command(cm);=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -2506,7 +2526,7 @@=0A= mtx_unlock(&sc->mfi_io_lock);=0A= }=0A= =0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= int=0A= @@ -2544,7 +2564,8 @@=0A= cm->cm_total_frame_size =3D MFI_IO_FRAME_SIZE;=0A= cm->cm_flags =3D MFI_CMD_POLLED | MFI_CMD_DATAOUT;=0A= =0A= - error =3D mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed dump blocks\n");=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTWRITE);=0A= bus_dmamap_unload(sc->mfi_buffer_dmat, cm->cm_dmamap);=0A= @@ -2587,7 +2608,8 @@=0A= cm->cm_total_frame_size =3D MFI_PASS_FRAME_SIZE;=0A= cm->cm_flags =3D MFI_CMD_POLLED | MFI_CMD_DATAOUT | MFI_CMD_SCSI;=0A= =0A= - error =3D mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed dump blocks\n");=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTWRITE);=0A= bus_dmamap_unload(sc->mfi_buffer_dmat, cm->cm_dmamap);=0A= @@ -3643,7 +3665,7 @@=0A= =0A= #if 0=0A= if (timedout)=0A= - MFI_DUMP_CMDS(SC);=0A= + MFI_DUMP_CMDS(sc);=0A= #endif=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -3656,7 +3678,7 @@=0A= mfi_timeout(void *data)=0A= {=0A= struct mfi_softc *sc =3D (struct mfi_softc *)data;=0A= - struct mfi_command *cm;=0A= + struct mfi_command *cm, *tmp;=0A= time_t deadline;=0A= int timedout =3D 0;=0A= =0A= @@ -3669,7 +3691,7 @@=0A= }=0A= }=0A= mtx_lock(&sc->mfi_io_lock);=0A= - TAILQ_FOREACH(cm, &sc->mfi_busy, cm_link) {=0A= + TAILQ_FOREACH_SAFE(cm, &sc->mfi_busy, cm_link, tmp) {=0A= if (sc->mfi_aen_cm =3D=3D cm || sc->mfi_map_sync_cm =3D=3D cm)=0A= continue;=0A= if (cm->cm_timestamp < deadline) {=0A= @@ -3682,6 +3704,13 @@=0A= );=0A= MFI_PRINT_CMD(cm);=0A= MFI_VALIDATE_CMD(sc, cm);=0A= + /*=0A= + * Fail the command instead of leaving it on=0A= + * the queue where it could remain stuck forever=0A= + */=0A= + mfi_remove_busy(cm);=0A= + cm->cm_error =3D ETIMEDOUT;=0A= + mfi_complete(sc, cm);=0A= timedout++;=0A= }=0A= }=0A= @@ -3689,7 +3718,7 @@=0A= =0A= #if 0=0A= if (timedout)=0A= - MFI_DUMP_CMDS(SC);=0A= + MFI_DUMP_CMDS(sc);=0A= #endif=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= --- sys/dev/mfi/mfi_tbolt.c.orig 2012-11-07 23:00:24.542124476 +0000=0A= +++ sys/dev/mfi/mfi_tbolt.c 2012-11-07 23:01:46.848207655 +0000=0A= @@ -162,14 +162,14 @@=0A= while (!( HostDiag & DIAG_WRITE_ENABLE)) {=0A= for (i =3D 0; i < 1000; i++);=0A= HostDiag =3D (uint32_t)MFI_READ4(sc, MFI_HDR);=0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%x, "=0A= - "hostdiag=3D%x\n", retry, HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%d, "=0A= + "hostdiag=3D%#x\n", retry, HostDiag);=0A= =0A= if (retry++ >=3D 100)=0A= return 1;=0A= }=0A= =0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: HostDiag=3D%x\n", = HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: HostDiag=3D%#x\n", = HostDiag);=0A= =0A= MFI_WRITE4(sc, MFI_HDR, (HostDiag | DIAG_RESET_ADAPTER));=0A= =0A= @@ -181,8 +181,8 @@=0A= while (HostDiag & DIAG_RESET_ADAPTER) {=0A= for (i =3D 0; i < 1000; i++) ;=0A= HostDiag =3D (uint32_t)MFI_READ4(sc, MFI_RSR);=0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%x, "=0A= - "hostdiag=3D%x\n", retry, HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%d, "=0A= + "hostdiag=3D%#x\n", retry, HostDiag);=0A= =0A= if (retry++ >=3D 1000)=0A= return 1;=0A= @@ -734,7 +734,8 @@=0A= =0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= =0A= - cmd =3D TAILQ_FIRST(&sc->mfi_cmd_tbolt_tqh);=0A= + if ((cmd =3D TAILQ_FIRST(&sc->mfi_cmd_tbolt_tqh)) =3D=3D NULL)=0A= + return NULL;=0A= TAILQ_REMOVE(&sc->mfi_cmd_tbolt_tqh, cmd, next);=0A= memset((uint8_t *)cmd->sg_frame, 0, MEGASAS_MAX_SZ_CHAIN_FRAME);=0A= memset((uint8_t *)cmd->io_request, 0,=0A= @@ -1119,9 +1120,9 @@=0A= * should be performed on the controller=0A= */=0A= if (cm->retry_for_fw_reset =3D=3D 3) {=0A= - device_printf(sc->mfi_dev, "megaraid_sas: command %d "=0A= - "was tried multiple times during adapter reset"=0A= - "Shutting down the HBA\n", cm->cm_index);=0A= + device_printf(sc->mfi_dev, "megaraid_sas: command %p "=0A= + "index=3D%d was tried multiple times during adapter "=0A= + "reset - Shutting down the HBA\n", cm, cm->cm_index);=0A= mfi_kill_hba(sc);=0A= sc->hw_crit_error =3D 1;=0A= return;=0A= @@ -1137,8 +1138,8 @@=0A= if (cm->cm_frame->dcmd.opcode !=3D=0A= MFI_DCMD_CTRL_EVENT_WAIT) {=0A= device_printf(sc->mfi_dev,=0A= - "APJ ****requeue command %d \n",=0A= - cm->cm_index);=0A= + "APJ ****requeue command %p "=0A= + "index=3D%d\n", cm, cm->cm_index);=0A= mfi_requeue_ready(cm);=0A= }=0A= }=0A= @@ -1357,6 +1358,8 @@=0A= device_printf(sc->mfi_dev, "failed to send map sync\n");=0A= free(ld_sync, M_MFIBUF);=0A= sc->mfi_map_sync_cm =3D NULL;=0A= + if ((cmd->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cmd);=0A= mfi_requeue_ready(cmd);=0A= goto out;=0A= }=0A= ------=_NextPart_000_0112_01CDBD48.EFF94C40-- From owner-freebsd-scsi@FreeBSD.ORG Thu Nov 8 00:52:21 2012 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0E19225C; Thu, 8 Nov 2012 00:52:21 +0000 (UTC) (envelope-from ambrisko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id D0AB18FC0C; Thu, 8 Nov 2012 00:52:20 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qA80qKCh088662; Thu, 8 Nov 2012 00:52:20 GMT (envelope-from ambrisko@freefall.freebsd.org) Received: (from ambrisko@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qA80qKr1088658; Thu, 8 Nov 2012 00:52:20 GMT (envelope-from ambrisko) Date: Thu, 8 Nov 2012 00:52:20 GMT Message-Id: <201211080052.qA80qKr1088658@freefall.freebsd.org> To: kpneal@pobox.com, ambrisko@FreeBSD.org, freebsd-scsi@FreeBSD.org From: ambrisko@FreeBSD.org Subject: Re: kern/172864: [mfi] mfip does not notice new drives X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 00:52:21 -0000 Synopsis: [mfi] mfip does not notice new drives State-Changed-From-To: open->closed State-Changed-By: ambrisko State-Changed-When: Thu Nov 8 00:51:09 UTC 2012 State-Changed-Why: Patch http://people.freebsd.org/~rstone/patches/mfi_hotplug.diff commited. http://www.freebsd.org/cgi/query-pr.cgi?pr=172864 From owner-freebsd-scsi@FreeBSD.ORG Thu Nov 8 00:54:15 2012 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 27FB52A5; Thu, 8 Nov 2012 00:54:15 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E6B628FC08; Thu, 8 Nov 2012 00:54:14 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qA80sEGp088840; Thu, 8 Nov 2012 00:54:14 GMT (envelope-from eadler@freefall.freebsd.org) Received: (from eadler@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qA80sE0h088836; Thu, 8 Nov 2012 00:54:14 GMT (envelope-from eadler) Date: Thu, 8 Nov 2012 00:54:14 GMT Message-Id: <201211080054.qA80sE0h088836@freefall.freebsd.org> To: kpneal@pobox.com, eadler@FreeBSD.org, freebsd-scsi@FreeBSD.org, ambrisko@FreeBSD.org From: eadler@FreeBSD.org Subject: Re: kern/172864: [mfi] mfip does not notice new drives X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 00:54:15 -0000 Synopsis: [mfi] mfip does not notice new drives State-Changed-From-To: closed->patched State-Changed-By: eadler State-Changed-When: Thu Nov 8 00:54:12 UTC 2012 State-Changed-Why: committed, but not MFCed Responsible-Changed-From-To: freebsd-scsi->ambrisko Responsible-Changed-By: eadler Responsible-Changed-When: Thu Nov 8 00:54:12 UTC 2012 Responsible-Changed-Why: committed, but not MFCed http://www.freebsd.org/cgi/query-pr.cgi?pr=172864 From owner-freebsd-scsi@FreeBSD.ORG Fri Nov 9 14:18:14 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B6322FAF; Fri, 9 Nov 2012 14:18:14 +0000 (UTC) (envelope-from prvs=1660defc6c=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 1D4048FC15; Fri, 9 Nov 2012 14:18:13 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50001000534.msg; Fri, 09 Nov 2012 14:18:10 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 09 Nov 2012 14:18:10 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1660defc6c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <93175830FB964224A4A0B2C4A6C662B3@multiplay.co.uk> From: "Steven Hartland" To: , "John Baldwin" Subject: Re: Changing mfi max_cmds for new tbolt based cards? Date: Fri, 9 Nov 2012 14:18:17 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 14:18:14 -0000 I've been doing a lot of work over the past few weeks find and fixing bugs in the mfi driver. One of these fixes was to ensure that mfi_max_cmds is used consistently throughout the driver to ensure validation checks work as expected. During this I noticed that mfi_max_cmds is very low for current generation controllers. In my case 2208 based controller reports 1008 cmd slots, so quite a jump from the previous max which I believe was 256? I was wondering if we should look to change the how our sysctl is setup? Recently John you made the following comment: > Mess with mfi_max_cmds at your own risk. The limit was added to work around > broken mfi(4) firmware revisions that would lock up when the entire command > queue (256) was used. Just a suggestion to be cautious. It is probably > safe to use more than 128, but I would be wary of using all of the slots on > your adapter. (A verbose boost will show you the number of command slots your > firmware supports.) > http://lists.freebsd.org/pipermail/freebsd-current/2012-September/036639.html Do you have an information about this specifically:- 1. Which generation of controller? 2. Which controller firmware was effected? 3. Was there a reproducible scenario which triggered this behaviour? I'm going to test stability here but my current thinking is to change the default to -1 (controller limit) for new tbolt based controllers. N.B. Resent from correct list account, sorry for the duplicate John. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Fri Nov 9 17:06:04 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 185B7C06; Fri, 9 Nov 2012 17:06:04 +0000 (UTC) (envelope-from prvs=1660defc6c=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C5BB18FC12; Fri, 9 Nov 2012 17:06:02 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50001002154.msg; Fri, 09 Nov 2012 17:05:59 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 09 Nov 2012 17:05:59 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1660defc6c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Steven Hartland" , "Doug Ambrisko" References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> <20121106180152.GA40422@ambrisko.com> <6B5B65F4FC854EB8BBC701500096602E@multiplay.co.uk> <0B4E8AFF9DA04C6EBD2496A8B58F1D67@multiplay.co.uk> Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Date: Fri, 9 Nov 2012 17:06:03 -0000 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0276_01CDBE9C.80113780" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 17:06:04 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0276_01CDBE9C.80113780 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit ----- Original Message ----- From: "Steven Hartland" ... > I've just had another panic, trace below, but it doesn't seem to be related > to my changes so I'd appreciate your feedback on them as they are for now. > > While the lock patch fixes the problems I've seen, its not clear to me > why mfi_tbolt_reset is acquiring the lock and hence requiring > mfi_process_fw_state_chg_isr to jump through hoops to ensure locking > around queue manipulation is done correctly. Given what its doing > (resetting the entire adapter) I wouldn't be surprised if it should > really be acquiring the config lock. > > Other things I've noticed / questions > * Should mfi_abort sleep even if its call to mfi_mapcmd fails? > * Should mfi_get_controller_info really ignore the error from mfi_mapcmd? > * Do these controllers not support none 512 byte requests? Currently > all syspd requests are done assuming 512 byte sectors which the disk may > not be. This will both reduce performance or potentially break totally > if the firmware isn't translating it under the surface correctly. > > Anyway the new panic manually transcribed is:- > panic: Bad linx elm 0xffffff0069b0fc0 next->prev != elm > ... > mfi_tbolt_get_cmd() > mfi_build_mpt_pass_thru() > mfi_tbolt_build_mpt_cmd() > mfi_tbolt_send_frame() > bus_dmamap_load() > mfi_mapcmd() > mfi_startio() > mfi_syspd_strategy() > g_disk_start() > g_io_schedule_down() > g_down_proc_body() > fork_exit() > fork_trampoline() > > Looks like mfi_cmd_tbolt_tqh has become corrupt some how, but as far as I > can tell all manip is done using the TAILQ macros and under mfi_io_lock > so its not obvious to me at this time why this is, any ideas? I've gone through looking for the possible cause of this and while there's nothing directly connected to the manip of this queue I've found and fixed quite a large number of additional problems which may have been indirectly causing this problem. The biggest change is to use mfi_max_cmds to limit the value stored in sc->mfi_max_fw_cmds as this is used extensively throughout the driver for allocation and range checks so having this inconsitently set opened up a large number of possible overrun errors. The new patch attached documents all the changes in detail. I've managed to do one test run so far which failed to reproduce any panics, so definitely moving in the right direction :) The machine has now been collected for repair by the supplier but I'm going to try and get them to put it online for more testing over the weekend. Given the failure rate so far if I can do another 4 runs with no panics I'd be happy that the majority of error conditions are working as expected. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. ------=_NextPart_000_0276_01CDBE9C.80113780 Content-Type: application/octet-stream; name="zz-mfi-queue.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zz-mfi-queue.patch" Fixes queuing issues where mfi_release_command blindly sets the cm_flags = =3D 0=0A= without first removing the command from the relavent queue.=0A= =0A= This was causing panics in the queue functions which check to ensure a = command=0A= is not on another queue.=0A= =0A= Fixed some cases where the error from mfi_mapcmd was lost and where the = command=0A= was never released / dequeued in error cases.=0A= =0A= Ensure that all failures to mfi_mapcmd are logged.=0A= =0A= Fixed possible null pointer exception in mfi_aen_setup if = mfi_get_log_state=0A= failed.=0A= =0A= Fixed mfi_parse_entries & mfi_aen_setup not returning possible errors.=0A= =0A= Corrected MFI_DUMP_CMDS calls with invalid vars SC vs sc.=0A= =0A= Commands which have timed out now set cm_error to ETIMEDOUT and call=0A= mfi_complete which prevents them getting stuck in the busy queue forever.=0A= =0A= Fixed possible use of NULL pointer in mfi_tbolt_get_cmd.=0A= =0A= Changed output formats to be more easily recognisable when debugging.=0A= =0A= Optimised mfi_cmd_pool_tbolt cleanup.=0A= =0A= Made information about driver limiting commands always display as for = modern=0A= cards this can be severe.=0A= =0A= Fixed mfi_tbolt_alloc_cmd out of memory case which previously didnt = return an=0A= error.=0A= =0A= Added malloc checks for request_desc_pool including free when subsiquent = errors=0A= are detected.=0A= =0A= Fixed overflow error in SIMD reply descriptor check.=0A= =0A= Fixed tbolt_cmd leak in mfi_build_and_issue_cmd if there's an error = during IO=0A= build.=0A= =0A= Elimintated double checks on sc->mfi_aen_cm & sc->mfi_map_sync_cm in=0A= mfi_shutdown.=0A= =0A= Move local hdr calculation after error check in mfi_aen_complete.=0A= =0A= Fixed wakeup on NULL in mfi_aen_complete.=0A= =0A= Fixed mfi_aen_cm cleanup in mfi_process_fw_state_chg_isr not checking if = it was=0A= NULL.=0A= =0A= Changed mfi_alloc_commands to error if bus_dmamap_create fails. = Previously we=0A= would try to continue with the number of allocated commands but lots of = places=0A= in the driver assume sc->mfi_max_fw_cmds is whats available so its = unsafe to do=0A= this without lots of changes.=0A= =0A= Removed mfi_total_cmds as its no longer used due the above change.=0A= =0A= Corrected mfi_tbolt_alloc_cmd to return ENOMEM where appropriate.=0A= =0A= Fixed timeouts actually firing at double what they should.=0A= =0A= Setting hw.mfi.max_cmds=3D-1 now configures to use the controller max.=0A= =0A= A few style (9) fixes e.g. braced single line conditions and double = blank lines=0A= --- sys/dev/mfi/mfi.c.orig 2012-11-07 23:00:24.540112877 +0000=0A= +++ sys/dev/mfi/mfi.c 2012-11-09 14:10:12.288110349 +0000=0A= @@ -143,7 +143,7 @@=0A= static int mfi_max_cmds =3D 128;=0A= TUNABLE_INT("hw.mfi.max_cmds", &mfi_max_cmds);=0A= SYSCTL_INT(_hw_mfi, OID_AUTO, max_cmds, CTLFLAG_RD, &mfi_max_cmds,=0A= - 0, "Max commands");=0A= + 0, "Max commands limit (-1 =3D controller limit)");=0A= =0A= static int mfi_detect_jbod_change =3D 1;=0A= TUNABLE_INT("hw.mfi.detect_jbod_change", &mfi_detect_jbod_change);=0A= @@ -366,7 +366,7 @@=0A= {=0A= uint32_t status;=0A= int error, commsz, framessz, sensesz;=0A= - int frames, unit, max_fw_sge;=0A= + int frames, unit, max_fw_sge, max_fw_cmds;=0A= uint32_t tb_mem_size =3D 0;=0A= =0A= if (sc =3D=3D NULL)=0A= @@ -461,7 +461,13 @@=0A= * instead of compile time.=0A= */=0A= status =3D sc->mfi_read_fw_status(sc);=0A= - sc->mfi_max_fw_cmds =3D status & MFI_FWSTATE_MAXCMD_MASK;=0A= + max_fw_cmds =3D status & MFI_FWSTATE_MAXCMD_MASK;=0A= + if (mfi_max_cmds > 0 && mfi_max_cmds < max_fw_cmds) {=0A= + device_printf(sc->mfi_dev, "FW MaxCmds =3D %d, limiting to %d\n",=0A= + max_fw_cmds, mfi_max_cmds);=0A= + sc->mfi_max_fw_cmds =3D mfi_max_cmds;=0A= + } else=0A= + sc->mfi_max_fw_cmds =3D max_fw_cmds;=0A= max_fw_sge =3D (status & MFI_FWSTATE_MAXSGL_MASK) >> 16;=0A= sc->mfi_max_sge =3D min(max_fw_sge, ((MFI_MAXPHYS / PAGE_SIZE) + 1));=0A= =0A= @@ -469,7 +475,8 @@=0A= =0A= if (sc->mfi_flags & MFI_FLAGS_TBOLT) {=0A= mfi_tbolt_init_globals(sc);=0A= - device_printf(sc->mfi_dev, "MaxCmd =3D %x MaxSgl =3D %x state =3D %x = \n",=0A= + device_printf(sc->mfi_dev, "MaxCmd =3D %d, Drv MaxCmd =3D %d, "=0A= + "MaxSgl =3D %d, state =3D %#x\n", max_fw_cmds,=0A= sc->mfi_max_fw_cmds, sc->mfi_max_sge, status);=0A= tb_mem_size =3D mfi_tbolt_get_memory_requirement(sc);=0A= =0A= @@ -779,21 +786,16 @@=0A= mfi_alloc_commands(struct mfi_softc *sc)=0A= {=0A= struct mfi_command *cm;=0A= - int i, ncmds;=0A= + int i, j;=0A= =0A= /*=0A= * XXX Should we allocate all the commands up front, or allocate on=0A= * demand later like 'aac' does?=0A= */=0A= - ncmds =3D MIN(mfi_max_cmds, sc->mfi_max_fw_cmds);=0A= - if (bootverbose)=0A= - device_printf(sc->mfi_dev, "Max fw cmds=3D %d, sizing driver "=0A= - "pool to %d\n", sc->mfi_max_fw_cmds, ncmds);=0A= -=0A= - sc->mfi_commands =3D malloc(sizeof(struct mfi_command) * ncmds, = M_MFIBUF,=0A= - M_WAITOK | M_ZERO);=0A= + sc->mfi_commands =3D malloc(sizeof(struct mfi_command) *=0A= + sc->mfi_max_fw_cmds, M_MFIBUF, M_WAITOK | M_ZERO);=0A= =0A= - for (i =3D 0; i < ncmds; i++) {=0A= + for (i =3D 0; i < sc->mfi_max_fw_cmds; i++) {=0A= cm =3D &sc->mfi_commands[i];=0A= cm->cm_frame =3D (union mfi_frame *)((uintptr_t)sc->mfi_frames +=0A= sc->mfi_cmd_size * i);=0A= @@ -809,10 +811,20 @@=0A= mtx_lock(&sc->mfi_io_lock);=0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= + } else {=0A= + device_printf(sc->mfi_dev, "Failed to allocate %d "=0A= + "command blocks, only allocated %d\n",=0A= + sc->mfi_max_fw_cmds, i - 1);=0A= + for (j =3D 0; j < i; j++) {=0A= + cm =3D &sc->mfi_commands[i];=0A= + bus_dmamap_destroy(sc->mfi_buffer_dmat,=0A= + cm->cm_dmamap);=0A= + }=0A= + free(sc->mfi_commands, M_MFIBUF);=0A= + sc->mfi_commands =3D NULL;=0A= + =0A= + return (ENOMEM);=0A= }=0A= - else=0A= - break;=0A= - sc->mfi_total_cmds++;=0A= }=0A= =0A= return (0);=0A= @@ -837,6 +849,23 @@=0A= cm->cm_sg->sg32[0].addr =3D 0;=0A= }=0A= =0A= + /*=0A= + * Command may be on other queues e.g. busy queue depending on the=0A= + * flow of a previous call to mfi_mapcmd, so ensure its dequeued=0A= + * properly=0A= + */=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cm);=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_READY) !=3D 0)=0A= + mfi_remove_ready(cm);=0A= +=0A= + /* We're not expecting it to be on any other queue but check */=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_MASK) !=3D 0) {=0A= + printf("command %p is still on another queue, flags =3D %#x\n",=0A= + cm, cm->cm_flags);=0A= + panic("command is still on a queue");=0A= + }=0A= +=0A= hdr_data =3D (uint32_t *)cm->cm_frame;=0A= hdr_data[0] =3D 0; /* cmd, sense_len, cmd_status, scsi_status */=0A= hdr_data[1] =3D 0; /* target_id, lun_id, cdb_len, sg_count */=0A= @@ -950,15 +979,12 @@=0A= cm->cm_data =3D NULL;=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= =0A= - if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= device_printf(sc->mfi_dev, "failed to send init command\n");=0A= - mtx_unlock(&sc->mfi_io_lock);=0A= - return (error);=0A= - }=0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= =0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= static int=0A= @@ -1046,27 +1072,26 @@=0A= class_locale.members.evt_class =3D mfi_event_class;=0A= =0A= if (seq_start =3D=3D 0) {=0A= - error =3D mfi_get_log_state(sc, &log_state);=0A= + if((error =3D mfi_get_log_state(sc, &log_state)) !=3D 0)=0A= + goto out;=0A= sc->mfi_boot_seq_num =3D log_state->boot_seq_num;=0A= - if (error) {=0A= - if (log_state)=0A= - free(log_state, M_MFIBUF);=0A= - return (error);=0A= - }=0A= =0A= /*=0A= * Walk through any events that fired since the last=0A= * shutdown.=0A= */=0A= - mfi_parse_entries(sc, log_state->shutdown_seq_num,=0A= - log_state->newest_seq_num);=0A= + if((error =3D mfi_parse_entries(sc, log_state->shutdown_seq_num,=0A= + log_state->newest_seq_num)) !=3D 0)=0A= + goto out;=0A= seq =3D log_state->newest_seq_num;=0A= } else=0A= seq =3D seq_start;=0A= - mfi_aen_register(sc, seq, class_locale.word);=0A= - free(log_state, M_MFIBUF);=0A= + error =3D mfi_aen_register(sc, seq, class_locale.word);=0A= +out:=0A= + if (log_state)=0A= + free(log_state, M_MFIBUF);=0A= =0A= - return 0;=0A= + return (error);=0A= }=0A= =0A= int=0A= @@ -1076,7 +1101,6 @@=0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= cm->cm_complete =3D NULL;=0A= =0A= -=0A= /*=0A= * MegaCli can issue a DCMD of 0. In this case do nothing=0A= * and return 0 to it as status=0A= @@ -1104,12 +1128,13 @@=0A= if (sc->mfi_cdev !=3D NULL)=0A= destroy_dev(sc->mfi_cdev);=0A= =0A= - if (sc->mfi_total_cmds !=3D 0) {=0A= - for (i =3D 0; i < sc->mfi_total_cmds; i++) {=0A= + if (sc->mfi_commands !=3D NULL) {=0A= + for (i =3D 0; i < sc->mfi_max_fw_cmds; i++) {=0A= cm =3D &sc->mfi_commands[i];=0A= bus_dmamap_destroy(sc->mfi_buffer_dmat, cm->cm_dmamap);=0A= }=0A= free(sc->mfi_commands, M_MFIBUF);=0A= + sc->mfi_commands =3D NULL;=0A= }=0A= =0A= if (sc->mfi_intr)=0A= @@ -1165,7 +1190,8 @@=0A= /* End LSIP200113393 */=0A= /* ThunderBolt INIT packet memory Free */=0A= if (sc->mfi_tb_init_busaddr !=3D 0)=0A= - bus_dmamap_unload(sc->mfi_tb_init_dmat, sc->mfi_tb_init_dmamap);=0A= + bus_dmamap_unload(sc->mfi_tb_init_dmat,=0A= + sc->mfi_tb_init_dmamap);=0A= if (sc->mfi_tb_init !=3D NULL)=0A= bus_dmamem_free(sc->mfi_tb_init_dmat, sc->mfi_tb_init,=0A= sc->mfi_tb_init_dmamap);=0A= @@ -1182,16 +1208,14 @@=0A= sc->mfi_tb_ioc_init_dmamap);=0A= if (sc->mfi_tb_ioc_init_dmat !=3D NULL)=0A= bus_dma_tag_destroy(sc->mfi_tb_ioc_init_dmat);=0A= - for (int i =3D 0; i < sc->mfi_max_fw_cmds; i++) {=0A= - if (sc->mfi_cmd_pool_tbolt !=3D NULL) {=0A= + if (sc->mfi_cmd_pool_tbolt !=3D NULL) {=0A= + for (int i =3D 0; i < sc->mfi_max_fw_cmds; i++) {=0A= if (sc->mfi_cmd_pool_tbolt[i] !=3D NULL) {=0A= free(sc->mfi_cmd_pool_tbolt[i],=0A= M_MFIBUF);=0A= sc->mfi_cmd_pool_tbolt[i] =3D NULL;=0A= }=0A= }=0A= - }=0A= - if (sc->mfi_cmd_pool_tbolt !=3D NULL) {=0A= free(sc->mfi_cmd_pool_tbolt, M_MFIBUF);=0A= sc->mfi_cmd_pool_tbolt =3D NULL;=0A= }=0A= @@ -1256,9 +1280,8 @@=0A= cm->cm_error =3D 0;=0A= mfi_complete(sc, cm);=0A= }=0A= - if (++ci =3D=3D (sc->mfi_max_fw_cmds + 1)) {=0A= + if (++ci =3D=3D (sc->mfi_max_fw_cmds + 1))=0A= ci =3D 0;=0A= - }=0A= }=0A= =0A= sc->mfi_comms->hw_ci =3D ci;=0A= @@ -1288,15 +1311,15 @@=0A= int error;=0A= =0A= =0A= - if (sc->mfi_aen_cm)=0A= + if (sc->mfi_aen_cm !=3D NULL) {=0A= sc->cm_aen_abort =3D 1;=0A= - if (sc->mfi_aen_cm !=3D NULL)=0A= mfi_abort(sc, &sc->mfi_aen_cm);=0A= + }=0A= =0A= - if (sc->mfi_map_sync_cm)=0A= + if (sc->mfi_map_sync_cm !=3D NULL) {=0A= sc->cm_map_abort =3D 1;=0A= - if (sc->mfi_map_sync_cm !=3D NULL)=0A= mfi_abort(sc, &sc->mfi_map_sync_cm);=0A= + }=0A= =0A= mtx_lock(&sc->mfi_io_lock);=0A= error =3D mfi_dcmd_command(sc, &cm, MFI_DCMD_CTRL_SHUTDOWN, NULL, 0);=0A= @@ -1310,9 +1333,8 @@=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= cm->cm_data =3D NULL;=0A= =0A= - if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= device_printf(sc->mfi_dev, "Failed to shutdown controller\n");=0A= - }=0A= =0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -1683,11 +1705,11 @@=0A= sc =3D cm->cm_sc;=0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= =0A= - hdr =3D &cm->cm_frame->header;=0A= -=0A= if (sc->mfi_aen_cm =3D=3D NULL)=0A= return;=0A= =0A= + hdr =3D &cm->cm_frame->header;=0A= +=0A= if (sc->cm_aen_abort ||=0A= hdr->cmd_status =3D=3D MFI_STAT_INVALID_STATUS) {=0A= sc->cm_aen_abort =3D 0;=0A= @@ -1713,8 +1735,8 @@=0A= }=0A= =0A= free(cm->cm_data, M_MFIBUF);=0A= - sc->mfi_aen_cm =3D NULL;=0A= wakeup(&sc->mfi_aen_cm);=0A= + sc->mfi_aen_cm =3D NULL;=0A= mfi_release_command(cm);=0A= =0A= /* set it up again so the driver can catch more events */=0A= @@ -1796,6 +1818,7 @@=0A= mtx_lock(&sc->mfi_io_lock);=0A= mfi_release_command(cm);=0A= mtx_unlock(&sc->mfi_io_lock);=0A= + error =3D EIO;=0A= break;=0A= }=0A= mtx_lock(&sc->mfi_io_lock);=0A= @@ -1824,7 +1847,7 @@=0A= }=0A= =0A= free(el, M_MFIBUF);=0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= static int=0A= @@ -1941,11 +1964,12 @@=0A= dcmd->mbox[0]=3Did;=0A= dcmd->header.scsi_status =3D 0;=0A= dcmd->header.pad0 =3D 0;=0A= - if (mfi_mapcmd(sc, cm) !=3D 0) {=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0) {=0A= device_printf(sc->mfi_dev,=0A= "Failed to get physical drive info %d\n", id);=0A= free(pd_info, M_MFIBUF);=0A= - return (0);=0A= + mfi_release_command(cm);=0A= + return (error);=0A= }=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTREAD);=0A= @@ -2211,11 +2235,14 @@=0A= if ((hdr->cmd_status !=3D MFI_STAT_OK) || (hdr->scsi_status !=3D 0)) {=0A= bio->bio_flags |=3D BIO_ERROR;=0A= bio->bio_error =3D EIO;=0A= - device_printf(sc->mfi_dev, "I/O error, status=3D %d "=0A= - "scsi_status=3D %d\n", hdr->cmd_status, hdr->scsi_status);=0A= + device_printf(sc->mfi_dev, "I/O error, status=3D%#x "=0A= + "scsi_status=3D%#x\n", hdr->cmd_status, hdr->scsi_status);=0A= mfi_print_sense(cm->cm_sc, cm->cm_sense);=0A= } else if (cm->cm_error !=3D 0) {=0A= bio->bio_flags |=3D BIO_ERROR;=0A= + bio->bio_error =3D cm->cm_error;=0A= + device_printf(sc->mfi_dev, "I/O error, error=3D%#x\n",=0A= + cm->cm_error);=0A= }=0A= =0A= mfi_release_command(cm);=0A= @@ -2251,6 +2278,9 @@=0A= =0A= /* Send the command to the controller */=0A= if (mfi_mapcmd(sc, cm) !=3D 0) {=0A= + device_printf(sc->mfi_dev, "Failed to startio\n");=0A= + if ((cm->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cm);=0A= mfi_requeue_ready(cm);=0A= break;=0A= }=0A= @@ -2374,7 +2404,7 @@=0A= cm->cm_extra_frames =3D (cm->cm_total_frame_size - 1) / MFI_FRAME_SIZE;=0A= =0A= if (sc->MFA_enabled)=0A= - mfi_tbolt_send_frame(sc, cm);=0A= + mfi_tbolt_send_frame(sc, cm);=0A= else=0A= mfi_send_frame(sc, cm);=0A= =0A= @@ -2466,7 +2496,7 @@=0A= {=0A= struct mfi_command *cm;=0A= struct mfi_abort_frame *abort;=0A= - int i =3D 0;=0A= + int i =3D 0, error;=0A= uint32_t context =3D 0;=0A= =0A= mtx_lock(&sc->mfi_io_lock);=0A= @@ -2490,7 +2520,8 @@=0A= cm->cm_data =3D NULL;=0A= cm->cm_flags =3D MFI_CMD_POLLED;=0A= =0A= - mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed to abort command\n");=0A= mfi_release_command(cm);=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -2506,7 +2537,7 @@=0A= mtx_unlock(&sc->mfi_io_lock);=0A= }=0A= =0A= - return (0);=0A= + return (error);=0A= }=0A= =0A= int=0A= @@ -2544,7 +2575,8 @@=0A= cm->cm_total_frame_size =3D MFI_IO_FRAME_SIZE;=0A= cm->cm_flags =3D MFI_CMD_POLLED | MFI_CMD_DATAOUT;=0A= =0A= - error =3D mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed dump blocks\n");=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTWRITE);=0A= bus_dmamap_unload(sc->mfi_buffer_dmat, cm->cm_dmamap);=0A= @@ -2587,7 +2619,8 @@=0A= cm->cm_total_frame_size =3D MFI_PASS_FRAME_SIZE;=0A= cm->cm_flags =3D MFI_CMD_POLLED | MFI_CMD_DATAOUT | MFI_CMD_SCSI;=0A= =0A= - error =3D mfi_mapcmd(sc, cm);=0A= + if ((error =3D mfi_mapcmd(sc, cm)) !=3D 0)=0A= + device_printf(sc->mfi_dev, "failed dump blocks\n");=0A= bus_dmamap_sync(sc->mfi_buffer_dmat, cm->cm_dmamap,=0A= BUS_DMASYNC_POSTWRITE);=0A= bus_dmamap_unload(sc->mfi_buffer_dmat, cm->cm_dmamap);=0A= @@ -3632,7 +3665,7 @@=0A= deadline =3D time_uptime - mfi_cmd_timeout;=0A= mtx_lock(&sc->mfi_io_lock);=0A= TAILQ_FOREACH(cm, &sc->mfi_busy, cm_link) {=0A= - if (cm->cm_timestamp < deadline) {=0A= + if (cm->cm_timestamp <=3D deadline) {=0A= device_printf(sc->mfi_dev,=0A= "COMMAND %p TIMEOUT AFTER %d SECONDS\n",=0A= cm, (int)(time_uptime - cm->cm_timestamp));=0A= @@ -3643,7 +3676,7 @@=0A= =0A= #if 0=0A= if (timedout)=0A= - MFI_DUMP_CMDS(SC);=0A= + MFI_DUMP_CMDS(sc);=0A= #endif=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= @@ -3656,7 +3689,7 @@=0A= mfi_timeout(void *data)=0A= {=0A= struct mfi_softc *sc =3D (struct mfi_softc *)data;=0A= - struct mfi_command *cm;=0A= + struct mfi_command *cm, *tmp;=0A= time_t deadline;=0A= int timedout =3D 0;=0A= =0A= @@ -3669,10 +3702,10 @@=0A= }=0A= }=0A= mtx_lock(&sc->mfi_io_lock);=0A= - TAILQ_FOREACH(cm, &sc->mfi_busy, cm_link) {=0A= + TAILQ_FOREACH_SAFE(cm, &sc->mfi_busy, cm_link, tmp) {=0A= if (sc->mfi_aen_cm =3D=3D cm || sc->mfi_map_sync_cm =3D=3D cm)=0A= continue;=0A= - if (cm->cm_timestamp < deadline) {=0A= + if (cm->cm_timestamp <=3D deadline) {=0A= if (sc->adpreset !=3D 0 && sc->issuepend_done =3D=3D 0) {=0A= cm->cm_timestamp =3D time_uptime;=0A= } else {=0A= @@ -3682,6 +3715,13 @@=0A= );=0A= MFI_PRINT_CMD(cm);=0A= MFI_VALIDATE_CMD(sc, cm);=0A= + /*=0A= + * Fail the command instead of leaving it on=0A= + * the queue where it could remain stuck forever=0A= + */=0A= + mfi_remove_busy(cm);=0A= + cm->cm_error =3D ETIMEDOUT;=0A= + mfi_complete(sc, cm);=0A= timedout++;=0A= }=0A= }=0A= @@ -3689,7 +3729,7 @@=0A= =0A= #if 0=0A= if (timedout)=0A= - MFI_DUMP_CMDS(SC);=0A= + MFI_DUMP_CMDS(sc);=0A= #endif=0A= =0A= mtx_unlock(&sc->mfi_io_lock);=0A= --- sys/dev/mfi/mfi_tbolt.c.orig 2012-11-07 23:00:24.542124476 +0000=0A= +++ sys/dev/mfi/mfi_tbolt.c 2012-11-09 14:16:13.036767564 +0000=0A= @@ -162,14 +162,14 @@=0A= while (!( HostDiag & DIAG_WRITE_ENABLE)) {=0A= for (i =3D 0; i < 1000; i++);=0A= HostDiag =3D (uint32_t)MFI_READ4(sc, MFI_HDR);=0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%x, "=0A= - "hostdiag=3D%x\n", retry, HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%d, "=0A= + "hostdiag=3D%#x\n", retry, HostDiag);=0A= =0A= if (retry++ >=3D 100)=0A= return 1;=0A= }=0A= =0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: HostDiag=3D%x\n", = HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: HostDiag=3D%#x\n", = HostDiag);=0A= =0A= MFI_WRITE4(sc, MFI_HDR, (HostDiag | DIAG_RESET_ADAPTER));=0A= =0A= @@ -181,8 +181,8 @@=0A= while (HostDiag & DIAG_RESET_ADAPTER) {=0A= for (i =3D 0; i < 1000; i++) ;=0A= HostDiag =3D (uint32_t)MFI_READ4(sc, MFI_RSR);=0A= - device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%x, "=0A= - "hostdiag=3D%x\n", retry, HostDiag);=0A= + device_printf(sc->mfi_dev, "ADP_RESET_TBOLT: retry time=3D%d, "=0A= + "hostdiag=3D%#x\n", retry, HostDiag);=0A= =0A= if (retry++ >=3D 1000)=0A= return 1;=0A= @@ -447,13 +447,21 @@=0A= sc->request_desc_pool =3D malloc(sizeof(=0A= union mfi_mpi2_request_descriptor) * sc->mfi_max_fw_cmds,=0A= M_MFIBUF, M_NOWAIT|M_ZERO);=0A= +=0A= + if (sc->request_desc_pool =3D=3D NULL) {=0A= + device_printf(sc->mfi_dev, "Could not alloc "=0A= + "memory for request_desc_pool\n");=0A= + return (ENOMEM);=0A= + }=0A= +=0A= sc->mfi_cmd_pool_tbolt =3D malloc(sizeof(struct mfi_cmd_tbolt*)=0A= * sc->mfi_max_fw_cmds, M_MFIBUF, M_NOWAIT|M_ZERO);=0A= =0A= - if (!sc->mfi_cmd_pool_tbolt) {=0A= - device_printf(sc->mfi_dev, "out of memory. Could not alloc "=0A= - "memory for cmd_list_fusion\n");=0A= - return 1;=0A= + if (sc->mfi_cmd_pool_tbolt =3D=3D NULL) {=0A= + free(sc->request_desc_pool, M_MFIBUF);=0A= + device_printf(sc->mfi_dev, "Could not alloc "=0A= + "memory for cmd_pool_tbolt\n");=0A= + return (ENOMEM);=0A= }=0A= =0A= for (i =3D 0; i < sc->mfi_max_fw_cmds; i++) {=0A= @@ -461,20 +469,24 @@=0A= struct mfi_cmd_tbolt),M_MFIBUF, M_NOWAIT|M_ZERO);=0A= =0A= if (!sc->mfi_cmd_pool_tbolt[i]) {=0A= - device_printf(sc->mfi_dev, "Could not alloc cmd list "=0A= - "fusion\n");=0A= + device_printf(sc->mfi_dev, "Could not alloc "=0A= + "cmd_pool_tbolt entry\n");=0A= =0A= for (j =3D 0; j < i; j++)=0A= free(sc->mfi_cmd_pool_tbolt[j], M_MFIBUF);=0A= =0A= + free(sc->request_desc_pool, M_MFIBUF);=0A= + sc->request_desc_pool =3D NULL;=0A= free(sc->mfi_cmd_pool_tbolt, M_MFIBUF);=0A= sc->mfi_cmd_pool_tbolt =3D NULL;=0A= +=0A= + return (ENOMEM);=0A= }=0A= }=0A= =0A= /*=0A= * The first 256 bytes (SMID 0) is not used. Don't add to the cmd=0A= - *list=0A= + * list=0A= */=0A= io_req_base =3D sc->request_message_pool_align=0A= + MEGASAS_THUNDERBOLT_NEW_MSG_SIZE;=0A= @@ -614,9 +626,16 @@=0A= static inline void=0A= mfi_tbolt_return_cmd(struct mfi_softc *sc, struct mfi_cmd_tbolt *cmd)=0A= {=0A= +/* TODO: remove this debugging */=0A= +struct mfi_cmd_tbolt *tbolt_cmd;=0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= =0A= cmd->sync_cmd_idx =3D sc->mfi_max_fw_cmds;=0A= +/* TODO: remove this debugging */=0A= +TAILQ_FOREACH(tbolt_cmd, &sc->mfi_cmd_tbolt_tqh, next) {=0A= + if (tbolt_cmd =3D=3D cmd)=0A= + panic("returning tbolt cmd=3D%p thats already present", cmd);=0A= +}=0A= TAILQ_INSERT_TAIL(&sc->mfi_cmd_tbolt_tqh, cmd, next);=0A= }=0A= =0A= @@ -652,13 +671,19 @@=0A= /* Read Reply descriptor */=0A= while ((val.u.low !=3D 0xFFFFFFFF) && (val.u.high !=3D 0xFFFFFFFF)) {=0A= smid =3D reply_desc->SMID;=0A= - if (!smid || smid > sc->mfi_max_fw_cmds + 1) {=0A= - device_printf(sc->mfi_dev, "smid is %x. Cannot "=0A= + if (!smid || smid > sc->mfi_max_fw_cmds) {=0A= + device_printf(sc->mfi_dev, "smid is %d. Cannot "=0A= "proceed. Returning \n", smid);=0A= return;=0A= }=0A= =0A= cmd_tbolt =3D sc->mfi_cmd_pool_tbolt[smid - 1];=0A= + if (cmd_tbolt->sync_cmd_idx =3D=3D sc->mfi_max_fw_cmds) {=0A= + device_printf(sc->mfi_dev, "cmd_tbolt %p "=0A= + "has invalid sync_cmd_idx=3D%d - returning\n",=0A= + cmd_tbolt, cmd_tbolt->sync_cmd_idx);=0A= + return;=0A= + }=0A= cmd_mfi =3D &sc->mfi_commands[cmd_tbolt->sync_cmd_idx];=0A= scsi_io_req =3D cmd_tbolt->io_request;=0A= =0A= @@ -666,10 +691,9 @@=0A= extStatus =3D cmd_mfi->cm_frame->dcmd.header.scsi_status;=0A= map_tbolt_cmd_status(cmd_mfi, status, extStatus);=0A= =0A= - if (cmd_mfi->cm_flags & MFI_CMD_SCSI &&=0A= + if ((cmd_mfi->cm_flags & MFI_CMD_SCSI) !=3D 0 &&=0A= (cmd_mfi->cm_flags & MFI_CMD_POLLED) !=3D 0) {=0A= /* polled LD/SYSPD IO command */=0A= - mfi_tbolt_return_cmd(sc, cmd_tbolt);=0A= /* XXX mark okay for now DJA */=0A= cmd_mfi->cm_frame->header.cmd_status =3D MFI_STAT_OK;=0A= } else {=0A= @@ -684,8 +708,8 @@=0A= =0A= /* complete the command */=0A= mfi_complete(sc, cmd_mfi);=0A= - mfi_tbolt_return_cmd(sc, cmd_tbolt);=0A= }=0A= + mfi_tbolt_return_cmd(sc, cmd_tbolt);=0A= =0A= sc->last_reply_idx++;=0A= if (sc->last_reply_idx >=3D sc->mfi_max_fw_cmds) {=0A= @@ -734,7 +758,8 @@=0A= =0A= mtx_assert(&sc->mfi_io_lock, MA_OWNED);=0A= =0A= - cmd =3D TAILQ_FIRST(&sc->mfi_cmd_tbolt_tqh);=0A= + if ((cmd =3D TAILQ_FIRST(&sc->mfi_cmd_tbolt_tqh)) =3D=3D NULL)=0A= + return NULL;=0A= TAILQ_REMOVE(&sc->mfi_cmd_tbolt_tqh, cmd, next);=0A= memset((uint8_t *)cmd->sg_frame, 0, MEGASAS_MAX_SZ_CHAIN_FRAME);=0A= memset((uint8_t *)cmd->io_request, 0,=0A= @@ -988,8 +1013,10 @@=0A= =0A= index =3D cmd->index;=0A= req_desc =3D mfi_tbolt_get_request_descriptor(sc, index-1);=0A= - if (mfi_tbolt_build_io(sc, mfi_cmd, cmd))=0A= + if (mfi_tbolt_build_io(sc, mfi_cmd, cmd) !=3D 0) {=0A= + mfi_tbolt_return_cmd(sc, cmd);=0A= return NULL;=0A= + }=0A= req_desc->header.SMID =3D index;=0A= return req_desc;=0A= }=0A= @@ -1119,9 +1146,9 @@=0A= * should be performed on the controller=0A= */=0A= if (cm->retry_for_fw_reset =3D=3D 3) {=0A= - device_printf(sc->mfi_dev, "megaraid_sas: command %d "=0A= - "was tried multiple times during adapter reset"=0A= - "Shutting down the HBA\n", cm->cm_index);=0A= + device_printf(sc->mfi_dev, "megaraid_sas: command %p "=0A= + "index=3D%d was tried multiple times during adapter "=0A= + "reset - Shutting down the HBA\n", cm, cm->cm_index);=0A= mfi_kill_hba(sc);=0A= sc->hw_crit_error =3D 1;=0A= return;=0A= @@ -1130,15 +1157,14 @@=0A= if ((cm->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0) {=0A= struct mfi_cmd_tbolt *cmd;=0A= mfi_remove_busy(cm);=0A= - cmd =3D sc->mfi_cmd_pool_tbolt[cm->cm_extra_frames -=0A= - 1 ];=0A= + cmd =3D sc->mfi_cmd_pool_tbolt[cm->cm_extra_frames - 1];=0A= mfi_tbolt_return_cmd(sc, cmd);=0A= if ((cm->cm_flags & MFI_ON_MFIQ_MASK) =3D=3D 0) {=0A= if (cm->cm_frame->dcmd.opcode !=3D=0A= MFI_DCMD_CTRL_EVENT_WAIT) {=0A= device_printf(sc->mfi_dev,=0A= - "APJ ****requeue command %d \n",=0A= - cm->cm_index);=0A= + "APJ ****requeue command %p "=0A= + "index=3D%d\n", cm, cm->cm_index);=0A= mfi_requeue_ready(cm);=0A= }=0A= }=0A= @@ -1196,22 +1222,22 @@=0A= }=0A= mtx_unlock(&sc->mfi_io_lock);=0A= if ((error =3D mfi_tbolt_init_MFI_queue(sc)) !=3D 0)=0A= - return;=0A= + return;=0A= =0A= mtx_lock(&sc->mfi_io_lock);=0A= =0A= sc->mfi_enable_intr(sc);=0A= sc->adpreset =3D 0;=0A= - free(sc->mfi_aen_cm->cm_data, M_MFIBUF);=0A= - mfi_remove_busy(sc->mfi_aen_cm);=0A= - cmd =3D sc->mfi_cmd_pool_tbolt[sc->mfi_aen_cm->cm_extra_frames=0A= - - 1];=0A= - mfi_tbolt_return_cmd(sc, cmd);=0A= - if (sc->mfi_aen_cm) {=0A= + if (sc->mfi_aen_cm !=3D NULL) {=0A= + free(sc->mfi_aen_cm->cm_data, M_MFIBUF);=0A= + mfi_remove_busy(sc->mfi_aen_cm);=0A= + cmd =3D sc->mfi_cmd_pool_tbolt[=0A= + sc->mfi_aen_cm->cm_extra_frames - 1];=0A= + mfi_tbolt_return_cmd(sc, cmd);=0A= mfi_release_command(sc->mfi_aen_cm);=0A= sc->mfi_aen_cm =3D NULL;=0A= }=0A= - if (sc->mfi_map_sync_cm) {=0A= + if (sc->mfi_map_sync_cm !=3D NULL) {=0A= mfi_release_command(sc->mfi_map_sync_cm);=0A= sc->mfi_map_sync_cm =3D NULL;=0A= }=0A= @@ -1357,6 +1383,8 @@=0A= device_printf(sc->mfi_dev, "failed to send map sync\n");=0A= free(ld_sync, M_MFIBUF);=0A= sc->mfi_map_sync_cm =3D NULL;=0A= + if ((cmd->cm_flags & MFI_ON_MFIQ_BUSY) !=3D 0)=0A= + mfi_remove_busy(cmd);=0A= mfi_requeue_ready(cmd);=0A= goto out;=0A= }=0A= --- sys/dev/mfi/mfi_debug.c.orig 2012-11-09 00:56:43.485019300 +0000=0A= +++ sys/dev/mfi/mfi_debug.c 2012-11-09 02:24:35.569778892 +0000=0A= @@ -271,7 +271,7 @@=0A= {=0A= int i;=0A= =0A= - for (i =3D 0; i < sc->mfi_total_cmds; i++)=0A= + for (i =3D 0; i < sc->mfi_max_fw_cmds; i++)=0A= mfi_print_generic_frame(sc, &sc->mfi_commands[i]);=0A= }=0A= =0A= --- sys/dev/mfi/mfivar.h.orig 2012-11-09 00:55:55.000000000 +0000=0A= +++ sys/dev/mfi/mfivar.h 2012-11-09 14:36:01.203796033 +0000=0A= @@ -267,10 +267,6 @@=0A= */=0A= struct mfi_command *mfi_commands;=0A= /*=0A= - * How many commands were actually allocated=0A= - */=0A= - int mfi_total_cmds;=0A= - /*=0A= * How many commands the firmware can handle. Also how big the reply=0A= * queue is, minus 1.=0A= */=0A= ------=_NextPart_000_0276_01CDBE9C.80113780-- From owner-freebsd-scsi@FreeBSD.ORG Fri Nov 9 17:25:09 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3CD5A274; Fri, 9 Nov 2012 17:25:09 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 0A1918FC0A; Fri, 9 Nov 2012 17:25:08 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 09 Nov 2012 09:26:27 -0800 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id qA9HP8eY013825; Fri, 9 Nov 2012 09:25:08 -0800 (PST) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id qA9HP8oW013824; Fri, 9 Nov 2012 09:25:08 -0800 (PST) (envelope-from ambrisko) Date: Fri, 9 Nov 2012 09:25:08 -0800 From: Doug Ambrisko To: Steven Hartland Subject: Re: mfi panic on recused on non-recusive mutex MFI I/O lock Message-ID: <20121109172508.GA13333@ambrisko.com> References: <2DC1C56CFFF24FE0B17C34AD21A7DFAA@multiplay.co.uk> <39D16C43C8274CE9B8F23C18459E2FD4@multiplay.co.uk> <20121105212911.GA17904@ambrisko.com> <27169C7FE704495087A093752D15E7B6@multiplay.co.uk> <20121106180152.GA40422@ambrisko.com> <6B5B65F4FC854EB8BBC701500096602E@multiplay.co.uk> <0B4E8AFF9DA04C6EBD2496A8B58F1D67@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 17:25:09 -0000 On Fri, Nov 09, 2012 at 05:06:03PM -0000, Steven Hartland wrote: | | ----- Original Message ----- | From: "Steven Hartland" | ... | >I've just had another panic, trace below, but it doesn't seem to be related | >to my changes so I'd appreciate your feedback on them as they are for now. | > | >While the lock patch fixes the problems I've seen, its not clear to me | >why mfi_tbolt_reset is acquiring the lock and hence requiring | >mfi_process_fw_state_chg_isr to jump through hoops to ensure locking | >around queue manipulation is done correctly. Given what its doing | >(resetting the entire adapter) I wouldn't be surprised if it should | >really be acquiring the config lock. | > | >Other things I've noticed / questions | >* Should mfi_abort sleep even if its call to mfi_mapcmd fails? | >* Should mfi_get_controller_info really ignore the error from mfi_mapcmd? | >* Do these controllers not support none 512 byte requests? Currently | >all syspd requests are done assuming 512 byte sectors which the disk may | >not be. This will both reduce performance or potentially break totally | >if the firmware isn't translating it under the surface correctly. | > | >Anyway the new panic manually transcribed is:- | >panic: Bad linx elm 0xffffff0069b0fc0 next->prev != elm | >... | >mfi_tbolt_get_cmd() | >mfi_build_mpt_pass_thru() | >mfi_tbolt_build_mpt_cmd() | >mfi_tbolt_send_frame() | >bus_dmamap_load() | >mfi_mapcmd() | >mfi_startio() | >mfi_syspd_strategy() | >g_disk_start() | >g_io_schedule_down() | >g_down_proc_body() | >fork_exit() | >fork_trampoline() | > | >Looks like mfi_cmd_tbolt_tqh has become corrupt some how, but as far as I | >can tell all manip is done using the TAILQ macros and under mfi_io_lock | >so its not obvious to me at this time why this is, any ideas? | | I've gone through looking for the possible cause of this and while there's | nothing directly connected to the manip of this queue I've found and fixed | quite a large number of additional problems which may have been indirectly | causing this problem. | | The biggest change is to use mfi_max_cmds to limit the value stored in | sc->mfi_max_fw_cmds as this is used extensively throughout the driver | for allocation and range checks so having this inconsitently set opened up | a large number of possible overrun errors. | | The new patch attached documents all the changes in detail. | | I've managed to do one test run so far which failed to reproduce any panics, | so definitely moving in the right direction :) | | The machine has now been collected for repair by the supplier but I'm going | to try and get them to put it online for more testing over the weekend. | | Given the failure rate so far if I can do another 4 runs with no panics I'd | be happy that the majority of error conditions are working as expected. Sounds like you have made some good progress. I looked at your prior locking change and they good. Haven't had time to go through the queue changes yet. Thanks, Doug A.