From owner-freebsd-scsi@FreeBSD.ORG Mon Feb 10 11:06:54 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D21F114F for ; Mon, 10 Feb 2014 11:06:54 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BC54C1FEC for ; Mon, 10 Feb 2014 11:06:54 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s1AB6sT5080190 for ; Mon, 10 Feb 2014 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s1AB6sNU080188 for freebsd-scsi@FreeBSD.org; Mon, 10 Feb 2014 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Feb 2014 11:06:54 GMT Message-Id: <201402101106.s1AB6sNU080188@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Feb 2014 11:06:54 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 17 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Feb 11 15:41:06 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8E969981 for ; Tue, 11 Feb 2014 15:41:06 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 108411FE9 for ; Tue, 11 Feb 2014 15:41:05 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id D14A19DC5A9 for ; Tue, 11 Feb 2014 16:41:03 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: The bloody RAID/JBOD virus Date: Tue, 11 Feb 2014 16:41:02 +0100 Message-Id: <359D6D68-B8AA-4295-8571-85F82E24D5E0@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Feb 2014 15:41:06 -0000 Hello, We are again evaluating hardware to use FreeBSD with ZFS as a storage = server. And, of course, we are again banging our heads against the bloody "intelligent" controllers. This machine has two controllers. The first one is recognized by the mps = driver, mps0: port 0x3f00-0x3fff mem = 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on = pci17 mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: = 185c and the second one (this is what I don't like at all) by the mfi driver. mfi0: port 0x4f00-0x4fff mem = 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on = pci22 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23=20 mfi0: FW MaxCmds =3D 240, limiting to 128 mfi0: MaxCmd =3D 240, Drv MaxCmd =3D 128, MaxSgl =3D 70, state =3D = 0xb73c00f0 So, again, we have the typical scenario of the RAID card between ZFS and = the disks. # camcontrol devlist at scbus0 target 1 lun 0 (pass0,da0) The machine has actually 24 disks.=20 # zpool status pool: clientes state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM clientes ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 mfisyspd0 ONLINE 0 0 0 mfisyspd1 ONLINE 0 0 0 mfisyspd2 ONLINE 0 0 0 mfisyspd3 ONLINE 0 0 0 mfisyspd4 ONLINE 0 0 0 mfisyspd5 ONLINE 0 0 0 mfisyspd6 ONLINE 0 0 0 mfisyspd7 ONLINE 0 0 0 mfisyspd8 ONLINE 0 0 0 mfisyspd9 ONLINE 0 0 0 mfisyspd10 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 mfisyspd11 ONLINE 0 0 0 mfisyspd12 ONLINE 0 0 0 mfisyspd13 ONLINE 0 0 0 mfisyspd14 ONLINE 0 0 0 mfisyspd15 ONLINE 0 0 0 mfisyspd16 ONLINE 0 0 0 mfisyspd17 ONLINE 0 0 0 mfisyspd18 ONLINE 0 0 0 mfisyspd19 ONLINE 0 0 0 mfisyspd20 ONLINE 0 0 0 mfisyspd21 ONLINE 0 0 0 spares mfisyspd22 AVAIL =20 errors: No known data errors But, again, defined as "jbod", which I don't like at all. At least in = the past this formula has been a proven disaster. In the past (similar = situations) I've been unable to hot swap disks without voodoo macumba = procedures (I consider needing a "mfiutil" voodoo macumba) and, of = course, the real CAM subsystem has no visibility at all. # mfiutil show adapter mfi0 Adapter: Product Name: ServeRAID M5210e Serial Number: 3CJ0SG =20 Firmware: 24.0.2-0013 RAID Levels: JBOD, RAID0, RAID1, RAID10 Battery Backup: not present NVRAM: 32K Onboard Memory: 0M Minimum Stripe: 64K Maximum Stripe: 64K Actually, in several machines I use a patched driver that ignores all = the raid crap and presents the disks as real SAS disks. But I am using = it on machines in which a failure is not a tragedy. Been working for = years without incidents, but OS updates are always risky. So, I would like to know: 1) Is this "mfisyspd" REALLY a disk? Won't I notice any differences? So = far, I've attached SSD disks and ZFS has created a pool with 512 byte = blocks. Note the difference between a "more or less like a disk JBOD" = (which I definitely do not want) and a real disk. 2) Is there a way to bypass all that or should I look for a replacement = HBA instead? Seems it's impossible to get manufacturers to ship simple = HBAs without that "intelligent" RAID thing. Of course I may be wrong, and this card might be what I really want, = with no interference from the RAID functionality. But, so far, every = time I've seen so-called JBODs defined with RAID cards, they were = actually 1 disk RAID0 logical volumes, which I don't want. At least loading the mfip driver gives me access to the pass devices, = which is some progress. But I'm still not sure. Sorry for the blunt message, but wherever I look I see all these cards = we should not use with ZFS. Thanks! Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 14 12:44:35 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BA81C3EC; Fri, 14 Feb 2014 12:44:35 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 794121389; Fri, 14 Feb 2014 12:44:35 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id DC0179DC7B1; Fri, 14 Feb 2014 13:44:27 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Subject: FreeBSD 10, ServeRAID M5210e, syspd corruption Date: Fri, 14 Feb 2014 13:44:26 +0100 Message-Id: To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) Cc: Stable Stable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 12:44:35 -0000 (crossposting to -Stable just in case) Hello, I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID = M5210e and 23 SSD disks. uname -a FreeBSD hostname 10.0-RELEASE FreeBSD 10.0-RELEASE #1: Fri Feb 14 = 09:35:12 CET 2014 toor@ hostname:/usr/obj/usr/src/sys/GENERIC amd64 The server has a SAS backplane and a controller recognized by the mfi = driver. mfi0 Adapter: Product Name: ServeRAID M5210e Serial Number: 3CJ0SG =20 Firmware: 24.0.2-0013 RAID Levels: JBOD, RAID0, RAID1, RAID10 Battery Backup: not present NVRAM: 32K Onboard Memory: 0M Minimum Stripe: 64K Maximum Stripe: 64K As I am intending to use ZFS, I need direct access to the disks, no need = for fancy RAID features. I have seen that the newest cards support a so-called "syspd" mode that = gives direct acess to the disks. However, in this configuration, syspd consistently corrupts data on the = disks. I have done tests with three models of disks: - Samsung SSD 840 BB0Q (1 TB) - OCZ-VERTEX4 1.5 (512 GB) - SEAGATE ST9146803SS FS03 (136 GB) In the three cases there is data corruption. Using FFS on the disks = results in a panic if I run a benchmark, for example, bonnie++. Using ZFS (I've been creating one disk pools to test) I don't get panics = but the data is consistently corrupted. The writes work, but whenever there is read activity (either bonnie++ reaching the "rewrite" phase, or = a ZFS scrub), ZFS detects data corruption. Trying the =FCber neat hw.mfi.allow_cam_disk_passthrough (which is = great, because ZFS can detect the SSDs and issue TRIM commands) I get the same result: data corruption. However, I have tried to create a one-disk raid0 volume, and in that = case it works like a charm, no corruption at all, so I can safely assume that this is not a defective backplane, expander or cabling.=20 So: mfisyspd -> CORRUPT da -> CORRUPT mfid -> NOT CORRUPT Any ideas? Could be a driver error or a firmware problem, I am clueless = for now. Anything I can test? The machine is not in production, I can try patches = or whatever. Thanks!! Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 14 13:44:42 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 521A66F4; Fri, 14 Feb 2014 13:44:42 +0000 (UTC) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A232018B8; Fri, 14 Feb 2014 13:44:41 +0000 (UTC) Received: by mail-lb0-f169.google.com with SMTP id q8so9500825lbi.0 for ; Fri, 14 Feb 2014 05:44:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=T3zoRPxjPz2DYss2XSE3Q05jpSTd1MVe2tdn+gJR8IM=; b=KBG5DomhFDxlhM1msUmQmbI/wlcphgrTAgE+kjiGdjQW+V6qwzYEv1tLjpef+U7gTo a52Nmi82+EBwjPbLD668MAzLgYXSU2x0q/Q5ZAhVFOuTq9MeLsV2IEfl7nLe5OuD1Kxr nAv9IX5NI3nbNCoGXswaubXlA2og+e657AlMDVtCaLv6Y9xC2R53AnptJ3eOuoVT6+fm EgcYIgN/gg01KGYVkhpr3dsZOReObaapeH55GeGmyWaxfU2d53ttEKbGiWUOQxN92lf7 y9QvRytMx0lKBRKUb2bLtSAhzGGG4jrH3yz6Zb7R33hZevN4MO+QjTsOMjr3CdpzYA7l Fj+g== MIME-Version: 1.0 X-Received: by 10.112.160.200 with SMTP id xm8mr5188105lbb.24.1392385479597; Fri, 14 Feb 2014 05:44:39 -0800 (PST) Received: by 10.112.35.167 with HTTP; Fri, 14 Feb 2014 05:44:39 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Feb 2014 13:44:39 +0000 Message-ID: Subject: Re: FreeBSD 10, ServeRAID M5210e, syspd corruption From: Tom Evans To: Borja Marcos Content-Type: text/plain; charset=UTF-8 Cc: freebsd-scsi@freebsd.org, Stable Stable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 13:44:42 -0000 On Fri, Feb 14, 2014 at 12:44 PM, Borja Marcos wrote: > (crossposting to -Stable just in case) > > Hello, > > I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID M5210e and 23 SSD disks. > I'm afraid I have no solution to offer you for this issue, but with this setup an mps(8) card (LSI SAS2008 and similar) in IT (passthru) mode would work excellently. Maybe easier to change the card than struggle to get it to do something it doesn't want to? Cheers Tom From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 14 14:00:11 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4079BF1E; Fri, 14 Feb 2014 14:00:10 +0000 (UTC) Received: from sarah.protected-networks.net (sarah.protected-networks.net [202.12.127.65]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 489601A03; Fri, 14 Feb 2014 14:00:10 +0000 (UTC) Received: from toshi.auburn.protected-networks.net (toshi.auburn.protected-networks.net [202.12.127.84]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "Iain Butler", Issuer "RSA Class 2 Personal CA v2" (verified OK)) (Authenticated sender: imb@protected-networks.net) by sarah.protected-networks.net (Postfix) with ESMTPSA id 6F97260F5; Fri, 14 Feb 2014 09:00:02 -0500 (EST) DomainKey-Signature: a=rsa-sha1; s=200509; d=protected-networks.net; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:cc:subject: references:in-reply-to:x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=BNvBkca/eP6BI7XgVAc0zhsy2PaCWk9wX+UB2B9XpNitsEnkZok1gnPCCpDVaaDaM I81OiWuWttVfZ2kO/5WC7q+uTDU5OW8AS+SnCkTyPGCixI5XWyAG6aclb1ZID0t Message-ID: <52FE2160.5030805@protected-networks.net> Date: Fri, 14 Feb 2014 09:00:00 -0500 From: Michael Butler User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Borja Marcos , freebsd-scsi@freebsd.org Subject: Re: FreeBSD 10, ServeRAID M5210e, syspd corruption References: In-Reply-To: X-Enigmail-Version: 1.6 OpenPGP: id=0442D492 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Stable Stable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 14:00:11 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/14/14 07:44, Borja Marcos wrote: > I have seen that the newest cards support a so-called "syspd" mode that gives direct acess to the disks. > However, in this configuration, syspd consistently corrupts data on the disks. Just a silly "left field" idea .. can you try adding .. vfs.unmapped_buf_allowed=0 .. to /boot/loader.conf and re-testing? As I say, just an idea .. Michael -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlL+IWAACgkQQv9rrgRC1JK/LwCfS8cO2BWAimSO/W7Y8E1VsjuK L6wAnRgl46sXeO1f04Dn1oF+wwHK1zng =k1oh -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 14 14:46:43 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 955B4B2C; Fri, 14 Feb 2014 14:46:43 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 535D81F9F; Fri, 14 Feb 2014 14:46:42 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 280D19DCE86; Fri, 14 Feb 2014 15:46:41 +0100 (CET) Subject: Re: FreeBSD 10, ServeRAID M5210e, syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset="utf8"; X-Pgp-Agent: GPGMail (null) From: Borja Marcos In-Reply-To: <52FE2160.5030805@protected-networks.net> Date: Fri, 14 Feb 2014 15:46:33 +0100 Content-Transfer-Encoding: 8bit Message-Id: References: <52FE2160.5030805@protected-networks.net> To: Michael Butler X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org, Stable Stable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 14:46:43 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 14, 2014, at 3:00 PM, Michael Butler wrote: Signed PGP part On 02/14/14 07:44, Borja Marcos wrote: > I have seen that the newest cards support a so-called "syspd" mode that gives direct acess to the disks. > However, in this configuration, syspd consistently corrupts data on the disks. Just a silly "left field" idea .. can you try adding .. vfs.unmapped_buf_allowed=0 Thanks. Same result unfortunately. Borja. -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlL+LE8ACgkQULpVo4XWgJ+GSwCguixlvwrVRgc73VgaqcSbEN1M 7fEAoKMa0ksOUd2DxQskoYGRTHpCzEOj =Othe -----END PGP SIGNATURE----- From owner-freebsd-scsi@FreeBSD.ORG Sat Feb 15 16:14:49 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0137BF8 for ; Sat, 15 Feb 2014 16:14:49 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 723EF139E for ; Sat, 15 Feb 2014 16:14:49 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s1FGEfvq029915 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 15 Feb 2014 16:14:41 GMT Date: Sat, 15 Feb 2014 16:14:40 +0000 From: Karl Pielorz To: freebsd-scsi@freebsd.org Subject: Wiring down devices with LSI 2308 / mps? Message-ID: <41337B9333D94F88B9469A60@study64.tdx.co.uk> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 16:14:49 -0000 Hi, I've got a SuperMicro board that has an onboard LSI 2308 onboard. This appears using the mps driver: " mps0: port 0xc200-0xc2ff mem 0xf33c0000-0xf33cffff,0xf3380000-0xf33bffff irq 44 at device 7.0 on pci0 mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities:185c " I'd like to wire the devices down on this - but that doesn't seem to work :( I tried this by putting entries in '/boot/device.hints' such as: " hint.scbus.0.at="mps0" hint.da.0.at="scbus0" hint.da.0.target="2" hint.da.0.unit="0" hint.da.1.at="scbus0" hint.da.1.target="1" hint.da.1.unit="0" ... " Testing though shows that maybe the controller itself is collapsing 'missing' ports/targets? e.g. Attach a disk to 'Port 0' - and it shows as: da2 at mps0 bus 0 scbus0 target 2 lun 0 Ok, target 2? - but at least that's mapped to da2 (wiring works) Take the disk off, and connect it to Port 1 - and it shows as: da2 at mps0 bus 0 scbus0 target 2 lun 0 Port 1 is target 2 as well? - Switch the drive to 'Port 2' - and you get: da2 at mps0 bus 0 scbus0 target 2 lun 0 So port 2 is also target 2? (if the others aren't occupied) Add a 2nd drive (to any port) and you get: da3 at mps0 bus 0 scbus0 target 3 lun 0 So the 'wiring down' is working - but it looks like the LSI collapses it's own 'missing drives'? - And starts at Target 2? I can't see any setting in the LSI bios that would change this? - I'm presuming it must be possible not to have it do this - though searching the lists didn't turn anything obvious up... Thanks, -Karl From owner-freebsd-scsi@FreeBSD.ORG Sat Feb 15 21:17:08 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7B29ABA9 for ; Sat, 15 Feb 2014 21:17:08 +0000 (UTC) Received: from nm25-vm6.bullet.mail.ne1.yahoo.com (nm25-vm6.bullet.mail.ne1.yahoo.com [98.138.91.118]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 33CF619F2 for ; Sat, 15 Feb 2014 21:17:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=gcom1024; t=1392498871; bh=+dVM74jjrVRRKJlyCyDwdtWJKbPQPtFjB4h+FY8uIjs=; h=Received:Received:Received:DKIM-Signature:X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=MTh3EcgYgxC3Yfom48vZ/IE7c0hsxIDJnCzI+zoPfsE5QnDYcyXPsT4hEfzqNHcRIUm7ccaiMJMjy0rZ7XpkKF5V/tJxQLA4vNRZaDpyIqxXL6x/iqxpo+Tagtn1BHYNme3sBbdi6fh5WxNK+jXHbyuyqALTy14EpKgQg2JyDto= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=gcom1024; d=yahoo.com; b=kc8vCenNSH6PrWnMcv7gkS9KGlXZ25RDb3iMS/cyH5u5RRuV1zHa8nzNdfN7hyy+18HC/pnqQnhvPgKHfs3EASOD26tzPfPC+06kQPYTpKsK8g7F5cQkyZXuCwzBB4JW5pkHY9QJAEgkUKbiU7fZ7v29WpUyX8PUIvT7smihMmo=; Received: from [98.138.101.128] by nm25.bullet.mail.ne1.yahoo.com with NNFMP; 15 Feb 2014 21:14:31 -0000 Received: from [98.138.84.44] by tm16.bullet.mail.ne1.yahoo.com with NNFMP; 15 Feb 2014 21:14:31 -0000 Received: from [127.0.0.1] by smtp112.mail.ne1.yahoo.com with NNFMP; 15 Feb 2014 21:14:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1392498871; bh=+dVM74jjrVRRKJlyCyDwdtWJKbPQPtFjB4h+FY8uIjs=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=FCdMbvYDggMmWgFX80lySEea9k19LF0Fk2vRH9To9D57FkLrRiUJak02qS9pN8PKTFTpbOZJ/GuldTjspNmBry/34ndCCVaqNUYS0o0MALT0RrI2O7raxjUX4g/EYkjF6lfg16R786QjtOZoC6XRB3j6aHFW/6i6+Ry5FQD1+0M= X-Yahoo-Newman-Id: 674809.89328.bm@smtp112.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 03iKl9QVM1nFjzKSYoPTqb5qIIBYs4M1G6gfJDTp7tOzZWV _WwSWMsJLY67TJ6ba3OYDmWPklPjRK8yA6rcM8P5lKA3fkl6NlRAPebIg5rw 64nEbKNj8NjxtPAkJILai4pgRlXKuixKFF.TAvaYwvB3ftXtDhI_Itb5TGRV OPTScC38Z1mdjfSvxP8uT_U5._2_Wwj.G19KCNgYly8.F.gkyRa3WWLokrwf ESeJrV3rZBX8Xzhly0b9dUMHbosEFBdKOmgYDgQNbl.B5_YeUElZ0CR7a7yE 9zhwjIgg47Pyp4wbce19dMhCS4i82CwSADulICk4BxkK4L41HYPiOdd1NXQI jfDu2OKU24z6bWL5f2Yg8g7FhREcrz53MBIxZf.asINJA4XZkvvMq5ay3fFj F.lo3n2a9AGgV6x8VplCv5vqM7AGUt0YHhFow4iQkI_W_UgHiBdjNE9U5nmy KJXszXOGWtxVgnVYsfE8y24aglpYQ_q5dIkBUj1g_RcKz6FbOLk8m6MsQurd 6jHuUfboV_Hx3gqrLfD3csJ.IwIhUX75mlgOwk3M4qsPBXn6.xqoQCYts4XR zU9cQTqkuqNeKQR2SL4NOo.fzvk34wc0J4TFmsPRIdWucVNrEE4Q2Utzqm6l _tSLr4nPGkiSB X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- X-Rocket-Received: from phobos.samsco.home (scott4long@168.103.85.57 with plain [98.138.105.21]) by smtp112.mail.ne1.yahoo.com with SMTP; 15 Feb 2014 13:14:31 -0800 PST Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Wiring down devices with LSI 2308 / mps? From: Scott Long In-Reply-To: <41337B9333D94F88B9469A60@study64.tdx.co.uk> Date: Sat, 15 Feb 2014 14:14:29 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <41337B9333D94F88B9469A60@study64.tdx.co.uk> To: Karl Pielorz X-Mailer: Apple Mail (2.1827) Cc: "FreeBSD-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 21:17:08 -0000 Hi, The LSI cards keep a persistent mapping of hard drive serial numbers to = target numbers. That way, when it=92s in RAID mode and you pull a drive = from one slot and put it into another slot, it=92ll recognize that it=92s = the same drive and part of the same array set. It=92s incredibly = annoying, and once the persistent mapping is created, it cannot be = cleared just by turning off the RAID mode. What we do at Netflix is use = the a tool provided by LSI to clear out the persistent mapping (and all = of NVRAM, actually), and then turn off the persistence feature. = Unfortunately, you=92ll need to contact LSI directly for this tool. = Once you have it, I can share a script that performs the necessary = actions. Contact me off-list if you=92d like more information. Scott On Feb 15, 2014, at 9:14 AM, Karl Pielorz = wrote: >=20 > Hi, >=20 > I've got a SuperMicro board that has an onboard LSI 2308 onboard. This = appears using the mps driver: >=20 > " > mps0: port 0xc200-0xc2ff mem = 0xf33c0000-0xf33cffff,0xf3380000-0xf33bffff irq 44 at device 7.0 on pci0 > mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd > mps0: = IOCCapabilities:185c > " >=20 > I'd like to wire the devices down on this - but that doesn't seem to = work :( >=20 > I tried this by putting entries in '/boot/device.hints' such as: >=20 > " > hint.scbus.0.at=3D"mps0" >=20 > hint.da.0.at=3D"scbus0" > hint.da.0.target=3D"2" > hint.da.0.unit=3D"0" >=20 > hint.da.1.at=3D"scbus0" > hint.da.1.target=3D"1" > hint.da.1.unit=3D"0" > ... > " >=20 > Testing though shows that maybe the controller itself is collapsing = 'missing' ports/targets? >=20 > e.g. Attach a disk to 'Port 0' - and it shows as: >=20 > da2 at mps0 bus 0 scbus0 target 2 lun 0 >=20 > Ok, target 2? - but at least that's mapped to da2 (wiring works) >=20 > Take the disk off, and connect it to Port 1 - and it shows as: >=20 > da2 at mps0 bus 0 scbus0 target 2 lun 0 >=20 > Port 1 is target 2 as well? - Switch the drive to 'Port 2' - and you = get: >=20 > da2 at mps0 bus 0 scbus0 target 2 lun 0 >=20 > So port 2 is also target 2? (if the others aren't occupied) >=20 > Add a 2nd drive (to any port) and you get: >=20 > da3 at mps0 bus 0 scbus0 target 3 lun 0 >=20 >=20 > So the 'wiring down' is working - but it looks like the LSI collapses = it's own 'missing drives'? - And starts at Target 2? >=20 > I can't see any setting in the LSI bios that would change this? - I'm = presuming it must be possible not to have it do this - though searching = the lists didn't turn anything obvious up... >=20 > Thanks, >=20 > -Karl >=20 > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to = "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Sat Feb 15 23:59:48 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9E05A10F for ; Sat, 15 Feb 2014 23:59:48 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 5FDFA15CB for ; Sat, 15 Feb 2014 23:59:47 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s1FNxkeN076635 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 15 Feb 2014 23:59:46 GMT Date: Sat, 15 Feb 2014 23:59:46 +0000 From: Karl Pielorz To: Scott Long Subject: Re: Wiring down devices with LSI 2308 / mps? Message-ID: <53DEE6652CDBF8E49ECF9FE3@study64.tdx.co.uk> In-Reply-To: References: <41337B9333D94F88B9469A60@study64.tdx.co.uk> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: "FreeBSD-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 23:59:48 -0000 --On 15 February 2014 14:14:29 -0700 Scott Long =20 wrote: > Hi, > > The LSI cards keep a persistent mapping of hard drive serial numbers to > target numbers. That way, when it=E2=80=99s in RAID mode and you pull a = drive > from one slot and put it into another slot, it=E2=80=99ll recognize that = it=E2=80=99s > the same drive and part of the same array set. It=E2=80=99s incredibly > annoying, and once the persistent mapping is created, it cannot be > cleared just by turning off the RAID mode. What we do at Netflix is use > the a tool provided by LSI to clear out the persistent mapping (and all > of NVRAM, actually), and then turn off the persistence feature. > Unfortunately, you=E2=80=99ll need to contact LSI directly for this tool. = Once > you have it, I can share a script that performs the necessary actions. > Contact me off-list if you=E2=80=99d like more information. Ha - at least I know it's not me going mad then :) I'm living with GPT labels at the moment (and labels on the drive caddies)=20 - if that doesn't work out (it's currently "bearable") I'll tap LSI for the = 'magic tools', and drop you an email if they give me a copy. You're right though it is *incredibly annoying* (especially when you don't=20 know it's doing it!). -Karl