From owner-freebsd-scsi@freebsd.org Tue Jul 10 17:32:28 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCF91103EFF5 for ; Tue, 10 Jul 2018 17:32:27 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E60681DF7 for ; Tue, 10 Jul 2018 17:32:27 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from [10.12.22.246] ([193.170.152.64]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0Lmvlg-1gHVRz3lnQ-00h6eO; Tue, 10 Jul 2018 19:32:23 +0200 Subject: Re: problems with SAS JBODs 2 To: Stephen Mcconnell , FreeBSD-scsi References: <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> <0af047d477d15ec364140653bd967c89@mail.gmail.com> From: Oliver Sech Message-ID: Date: Tue, 10 Jul 2018 19:32:23 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <0af047d477d15ec364140653bd967c89@mail.gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:vJlQGIsGuWSol0GZc/SGMDcStKitTIyqAdiP7ULzS7UcPA/XL3B 4tQU5WM0t7E/eu4WkoEc9DTurvdiw3HyRrFOI+JCX3E4KF2xqw8TBYHGeOUTaBGn4ZHKdFC IWcNXv9zT7kOpkkT1bpx9wrul3k5oaCnMu2GpdoYr7UQFluBGdAXEqTJMG89VKftTgQgR8Z 6YZHuvyRP76RjxSGiUXyA== X-UI-Out-Filterresults: notjunk:1;V01:K0:TGMrTMRuaHo=:/tuvrkT6sbZhUIdEhW5o/r Kcrp3QAeCEdSDB6sRUV7YleEWwfPhhiA/Ac8Aq/iBnOmpu3UVLjftlJ3U4AXIYMDeFEMH9Yre 1sLtvXOgKelQl6k6gTl79lsaueZTdRFwe/2RUncX26TcgcNXKzyHMlbyWq+v1+VVikFruRVWX 796VVIxS8wPrVqWHqXKCs62+ATPhsIb9h49/bslQp89oMl+eD6lwfHmP+20SwMNGTDrin3j80 lydpyZjP6Mg1in9/875JbHn+rRlQubSk7aL6gqveGGD1122Ai2pXvmyDjs/dCg1P7oBVTyalB bHNqFP4JjfgZPC/AlRbLqphKer+m6bJ+poJyti44ueEffqijj1+d71WkyVCJqpxwIInsJfl6u BMuBp1atCKnw5JlDgSeKBPRJvcZ3oZYHGxWd1Byd3Zl7rGJBmMHC1A7bHKkVPS7iFGXgNAVga dext3FwigpQ+6CYEBJ/xpXRrrCYJi840oSsPs8yQIkrxKoLxCSQyQ2tx4a3ZVPKruazhCoIub 6Kq4HwOgsjY0Znx45fhEVRdhwe+2CwfZbxv6Es5maFo3mnR2vStsGcvpBXmpE7ekBVTa7ceCW gsHnzfsoyXGP5PQIjJ6yVFLr/WFZCBLVpo5wgn9hu5TMQbFTwXNbIrWda6GwMKdLwLjMxx6PQ VWGB4DjEo74FW+MZf/TK4eEmFTglrR0BcFCu9b9UD/DsHBjtffy7IXRThLlFISkRRQMPPpRcA L4ipCrT8NQYxWDU52TVyXl4sVt8SQ9m2lLjeJ5LMGV3qUSE5/fQDE3RFFtMzPVdzx0BV446d3 Yy3Cz6s X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2018 17:32:28 -0000 sorry for sending dead links earlier... (Here is a link for the previous files: https://www.dropbox.com/s/5dlwizrzy48vme3/freebsd_sas.zip?dl=0 ) Here is the link for the new logs: https://www.dropbox.com/s/7bbt1fipg2a50oq/freebsd_sas2.zip?dl=0 notes: logfile: "1_clean_boot_without_shelves_dmesg" while booting with no shelves are attached it actually resets something: mpr0: mpr_mapping_check_devices: Enclosure XX is missing from the topology. Update its missing count. mpr0: _mapping_commit_enc_entry: Writing DPM entry XX for enclosure. logfile: "3_shelf_disconnected_geom" the only disks that really are connected are ada0, ada1, da0 everything else cannot get accessed Hardware: Promise SAS2 VTrak 830 (Full of SATA disks) + LSI 9305-16e Oliver On 07/10/2018 05:48 PM, Stephen Mcconnell wrote: > Ken, I looked at the logs and I don't see anything in them that suggests > that the driver is not adding any of the devices. In fact, I don't see > anything that looks strange at all. This looks like a different problem than > the other one you mentioned. What do you think? > > Steve > >> -----Original Message----- >> From: Stephen Mcconnell [mailto:stephen.mcconnell@broadcom.com] >> Sent: Tuesday, July 10, 2018 9:28 AM >> To: 'Oliver Sech'; 'FreeBSD-scsi' >> Subject: RE: problems with SAS JBODs 2 >> >> Hi Oliver, I can't get to your links. Can you try to send the logs in >> another >> way? >> >> Steve >> >>> -----Original Message----- >>> From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- >>> scsi@freebsd.org] On Behalf Of Oliver Sech >>> Sent: Tuesday, July 10, 2018 9:14 AM >>> To: FreeBSD-scsi >>> Subject: Re: problems with SAS JBODs 2 >>> >>> I tested a few additional things. I don't think this is a multipath, >>> daisy >> chain >>> nor a SAS wide ports problem. >>> I can reproduce the problem with just a single connection to an >>> Expander/JBOD. >>> >>> Test: >>> * physically disconnect all shelves >>> * reboot system >>> * connect one shelf via SAS cable >>> * check number of disks (after a reboot everything always shows up) >>> * disconnect the shelf and wait (geom disk list still shows most disks.) >>> * connect the shelf (missing disks) >>> >>> Tested Hardware: >>> * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e ( internal >> daisy >>> chain + wide links) >>> * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e (straight HBA >> <- >>>> EXPANDER connection. (no wide links, no daisy chain)) >>> * Supermicro SAS2 SC847E26-RJBOD1 + SAS3 LSI 9305-16e (internal >>> daisy >>> chain) >>> * Promise SAS2 VTrak 830 + SAS3 LSI 9305-16e (straight HBA >>> <-> >>> EXPANDER connection.) >>> >>> >>> >>> On 07/04/2018 12:15 PM, Oliver Sech wrote: >>>>> 1) Are the expanders daisy chained? Some SAS expanders don't work >>> reliably >>>>> when daisy chained. Best to direct connect each one to the server. >>>> At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 >>> lanes?). >>>> Unfortunately the JBOD has 24 slots in the front and 20 in the back >>>> and, >>> those are connected via a internal SAS daisy chaining. >>>> I could rewire and connect each backplane directly to the server, but >>> unfortunately I do not have enough ports.. >>>> >>>> JOBD Model: Supermicro 847E2C-R1K28JBOD >>>> >>>>> 2) Are the expanders connected in multipath or single path? You need >>>>> geom_multipath if you're going to do that. >>>> See answer 1. There is a single path from the host to the first >>>> expander. >>>> >>>>> 3) Are you attempting to use wide ports (two SAS cables connecting >> each >>>>> expander to the HBA). If do, you'll need to make sure that each pair >>>>> of >>>>> SAS cables goes to the same HBA chip (not merely the same card, as >> some >>>>> cards contain two HBA chips). >>>> see 1. The last time I opened one of those JBODs there were 8 SAS >>>> cables >>> between the Front and Back expander. I assume that wide ports are being >>> used. >>>> (2 expanders per backplane as well) >>>> >>>>> 4) Are you trying to remove an expander while ZFS is active on that >>>>> expander? That will suspend your pool, and ZFS doesn't always >>>>> recover >>> from >>>>> a suspended state. >>>> I'm testing with a new unused disk shelf that was never part of the >>>> ZFS >>> pool. There were >>>> _______________________________________________ >>>> freebsd-scsi@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >>>> To unsubscribe, send any mail to >>>> "freebsd-scsi-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-scsi@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi >>> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"