From owner-freebsd-scsi@freebsd.org Tue Jul 10 15:13:57 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5554A102F9D0 for ; Tue, 10 Jul 2018 15:13:57 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF2517B47E for ; Tue, 10 Jul 2018 15:13:56 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from [10.12.22.246] ([193.170.152.64]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LeSOH-1gQa1J1yij-00q7ur for ; Tue, 10 Jul 2018 17:13:48 +0200 Subject: Re: problems with SAS JBODs 2 From: Oliver Sech References: <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> To: FreeBSD-scsi Message-ID: Date: Tue, 10 Jul 2018 17:13:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K1:r02C5jHAr7PPxvUvgH1h798kBhZAKUUfXSeIkK3F4w4ZD0rIBaD Hb5SVz3ZA4b0MsBzu+hELGF64RjWMDcQumJZ3dmS76M+v5dBfqi2lM6jz7sRlehfYsBJ9Wg yz2ddWXOUgKXH5mVLVprdcLkjm9DKbGFqJ/iMXJSyfhRpSbt7BhJZGXvEgh5NEta3ESl7Zg PC6kE5sR5JMLfoowj7WOA== X-UI-Out-Filterresults: notjunk:1;V01:K0:/7z0975Yk5M=:cA5gEiAshfzdlJjAppK67R Oz+7tyxU8gPnhauSCp0nJ41Dwmm7E6+FC0oKLm70Sjw8Tv2CcOnLmpIQQMJzRGFG3NPZ17olZ 7zxaB69siZ0oqFjL+8m2l3ZGRVj1R2XRnhNZSwxxaM9p/ZyvaJPZFH70oWf00mJtTVx0GRGQC tqqBCtU1qAhrf0K0y7M/M01JeBLOZ1TwVOlLZWjOHt8ASjI5g1nuQwesDM2rlnHkuLC9qBkpd FBpg4iTerjj1pUZzDfXaD3iN+S3KILNudrZMFQFWEKV8ukzGL4JMhnTZBs/qICe2CrqnXoMs4 Xo3zBkZxEVfENfkq72raYglmGegcGOtHW48gSQZ0XI8J9rRAgD5L8uMqQNBmAqRyuELQP6lmc n1lUHqyvvIkQRmgQPJZbNQgpSQbljIRKwX5MoVrTzKoAoMcKGrW24Jq0MiZoLalss55XbcCCN AD0/a4jqo9vfkI0WNetfaTiTOYU9Gj5Ubgpne8bRBdfq6CApadL2mq4zTbPl2QZ5GGKXOUYjM gePxlz/KLEH8qCfd9LuWaPsiOiCx3QJkc8VVzEy97c4XI2QELPyBDoaviOp2VcPjAND7Q/dL8 tf7GPcxGIaUicHUSQjwVACZlCCL+IqhrdLjkyD2tV51E12Fw8NdNMQeHTD9lCaunWm2MYnIt5 UGNHFp3rgXUeQ1+wKsCjm/pdiOD/xGM9zXHykmsAJ5yRjGEkXDwzU3rsyeE3XaOedN9Yf/Grr ow9DpzgtljzJHf8O1FGSfZXyuhnLskjM2kQLyjwE30F7ym1I91EHhpIaxx2XPx3nxVabSMNLW jGTRrHY X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2018 15:13:57 -0000 I tested a few additional things. I don't think this is a multipath, daisy chain nor a SAS wide ports problem. I can reproduce the problem with just a single connection to an Expander/JBOD. Test: * physically disconnect all shelves * reboot system * connect one shelf via SAS cable * check number of disks (after a reboot everything always shows up) * disconnect the shelf and wait (geom disk list still shows most disks.) * connect the shelf (missing disks) Tested Hardware: * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e ( internal daisy chain + wide links) * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e (straight HBA <-> EXPANDER connection. (no wide links, no daisy chain)) * Supermicro SAS2 SC847E26-RJBOD1 + SAS3 LSI 9305-16e (internal daisy chain) * Promise SAS2 VTrak 830 + SAS3 LSI 9305-16e (straight HBA <-> EXPANDER connection.) On 07/04/2018 12:15 PM, Oliver Sech wrote: >> 1) Are the expanders daisy chained?  Some SAS expanders don't work reliably >> when daisy chained.   Best to direct connect each one to the server. > At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 lanes?). > Unfortunately the JBOD has 24 slots in the front and 20 in the back and, those are connected via a internal SAS daisy chaining. > I could rewire and connect each backplane directly to the server, but unfortunately I do not have enough ports.. > > JOBD Model: Supermicro 847E2C-R1K28JBOD > >> 2) Are the expanders connected in multipath or single path?  You need >> geom_multipath if you're going to do that. > See answer 1. There is a single path from the host to the first expander. > >> 3) Are you attempting to use wide ports (two SAS cables connecting each >> expander to the HBA).  If do, you'll need to make sure that each pair of >> SAS cables goes to the same HBA chip (not merely the same card, as some >> cards contain two HBA chips). > see 1. The last time I opened one of those JBODs there were 8 SAS cables between the Front and Back expander. I assume that wide ports are being used. > (2 expanders per backplane as well) > >> 4) Are you trying to remove an expander while ZFS is active on that >> expander?  That will suspend your pool, and ZFS doesn't always recover from >> a suspended state. > I'm testing with a new unused disk shelf that was never part of the ZFS pool. There were > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"