From owner-freebsd-scsi@freebsd.org Wed Jul 4 10:15:40 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61CE810252F6 for ; Wed, 4 Jul 2018 10:15:40 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A578B72379; Wed, 4 Jul 2018 10:15:39 +0000 (UTC) (envelope-from crimsonthunder@gmx.net) Received: from [10.12.22.246] ([193.170.152.64]) by mail.gmx.com (mrgmx003 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M4Gyx-1gSD2w1k4R-00rpLX; Wed, 04 Jul 2018 12:15:37 +0200 Subject: Re: problems with SAS JBODs 2 To: Alan Somers Cc: FreeBSD-scsi References: From: Oliver Sech Message-ID: <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> Date: Wed, 4 Jul 2018 12:15:36 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:gONWIpw357uDHwszQA/HlgHPawiUP7HBEm6/+oMHcSWHolxNmR8 5hTk1TyPzIp7K9WrUhWKad5JxU9l3kMARjGzAt4kzehB14EL3kfhl5KiRaSfBJo+xyq92D+ Ehtst7pw9kH4R9ziq9d6zGn0HX2aujZ2B0KN1Z5006oH0b4vUD6pTCP8bwMcsk1IDCaJXsr 0rEv2cCGmLYUkO2AIwcxA== X-UI-Out-Filterresults: notjunk:1;V01:K0:GuyHOPMS98c=:/jtz4jUYIX/wRiIVf4VbU3 9xcQbSFtKlsDpiGAwshlFZaKnTZo6I4Yyq+Gc/4yeMk+SSOId1/LY8ZfFlN8gt8S7TM50jsOQ GeznOlQscWKOweeFwIRtZTuJTbs5+gP85IV9/T0VgkdNu5HQT4H2jCmlkl+v1HuRS9Rz6w+QE kUGDmWS4cIJlhxENy6/wDz/QDgFlIBmX1H/GgWVHddJCGOneXE8UeWk9fD0+HT5A6TxipXKPL M0F3DS380xvksHbqLzRQ1zUCZNrbCwozwMTabRyL9Ro67FIwQB1qvBnMppli+21PDK/nsxEzg vdXcBEUvTvjWaiwCgMU5IhFT4PR3gNGb3jTGgOa2M20f03xTvp/tH6748iTHfQjMGG+u/d+nD h6sH0npLyB72Na0zMj9W+oFjQTSep0j//jjoqxcotpUh+4jvzkt3Te4ydq6f3ZoCkHDebeTQQ Q8d1G16Ki883a+DW7AfAjrf349LAxTo1pbU6VBXmM+RL3XHNbYGvdA1O+y8czPIKvuhUFu1Eu Ky17QY9VFT5E0fKbWGCw6jztLgcZLZWcv6/oTMsRILNpXkuzAsG9TLdUhTnWVWTI14SVfm3Zj YtssUM8QdrHG1QwJS/fT2ALPA6v7P2fQUl0MCSEUIj/U9U9We4UXEmeuOfFSL0HQq36EobUlq wnMxNFuzHV8FVfMw4pDZ+zVGGh19aSfurtJlRbth/KQ/0gc5XcH6Cj5jErH7dDT36THz/havm m+pbByDhE898XEtK9qt0qb6J6uowB3O9nCWy43Rss6H+n3KNevU+GAHaK4Cljv37R8q6PvRfV b7PI86z X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jul 2018 10:15:40 -0000 > 1) Are the expanders daisy chained? Some SAS expanders don't work reliably > when daisy chained. Best to direct connect each one to the server. At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 lanes?). Unfortunately the JBOD has 24 slots in the front and 20 in the back and, those are connected via a internal SAS daisy chaining. I could rewire and connect each backplane directly to the server, but unfortunately I do not have enough ports.. JOBD Model: Supermicro 847E2C-R1K28JBOD > 2) Are the expanders connected in multipath or single path? You need > geom_multipath if you're going to do that. See answer 1. There is a single path from the host to the first expander. > 3) Are you attempting to use wide ports (two SAS cables connecting each > expander to the HBA). If do, you'll need to make sure that each pair of > SAS cables goes to the same HBA chip (not merely the same card, as some > cards contain two HBA chips). see 1. The last time I opened one of those JBODs there were 8 SAS cables between the Front and Back expander. I assume that wide ports are being used. (2 expanders per backplane as well) > 4) Are you trying to remove an expander while ZFS is active on that > expander? That will suspend your pool, and ZFS doesn't always recover from > a suspended state. I'm testing with a new unused disk shelf that was never part of the ZFS pool. There were