From owner-freebsd-scsi@freebsd.org Tue Jul 3 12:29:07 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24F42103ED46 for ; Tue, 3 Jul 2018 12:29:07 +0000 (UTC) (envelope-from CrimsonThunder@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 725077E3B9 for ; Tue, 3 Jul 2018 12:29:06 +0000 (UTC) (envelope-from CrimsonThunder@gmx.net) Received: from [193.170.152.64] ([193.170.152.64]) by web-mail.gmx.net (3c-app-gmx-bs01.server.lan [172.19.170.50]) (via HTTP); Tue, 3 Jul 2018 14:28:58 +0200 MIME-Version: 1.0 Message-ID: From: "Oliver Sech" To: freebsd-scsi@freebsd.org Subject: problems with SAS JBODs 2 Content-Type: text/plain; charset=UTF-8 Date: Tue, 3 Jul 2018 14:28:58 +0200 Importance: normal Sensitivity: Normal Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-Provags-ID: V03:K1:rYQxv3CGKWSL7s+RhGFtTT9jCBYRww7J/UgGFudYROqRYjXXbp1//Fa+JHCit7GWFtZB0 bHKcLyb9cxmbv1SYv9rss8e2awS6oiHGxnynG7L2PLeBFhRUEazBSiKVvE3VAKDfQvyMxUMFOVSf ewoFnITzVyrlLfXheuJmZRkhrucH+waQ3fdeELdWOs0wTCnCEat/nAhaf9EyVFB8dvwjrmXTbIy/ D2skSVKQG/rjELv7SyYpfb8faZ0LBwI3nOl07/H9sh8HiGskloX5XOYvmKhEMJwWO4rftmFNKCxv qo= X-UI-Out-Filterresults: notjunk:1;V01:K0:pMcRsJJ3n/A=:q3clALHsU/7+wCST7VHZIW HZlMntZDLytHpPls9/Dp0Wm4vEi4UsbmdLVfMezB3AfSNPvOuOFIWSXkWZpggrEiapDsomyMJ XPrp0D4cEh5WQ89pywPEsRbUPjYoEd7S7/xKlSh5uAGixOO+dBR4BterpLTbxzo0I5Lq0B3to qAplLb3W2tRvnBs69jJT/2Fw8CGP6JrZHNaWJAGFvTGOYRJzrQm86QNnY1kF8bohqX/mEr+s4 RsiWFoyiNc126YEfk7BiAnrVb24B6SVaQPHfw8ymXFLTTUYUAqETZK0FWRl3tgs0om1LuK/qq 4UNfX06N2y9qvy1qPzKlOhgVCBacsBlaKgl8jYQFnhFrGU/KBq8vMggJU7VGoeGT4E2BBpvn2 IRoI9iE5X+fk7FzvKLiaGhVTPAMR3jskCGjXa1Z3Wqh9yU8TWg9WrwaYQhWX6uCydgq2iM0/T k2e/tCDPH75qCKpfh4t/D/5HJiEdBXXowWjoeb7elmQ5nrhKFU3J X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2018 12:29:07 -0000 Hi! =C2=A0 I use FreeBSD with for a large ZFS pool (over 1PB) and I recently encounte= red a lot of problems with the JBODs=2E Generally everything works fine unt= il I replug the shelves=2E =C2=A0 When I start with a clean system and attach a single shelf every thing see= ms fine=2E -> 44 disks show up, I can use the enclosure services (sesutil) and the sy= stem continues to run without problems=2E Once I disconnect the SAS cable, wait until all devices disapear and recon= nect I get all sorts of problems=2E -> a random number of disks shows up and the enclosure "ses" do not show u= p Once I restart the system I can start over again=2E =C2=A0 On the server with the large pool there are only certain ports on the HBA = that I can use, otherwise disks will be missing after a reboot and my ZFS p= ool won't go online=2E I tried different firmware on the HBA=2E I tried the mpr=2Eko module from = the broadcom site=2E (I replaced the one in /boot/kernel?) I tested all the things above with a Linux as OS and everything seems to w= ork=2E =C2=A0 =C2=A0 Is there anything I'm missing? A command that can reset the SAS components= ? =C2=A0 =C2=A0 FreeBSD version: 11=2E1-RELEASE-p11 HBA: broadcom lsi 9305-16e (latest firmware) JBOD:SC847E2C-R1K28JBOD (two expanders, internally daisy chained)