Date: Tue, 3 Jul 2018 08:06:22 -0600 From: Alan Somers <asomers@freebsd.org> To: Oliver Sech <CrimsonThunder@gmx.net> Cc: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: Re: problems with SAS JBODs 2 Message-ID: <CAOtMX2h8r31AeNCKyckK2P0VLn1CKFogo9bWom2So1x2ngpa4A@mail.gmail.com> In-Reply-To: <trinity-14d18077-ea73-40f6-9e87-d2d4000b1f7e-1530620937871@3c-app-gmx-bs01> References: <trinity-14d18077-ea73-40f6-9e87-d2d4000b1f7e-1530620937871@3c-app-gmx-bs01>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jul 3, 2018 at 6:28 AM, Oliver Sech <CrimsonThunder@gmx.net> wrote: > Hi! > > I use FreeBSD with for a large ZFS pool (over 1PB) and I recently > encountered a lot of problems with the JBODs. Generally everything works > fine until I replug the shelves. > > When I start with a clean system and attach a single shelf every thing > seems fine. > -> 44 disks show up, I can use the enclosure services (sesutil) and the > system continues to run without problems. > Once I disconnect the SAS cable, wait until all devices disapear and > reconnect I get all sorts of problems. > -> a random number of disks shows up and the enclosure "ses" do not show up > Once I restart the system I can start over again. > > On the server with the large pool there are only certain ports on the HBA > that I can use, otherwise disks will be missing after a reboot and my ZFS > pool won't go online. > I tried different firmware on the HBA. I tried the mpr.ko module from the > broadcom site. (I replaced the one in /boot/kernel?) > I tested all the things above with a Linux as OS and everything seems to > work. > > > Is there anything I'm missing? A command that can reset the SAS components? > > > FreeBSD version: 11.1-RELEASE-p11 > HBA: broadcom lsi 9305-16e (latest firmware) > JBOD:SC847E2C-R1K28JBOD (two expanders, internally daisy chained) > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > 1) Are the expanders daisy chained? Some SAS expanders don't work reliably when daisy chained. Best to direct connect each one to the server. 2) Are the expanders connected in multipath or single path? You need geom_multipath if you're going to do that. 3) Are you attempting to use wide ports (two SAS cables connecting each expander to the HBA). If do, you'll need to make sure that each pair of SAS cables goes to the same HBA chip (not merely the same card, as some cards contain two HBA chips). 4) Are you trying to remove an expander while ZFS is active on that expander? That will suspend your pool, and ZFS doesn't always recover from a suspended state. -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2h8r31AeNCKyckK2P0VLn1CKFogo9bWom2So1x2ngpa4A>