From owner-freebsd-scsi@freebsd.org Tue Jul 3 14:06:26 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE51D102337B for ; Tue, 3 Jul 2018 14:06:25 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf0-x236.google.com (mail-lf0-x236.google.com [IPv6:2a00:1450:4010:c07::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4396282BA9 for ; Tue, 3 Jul 2018 14:06:25 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf0-x236.google.com with SMTP id u202-v6so1715022lff.9 for ; Tue, 03 Jul 2018 07:06:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=v4YdNpGo5eTww084qOR9pZjIqbjpaL4JLHsLtjcYaqw=; b=HqY863BmtgRHO0Wq4bFunOKy/GlLNn8peCnkvc+XQsIOlZcuMQVkiKk9OWrgQqOsMd KaGcj2H82i+s2TStYOXZ9pk7MusRh3EOUHdPOALCGC3Qe2jWGV8LGgjvOmm13fjmmYJb XI7h8e4aQGCaz6cihS02FyxTT+blNyKUPr34m4YWv7UCAnmG3fB+jBO09qk/7bcoKACT A/6QFqIU076UgDiwxo3p9lzHdYKF1vZ4ObpvlUNERgidZLb9md15HLsp+TQSeUer/8eA RtQu9FDZIkNuLfPAWHQz/4EZOE3d2o21s3rFU93h6tPVcY58lh02fyf1TUg4o0CrC+af G6DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=v4YdNpGo5eTww084qOR9pZjIqbjpaL4JLHsLtjcYaqw=; b=fwDSX0ZR6jCmDTb3VOxviQpj/Ixr5tATT3e1/7DCuJcl2T6GQNkqYRyBSHXA1jcaE4 gStQm6/WxcYqMPliArX1CutKw5ftgrCTCeY/AClBYt5hMlxJYHy2v5HXVTjSQvln06CJ N6v9MUwnKYdOJXUx/GD5q45KshTPmFS1zL2Xvx4nt4eWeYDwN+Sso+BphkjxXdNwVDdi GBa+KQhVEorSoNYjku014ocF4bPNaFhpkMW4Pwc6MuHqNBXST9y/CJBQcOVR7gYQiR5M ZcTTXmLr8FkFSbnQ+XFo29vRRwlSPuInjoTRbV7/iiibBX1ZlxXXuey0pi+wfzBE+lVt hjkg== X-Gm-Message-State: APt69E1sv1FOJlLXboC7eC6Z4jMh6aBfwwZzKkkdvTY1vQA8WR6CDH3a I0AXk7BScoNcdgcMPIq6ifk/yI48kV3Qpre2G48= X-Google-Smtp-Source: AAOMgpcxluot58m9uzrFun9yvHV4R3BRDYOa1pHq2SK3kT7Qs+K6OjpvlKZexk7/92jKf5h6VXjvlTF5JgGFr/qdfEU= X-Received: by 2002:a19:eac1:: with SMTP id y62-v6mr7109314lfi.138.1530626783342; Tue, 03 Jul 2018 07:06:23 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 2002:ab3:1b91:0:0:0:0:0 with HTTP; Tue, 3 Jul 2018 07:06:22 -0700 (PDT) In-Reply-To: References: From: Alan Somers Date: Tue, 3 Jul 2018 08:06:22 -0600 X-Google-Sender-Auth: 45xipcwAtgDccMcAMAxGnRPFW0E Message-ID: Subject: Re: problems with SAS JBODs 2 To: Oliver Sech Cc: FreeBSD-scsi Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2018 14:06:26 -0000 On Tue, Jul 3, 2018 at 6:28 AM, Oliver Sech wrote: > Hi! > > I use FreeBSD with for a large ZFS pool (over 1PB) and I recently > encountered a lot of problems with the JBODs. Generally everything works > fine until I replug the shelves. > > When I start with a clean system and attach a single shelf every thing > seems fine. > -> 44 disks show up, I can use the enclosure services (sesutil) and the > system continues to run without problems. > Once I disconnect the SAS cable, wait until all devices disapear and > reconnect I get all sorts of problems. > -> a random number of disks shows up and the enclosure "ses" do not show up > Once I restart the system I can start over again. > > On the server with the large pool there are only certain ports on the HBA > that I can use, otherwise disks will be missing after a reboot and my ZFS > pool won't go online. > I tried different firmware on the HBA. I tried the mpr.ko module from the > broadcom site. (I replaced the one in /boot/kernel?) > I tested all the things above with a Linux as OS and everything seems to > work. > > > Is there anything I'm missing? A command that can reset the SAS components? > > > FreeBSD version: 11.1-RELEASE-p11 > HBA: broadcom lsi 9305-16e (latest firmware) > JBOD:SC847E2C-R1K28JBOD (two expanders, internally daisy chained) > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > 1) Are the expanders daisy chained? Some SAS expanders don't work reliably when daisy chained. Best to direct connect each one to the server. 2) Are the expanders connected in multipath or single path? You need geom_multipath if you're going to do that. 3) Are you attempting to use wide ports (two SAS cables connecting each expander to the HBA). If do, you'll need to make sure that each pair of SAS cables goes to the same HBA chip (not merely the same card, as some cards contain two HBA chips). 4) Are you trying to remove an expander while ZFS is active on that expander? That will suspend your pool, and ZFS doesn't always recover from a suspended state. -Alan