From owner-freebsd-scsi@freebsd.org Tue Jul 10 15:48:31 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A4C71032E45 for ; Tue, 10 Jul 2018 15:48:31 +0000 (UTC) (envelope-from stephen.mcconnell@broadcom.com) Received: from mail-it0-x233.google.com (mail-it0-x233.google.com [IPv6:2607:f8b0:4001:c0b::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 073617CA93 for ; Tue, 10 Jul 2018 15:48:30 +0000 (UTC) (envelope-from stephen.mcconnell@broadcom.com) Received: by mail-it0-x233.google.com with SMTP id w16-v6so9825749ita.0 for ; Tue, 10 Jul 2018 08:48:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to; bh=l/mkfNvLY1riVT1IboUU7yHXw1OXCvBL2Z4sdcKd2EA=; b=JMgvOBwru//nzHDLHO6QYrKBFY8KnkqmqOU5jau/BdDubZyCtl99l+YY1FeNsinK5E akbO22mdBrTM+2TP0+c48nhl+sA0Oy5/kwD/ADb+Fio8rbbjSrZeHczUfMDLhcqgs0As FiAsgzF7N0tWLfbruJZ8gIf1ZrSO84q0J56GkIcX8iHuabGUJS026GwUrBo8827SC+jY WVSj0bj6HJC3bGU7Wr2z0pexYloGMOze3MDVEwpDHVaKoUsao5rWwuW1IVak+OLjqCdo W4KfQ5STLpw0DlSk6gqtUeEaEzjMrxFXkJAgQuAv+czUDGQUN5txcqA68XzPUyJeHaHy chhA== X-Gm-Message-State: APt69E0zwp31d343f679HADWRuS9x07NBslVOZ2Panh0YfVXU/McXzdy zpkJlQ8uWLBDTfNPigyybqZ+VUC88cubPt+ZQTCxhA== X-Google-Smtp-Source: AAOMgpeM8ZUaISuH/DsKYBiREJ9buYTg3tMVniyttx/2GVEaUfTALv/6ultkHTt85vaUdJFc6rjcnST13VjQmkAbgpI= X-Received: by 2002:a24:edce:: with SMTP id r197-v6mr12288983ith.23.1531237710325; Tue, 10 Jul 2018 08:48:30 -0700 (PDT) From: Stephen Mcconnell References: <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> 3caf8ccd6fde8cfc4db25bae5327c46b@mail.gmail.com In-Reply-To: 3caf8ccd6fde8cfc4db25bae5327c46b@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHJZ/UmTT9Y1rodqvzH7TRwbPT2YALnpLW+Ap4aqgkCOUyURaRf/K0AgAAFNrA= Date: Tue, 10 Jul 2018 09:48:29 -0600 Message-ID: <0af047d477d15ec364140653bd967c89@mail.gmail.com> Subject: RE: problems with SAS JBODs 2 To: Oliver Sech , FreeBSD-scsi Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2018 15:48:31 -0000 Ken, I looked at the logs and I don't see anything in them that suggests that the driver is not adding any of the devices. In fact, I don't see anything that looks strange at all. This looks like a different problem than the other one you mentioned. What do you think? Steve > -----Original Message----- > From: Stephen Mcconnell [mailto:stephen.mcconnell@broadcom.com] > Sent: Tuesday, July 10, 2018 9:28 AM > To: 'Oliver Sech'; 'FreeBSD-scsi' > Subject: RE: problems with SAS JBODs 2 > > Hi Oliver, I can't get to your links. Can you try to send the logs in > another > way? > > Steve > > > -----Original Message----- > > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > > scsi@freebsd.org] On Behalf Of Oliver Sech > > Sent: Tuesday, July 10, 2018 9:14 AM > > To: FreeBSD-scsi > > Subject: Re: problems with SAS JBODs 2 > > > > I tested a few additional things. I don't think this is a multipath, > > daisy > chain > > nor a SAS wide ports problem. > > I can reproduce the problem with just a single connection to an > > Expander/JBOD. > > > > Test: > > * physically disconnect all shelves > > * reboot system > > * connect one shelf via SAS cable > > * check number of disks (after a reboot everything always shows up) > > * disconnect the shelf and wait (geom disk list still shows most disks.) > > * connect the shelf (missing disks) > > > > Tested Hardware: > > * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e ( internal > daisy > > chain + wide links) > > * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e (straight HBA > <- > > > EXPANDER connection. (no wide links, no daisy chain)) > > * Supermicro SAS2 SC847E26-RJBOD1 + SAS3 LSI 9305-16e (internal > > daisy > > chain) > > * Promise SAS2 VTrak 830 + SAS3 LSI 9305-16e (straight HBA > > <-> > > EXPANDER connection.) > > > > > > > > On 07/04/2018 12:15 PM, Oliver Sech wrote: > > >> 1) Are the expanders daisy chained? Some SAS expanders don't work > > reliably > > >> when daisy chained. Best to direct connect each one to the server. > > > At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 > > lanes?). > > > Unfortunately the JBOD has 24 slots in the front and 20 in the back > > > and, > > those are connected via a internal SAS daisy chaining. > > > I could rewire and connect each backplane directly to the server, but > > unfortunately I do not have enough ports.. > > > > > > JOBD Model: Supermicro 847E2C-R1K28JBOD > > > > > >> 2) Are the expanders connected in multipath or single path? You need > > >> geom_multipath if you're going to do that. > > > See answer 1. There is a single path from the host to the first > > > expander. > > > > > >> 3) Are you attempting to use wide ports (two SAS cables connecting > each > > >> expander to the HBA). If do, you'll need to make sure that each pair > > >> of > > >> SAS cables goes to the same HBA chip (not merely the same card, as > some > > >> cards contain two HBA chips). > > > see 1. The last time I opened one of those JBODs there were 8 SAS > > > cables > > between the Front and Back expander. I assume that wide ports are being > > used. > > > (2 expanders per backplane as well) > > > > > >> 4) Are you trying to remove an expander while ZFS is active on that > > >> expander? That will suspend your pool, and ZFS doesn't always > > >> recover > > from > > >> a suspended state. > > > I'm testing with a new unused disk shelf that was never part of the > > > ZFS > > pool. There were > > > _______________________________________________ > > > freebsd-scsi@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > > To unsubscribe, send any mail to > > > "freebsd-scsi-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-scsi@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"