From owner-freebsd-questions@freebsd.org Sat Dec 14 11:56:36 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6FD611CB06C for ; Sat, 14 Dec 2019 11:56:36 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47ZmGW4dZmz4Jjk for ; Sat, 14 Dec 2019 11:56:35 +0000 (UTC) (envelope-from tomek@cedro.info) Received: by mail-ot1-x333.google.com with SMTP id 59so2536379otp.12 for ; Sat, 14 Dec 2019 03:56:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=xlNY5iZEW4UHPdu72BCkoHxhgAelmgQTienHVYRN0EA=; b=ckieyI3BNS4R4Ar+uE0FQDinoadTk8M3JyIbrOqvUf9wQh65LB+NYi+vEdf7e2z7rc /xrBRQPdm3UntIdXOmxwNDJyQSmua/1fWg0P+7aZHhgIWY4wfbPMDm29tk0VhcEJ+HZ1 wjNjzlNr0xSCwACjSLNpF+EBT2kBgkCzwaLe1MPMAAE7YLHMzGPEpIqkxM+WybTA6W9A q9BMKmajfqQOOYgeKrK2k1X3UGHI9GGVYSJsu6GWBRTQa9e1Dm5QyNbJbRtfUgImQq9E XiHqGbTG+IjXmJHc0Cp2Mf24N/t7bVAgU3qqTXJizMlXrydfFwFfMMn5bqJDL9LXQ/jJ HLvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=xlNY5iZEW4UHPdu72BCkoHxhgAelmgQTienHVYRN0EA=; b=pAvohocun2W8Kv/fSgymISpXRiWXlqdhbpyiRNQAsQdChLV4rDGg8SQlu2OXuzeOtc 1c8QVNVgIjwO7aIZWExsOAneIxJyQ/zGWLA6TZQ7/bIAHawk0LLuP5afFhIBWQ76h18R JDdmPk1L/+QEqajSVYEmWB7aPU0W3zceF06z8ln+SeBKsG3NqafeyNRLB8g7QoJj8q7C rNK0SjIqN28RY236R/MCx3H5NoaBkW5fMQ4//kNJOPC4XgxRn2j82KUXGMG6fSNRqd6M Zp70LkavhXBsO+kKGGXe7jxWlUT6Mhw1yn7EvBsaf7tuue8hAYHfQLg6UTm9YZwzUxeh tn+g== X-Gm-Message-State: APjAAAXQwg9e1IAhYLwjB+EFvqbd7pRF5+FjF9MT52iV0ERARtL33eg6 TsCVuzjo/jVlbVSSIeMx1lqwmYNcpwM= X-Google-Smtp-Source: APXvYqxHvPgjJV+msX0/uqVWzqLSLRDThsI/TB7l6+zn82WAp07lxyhK0MsFYPgQYX0LIe4P8tC0lA== X-Received: by 2002:a9d:5888:: with SMTP id x8mr19219975otg.361.1576324593530; Sat, 14 Dec 2019 03:56:33 -0800 (PST) Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com. [209.85.210.43]) by smtp.gmail.com with ESMTPSA id h132sm4400827oif.44.2019.12.14.03.56.32 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 14 Dec 2019 03:56:32 -0800 (PST) Received: by mail-ot1-f43.google.com with SMTP id k14so2603443otn.4 for ; Sat, 14 Dec 2019 03:56:32 -0800 (PST) X-Received: by 2002:a9d:6a50:: with SMTP id h16mr20726051otn.267.1576324592339; Sat, 14 Dec 2019 03:56:32 -0800 (PST) MIME-Version: 1.0 References: <28a92269-832b-61d0-3d25-68be2439dd9c@pp.dyndns.biz> <7a20b370-6dac-3caf-3f45-fe485c56b372@pp.dyndns.biz> In-Reply-To: <7a20b370-6dac-3caf-3f45-fe485c56b372@pp.dyndns.biz> From: Tomasz CEDRO Date: Sat, 14 Dec 2019 12:56:20 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Root volume renumbered unexpectedly, no longer boots To: =?UTF-8?Q?Morgan_Wesstr=C3=B6m?= Cc: FreeBSD Questions Mailing List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47ZmGW4dZmz4Jjk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=ckieyI3B; dmarc=none; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::333) smtp.mailfrom=tomek@cedro.info X-Spamd-Result: default: False [-4.53 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[cedro.info]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[cedro.info:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.23)[ip: (-7.00), ipnet: 2607:f8b0::/32(-2.20), asn: 15169(-1.91), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Dec 2019 11:56:36 -0000 On Sat, Dec 14, 2019 at 11:05 AM Morgan Wesstr=C3=B6m wrote: > > 48 disks, actually. :) Half of those are on an external JBOD connecte= d > > via an LSI FC controller. This server is significantly older than my > > association with it, so I'm uncertain about how the internal 24 drives = are > > connected. If it helps, dmesg only reports ses0 and ses1 drivers. The > > boot disk and the first 24 ZFS drives are all on ses0. I think that > > implies only two controllers in use, not three. > > > > Yeah, I see now that your mainboard has a SAS controller in addition to > the Intel ICH10 controller. That BIOS setting I suggested would only > affect disks attached to the Intel controller. The SAS controller should > have it's own option BIOS settings, available through some key combo > during boot. I have no experience with these controllers though so I > couldn't tell if something in their settings suddenly made them report > disks in a different order. Hopefully someone who knows how they work > will read this and help you. I though that too. Also the first boot disk seems attached to that controll= er. It depends on how the disks were replaced. One by one or all of them at onc= e? It would be best to go to RAID/SAS BIOS and see settings there. Disk order should not change. Usually RAID BIOS will show warnings about missing disks or RAID state change but it should keep the disk order. Except when all disks are set to create one large RAID array. Then controller decides on how to restore the data on the disks. Also this takes some time until array is can get back to full operations. On a HP ProLiant server I also had to boot special CD with RAID utility to set up this controller in a way that was not possible from RAID BIOS screen. Anyways with this kind of play it is always best to create a backup in the first place. Good luck! :-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info