From owner-freebsd-questions@freebsd.org Fri Dec 13 23:26:22 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B02FA1D9CF6 for ; Fri, 13 Dec 2019 23:26:22 +0000 (UTC) (envelope-from matt@conundrum.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47ZRcs5c05z4f4Y for ; Fri, 13 Dec 2019 23:26:21 +0000 (UTC) (envelope-from matt@conundrum.com) Received: by mail-lj1-x22b.google.com with SMTP id m6so491045ljc.1 for ; Fri, 13 Dec 2019 15:26:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=conundrum-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=57NJUXU/Zb5iMeXAn48dC7M8cZObn8vWc6uFyMFuSS4=; b=kD1aAgNOOOhv7UsyHLcWv+AkIdRSEoBHV/CFxBvUgAB8SV613fw2WG+xyxw8RP5ZhO pV5eloM5rsfTQbT1nCKYW8pW12xSoPKOWco/zeNlg/1QOUt4BAjYXXNsA0p0HSc2wqZZ ltuT6p7haxuAfNTWIEbokvLMdcMiFZP9FZSfXzU+8BlXKmjlld04YwNe/7mhy+5DlZlR t0rQlQ8mlkZ1HA1uehYHpK97Z+Nfqv3/418//rDpKtjpO82lFtG5fWJcSYN9KTrHSiYJ FHGlO9xupuim8AOGEGDJxB9KhdpbN0v6da0udrj+FMMnpBXwLx+H88oHOJqOPqZUl2NZ rzRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=57NJUXU/Zb5iMeXAn48dC7M8cZObn8vWc6uFyMFuSS4=; b=JywTgHz2NY48ZfNMeyK1dMKFz2cPkM/VqrxIaNyiYfmFPRJo9p2aRQgBYB8qJoXysP 5rlASq2YeQ6rDBrnFRQEjwDpHTD/ELh9NXziKfaD4GzqOXjueceCdIXkozh0kU1W74mH XA+j0JH3qjcCSRLwYRYRVU5wuDj1sM0+ovCLnuNoNQXMzCdXIspkxlBO6PEw7rqbJdM6 MofBULticuYVoBUNZ6Ym9eK1e3JGTgUIPwB9yaKwnd4eKchn4eeEJN6efkrwAe3orVBZ 7oKBrB5m304nFZURmSuiAy7boG+u+v9WNAXbtao4INOyzirPF50+cRdg2+rHDZqFF6rM qD2Q== X-Gm-Message-State: APjAAAWhKRr3N9qwD3XoRM4SkwsKvB9MWgX+qtT2mx2T1wTohxwmRS7u bCCcviPCSPxuKsa+6HKbCGPRem53f1WiVE8ecHXseNZttOe8LA== X-Google-Smtp-Source: APXvYqwswmzPCvOQlfAmwMkNq5dMK5TOwu2H+ry734VXtvKXShsZUSn6DPlO6oghxxmAlCZYgH5jt+TLMcOh5Zxwags= X-Received: by 2002:a2e:6f19:: with SMTP id k25mr11041761ljc.84.1576279579701; Fri, 13 Dec 2019 15:26:19 -0800 (PST) MIME-Version: 1.0 References: <28a92269-832b-61d0-3d25-68be2439dd9c@pp.dyndns.biz> In-Reply-To: From: Matthew Pounsett Date: Fri, 13 Dec 2019 18:26:08 -0500 Message-ID: Subject: Re: Root volume renumbered unexpectedly, no longer boots To: =?UTF-8?Q?Morgan_Wesstr=C3=B6m?= Cc: freebsd-questions@freebsd.org X-Rspamd-Queue-Id: 47ZRcs5c05z4f4Y X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=conundrum-com.20150623.gappssmtp.com header.s=20150623 header.b=kD1aAgNO; dmarc=none; spf=none (mx1.freebsd.org: domain of matt@conundrum.com has no SPF policy when checking 2a00:1450:4864:20::22b) smtp.mailfrom=matt@conundrum.com X-Spamd-Result: default: False [-5.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[conundrum-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[conundrum.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[conundrum-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[b.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.79)[ip: (-9.32), ipnet: 2a00:1450::/32(-2.67), asn: 15169(-1.91), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Dec 2019 23:26:22 -0000 On Fri, 13 Dec 2019 at 17:31, Morgan Wesstr=C3=B6m < freebsd-database@pp.dyndns.biz> wrote: > > The SSD is on the mainboard controller. I have no idea what the SATA > > controller's original mode was, but just now it was set to IDE. I trie= d > > switching it to AHCI, but that didn't improve anything, and generated a > new > > "AHCI BIOS not installed" error during boot, so I switched it back to > IDE. > > Either of the RAID settings seem like bad choices, since I want direct > > access to the physical drives for ZFS. > > I've seen systems that emulate 2-port master/slave controllers in > IDE/Legacy mode and renumber drives differently in that mode than in > AHCI mode. > Yeah.. I didn't see any difference moving back and forth though. > > That BIOS setting only affects the mainboard controller and with a 24 > disk ZFS pool I assume those disks are connected to some other > controller. Regardless, I'd stay away from the RAID mode too though > since there's a risk of overwriting some sectors of your system disk. It > would've be interesting to know if that setting has changed though... > 48 disks, actually. :) Half of those are on an external JBOD connected via an LSI FC controller. This server is significantly older than my association with it, so I'm uncertain about how the internal 24 drives are connected. If it helps, dmesg only reports ses0 and ses1 drivers. The boot disk and the first 24 ZFS drives are all on ses0. I think that implies only two controllers in use, not three. Is there any indication that the BIOS settings were reset during the > disk swap? (Date/time being way off for example). None that I can see. And, da4 vanished from the BIOS list of bootable devices several reboots after powering the machine back on, so I don't have any reason to think that was directly related to powering the machine off, or to the drive reordering. Its disappearance was coincident with booting the system off a USB installer, but timing is the only connection I can think of between those two things. Before anyone goes there... I did not start the installer, so I have no reason to think it has modified da4's MBR. I'm only booting off that stick to get to a usable shell. The only symptom of a problem that resulted during the power-off/drive swap was that da0 moved to da4 in the first place, and that trying to boot ufs:/dev/da4p2 from the boot prompt seems to fail. > Did you verify that > the SSD is still connected to the lowest numbered port? ATA_STATIC_ID > was removed back in 2015 if I remember correctly so unless you run an > ancient FreeBSD version I wouldn't rely on that mechanism. > Was it? Okay.. I checked the options file in the 11.2 source code earlier today and it still appears in that file, so I thought it was still in use. I haven't had cause to manually compile a kernel since about 2009, so I'm a bit out of date on what the options are. The boot drive wasn't moved... it's mounted inside the chassis, and we were swapping drives in the front-facing array of the JBOD (not even in the main chassis). So, at least physically, it's still connected to the same port. Looks like I was wrong about it being an SSD, but I think that's beside the point. dmesg currently reports the following things about da4 (hand-retyped, not cut and pasted, because I'm on a Java console): ses0: da4,pass4: Element descriptor: 'Slot 01' ses0: da4,pass4: SAS Device Slot Element: 1 Phys at Slot 0 > > Perhaps you can tell us what kind of system this is in case someone on > the list has a similar system and know its quirks. Also, if you're > running a GENERIC kernel or not. > Ah, sorry. I thought I mentioned that this is running 11.2-p7 GENERIC. I apparently forgot to mention the generic part. It has been progressively upgraded over the years. My guess is that the last time it saw a completely fresh install was sometime around the 9.x era, but that's just a guess. The system mainboard is a Supermicro X8DT3. The BIOS is significantly out of date (1.1 dated 2010.. current is 2.2) but we haven't had any BIOS related issues to cause us to want to update. More=E2=80=94possibly relevant=E2=80=94hand transcribing from dmesg: mps0: port 0xd000-0xd0ff mem 0xfae3c000-0xfa3ffff,0xfae40000-0xfae7ffff irq 16 at device 0.0 on pci2 mpt0: port 0xfc000-0xc0ff mem 0xfabec000-0xfabeffff,xfabf0000-0xfabfffff irq 16 at device 0.0 on pci5 ses0: Fixed Enclosure Services SPC-3 SCSI device ses1: Fixed Enclosure Services SPC-3 SCSI device