From nobody Wed Oct 18 14:30:09 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S9YCp5tV9z4x5wy for ; Wed, 18 Oct 2023 14:30:14 +0000 (UTC) (envelope-from azathanai@icloud.com) Received: from qs51p00im-qukt01072702.me.com (qs51p00im-qukt01072702.me.com [17.57.155.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4S9YCp0BZTz3LWC for ; Wed, 18 Oct 2023 14:30:14 +0000 (UTC) (envelope-from azathanai@icloud.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=icloud.com header.s=1a1hai header.b=UuyWFw0F; spf=pass (mx1.freebsd.org: domain of azathanai@icloud.com designates 17.57.155.17 as permitted sender) smtp.mailfrom=azathanai@icloud.com; dmarc=pass (policy=quarantine) header.from=icloud.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1697639412; bh=1wXGT2JMokfH6AeNStK/56gEQ9U6iSP8MBorwHdO+Gw=; h=From:To:Subject:Date:Message-ID:Content-Type:MIME-Version; b=UuyWFw0F/2bGEg1N1zFoKTjSLd4855Y72dZ4jSbmbm4NXaFSsOuenHZ1BTTeZPt8h reuMcuXSTVJkp2hc/KmeagW6nXS5LIHWRU/tvVD06HcAfJG2vK+CNsanGBIeW7ISNY v7XdGRMdiPSf/6Aw2FJi/ozjwS/xbE3fRGn5th7ezg63TnHJvdQcd60aMckWaUmbJg ikwGNKOKah9v8INcPWrE8Ilt/aPNT5cIzGAU/ZCgiL2S1vYUDBs/6cdCP8N0pZAV7f UaoPKSMv0g/FgFShjWyMe1UJBPcCLyZqCMLrOnmQHKSGdS+DSdrmoMuRe7lB762xiH PrJS4C4HON0EQ== Received: from AM6PR02MB4471.eurprd02.prod.outlook.com (qs51p00im-dlb-asmtp-mailmevip.me.com [17.57.155.28]) by qs51p00im-qukt01072702.me.com (Postfix) with ESMTPSA id 552641680488; Wed, 18 Oct 2023 14:30:11 +0000 (UTC) From: "azathanai@icloud.com" To: Miroslav Lachman <000.fbsd@quip.cz> CC: "freebsd-hackers@FreeBSD.org" Subject: Re: Question of installing over SAS controller and USB flash Thread-Topic: Question of installing over SAS controller and USB flash Thread-Index: AQHZ+7gox8b2WXJ7x0SXInCUTrHiVLBDh8MAgADxNbGAACMPAIAEK6+cgAbeJC4= X-MS-Exchange-MessageSentRepresentingType: 1 Date: Wed, 18 Oct 2023 14:30:09 +0000 Message-ID: References: <84c06802-a6d9-fa5d-8dab-9ebe18074321@quip.cz> In-Reply-To: Accept-Language: en-US, bg-BG Content-Language: en-US X-MS-Has-Attach: X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: X-MS-Exchange-Organization-RecordReviewCfmType: 0 msip_labels: Content-Type: multipart/alternative; boundary="_000_AM6PR02MB44715B177C56DDBB0CC4CD4AA5D5AAM6PR02MB4471eurp_" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-Proofpoint-GUID: xNs8NxVarmvGvqZDbkleLNexQxAqGuYV X-Proofpoint-ORIG-GUID: xNs8NxVarmvGvqZDbkleLNexQxAqGuYV X-Proofpoint-Virus-Version: =?UTF-8?Q?vendor=3Dfsecure_engine=3D1.1.170-22c6f66c430a71ce266a39bfe25bc?= =?UTF-8?Q?2903e8d5c8f:6.0.138,18.0.790,17.0.605.474.0000000_definitions?= =?UTF-8?Q?=3D2022-01-12=5F02:2020-02-14=5F02,2022-01-12=5F02,2020-01-23?= =?UTF-8?Q?=5F02_signatures=3D0?= X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2310180119 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.11 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[icloud.com:d:+,icloud.com:s:+]; FROM_DN_EQ_ADDR(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.91)[-0.910]; DMARC_POLICY_ALLOW(-0.50)[icloud.com,quarantine]; RWL_MAILSPIKE_VERYGOOD(-0.20)[17.57.155.17:from]; R_DKIM_ALLOW(-0.20)[icloud.com:s=1a1hai]; R_SPF_ALLOW(-0.20)[+ip4:17.57.155.0/24]; RCVD_IN_DNSWL_LOW(-0.10)[17.57.155.17:from]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_ENVFROM(0.00)[icloud.com]; ASN(0.00)[asn:714, ipnet:17.57.155.0/24, country:US]; FREEMAIL_FROM(0.00)[icloud.com]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[icloud.com:dkim]; DKIM_TRACE(0.00)[icloud.com:+]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4S9YCp0BZTz3LWC --_000_AM6PR02MB44715B177C56DDBB0CC4CD4AA5D5AAM6PR02MB4471eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable OK. I've flashed the card with the mini-memstick and now it boots and starts th= e installation for FreeBSD, which is fine. However when it goes on partitio= ning of the drives, I'm not sure what to do. 1. I can mark all drives on the sas controller for auto-zfs, but if the = flash card is not "included" then the machine will reboot to the installati= on again. 2. I can include the flash card in the installation of auto-zfs, but tha= t means to have the card as root space and will not be able to use the big = drives. 3. I need to manually setup the partitions, having the efi and /boot on = the flash card and eventually mirrored on all other drives and have the / o= n the SAS drives, but doesn't have a clue how to do that. Is 3 the correct way to proceed and is any help on that possible? Cyberly yours, Nikolay Balov ________________________________ From: owner-freebsd-hackers@freebsd.org = on behalf of azathanai@icloud.com Sent: Wednesday, October 18, 2023 09:14 To: Miroslav Lachman <000.fbsd@quip.cz> Cc: freebsd-hackers@FreeBSD.org Subject: Re: Question of installing over SAS controller and USB flash Hey Miroslav, thank you for the help. I've tried with the first part of your suggestion - gpart bootcode, but tha= t was not enough. I suppose I do need the /boot on the flash card, as I've = installed Linux and put the /boot "folder" on the flash and everything was = working flawlessly. I'll try your suggestion with the memstick. Just a question there. Do I need to make the "gpart bootcode" part if I'm creating the memstick, o= r the memstick has everything needed to boot from it, i.e. it has the bootc= odes? Thanks again for your help and willingness to waste your time on a noob que= stions! Cyberly yours, Nikolay Balov ________________________________ From: owner-freebsd-hackers@freebsd.org = on behalf of Miroslav Lachman <000.fbsd@quip.cz> Sent: Wednesday, October 11, 2023 16:49 To: azathanai@icloud.com ; Warner Losh Cc: freebsd-hackers@FreeBSD.org Subject: Re: Question of installing over SAS controller and USB flash On 11/10/2023 14:02, azathanai@icloud.com wrote: > Hey Warner, > the box is very featureLESS (an old Xyratex HS-1235T box, as far as I > researched it). > T2029_QR v1.3.ai (hackspace.org.uk) > > (5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.com) > > > Through the POST process and the BIOS there is no option to get into the > SAS controller and enable its BIOS. I would love to install FreeBSD > directly on the drives and not to deal with the flash card, but I > suppose the lack of controller BIOS is the reason the box comes with > reader and card directly. [..] This depends on at what stage the computer starts to see the disk behind the SAS controller. I had an old HP server that only saw one disk out of four in the BIOS, so it couldn't boot from RAIDZ, but it was enough to make a 10GB partition on each disk as a 4 way ZFS mirror, boot from that and use the rest of each disk for the main RAIDZ storage (4x 8TB). Maybe it would have been enough to use a USB flash drive for the freebsd-boot partition in GPT scheme and then continue boot from the disks on the SAS controller. gpart create -s GPT da0 gpart bootcode -b /boot/pmbr da0 gpart add -b 40 -s 512 -t freebsd-boot da0 gpart bootcode -p /boot/zfsgptboot -i 1 da0 As long as the computer boots from the installer CD, FreeBSD with RAIDZ can be installed on the disks behind the SAS controller without any modification, only the USB flash drive must provide the boot. If the USB flash drive must also contain a UFS /boot partition, then copy an image that can be downloaded from the web as a "mini-memstick" to the USB flash drive https://www.freebsd.org/releases/13.2R/announce/ (use dd according to the instructions) Or again, create a complete installation from the FreeBSD installer on the USB flash drive. But the installation to big drives with RAIDZ should be done normally from the FreeBSD installer. No modifications needed. Kind regards Miroslav Lachman --_000_AM6PR02MB44715B177C56DDBB0CC4CD4AA5D5AAM6PR02MB4471eurp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
OK.
I've flashed the card with the mini-memstick and now it boots and starts th= e installation for FreeBSD, which is fine. However when it goes on par= titioning of the drives, I'm not sure what to do.
  1. I c= an mark all drives on the sas controller for auto-zfs, but if the flash car= d is not "included" then the machine will reboot to the installation again. 
  2. I c= an include the flash card in the installation of auto-zfs, but that means t= o have the card as root space and will not be able to use the big drives.
  3. I n= eed to manually setup the partitions, having the efi and /boot on the flash= card and eventually mirrored on all other drives and have the / on the SAS drives, but doesn't have a clue how= to do that.
Is 3 the correct way to proceed and is any help = on that possible?


Cyberly yours,
Nikolay Balov
From: owner-freebsd-hackers= @freebsd.org <owner-freebsd-hackers@freebsd.org> on behalf of azathan= ai@icloud.com <azathanai@icloud.com>
Sent: Wednesday, October 18, 2023 09:14
To: Miroslav Lachman <000.fbsd@quip.cz>
Cc: freebsd-hackers@FreeBSD.org <freebsd-hackers@FreeBSD.org><= br> Subject: Re: Question of installing over SAS controller and USB flas= h
 
Hey Miroslav,
thank you for the help.
I've tried with the first part of your suggestion - gpart bootcode, but tha= t was not enough. I suppose I do need the /boot on the flash card, as I've = installed Linux and put the /boot "folder" on the flash and every= thing was working flawlessly.
I'll try your suggestion with the memstick. Just a question there.
Do I need to make the "gpart bootcode" part if I'm creating the m= emstick, or the memstick has everything needed to boot from it, i.e. it has= the bootcodes?

Thanks again for your help and willingness to waste your tim= e on a noob questions!

Cyberly yours,
Nikolay Balov


From: owner-freeb= sd-hackers@freebsd.org <owner-freebsd-hackers@freebsd.org> on behalf = of Miroslav Lachman <000.fbsd@quip.cz>
Sent: Wednesday, October 11, 2023 16:49
To: azathanai@icloud.com <azathanai@icloud.com>; Warner L= osh <imp@bsdimp.com>
Cc: freebsd-hackers@FreeBSD.org <freebsd-hackers@FreeBSD.org= >
Subject: Re: Question of installing over SAS controller and USB= flash
 
On 11/10/2023 14:02, azathanai@icloud.c= om wrote:
> Hey Warner,
> the box is very featureLESS (an old Xyratex HS-1235T box, as far as I<= br> > researched it).
> T2029_QR v1.3.ai (hackspace.org.uk)
> <https://wiki.lo= ndon.hackspace.org.uk/w/images/b/b2/HS-1235T-ATX_Quick_Reference_Sheet.pdf<= /a>>
> (5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.co= m)
> <https://ww= w.reddit.com/r/homelab/comments/8loge6/xyratex_094403703_motherboard_myster= y_solved/>
>
> Through the POST process and the BIOS there is no option to get into t= he
> SAS controller and enable its BIOS. I would love to install FreeBSD > directly on the drives and not to deal with the flash card, but I
> suppose the lack of controller BIOS is the reason the box comes with > reader and card directly.

[..]

This depends on at what stage the computer starts to see the disk behind the SAS controller. I had an old HP server that only saw one disk out of four in the BIOS, so it couldn't boot from RAIDZ, but it was enough to
make a 10GB partition on each disk as a 4 way ZFS mirror, boot from that and use the rest of each disk for the main RAIDZ storage (4x 8TB).

Maybe it would have been enough to use a USB flash drive for the
freebsd-boot partition in GPT scheme and then continue boot from the
disks on the SAS controller.

gpart create -s GPT da0
gpart bootcode -b /boot/pmbr da0
gpart add -b 40 -s 512 -t freebsd-boot da0
gpart bootcode -p /boot/zfsgptboot -i 1 da0

As long as the computer boots from the installer CD, FreeBSD with RAIDZ
can be installed on the disks behind the SAS controller without any
modification, only the USB flash drive must provide the boot.

If the USB flash drive must also contain a UFS /boot partition, then
copy an image that can be downloaded from the web as a "mini-memstick&= quot;
to the USB flash drive
https://www.freebsd.org/releases/13.2R/announce/ (use dd= according to
the instructions)
Or again, create a complete installation from the FreeBSD installer on
the USB flash drive.

But the installation to big drives with RAIDZ should be done normally
from the FreeBSD installer. No modifications needed.

Kind regards
Miroslav Lachman


--_000_AM6PR02MB44715B177C56DDBB0CC4CD4AA5D5AAM6PR02MB4471eurp_--