From owner-freebsd-doc@FreeBSD.ORG Wed Mar 3 04:30:26 2004 Return-Path: Delivered-To: freebsd-doc@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EBFFE16A4CE for ; Wed, 3 Mar 2004 04:30:26 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E19B843D1D for ; Wed, 3 Mar 2004 04:30:26 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i23CUQbv038647 for ; Wed, 3 Mar 2004 04:30:26 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i23CUQAl038646; Wed, 3 Mar 2004 04:30:26 -0800 (PST) (envelope-from gnats) Date: Wed, 3 Mar 2004 04:30:26 -0800 (PST) Message-Id: <200403031230.i23CUQAl038646@freefall.freebsd.org> To: freebsd-doc@FreeBSD.org From: Jean-Francois Dockes Subject: Re: docs/62834: Diskless operation handbook section update X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Jean-Francois Dockes List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2004 12:30:27 -0000 The following reply was made to PR docs/62834; it has been noted by GNATS. From: Jean-Francois Dockes To: freebsd-gnats-submit@FreeBSD.org Cc: Subject: Re: docs/62834: Diskless operation handbook section update Date: Wed, 3 Mar 2004 13:28:57 +0100 I have integrated initial remarks to the handbook advanced networking diskless section update. The revised text can be found at: http://perso.wanadoo.fr/dockes/diskless/network-diskless.html The unified diffs follow, or can be found at: http://perso.wanadoo.fr/dockes/diskless/udiffs.txt I really believe that the new version is better than the previous one (which I wrote). It corrects some errors and contains updated material for FreeBSD 5.x I would appreciate it if someone could find the time to review and possibly commit the new text. Regards, J.F. Dockes --- chapter.sgml.old Wed Mar 3 12:48:33 2004 +++ chapter.sgml Wed Mar 3 12:58:57 2004 @@ -2402,10 +2402,10 @@ diskless operation A FreeBSD machine can boot over the network and operate without a - local disk, using filesystems mounted from an NFS server. No system - modification is necessary, beyond standard configuration files. - Such a system is easy to set up because all the necessary elements - are readily available: + local disk, using filesystems mounted from an NFS server. No + system modification is necessary, beyond standard configuration + files. Such a system is relatively easy to set up because all + the necessary elements are readily available: There are at least two possible methods to load the kernel over @@ -2451,10 +2451,10 @@ There are many ways to set up diskless workstations. Many elements are involved, and most can be customized to suit local - taste. The following will describe the setup of a complete system, - emphasizing simplicity and compatibility with the - standard FreeBSD startup scripts. The system described has the - following characteristics: + taste. The following will describe variations on the setup of a + complete system, emphasizing simplicity and compatibility with + the standard FreeBSD startup scripts. The system described has + the following characteristics: @@ -2466,12 +2466,14 @@ configuration files overridden by ones specific to diskless operation or, possibly, to the workstation they belong to. The parts of the root which have to be - writable are overlaid with &man.mfs.8; filesystems. Any changes - will be lost when the system reboots. + writable are overlaid with &man.mfs.8; (FreeBSD 4.X) or + &man.md.4; (FreeBSD 5.X) filesystems. Any changes will be lost + when the system reboots. - The kernel is loaded by etherboot - , using DHCP (or BOOTP) and TFTP. + The kernel is transferred and loaded either with + etherboot, or PXE + as some situations may mandate the use of either method. @@ -2480,6 +2482,111 @@ other hosts. + All the information in this section has been tested + using FreeBSD releases 4.9-RELEASE-p2 and 5.2-RELEASE. The text is + primarily structured for 4.X usage. Notes have been inserted where + appropriate to indicate 5.X changes. + + + + + Background information + + Setting up diskless workstations is both relatively + straightforward and prone to errors. These are sometimes + difficult to diagnose for a number of reasons. For example: + + + Compile time options may determine the possible values + or syntax of runtime parameters. + + + Error messages are often cryptic or totally + absent. + + + + In this context, having some knowledge of the background + mechanisms involved is very useful to solve the problems that + may arise. + + Several operations need to be performed for a successful + bootstrap: + + + + The machine needs to obtain initial + parameters such as its IP address, executable file name, + server name, root path. This is done using the DHCP or + BOOTP protocols. DHCP is a compatible extension of BOOTP, + and uses the same port numbers and basic packet + format. + It is possible to configure a system to use only BOOTP. + The &man.bootpd.8; server program is included in the + base FreeBSD system. + + However, DHCP has a number of advantages over BOOTP (nicer + configuration files, possibility of using PXE, plus many + others not directly related to diskless operation), and + we shall describe mainly a DHCP configuration, with + equivalent exemples using &man.bootpd.8; when possible. + The sample configuration will use the ISC DHCP software + package (release 3.0.1.r11 was installed on the test + server: ). + + + + The machine needs to transfer one or several + programs to local memory. Either TFTP or NFS are used. The + choice between TFTP and NFS is a compile time option in + several places. A common source of error is to specify + file names for the wrong protocol: TFTP typically + transfers all files from a single directory on the server, + and would expect file names relative to this + directory. NFS needs absolute file paths. + + + + The possible intermediate bootstrap programs and the + kernel need to be initialized and executed. There are + several important variations in this area: + + + PXE will load &man.pxeboot.8;, + which is a modified version of the FreeBSD third stage + loader. The loader will obtain most parameters necessary + to system startup, and leave them in the kernel + environment before transferring control. It is possible to + use a GENERIC kernel in this case. + + + + etherboot, will directly + load the kernel, with less preparation. A kernel with + specific options will need to be built. + + + + PXE and etherboot work + equally well with 4.X systems. Because 5.X kernels + normally let the loader do more work for them, PXE is + preferred for 5.X systems. + If your BIOS and network + cards support PXE, you should probably use it. However, + it is still possible to start a 5.2 system with + etherboot + + + + + Finally, the machine needs to access its filesystems. NFS + is used in all cases. + + + + See also &man.diskless.8; + + Setup Instructions @@ -2491,31 +2598,7 @@ booting - There are two protocols that are commonly used to boot a - workstation that retrieves its configuration over the network: BOOTP - and DHCP. They are used at several points in the workstation - bootstrap: - - etherboot uses - DHCP (by default) or BOOTP (needs a configuration option) to - find the kernel. (PXE uses DHCP). - - The kernel uses BOOTP to locate the NFS - root. - - - - It is possible to configure a system to use only BOOTP. - The &man.bootpd.8; server program is included in the - base FreeBSD system. - - However, DHCP has a number of advantages over BOOTP (nicer - configuration files, possibility of using PXE, plus many others - not directly related to diskless operation), and we shall describe - both a pure BOOTP, and a BOOTP+DHCP configuration, with an - emphasis on the latter, which will use the ISC DHCP software - package. - + Configuration Using ISC DHCP @@ -2526,7 +2609,7 @@ The isc-dhcp server can answer both BOOTP and DHCP requests. - As of release 4.4, isc-dhcp + As of release 4.9, isc-dhcp 3.0 is not part of the base system. You will first need to install the net/isc-dhcp3-server port or the @@ -2536,7 +2619,9 @@ Once isc-dhcp is installed, it needs a configuration file to run, (normally named /usr/local/etc/dhcpd.conf). Here follows - a commented example: + a commented example, where host margaux + uses etherboot and host + corbieres uses PXE. default-lease-time 600; @@ -2556,9 +2641,16 @@ hardware ethernet 01:23:45:67:89:ab; fixed-address margaux.example.com; next-server 192.168.4.4; - filename "/tftpboot/kernel.diskless"; + filename "/data/misc/kernel.diskless"; option root-path "192.168.4.4:/data/misc/diskless"; } + host corbieres { + hardware ethernet 00:02:b3:27:62:df; + fixed-address corbieres.example.com; + filename "pxeboot"; + next-server 192.168.4.4; + option root-path "192.168.4.4:/data/misc/diskless"; + } } @@ -2573,26 +2665,37 @@ The - next-server directive designates - the TFTP server (the default is to use the same host as the - DHCP server). + next-server directive designates the TFTP + or NFS server to use for loading loader or kernel file (the + default is to use the same host as the DHCP server). - The - filename directive defines the file that - etherboot will load as a - kernel. - PXE appears to prefer a relative file - name, and it loads pxeboot, not the - kernel (option filename - "pxeboot"). - - + The + filename directive defines the file that + etherboot or PXE will load for the + next execution step. It must be specified according to the + transfer method used. etherboot can + be compiled to use NFS or TFTP. The FreeBSD port + configures NFS by default. + + + PXE uses TFTP, + which is why a relative file name is used here (this may + depend on the TFTP server configuration, but would be fairly + typical). Also, PXE loads pxeboot, not + the kernel. There are other interesting possibilities, like + loading pxeboot from a FreeBSD cdrom + /boot directory (as &man.pxeboot.8; can + load a GENERIC kernel, this makes it possible to use PXE to + boot from a remote cdrom). The - root-path option defines the path to - the root filesystem, in usual NFS notation. + root-path option defines the path to the + root filesystem, in usual NFS notation. When using PXE, it is + possible to leave off the host's IP as long as you do not + enable the kernel option BOOTP. The NFS server will then be + the same as the TFTP one. @@ -2605,7 +2708,7 @@ Here follows an equivalent bootpd - configuration. This would be found in + configuration (reduced to one client). This would be found in /etc/bootptab. Please note that etherboot @@ -2656,6 +2759,10 @@ of everything. Else refer to for information about ports and packages. + You can change the etherboot + configuration (ie: to use TFTP instead of NFS) by editing the + Config file in the etherboot source directory. + For our setup, we shall use a boot floppy. For other methods (PROM, or dos program), please refer to the etherboot documentation. @@ -2677,6 +2784,30 @@ + + Booting with PXE + + By default, the &man.pxeboot.8; loader loads the kernel via NFS. + It can be compiled to use TFTP instead by specifying the + LOADER_TFTP_SUPPORT option in + /etc/make.conf . See the comments in + /etc/defaults/make.conf (or + /usr/share/examples/etc/make.conf for 5.x + systems) for instructions. + + There are two other undocumented + make.conf options which may be useful for + setting up a serial console diskless machine: + BOOT_PXELDR_PROBE_KEYBOARD, and BOOT_PXELDR_ALWAYS_SERIAL (the + latter only exists on FreeBSD 5.x). + + + To use PXE when the machine starts, you will usually + need to select the Boot from network option + in your BIOS setup, or type a function key during the PC + initialization. + + Configuring the TFTP and NFS Servers @@ -2690,8 +2821,10 @@ diskless operation - You need to enable tftpd on the TFTP - server: + If any of the boostrap programs (PXE, pxeboot, or etherboot) + are configured to use TFTP, you need to enable + tftpd on the file server: + Create a directory from which tftpd @@ -2702,7 +2835,7 @@ Add this line to your /etc/inetd.conf: - tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot + tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot It appears that at least some PXE versions want the TCP version of TFTP. In this case, add a second line, @@ -2722,7 +2855,7 @@ location is set in both inetd.conf and dhcpd.conf. - You also need to enable NFS and export the + In all cases, you also need to enable NFS and export the appropriate filesystem on the NFS server. @@ -2738,7 +2871,7 @@ point and replace margaux with the name of the diskless workstation): - /data/misc -alldirs -ro margaux + /data/misc -alldirs -ro margaux corbieres Tell mountd to reread its configuration @@ -2759,23 +2892,45 @@ kernel configuration - Create a kernel configuration file for the diskless client - with the following options (in addition to the usual - ones): + If using etherboot, you need to + create a kernel configuration file for the diskless client + with the following options (in addition to the usual ones): - options BOOTP # Use BOOTP to obtain IP address/hostname - options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info - options BOOTP_COMPAT # Workaround for broken bootp daemons. + options BOOTP # Use BOOTP to obtain IP address/hostname + options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info - You may also want to use BOOTP_NFSV3 and - BOOTP_WIRED_TO (refer to LINT). - + You may also want to use BOOTP_NFSV3, + BOOT_COMPAT and + BOOTP_WIRED_TO (refer to + LINT). + + These option names are historical and slightly + misleading as they actually enable indifferent use of DHCP or BOOTP + inside the kernel (it is also possible to force strict BOOTP or + DHCP use). + Build the kernel (See ), - and copy it to the tftp directory, under the name listed - in dhcpd.conf. + and copy it to the place specified in + dhcpd.conf. + When using PXE, building a kernel with the above + options is not strictly necessary. Enabling them will cause more + DHCP requests to be issued during kernel startup, with a small + risk of inconsistency between the new values and those retrieved + by pxeboot in some special cases. The advantage of using them is + that the host name will be set as a side effect. Else you will + need to set the host name by another method, for example in a + client-specific rc.conf file. + + In order to be loadable with + etherboot, a 5.x kernel needs to have + the device hints compiled in. You would typically set the + following option in the configuration file (see the NOTES + configuration comments file). + hints "GENERIC.hints" + @@ -2798,6 +2953,13 @@ the place where the filesystem will be created (the DEST variable). + The clone_root script distributed + with FreeBSD 5.2 is out of date relative to the 4.x version, and + also needs a few specific 5.x adjustments. Please refer to PR + + misc/62417. + + Refer to the comments at the top of the script for instructions. They explain how the base filesystem is built, and how files may be selectively overridden by versions specific @@ -2812,39 +2974,68 @@ other examples in the diskless directory, they actually document a configuration method which is distinct from the one used by clone_root and - /etc/rc.diskless[12], which is a little - confusing. Use them for reference only, except if you prefer - the method that they describe, in which case you will need + the system startup scripts in /etc, which + is a little confusing. Use them for reference only, except if + you prefer the method that they describe, in which case you will need customized rc scripts. + + When using PXE, it may happen that the loader will + get a wrong idea of the root filesystem to be used by the + kernel (this will be the case if the cloned root + /etc/fstab is a copy of the server's, + as the client-specific version will not be visible by the + loader, which looks for hints there). + This will cause + error messages at the end of the kernel initialization, + when the kernel tries to mount the wrong root and fails, + or even actual errors if it succeeds because a local disk + happens to be present. + + It is possible to prevent this problem from happening + by inserting the following line in + /boot/loader.conf: + + vfs.root.mountfrom="" + + + + Configuring Swap If needed, a swap file located on the server can be - accessed via NFS. The exact bootptab - or dhcpd.conf options are not clearly - documented at this time. The following configuration - suggestions have been reported to work in some installations - using isc-dhcp 3.0rc11. + accessed via NFS. One of the methods commonly used to do this + has been discontinued in release 5.X. + + + NFS swap with FreeBSD 4.x + + The swap file location and size can be specified with + BOOTP/DHCP FreeBSD-specific options 128 and 129. Examples of + configuration files for isc-dhcp 3 or bootpd follow. + Add the following lines to dhcpd.conf: - # Global section - option swap-path code 128 = string; - option swap-size code 129 = integer 32; - - host margaux { - ... # Standard lines, see above - option swap-path "192.168.4.4:/netswapvolume/netswap"; - option swap-size 64000; - } + # Global section + option swap-path code 128 = string; + option swap-size code 129 = integer 32; + + host margaux { + ... # Standard lines, see above + option swap-path "192.168.4.4:/netswapvolume/netswap"; + option swap-size 64000; + } - The idea is that, at least for a FreeBSD client, - DHCP/BOOTP option code 128 is the path to the NFS swap file, - and option code 129 is the swap size in kilobytes. Older - versions of dhcpd allowed a syntax of + + swap-path is the path to a directory + where swap files will be located. Each file will be named + swap.clientIp. + + Older versions of dhcpd used a syntax of option option-128 "..., which is no longer supported. /etc/bootptab would use the @@ -2860,12 +3051,12 @@ On the NFS swap file server, create the swap file(s) - - &prompt.root; mkdir /netswapvolume/netswap - &prompt.root; cd /netswapvolume/netswap - &prompt.root; dd if=/dev/zero bs=1024 count=64000 of=swap.192.168.4.6 - &prompt.root; chmod 0600 swap.192.168.4.6 - + + &prompt.root; mkdir /netswapvolume/netswap + &prompt.root; cd /netswapvolume/netswap + &prompt.root; dd if=/dev/zero bs=1024 count=64000 of=swap.192.168.4.6 + &prompt.root; chmod 0600 swap.192.168.4.6 + 192.168.4.6 is the IP address for the diskless client. @@ -2874,13 +3065,31 @@ On the NFS swap file server, add the following line to /etc/exports: - /netswapvolume -maproot=0:10 -alldirs margaux +/netswapvolume -maproot=0:10 -alldirs margaux Then tell mountd to reread the exports file, as above. + + + + NFS swap with FreeBSD 5.x + + The kernel does not support enabling NFS swap at boot + time. Swap must be enabled by the startup scripts, by + mounting a writeable file system and creating and enabling a + swap file. This can be done, for example, in + /etc/rc.local. Example: + + + swapfile=/writableVolume/swap/swap.MyIP + dd if=/dev/zero of=$swapfile bs=1k count=1 oseek=100000 + swapon $swapfile + rm -rf $swapfile + + @@ -2915,7 +3124,7 @@ to create the correct device entries (FreeBSD 5.0 and later use &man.devfs.5; to allocate device nodes transparently for the user, running MAKEDEV on these - versions is useless). + versions is pointless).