Date: Wed, 3 Feb 2016 21:24:04 GMT From: kczekirda@FreeBSD.org To: svn-soc-all@FreeBSD.org Subject: socsvn commit: r298342 - soc2015/kczekirda/asiabsdcon2016 Message-ID: <201602032124.u13LO4Dl054167@socsvn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: kczekirda Date: Wed Feb 3 21:24:03 2016 New Revision: 298342 URL: http://svnweb.FreeBSD.org/socsvn/?view=rev&rev=298342 Log: expand Added: soc2015/kczekirda/asiabsdcon2016/tftp.png (contents, props changed) Modified: soc2015/kczekirda/asiabsdcon2016/paper.pdf soc2015/kczekirda/asiabsdcon2016/paper.tex Modified: soc2015/kczekirda/asiabsdcon2016/paper.pdf ============================================================================== Binary file (source and/or target). No diff available. Modified: soc2015/kczekirda/asiabsdcon2016/paper.tex ============================================================================== --- soc2015/kczekirda/asiabsdcon2016/paper.tex Wed Feb 3 20:55:08 2016 (r298341) +++ soc2015/kczekirda/asiabsdcon2016/paper.tex Wed Feb 3 21:24:03 2016 (r298342) @@ -75,7 +75,7 @@ \section{Servers side} -The Preboot eXecution Environment allows to boot from a network interface. Host broadcasts a DHCP discover a request and a DHCP server responds with a DHCP packet that includes PXE options (the name of a boot server and a boot file). The client downloads his boot file by using TFTP and then executes it. In this project it is iPXE loader. In the next step iPXE loads MEMDISK kernel with the location of modified mfsBSD iso file as its parameter and then nodes mount shared storage via NFS protocol. +The Preboot eXecution Environment allows to boot from a network interface. Host broadcasts a DHCP discover a request and a DHCP server responds with a DHCP packet that includes PXE options (the name of a boot server and a boot file). The client downloads his boot file by using TFTP and then executes it. In this project it is iPXE loader and this is classical chainloading of iPXE. In the next step iPXE loads MEMDISK kernel with the location of modified mfsBSD iso file as its parameter and then nodes mount shared storage via NFS protocol. As you can see, there is a lot of services to configure: @@ -88,28 +88,157 @@ \end{itemize} \subsection{DHCP Server} -Firts step of booting node from network is DHCP service. DHCP server responds with a DHCP packet that included PXE options, in this case the name of TFTP boot server and a boot file. +Firts step of booting node from the network is DHCP service. DHCP server responds with a DHCP packet that included PXE options, in this case the name of TFTP boot server and a boot file. + +An example of the dhcp server configuration: + +{\tt \small\begin{verbatim} +subnet 192.168.22.0 netmask 255.255.255.0 { + range 192.168.22.10 192.168.22.50; + option routers 192.168.22.1; + option domain-name-servers 192.168.22.1; + next-server 192.168.22.19; + if exists user-class and ( option \\ + user-class = "iPXE" ) { + filename "http://192.168.22.3/menu.ipxe"; + } + else { + filename "undionly.kpxe"; + } +} +\end{verbatim} +} + +In this case we can see, that TFTP server is located on 192.168.22.19 IP address, filename is different and depends on client user-class. iPXE image (filename "undionly.kpxe") is handed when the DHCP request comes from a legacy PXE client. In the next step request sends iPXE DHCP client with user-class iPXE and answer in filename option is the url with menu.ipxe script. \subsection{TFTP Server} -On the TFTP server I only store iPXE loader compiled from the port. +Trivial File Transfer Protocol (TFTP) is a service used for transwer iPXE image compiled from the port. Nodes download the image from the TFTP server each time that they boot. In my project I use FreeNAS and TFTP configuration screen shows default configuration and it is sufficient. + +\begin{figure}[h] +\begin{center} + \centering + \includegraphics[width=0.5\textwidth]{tftp.png} +\end{center} +\end{figure} \subsection{HTTP Server} -On the HTTP server I only store iso image of custom mfsBSD, iPXE loader runs it if node should to take new task. +HTTP server is used for serving iso image of custom mfsBSD and initial script: menu.ipxe. In my case it's apache in the jail on the FreeNAS box. \subsection{NFS Server} -The NFS server it's a storage for source code and obj files if node have not enough RAM memory. NFS service is provided by FreeNAS and stored on the ZFS filesystem. The dataset have enabled deduplication. +The NFS service is provided by FreeNAS. It's a storage for source code. If node have not enough RAM memory can also save obj files there. NFS export is stored on the ZFS filesystem. The dataset have enabled deduplication. This configuration allows to have access to every revision of the source code without switching beetween revisions in repository. \subsection{Management} -The frontend of management application is writen in python with bottle framework. This is the place, where I can manage my nodes and revisions. Screenshot is below. +The frontend of management application is writen in python with bottle framework. Informations about nodes and revisions are saved in the sqlite database. The management is the place, where user can manage nodes and revisions and it works as http server. Application supports methods: + +\begin{itemize} +\item / to provide default ipxe script +\item /admin - it's main dashboard +\item /admin/add\_node +\item /admin/edit\_node/:id +\item /admin/delete\_node/:id +\item /admin/add\_task +\item /admin/delete\_task/:id +\item /menu/:mac to send static ipxe script which name is saved in the database +\item /static/ to provide static files +\item /admin/take\_task/:mac to start environment preparing +\item /admin/change\_boot/:host/:new to change boot ipxe script +\item /admin/change\_task\_status/:revision/:new\_status +\item /admin/change\_node\_status/:hostname/:new\_status +\end{itemize} + +Example screenshot you can see below. \section{Client side} -From client side there is only one thing I have to carry on - setup boot order - from network. The iPXE uses scripts to decide which is next step on booting - hard drive or network. +From client side there is only one thing I have to carry on - set network card as first booting device. The iPXE uses script decides which is the next step on booting - hard drive or network. + +\section{mfsBSD configuration} + +mfsBSD configuration is very simple, because I added only this lines to mfsbsd/conf/rc.local.sample file: + +{\tt \small\begin{verbatim} +sleep 10 +mkdir /cluster +mount -t nfs -o nolockd 192.168.22.19:/mnt/tank \\ + /freebsd/$(hostname)/cluster /cluster +sh -x /cluster/run.sh > /cluster/run.log 2>&1 & +\end{verbatim} +} + +Node mounts storage and runs cluster script, where are other instructions. + +\section{iPXE scripts} + +For control the boot process on nodes I have four ipxe scripts (take\_task, wait, cluster and hdd). The first of them is take\_task.ipxe: +{\tt \small\begin{verbatim} +#!ipxe + +set www 192.168.22.3 +set port 8080 + +chain http://${www}:${port}/admin/ \\ + take_task/${net0/mac} +\end{verbatim} +} + +In this script node sends request to management application and tell them that it is clean and it is ready to take new task. Very important parameter is mac address of the network card. The management uses this parameter to search which is the next one ipxe script (wait, cluster or hdd). + +The second script is wait.ipxe: + +{\tt \small\begin{verbatim} +#!ipxe +set www 192.168.22.3 +set port 8080 +set timeout 120000 + +:menu +menu Creating environment, please wait... +item next Please wait... +choose --timeout ${timeout} selected +goto ${selected} + +:next +chain http://${www}:${port}/menu/${net0/mac} +\end{verbatim} +} + +This script is the infinite loop. Every 120 seconds node asks the management for new ipxe script. During this time the management is preparing environment (creating directories for revision, copying the source tree etc). + +When server finish preparing environment ipxe script for node changes to cluster.ipxe: + +{\tt \small\begin{verbatim} +#!ipxe +set timeout 10000 + +set www 192.168.22.3 +set iso mfsbsd_cluster.iso + +sanboot --drive 0x81 --no-describe http://${www}/${iso} +\end{verbatim} +} + +and node boots from mfsBSD iso and do tests. + +When all tests is fine cluster script change node status (and ipxe script) to hdd: + +{\tt \small\begin{verbatim} +#!ipxe +set timeout 10000 + +sanboot --drive 0x80 --no-describe +\end{verbatim} +} + +and node boots from HDD drive. + +The last change is reseting node and set take\_task.ipxe as a script to run. \section{Workflow} +This is complete workflow for the node and revision. + \subsection{The node} \begin{itemize} -\item the node starts from netbooting and take\_task.ipxe file +\item the node starts netbooting from take task status \item in first step of PXE booting node sends DHCP request and DHCP server respond with \texttt{next-server} and \texttt{filename} options and node knows what and from to download. \item the node downloads iPXE loader binary by TFTP protocol and executes it \item iPXE sends DHCP request and gives an answer with different filename option - url to iPXE starting script @@ -125,8 +254,6 @@ If any step from building, installing or booting stage fails then the node starts netbooting and takes new task. - - \subsection{Revision} \begin{itemize} Added: soc2015/kczekirda/asiabsdcon2016/tftp.png ============================================================================== Binary file. No diff available.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201602032124.u13LO4Dl054167>