From owner-freebsd-fs@freebsd.org Sun Apr 12 11:30:30 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF0762BABC2 for ; Sun, 12 Apr 2020 11:30:30 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490V0z08yQz415D for ; Sun, 12 Apr 2020 11:30:26 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id B6C1F40010 for ; Sun, 12 Apr 2020 13:30:21 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 98ED940015; Sun, 12 Apr 2020 13:30:21 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id E736540010; Sun, 12 Apr 2020 13:30:19 +0200 (CEST) From: Peter Eriksson Message-Id: <92901912-F3AA-4C44-BF6E-3F4E0F996BFF@lysator.liu.se> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow Date: Sun, 12 Apr 2020 13:30:17 +0200 In-Reply-To: <708dabd7-838c-2b34-87dc-2151536fa8bc@wp.pl> To: freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <708dabd7-838c-2b34-87dc-2151536fa8bc@wp.pl> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 490V0z08yQz415D X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-3.32 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-1.02)[ip: (-3.21), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.85), country: EU(-0.01)]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; FREEMAIL_CC(0.00)[wp.pl]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:30:30 -0000 Sure You can find my stuff at my Github page: https://github.com/ptrrkssn/freebsd-stuff = The things needed are the /etc/rc.d scripts (zfs, nfsd, samba_server, = cold) scripts in =E2=80=9Crc.d=E2=80=9D and the =E2=80=9Czfs-mountall=E2=80= =9D in =E2=80=9Cscripts=E2=80=9D that goes into /sbin. What they basically do are use a helper function =E2=80=9Cwaitfor=E2=80=9D= to wait for a =E2=80=9C${name}.done=E2=80=9D file to appear in /var/run = and then just=20 (waitfor zfs ; run_rc_command =E2=80=9C$1=E2=80=9D) & Instead of just the =E2=80=9Crun_rc_command =E2=80=9C$1=E2=80=9D. And = then we have the scripts create those =E2=80=9C${name}.done=E2=80=9D = files when they have started their daemons. Not foolproof (should check that the daemon really is successfully = running and more before signalling =E2=80=9Cdone=E2=80=9D). scripts/zfs-mountall=20 Mounts all filesystems under a specific DATASET sequentially. Basically = what a =E2=80=9Czfs mount -r=E2=80=9D would have done if that option had = existed. rc.d/zfs Executes /sbin/zfs-mountall to make sure all filesystems in the = =E2=80=98zroot=E2=80=99 (or whatever dataset / is mounted from) is = mounted first. This is needed to make sure all important filesystems are = mounted (like various /var filesystems etc). Then basically does: ( rm -f /var/run/zfs*.done zfs mount -va >/var/log/zfs/mount.log && touch = /var/run/zfs-mount.done zfs share -a >/var/log/zfs/share.log && touch /var/run/zfs.done ) & rc.d/mountd, rc.d/nfsd, rc.d/ctld & rc.d/samba_server all contains a = function: waitfor() { FIRST=3Dyes while [ ! -f "/var/run/$1.done" ]; do if [ $FIRST =3D yes ]; then echo "${name}: Waiting for '$1' to finish..." FIRST=3Dno fi sleep 1 done } It probably should go into /etc/rc.subr or somewhere else but=E2=80=A6 = =E2=80=9Cwaitfor=E2=80=9D basically is loop and waits for the = =E2=80=9C${name}.done=E2=80=9D file to appear in /var/run. rc.d/mountd then ends with this instead of just: =E2=80=98 = run_rc_command =E2=80=9C$1=E2=80=9D =E2=80=98 case "$1" in faststart) rm -f "/var/run/${name}.done" (waitfor zfs ; run_rc_command "$1" ; touch = "/var/run/${name}.done") & ;; *) run_rc_command "$1" ;; esac rc.d/nfsd ends with: case "$1" in faststart) (waitfor "mountd" ; run_rc_command "$1") & ;; *) run_rc_command "$1" ;; esac (And similar for cold) And rc.d/samba_server has a bit more modifications (we only want = =E2=80=9Csmbd=E2=80=9D to delay it=E2=80=99s startup, not winbindd or = numbed) so does this in the middle of the script. Also just waits for = the =E2=80=9Czfs mount -a=E2=80=9D to be finished, doesn=E2=80=99t have = to wait for =E2=80=9Czfs share -a=E2=80=9D (NFS)): # Daemon should be enabled and running = =20 if ( [ -n "${rcvar}" ] && checkyesno "${rcvar}" ) || [ -n = "$force_run" ]; then if [ "${rc_arg}" =3D "start" -a "${name}" =3D "smbd" ]; then (waitfor "zfs-mount" ; run_rc_command = "${_rc_prefix}${rc_arg}" ${rc_extra_args}) & else run_rc_command "${_rc_prefix}${rc_arg}" ${rc_extra_args} # If any of the commands failed, take it as a global = result =20 result=3D$((${result} || $?)) fi fi =20 - Peter > On 12 Apr 2020, at 09:17, Ireneusz Pluta/wp.pl wrote: >=20 > W dniu 2020-04-11 o 23:24, Peter Eriksson pisze: >> (We have modified our rc.d startup scripts so we do the =E2=80=9Czfs = mount -a=E2=80=9D part in the background so the rest of the system = doesn=E2=80=99t have to wait for it (so we can get a ssh login prompt = and login and check the progress of the filesystem mounts. (And then = =E2=80=9Cnfs exports=E2=80=9D, nfsd and samba also waits in the = background for that to finish before starting up. A bit of a hack but it = works (a real parallel server manager with dependencies would have been = better > would you share these modifications? >=20 >> :-)