Date: Tue, 19 Feb 2019 10:17:17 +0100 From: Ole <ole@free.de> To: freebsd-questions@freebsd.org Subject: Re: ZFS deadlock on parallel ZFS operations FreeBSD 11.2 and 12.0 Message-ID: <20190219101717.61526ab1.ole@free.de> In-Reply-To: <20190215113423.01edabe9.ole@free.de> References: <20190215113423.01edabe9.ole@free.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/YBhg/WnD.5x0KVMWTOJgpCs Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi, ok now I got a again unkillable ZFS process. It is only one 'zfs send' command. Any Idea how to kill this process without powering off the machine? oot@jails1:/usr/home/admin # ps aux | grep 'zfs send' root 17617 0.0 0.0 12944 3856 - Is Sat04 0:00.00 sudo z= fs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf@= 2019- root 17618 0.0 0.0 12980 4036 - D Sat04 0:00.01 zfs se= nd -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf@2019-= 02-16 root 19299 0.0 0.0 11320 2588 3 S+ 09:53 0:00.00 grep z= fs send root@jails1:/usr/home/admin # kill -9 17618 root@jails1:/usr/home/admin # ps aux | grep 'zfs send' root 17617 0.0 0.0 12944 3856 - Is Sat04 0:00.00 sudo z= fs send -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf@= 2019- root 17618 0.0 0.0 12980 4036 - D Sat04 0:00.01 zfs se= nd -e -I cryptopool/iocage/jails/2fe7ae89-760e-423c-8e7f-4f504e0f08bf@2019-= 02-16 root 19304 0.0 0.0 11320 2588 3 S+ 09:53 0:00.00 grep z= fs send It is a FreeBSD 12.0 VM-Image running in a Bhyve VM. There is basicly only py36-iocage installed, and there are 7 running Jails.=20 There is 30G RAM and sysctl vfs.zfs.arc_max ist set to 20G. It seems that=20 the whole zpool is in some kind of deadlock. All Jails are crashed,=20 unkillable and I can not run any command inside.=20 regards Ole Fri, 15 Feb 2019 11:34:23 +0100 - Ole <ole@free.de>: > Hi, >=20 > I observed that FreeBSD Systems with ZFS will run into a deadlock if > there are many parallel zfs send/receive/snapshot processes. >=20 > I observed this on bare metal and virtual machines with FreeBSD 11.2 > and 12.0. With RAM from 20 to 64G. >=20 > If the system is also on ZFS the whole system crashes. With only jails > on ZFS they freeze, but the Host system stays stable. But you can't > kill -9 the zfs processes. Only a poweroff stops the machine. >=20 > On a FreeBSD 12.0 VM (bhyve), 30G RAM, 5 CPUs, about 30 zfs > operations, mostly send and receive will crash the system. >=20 > There is no heavy load on the machine: >=20 > # top | head -8 > last pid: 91503; load averages: 0.34, 0.31, 0.29 up 0+22:50:47 > 11:24:00 536 processes: 1 running, 529 sleeping, 6 zombie > CPU: 0.9% user, 0.0% nice, 1.5% system, 0.2% interrupt, 97.4% idle > Mem: 165M Active, 872M Inact, 19G Wired, 264M Buf, 9309M Free > ARC: 11G Total, 2450M MFU, 7031M MRU, 216M Anon, 174M Header, 1029M > Other 8423M Compressed, 15G Uncompressed, 1.88:1 Ratio > Swap: 1024M Total, 1024M Free >=20 > I wonder if this is a BUG or normal behaviour. I could live with a > limited amount of parallel ZFS operation, but I don't want the whole > system to crash.=20 >=20 > Reducing the vfs.zfs.arc_max wont help. >=20 > Any Idea to handle with this? >=20 > regards > Ole --Sig_/YBhg/WnD.5x0KVMWTOJgpCs Content-Type: application/pgp-signature Content-Description: Digitale Signatur von OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE60BGd7KVfL83NXCUJZaRRqjklFAFAlxryZ0ACgkQJZaRRqjk lFA8tw//Tmj/CLRBuFYRnoXhFLL9rhoR3whmJaB0KvgYLwuYKPCPNLRe7pO0qdRt FXXVidDp0pmQv8+p+tC9AXJXZOEec3h2+7M0Akv/s0V7LVYx0bOqs8rXRAGK2sBX MBI8Cdo3WKkaod+/FY4svUSckDqW7hqQziCHxCCz4WXTE7XGcPpNHASbkA+1rz1X baYByV1Jz1rh5ywA5JTf1dhPn8jdfA4Rr2atElr+RVnvWuSh386rWdQcRH3bbM45 2CPeODiF3VCi2xIQ/bpaSWs6lTZFdKCDw34GuXjgwqkZsrpRS5aG5osZeL8eubGs U76qyyCYfq1Pt4h7dfGZmP4kLO57w60PPzTx/QiQXyDMVj0PQfUiveOCmpKVK0ce Ot7habbNPdxFjyZMZVGgKKS1E/h7UM4CasuypEbbqRdIU6/gYoWrlDV9dCGk/O64 L6WQ3+jrsM1ORzjJ30txCWsLParQBlx+x2XO8/w+j5lH0AfiHVEfcM/dsOm1IHwY awa/qD8DpDpPEFFnij/bEL5gwjB+ef7n7S+VjCAXd53LCbmZFZ9qenbZWv7GRwIf z8thnA5+lnBZTMk7qPajvgxvsFjHXVGsgozhaUnsZp7T4Tf6Xny7Z1+Q8G5J7arv Es3KHXyxI2vYwanM2FB/MlbdnE1aDx6+36IogCfM6EqcfZ2KZOQ= =Wp9M -----END PGP SIGNATURE----- --Sig_/YBhg/WnD.5x0KVMWTOJgpCs--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190219101717.61526ab1.ole>