Date: Tue, 04 Jun 2019 21:05:26 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 238326] Kernel crash on jail stop (VIMAGE/VNET) Message-ID: <bug-238326-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D238326 Bug ID: 238326 Summary: Kernel crash on jail stop (VIMAGE/VNET) Product: Base System Version: 12.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: paul.le.gauret@gmail.com Jails managed through the native jails tools with the attached jail.conf fi= le (so called "thin jails", VNET). I can reliably crash the system by issuing a "service jail restart". The crash happens when jails are being stopped. [root@beastie ~]# uname -a FreeBSD beastie 12.0-RELEASE-p4 FreeBSD 12.0-RELEASE-p4 GENERIC amd64 [root@beastie ~]# service jail restart Stopping jails: dns db ldap web nextcloud imap smtp testpacket_write_wait: Connection to xxx.xxx.xxx.xxx port 22: Broken pipe As a workaround I can avoid the crash by inserting a exec.poststop =3D "sleep 2"; statement in jail.conf. 1 second was not enough to avoid the crash. Also worth noting that the issue does not appear if I totally disable networking, hence my guess this is somehow VIMAGE/VNET related. I've managed to obtain a core dump but my attempts at debugging didn't go f= ar as I'm not experienced with this. Of course happy to do more debugging if t= his can help identifying the issue. The system is not yet live so I can pretty = much try anything on it. [root@beastie /var/crash]# kgdb /boot/kernel/kernel /var/crash/vmcore.0 GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.htm= l> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...(no debugging symbols found)...d= one. 0xffffffff80bcd0bd in sched_switch () (kgdb) bt #0 0xffffffff80bcd0bd in sched_switch () #1 0xffffffff80ba6de1 in mi_switch () #2 0xffffffff80bf554c in sleepq_wait () #3 0xffffffff80ba6817 in _sleep () #4 0xffffffff80bfae71 in taskqueue_thread_loop () #5 0xffffffff80b5bf33 in fork_exit () #6 <signal handler called> My jail.conf file below. Note that the crash happens regardless of whether = the jails with special permissions (db, builder) are configured. I also tried tweaking the mount/umount order without success. $release=3D"12.0-RELEASE"; # Release used to create the jail if not already existing host.hostname =3D "${name}"; path =3D "/jails/${name}"; exec.consolelog =3D "/var/log/jail.${name}.console.log"; vnet =3D "new"; vnet.interface =3D "epair${jailnum}b"; ### Create the jail if not already exsisting ### exec.prestart =3D "if [ ! -d "/jails/THINJAILS/${name}" ]; then zfs clone zroot/jails/TEMPLATES/skeleton-${release}@skeleton zroot/jails/THINJAILS/${name}; fi"; exec.prestart +=3D "if [ ! -d "/jails/${name}" ]; then mkdir /jails/${name}= ; fi"; ### Mount filesystems for shared base (RO) and individual jail skeleton (RW) exec.prestart +=3D "mount_nullfs -o ro /jails/TEMPLATES/base-${release} /jails/${name}"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/etc /jails/${name}/etc"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/home /jails/${name}/home"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/root /jails/${name}/root"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/tmp /jails/${name}/tmp"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/var /jails/${name}/var"; exec.prestart +=3D "mount_nullfs -o rw /jails/THINJAILS/${name}/usr/local /jails/${name}/usr/local"; ### Mount /usr/ports from hosts exec.prestart +=3D "mount_nullfs -o ro /usr/ports /jails/${name}/usr/ports"; ### Mount data filesystem if exists exec.prestart +=3D "if [ -d "/jails/DATA/${name}" ]; then mount_nullfs -o rw /jails/DATA/${name} /jails/${name}/data; fi"; ### Mount devfs with the default ruleset (11) now that the jail filesystems= are mounted exec.prestart +=3D "mount -t devfs -o ruleset=3D4 devfs /jails/${name}/dev"; ### Create an ethernet pair and add to bridge0 (internal bridge) exec.prestart +=3D "ifconfig epair${jailnum} create up"; exec.prestart +=3D "ifconfig epair${jailnum}a description '${name} - host interface'"; exec.prestart +=3D "ifconfig epair${jailnum}b description '${name} - jail interface'"; exec.prestart +=3D "ifconfig bridge0 addm epair${jailnum}a"; exec.start =3D "ifconfig epair${jailnum}b inet 192.168.1.${jailnum}/24"; exec.start +=3D "ifconfig epair${jailnum}b inet6 xxx:xxx:xxx:xxx::${jailnum}:1/64"; exec.start +=3D "route add default 192.168.1.1"; exec.start +=3D "route -6 add default xxx:xxx:xxx:xxx::1:1"; ### Proceed with boot through rc exec.start +=3D "/bin/sh /etc/rc"; ### Shutdown the jail exec.stop =3D "/bin/sh /etc/rc.shutdown"; ### Give the jail a couple of seconds to shutdown (avoid issues unmounting filesystems) #exec.poststop =3D "sleep 2"; ### Destroy jail network interface exec.poststop +=3D "ifconfig epair${jailnum}a destroy"; ### Unmount filesystems exec.poststop +=3D "if [ -d "/jails/DATA/${name}" ]; then umount /jails/${name}/data; fi"; exec.poststop +=3D "umount -f /jails/${name}/dev"; exec.poststop +=3D "umount -f /jails/${name}/usr/ports"; exec.poststop +=3D "umount -f /jails/${name}/etc"; exec.poststop +=3D "umount -f /jails/${name}/home"; exec.poststop +=3D "umount -f /jails/${name}/root"; exec.poststop +=3D "umount -f /jails/${name}/tmp"; exec.poststop +=3D "umount -f /jails/${name}/var"; exec.poststop +=3D "umount -f /jails/${name}/usr/local"; exec.poststop +=3D "umount -f /jails/${name}"; dns { $jailnum=3D2; }; db { $jailnum=3D3; allow.sysvipc; }; ldap { $jailnum=3D4; }; web { $jailnum=3D5; }; nextcloud { $jailnum=3D6; }; imap { $jailnum=3D7; }; smtp { $jailnum=3D8; }; test { $jailnum=3D98; }; builder { $jailnum=3D99; enforce_statfs=3D0; allow.mount; allow.mount.nullfs; allow.mount.tmpfs; allow.mount.devfs; allow.chflags; }; --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-238326-227>