Date: Wed, 18 Sep 2013 16:29:16 +0300 From: Volodymyr Kostyrko <c.kworr@gmail.com> To: freebsd-fs@FreeBSD.org Subject: possible hang in unionfs Message-ID: <5239AAAC.2@gmail.com>
next in thread | raw e-mail | index | archive | help
Hello. I'm supporting servers that share some projects, and I want those projects to be contamined and separated. I'm using this scheme: /bin /var/jail/project/bin nullfs ro,late 0 0 /tmp /var/jail/project/tmp unionfs ro,late,below 0 0 /var /var/jail/project/var unionfs ro,late,below 0 0 /usr /var/jail/project/usr nullfs ro,late 0 0 /usr/local /var/jail/project/usr/local nullfs ro,late 0 0 /lib /var/jail/project/lib nullfs ro,late 0 0 My base system is ZFS hence mounting /usr and /usr/local as they are different fs. And because I want to access postgres/mysql/redis through socket I need tmp from the main container. And the sites want write access to tmp too. Mostly everything works, but sometimes the machine gets locked. Everything stops and I can't even login. I can see login process is waiting in [zfs]. I rebuild everything with DEBUG and now I see this: Sep 14 11:19:36 slylandro kernel: lock order reversal: Sep 14 11:19:36 slylandro kernel: 1st 0xfffffe002e41b878 tmpfs (tmpfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1907 Sep 14 11:19:36 slylandro kernel: 2nd 0xfffffe019b06c680 zfs (zfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1938 Sep 14 11:19:36 slylandro kernel: KDB: stack backtrace: Sep 14 11:19:36 slylandro kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff9f8492d2d0 Sep 14 11:19:36 slylandro kernel: kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff9f8492d380 Sep 14 11:19:36 slylandro kernel: witness_checkorder() at witness_checkorder+0xc0a/frame 0xffffff9f8492d400 Sep 14 11:19:36 slylandro kernel: __lockmgr_args() at __lockmgr_args+0x744/frame 0xffffff9f8492d520 Sep 14 11:19:36 slylandro kernel: vop_stdlock() at vop_stdlock+0x3c/frame 0xffffff9f8492d540 Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d570 Sep 14 11:19:36 slylandro kernel: unionfs_lock() at unionfs_lock+0x4c6/frame 0xffffff9f8492d600 Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d630 Sep 14 11:19:36 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 0xffffff9f8492d690 Sep 14 11:19:36 slylandro kernel: unionfs_root() at unionfs_root+0x43/frame 0xffffff9f8492d6c0 Sep 14 11:19:36 slylandro kernel: lookup() at lookup+0x99d/frame 0xffffff9f8492d750 Sep 14 11:19:36 slylandro kernel: namei() at namei+0x589/frame 0xffffff9f8492d800 Sep 14 11:19:36 slylandro kernel: unp_connect() at unp_connect+0x1b3/frame 0xffffff9f8492da60 Sep 14 11:19:36 slylandro kernel: uipc_connect() at uipc_connect+0x47/frame 0xffffff9f8492da90 Sep 14 11:19:36 slylandro kernel: kern_connect() at kern_connect+0xdb/frame 0xffffff9f8492dae0 Sep 14 11:19:36 slylandro kernel: sys_connect() at sys_connect+0x96/frame 0xffffff9f8492db20 Sep 14 11:19:36 slylandro kernel: amd64_syscall() at amd64_syscall+0x259/frame 0xffffff9f8492dc30 Sep 14 11:19:36 slylandro kernel: Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff9f8492dc30 Sep 14 11:19:36 slylandro kernel: --- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff6c18, rbp = 0x7fffffff6c40 --- Sep 14 11:19:39 slylandro kernel: lock order reversal: Sep 14 11:19:39 slylandro kernel: 1st 0xfffffe0089d87878 zfs (zfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:312 Sep 14 11:19:39 slylandro kernel: 2nd 0xfffffe0280cc9680 unionfs (unionfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_subr.c:366 Sep 14 11:19:39 slylandro kernel: 3rd 0xfffffe002e3ec098 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2335 Sep 14 11:19:39 slylandro kernel: KDB: stack backtrace: Sep 14 11:19:39 slylandro kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff9f835fe120 Sep 14 11:19:39 slylandro kernel: kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff9f835fe1d0 Sep 14 11:19:39 slylandro kernel: witness_checkorder() at witness_checkorder+0xc0a/frame 0xffffff9f835fe250 Sep 14 11:19:39 slylandro kernel: __lockmgr_args() at __lockmgr_args+0x744/frame 0xffffff9f835fe370 Sep 14 11:19:39 slylandro kernel: vop_stdlock() at vop_stdlock+0x3c/frame 0xffffff9f835fe390 Sep 14 11:19:39 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f835fe3c0 Sep 14 11:19:39 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 0xffffff9f835fe420 Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x21b/frame 0xffffff9f835fe480 Sep 14 11:19:39 slylandro kernel: unionfs_noderem() at unionfs_noderem+0x1cf/frame 0xffffff9f835fe4f0 Sep 14 11:19:39 slylandro kernel: unionfs_reclaim() at unionfs_reclaim+0x14/frame 0xffffff9f835fe500 Sep 14 11:19:39 slylandro kernel: VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb9/frame 0xffffff9f835fe530 Sep 14 11:19:39 slylandro kernel: vgonel() at vgonel+0x16e/frame 0xffffff9f835fe5a0 Sep 14 11:19:39 slylandro kernel: vrecycle() at vrecycle+0x3e/frame 0xffffff9f835fe5d0 Sep 14 11:19:39 slylandro kernel: unionfs_inactive() at unionfs_inactive+0x23/frame 0xffffff9f835fe5e0 Sep 14 11:19:39 slylandro kernel: VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb9/frame 0xffffff9f835fe610 Sep 14 11:19:39 slylandro kernel: vinactive() at vinactive+0xc6/frame 0xffffff9f835fe660 Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x263/frame 0xffffff9f835fe6c0 Sep 14 11:19:39 slylandro kernel: lookup() at lookup+0xd2c/frame 0xffffff9f835fe750 Sep 14 11:19:39 slylandro kernel: namei() at namei+0x589/frame 0xffffff9f835fe800 Sep 14 11:19:39 slylandro kernel: unp_connect() at unp_connect+0x1b3/frame 0xffffff9f835fea60 Sep 14 11:19:39 slylandro kernel: uipc_connect() at uipc_connect+0x47/frame 0xffffff9f835fea90 Sep 14 11:19:39 slylandro kernel: kern_connect() at kern_connect+0xdb/frame 0xffffff9f835feae0 Sep 14 11:19:39 slylandro kernel: sys_connect() at sys_connect+0x96/frame 0xffffff9f835feb20 Sep 14 11:19:39 slylandro kernel: amd64_syscall() at amd64_syscall+0x259/frame 0xffffff9f835fec30 Sep 14 11:19:39 slylandro kernel: Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff9f835fec30 Sep 14 11:19:39 slylandro kernel: --- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff7628, rbp = 0x7fffffff7650 --- The machine is on 9.2-RC4. Has everyone faced something like this? Is there any other way to share host and slave containers filesystems? -- Sphinx of black quartz, judge my vow.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5239AAAC.2>