From owner-freebsd-fs@FreeBSD.ORG Wed Sep 18 13:29:20 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 727F98B0 for ; Wed, 18 Sep 2013 13:29:20 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-la0-x231.google.com (mail-la0-x231.google.com [IPv6:2a00:1450:4010:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DB2982A54 for ; Wed, 18 Sep 2013 13:29:19 +0000 (UTC) Received: by mail-la0-f49.google.com with SMTP id ev20so5582605lab.22 for ; Wed, 18 Sep 2013 06:29:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=O0I9ekqx9i3szzeYSxKQMwjTC63ce4Rk9M8j/T+SkR4=; b=cW7PDiRNnR3jkNpYaPrhLSAQ40KRfG56aFZgRNkIMCmC/zLaoOf1TQMlTM0QYxC1Fo +P9jBm2K7FnCWNwqN+EVKOuOA8d3LmIfGIO1EuaOJIwSYJ1EQCBRJD9sWHHIiS8MF9Mv JqM5NB1EPRW6qEKJg3pEjvNoqvnQCKvmANeBdUlN8+MTJwZb6dp2UL31qE5SB7q81irk QmrTHglEpewUJVYcto9/u1+b/7vXk7BoGhjz136t6voX2hvazRMWYCKdOO22f5lBmj1W eh8ll35QnjwrMffSilhcSNBHhEUsG7jRGZo1P5KIkpUKyLPeIdIHV76FPxm65A36Nwmi Z4NA== X-Received: by 10.112.155.39 with SMTP id vt7mr11114312lbb.29.1379510957863; Wed, 18 Sep 2013 06:29:17 -0700 (PDT) Received: from [192.168.1.130] (mau.donbass.com. [92.242.127.250]) by mx.google.com with ESMTPSA id vs11sm990601lac.3.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 18 Sep 2013 06:29:17 -0700 (PDT) Message-ID: <5239AAAC.2@gmail.com> Date: Wed, 18 Sep 2013 16:29:16 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130909 Thunderbird/17.0.8 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Subject: possible hang in unionfs Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Sep 2013 13:29:20 -0000 Hello. I'm supporting servers that share some projects, and I want those projects to be contamined and separated. I'm using this scheme: /bin /var/jail/project/bin nullfs ro,late 0 0 /tmp /var/jail/project/tmp unionfs ro,late,below 0 0 /var /var/jail/project/var unionfs ro,late,below 0 0 /usr /var/jail/project/usr nullfs ro,late 0 0 /usr/local /var/jail/project/usr/local nullfs ro,late 0 0 /lib /var/jail/project/lib nullfs ro,late 0 0 My base system is ZFS hence mounting /usr and /usr/local as they are different fs. And because I want to access postgres/mysql/redis through socket I need tmp from the main container. And the sites want write access to tmp too. Mostly everything works, but sometimes the machine gets locked. Everything stops and I can't even login. I can see login process is waiting in [zfs]. I rebuild everything with DEBUG and now I see this: Sep 14 11:19:36 slylandro kernel: lock order reversal: Sep 14 11:19:36 slylandro kernel: 1st 0xfffffe002e41b878 tmpfs (tmpfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1907 Sep 14 11:19:36 slylandro kernel: 2nd 0xfffffe019b06c680 zfs (zfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:1938 Sep 14 11:19:36 slylandro kernel: KDB: stack backtrace: Sep 14 11:19:36 slylandro kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff9f8492d2d0 Sep 14 11:19:36 slylandro kernel: kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff9f8492d380 Sep 14 11:19:36 slylandro kernel: witness_checkorder() at witness_checkorder+0xc0a/frame 0xffffff9f8492d400 Sep 14 11:19:36 slylandro kernel: __lockmgr_args() at __lockmgr_args+0x744/frame 0xffffff9f8492d520 Sep 14 11:19:36 slylandro kernel: vop_stdlock() at vop_stdlock+0x3c/frame 0xffffff9f8492d540 Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d570 Sep 14 11:19:36 slylandro kernel: unionfs_lock() at unionfs_lock+0x4c6/frame 0xffffff9f8492d600 Sep 14 11:19:36 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f8492d630 Sep 14 11:19:36 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 0xffffff9f8492d690 Sep 14 11:19:36 slylandro kernel: unionfs_root() at unionfs_root+0x43/frame 0xffffff9f8492d6c0 Sep 14 11:19:36 slylandro kernel: lookup() at lookup+0x99d/frame 0xffffff9f8492d750 Sep 14 11:19:36 slylandro kernel: namei() at namei+0x589/frame 0xffffff9f8492d800 Sep 14 11:19:36 slylandro kernel: unp_connect() at unp_connect+0x1b3/frame 0xffffff9f8492da60 Sep 14 11:19:36 slylandro kernel: uipc_connect() at uipc_connect+0x47/frame 0xffffff9f8492da90 Sep 14 11:19:36 slylandro kernel: kern_connect() at kern_connect+0xdb/frame 0xffffff9f8492dae0 Sep 14 11:19:36 slylandro kernel: sys_connect() at sys_connect+0x96/frame 0xffffff9f8492db20 Sep 14 11:19:36 slylandro kernel: amd64_syscall() at amd64_syscall+0x259/frame 0xffffff9f8492dc30 Sep 14 11:19:36 slylandro kernel: Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff9f8492dc30 Sep 14 11:19:36 slylandro kernel: --- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff6c18, rbp = 0x7fffffff6c40 --- Sep 14 11:19:39 slylandro kernel: lock order reversal: Sep 14 11:19:39 slylandro kernel: 1st 0xfffffe0089d87878 zfs (zfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_vnops.c:312 Sep 14 11:19:39 slylandro kernel: 2nd 0xfffffe0280cc9680 unionfs (unionfs) @ /usr/src/sys/modules/unionfs/../../fs/unionfs/union_subr.c:366 Sep 14 11:19:39 slylandro kernel: 3rd 0xfffffe002e3ec098 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2335 Sep 14 11:19:39 slylandro kernel: KDB: stack backtrace: Sep 14 11:19:39 slylandro kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff9f835fe120 Sep 14 11:19:39 slylandro kernel: kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff9f835fe1d0 Sep 14 11:19:39 slylandro kernel: witness_checkorder() at witness_checkorder+0xc0a/frame 0xffffff9f835fe250 Sep 14 11:19:39 slylandro kernel: __lockmgr_args() at __lockmgr_args+0x744/frame 0xffffff9f835fe370 Sep 14 11:19:39 slylandro kernel: vop_stdlock() at vop_stdlock+0x3c/frame 0xffffff9f835fe390 Sep 14 11:19:39 slylandro kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff9f835fe3c0 Sep 14 11:19:39 slylandro kernel: _vn_lock() at _vn_lock+0x54/frame 0xffffff9f835fe420 Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x21b/frame 0xffffff9f835fe480 Sep 14 11:19:39 slylandro kernel: unionfs_noderem() at unionfs_noderem+0x1cf/frame 0xffffff9f835fe4f0 Sep 14 11:19:39 slylandro kernel: unionfs_reclaim() at unionfs_reclaim+0x14/frame 0xffffff9f835fe500 Sep 14 11:19:39 slylandro kernel: VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb9/frame 0xffffff9f835fe530 Sep 14 11:19:39 slylandro kernel: vgonel() at vgonel+0x16e/frame 0xffffff9f835fe5a0 Sep 14 11:19:39 slylandro kernel: vrecycle() at vrecycle+0x3e/frame 0xffffff9f835fe5d0 Sep 14 11:19:39 slylandro kernel: unionfs_inactive() at unionfs_inactive+0x23/frame 0xffffff9f835fe5e0 Sep 14 11:19:39 slylandro kernel: VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb9/frame 0xffffff9f835fe610 Sep 14 11:19:39 slylandro kernel: vinactive() at vinactive+0xc6/frame 0xffffff9f835fe660 Sep 14 11:19:39 slylandro kernel: vputx() at vputx+0x263/frame 0xffffff9f835fe6c0 Sep 14 11:19:39 slylandro kernel: lookup() at lookup+0xd2c/frame 0xffffff9f835fe750 Sep 14 11:19:39 slylandro kernel: namei() at namei+0x589/frame 0xffffff9f835fe800 Sep 14 11:19:39 slylandro kernel: unp_connect() at unp_connect+0x1b3/frame 0xffffff9f835fea60 Sep 14 11:19:39 slylandro kernel: uipc_connect() at uipc_connect+0x47/frame 0xffffff9f835fea90 Sep 14 11:19:39 slylandro kernel: kern_connect() at kern_connect+0xdb/frame 0xffffff9f835feae0 Sep 14 11:19:39 slylandro kernel: sys_connect() at sys_connect+0x96/frame 0xffffff9f835feb20 Sep 14 11:19:39 slylandro kernel: amd64_syscall() at amd64_syscall+0x259/frame 0xffffff9f835fec30 Sep 14 11:19:39 slylandro kernel: Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff9f835fec30 Sep 14 11:19:39 slylandro kernel: --- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x801f4d8ac, rsp = 0x7fffffff7628, rbp = 0x7fffffff7650 --- The machine is on 9.2-RC4. Has everyone faced something like this? Is there any other way to share host and slave containers filesystems? -- Sphinx of black quartz, judge my vow.