From owner-freebsd-bugs@FreeBSD.ORG Thu Apr 21 10:20:09 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B9201065673 for ; Thu, 21 Apr 2011 10:20:09 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 478FF8FC15 for ; Thu, 21 Apr 2011 10:20:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p3LAK9fm069455 for ; Thu, 21 Apr 2011 10:20:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p3LAK9f2069454; Thu, 21 Apr 2011 10:20:09 GMT (envelope-from gnats) Resent-Date: Thu, 21 Apr 2011 10:20:09 GMT Resent-Message-Id: <201104211020.p3LAK9f2069454@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Konstantin Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 706371065675 for ; Thu, 21 Apr 2011 10:12:48 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 54BC88FC16 for ; Thu, 21 Apr 2011 10:12:48 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p3LACmd4042083 for ; Thu, 21 Apr 2011 10:12:48 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p3LAClYu042077; Thu, 21 Apr 2011 10:12:47 GMT (envelope-from nobody) Message-Id: <201104211012.p3LAClYu042077@red.freebsd.org> Date: Thu, 21 Apr 2011 10:12:47 GMT From: Konstantin To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/156545: mv could break UFS on SMP systems X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Apr 2011 10:20:09 -0000 >Number: 156545 >Category: kern >Synopsis: mv could break UFS on SMP systems >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Apr 21 10:20:08 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Konstantin >Release: 8.X/7.X >Organization: Kasperksy Lab >Environment: # uname -a FreeBSD test-host 8.2-RELEASE FreeBSD 8.2-RELEASE #1: Thu Apr 7 14:23:20 UTC 2011 root@host:/usr/obj/amd64/usr/src/sys/GENERIC amd64 >Description: We have found out a strange mv(1) behaviour on SMP systems with UFS. If two or more processes are trying to move the same dir from one source, it could break UFS. It works on 7.X/8.X amd64 systems. Actually, i think, it works on all systems with SMP on all platforms. I guess that the problem is somewhere in ./sys/ufs/ufs/ufs_vnops.c because i can't repeat it for an example on ZFS partition. After UFS break you will see this: # mount /dev/da0p2 on / (ufs, local, noatime) devfs on /dev (devfs, local, multilabel) /dev/da0p4 on /data (ufs, local, noatime, soft-updates) # cd /data/test # ls -lai total 40 11540480 drwxr-xr-x 5 root wheel 512 Apr 21 17:30 . 2 drwxr-xr-x 8 root wheel 512 Apr 21 17:29 .. 11543482 drwxr-xr-x 3 root wheel 16384 Apr 21 17:37 dst1 11543483 drwxr-xr-x 3 root wheel 512 Apr 21 17:37 dst2 11540481 drwxr-xr-x 2 root wheel 16384 Apr 21 17:37 src 4 -rw-r--r-- 1 root wheel 738 Apr 21 17:33 test.pl # ls -lai dst1 total 20 11543482 drwxr-xr-x 3 root wheel 16384 Apr 21 17:37 . 11540480 drwxr-xr-x 5 root wheel 512 Apr 21 17:30 .. 11541475 drwx------ 3 root wheel 512 Apr 21 17:36 03qOT # ls -lai dst2 total 6 11543483 drwxr-xr-x 3 root wheel 512 Apr 21 17:37 . 11540480 drwxr-xr-x 5 root wheel 512 Apr 21 17:30 .. 11541475 drwx------ 3 root wheel 512 Apr 21 17:36 03qOT # rm -rf dst1/03qOT rm: dst1/03qOT: Invalid argument # rm -rf dst2/03qOT rm: dst2/03qOT: Invalid argument The only way to fix it is to unmount the partition and run the fsck on it. # fsck -vy /data start /data wait fsck_ufs -y /dev/da0p4 ** /dev/da0p4 ** Last Mounted on /data ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames /test/dst1/03qOT IS AN EXTRANEOUS HARD LINK TO DIRECTORY /test/dst1/03qOT REMOVE? yes BAD INODE NUMBER FOR '..' I=11541475 OWNER=root MODE=40700 SIZE=512 MTIME=Apr 21 17:36 2011 DIR=/test/dst2/03qOT UNEXPECTED SOFT UPDATE INCONSISTENCY FIX? yes ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts LINK COUNT DIR I=11541475 OWNER=root MODE=40700 SIZE=512 MTIME=Apr 21 17:36 2011 COUNT 3 SHOULD BE 2 ADJUST? yes LINK COUNT DIR I=11543482 OWNER=root MODE=40755 SIZE=16384 MTIME=Apr 21 17:37 2011 COUNT 3 SHOULD BE 2 ADJUST? yes ** Phase 5 - Check Cyl groups 7757 files, 87291 used, 134202352 free (256 frags, 16775262 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** >How-To-Repeat: You can use this script to repeat the problem: #!/usr/local/bin/perl use strict; my $iterations = 50; my $dirs = 1000; mkdir 'src'; for(1..$iterations) { print "Iterations $_\n"; do_test(); } sub do_test { print "Creating $dirs sample directories...\n"; for(1..$dirs) { my $dirname = `mktemp -d src/XXXXX`; chomp $dirname; system("touch $dirname/data && touch $dirname/meta"); print "$_ "; } print "done\n"; my ($pid1, $pid2) = (do_job('src', 'dst1'), do_job('src', 'dst2')); waitpid($pid1, 0); waitpid($pid2, 0); } sub do_job { my ($src, $dst) = @_; my $pid = fork(); return $pid if $pid != 0; system("mkdir -p $dst && ls $src | xargs -I _ mv -n $src/_ $dst"); system("rm -rf $dst/*") == 0 or die; exit(0); } >Fix: >Release-Note: >Audit-Trail: >Unformatted: