From owner-freebsd-fs@FreeBSD.ORG Sat May 3 12:51:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F270106564A for ; Sat, 3 May 2008 12:51:05 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 4BE898FC19 for ; Sat, 3 May 2008 12:51:04 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id m43Cp1q9011127 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 3 May 2008 14:51:02 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id m43Copdt001744 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 3 May 2008 14:50:52 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id m43CopgF043266; Sat, 3 May 2008 14:50:51 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id m43Copwa043265; Sat, 3 May 2008 14:50:51 +0200 (CEST) (envelope-from ticso) Date: Sat, 3 May 2008 14:50:51 +0200 From: Bernd Walter To: Eric Anderson Message-ID: <20080503125050.GG40730@cicely12.cicely.de> References: <48070DCF.9090902@fsn.hu> <4CA7BA82-E95C-45FF-9B94-8EF27B6DB024@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4CA7BA82-E95C-45FF-9B94-8EF27B6DB024@freebsd.org> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: freebsd-fs@freebsd.org Subject: Re: Consistent inodes between distinct machines X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 May 2008 12:51:05 -0000 On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote: > On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote: > > >Hello, > > > >I have several NFS servers, where the service must be available > >0-24. The servers are mounted read only on the clients and I've > >solved the problem of maintaining consistent inodes between them by > >rsyncing an UFS image and mounting it via md on the NFS servers. > >The machines have a common IP address with CARP, so if one of them > >falls out, the other(s) can take over. > > > >This works nice, but rsyncing multi gigabyte files are becoming more > >and more annoying, so I've wondered whether it would be possible to > >get constant inodes between machines via alternative ways. > > > Why not avoid syncing multi-gigabyte files by splitting your huge FS > image into many smaller say 512MB files, then use md and geom concat/ > stripe/etc to make them all one image that you mount? Where would be the positive effect by doing this? FFS distributes data over the media, so all the small files changes in almost every case and you have to checksum-compare the whole virtual disk anyway. With multiple files the syncing is more complex. For example a normal rsync run can garantie that you get a complete file synced or none at all, but this doesn't work out of the box with multiple files, so you risk half updated data. Nevertheless I think that the UFS/NFS combo is not very good for this problem. With ZFS send/receive however inode numbers are consistent. Together with the differential stream creation it is quite efficient to sync large volumes as well. [75]cicely14# zfs send data/arm-elf@2008-05-03 | zfs receive -v data/test receiving full stream of data/arm-elf@2008-05-03 into data/test@2008-05-03 received 126Mb stream in 28 seconds (4.50Mb/sec) 0.008u 5.046s 0:27.93 18.0% 53+2246k 0+0io 0pf+0w [56]cicely14# ls -ali /usr/local/arm-elf/bin/ total 22585 147 drwxr-xr-x 2 root wheel 20 Mar 25 2006 . 3 drwxr-xr-x 11 root wheel 11 Dec 25 04:58 .. 154 -rwxr-xr-x 1 root wheel 1514107 Mar 25 2006 arm-elf-addr2line 150 -rwxr-xr-x 2 root wheel 1495219 Mar 25 2006 arm-elf-ar 159 -rwxr-xr-x 2 root wheel 2275463 Mar 25 2006 arm-elf-as 158 -rwxr-xr-x 1 root wheel 1481234 Mar 25 2006 arm-elf-c++filt 163 -rwxr-xr-x 1 root wheel 300233 Mar 25 2006 arm-elf-cpp 164 -rwxr-xr-x 2 root wheel 296938 Mar 25 2006 arm-elf-gcc 164 -rwxr-xr-x 2 root wheel 296938 Mar 25 2006 arm-elf-gcc-4.1.0 162 -rwxr-xr-x 1 root wheel 15949 Mar 25 2006 arm-elf-gccbug 161 -rwxr-xr-x 1 root wheel 126715 Mar 25 2006 arm-elf-gcov 160 -rwxr-xr-x 2 root wheel 2162285 Mar 25 2006 arm-elf-ld 156 -rwxr-xr-x 2 root wheel 1541809 Mar 25 2006 arm-elf-nm 153 -rwxr-xr-x 1 root wheel 1871104 Mar 25 2006 arm-elf-objcopy 149 -rwxr-xr-x 2 root wheel 2008424 Mar 25 2006 arm-elf-objdump 152 -rwxr-xr-x 2 root wheel 1495214 Mar 25 2006 arm-elf-ranlib 155 -rwxr-xr-x 1 root wheel 389000 Mar 25 2006 arm-elf-readelf 148 -rwxr-xr-x 1 root wheel 1430608 Mar 25 2006 arm-elf-size 151 -rwxr-xr-x 1 root wheel 1412788 Mar 25 2006 arm-elf-strings 157 -rwxr-xr-x 2 root wheel 1871103 Mar 25 2006 arm-elf-strip [57]cicely14# ls -ali /data/test/bin/ total 22585 147 drwxr-xr-x 2 root wheel 20 Mar 25 2006 . 3 drwxr-xr-x 11 root wheel 11 Dec 25 04:58 .. 154 -rwxr-xr-x 1 root wheel 1514107 Mar 25 2006 arm-elf-addr2line 150 -rwxr-xr-x 2 root wheel 1495219 Mar 25 2006 arm-elf-ar 159 -rwxr-xr-x 2 root wheel 2275463 Mar 25 2006 arm-elf-as 158 -rwxr-xr-x 1 root wheel 1481234 Mar 25 2006 arm-elf-c++filt 163 -rwxr-xr-x 1 root wheel 300233 Mar 25 2006 arm-elf-cpp 164 -rwxr-xr-x 2 root wheel 296938 Mar 25 2006 arm-elf-gcc 164 -rwxr-xr-x 2 root wheel 296938 Mar 25 2006 arm-elf-gcc-4.1.0 162 -rwxr-xr-x 1 root wheel 15949 Mar 25 2006 arm-elf-gccbug 161 -rwxr-xr-x 1 root wheel 126715 Mar 25 2006 arm-elf-gcov 160 -rwxr-xr-x 2 root wheel 2162285 Mar 25 2006 arm-elf-ld 156 -rwxr-xr-x 2 root wheel 1541809 Mar 25 2006 arm-elf-nm 153 -rwxr-xr-x 1 root wheel 1871104 Mar 25 2006 arm-elf-objcopy 149 -rwxr-xr-x 2 root wheel 2008424 Mar 25 2006 arm-elf-objdump 152 -rwxr-xr-x 2 root wheel 1495214 Mar 25 2006 arm-elf-ranlib 155 -rwxr-xr-x 1 root wheel 389000 Mar 25 2006 arm-elf-readelf 148 -rwxr-xr-x 1 root wheel 1430608 Mar 25 2006 arm-elf-size 151 -rwxr-xr-x 1 root wheel 1412788 Mar 25 2006 arm-elf-strings 157 -rwxr-xr-x 2 root wheel 1871103 Mar 25 2006 arm-elf-strip -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.