From owner-freebsd-fs@FreeBSD.ORG Sun Mar 3 02:26:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EE5D17EB for ; Sun, 3 Mar 2013 02:26:33 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id B9006E70 for ; Sun, 3 Mar 2013 02:26:33 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.3]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id r232E5us019391 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Sat, 2 Mar 2013 18:14:05 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Sat, 2 Mar 2013 18:14:06 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <1934743591.20130302181406@takeda.tk> To: tech mailinglists Subject: Re: I am to silly to mount a zpool while boot In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2013 02:26:34 -0000 Hello tech, Friday, March 1, 2013, 3:26:37 AM, you wrote: > I think that I only can be an idiot to get in such a problem but I am > not able to mount a zpool via fstab while boot. > I have a FreeBSD i386 PV Xen DomU running with 3 disks xbd0 (ext2 for > /boot), xbd1 (UFS for /) and xbd2 (ZFS/zpool with name home to mount > at /home). > I now tried everything I could find. So my fstab entry looks like this: > home /home zfs rw,late 0 0 > The real problem is that after a reboot the zpool is no longer > imported, I really don't know why I always have to reimport the pool > via zpool import -d /dev home. Because of this the filesystem never > can be mounted via fstab while boot and I get dropped into a shell > where I need to do this always manually. > So why the pool always isn't imported after boot and how can I solve this issue? > And is the fstab entry correct itself? So would it work when the pool > gets imported with it's name befor the fstab entry is parsed? > Hope that someone give me a few hints or a solution. Few things: - you don't need fstab entry for ZFS - make sure you have zfs_enable="YES" in /etc/rc.conf - make sure (perhaps this is your issue) /boot/zfs/zpool.cache is writable. This is where ZFS remembers (among many things) what you had previously imported. BTW why your /boot is ext2? - check that "zfs get canmount home" returns "on" -- Best regards, Derek mailto:takeda@takeda.tk If brute force doesn't solve your problems, then you aren't using enough. From owner-freebsd-fs@FreeBSD.ORG Sun Mar 3 02:26:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4EF7D7EC for ; Sun, 3 Mar 2013 02:26:34 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id 119BEE71 for ; Sun, 3 Mar 2013 02:26:34 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.3]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id r232E0O9019383 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Sat, 2 Mar 2013 18:14:00 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Sat, 2 Mar 2013 18:03:48 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <192501837.20130302180348@takeda.tk> To: Tom Evans Subject: Re: I am to silly to mount a zpool while boot In-Reply-To: References: <513098FF.8030806@brockmann-consult.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2013 02:26:34 -0000 Hello Tom, Friday, March 1, 2013, 4:12:03 AM, you wrote: > I have UFS root, ZFS for /usr, /var etc, due to BIOS/loader issues > when initially trying to get ZFS boot working on this box. > This is the total contents of fstab: > /dev/gpt/root / ufs rw 1 > /dev/gpt/swap1 none swap sw 0 > /dev/gpt/swap2 none swap sw 0 > The ZFS fs is mounted by the mountpoint property: >> $ zfs get mountpoint tank > NAME PROPERTY VALUE SOURCE > tank mountpoint /tank default > ZFS is loaded as usual, by adding zfs_load="YES" to /boot/loader.conf > and zfs_enable="YES" to /etc/rc.conf Since you have UFS root, I don't think you need zfs_load="YES" in /boot/loader.conf, rc.conf should be enough. -- Best regards, Derek mailto:takeda@takeda.tk Everybody repeat after me..."We are all individuals." From owner-freebsd-fs@FreeBSD.ORG Sun Mar 3 14:25:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 41B3712F for ; Sun, 3 Mar 2013 14:25:13 +0000 (UTC) (envelope-from mailinglists.tech@gmail.com) Received: from mail-ee0-f49.google.com (mail-ee0-f49.google.com [74.125.83.49]) by mx1.freebsd.org (Postfix) with ESMTP id AD8A2950 for ; Sun, 3 Mar 2013 14:25:12 +0000 (UTC) Received: by mail-ee0-f49.google.com with SMTP id d41so3271522eek.36 for ; Sun, 03 Mar 2013 06:25:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=RN+BQKihw9kSM/NRO0Pqyhj1bpaBGC6zOZ8B1XCKd/0=; b=dXPuhP3hf7fv9MctffirGKJAn1Slx3NhO2YN7Du4wB8QbkdpFk9L4ATdgyfDpI20jL n59Q9tPrprgwwbDfYiHQ8k0sEgVtXiIK2hkaHiJWS8M8c2zIiQ0e5Od+Ac5THFCPLRS5 V20l6p5PNUDiGCczvQSeAqaRe+dSwM7Hpk3/r+CBxeioLfhOaFZAK5i3QrjCCWMp7lUp MWrBfWPsqOvagvUGULvYQlgET7KV/snw/JBvSAJhI2xNGh+PNdCXV0UQtnUPZiapB1+k 5VdCdIfSLbD2meyHZlmcejELcYOkvp5gux0AEvD5PffdLqwGSwpbNTes+ERjRUSbIQNl E8SA== X-Received: by 10.14.4.69 with SMTP id 45mr49041651eei.0.1362320710742; Sun, 03 Mar 2013 06:25:10 -0800 (PST) Received: from [127.0.0.1] (ashlynn.lippux.de. [5.9.218.242]) by mx.google.com with ESMTPS id m46sm27303126eeo.16.2013.03.03.06.25.08 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 03 Mar 2013 06:25:09 -0800 (PST) Message-ID: <51335D43.30108@gmail.com> Date: Sun, 03 Mar 2013 15:25:07 +0100 From: tech mailinglists User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3 MIME-Version: 1.0 To: Derek Kulinski Subject: Re: I am to silly to mount a zpool while boot References: <1934743591.20130302181406@takeda.tk> In-Reply-To: <1934743591.20130302181406@takeda.tk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2013 14:25:13 -0000 Am 03.03.2013 03:14, schrieb Derek Kulinski: > Hello tech, > > Friday, March 1, 2013, 3:26:37 AM, you wrote: > >> I think that I only can be an idiot to get in such a problem but I am >> not able to mount a zpool via fstab while boot. >> I have a FreeBSD i386 PV Xen DomU running with 3 disks xbd0 (ext2 for >> /boot), xbd1 (UFS for /) and xbd2 (ZFS/zpool with name home to mount >> at /home). >> I now tried everything I could find. So my fstab entry looks like this: >> home /home zfs rw,late 0 0 >> The real problem is that after a reboot the zpool is no longer >> imported, I really don't know why I always have to reimport the pool >> via zpool import -d /dev home. Because of this the filesystem never >> can be mounted via fstab while boot and I get dropped into a shell >> where I need to do this always manually. >> So why the pool always isn't imported after boot and how can I solve this issue? >> And is the fstab entry correct itself? So would it work when the pool >> gets imported with it's name befor the fstab entry is parsed? >> Hope that someone give me a few hints or a solution. > Few things: > - you don't need fstab entry for ZFS > - make sure you have zfs_enable="YES" in /etc/rc.conf > - make sure (perhaps this is your issue) /boot/zfs/zpool.cache is > writable. This is where ZFS remembers (among many things) what you > had previously imported. BTW why your /boot is ext2? > - check that "zfs get canmount home" returns "on" > Hello all, I was able to solve the problem actually. The thing was that two zfs cache files was existing on two partitions and so the imported pools were not be found. The give an answer to the question. I am actually in the case that I have a Xen paravirtualized FreeBSD i386 DomU which gets booted with PV GRUB. PV GRUB is GRUB compiled against Mini OS and has the intension to solve problems which exist with the kernel and initrd configuration file options and PyGRUB (GRUB written in Python). I first tried to boot from a ZFS volume which failed I think I made some mistakes and will try to figure it out again. UFS seems to be not usable with PV GRUB it detects UFS2 but can read files from there. This problem also couldn't be solved by chainloading because it seems that PV GRUB can not handle the loader files. So I thought to create and ext2 filesystem to bootstrap the system which works for now. There is a PV GRUB version which can handle ZFS but I wasn't able actually to create a ZFS pool which works as system disk. So I will try to figure this out. Best Regards From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 09:28:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AED6361A for ; Mon, 4 Mar 2013 09:28:35 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.9]) by mx1.freebsd.org (Postfix) with ESMTP id 327F68A6 for ; Mon, 4 Mar 2013 09:28:34 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mreu2) with ESMTP (Nemesis) id 0Lkkvg-1UkgaL0VKm-00b3t8; Mon, 04 Mar 2013 10:28:33 +0100 Message-ID: <5134693B.30408@brockmann-consult.de> Date: Mon, 04 Mar 2013 10:28:27 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: benefit of GEOM labels for ZFS, was Hard drive device names... serial numbers References: <512FE773.3060903@physics.umn.edu> In-Reply-To: <512FE773.3060903@physics.umn.edu> X-Enigmail-Version: 1.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:uy8FdlXG9bjXjjw1TLVEFoTIbkHmF9KbC9nLx/zfqMS gqmjdhe2UyapuzbujniGjifMtRFLXPy0P3GOrfs1MCVyV14HUG euArSWnStH0tzYHAXAKNg8LqltSFRXaWWyyazmP7cRME+gZ6VS 4JdEzy42nJOA/ZHm+fJ3G7LkC5/WrXYzWeM1fXnRM3zgt4hszp 6Rb3wPyIXS9dZs5fi/y3OYWxwCOLvrNAwjibp6ocX0iLf6LFQN 2HE+r83jp+kan3JqL3E4kG8p6RSzlswrf+Epdwc/F7RW5vsXgo LR39iQ0u0aUzN/hFye6cupaPsl/u28vNmjIIk3YJLILhIPSnHc uLz6VDUKDPHZ2w8tSY8XstQJz6noJdnzuc3GnFSPP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 09:28:35 -0000 I just use zfs offline, then dd to read the disk, and pull the one that blinks. :) zfs offline means the disk won't blink ever without me causing it, so there's no confusion. I would only use a labelling system if I could easily label the disks on the front too, but I don't have small enough labels... the disks have too much vent space, so I assume the labels would just fall off, block airflow, and be a hassle. And the servers are local, so dd isn't a problem. On 2013-03-01 00:25, Graham Allan wrote: > Sorry to come in late on this thread but I've been struggling with > thinking about the same issue, from a different perspective. > > Several months ago we created our first "large" ZFS storage system, > using 42 drives plus a few SSDs in one of the oft-used Supermicro > 45-drive chassis. It has been working really nicely but has led to > some puzzling over the best way to do some things when we build more. > > We made our pool using geom drive labels. Ever since, I've been > wondering if this really gives any advantage - at least for this type > of system. If you need to replace a drive, you don't really know which > enclosure slot any given da device is, and so our answer has been to > dig around using sg3_utils commands wrapped in a bit of perl, to try > and correlate the da device to the slot via the drive serial number. > > At this point, having a geom label just seems like an extra bit of > indirection to increase my confusion :-) Although setting the geom > label to the drive serial number might be a serious improvement... > > We're about to add a couple more of these shelves to the system, > giving a total of 135 drives (although each shelf would be a separate > pool), and given that they will be standard consumer grade drives, > some frequency of replacement is a given. > > Does anyone have any good tips on how to manage a large number of > drives in a zfs pool like this? > > Thanks, > > Graham -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de -------------------------------------------- From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 10:39:07 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1B301475 for ; Mon, 4 Mar 2013 10:39:07 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) by mx1.freebsd.org (Postfix) with ESMTP id 7FF17B07 for ; Mon, 4 Mar 2013 10:39:05 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r24AcsHk076938 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 4 Mar 2013 12:38:55 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <513479BE.1060109@digsys.bg> Date: Mon, 04 Mar 2013 12:38:54 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.12) Gecko/20130125 Thunderbird/10.0.12 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: benefit of GEOM labels for ZFS, was Hard drive device names... serial numbers References: <512FE773.3060903@physics.umn.edu> <5134693B.30408@brockmann-consult.de> In-Reply-To: <5134693B.30408@brockmann-consult.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 10:39:07 -0000 What you do when the disk is dead, and you don't know which one it is in a rather big rack full of disks? Perhaps, you offline each and every disk in the system until you eliminate all but one? :) Hint: all disks die sooner or later. Dnaiel On 04.03.13 11:28, Peter Maloney wrote: > I just use zfs offline, then dd to read the disk, and pull the one that > blinks. :) zfs offline means the disk won't blink ever without me > causing it, so there's no confusion. > > I would only use a labelling system if I could easily label the disks on > the front too, but I don't have small enough labels... the disks have > too much vent space, so I assume the labels would just fall off, block > airflow, and be a hassle. And the servers are local, so dd isn't a problem. > > From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 11:06:41 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D434DE9B for ; Mon, 4 Mar 2013 11:06:41 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id C6C30E43 for ; Mon, 4 Mar 2013 11:06:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r24B6fBO038733 for ; Mon, 4 Mar 2013 11:06:41 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r24B6fhw038731 for freebsd-fs@FreeBSD.org; Mon, 4 Mar 2013 11:06:41 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 4 Mar 2013 11:06:41 GMT Message-Id: <201303041106.r24B6fhw038731@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 11:06:41 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/174060 fs [ext2fs] Ext2FS system crashes (buffer overflow?) o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 299 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 16:10:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 10ACD42B for ; Mon, 4 Mar 2013 16:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id DADEA3C6 for ; Mon, 4 Mar 2013 16:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r24GA10v096621 for ; Mon, 4 Mar 2013 16:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r24GA1pB096620; Mon, 4 Mar 2013 16:10:01 GMT (envelope-from gnats) Date: Mon, 4 Mar 2013 16:10:01 GMT Message-Id: <201303041610.r24GA1pB096620@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Michael Moll Subject: Re: kern/162362: [snapshots] [panic] ufs with snapshot(s) panics when getting full X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Michael Moll List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 16:10:02 -0000 The following reply was made to PR kern/162362; it has been noted by GNATS. From: Michael Moll To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/162362: [snapshots] [panic] ufs with snapshot(s) panics when getting full Date: Mon, 4 Mar 2013 17:05:19 +0100 FYI, I can't reproduce this on a recent 9-STABLE anymore. It's OK to close this PR. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 16:37:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C3571ED1 for ; Mon, 4 Mar 2013 16:37:59 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp2.bway.net (smtp2.bway.net [216.220.96.28]) by mx1.freebsd.org (Postfix) with ESMTP id 8611E743 for ; Mon, 4 Mar 2013 16:37:59 +0000 (UTC) Received: from toasty.sporklab.com (foon.sporktines.com [96.57.144.66]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp2.bway.net (Postfix) with ESMTPSA id 54A4395853; Mon, 4 Mar 2013 11:27:59 -0500 (EST) References: <512FE773.3060903@physics.umn.edu> <5134693B.30408@brockmann-consult.de> <513479BE.1060109@digsys.bg> In-Reply-To: <513479BE.1060109@digsys.bg> Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii Message-Id: <6F419054-DC4E-43A7-8879-37BE54D10A47@bway.net> Content-Transfer-Encoding: quoted-printable From: Charles Sprickman Subject: Re: benefit of GEOM labels for ZFS, was Hard drive device names... serial numbers Date: Mon, 4 Mar 2013 11:27:58 -0500 To: Daniel Kalchev X-Mailer: Apple Mail (2.1085) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 16:37:59 -0000 On Mar 4, 2013, at 5:38 AM, Daniel Kalchev wrote: > What you do when the disk is dead, and you don't know which one it is = in a rather big rack full of disks? >=20 > Perhaps, you offline each and every disk in the system until you = eliminate all but one? :) One thing that I recently discovered is led(4). I have no idea how ubiquitous support for this is, but I see it on both = a Supermicro and a Tyan board. In /dev/led, things are labelled like = so: [spork@util2 ~]$ ls -l /dev/led/ total 0 crw------- 1 root wheel 0, 40 Feb 9 00:20 ahcich0.fault crw------- 1 root wheel 0, 39 Feb 9 00:20 ahcich0.locate crw------- 1 root wheel 0, 42 Feb 9 00:20 ahcich1.fault crw------- 1 root wheel 0, 41 Feb 9 00:20 ahcich1.locate crw------- 1 root wheel 0, 44 Feb 9 00:20 ahcich2.fault crw------- 1 root wheel 0, 43 Feb 9 00:20 ahcich2.locate crw------- 1 root wheel 0, 46 Feb 9 00:20 ahcich3.fault crw------- 1 root wheel 0, 45 Feb 9 00:20 ahcich3.locate crw------- 1 root wheel 0, 48 Feb 9 00:20 ahcich4.fault crw------- 1 root wheel 0, 47 Feb 9 00:20 ahcich4.locate crw------- 1 root wheel 0, 50 Feb 9 00:20 ahcich5.fault crw------- 1 root wheel 0, 49 Feb 9 00:20 ahcich5.locate If you pair that up with boot messages, you can probably sort out which = drive is which: [spork@util2 ~]$ grep "at ahcich1" /var/run/dmesg.boot=20 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 And then you can blink the "locate" LED on the sas/sata backplane: root@util2:/home/spork # echo f > /dev/led/ahcich0.locate And disable the blinking after you're done: root@util2:/home/spork # echo 0 > /dev/led/ahcich0.locate I'm sure this is all very hardware dependent, but if you have supported = hardware, it's an easy way to find what's where. Charles > Hint: all disks die sooner or later. >=20 > Dnaiel >=20 > On 04.03.13 11:28, Peter Maloney wrote: >> I just use zfs offline, then dd to read the disk, and pull the one = that >> blinks. :) zfs offline means the disk won't blink ever without me >> causing it, so there's no confusion. >>=20 >> I would only use a labelling system if I could easily label the disks = on >> the front too, but I don't have small enough labels... the disks have >> too much vent space, so I assume the labels would just fall off, = block >> airflow, and be a hassle. And the servers are local, so dd isn't a = problem. >>=20 >>=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 17:32:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 17CA0642 for ; Mon, 4 Mar 2013 17:32:15 +0000 (UTC) (envelope-from lkchen@k-state.edu) Received: from ksu-out.merit.edu (ksu-out.merit.edu [207.75.117.132]) by mx1.freebsd.org (Postfix) with ESMTP id D628F9F1 for ; Mon, 4 Mar 2013 17:32:14 +0000 (UTC) X-Merit-ExtLoop1: 1 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgEFABbaNFHPS3TT/2dsb2JhbAA6Cg6CNYQNuSCDYhZzgh8BAQQBI1YFBw8EJRkCWQaIIAayW4kziDqNUoNvgRMDiGyeRoFSd12CCQ X-IronPort-AV: E=Sophos;i="4.84,781,1355115600"; d="scan'208,217";a="207009898" X-MERIT-SOURCE: KSU Received: from ksu-sfpop-mailstore02.merit.edu ([207.75.116.211]) by sfpop-ironport07.merit.edu with ESMTP; 04 Mar 2013 12:32:08 -0500 Date: Mon, 4 Mar 2013 12:32:07 -0500 (EST) From: "Lawrence K. Chen, P.Eng." To: Gustau =?utf-8?Q?P=C3=A9rez?= i Querol Message-ID: <629654633.22396164.1362418327720.JavaMail.root@k-state.edu> In-Reply-To: <513216C6.6030108@entel.upc.edu> Subject: Re: benefit of GEOM labels for ZFS, was Hard drive device names... serial numbers MIME-Version: 1.0 X-Originating-IP: [129.130.0.181] X-Mailer: Zimbra 7.2.2_GA_2852 (ZimbraWebClient - GC25 ([unknown])/7.2.2_GA_2852) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 17:32:15 -0000 ----- Original Message ----- > Al 01/03/2013 23:52, En/na Lawrence K. Chen, P.Eng. ha escrit: > > I only have 15 drives...(12 HDDs and 3 SSDs) but the ordering of > > drives seemed to randomize on every boot (wonder now if the > > controller was doing some kind of staggering in spin ups. And, > > their other drivers cope with it. They provide a v1.1 driver for > > FreeBSD 7.2 or source to the v1.0 driver.) And, then everything > > moved around when I changed controllers a few times. > > > I had resorted at one point to putting device.hints to force all > > the > > drives to keep their mapping. Which caused problems elsewhere, and > > a mess when I added another controller. But, then I changed to > > more > > meaningful GPT labels and exported and re-imported my zpools with > > '-d /dev/gpt', and now things are ok. > > That reordering issue is what made me switch to geom labels first > (IIRC I did by 5.x era) and next switched to GPT labels. The GPT > allows me first to easily used those drives with ZFS and specially > because those labels belong to the partition scheme and thus are > filesystem independent. > What I found specially useful is to be able to identify those drives. > When I drive fails I know where it is because of the simple name > convention I use; but having a blinking led helps a lot, specially > when writing the recovery plan for the rest of the team. > Gus The GPT labels have come to be even more handy, because I had to switch out one (of two 5-bay) enclosures for a cheaper Rosewill one, due to a failing powersupply. The Rosewill one doesn't have LEDs. L From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 18:05:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7199BD81 for ; Mon, 4 Mar 2013 18:05:25 +0000 (UTC) (envelope-from lkchen@k-state.edu) Received: from ksu-out.merit.edu (ksu-out.merit.edu [207.75.117.132]) by mx1.freebsd.org (Postfix) with ESMTP id 3E8C1B3F for ; Mon, 4 Mar 2013 18:05:25 +0000 (UTC) X-Merit-ExtLoop1: 1 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av8EAN/hNFHPS3TT/2dsb2JhbABEhlC9AhZzgh8BAQQBI0QSBQcPDgoCAg0ZAlkGiCAGsi+JM4g6gSOQHoETA4hsnkaBUoFUggk X-IronPort-AV: E=Sophos;i="4.84,781,1355115600"; d="scan'208";a="906285647" X-MERIT-SOURCE: KSU Received: from ksu-sfpop-mailstore02.merit.edu ([207.75.116.211]) by sfpop-ironport05.merit.edu with ESMTP; 04 Mar 2013 13:05:13 -0500 Date: Mon, 4 Mar 2013 13:05:13 -0500 (EST) From: "Lawrence K. Chen, P.Eng." To: Charles Sprickman Message-ID: <1234083675.22411680.1362420313611.JavaMail.root@k-state.edu> In-Reply-To: <6F419054-DC4E-43A7-8879-37BE54D10A47@bway.net> Subject: Re: benefit of GEOM labels for ZFS, was Hard drive device names... serial numbers MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [129.130.0.181] X-Mailer: Zimbra 7.2.2_GA_2852 (ZimbraWebClient - GC25 ([unknown])/7.2.2_GA_2852) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 18:05:25 -0000 ----- Original Message ----- > On Mar 4, 2013, at 5:38 AM, Daniel Kalchev wrote: > > > What you do when the disk is dead, and you don't know which one it > > is in a rather big rack full of disks? > > > > Perhaps, you offline each and every disk in the system until you > > eliminate all but one? :) > > One thing that I recently discovered is led(4). > > I have no idea how ubiquitous support for this is, but I see it on > both a Supermicro and a Tyan board. In /dev/led, things are > labelled like so: > > [spork@util2 ~]$ ls -l /dev/led/ > total 0 > crw------- 1 root wheel 0, 40 Feb 9 00:20 ahcich0.fault > crw------- 1 root wheel 0, 39 Feb 9 00:20 ahcich0.locate > crw------- 1 root wheel 0, 42 Feb 9 00:20 ahcich1.fault > crw------- 1 root wheel 0, 41 Feb 9 00:20 ahcich1.locate > crw------- 1 root wheel 0, 44 Feb 9 00:20 ahcich2.fault > crw------- 1 root wheel 0, 43 Feb 9 00:20 ahcich2.locate > crw------- 1 root wheel 0, 46 Feb 9 00:20 ahcich3.fault > crw------- 1 root wheel 0, 45 Feb 9 00:20 ahcich3.locate > crw------- 1 root wheel 0, 48 Feb 9 00:20 ahcich4.fault > crw------- 1 root wheel 0, 47 Feb 9 00:20 ahcich4.locate > crw------- 1 root wheel 0, 50 Feb 9 00:20 ahcich5.fault > crw------- 1 root wheel 0, 49 Feb 9 00:20 ahcich5.locate > > If you pair that up with boot messages, you can probably sort out > which drive is which: > > [spork@util2 ~]$ grep "at ahcich1" /var/run/dmesg.boot > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > > And then you can blink the "locate" LED on the sas/sata backplane: > > root@util2:/home/spork # echo f > /dev/led/ahcich0.locate > > And disable the blinking after you're done: > > root@util2:/home/spork # echo 0 > /dev/led/ahcich0.locate > > I'm sure this is all very hardware dependent, but if you have > supported hardware, it's an easy way to find what's where. > > Charles > > Hmmm, interesting.... > ls -l /dev/led total 0 crw------- 1 root wheel 0, 59 Feb 27 23:28 ahcich0.fault crw------- 1 root wheel 0, 58 Feb 27 23:28 ahcich0.locate crw------- 1 root wheel 0, 61 Feb 27 23:28 ahcich1.fault crw------- 1 root wheel 0, 60 Feb 27 23:28 ahcich1.locate crw------- 1 root wheel 0, 63 Feb 27 23:28 ahcich2.fault crw------- 1 root wheel 0, 62 Feb 27 23:28 ahcich2.locate crw------- 1 root wheel 0, 65 Feb 27 23:28 ahcich3.fault crw------- 1 root wheel 0, 64 Feb 27 23:28 ahcich3.locate crw------- 1 root wheel 0, 67 Feb 27 23:28 ahcich4.fault crw------- 1 root wheel 0, 66 Feb 27 23:28 ahcich4.locate crw------- 1 root wheel 0, 69 Feb 27 23:28 ahcich5.fault crw------- 1 root wheel 0, 68 Feb 27 23:28 ahcich5.locate crw------- 1 root wheel 0, 48 Feb 27 23:28 siisch0 crw------- 1 root wheel 0, 49 Feb 27 23:28 siisch1 crw------- 1 root wheel 0, 54 Feb 27 23:28 siisch2 crw------- 1 root wheel 0, 55 Feb 27 23:28 siisch3 crw------- 1 root wheel 0, 56 Feb 27 23:28 siisch4 crw------- 1 root wheel 0, 57 Feb 27 23:28 siisch5 ahcich is the mobo controller, wonder where the LEDs are...don't recall seeing any when I've been poking around inside. But, the 10 external drives are on the siisch controllers. Actually 5 on siisch0 and 5 on siisch1. > dmesg | grep 'siisch0' siisch0: at channel 0 on siis0 pmp0 at siisch0 bus 0 scbus4 target 15 lun 0 ada4 at siisch0 bus 0 scbus4 target 0 lun 0 ada5 at siisch0 bus 0 scbus4 target 1 lun 0 ada6 at siisch0 bus 0 scbus4 target 2 lun 0 ada7 at siisch0 bus 0 scbus4 target 3 lun 0 ada8 at siisch0 bus 0 scbus4 target 4 lun 0 The enclosure (SANS Digital) on siisch0 has 2 LEDs at each drive position and a LED on the front for each. The enclosure on siisch1 was cheaper, and lacks the LEDs, but uses the same carriers, etc. I suppose if it really mattered, I could move the SANS Digital enclosure from my old fileserver to here, making both enclosures on old server the cheaper Rosewill ones.... Though I haven't figured out how to get persistent drive identifiers or such on ubuntu....and, since I'm just doing RAID1 across, and separate VGs....when a drive fails in a RAID set...either there's an LED or its the corresponding one in the other enclosure.... siisch[01] is a Sil3132 and siisch[2345] is a Sil3124 Though I'm thinking of switch one or both for asm1061 based controllers. Had put the idea on hold at first, but now that my video card is no longer supported by the latest nvidia-driver...I'm considering whether I want to risk getting a newer card (it was the 6th card I tried before I had something mostly working...) Only some of my drives are SATA III though. But all 3 SSDs are....and think the ones I'm using for ZIL/L2ARC might benefit. Perhaps I need 3 asm1061's.... L From owner-freebsd-fs@FreeBSD.ORG Mon Mar 4 22:32:09 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6F02197F for ; Mon, 4 Mar 2013 22:32:09 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id F396318D6 for ; Mon, 4 Mar 2013 22:32:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r24MVpCK035394 for ; Tue, 5 Mar 2013 02:31:51 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Tue, 5 Mar 2013 02:31:51 +0400 (MSK) From: Dmitry Morozovsky To: freebsd-fs@FreeBSD.org Subject: carp on stable/9: is there a way to keep jumbo? Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (woozle.rinet.ru [0.0.0.0]); Tue, 05 Mar 2013 02:31:52 +0400 (MSK) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2013 22:32:09 -0000 Dear collesagues, yes, I know glebius@ overhauled carp in -current, but I'm a bit nervous to deploy bleeding edge system on a NAS/SAN ;) So, my question is about current state of carp in stable/9: building HA pair I found that carp interfaces lose jumbo capabilities: root@cthulhu4:~# ifconfig | grep mtu em0: flags=8943 metric 0 mtu 9000 em1: flags=8943 metric 0 mtu 9000 lo0: flags=8049 metric 0 mtu 16384 lagg0: flags=8943 metric 0 mtu 9000 carp0: flags=49 metric 0 mtu 1500 carp1: flags=49 metric 0 mtu 1500 root@cthulhu4:~# ifconfig carp1 mtu 9000 ifconfig: ioctl (set mtu): Invalid argument Is it unavoidable at the moment, or am I missing something obvious? Thanks! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Tue Mar 5 02:05:04 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 87611B97 for ; Tue, 5 Mar 2013 02:05:04 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-we0-x234.google.com (mail-we0-x234.google.com [IPv6:2a00:1450:400c:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id F2906869 for ; Tue, 5 Mar 2013 02:05:03 +0000 (UTC) Received: by mail-we0-f180.google.com with SMTP id k14so5181208wer.25 for ; Mon, 04 Mar 2013 18:05:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:reply-to:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=OAzXE2vTqMtLwkfC0mzn9GPWSfOXhKHotswhKoPr/cQ=; b=K7+bfc14t4SC9TrjxrPzVMyp/YA/ShqNPvCPgJMpJeSlJC1z9bmnKDH37aV6OYOv4c l0zaA5xZm90dtcQ3aN56sVaCiNZBdGUjDydm/3XvDnRuXztzYpOT++QjzCe+NPQ4gmfn 3gR2IMT5SE04ZizBt98B7GjGpWcSVoBJUYxeUFQbPE6KaSP0eZnwLGXtCf0E2JbFRCDt p+Bp7uIlPLu8S/adId6IAvUelzvxv5kyoFhGsesLSYI0/PFHh6nhPtiLCOGybmd24DYm ULM+0L2SuGnp/uQ7YP/ATabgcZIk5rdIleU8f7XAc3CtrYyMWj5sU0ZhYtgLB7HpstNS aFqw== MIME-Version: 1.0 X-Received: by 10.194.21.233 with SMTP id y9mr29002752wje.47.1362449103221; Mon, 04 Mar 2013 18:05:03 -0800 (PST) Received: by 10.180.212.51 with HTTP; Mon, 4 Mar 2013 18:05:03 -0800 (PST) In-Reply-To: References: Date: Tue, 5 Mar 2013 10:05:03 +0800 Message-ID: Subject: Re: carp on stable/9: is there a way to keep jumbo? From: Marcelo Araujo To: Dmitry Morozovsky Content-Type: text/plain; charset=KOI8-R X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Mar 2013 02:05:04 -0000 2013/3/5 Dmitry Morozovsky > Dear collesagues, > > yes, I know glebius@ overhauled carp in -current, but I'm a bit nervous to > deploy bleeding edge system on a NAS/SAN ;) > > So, my question is about current state of carp in stable/9: building HA > pair I > found that carp interfaces lose jumbo capabilities: > > Hello Dmitry, I made a patch for 9.1-RELEASE, it is totally based on glebius@ work, or partially :). I'm using it nowadays and it just works pretty fine for me. I didn't test with JUMBO frame, but you can give a try and let us know if it works or not. PATCH: http://people.freebsd.org/~araujo/carpdev/ Best Regards, -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Tue Mar 5 20:12:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7A8B7B1A for ; Tue, 5 Mar 2013 20:12:41 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3E8E4AAA for ; Tue, 5 Mar 2013 20:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=cXE3PpzwTRudZMrAfe/3dUFnRowVw98QEvJ5tRe1nSo=; b=CV7+SE72M33B2G3vrS60TEKH/e20M3eBZO+PQU2J6ODguJKcb+QUy2tEusOReGCKeLxs2wwv5i+0ecAcP8+qNCh0sijPmIg5tvfgvUj0ro1V1BlkgDmgKr+iVUE3etcVcR3xqVLEZUB+mp+x4LdfYvxwTtGQskmcVYQmr4sdhso=; Received: from localhost.lerctr.org ([127.0.0.1]:17027 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UCyE4-000A7I-78 for freebsd-fs@freebsd.org; Tue, 05 Mar 2013 14:12:40 -0600 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 05 Mar 2013 14:12:40 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 05 Mar 2013 14:12:40 -0600 From: Larry Rosenman To: Subject: zfs send/recv invalid data Message-ID: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Mar 2013 20:12:41 -0000 I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to 10.0) of a -R -I stream. What data do I need to gather to figure out what side and what's wrong? I've already started zpool scrubs on both sides. I can insert a tee to grab the stream on either/both sides if that would help. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Tue Mar 5 20:47:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CF9EC3D4 for ; Tue, 5 Mar 2013 20:47:05 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews05.kpnxchange.com (cpsmtpb-ews05.kpnxchange.com [213.75.39.8]) by mx1.freebsd.org (Postfix) with ESMTP id 4887CCC0 for ; Tue, 5 Mar 2013 20:47:04 +0000 (UTC) Received: from cpsps-ews25.kpnxchange.com ([10.94.84.191]) by cpsmtpb-ews05.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 5 Mar 2013 21:44:31 +0100 Received: from CPSMTPM-TLF102.kpnxchange.com ([195.121.3.5]) by cpsps-ews25.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 5 Mar 2013 21:44:31 +0100 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF102.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 5 Mar 2013 21:45:56 +0100 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 1D34460DA; Tue, 5 Mar 2013 21:45:56 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Larry Rosenman" Subject: Re: zfs send/recv invalid data References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> Date: Tue, 05 Mar 2013 21:45:55 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> User-Agent: Opera Mail/12.14 (FreeBSD) X-OriginalArrivalTime: 05 Mar 2013 20:45:56.0482 (UTC) FILETIME=[6F882620:01CE19E2] X-RcptDomain: freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Mar 2013 20:47:05 -0000 On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman wrote: > I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to > 10.0) of a -R -I stream. > > What data do I need to gather to figure out what side and what's wrong? > > I've already started zpool scrubs on both sides. > > I can insert a tee to grab the stream on either/both sides if that would > help. > > > Is the problem repeatable or is it just a network glitch? Ronald. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 5 20:52:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7C603575 for ; Tue, 5 Mar 2013 20:52:34 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3D122CF5 for ; Tue, 5 Mar 2013 20:52:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=LTuv0eNiIb8asbUFi1eEbXoSkozuUrQ5RPdYbDqp7/A=; b=cbMrfUJaHq+0kQn/t4OPj3LozC4qbQ9nRCARMJxu1wM00YpxGek0hDfg3jI0KefC09hcRMPbf23zfXzZk4mxN6YRCZF3U8ng8BxoNVJffPCPFmpfBnCxO9eS1kCgwCo4iwjqXy32V9sEqNz9Ng4RooZZrDvPstqozMgmwg7vfg4=; Received: from localhost.lerctr.org ([127.0.0.1]:53649 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UCyqf-000AWP-5X; Tue, 05 Mar 2013 14:52:33 -0600 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 05 Mar 2013 14:52:32 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 05 Mar 2013 14:52:32 -0600 From: Larry Rosenman To: Ronald Klop Subject: Re: zfs send/recv invalid data In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> Message-ID: <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Mar 2013 20:52:34 -0000 On 2013-03-05 14:45, Ronald Klop wrote: > On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman > wrote: > >> I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to >> 10.0) of a -R -I stream. >> >> What data do I need to gather to figure out what side and what's >> wrong? >> >> I've already started zpool scrubs on both sides. >> >> I can insert a tee to grab the stream on either/both sides if that >> would help. >> >> >> > > Is the problem repeatable or is it just a network glitch? > > Ronald. Repeatable....... -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 00:13:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 257AC70 for ; Wed, 6 Mar 2013 00:13:51 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id CAB1263E for ; Wed, 6 Mar 2013 00:13:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=ZmheHPK/chYQ1nIz/4t0RWHwGzljH68koPoXWyuDqHY=; b=UqlSGHDgs11Zpa1Fv0pcKo76/qHJBqYeb4XZkJAB9Vo1uBDn5YcsJU5nUBP3qXq80YW4ukmuv1NII+3Ljpmju8AdcH/mweOtkbd47nXygRNohWjZVamzNbpqqWYgnGduf5rtStpWLAZ01dyZeEGCcLz7GXe8sqcP13we0bIsi/M=; Received: from localhost.lerctr.org ([127.0.0.1]:62515 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UD1zQ-000CGc-RC; Tue, 05 Mar 2013 18:13:49 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 05 Mar 2013 18:13:47 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 05 Mar 2013 18:13:47 -0600 From: Larry Rosenman To: Ronald Klop Subject: Re: zfs send/recv invalid data In-Reply-To: <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> Message-ID: <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 00:13:51 -0000 On 2013-03-05 14:52, Larry Rosenman wrote: > On 2013-03-05 14:45, Ronald Klop wrote: >> On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman >> wrote: >> >>> I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to >>> 10.0) of a -R -I stream. >>> What data do I need to gather to figure out what side and what's >>> wrong? >>> I've already started zpool scrubs on both sides. >>> I can insert a tee to grab the stream on either/both sides if that >>> would help. >>> >>> >>> >> Is the problem repeatable or is it just a network glitch? >> Ronald. > Repeatable....... Here is the exact error message: receiving incremental stream of vault/home/ctr@2013-03-05-test3 into zroot/backups/TBH/home/ctr@2013-03-05-test3 cannot receive incremental stream: invalid backup stream this is the script I'm running: #!/bin/sh DATE=`date "+%Y-%m-%d-BUG-REPRO"` DATE2=`date -v "-1d" "+%Y-%m-%d"` # snap the source ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} # zfs copy the source to here. ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} vault@${DATE} | \ tee /tmp/backup.stream.send.${DATE} | \ ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs recv -u -v -d zroot/backups/TBH\"" # make sure we NEVER allow the backup stuff to automount. /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh both streams are in http://www.lerctr.org/~ler/ZFS_RECV as well as a zfs list -t all from both sides..... HELP.... -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 00:48:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CF937B14 for ; Wed, 6 Mar 2013 00:48:39 +0000 (UTC) (envelope-from prvs=1777d4b2b8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 51D5D783 for ; Wed, 6 Mar 2013 00:48:39 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002568159.msg for ; Wed, 06 Mar 2013 00:48:38 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 06 Mar 2013 00:48:38 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1777d4b2b8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <612228ABB8BC423D907C9538783667A4@multiplay.co.uk> From: "Steven Hartland" To: "Larry Rosenman" , "Ronald Klop" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> Subject: Re: zfs send/recv invalid data Date: Wed, 6 Mar 2013 00:47:54 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 00:48:39 -0000 ----- Original Message ----- From: "Larry Rosenman" > Here is the exact error message: > receiving incremental stream of vault/home/ctr@2013-03-05-test3 into > zroot/backups/TBH/home/ctr@2013-03-05-test3 > cannot receive incremental stream: invalid backup stream > > this is the script I'm running: > #!/bin/sh > DATE=`date "+%Y-%m-%d-BUG-REPRO"` > DATE2=`date -v "-1d" "+%Y-%m-%d"` > # snap the source > ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} > # zfs copy the source to here. > ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} > vault@${DATE} | \ > tee /tmp/backup.stream.send.${DATE} | \ > ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs > recv -u -v -d zroot/backups/TBH\"" > # make sure we NEVER allow the backup stuff to automount. > /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ > awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh > > both streams are in http://www.lerctr.org/~ler/ZFS_RECV > > as well as a zfs list -t all from both sides..... How recent is you receiving box? Does verbose give you any more information? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 00:53:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 03950E74 for ; Wed, 6 Mar 2013 00:53:01 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id C1E287C2 for ; Wed, 6 Mar 2013 00:53:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:References:Message-ID:In-Reply-To:Subject:cc:To:From:Date; bh=PlPMrofuptvChD68UeBWFf3wqTgJ5qeTizLBacyYG68=; b=QOgHhLTT+XJA6RjRdCcW115eRhlUi9oc53pRi0R/K/OAzIiphxZECSPDCwucrUOtPfERJlmy228vVs3svRrdtSiKxQIErz720MifcVTE0yTI8YNPWhIa+2xcP+RHZiCA7/APUNiBYweDuD9UaRrZEOjhgRW5eZftvdOY4PffWSw=; Received: from lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net ([2001:470:1f0e:3ad::2]:30174) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UD2bJ-000CdF-ST; Tue, 05 Mar 2013 18:52:58 -0600 Date: Tue, 5 Mar 2013 18:52:54 -0600 (CST) From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <612228ABB8BC423D907C9538783667A4@multiplay.co.uk> Message-ID: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <612228ABB8BC423D907C9538783667A4@multiplay.co.uk> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 00:53:01 -0000 On Wed, 6 Mar 2013, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" >> Here is the exact error message: >> receiving incremental stream of vault/home/ctr@2013-03-05-test3 into >> zroot/backups/TBH/home/ctr@2013-03-05-test3 >> cannot receive incremental stream: invalid backup stream >> >> this is the script I'm running: >> #!/bin/sh >> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >> DATE2=`date -v "-1d" "+%Y-%m-%d"` >> # snap the source >> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >> # zfs copy the source to here. >> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} vault@${DATE} | >> \ >> tee /tmp/backup.stream.send.${DATE} | \ >> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs >> recv -u -v -d zroot/backups/TBH\"" >> # make sure we NEVER allow the backup stuff to automount. >> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >> >> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >> >> as well as a zfs list -t all from both sides..... > > How recent is you receiving box? FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon Mar 4 19:59:08 CST 2013 root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 > > Does verbose give you any more information? that is -v output. Other ideas? > > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. and the > person or entity to whom it is addressed. In the event of misdirection, the > recipient is prohibited from using, copying, printing or otherwise > disseminating it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 02:35:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 90806338 for ; Wed, 6 Mar 2013 02:35:44 +0000 (UTC) (envelope-from prvs=1777d4b2b8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 131A5B0E for ; Wed, 6 Mar 2013 02:35:43 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002570239.msg for ; Wed, 06 Mar 2013 02:35:42 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 06 Mar 2013 02:35:42 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1777d4b2b8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Larry Rosenman" , "Ronald Klop" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> Subject: Re: zfs send/recv invalid data Date: Wed, 6 Mar 2013 02:35:46 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 02:35:44 -0000 ----- Original Message ----- From: "Larry Rosenman" To: "Ronald Klop" Cc: Sent: Wednesday, March 06, 2013 12:13 AM Subject: Re: zfs send/recv invalid data > On 2013-03-05 14:52, Larry Rosenman wrote: >> On 2013-03-05 14:45, Ronald Klop wrote: >>> On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman >>> wrote: >>> >>>> I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to >>>> 10.0) of a -R -I stream. >>>> What data do I need to gather to figure out what side and what's >>>> wrong? >>>> I've already started zpool scrubs on both sides. >>>> I can insert a tee to grab the stream on either/both sides if that >>>> would help. >>>> >>>> >>>> >>> Is the problem repeatable or is it just a network glitch? >>> Ronald. >> Repeatable....... > > > Here is the exact error message: > receiving incremental stream of vault/home/ctr@2013-03-05-test3 into > zroot/backups/TBH/home/ctr@2013-03-05-test3 > cannot receive incremental stream: invalid backup stream > > this is the script I'm running: > #!/bin/sh > DATE=`date "+%Y-%m-%d-BUG-REPRO"` > DATE2=`date -v "-1d" "+%Y-%m-%d"` > # snap the source > ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} > # zfs copy the source to here. > ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} > vault@${DATE} | \ > tee /tmp/backup.stream.send.${DATE} | \ > ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs > recv -u -v -d zroot/backups/TBH\"" > # make sure we NEVER allow the backup stuff to automount. > /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ > awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh > > both streams are in http://www.lerctr.org/~ler/ZFS_RECV Your send and receive sides differ, which indicates your ssh shell my not be clean. Looking at the receive side its got what looks like a mail message appended. I suspect if you manually copy the receive copy to the 10 machine and the receive it will work fine. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 02:42:12 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C6D12466 for ; Wed, 6 Mar 2013 02:42:12 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5FCE9B3C for ; Wed, 6 Mar 2013 02:42:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=rOrT9qt6sAZyCKBVFFSxU2D4xA44qV7JX2f4+wog2cQ=; b=DSrm1lATp567Ji+NfmLwR2gx9n0noOLqpoRz1iI5lLG7QO+vrsjAuXg/hw3FpuFX7TSNa3kNRhdiUh3KV7N/SeugMxLEdZDMMzhAADkkfvcXVDbREkw2A06FkJjo8TFTLmbrh5+AaJCdmhdhDGjTmOI4mZFQPjPaCnh0kkMHheQ=; Received: from localhost.lerctr.org ([127.0.0.1]:48973 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UD4Ix-000DXn-Mq; Tue, 05 Mar 2013 20:42:08 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 05 Mar 2013 20:42:06 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 05 Mar 2013 20:42:06 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> Message-ID: <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 02:42:12 -0000 On 2013-03-05 20:35, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" > To: "Ronald Klop" > Cc: > Sent: Wednesday, March 06, 2013 12:13 AM > Subject: Re: zfs send/recv invalid data > > >> On 2013-03-05 14:52, Larry Rosenman wrote: >>> On 2013-03-05 14:45, Ronald Klop wrote: >>>> On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman >>>> wrote: >>>> >>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs recv >>>>> (to 10.0) of a -R -I stream. >>>>> What data do I need to gather to figure out what side and what's >>>>> wrong? >>>>> I've already started zpool scrubs on both sides. >>>>> I can insert a tee to grab the stream on either/both sides if that >>>>> would help. >>>>> >>>>> >>>>> >>>> Is the problem repeatable or is it just a network glitch? >>>> Ronald. >>> Repeatable....... >> >> Here is the exact error message: >> receiving incremental stream of vault/home/ctr@2013-03-05-test3 into >> zroot/backups/TBH/home/ctr@2013-03-05-test3 >> cannot receive incremental stream: invalid backup stream >> this is the script I'm running: >> #!/bin/sh >> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >> DATE2=`date -v "-1d" "+%Y-%m-%d"` >> # snap the source >> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >> # zfs copy the source to here. >> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} >> vault@${DATE} | \ >> tee /tmp/backup.stream.send.${DATE} | \ >> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | >> zfs recv -u -v -d zroot/backups/TBH\"" >> # make sure we NEVER allow the backup stuff to automount. >> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >> both streams are in http://www.lerctr.org/~ler/ZFS_RECV > > Your send and receive sides differ, which indicates your ssh > shell my not be clean. > > Looking at the receive side its got what looks like a mail > message appended. > > I suspect if you manually copy the receive copy to the 10 machine and > the receive it will work fine. we're copying mail files........ and it still fails.... > > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 02:54:04 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8082E54F for ; Wed, 6 Mar 2013 02:54:04 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0F1D3B79 for ; Wed, 6 Mar 2013 02:54:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:References:Message-ID:In-Reply-To:Subject:cc:To:From:Date; bh=1u5jxRW5bJMogSrGWQKCmbeUVE4GxoqPfA+aUO3tqa4=; b=kEmXHmOIZ21lFFLVouJ/ALcLKVqt21T97turqV2ARd7lbYqMMYOFGmdxiHITfM1HecYlJ4wU3teew6F4y32lx7cfORAkCca8mUZXSPBNgAaM88TVeJcPo/iW2ZadRTyXxPZaT1x+NbHKZ2EcvRvGS0nA+Y+62qc+09vY3s2cHUo=; Received: from lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net ([2001:470:1f0e:3ad::2]:39702) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UD4UL-000DfG-GI; Tue, 05 Mar 2013 20:54:00 -0600 Date: Tue, 5 Mar 2013 20:53:50 -0600 (CST) From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> Message-ID: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 02:54:04 -0000 On Tue, 5 Mar 2013, Larry Rosenman wrote: > On 2013-03-05 20:35, Steven Hartland wrote: >> ----- Original Message ----- From: "Larry Rosenman" >> To: "Ronald Klop" >> Cc: >> Sent: Wednesday, March 06, 2013 12:13 AM >> Subject: Re: zfs send/recv invalid data >> >> >>> On 2013-03-05 14:52, Larry Rosenman wrote: >>>> On 2013-03-05 14:45, Ronald Klop wrote: >>>>> On Tue, 05 Mar 2013 21:12:40 +0100, Larry Rosenman >>>>> wrote: >>>>> >>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to >>>>>> 10.0) of a -R -I stream. >>>>>> What data do I need to gather to figure out what side and what's wrong? >>>>>> I've already started zpool scrubs on both sides. >>>>>> I can insert a tee to grab the stream on either/both sides if that >>>>>> would help. >>>>>> >>>>>> >>>>>> >>>>> Is the problem repeatable or is it just a network glitch? >>>>> Ronald. >>>> Repeatable....... >>> >>> Here is the exact error message: >>> receiving incremental stream of vault/home/ctr@2013-03-05-test3 into >>> zroot/backups/TBH/home/ctr@2013-03-05-test3 >>> cannot receive incremental stream: invalid backup stream >>> this is the script I'm running: >>> #!/bin/sh >>> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >>> DATE2=`date -v "-1d" "+%Y-%m-%d"` >>> # snap the source >>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >>> # zfs copy the source to here. >>> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} vault@${DATE} >>> | \ >>> tee /tmp/backup.stream.send.${DATE} | \ >>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs >>> recv -u -v -d zroot/backups/TBH\"" >>> # make sure we NEVER allow the backup stuff to automount. >>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >> >> Your send and receive sides differ, which indicates your ssh >> shell my not be clean. >> >> Looking at the receive side its got what looks like a mail >> message appended. >> >> I suspect if you manually copy the receive copy to the 10 machine and >> the receive it will work fine. > > we're copying mail files........ > > and it still fails.... > I've put more example send/recv files in that directory. we're copying home dirs, which include lots of mail. (this one is my wife's) Ideas? I *CAN* give access to both sides via ssh..... -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 06:43:20 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3AED3E2C; Wed, 6 Mar 2013 06:43:20 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 05408405; Wed, 6 Mar 2013 06:43:19 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r266hBWU015053; Tue, 5 Mar 2013 22:43:15 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303060643.r266hBWU015053@gw.catspoiler.org> Date: Tue, 5 Mar 2013 22:43:11 -0800 (PST) From: Don Lewis Subject: Re: Panic in ffs_valloc (Was: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS!) To: lev@FreeBSD.org In-Reply-To: <1352492388.20130302002244@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-5 Content-Transfer-Encoding: 8BIT Cc: mckusick@mckusick.com, freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 06:43:20 -0000 On 2 Mar, Lev Serebryakov wrote: > Hello, Kirk. > You wrote 1 марта 2013 г., 22:00:51: > >>> As far, as I understand, if this theory is right (file system >>> corruption which left unnoticed by "standard" fsck), it is bug in FFS >>> SU+J too, as it should not be corrupted by reordered writes (if >>> writes is properly reported as completed even if they were >>> reordered). > KM> If the bitmaps are left corrupted (in particular if blocks are marked > KM> free that are actually in use), then that panic can occur. Such a state > KM> should never be possible when running with SU even if you have crashed > KM> multiple times and restarted without running fsck. > I run fsck every time (ok, every half-year) when server crashes due > to my awkward experiments on live system, but I run it as it runs: > with journal after upgrade to 9-STABLE, not full old-fashioned run. > > KM> To reduce the number of possible points of failure, I suggest that > KM> you try running with just SU (i.e., turn off the SU+J jornalling). > KM> you can do this with `tunefs -j disable /dev/fsdisk'. This will > KM> turn off journalling, but not soft updates. You can verify this > KM> by then running `tunefs -p /dev/fsdisk' to ensure that soft updates > KM> are still enabled. > And wait another half a year :) > > I'm trying to reproduce this situation on VM (VirtualBox with > virtual HDDs), but no luck (yet?). > > KM> I will MFC 246876 and 246877 once they have been in head long enough > KM> to have confidence that they will not cause trouble. That means at > KM> least a month (well more than the two weeks they have presently been > KM> there). > > KM> Note these changes only pass the barrier request down to the GEOM > KM> layer. I don't know whether it actually makes it to the drive layer > KM> and if it does whether the drive layer actually implements it. My > KM> goal was to get the ball rolling. > I'm have controversial feelings about this barriers. IMHO, all > writes to UFS (FFS) could and should be divided into two classes: > data writes and metadata (including journal, as FFS doesn't have data > journaling) writes. IMHO (it is last time I type these 4 letters, > but, please, add it when you read this before and after each my > sentence, as I'm not FS expert at any grade), data writes could be > done as best effort till fsync() is called (or file is opened with > appropriate flag, which is equivalent to automatic fsync() after each > write). They could be delayed, reordered, etc. But metadata should > have some strong guarantees (and fsync()'ed data too, of course). > Such division could allow best possible performance & consistent FS > metadata (maybe not consistent user data -- but every application > which needs strong guarantees, like RDBMS, use fsync() anyway). When growing a file, the data *must* be written before writing the block pointer that points to it. If this ordering isn't obeyed, then a system crash that occurs between the block pointer write and the data write would result in the file containing whatever garbage was in the data block. That garbage could be the confidential contents of some other user's previously deleted file. > Now you add "BARRIER" write. It looks too strong to use it often. > It will force writing of ALL data from caches, even if your intention > is to write only 2 or 3 blocks of metadata. It could solve problems > with FS metadata, but it will degrade performance, especially in > multithreaded load. Update of inode map for creating 0 bytes file > flag by one process (protected with barrier) will flush whole data > cache (maybe, hundred of meagbytes) of other one. I'm not a fan of barriers for that reason. My understanding is that older versions of the Linux kernel were pretty lax about write ordering in the ext3 filesystem and that recent kernels enable barriers for ext3 and ext4. I've heard a lot of complaints about mysql peformance tanking with these newer kernels and the commonly suggested workaround is to disable barriers and make sure the system is on an UPS (though I've had plenty of UPS failures over the years). Fortunately in the case of 246877, the barrier operation should happen infrequently, especially after the filesystem has been in use for a while. > It is better than noting, but, it is not best solution. Every write > should be marked as "critical" or "loose" and critical-marked buffers > (BIOs) must be written ASAP and before all other _crtitcal_ BIOs (not > all BIOs after it with or without flag). So, barrier should affect > only other barriers (ordered writes). Default, "loose" semantic (for > data) will exactly what we have now. > > It is very hard to implement contract "It only ensure that buffers > written before that buffer will get to the media before any buffers > written after that buffer" in any other way but full flush, which, as I > stated above, will hurt performace in such cases as effective > RAID5-like implementations which gain a lot from combining wrties > together by spatial (not time) property. > > And for full flush (which is needed sometimes, of course) we > already have BIO_FLUSH command. > > Anyway, I'll support new semantic in geom_raid5 ASAP. But, > unfortunately, now it could be supported as it is simple write > followed by BIO_FLUSH -- not very effective :( > From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 06:52:15 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7C672FE9; Wed, 6 Mar 2013 06:52:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 35E2262D; Wed, 6 Mar 2013 06:52:15 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:9421:367:9d7d:512b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 843EF4AC57; Wed, 6 Mar 2013 10:52:07 +0400 (MSK) Date: Wed, 6 Mar 2013 10:52:05 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <958644234.20130306105205@serebryakov.spb.ru> To: Don Lewis Subject: Re: Panic in ffs_valloc (Was: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS!) In-Reply-To: <201303060643.r266hBWU015053@gw.catspoiler.org> References: <1352492388.20130302002244@serebryakov.spb.ru> <201303060643.r266hBWU015053@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-5 Content-Transfer-Encoding: quoted-printable Cc: mckusick@mckusick.com, freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 06:52:15 -0000 Hello, Don. You wrote 6 =DC=D0=E0=E2=D0 2013 =D3., 10:43:11: DL> When growing a file, the data *must* be written before writing the block DL> pointer that points to it. If this ordering isn't obeyed, then a system DL> crash that occurs between the block pointer write and the data write DL> would result in the file containing whatever garbage was in the data DL> block. That garbage could be the confidential contents of some other DL> user's previously deleted file. It is why confidential data should be zeored-out before file deletion :) But here is another way: add "stream id" for all writes. FS could mark each write by vnode address (or inode number + FS id) and drivers, which need high performance could add additional logic to do selective barriers. All other FSes (which don't need barriers) and drivers (which don't need this optimization) will work as it is now. It doesn't look like tight coupling, as this stream id could be anything FS want (information of FS structure will not leak though it) and 0 if FS don't want to use this feature. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 07:15:23 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 86796950; Wed, 6 Mar 2013 07:15:23 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 5220C6F9; Wed, 6 Mar 2013 07:15:23 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r267FDHS015118; Tue, 5 Mar 2013 23:15:17 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303060715.r267FDHS015118@gw.catspoiler.org> Date: Tue, 5 Mar 2013 23:15:13 -0800 (PST) From: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! To: lev@FreeBSD.org In-Reply-To: <612776324.20130301152756@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-5 Content-Transfer-Encoding: 8BIT Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 07:15:23 -0000 On 1 Mar, Lev Serebryakov wrote: > Hello, Ivan. > You wrote 28 февраля 2013 г., 21:01:46: > >>> One time, Kirk say, that delayed writes are Ok for SU until bottom >>> layer doesn't lie about operation completeness. geom_raid5 could >>> delay writes (in hope that next writes will combine nicely and allow >>> not to do read-calculate-write cycle for read alone), but it never >>> mark BIO complete until it is really completed (layers down to >>> geom_raid5 returns completion). So, every BIO in wait queue is "in >>> flight" from GEOM/VFS point of view. Maybe, it is fatal for journal :( > IV> It shouldn't be - it could be a bug. > I understand, that it proves nothing, but I've tried to repeat > "previous crash corrupt FS in journal-undetectable way" theory by > killing virtual system when there is massive writing to > geom_radi5-based FS (on virtual drives, unfortunately). I've done 15 > tries (as it is manual testing, it takes about 1-1.5 hours total), > but every time FS was Ok after double-fsck (first with journal and > last without one). Of course, there was MASSIVE loss of data, as > timeout and size of cache in geom_raid5 was set very high (sometimes > FS becomes empty after unpacking 50% of SVN mirror seed, crash and > check) but FS was consistent every time! Did you have any power failures that took down the system sometime before this panic occured? By default FreeBSD enables write caching on ATA drives. kern.cam.ada.write_cache: 1 kern.cam.ada.0.write_cache: -1 (-1 => use system default value) That means that the drive will immediately acknowledge writes and is free to reorder them as it pleases. When UFS+SU allocates a new inode, it first clears the available bit in the bitmap and writes the bitmap block to disk before it writes the new inode contents to disk. When a file is deleted, the inode is zeroed on disk before the available bit is set in the bitmap and the bitmap block is written. That means that if an inode is marked as available in the bitmap, then it should be zero. The system panic that you experienced happened when the system was attempting to allocate an inode for a new file and when it peeked at an inode that was marked as available, it found that the inode was non-zero. What might have happened is that sometime in the past, the system was in the process of creating a new file when a power failure ocurred. It found an available inode, marked it as unavailable in the bitmap, and write the bitmap block to the drive. Because write caching was enabled, the bitmap block was cached in the drive's write cache, and the drive said that the write was complete. After getting this response, UFS+SU wrote the new inode contents to the drive, which was also cached. The drive then wrote the inode contents to the drive. At this point the power failed, losing all of the contents of the drive write cache before the bitmap block was updated. When the system was powered up again, fsck just replayed the journal because you were using SU+J, and didn't detect the inconsistency between the bitmap and the actual inode contents (which would require a full fsck). This damage could remain latent for quite some time, and wouldn't be found until the filesystem tried to allocate the inode in question. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 07:32:55 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 94015FC0; Wed, 6 Mar 2013 07:32:55 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 58F607A9; Wed, 6 Mar 2013 07:32:55 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:9421:367:9d7d:512b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 3CB884AC58; Wed, 6 Mar 2013 11:32:53 +0400 (MSK) Date: Wed, 6 Mar 2013 11:32:50 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <1644513757.20130306113250@serebryakov.spb.ru> To: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! In-Reply-To: <201303060715.r267FDHS015118@gw.catspoiler.org> References: <612776324.20130301152756@serebryakov.spb.ru> <201303060715.r267FDHS015118@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-5 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 07:32:55 -0000 Hello, Don. You wrote 6 =DC=D0=E0=E2=D0 2013 =D3., 11:15:13: DL> Did you have any power failures that took down the system sometime DL> before this panic occured? By default FreeBSD enables write caching on I had other panic due to my inaccurate hands... But I don't remeber any power failures, as I havee UPS and this one works (I check it every month). DL> That means that the drive will immediately acknowledge writes and is DL> free to reorder them as it pleases. DL> When UFS+SU allocates a new inode, it first clears the available bit in DL> the bitmap and writes the bitmap block to disk before it writes the new DL> inode contents to disk. When a file is deleted, the inode is zeroed on DL> disk before the available bit is set in the bitmap and the bitmap block DL> is written. That means that if an inode is marked as available in the DL> bitmap, then it should be zero. The system panic that you experienced DL> happened when the system was attempting to allocate an inode for a new DL> file and when it peeked at an inode that was marked as available, it DL> found that the inode was non-zero. DL> What might have happened is that sometime in the past, the system was in >[SKIPPED] DL> tried to allocate the inode in question. This scenario looks plausible, but it raises another question: does barriers will protect against it? It doesn't look so, as now barrier write is issued only when new inode BLOCK is allocated. And it leads us to my other question: why did not mark such vital writes with flag, which will force driver to mark them as "uncacheable" (And same for fsync()-inducted writes). Again, not BIO_FLUSH, which should flush whole cache, but flag for BIO. I was told my mav@ (ahci driver author) that ATA has such capability. And I'm sure, that SCSI/SAS drives should have one too. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 08:15:28 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F2A7192C; Wed, 6 Mar 2013 08:15:27 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id BBF69932; Wed, 6 Mar 2013 08:15:27 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r268FIl5015220; Wed, 6 Mar 2013 00:15:22 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303060815.r268FIl5015220@gw.catspoiler.org> Date: Wed, 6 Mar 2013 00:15:18 -0800 (PST) From: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! To: lev@FreeBSD.org In-Reply-To: <1644513757.20130306113250@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-5 Content-Transfer-Encoding: 8BIT Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 08:15:28 -0000 On 6 Mar, Lev Serebryakov wrote: > Hello, Don. > You wrote 6 марта 2013 г., 11:15:13: > > DL> Did you have any power failures that took down the system sometime > DL> before this panic occured? By default FreeBSD enables write caching on > I had other panic due to my inaccurate hands... But I don't remeber > any power failures, as I havee UPS and this one works (I check it every > month). > > DL> That means that the drive will immediately acknowledge writes and is > DL> free to reorder them as it pleases. > > DL> When UFS+SU allocates a new inode, it first clears the available bit in > DL> the bitmap and writes the bitmap block to disk before it writes the new > DL> inode contents to disk. When a file is deleted, the inode is zeroed on > DL> disk before the available bit is set in the bitmap and the bitmap block > DL> is written. That means that if an inode is marked as available in the > DL> bitmap, then it should be zero. The system panic that you experienced > DL> happened when the system was attempting to allocate an inode for a new > DL> file and when it peeked at an inode that was marked as available, it > DL> found that the inode was non-zero. > > DL> What might have happened is that sometime in the past, the system was in >>[SKIPPED] > DL> tried to allocate the inode in question. > This scenario looks plausible, but it raises another question: does > barriers will protect against it? It doesn't look so, as now barrier > write is issued only when new inode BLOCK is allocated. And it leads > us to my other question: why did not mark such vital writes with > flag, which will force driver to mark them as "uncacheable" (And same > for fsync()-inducted writes). Again, not BIO_FLUSH, which should > flush whole cache, but flag for BIO. I was told my mav@ (ahci driver > author) that ATA has such capability. And I'm sure, that SCSI/SAS drives > should have one too. In the existing implementation, barriers wouldn't help since they aren't used in enough nearly enough places. UFS+SU currently expects the drive to tell it when the data actually hits the platter so that it can control the write ordering. In theory, barriers could be used instead, but performance would be terrible if they got turned into cache flushes. With NCQ or TCQ, the drive can have a sizeable number of writes internally queued and it is free to reorder them as it pleases even with write caching disabled, but if write caching is disabled it has to delay the notification of their completion until the data is on the platters so that UFS+SU can enforce the proper dependency ordering. Many years ago, when UFS+SU was fairly new, I experimented with enabling and disabling write caching on a SCSI drive with TCQ. Performance was about the same either way. I always disabled write caching on my SCSI drives after that because that is what UFS+SU expectes so that it can avoid inconsistencies in the case of power failure. I don't know enough about ATA to say if it supports marking individual writes as uncacheable. To support consistency on a drive with write caching enabled, UFS+SU would have to mark many of its writes as uncacheable. Even if this works, calls to fsync() would have to be turned into cache flushes to force the file data (assuming that it was was written with a cacheable write) to be written to the platters and only return to the userland program after the data is written. If drive write caching is off, then UFS+SU keeps track of the outstanding writes and an fsync() call won't return until the drive notifies UFS+SU that the data blocks for that file are actually written. In this case, the fsync() call doesn't need to get propagated down to the drive. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 08:23:32 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A038FAEA; Wed, 6 Mar 2013 08:23:32 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 2CA1F976; Wed, 6 Mar 2013 08:23:31 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r268NNor015235; Wed, 6 Mar 2013 00:23:27 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303060823.r268NNor015235@gw.catspoiler.org> Date: Wed, 6 Mar 2013 00:23:23 -0800 (PST) From: Don Lewis Subject: Re: Panic in ffs_valloc (Was: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS!) To: lev@FreeBSD.org In-Reply-To: <958644234.20130306105205@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-5 Content-Transfer-Encoding: 8BIT Cc: mckusick@mckusick.com, freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 08:23:32 -0000 On 6 Mar, Lev Serebryakov wrote: > Hello, Don. > You wrote 6 марта 2013 г., 10:43:11: > > DL> When growing a file, the data *must* be written before writing the block > DL> pointer that points to it. If this ordering isn't obeyed, then a system > DL> crash that occurs between the block pointer write and the data write > DL> would result in the file containing whatever garbage was in the data > DL> block. That garbage could be the confidential contents of some other > DL> user's previously deleted file. > It is why confidential data should be zeored-out before file deletion > :) Performance when deleting multi-gigabyte, low-value files would kind of suck if we did that ... From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 08:30:03 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5A431C33; Wed, 6 Mar 2013 08:30:03 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1DCC99B8; Wed, 6 Mar 2013 08:30:03 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:9421:367:9d7d:512b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 1DE504AC57; Wed, 6 Mar 2013 12:29:55 +0400 (MSK) Date: Wed, 6 Mar 2013 12:29:52 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <92952324.20130306122952@serebryakov.spb.ru> To: Don Lewis Subject: Re: Panic in ffs_valloc (Was: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS!) In-Reply-To: <201303060823.r268NNor015235@gw.catspoiler.org> References: <958644234.20130306105205@serebryakov.spb.ru> <201303060823.r268NNor015235@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-5 Content-Transfer-Encoding: quoted-printable Cc: mckusick@mckusick.com, freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 08:30:03 -0000 Hello, Don. You wrote 6 =DC=D0=E0=E2=D0 2013 =D3., 12:23:23: >> DL> When growing a file, the data *must* be written before writing the b= lock >> DL> pointer that points to it. If this ordering isn't obeyed, then a sy= stem >> DL> crash that occurs between the block pointer write and the data write >> DL> would result in the file containing whatever garbage was in the data >> DL> block. That garbage could be the confidential contents of some other >> DL> user's previously deleted file. >> It is why confidential data should be zeored-out before file deletion >> :) DL> Performance when deleting multi-gigabyte, low-value files would kind of DL> suck if we did that ... It should be application-level decision. And user-level, really :) Yes, I'm paranoid, and delete all sensitive data with special software, which does several passes of writing zeroes, ones and random garbage :) --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 08:38:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 31F9CDAD for ; Wed, 6 Mar 2013 08:38:22 +0000 (UTC) (envelope-from prvs=1777d4b2b8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B5806A01 for ; Wed, 6 Mar 2013 08:38:21 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002574306.msg for ; Wed, 06 Mar 2013 08:38:18 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 06 Mar 2013 08:38:18 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1777d4b2b8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> From: "Steven Hartland" To: "Larry Rosenman" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> Subject: Re: zfs send/recv invalid data Date: Wed, 6 Mar 2013 08:38:22 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 08:38:22 -0000 ----- Original Message ----- From: "Larry Rosenman" >>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs recv (to >>>>>>> 10.0) of a -R -I stream. >>>>>>> What data do I need to gather to figure out what side and what's wrong? >>>>>>> I've already started zpool scrubs on both sides. >>>>>>> I can insert a tee to grab the stream on either/both sides if that >>>>>>> would help. >>>>>>> >>>>>>> >>>>>>> >>>>>> Is the problem repeatable or is it just a network glitch? >>>>>> Ronald. >>>>> Repeatable....... >>>> >>>> Here is the exact error message: >>>> receiving incremental stream of vault/home/ctr@2013-03-05-test3 into >>>> zroot/backups/TBH/home/ctr@2013-03-05-test3 >>>> cannot receive incremental stream: invalid backup stream >>>> this is the script I'm running: >>>> #!/bin/sh >>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >>>> DATE2=`date -v "-1d" "+%Y-%m-%d"` >>>> # snap the source >>>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >>>> # zfs copy the source to here. >>>> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} vault@${DATE} >>>> | \ >>>> tee /tmp/backup.stream.send.${DATE} | \ >>>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} | zfs >>>> recv -u -v -d zroot/backups/TBH\"" >>>> # make sure we NEVER allow the backup stuff to automount. >>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >>>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >>> >>> Your send and receive sides differ, which indicates your ssh >>> shell my not be clean. >>> >>> Looking at the receive side its got what looks like a mail >>> message appended. >>> >>> I suspect if you manually copy the receive copy to the 10 machine and >>> the receive it will work fine. >> >> we're copying mail files........ >> >> and it still fails.... >> > I've put more example send/recv files in that directory. > > we're copying home dirs, which include lots of mail. > > (this one is my wife's) > > Ideas? > > I *CAN* give access to both sides via ssh..... The copy of the data stream on both sides should be identical though and its not, which leads me to believe something is corrupting the data on the way. Try the following:- >From source:- zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream scp test.stream home.lerctr.org:~/ >From target: zfs recv -u -v -d zroot/backups/TBH < test.stream If this works then there is something unclean about your ssh shell. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 08:41:44 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B7998FBD; Wed, 6 Mar 2013 08:41:44 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 42F9EA2D; Wed, 6 Mar 2013 08:41:44 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:9421:367:9d7d:512b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 2FD304AC57; Wed, 6 Mar 2013 12:41:42 +0400 (MSK) Date: Wed, 6 Mar 2013 12:41:39 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <1198028260.20130306124139@serebryakov.spb.ru> To: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! In-Reply-To: <201303060815.r268FIl5015220@gw.catspoiler.org> References: <1644513757.20130306113250@serebryakov.spb.ru> <201303060815.r268FIl5015220@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-5 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 08:41:44 -0000 Hello, Don. You wrote 6 =DC=D0=E0=E2=D0 2013 =D3., 12:15:18: >> This scenario looks plausible, but it raises another question: does >> barriers will protect against it? It doesn't look so, as now barrier >> write is issued only when new inode BLOCK is allocated. And it leads >> us to my other question: why did not mark such vital writes with >> flag, which will force driver to mark them as "uncacheable" (And same >> for fsync()-inducted writes). Again, not BIO_FLUSH, which should >> flush whole cache, but flag for BIO. I was told my mav@ (ahci driver >> author) that ATA has such capability. And I'm sure, that SCSI/SAS drives >> should have one too. DL> In the existing implementation, barriers wouldn't help since they aren't DL> used in enough nearly enough places. UFS+SU currently expects the drive DL> to tell it when the data actually hits the platter so that it can DL> control the write ordering. In theory, barriers could be used instead, DL> but performance would be terrible if they got turned into cache flushes. Yep! So, we need stream (file/vnode/inode)-related barriers or simple per-request (bp/bio) flag added. DL> With NCQ or TCQ, the drive can have a sizeable number of writes DL> internally queued and it is free to reorder them as it pleases even with DL> write caching disabled, but if write caching is disabled it has to delay DL> the notification of their completion until the data is on the platters DL> so that UFS+SU can enforce the proper dependency ordering. But, again, performance would be terrible :( I've checked it. On very sparse multi-threaded patterns (multiple torrents download on fast channel in my simple home case, and, I think, things could be worse in case of big file server in organization) and "simple" SATA drives it significant worse in my experience :( DL> I don't know enough about ATA to say if it supports marking individual DL> writes as uncacheable. To support consistency on a drive with write DL> caching enabled, UFS+SU would have to mark many of its writes as DL> uncacheable. Even if this works, calls to fsync() would have to be I don't see this as a big problem. I've done some experiments about one and half year ago by adding counter all overs UFS/FFS code when it writes metadata, and it was about 1% of writes on busy file system (torrents, csup update, buildworld, all on one big FS). DL> turned into cache flushes to force the file data (assuming that it was DL> was written with a cacheable write) to be written to the platters and DL> only return to the userland program after the data is written. If drive DL> write caching is off, then UFS+SU keeps track of the outstanding writes DL> and an fsync() call won't return until the drive notifies UFS+SU that DL> the data blocks for that file are actually written. In this case, the DL> fsync() call doesn't need to get propagated down to the drive. I see. But then we should turn off disc cache by default. And write some whitepaper about this situation. I don't know what is better for commodity SATA drives, really. And I'm not sure, that I understand UFS/FFS code good enough to do proper experiment by adding such flag to whole our storage stack :( And second problem: SSD. I know nothing about their caching strategies, and SSDs has very big RAM buffers compared to commodity HDDs (something like 512MiB vs 64MiB). --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 10:01:18 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8636FCC5; Wed, 6 Mar 2013 10:01:18 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 4E327E6C; Wed, 6 Mar 2013 10:01:17 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r26A18n3015414; Wed, 6 Mar 2013 02:01:12 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303061001.r26A18n3015414@gw.catspoiler.org> Date: Wed, 6 Mar 2013 02:01:08 -0800 (PST) From: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! To: lev@FreeBSD.org In-Reply-To: <1198028260.20130306124139@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 10:01:18 -0000 On 6 Mar, Lev Serebryakov wrote: > DL> With NCQ or TCQ, the drive can have a sizeable number of writes > DL> internally queued and it is free to reorder them as it pleases even with > DL> write caching disabled, but if write caching is disabled it has to delay > DL> the notification of their completion until the data is on the platters > DL> so that UFS+SU can enforce the proper dependency ordering. > But, again, performance would be terrible :( I've checked it. On > very sparse multi-threaded patterns (multiple torrents download on > fast channel in my simple home case, and, I think, things could be > worse in case of big file server in organization) and "simple" SATA > drives it significant worse in my experience :( I'm surprised that a typical drive would have enough onboard cache for write caching to help signficantly in that situation. Is the torrent software doing a lot of fsync() calls? Those would essentially turn into NOPs if write caching is enabled, but would stall the thread until the data hits the platter if write caching is disabled. One limitation of NCQ is that it only supports 32 simultaneous commands. With write caching enabled, you might be able to stuff more writes into the drive's onboard memory so that it can do a better job of optimizing the ordering and increase it's number of I/O's per second, though I wouldn't expect miracles. A SAS drive and controller with TCQ would support more simultaneous commands and might also perform better. Creating a file by writing it in random order is fairly expensive. Each time a new block is written by the application, UFS+SU has to first find a free block by searching the block bitmaps, mark that block as allocated, wait for that write of the bitmap block to complete, write the data to that block, wait for that to complete, and then write the block pointer to the inode or an indirect block. Because of the random write ordering, there is probably not enough locality to do coalesce multiple updates to the bitmap and indirect blocks into one write before the syncer interval expires. These operations all happen in the background after the write() call, but once you hit the I/O per second limit of the drive, eventually enough backlog builds to stall the application. Also, if another update needs to be done to a block that the syncer has queued for writing, that may also cause a stall until the write completes. If you hack the torrent software to create and pre-zero each file before it starts downloading it, then each bitmap and indirect block will probably only get written once during that operation and won't get written again during the actual download, and zeroing the data blocks will be sequential and fast. During the download, the only writes will be to the data blocks, so you might see something like a 3x performance improvement. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 10:46:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 42F79A4C for ; Wed, 6 Mar 2013 10:46:17 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id F1E24D6 for ; Wed, 6 Mar 2013 10:46:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=VD1TsKXq6iry1WuQv73sn2/3b0ZMVKagI35GYWLrZp4=; b=IpS5OK4iQAlZs6dcMrsfAzv9XTV6OBcMy/vwyS5PKKyIl1+zkIgdHTkpExBk7siewrWdOYoF4kq3KloUsBjuVCj1Lj091JXrU/1R0baCLs+YrtlZTMCBrVvzQ1vx7l45vn04OuvX3kgEfYSDQvXTrTuONbzh4b0a167pQ++cu6U=; Received: from localhost.lerctr.org ([127.0.0.1]:32334 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDBrP-000HLf-Lr; Wed, 06 Mar 2013 04:46:13 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 04:46:10 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 04:46:10 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 10:46:17 -0000 On 2013-03-06 02:38, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" >>>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs >>>>>>>> recv (to 10.0) of a -R -I stream. >>>>>>>> What data do I need to gather to figure out what side and >>>>>>>> what's wrong? >>>>>>>> I've already started zpool scrubs on both sides. >>>>>>>> I can insert a tee to grab the stream on either/both sides if >>>>>>>> that would help. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Is the problem repeatable or is it just a network glitch? >>>>>>> Ronald. >>>>>> Repeatable....... >>>>> Here is the exact error message: >>>>> receiving incremental stream of vault/home/ctr@2013-03-05-test3 >>>>> into zroot/backups/TBH/home/ctr@2013-03-05-test3 >>>>> cannot receive incremental stream: invalid backup stream >>>>> this is the script I'm running: >>>>> #!/bin/sh >>>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >>>>> DATE2=`date -v "-1d" "+%Y-%m-%d"` >>>>> # snap the source >>>>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >>>>> # zfs copy the source to here. >>>>> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} >>>>> vault@${DATE} | \ >>>>> tee /tmp/backup.stream.send.${DATE} | \ >>>>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} >>>>> | zfs recv -u -v -d zroot/backups/TBH\"" >>>>> # make sure we NEVER allow the backup stuff to automount. >>>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >>>>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >>>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >>>> Your send and receive sides differ, which indicates your ssh >>>> shell my not be clean. >>>> Looking at the receive side its got what looks like a mail >>>> message appended. >>>> I suspect if you manually copy the receive copy to the 10 machine >>>> and >>>> the receive it will work fine. >>> >>> we're copying mail files........ >>> >>> and it still fails.... >>> >> I've put more example send/recv files in that directory. >> we're copying home dirs, which include lots of mail. >> (this one is my wife's) >> Ideas? >> I *CAN* give access to both sides via ssh..... > > The copy of the data stream on both sides should be identical > though and its not, which leads me to believe something is > corrupting the data on the way. Try the following:- > >> From source:- > zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream > scp test.stream home.lerctr.org:~/ >> From target: > zfs recv -u -v -d zroot/backups/TBH < test.stream > > If this works then there is something unclean about your ssh > shell. > > Regards > Steve > send side: # zfs send -R -D -I vault@2013-03-05 vault@2013-03-06 >/tmp/send.stream # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # scp /tmp/send.stream home:/tmp/send.stream send.stream 100% 1180MB 2.5MB/s 07:44 # uname -a FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: Mon Mar 4 18:08:11 CST 2013 root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 # Receive side: # uname -a FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon Mar 4 19:59:08 CST 2013 root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream # So, you are correct that something(tm) is unclean about the ssh path. adding -current and -stable for diagnosing ssh issue. sshd config on the 8.3-STABLE box: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01 02:06:04Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # Disable legacy (protocol version 1) support in the server for new # installations. In future the default will change to require explicit # activation of protocol 1 Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys #AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # sshd config on the 10.0-CURRENT: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03 16:51:41Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 #Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # Ideas from the ssh folks? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 10:51:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7A524E00 for ; Wed, 6 Mar 2013 10:51:39 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vc0-f174.google.com (mail-vc0-f174.google.com [209.85.220.174]) by mx1.freebsd.org (Postfix) with ESMTP id 3A507132 for ; Wed, 6 Mar 2013 10:51:38 +0000 (UTC) Received: by mail-vc0-f174.google.com with SMTP id n11so4662421vch.19 for ; Wed, 06 Mar 2013 02:51:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=6wVU27BX+Frw0xd0Za/hk9eVbvLXK3MmvnsAxFjXEvQ=; b=GNWZv/BIOGCSTmedxVukd9c/HsKhINk7X4jPoJaOKUEDtA07sA6NNzdd8/eh0aSL9W 76VyX0jkNlq0qYwrb46qwkVCcfSKOEdPAp5FtZ/Etc1c+C14PU18ZOgjsS8MB3Jihmsh urjMFkyD+Od0LXeVMX3HgVCTydEV9LhYY7hTid+9AV76kixqCMOF01VtiZXSNkdF015K goxEijN6wabCBuUzx2HUzCOKEUJ8WR5TPs8JeKPye8FnYyRtpST8lAiWXzjqBJl11Fjr 6vbkixWispZdSpDEFdmwvoxtEtolxgyOV8bntszvSy+LB3wPCUoX/4qiALK4c/WcPNZn 6XcQ== MIME-Version: 1.0 X-Received: by 10.58.205.179 with SMTP id lh19mr11300827vec.7.1362567098249; Wed, 06 Mar 2013 02:51:38 -0800 (PST) Received: by 10.58.223.170 with HTTP; Wed, 6 Mar 2013 02:51:38 -0800 (PST) In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> Date: Wed, 6 Mar 2013 10:51:38 +0000 Message-ID: Subject: Re: zfs send/recv invalid data From: Tom Evans To: Larry Rosenman Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 10:51:39 -0000 On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman wrote: > So, you are correct that something(tm) is unclean about the ssh path. > And what is it that is added? Surely that is trivial to determine? echo hello > foo cat foo | ssh host cat Cheers Tom From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 10:56:37 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 47BB11ED for ; Wed, 6 Mar 2013 10:56:37 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 13BDE176 for ; Wed, 6 Mar 2013 10:56:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=bX47YHsqvdIwf4mSHqU2DswIw3M+W3RvvAx+gKKpl+k=; b=EBf2mwHvwTzSLAbSTmfiI7npxN+U/PSKuZf5+nPSu/MfFh4dACZ5OUSb04elIUBwL33IxhrBb6mLPGztlKrj4svY4FVURdFluRwYWJavPdliYS4Vjx534E8H3JGjKh7iJfIxCZNMkhE6/ATMDr+L3JK23Q7XgQyzPT1IJ9T6HLE=; Received: from localhost.lerctr.org ([127.0.0.1]:28254 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDC1R-000HVZ-UN; Wed, 06 Mar 2013 04:56:34 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 04:56:32 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 04:56:32 -0600 From: Larry Rosenman To: Tom Evans Subject: Re: zfs send/recv invalid data In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> Message-ID: <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 10:56:37 -0000 On 2013-03-06 04:51, Tom Evans wrote: > On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman > wrote: >> So, you are correct that something(tm) is unclean about the ssh path. >> > > And what is it that is added? Surely that is trivial to determine? > > echo hello > foo > cat foo | ssh host cat > > Cheers > > Tom it seems to be bytestream dependent, as this simple case works, and the vast majority of the sends work, but the particular case in the thread does NOT. # echo hello >foo # cat foo | ssh home cat hello # -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 11:07:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9E40C8A5 for ; Wed, 6 Mar 2013 11:07:32 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6C205208 for ; Wed, 6 Mar 2013 11:07:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=Zvwh+9YqwXiDTFfmDhtyH+ExUc6KMRmXmGHIHBa/wr8=; b=Meve4udwnvZpnB/hC2VKGaSB4PP6t496TiDOQRi7nA0svMVZURYUDOD3BSAac/AqtKa6YjNQVkF1si1PAmn+18Mt6MteVhtz4/PgfCnAAB8zXYBUfkXmqFv1sp+USfzmO/vbDoxHYRdmww3culy8+gKiYKHI/U55pPjCicPSU9s=; Received: from localhost.lerctr.org ([127.0.0.1]:29268 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDCC1-000Hcc-Ff; Wed, 06 Mar 2013 05:07:30 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 05:07:28 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 05:07:28 -0600 From: Larry Rosenman To: Tom Evans Subject: Re: zfs send/recv invalid data In-Reply-To: <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> Message-ID: <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 11:07:32 -0000 On 2013-03-06 04:56, Larry Rosenman wrote: > On 2013-03-06 04:51, Tom Evans wrote: >> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >> wrote: >>> So, you are correct that something(tm) is unclean about the ssh >>> path. >>> >> And what is it that is added? Surely that is trivial to determine? >> echo hello > foo >> cat foo | ssh host cat >> Cheers >> Tom > it seems to be bytestream dependent, as this simple case works, and > the vast majority of the sends work, but the particular > case in the thread does NOT. > > # echo hello >foo > # cat foo | ssh home cat > hello > # stranger: $ cd /tmp $ cat send.stream | ssh home openssl md5 (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 $ openssl md5 send.stream MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 $ so I'm not sure what is tripping it in the stream to a pipe case. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 11:46:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AFE174B4 for ; Wed, 6 Mar 2013 11:46:32 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4BCE3399 for ; Wed, 6 Mar 2013 11:46:31 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1UDCo7-00038q-A8 for freebsd-fs@freebsd.org; Wed, 06 Mar 2013 12:46:51 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 06 Mar 2013 12:46:51 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 06 Mar 2013 12:46:51 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Subject: ZFS deadlock (?) Date: Wed, 06 Mar 2013 12:46:11 +0100 Lines: 49 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig725D25E923B3C6A3C5A88E96" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:14.0) Gecko/20120812 Thunderbird/14.0 X-Enigmail-Version: 1.4.3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 11:46:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig725D25E923B3C6A3C5A88E96 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Maybe it's just me, but I really still have a hard time trusting ZFS. Here's what happened today in a machine with FreeNAS 8.3: 41526 root 1 45 0 17856K 1860K tx->tx 0 0:00 0.00% /sbin/zfs create -o compression=3Dlzjb -V 100g store/test 41641 root 1 44 0 17856K 1860K tx->tx 0 0:00 0.00% /sbin/zfs create -o compression=3Dlzjb -V 200g store/bla 41492 root 1 45 0 17856K 1860K tx->tx 2 0:00 0.00% /sbin/zfs create -o compression=3Dlzjb -V 100g store/test 41392 root 1 44 0 17856K 1860K tx->tx 0 0:00 0.00% /sbin/zfs create -o compression=3Dlzjb -V 100G store/test These processes were started from the web GUI and are apparently deadlocked or not proceeding because of some other reason. I tried creating the "test" volume first (the earliest PID), and when it didn't finish in a few minutes I cancelled it from the web GUI and started again. Apparently, all of these processes are in some kind of deadlock state. No IO is happening on the machine. An attempt to do a "zpool scrub store" resulted in zpool also being (dead)locked, in a different place: [ctrl-t] load: 0.00 cmd: zpool 41724 [scl->scl_cv)] 37.22r 0.00u 0.00s 0% 1696k Now, all IO on the file system is (dead) locked and cannot proceed. There is nothing significant in /var/log/messages. --------------enig725D25E923B3C6A3C5A88E96 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlE3LIsACgkQ/QjVBj3/HSw0RQCeOlWSrQs/hLfEWvz733H3+zZ2 Nd8AnAlMM8FNn7n+5OLlmkzbBB+mttvo =sozH -----END PGP SIGNATURE----- --------------enig725D25E923B3C6A3C5A88E96-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 11:48:10 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6B2E453D for ; Wed, 6 Mar 2013 11:48:10 +0000 (UTC) (envelope-from mattblists@icritical.com) Received: from mail2.icritical.com (mail2.icritical.com [212.57.248.50]) by mx1.freebsd.org (Postfix) with SMTP id C11B33D4 for ; Wed, 6 Mar 2013 11:48:09 +0000 (UTC) Received: (qmail 2103 invoked from network); 6 Mar 2013 11:41:47 -0000 Received: from localhost (127.0.0.1) by mail2.icritical.com with SMTP; 6 Mar 2013 11:41:47 -0000 Received: (qmail 2058 invoked by uid 599); 6 Mar 2013 11:41:46 -0000 Received: from unknown (HELO PDC002.icritical.int) (212.57.254.146) by mail2.icritical.com (qpsmtpd/0.28) with ESMTP; Wed, 06 Mar 2013 11:41:46 +0000 Message-ID: <51372B71.7030107@icritical.com> Date: Wed, 6 Mar 2013 11:41:37 +0000 From: Matt Burke User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130122 Thunderbird/17.0.2 MIME-Version: 1.0 To: Subject: No stats on BIO_FLUSH Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-TLS-Incoming: YES X-Virus-Scanned: by iCritical at mail2.icritical.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 11:48:10 -0000 Is there any reason BIO_FLUSH operations aren't recorded by devstats? The patch below causes flushes to be recorded under DEVSTAT_NO_DATA without any apparent ill effects. --- a/sys/geom/geom_disk.c +++ b/sys/geom/geom_disk.c @@ -252,7 +252,7 @@ g_disk_done(struct bio *bp) if (bp2->bio_error == 0) bp2->bio_error = bp->bio_error; bp2->bio_completed += bp->bio_completed; - if ((bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) != 0 && + if ((bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE|BIO_FLUSH)) != 0 && (sc = bp2->bio_to->geom->softc) != NULL && (dp = sc->dp) != NULL) { devstat_end_transaction_bio(dp->d_devstat, bp); @@ -403,6 +403,7 @@ g_disk_start(struct bio *bp) } bp2->bio_done = g_disk_done; bp2->bio_disk = dp; + devstat_start_transaction_bio(dp->d_devstat, bp2); g_disk_lock_giant(dp); dp->d_strategy(bp2); g_disk_unlock_giant(dp); -- Sorry for the following... iCritical is a brand of Critical Software Ltd. Registered in England & Wales: 04909220. Registered Office: IC2, Keele Science Park, Keele, Staffordshire, ST5 5NH. This message has been scanned for security threats by iCritical. The information contained in this message is confidential and intended for the addressee only. If you have received this message in error, or there are any problems with its content, please contact the sender. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 12:53:44 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7DA67461; Wed, 6 Mar 2013 12:53:44 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 08C7D918; Wed, 6 Mar 2013 12:53:43 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:9421:367:9d7d:512b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 12C844AC57; Wed, 6 Mar 2013 16:53:39 +0400 (MSK) Date: Wed, 6 Mar 2013 16:53:37 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <1402477662.20130306165337@serebryakov.spb.ru> To: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! In-Reply-To: <201303061001.r26A18n3015414@gw.catspoiler.org> References: <1198028260.20130306124139@serebryakov.spb.ru> <201303061001.r26A18n3015414@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 12:53:44 -0000 Hello, Don. You wrote 6 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2013 =D0=B3., 14:01:08: >> DL> With NCQ or TCQ, the drive can have a sizeable number of writes >> DL> internally queued and it is free to reorder them as it pleases even = with >> DL> write caching disabled, but if write caching is disabled it has to d= elay >> DL> the notification of their completion until the data is on the platte= rs >> DL> so that UFS+SU can enforce the proper dependency ordering. >> But, again, performance would be terrible :( I've checked it. On >> very sparse multi-threaded patterns (multiple torrents download on >> fast channel in my simple home case, and, I think, things could be >> worse in case of big file server in organization) and "simple" SATA >> drives it significant worse in my experience :( DL> I'm surprised that a typical drive would have enough onboard cache for DL> write caching to help signficantly in that situation. Is the torrent It is 5x64MiB in my case, oh, effectively, 4x64MiB :) Really, I could repeat experiment with some predictable and repeatable benchmark. What in out ports could be used for massively-parallel (16+ files) random (with blocks like 64KiB and file sizes like 2+GiB) but "repeatable" benchmark? DL> software doing a lot of fsync() calls? Those would essentially turn Nope. It trys to avoid fsync(), of course DL> Creating a file by writing it in random order is fairly expensive. Each DL> time a new block is written by the application, UFS+SU has to first find DL> a free block by searching the block bitmaps, mark that block as DL> allocated, wait for that write of the bitmap block to complete, write DL> the data to that block, wait for that to complete, and then write the DL> block pointer to the inode or an indirect block. Because of the random DL> write ordering, there is probably not enough locality to do coalesce DL> multiple updates to the bitmap and indirect blocks into one write before DL> the syncer interval expires. These operations all happen in the DL> background after the write() call, but once you hit the I/O per second DL> limit of the drive, eventually enough backlog builds to stall the DL> application. Also, if another update needs to be done to a block that DL> the syncer has queued for writing, that may also cause a stall until the DL> write completes. If you hack the torrent software to create and DL> pre-zero each file before it starts downloading it, then each bitmap and DL> indirect block will probably only get written once during that operation DL> and won't get written again during the actual download, and zeroing the DL> data blocks will be sequential and fast. During the download, the only DL> writes will be to the data blocks, so you might see something like a 3x DL> performance improvement. My client (transmission, from ports) is configured to do "real preallocation" (not sparse one), but it doesn't help much. It surely limited by disk I/O :( But anyway, torrent client is bad benchmark if we start to speak about some real experiments to decide what could be improved in FFS/GEOM stack, as it is not very repeatable. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 17:17:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1A8CBBA7 for ; Wed, 6 Mar 2013 17:17:11 +0000 (UTC) (envelope-from nwf@cs.jhu.edu) Received: from blaze.cs.jhu.edu (blaze.cs.jhu.edu [128.220.13.50]) by mx1.freebsd.org (Postfix) with ESMTP id E33E989C for ; Wed, 6 Mar 2013 17:17:10 +0000 (UTC) Received: from gradx.cs.jhu.edu (gradx.cs.jhu.edu [128.220.13.52]) by blaze.cs.jhu.edu (8.14.3/8.14.3) with ESMTP id r26HAI2d008374 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 6 Mar 2013 12:10:18 -0500 (EST) Received: from gradx.cs.jhu.edu (localhost [127.0.0.1]) by gradx.cs.jhu.edu (8.14.5/8.14.5) with ESMTP id r26HAIx5029835; Wed, 6 Mar 2013 12:10:18 -0500 Received: (from nwf@localhost) by gradx.cs.jhu.edu (8.14.5/8.14.5/Submit) id r26HAIf1029834; Wed, 6 Mar 2013 12:10:18 -0500 Date: Wed, 6 Mar 2013 12:10:18 -0500 From: Nathaniel W Filardo To: freebsd-fs@freebsd.org, rercola@acm.jhu.edu Subject: Cyclic permutations of "zpool replace" on raidz devices lead to corrupt data? Message-ID: <20130306171017.GP17094@gradx.cs.jhu.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="b5Ibld7S3Mj9Y6Fc" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 17:17:11 -0000 --b5Ibld7S3Mj9Y6Fc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Greetings freebsd-fs, I had a zpool that looked like this: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada1 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ada2a ONLINE 0 0 0 ada4a ONLINE 0 0 0 cache ada2d ONLINE 0 0 0 ada4d ONLINE 0 0 0 and, in a fit of OCD, I decided to attach a spare disk on ata6 and use it to reorder the disks so that they were ada{0,1,3,5}. I had thought this would be painless, by running (and waiting for each resilver to complete) zpool replace tank0 ada5 ada6 zpool replace tank0 ada1 ada5 zpool replace tank0 ada0 ada1 zpool replace tank0 ada6 ada0 Nothing funny, just a cyclic permutation. I realize now that I should have run a "zpool scrub" between each pass, but I didn't, so, oops. (The last of these commands has run to completion, but never removed the replacing-0 node in the vdev tree; the pool is currently resilvering itself again after the panic reported later in this mail.) In any case, while I do not have exact numbers to report, the following symptoms occurred during this chain of events. "zpool replace tank0 ada5 ada6" seemed to run without problem. "zpool replace tank0 ada1 ada5" discovered 170-something checksum errors on ada6. "zpool replace tank0 ada0 ada1" discovered 35-ish checksum errors on ada5. "zpool replace tank0 ada6 ada0" discovered 9 checksum errors on ada1 and reported 8 checksum errors for the raidz1 vdev, including the corruption a file in my freebsd svn mirror. I then removed the svn mirror, which seemed to go off without a hitch, and started to rebuild it. Much later, having decided to wait on rebuilding the mirror, when shuffling files off of its host filesystem to another (from tank0/mirrors/freebsd to tank0/mirrors/misc, in prepraration for deleting the former, though this has not been done), I was met with panic: trap: fast data access mmu miss (kernel) cpuid = 0 KDB: stack backtrace: panic() at panic+0x290 trap() at trap+0x554 -- fast data access mmu miss tar=0 %o7=0xc09b8df4 -- userland() at ddt_phys_decref user trace: trap %o7=0xc09b8df4 pc 0xc0948e00, sp 0xf3a38b21 done Uptime: 76d13h22m31s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... As a wild guess, this seems likely to be http://mail.opensolaris.org/pipermail/zfs-discuss/2012-February/050972.html in which a corrupt DDT yields a NULL pointer dereference when a DDT entry is not found. My suspicion (and it is just a guess at this point) is that somebody somewhere in the stack is holding on to the "old" zpool configuration across replace operations and issuing writes to the incorrect device(s). A bit about the machine, in case it matters: It's a Sun V240 running 9-CURRENT (git rev id 1b82c3b) with 16GB of RAM. All the devices in this pool are connected by mvs0, a "Marvell 88SX6081 SATA controller". There has been no prior indication of checksum errors on any of the devices, despite routine scrubbing every two weeks for as long as I can remember. The disks themselves are all WDC WD7500AADS-00L5B1; ada2 is an OCZ-VERTEX2 and ada4 is an OCZ-SOLID3. At no point during this (including across the panic reboot) did the disks ever lose power. A friend is helping me to test my hypothesis, but on Illumos (we do not have easy access to another FBSD machine with sufficient spare disks). We shall report our findings. Thoughts? Thanks in advance. --nwf; --b5Ibld7S3Mj9Y6Fc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) iEYEARECAAYFAlE3eHkACgkQTeQabvr9Tc8TLQCdFH5GtRwqF5J62AmZRqugcHvT J9wAni3HHkjuF7eO4f5/tkdZSFI0nNgn =fAzy -----END PGP SIGNATURE----- --b5Ibld7S3Mj9Y6Fc-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 18:59:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DD3A84D5 for ; Wed, 6 Mar 2013 18:59:11 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 829D1F16 for ; Wed, 6 Mar 2013 18:59:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=4LV5yZA6+67yrWEu01+3xy7aR22xgJoMZiz7ZWC6ffM=; b=fzHxLTmbojhTISdwhtb4+j01qNeFhpmouj4H+s48vknC5Da2GNdosbywKFrDu34o3/Qo+y3HGi1X/v2hwRgfJlQ+fZ1bK254FK44QiuhJK+MS2N2thuWMvdIbBjkpVpdbpBl/NUbxwZnJXmOh81mifpuq9K5eTL2h5oANxo168E=; Received: from localhost.lerctr.org ([127.0.0.1]:45090 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDJYN-000Lr7-Im; Wed, 06 Mar 2013 12:59:08 -0600 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 12:59:02 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 12:59:02 -0600 From: Larry Rosenman To: Martin Simmons Subject: Re: zfs send/recv invalid data In-Reply-To: <201303061857.r26IvLnc024186@higson.cam.lispworks.com> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 18:59:11 -0000 On 2013-03-06 12:57, Martin Simmons wrote: >>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: >> >> On 2013-03-06 04:56, Larry Rosenman wrote: >>> On 2013-03-06 04:51, Tom Evans wrote: >>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >>>> wrote: >>>>> So, you are correct that something(tm) is unclean about the ssh >>>>> path. >>>>> >>>> And what is it that is added? Surely that is trivial to determine? >>>> echo hello > foo >>>> cat foo | ssh host cat >>>> Cheers >>>> Tom >>> it seems to be bytestream dependent, as this simple case works, and >>> the vast majority of the sends work, but the particular >>> case in the thread does NOT. >>> >>> # echo hello >foo >>> # cat foo | ssh home cat >>> hello >>> # >> stranger: >> $ cd /tmp >> $ cat send.stream | ssh home openssl md5 >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> $ openssl md5 send.stream >> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> $ >> >> so I'm not sure what is tripping it in the stream to a pipe case. > > Is that running under the same uid? The script you posted earlier > runs the > pipe under ssh root@tbh.lerctr.org. > > __Martin Ooo, good point. Would csh vs. sh make a difference? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 19:08:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EB9C9779 for ; Wed, 6 Mar 2013 19:08:57 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [193.34.186.230]) by mx1.freebsd.org (Postfix) with ESMTP id 5E461F84 for ; Wed, 6 Mar 2013 19:08:55 +0000 (UTC) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.5/8.14.5) with ESMTP id r26IvOCW045068; Wed, 6 Mar 2013 18:57:24 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id r26IvLj1024190; Wed, 6 Mar 2013 18:57:21 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id r26IvLnc024186; Wed, 6 Mar 2013 18:57:21 GMT Date: Wed, 6 Mar 2013 18:57:21 GMT Message-Id: <201303061857.r26IvLnc024186@higson.cam.lispworks.com> From: Martin Simmons To: Larry Rosenman In-reply-to: <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> (message from Larry Rosenman on Wed, 06 Mar 2013 05:07:28 -0600) Subject: Re: zfs send/recv invalid data References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 19:08:58 -0000 >>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: > > On 2013-03-06 04:56, Larry Rosenman wrote: > > On 2013-03-06 04:51, Tom Evans wrote: > >> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman > >> wrote: > >>> So, you are correct that something(tm) is unclean about the ssh > >>> path. > >>> > >> And what is it that is added? Surely that is trivial to determine? > >> echo hello > foo > >> cat foo | ssh host cat > >> Cheers > >> Tom > > it seems to be bytestream dependent, as this simple case works, and > > the vast majority of the sends work, but the particular > > case in the thread does NOT. > > > > # echo hello >foo > > # cat foo | ssh home cat > > hello > > # > stranger: > $ cd /tmp > $ cat send.stream | ssh home openssl md5 > (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 > $ openssl md5 send.stream > MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 > $ > > so I'm not sure what is tripping it in the stream to a pipe case. Is that running under the same uid? The script you posted earlier runs the pipe under ssh root@tbh.lerctr.org. __Martin From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 19:15:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BEE519C2 for ; Wed, 6 Mar 2013 19:15:11 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6BDFDFD8 for ; Wed, 6 Mar 2013 19:15:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=A3M7c4fG/WrjrZHUSCYetbJgTU5RYlkgGIuJV4P/Kt8=; b=V+qGr68kRJlWyBJfb7kySjeinU44kGzGjQ859Zk8qJ4/3X77wirSMYUK7tfVZHRGhrWylqPEiMm/2OKq5D0HOrOYlxknFVfF5Er/uGMlqGHUOpZTiIa6yzHFUBimgv7Fp9fU0fMVM2CrvUWOZbmDEUlI3hFFOtVTbFIrh2d2iNk=; Received: from localhost.lerctr.org ([127.0.0.1]:63037 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDJnr-000M2d-Pf; Wed, 06 Mar 2013 13:15:08 -0600 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 13:15:02 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 13:15:02 -0600 From: Larry Rosenman To: Martin Simmons Subject: Re: zfs send/recv invalid data In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> Message-ID: <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 19:15:11 -0000 On 2013-03-06 12:59, Larry Rosenman wrote: > On 2013-03-06 12:57, Martin Simmons wrote: >>>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: >>> On 2013-03-06 04:56, Larry Rosenman wrote: >>>> On 2013-03-06 04:51, Tom Evans wrote: >>>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >>>>> wrote: >>>>>> So, you are correct that something(tm) is unclean about the ssh >>>>>> path. >>>>>> >>>>> And what is it that is added? Surely that is trivial to determine? >>>>> echo hello > foo >>>>> cat foo | ssh host cat >>>>> Cheers >>>>> Tom >>>> it seems to be bytestream dependent, as this simple case works, and >>>> the vast majority of the sends work, but the particular >>>> case in the thread does NOT. >>>> # echo hello >foo >>>> # cat foo | ssh home cat >>>> hello >>>> # >>> stranger: >>> $ cd /tmp >>> $ cat send.stream | ssh home openssl md5 >>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>> $ openssl md5 send.stream >>> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>> $ >>> so I'm not sure what is tripping it in the stream to a pipe case. >> Is that running under the same uid? The script you posted earlier >> runs the >> pipe under ssh root@tbh.lerctr.org. >> __Martin > Ooo, good point. Would csh vs. sh make a difference? Doesn't seem to make a difference.... # cat send.stream | ssh root@home openssl md5 (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 # uname -a FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: Mon Mar 4 18:08:11 CST 2013 root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 # # exec csh root@borg:/home/ler # pwd /home/ler root@borg:/home/ler # ssh tbh "cat /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org openssl md5" (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 root@borg:/home/ler # uname -a FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon Mar 4 19:59:08 CST 2013 root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 root@borg:/home/ler # -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 19:29:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BC401D27 for ; Wed, 6 Mar 2013 19:29:33 +0000 (UTC) (envelope-from mitnick.lyu@gmail.com) Received: from mail-qa0-f49.google.com (mail-qa0-f49.google.com [209.85.216.49]) by mx1.freebsd.org (Postfix) with ESMTP id 8118B135 for ; Wed, 6 Mar 2013 19:29:33 +0000 (UTC) Received: by mail-qa0-f49.google.com with SMTP id o13so2764464qaj.15 for ; Wed, 06 Mar 2013 11:29:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=cOyJhTd7WHTwQE7ixirEka5fqzbcKyHqyyD2ouT+jfY=; b=buYuSrJCJLWzbZMWWVY+ewp6I6wLSUBLyEOoH7T8hCIndBO+FasRxXTu6g+JC7FKr0 A76DeDXgbNr6+oxvQNn1X9FzxhzJu7XpbOPHZLZeRbufm+2x5J0IfT8eoCZMi4jZgL2p qPHOYtQ2NFDvm26JIg6bUB9lN5xENlnhMQThVHYAbx2bMkE+r/2ntCGUnz4qJhixk9yF jWA6nCLkf9PzWQ5kx82iIjoby5fCwFZr30tUn2t3EGSfDxr4JjmGnqKue7wI9UbnLWth bubHvi31HUbCizbY0ha04vR9q/fTCqYvDKw93ZKAVFEOse3eOz4/eK59i5z/qJ4SLeR1 f4VA== MIME-Version: 1.0 X-Received: by 10.224.117.66 with SMTP id p2mr3445537qaq.45.1362598167218; Wed, 06 Mar 2013 11:29:27 -0800 (PST) Received: by 10.49.86.4 with HTTP; Wed, 6 Mar 2013 11:29:27 -0800 (PST) Date: Thu, 7 Mar 2013 03:29:27 +0800 Message-ID: Subject: GSoC 2013 - Extend UFS2 with on-disk indexing From: Lyu Mitnick To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 19:29:33 -0000 Hello FreeBSDer, I am Mitnick, a student who want to apply for GSoC 2013 and contribute to FreeBSD. I have take a look of the FreeBSD idea page. A project "Extend UFS2 with on-disk indexing" attracted my attention. As far as I know is that the project is an extension of "Dynamic memory allocation for dirhash in UFS2". I am wondering to know the project status and the potential mentor of the project. I have experience in C and have traced the source code of a file system. However I don't have any programming experience with UFS2. Any suggestion on what steps I should take to start is welcome!! Thanks very much. Mitnick From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 19:36:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3DF30EDD for ; Wed, 6 Mar 2013 19:36:13 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews03.kpnxchange.com (cpsmtpb-ews03.kpnxchange.com [213.75.39.6]) by mx1.freebsd.org (Postfix) with ESMTP id 75F0719F for ; Wed, 6 Mar 2013 19:36:12 +0000 (UTC) Received: from cpsps-ews07.kpnxchange.com ([10.94.84.174]) by cpsmtpb-ews03.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 6 Mar 2013 20:33:37 +0100 Received: from CPSMTPM-TLF103.kpnxchange.com ([195.121.3.6]) by cpsps-ews07.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 6 Mar 2013 20:33:37 +0100 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF103.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 6 Mar 2013 20:35:03 +0100 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 62FF8678A; Wed, 6 Mar 2013 20:35:03 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Martin Simmons" , "Larry Rosenman" Subject: Re: zfs send/recv invalid data References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> Date: Wed, 06 Mar 2013 20:35:02 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> User-Agent: Opera Mail/12.14 (FreeBSD) X-OriginalArrivalTime: 06 Mar 2013 19:35:03.0909 (UTC) FILETIME=[B336C150:01CE1AA1] X-RcptDomain: freebsd.org Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 19:36:13 -0000 On Wed, 06 Mar 2013 20:15:02 +0100, Larry Rosenman wrote: > On 2013-03-06 12:59, Larry Rosenman wrote: >> On 2013-03-06 12:57, Martin Simmons wrote: >>>>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: >>>> On 2013-03-06 04:56, Larry Rosenman wrote: >>>>> On 2013-03-06 04:51, Tom Evans wrote: >>>>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >>>>>> wrote: >>>>>>> So, you are correct that something(tm) is unclean about the ssh >>>>>>> path. >>>>>>> >>>>>> And what is it that is added? Surely that is trivial to determine? >>>>>> echo hello > foo >>>>>> cat foo | ssh host cat >>>>>> Cheers >>>>>> Tom >>>>> it seems to be bytestream dependent, as this simple case works, and >>>>> the vast majority of the sends work, but the particular >>>>> case in the thread does NOT. >>>>> # echo hello >foo >>>>> # cat foo | ssh home cat >>>>> hello >>>>> # >>>> stranger: >>>> $ cd /tmp >>>> $ cat send.stream | ssh home openssl md5 >>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>> $ openssl md5 send.stream >>>> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>> $ >>>> so I'm not sure what is tripping it in the stream to a pipe case. >>> Is that running under the same uid? The script you posted earlier >>> runs the >>> pipe under ssh root@tbh.lerctr.org. >>> __Martin >> Ooo, good point. Would csh vs. sh make a difference? > Doesn't seem to make a difference.... > > # cat send.stream | ssh root@home openssl md5 > (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 > # uname -a > FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 > r247820: Mon Mar 4 18:08:11 CST 2013 > root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 > # > # exec csh > root@borg:/home/ler # pwd > /home/ler > root@borg:/home/ler # ssh tbh "cat > /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org > openssl md5" > (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 > root@borg:/home/ler # uname -a > FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: > Mon Mar 4 19:59:08 CST 2013 > root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 > root@borg:/home/ler # > Hi, My script to send/rcv locally has a commented mbuffer in it because I had trouble with it in the past. zfs send $SEND_ARGS $VOL_FROM@$PREFIX$NEW \ | zfs receive -F $VOL_TO # | mbuffer -q | zfs receive -F $VOL_TO Maybe zfs receive expects a certain size of blocks of data and |ssh (or in my case |mbuffer) changes that. This is all a wild guess. More debugging should tell the difference, but it might give a direction to look for. The error "invalid backup stream" can be given at a couple of places. Can you use a debugger or ktrace or something else to get more information about what is happening? $ 20:32:24 ronald@sjakie [/usr/src/cddl/contrib/opensolaris/lib/libzfs/common] grep EZFS_BADSTREAM * libzfs.h: EZFS_BADSTREAM, /* bad backup stream */ libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, dgettext(TEXT_DOMAIN, libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, errbuf); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, errbuf)); libzfs_util.c: case EZFS_BADSTREAM: Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 22:20:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6E99231A for ; Wed, 6 Mar 2013 22:20:36 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 26EB9C1A for ; Wed, 6 Mar 2013 22:20:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:References:Message-ID:In-Reply-To:Subject:cc:To:From:Date; bh=j9Iqh8w1B26ImaL/fpyK227Vcr0JdCCEZSsTj3AD0CM=; b=OZtEeCHictLvRkU4/+cTQMVUNgrUXV7ff8J5F/hMGdvpDoAiFT9SWjsibpmnIW+7np+T4hF6XTnX0ouoeM9rt4P1xkCWdjWrTD2Sy/58WTNmEl/+SpfXx35LznCd6mRdEcm677JYwPKI7XUqaaQ/53cxV3WFqS0MFGBD8gPDH80=; Received: from lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net ([2001:470:1f0e:3ad::2]:39595) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDMhI-000O16-26; Wed, 06 Mar 2013 16:20:29 -0600 Date: Wed, 6 Mar 2013 16:20:24 -0600 (CST) From: Larry Rosenman To: Ronald Klop Subject: Re: zfs send/recv invalid data In-Reply-To: Message-ID: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 22:20:36 -0000 On Wed, 6 Mar 2013, Ronald Klop wrote: > On Wed, 06 Mar 2013 20:15:02 +0100, Larry Rosenman wrote: > >> On 2013-03-06 12:59, Larry Rosenman wrote: >>> On 2013-03-06 12:57, Martin Simmons wrote: >>>>>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: >>>>> On 2013-03-06 04:56, Larry Rosenman wrote: >>>>>> On 2013-03-06 04:51, Tom Evans wrote: >>>>>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >>>>>>> wrote: >>>>>>>> So, you are correct that something(tm) is unclean about the ssh >>>>>>>> path. >>>>>>>> >>>>>>> And what is it that is added? Surely that is trivial to determine? >>>>>>> echo hello > foo >>>>>>> cat foo | ssh host cat >>>>>>> Cheers >>>>>>> Tom >>>>>> it seems to be bytestream dependent, as this simple case works, and >>>>>> the vast majority of the sends work, but the particular >>>>>> case in the thread does NOT. >>>>>> # echo hello >foo >>>>>> # cat foo | ssh home cat >>>>>> hello >>>>>> # >>>>> stranger: >>>>> $ cd /tmp >>>>> $ cat send.stream | ssh home openssl md5 >>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> $ openssl md5 send.stream >>>>> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> $ >>>>> so I'm not sure what is tripping it in the stream to a pipe case. >>>> Is that running under the same uid? The script you posted earlier runs >>>> the >>>> pipe under ssh root@tbh.lerctr.org. >>>> __Martin >>> Ooo, good point. Would csh vs. sh make a difference? >> Doesn't seem to make a difference.... >> >> # cat send.stream | ssh root@home openssl md5 >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> # uname -a >> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: >> Mon Mar 4 18:08:11 CST 2013 >> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 >> # >> # exec csh >> root@borg:/home/ler # pwd >> /home/ler >> root@borg:/home/ler # ssh tbh "cat >> /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org >> openssl md5" >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> root@borg:/home/ler # uname -a >> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon >> Mar 4 19:59:08 CST 2013 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >> root@borg:/home/ler # >> > > > Hi, > > My script to send/rcv locally has a commented mbuffer in it because I had > trouble with it in the past. > > zfs send $SEND_ARGS $VOL_FROM@$PREFIX$NEW \ > | zfs receive -F $VOL_TO > # | mbuffer -q | zfs receive -F $VOL_TO > > > Maybe zfs receive expects a certain size of blocks of data and |ssh (or in my > case |mbuffer) changes that. This is all a wild guess. More debugging should > tell the difference, but it might give a direction to look for. > > > The error "invalid backup stream" can be given at a couple of places. Can you > use a debugger or ktrace or something else to get more information about what > is happening? > > $ 20:32:24 ronald@sjakie > [/usr/src/cddl/contrib/opensolaris/lib/libzfs/common] > grep EZFS_BADSTREAM * > libzfs.h: EZFS_BADSTREAM, /* bad backup stream */ > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > dgettext(TEXT_DOMAIN, > libzfs_sendrecv.c: error = zfs_error(hdl, > EZFS_BADSTREAM, errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_util.c: case EZFS_BADSTREAM: Hrm. Anyone have an idea on what to do with ktrace or dtrace or gdb or whatever to debug this? I'm at a loss on where to start on it :( -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 23:00:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EE9AA6B7 for ; Wed, 6 Mar 2013 23:00:08 +0000 (UTC) (envelope-from prvs=1777d4b2b8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 7FDBEE9D for ; Wed, 6 Mar 2013 23:00:08 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002587408.msg for ; Wed, 06 Mar 2013 23:00:06 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 06 Mar 2013 23:00:06 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1777d4b2b8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> From: "Steven Hartland" To: "Larry Rosenman" , "Martin Simmons" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> Subject: Re: zfs send/recv invalid data Date: Wed, 6 Mar 2013 23:00:10 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 23:00:09 -0000 ----- Original Message ----- From: "Larry Rosenman" > # cat send.stream | ssh root@home openssl md5 > (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 > # uname -a > FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 > r247820: Mon Mar 4 18:08:11 CST 2013 > root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 > # > # exec csh > root@borg:/home/ler # pwd > /home/ler > root@borg:/home/ler # ssh tbh "cat > /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org > openssl md5" > (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 > root@borg:/home/ler # uname -a > FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: > Mon Mar 4 19:59:08 CST 2013 > root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 > root@borg:/home/ler # When you run a command from ssh it will run as the target users shell so for csh if there is something bad in .cshrc or iirc .profile then you may get something output purely by logging in. So ensure you shell config's for the user on the target machine are clean. You can confirm this by comparing the output from the following two commands run from the sending machine:- md5 send.stream cat send.stream | ssh root@home.lerctr.org "cat" |md5 If these two commands produce different md5's then the login shell for root on home.lerctr.org is outputing something even when there is no terminal. You should be able to see what the shell is outputting using:- echo -n '' | ssh root@home.lerctr.org "cat" > 1.log After running this 1.log should be 0 bytes if its not your shell is definitely your problem and I'd start by looking at .cshrc and .profile for this user, or even the globals for these in /etc/ Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 23:16:24 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 25EFE94C for ; Wed, 6 Mar 2013 23:16:24 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id A8386F58 for ; Wed, 6 Mar 2013 23:16:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=HSKv2FLcgA6iPKpSQbJ3i1IyMTfpubXUYIQ8XXxlsQc=; b=Ss03tf0Vh0ESppbcxI4TCp0G2k05CraxRaGLRRsYWdqIULv7bMR6AMKya2/eB6Q4kxwrl785ofwRYxeHpN4YgWk2LMHUGVHw8gPNzvWuH9kxyUWlQbL5cve5uZWm1y7UqYqjNLgrodn6wroVl1fneJnY9T6bpVJSaH1pTIznBJI=; Received: from localhost.lerctr.org ([127.0.0.1]:21860 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDNZH-000OVR-K7; Wed, 06 Mar 2013 17:16:16 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 17:16:14 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 17:16:14 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 23:16:24 -0000 On 2013-03-06 17:00, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" >> # cat send.stream | ssh root@home openssl md5 >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> # uname -a >> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 >> r247820: Mon Mar 4 18:08:11 CST 2013 >> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 >> # >> # exec csh >> root@borg:/home/ler # pwd >> /home/ler >> root@borg:/home/ler # ssh tbh "cat >> /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org >> openssl md5" >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> root@borg:/home/ler # uname -a >> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 >> r247826: Mon Mar 4 19:59:08 CST 2013 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >> root@borg:/home/ler # > > When you run a command from ssh it will run as the target users shell > so for csh if there is something bad in .cshrc or iirc .profile then > you may get something output purely by logging in. > > So ensure you shell config's for the user on the target machine are > clean. > > You can confirm this by comparing the output from the following > two commands run from the sending machine:- > md5 send.stream > cat send.stream | ssh root@home.lerctr.org "cat" |md5 > > If these two commands produce different md5's then the login > shell for root on home.lerctr.org is outputing something > even when there is no terminal. > > You should be able to see what the shell is outputting using:- > echo -n '' | ssh root@home.lerctr.org "cat" > 1.log > > After running this 1.log should be 0 bytes if its not your > shell is definitely your problem and I'd start by looking > at .cshrc and .profile for this user, or even the globals > for these in /etc/ > I've done that, and get identical md5's..... I'm at a loss. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 23:48:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 04EBA6FA for ; Wed, 6 Mar 2013 23:48:38 +0000 (UTC) (envelope-from prvs=1777d4b2b8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 89BF0234 for ; Wed, 6 Mar 2013 23:48:36 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002588037.msg for ; Wed, 06 Mar 2013 23:48:36 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 06 Mar 2013 23:48:36 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1777d4b2b8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> From: "Steven Hartland" To: "Larry Rosenman" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> Subject: Re: zfs send/recv invalid data Date: Wed, 6 Mar 2013 23:48:38 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 23:48:38 -0000 ----- Original Message ----- From: "Larry Rosenman" To: "Steven Hartland" Cc: "Martin Simmons" ; ; ; Sent: Wednesday, March 06, 2013 11:16 PM Subject: Re: zfs send/recv invalid data > On 2013-03-06 17:00, Steven Hartland wrote: >> ----- Original Message ----- From: "Larry Rosenman" >>> # cat send.stream | ssh root@home openssl md5 >>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>> # uname -a >>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 >>> r247820: Mon Mar 4 18:08:11 CST 2013 >>> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 >>> # >>> # exec csh >>> root@borg:/home/ler # pwd >>> /home/ler >>> root@borg:/home/ler # ssh tbh "cat >>> /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org >>> openssl md5" >>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>> root@borg:/home/ler # uname -a >>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 >>> r247826: Mon Mar 4 19:59:08 CST 2013 >>> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >>> root@borg:/home/ler # >> >> When you run a command from ssh it will run as the target users shell >> so for csh if there is something bad in .cshrc or iirc .profile then >> you may get something output purely by logging in. >> >> So ensure you shell config's for the user on the target machine are >> clean. >> >> You can confirm this by comparing the output from the following >> two commands run from the sending machine:- >> md5 send.stream >> cat send.stream | ssh root@home.lerctr.org "cat" |md5 >> >> If these two commands produce different md5's then the login >> shell for root on home.lerctr.org is outputing something >> even when there is no terminal. >> >> You should be able to see what the shell is outputting using:- >> echo -n '' | ssh root@home.lerctr.org "cat" > 1.log >> >> After running this 1.log should be 0 bytes if its not your >> shell is definitely your problem and I'd start by looking >> at .cshrc and .profile for this user, or even the globals >> for these in /etc/ >> > I've done that, and get identical md5's..... > > I'm at a loss. I believe what you ran was similar but not identical, if you check the quoting subtly different. I'm not sure that will have any effect but both of the above are worth running just to confirm. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 00:05:40 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1726CE95; Thu, 7 Mar 2013 00:05:40 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 6BC7B2F0; Thu, 7 Mar 2013 00:05:39 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id r2705Xjb038999; Wed, 6 Mar 2013 17:05:33 -0700 (MST) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id r2705XOL038998; Wed, 6 Mar 2013 17:05:33 -0700 (MST) (envelope-from ken) Date: Wed, 6 Mar 2013 17:05:33 -0700 From: "Kenneth D. Merry" To: fs@freebsd.org, arch@freebsd.org Subject: patches to add new stat(2) file flags Message-ID: <20130307000533.GA38950@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="u3/rZRmxL6MmkK24" Content-Disposition: inline User-Agent: Mutt/1.4.2i X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 00:05:40 -0000 --u3/rZRmxL6MmkK24 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi folks, I have attached diffs against head for some additional stat(2) file flags. The primary purpose of these flags is to improve compatibility with CIFS, both from the client and the server side. There are more attributes supported by CIFS than the attached patch adds, but these are the attributes that ZFS supports natively. (So no changes in the on-disk format for ZFS at least.) These are also the flags that the Solaris CIFS server supports. I also implemented the flags in a way that is compatible with the way that they are implemented in the MacOS X sys/stat.h and the MacOS X version of our smbfs code. So you can, for instance, set the hidden flag with our version of smbfs just like you can in MacOS X: # ls -lao total 32 drwxr-xr-x 1 root wheel - 16384 Mar 6 15:53 . drwxr-xr-x 1 root wheel - 16384 Dec 31 1969 .. -rwxr-xr-x 1 root wheel - 0 Mar 6 15:53 foo # chflags hidden foo # ls -lao total 32 drwxr-xr-x 1 root wheel - 16384 Mar 6 15:53 . drwxr-xr-x 1 root wheel - 16384 Dec 31 1969 .. -rwxr-xr-x 1 root wheel hidden 0 Mar 6 15:53 foo But you can also set the system flag, which MacOS X doesn't yet support: # chflags system foo # ls -lao total 32 drwxr-xr-x 1 root wheel - 16384 Mar 6 15:53 . drwxr-xr-x 1 root wheel - 16384 Dec 31 1969 .. -rwxr-xr-x 1 root wheel system,hidden 0 Mar 6 15:53 foo There are other Windows/CIFS file attributes that would be good to implement, most notably the NOT_CONTENT_INDEXED flag, but I limited the list for now to avoid more invasive changes to ZFS in particular. (Since we would want to at least attempt to synchronize with Illumos on that.) My local commit message is below, it gives more details. Any objections to committing this? Change 659250 by kenm@ken.spectrabsd7 on 2013/02/27 15:02:04 Expand the use of stat(2) flags to allow storing some Windows/DOS and CIFS file attributes as BSD stat(2) flags. This work is intended to be compatible with ZFS, the Solaris CIFS server's interaction with ZFS, with MacOS X, and of course with Windows. The Windows attributes that are implemented were chosen based on the attributes that ZFS already supports. The flag values and names were chosen to be compatible with some existing flags in MacOS X. There are several new flags, and two existing flags (UF_IMMUTABLE and SF_ARCHIVED) has been implemented in a couple of new filesystems. Although it is not included in this change, __FreeBSD_version in sys/param.h should be bumped when this is committed to FreeBSD proper, since it is a system API change. The summary of the flags is as follows: UF_SYSTEM: Command line name: "system" or "usystem" ZFS name: XAT_SYSTEM, ZFS_SYSTEM Windows: FILE_ATTRIBUTE_SYSTEM This flag means that the file is used by the operating system. FreeBSD does not enforce any special handling when this flag is set. UF_SPARSE: Command line name: "sparse" or "usparse" ZFS name: XAT_SPARSE, ZFS_SPARSE Windows: FILE_ATTRIBUTE_SPARSE_FILE This flag means that the file is sparse. Although ZFS may modify this in some situations, there is not generally any special handling for this flag. UF_OFFLINE: Command line name: "offline" or "uoffline" ZFS name: XAT_OFFLINE, ZFS_OFFLINE Windows: FILE_ATTRIBUTE_OFFLINE This flag means that the file has been moved to offline storage. FreeBSD does not have any special handling for this flag. UF_REPARSE: Command line name: "reparse" or "ureparse" ZFS name: XAT_REPARSE, ZFS_REPARSE Windows: FILE_ATTRIBUTE_REPARSE_POINT This flag means that the file is a Windows reparse point. ZFS has special handling code for reparse points, but we don't currently have the other supporting infrastructure for them. UF_HIDDEN: Command line name: "hidden" or "uhidden" ZFS name: XAT_HIDDEN, ZFS_HIDDEN Windows: FILE_ATTRIBUTE_HIDDEN This flag means that the file may be excluded from a directory listing if the application honors it. FreeBSD has no special handling for this flag. The name and bit definition for UF_HIDDEN are identical to the definition in MacOS X. UF_IMMUTABLE: Command line name: "uchg", "uimmutable" ZFS name: XAT_READONLY, ZFS_READONLY Windows: FILE_ATTRIBUTE_READONLY This flag means that the file may not be modified. This is not a new flag, but where applicable it is mapped to the Windows readonly bit. ZFS and UFS now both support the flag and enforce it. The behavior of this flag is compatible with MacOS X. SF_ARCHIVED: Command line name: "arch", "archived" ZFS_NAME: XAT_ARCHIVE, ZFS_ARCHIVE Windows name: FILE_ATTRIBUTE_ARCHIVE The SF_ARCHIVED flag means that the file has been archived. The meaning is the inverse of the Windows FILE_ATTRIBUTE_ARCHIVE, and the ZFS XAT_ARCHIVE and ZFS_ARCHIVE attribute. Accordingly, we invert the internal flags in ZFS and SMBFS to get the value of SF_ARCHIVED. chflags.1: Document the new command line flag names (e.g. "system", "hidden") available to the user. strtofflags.c: Implement the mapping between the new command line flag names and new stat(2) flags. chflags.2: Document all of the new stat(2) flags, and explain the intended behavior in a little more detail. Explain how they map to Windows file attributes. Different filesystems behave differently with respect to flags, so warn the application developer to take care when using them. zfs_vnops.c: Add support for getting and setting the UF_IMMUTABLE, UF_SYSTEM, UF_HIDDEN, UF_REPARSE, UF_OFFLINE, UF_SPARSE and SF_ARCHIVED flags. All of these flags are implemented using attributes that ZFS already supports, so the on-disk format has not changed. ZFS currently doesn't allow setting the UF_REPARSE flag, and we don't really have the other infrastructure to support reparse points. As mentioned above, the ZFS semantics for the ARCHIVE flag are the same as in Windows, but are the inverse of the semantics for SF_ARCHIVED. Thus we clear SF_ARCHIVED when ZFS sets its archive attribute and vice versa. msdosfs_vnops.c: Add support for getting and setting UF_HIDDEN and UF_SYSTEM in MSDOSFS. It already supports SF_ARCHIVED. It currently supports the readonly bit with standard Unix permissions, but could be extended to support UF_IMMUTABLE. smbfs_node.c, smbfs_vnops.c: Add support for UF_HIDDEN, UF_SYSTEM, UF_IMMUTABLE and SF_ARCHIVED in SMBFS. This is similar to changes that Apple has made in their version of SMBFS (as of smb-583.8, posted on opensource.apple.com). The semantics should be the same as what Apple has done, but the implementation is different because they have made significant changes to the codebase. stat.h: Add definitions for UF_SYSTEM, UF_SPARSE, UF_OFFLINE, UF_REPARSE, and UF_HIDDEN. The definition of UF_HIDDEN is the same as the MacOS X definition. Add commented-out definitions of UF_COMPRESSED and UF_TRACKED. They are defined in MacOS X (as of 10.8.2), but we do not implement them (yet). ufs_vnops.c: Add support for getting and setting UF_HIDDEN, UF_SYSTEM, UF_SPARSE, UF_OFFLINE, and UF_REPARSE in UFS. These new flags are only stored, UFS does not take any action if the flag is set. Ken -- Kenneth Merry ken@FreeBSD.ORG --u3/rZRmxL6MmkK24 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="file_flags_head.20130306.1.txt" --- //depot/users/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.86 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-06 14:47:25.987128763 -0700 @@ -117,6 +117,16 @@ set the user immutable flag (owner or super-user only) .It Cm uunlnk , uunlink set the user undeletable flag (owner or super-user only) +.It Cm system , usystem +set the Windows system flag (owner or super-user only) +.It Cm sparse , usparse +set the sparse file attribute (owner or super-user only) +.It Cm offline , uoffline +set the offline file attribute (owner or super-user only) +.It Cm reparse , ureparse +set the Windows reparse point file attribute (owner or super-user only) +.It Cm hidden , uhidden +set the hidden file attribute (owner or super-user only) .El .Pp Putting the letters --- //depot/users/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.178 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-06 15:09:32.842437917 -0700 @@ -68,7 +68,17 @@ { "nodump", 1, UF_NODUMP }, { "noopaque", 0, UF_OPAQUE }, { "nouunlnk", 0, UF_NOUNLINK }, - { "nouunlink", 0, UF_NOUNLINK } + { "nouunlink", 0, UF_NOUNLINK }, + { "nosystem", 0, UF_SYSTEM, }, + { "nousystem", 0, UF_SYSTEM, }, + { "nosparse", 0, UF_SPARSE, }, + { "nousparse", 0, UF_SPARSE, }, + { "nooffline", 0, UF_OFFLINE, }, + { "nouoffline", 0, UF_OFFLINE, }, + { "noreparse", 0, UF_REPARSE, }, + { "noureparse", 0, UF_REPARSE, }, + { "nohidden", 0, UF_HIDDEN, }, + { "nouhidden", 0, UF_HIDDEN, } }; #define nmappings (sizeof(mapping) / sizeof(mapping[0])) --- //depot/users/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.257 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-06 14:49:39.357125717 -0700 @@ -75,16 +75,49 @@ Do not dump the file. .It Dv UF_IMMUTABLE The file may not be changed. +Filesystems may use this flag to maintain compatibility with the Windows and +CIFS FILE_ATTRIBUTE_READONLY attribute. .It Dv UF_APPEND The file may only be appended to. .It Dv UF_NOUNLINK The file may not be renamed or deleted. .It Dv UF_OPAQUE The directory is opaque when viewed through a union stack. +.It Dv UF_SYSTEM +The file has the Windows and CIFS FILE_ATTRIBUTE_SYSTEM attribute. +Filesystems in FreeBSD may store and display this flag, but do not provide +any special handling when it is set. +.It Dv UF_SPARSE +The file has the Windows FILE_ATTRIBUTE_SPARSE_FILE attribute. +This may also be used by a filesystem to indicate a sparse file. +.It Dv UF_OFFLINE +The file is offline, or has the Windows and CIFS FILE_ATTRIBUTE_OFFLINE +attribute. +Filesystems in FreeBSD store and display this flag, but do not provide any +special handling when it is set. +.It Dv UF_REPARSE +The file contains a Windows reparse point and has the Windows and CIFS +FILE_ATTRIBUTE_REPARSE_POINT attribute. +.It Dv UF_HIDDEN +The file may be hidden from directory listings at the application's +discretion. +The file has the Windows FILE_ATTRIBUTE_HIDDEN attribute. .It Dv SF_ARCHIVED -The file may be archived. +The file has been archived. +This flag means the opposite of the Windows and CIFS FILE_ATTRIBUTE_ARCHIVE +attribute. +That attribute means that the file should be archived, whereas +.Dv SF_ARCHIVED +means that the file has been archived. +Filesystems in FreeBSD may or may not have special handling for this flag. +For instance, ZFS tracks changes to files and will clear this bit when a +file is updated. +UFS only stores the flag, and relies on the application to change it when +needed. .It Dv SF_IMMUTABLE The file may not be changed. +This flag also indicates that the Windows ans CIFS FILE_ATTRIBUTE_READONLY +attribute is set. .It Dv SF_APPEND The file may only be appended to. .It Dv SF_NOUNLINK @@ -121,6 +154,13 @@ .Xr init 8 for details.) .Pp +The implementation of all flags is filesystem-dependent. +See the description of the +.Dv SF_ARCHIVED +flag above for one example of the differences in behavior. +Care should be exercised when writing applications to account for +support or lack of support of these flags in various filesystems. +.Pp The .Dv SF_SNAPSHOT flag is maintained by the system and cannot be toggled. --- //depot/users/kenm/FreeBSD-test3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.333 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c 2013-03-06 14:49:41.836130019 -0700 @@ -6057,23 +6057,49 @@ XVA_SET_REQ(&xvap, XAT_APPENDONLY); XVA_SET_REQ(&xvap, XAT_NOUNLINK); XVA_SET_REQ(&xvap, XAT_NODUMP); + XVA_SET_REQ(&xvap, XAT_READONLY); + XVA_SET_REQ(&xvap, XAT_ARCHIVE); + XVA_SET_REQ(&xvap, XAT_SYSTEM); + XVA_SET_REQ(&xvap, XAT_HIDDEN); + XVA_SET_REQ(&xvap, XAT_REPARSE); + XVA_SET_REQ(&xvap, XAT_OFFLINE); + XVA_SET_REQ(&xvap, XAT_SPARSE); + error = zfs_getattr(ap->a_vp, (vattr_t *)&xvap, 0, ap->a_cred, NULL); if (error != 0) return (error); /* Convert ZFS xattr into chflags. */ -#define FLAG_CHECK(fflag, xflag, xfield) do { \ - if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) \ +#define FLAG_CHECK(fflag, xflag, xfield, invert) do { \ + if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) { \ + if (!invert) \ + fflags |= (fflag); \ + } else if (invert) \ fflags |= (fflag); \ } while (0) FLAG_CHECK(SF_IMMUTABLE, XAT_IMMUTABLE, - xvap.xva_xoptattrs.xoa_immutable); + xvap.xva_xoptattrs.xoa_immutable, 0); FLAG_CHECK(SF_APPEND, XAT_APPENDONLY, - xvap.xva_xoptattrs.xoa_appendonly); + xvap.xva_xoptattrs.xoa_appendonly, 0); FLAG_CHECK(SF_NOUNLINK, XAT_NOUNLINK, - xvap.xva_xoptattrs.xoa_nounlink); + xvap.xva_xoptattrs.xoa_nounlink, 0); + FLAG_CHECK(SF_ARCHIVED, XAT_ARCHIVE, + xvap.xva_xoptattrs.xoa_archive, 1); FLAG_CHECK(UF_NODUMP, XAT_NODUMP, - xvap.xva_xoptattrs.xoa_nodump); + xvap.xva_xoptattrs.xoa_nodump, 0); + FLAG_CHECK(UF_IMMUTABLE, XAT_READONLY, + xvap.xva_xoptattrs.xoa_readonly, 0); + FLAG_CHECK(UF_SYSTEM, XAT_SYSTEM, + xvap.xva_xoptattrs.xoa_system, 0); + FLAG_CHECK(UF_HIDDEN, XAT_HIDDEN, + xvap.xva_xoptattrs.xoa_hidden, 0); + FLAG_CHECK(UF_REPARSE, XAT_REPARSE, + xvap.xva_xoptattrs.xoa_reparse, 0); + FLAG_CHECK(UF_OFFLINE, XAT_OFFLINE, + xvap.xva_xoptattrs.xoa_offline, 0); + FLAG_CHECK(UF_SPARSE, XAT_SPARSE, + xvap.xva_xoptattrs.xoa_sparse, 0); + #undef FLAG_CHECK *vap = xvap.xva_vattr; vap->va_flags = fflags; @@ -6111,7 +6137,16 @@ return (EOPNOTSUPP); fflags = vap->va_flags; - if ((fflags & ~(SF_IMMUTABLE|SF_APPEND|SF_NOUNLINK|UF_NODUMP)) != 0) + /* + * XXX KDM + * We need to figure out whether it makes sense to allow + * UF_REPARSE through, since we don't really have other + * facilities to handle reparse points and zfs_setattr() + * doesn't currently allow setting that attribute anyway. + */ + if ((fflags & ~(SF_IMMUTABLE|SF_APPEND|SF_ARCHIVED|SF_NOUNLINK| + UF_NODUMP|UF_IMMUTABLE|UF_SYSTEM|UF_HIDDEN|UF_REPARSE| + UF_OFFLINE|UF_SPARSE)) != 0) return (EOPNOTSUPP); /* * Unprivileged processes are not permitted to unset system @@ -6148,23 +6183,45 @@ } } -#define FLAG_CHANGE(fflag, zflag, xflag, xfield) do { \ - if (((fflags & (fflag)) && !(zflags & (zflag))) || \ - ((zflags & (zflag)) && !(fflags & (fflag)))) { \ - XVA_SET_REQ(&xvap, (xflag)); \ - (xfield) = ((fflags & (fflag)) != 0); \ +#define FLAG_CHANGE(fflag, zflag, xflag, xfield, invert) do { \ + if (!invert) { \ + if (((fflags & (fflag)) && !(zflags & (zflag))) || \ + ((zflags & (zflag)) && !(fflags & (fflag)))) { \ + XVA_SET_REQ(&xvap, (xflag)); \ + (xfield) = ((fflags & (fflag)) != 0); \ + } \ + } else { \ + if (((fflags & (fflag)) && (zflags & (zflag))) || \ + (!(zflags & (zflag)) && !(fflags & (fflag)))) { \ + XVA_SET_REQ(&xvap, (xflag)); \ + (xfield) = ((fflags & (fflag)) == 0); \ + } \ } \ } while (0) /* Convert chflags into ZFS-type flags. */ /* XXX: what about SF_SETTABLE?. */ FLAG_CHANGE(SF_IMMUTABLE, ZFS_IMMUTABLE, XAT_IMMUTABLE, - xvap.xva_xoptattrs.xoa_immutable); + xvap.xva_xoptattrs.xoa_immutable, 0); FLAG_CHANGE(SF_APPEND, ZFS_APPENDONLY, XAT_APPENDONLY, - xvap.xva_xoptattrs.xoa_appendonly); + xvap.xva_xoptattrs.xoa_appendonly, 0); FLAG_CHANGE(SF_NOUNLINK, ZFS_NOUNLINK, XAT_NOUNLINK, - xvap.xva_xoptattrs.xoa_nounlink); + xvap.xva_xoptattrs.xoa_nounlink, 0); + FLAG_CHANGE(SF_ARCHIVED, ZFS_ARCHIVE, XAT_ARCHIVE, + xvap.xva_xoptattrs.xoa_archive, 1); FLAG_CHANGE(UF_NODUMP, ZFS_NODUMP, XAT_NODUMP, - xvap.xva_xoptattrs.xoa_nodump); + xvap.xva_xoptattrs.xoa_nodump, 0); + FLAG_CHANGE(UF_IMMUTABLE, ZFS_READONLY, XAT_READONLY, + xvap.xva_xoptattrs.xoa_readonly, 0); + FLAG_CHANGE(UF_SYSTEM, ZFS_SYSTEM, XAT_SYSTEM, + xvap.xva_xoptattrs.xoa_system, 0); + FLAG_CHANGE(UF_HIDDEN, ZFS_HIDDEN, XAT_HIDDEN, + xvap.xva_xoptattrs.xoa_hidden, 0); + FLAG_CHANGE(UF_REPARSE, ZFS_REPARSE, XAT_REPARSE, + xvap.xva_xoptattrs.xoa_hidden, 0); + FLAG_CHANGE(UF_OFFLINE, ZFS_OFFLINE, XAT_OFFLINE, + xvap.xva_xoptattrs.xoa_offline, 0); + FLAG_CHANGE(UF_SPARSE, ZFS_SPARSE, XAT_SPARSE, + xvap.xva_xoptattrs.xoa_sparse, 0); #undef FLAG_CHANGE } return (zfs_setattr(vp, (vattr_t *)&xvap, 0, cred, NULL)); --- //depot/users/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.370 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-06 14:49:47.179130318 -0700 @@ -345,8 +345,17 @@ vap->va_birthtime.tv_nsec = 0; } vap->va_flags = 0; + /* + * The DOS Archive attribute means that a file needs to be + * archived. The BSD SF_ARCHIVED attribute means that a file has + * been archived. Thus the inversion here. + */ if ((dep->de_Attributes & ATTR_ARCHIVE) == 0) vap->va_flags |= SF_ARCHIVED; + if (dep->de_Attributes & ATTR_HIDDEN) + vap->va_flags |= UF_HIDDEN; + if (dep->de_Attributes & ATTR_SYSTEM) + vap->va_flags |= UF_SYSTEM; vap->va_gen = 0; vap->va_blocksize = pmp->pm_bpcluster; vap->va_bytes = @@ -420,12 +429,21 @@ if (error) return (error); } - if (vap->va_flags & ~SF_ARCHIVED) + if (vap->va_flags & ~(SF_ARCHIVED|UF_HIDDEN|UF_SYSTEM)) return EOPNOTSUPP; if (vap->va_flags & SF_ARCHIVED) dep->de_Attributes &= ~ATTR_ARCHIVE; else if (!(dep->de_Attributes & ATTR_DIRECTORY)) dep->de_Attributes |= ATTR_ARCHIVE; + if (vap->va_flags & UF_HIDDEN) + dep->de_Attributes |= ATTR_HIDDEN; + else + dep->de_Attributes &= ~ATTR_HIDDEN; + if (vap->va_flags & UF_SYSTEM) + dep->de_Attributes |= ATTR_SYSTEM; + else + dep->de_Attributes &= ~ATTR_SYSTEM; + dep->de_flag |= DE_MODIFIED; } --- //depot/users/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_node.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_node.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.417 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_node.c 2013-03-06 14:49:49.571128296 -0700 @@ -370,10 +370,13 @@ if (diff > 2) /* XXX should be configurable */ return ENOENT; va->va_type = vp->v_type; /* vnode type (for create) */ + va->va_flags = 0; /* flags defined for file */ if (vp->v_type == VREG) { va->va_mode = smp->sm_file_mode; /* files access mode and type */ - if (np->n_dosattr & SMB_FA_RDONLY) + if (np->n_dosattr & SMB_FA_RDONLY) { va->va_mode &= ~(S_IWUSR|S_IWGRP|S_IWOTH); + va->va_flags |= UF_IMMUTABLE; + } } else if (vp->v_type == VDIR) { va->va_mode = smp->sm_dir_mode; /* files access mode and type */ } else @@ -390,7 +393,17 @@ va->va_mtime = np->n_mtime; va->va_atime = va->va_ctime = va->va_mtime; /* time file changed */ va->va_gen = VNOVAL; /* generation number of file */ - va->va_flags = 0; /* flags defined for file */ + if (np->n_dosattr & SMB_FA_HIDDEN) + va->va_flags |= UF_HIDDEN; + if (np->n_dosattr & SMB_FA_SYSTEM) + va->va_flags |= UF_SYSTEM; + /* + * The meaning of the SF_ARCHIVED flag is the inverse of the CIFS + * archive flag. SF_ARCHIVED means that the file has been archived. + * SMB_FA_ARCHIVED means that the file needs to be archived. + */ + if ((vp->v_type != VDIR) && ((np->n_dosattr & SMB_FA_ARCHIVE) == 0)) + va->va_flags |= SF_ARCHIVED; va->va_rdev = NODEV; /* device the special file represents */ va->va_bytes = va->va_size; /* bytes of disk space held by file */ va->va_filerev = 0; /* file modification number */ --- //depot/users/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.493 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/smbfs/smbfs_vnops.c 2013-03-06 15:05:41.942127128 -0700 @@ -305,16 +305,30 @@ int old_n_dosattr; SMBVDEBUG("\n"); - if (vap->va_flags != VNOVAL) - return EOPNOTSUPP; isreadonly = (vp->v_mount->mnt_flag & MNT_RDONLY); /* * Disallow write attempts if the filesystem is mounted read-only. */ if ((vap->va_uid != (uid_t)VNOVAL || vap->va_gid != (gid_t)VNOVAL || vap->va_atime.tv_sec != VNOVAL || vap->va_mtime.tv_sec != VNOVAL || - vap->va_mode != (mode_t)VNOVAL) && isreadonly) + vap->va_mode != (mode_t)VNOVAL || vap->va_flags != VNOVAL) && + isreadonly) return EROFS; + + /* + * We only support setting five flags. Don't allow setting others. + * + * We map both SF_IMMUTABLE and UF_IMMUTABLE to SMB_FA_RDONLY for + * setting attributes. This is compatible with the MacOS X version + * of this code. SMB_FA_RDONLY translates only to UF_IMMUTABLE + * when getting attributes. + */ + if (vap->va_flags != VNOVAL) { + if (vap->va_flags & ~(UF_IMMUTABLE|UF_HIDDEN|UF_SYSTEM| + SF_ARCHIVED|SF_IMMUTABLE)) + return EINVAL; + } + scred = smbfs_malloc_scred(); smb_makescred(scred, td, ap->a_cred); if (vap->va_size != VNOVAL) { @@ -353,12 +367,47 @@ goto out; } } - if (vap->va_mode != (mode_t)VNOVAL) { + if ((vap->va_flags != VNOVAL) || (vap->va_mode != (mode_t)VNOVAL)) { old_n_dosattr = np->n_dosattr; - if (vap->va_mode & S_IWUSR) - np->n_dosattr &= ~SMB_FA_RDONLY; - else - np->n_dosattr |= SMB_FA_RDONLY; + + if (vap->va_mode != (mode_t)VNOVAL) { + if (vap->va_mode & S_IWUSR) + np->n_dosattr &= ~SMB_FA_RDONLY; + else + np->n_dosattr |= SMB_FA_RDONLY; + } + + if (vap->va_flags != VNOVAL) { + if (vap->va_flags & UF_HIDDEN) + np->n_dosattr |= SMB_FA_HIDDEN; + else + np->n_dosattr &= ~SMB_FA_HIDDEN; + + if (vap->va_flags & UF_SYSTEM) + np->n_dosattr |= SMB_FA_SYSTEM; + else + np->n_dosattr &= ~SMB_FA_SYSTEM; + + if (vap->va_flags & SF_ARCHIVED) + np->n_dosattr &= ~SMB_FA_ARCHIVE; + else + np->n_dosattr |= SMB_FA_ARCHIVE; + + /* + * We only support setting the immutable / readonly + * bit for regular files. According to comments in + * the MacOS X version of this code, supporting the + * readonly bit on directories doesn't do the same + * thing in Windows as in Unix. + */ + if (vp->v_type == VREG) { + if (vap->va_flags & (UF_IMMUTABLE|SF_IMMUTABLE)) + np->n_dosattr |= SMB_FA_RDONLY; + else + np->n_dosattr &= ~SMB_FA_RDONLY; + } + } + if (np->n_dosattr != old_n_dosattr) { error = smbfs_smb_setpattr(np, np->n_dosattr, NULL, scred); if (error) --- //depot/users/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.563 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-06 15:05:45.936126193 -0700 @@ -268,6 +268,22 @@ #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ #define UF_NOUNLINK 0x00000010 /* file may not be removed or renamed */ /* + * These two bits are defined in MacOS X. They are not currently used in + * FreeBSD. + */ +#if 0 +#define UF_COMPRESSED 0x00000020 /* file is compressed */ +#define UF_TRACKED 0x00000040 /* renames and deletes are tracked */ +#endif + +#define UF_SYSTEM 0x00000080 /* Windows system file bit */ +#define UF_SPARSE 0x00000100 /* sparse file */ +#define UF_OFFLINE 0x00000200 /* file is offline */ +#define UF_REPARSE 0x00000400 /* Windows reparse point file bit */ +/* This is the same as the MacOS X definition of UF_HIDDEN */ +#define UF_HIDDEN 0x00008000 /* file is hidden */ + +/* * Super-user changeable flags. */ #define SF_SETTABLE 0xffff0000 /* mask of superuser changeable flags */ --- //depot/users/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-04 17:51:12.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-04 17:51:12.000000000 -0700 --- /tmp/tmp.49594.581 2013-03-06 16:42:43.000000000 -0700 +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-06 15:06:27.004132409 -0700 @@ -529,8 +529,9 @@ } if (vap->va_flags != VNOVAL) { if ((vap->va_flags & ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | - UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | - SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) + UF_OPAQUE | UF_NOUNLINK | UF_HIDDEN | UF_SYSTEM | + UF_SPARSE | UF_OFFLINE | UF_REPARSE | SF_ARCHIVED | + SF_IMMUTABLE | SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) return (EOPNOTSUPP); if (vp->v_mount->mnt_flag & MNT_RDONLY) return (EROFS); --u3/rZRmxL6MmkK24-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 01:14:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 03FB88D1 for ; Thu, 7 Mar 2013 01:14:15 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id A151A748 for ; Thu, 7 Mar 2013 01:14:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=Dv4UxrXV9rxs6UhKdOfvkRT5oqPD6tI0D4ACGMddGxI=; b=l3FIpCXlTMcrFP8tBp9pe+snwFyEFCyG5rlUkNvIPPwXEaJUL7tIW+9uHZypp8yrebOMjvW/3zDpar8BRk5lYeZ5CfTwvIFgnpvbGwwMAojYIkcvQGo28fKHZyPVxfIH4TtiGld5xdMnfk+PQ9PPQNkAedjH+VFbb7090YcaOPM=; Received: from localhost.lerctr.org ([127.0.0.1]:59769 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDPPK-000PgK-Lc; Wed, 06 Mar 2013 19:14:09 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 19:14:05 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 19:14:05 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 01:14:15 -0000 On 2013-03-06 17:48, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" > To: "Steven Hartland" > Cc: "Martin Simmons" ; > ; ; > > Sent: Wednesday, March 06, 2013 11:16 PM > Subject: Re: zfs send/recv invalid data > > >> On 2013-03-06 17:00, Steven Hartland wrote: >>> ----- Original Message ----- From: "Larry Rosenman" >>>> # cat send.stream | ssh root@home openssl md5 >>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>> # uname -a >>>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 >>>> r247820: Mon Mar 4 18:08:11 CST 2013 >>>> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER >>>> amd64 >>>> # >>>> # exec csh >>>> root@borg:/home/ler # pwd >>>> /home/ler >>>> root@borg:/home/ler # ssh tbh "cat >>>> /home/ler/public_html/ZFS_RECV/send.stream | ssh >>>> root@home.lerctr.org openssl md5" >>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>> root@borg:/home/ler # uname -a >>>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 >>>> r247826: Mon Mar 4 19:59:08 CST 2013 >>>> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >>>> root@borg:/home/ler # >>> When you run a command from ssh it will run as the target users >>> shell >>> so for csh if there is something bad in .cshrc or iirc .profile then >>> you may get something output purely by logging in. >>> So ensure you shell config's for the user on the target machine are >>> clean. >>> You can confirm this by comparing the output from the following >>> two commands run from the sending machine:- >>> md5 send.stream >>> cat send.stream | ssh root@home.lerctr.org "cat" |md5 >>> If these two commands produce different md5's then the login >>> shell for root on home.lerctr.org is outputing something >>> even when there is no terminal. >>> You should be able to see what the shell is outputting using:- >>> echo -n '' | ssh root@home.lerctr.org "cat" > 1.log >>> After running this 1.log should be 0 bytes if its not your >>> shell is definitely your problem and I'd start by looking >>> at .cshrc and .profile for this user, or even the globals >>> for these in /etc/ >>> >> I've done that, and get identical md5's..... >> I'm at a loss. > > I believe what you ran was similar but not identical, if you > check the quoting subtly different. > > I'm not sure that will have any effect but both of the above > are worth running just to confirm. > > Regards > Steve $ ^DConnection to thebighonker.lerctr.org closed. $ ssh root@tbh Last login: Tue Mar 5 06:51:46 2013 from cpe-72-182-19-1 FreeBSD 8.3-STABLE (THEBIGHONKER) #54 r247820: Mon Mar 4 18:08:11 CST 2013 Welcome to FreeBSD! root@thebighonker:~ # pwd /root root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send send-file-snap-list send.stream send.uname root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send.stream MD5 (/home/ler/public_html/ZFS_RECV/send.stream) = 9cd1d73ea8411f1c222bc90e7bea3d33 root@thebighonker:~ # cat /home/ler/public_html/ZFS_RECV/send.stream |ssh root@home.lerctr.org "cat"|md5 9cd1d73ea8411f1c222bc90e7bea3d33 root@thebighonker:~ # looks like it's clean..... Other ideas? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 11:22:03 2013 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 48D68647; Thu, 7 Mar 2013 11:22:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail110.syd.optusnet.com.au (mail110.syd.optusnet.com.au [211.29.132.97]) by mx1.freebsd.org (Postfix) with ESMTP id E9A08237; Thu, 7 Mar 2013 11:22:02 +0000 (UTC) Received: from c211-30-173-106.carlnfd1.nsw.optusnet.com.au (c211-30-173-106.carlnfd1.nsw.optusnet.com.au [211.30.173.106]) by mail110.syd.optusnet.com.au (Postfix) with ESMTPS id DD323780A45; Thu, 7 Mar 2013 22:21:51 +1100 (EST) Date: Thu, 7 Mar 2013 22:21:38 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Kenneth D. Merry" Subject: Re: patches to add new stat(2) file flags In-Reply-To: <20130307000533.GA38950@nargothrond.kdm.org> Message-ID: <20130307214649.X981@besplex.bde.org> References: <20130307000533.GA38950@nargothrond.kdm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=bNdOu4CZ c=1 sm=1 a=n2O7wv11oSwA:10 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=YOiZBDKP_E4A:10 a=vDvlQP4Unmb3WPkJY9MA:9 a=CjuIK1q_8ugA:10 a=TEtd8y5WR3g2ypngnwZWYw==:117 Cc: arch@FreeBSD.org, fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 11:22:03 -0000 On Wed, 6 Mar 2013, Kenneth D. Merry wrote: > I have attached diffs against head for some additional stat(2) file flags. > > The primary purpose of these flags is to improve compatibility with CIFS, > both from the client and the server side. > ... > UF_IMMUTABLE: Command line name: "uchg", "uimmutable" > ZFS name: XAT_READONLY, ZFS_READONLY > Windows: FILE_ATTRIBUTE_READONLY > > This flag means that the file may not be modified. > This is not a new flag, but where applicable it is > mapped to the Windows readonly bit. ZFS and UFS > now both support the flag and enforce it. > > The behavior of this flag is compatible with MacOS X. This is incompatible with mapping the DOS read-only attribute to the non-writeable file permission in msdosfs. msdosfs does this mainly to get at least one useful file permission, but the semantics are subtly different from all of file permissions, UF_IMMUTABLE and SF_IMMUTABLE. I think it should be a new flag. > SF_ARCHIVED: Command line name: "arch", "archived" > ZFS_NAME: XAT_ARCHIVE, ZFS_ARCHIVE > Windows name: FILE_ATTRIBUTE_ARCHIVE > > The SF_ARCHIVED flag means that the file has been > archived. The meaning is the inverse of the Windows > FILE_ATTRIBUTE_ARCHIVE, and the ZFS XAT_ARCHIVE and > ZFS_ARCHIVE attribute. Accordingly, we invert the > internal flags in ZFS and SMBFS to get the value of > SF_ARCHIVED. msdosfs inverts this too. Another problem with it is that (I think) changing it is not a privileged operation in WinDOS, so it should be mapped to UF_ARCHIVE and not reversed. Since you are adding lots of flags, you could easily fix this. You made all the new flags UF_*. Even removal of SF_ARCHIVED wouldn't be missed, since ffs doesn't maintain it and no FreeBSD utility in /usr/src supports it (except to change and print it). > msdosfs_vnops.c: Add support for getting and setting > UF_HIDDEN and UF_SYSTEM in MSDOSFS. There is also a PR that maps the SYSTEM attribute to SF_IMMUTABLE. That would be useful but I think it is not orthogonal enough, and SF_IMMUTABLE is probably too strict. Mapping it to UF_SYSTEM ... > It already supports SF_ARCHIVED. It > currently supports the readonly bit with > standard Unix permissions, but could be > extended to support UF_IMMUTABLE. ... and not mapping READONLY leaves no mapping to an immutable bit. If you map READONLY to a new flag, it can still be used to give a certain amount of immutability that is weaker than [US]F_IMMUTABLE. Under FreeBSD, the immutable flags prevent changing anything, but under WinXP, I didn't notice the READONLY attribute preventing Cygwin (on ntfs and FAT32 file systems) changing things like acls and times, but that might be because Cygwin works around what it considers excessive security. Bruce From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 13:37:25 2013 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6E213489; Thu, 7 Mar 2013 13:37:25 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.freebsd.org (Postfix) with ESMTP id C45F9A37; Thu, 7 Mar 2013 13:37:24 +0000 (UTC) Received: from c211-30-173-106.carlnfd1.nsw.optusnet.com.au (c211-30-173-106.carlnfd1.nsw.optusnet.com.au [211.30.173.106]) by mail03.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id r27DbFZ3029843 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 8 Mar 2013 00:37:15 +1100 Date: Fri, 8 Mar 2013 00:37:15 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Kenneth D. Merry" Subject: Re: patches to add new stat(2) file flags In-Reply-To: <20130307000533.GA38950@nargothrond.kdm.org> Message-ID: <20130307222553.P981@besplex.bde.org> References: <20130307000533.GA38950@nargothrond.kdm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=DdhPMYRW c=1 sm=1 a=n2O7wv11oSwA:10 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=YOiZBDKP_E4A:10 a=-GnrcQN2Q_ltR23H6AcA:9 a=CjuIK1q_8ugA:10 a=TEtd8y5WR3g2ypngnwZWYw==:117 Cc: arch@FreeBSD.org, fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 13:37:25 -0000 On Wed, 6 Mar 2013, Kenneth D. Merry wrote: > I have attached diffs against head for some additional stat(2) file flags. > > The primary purpose of these flags is to improve compatibility with CIFS, > both from the client and the server side. > ... I missed looking at the diffs in my previous reply. % --- //depot/users/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.86 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-06 14:47:25.987128763 -0700 % @@ -117,6 +117,16 @@ % set the user immutable flag (owner or super-user only) % .It Cm uunlnk , uunlink % set the user undeletable flag (owner or super-user only) % +.It Cm system , usystem % +set the Windows system flag (owner or super-user only) This begins unsorting of the list. It's not just a Windows flag, since it also works in DOS. "Owner or" is too strict for msdosfs, since files can only have a single owner so it is controlling access using groups is needed. I use owner root and group msdosfs for msdosfs mounts. This works for normal operations like open/read/write, but fails for most attributes including file flags. msdosfs doesn't support many attributes but this change is supposed to add support for 3 new file flags so it would be good if it didn't restrict the support to root. % +.It Cm sparse , usparse % +set the sparse file attribute (owner or super-user only) % +.It Cm offline , uoffline % +set the offline file attribute (owner or super-user only) % +.It Cm reparse , ureparse % +set the Windows reparse point file attribute (owner or super-user only) % +.It Cm hidden , uhidden % +set the hidden file attribute (owner or super-user only) The additions at the end are also internally unsorted. Previously only "opaque" and "nodump" were unsorted. They are UF flags sorted with the SF flags, and "no" in "nodump" is not ignored for the purposes of sorting. Not having "u" in the old and new UF flag names messes up the sort order (unless you add futher confusion by ignoring "u" for the purposes of sorting) and makes it harder to add SF variants of the flags. % .El % .Pp % Putting the letters % --- //depot/users/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.178 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c 2013-03-06 15:09:32.842437917 -0700 % @@ -68,7 +68,17 @@ % { "nodump", 1, UF_NODUMP }, % { "noopaque", 0, UF_OPAQUE }, % { "nouunlnk", 0, UF_NOUNLINK }, % - { "nouunlink", 0, UF_NOUNLINK } % + { "nouunlink", 0, UF_NOUNLINK }, % + { "nosystem", 0, UF_SYSTEM, }, % + { "nousystem", 0, UF_SYSTEM, }, % + { "nosparse", 0, UF_SPARSE, }, % + { "nousparse", 0, UF_SPARSE, }, % + { "nooffline", 0, UF_OFFLINE, }, % + { "nouoffline", 0, UF_OFFLINE, }, % + { "noreparse", 0, UF_REPARSE, }, % + { "noureparse", 0, UF_REPARSE, }, % + { "nohidden", 0, UF_HIDDEN, }, % + { "nouhidden", 0, UF_HIDDEN, } This is totally disordered too. The old table was sorted except for "nosnapshot". Another bug is that "nosnapshot" is supported here (so chflags(1) can show it), but is not documented in chflags(1). % }; % #define nmappings (sizeof(mapping) / sizeof(mapping[0])) % % --- //depot/users/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.257 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-06 14:49:39.357125717 -0700 % @@ -75,16 +75,49 @@ % Do not dump the file. % .It Dv UF_IMMUTABLE % The file may not be changed. % +Filesystems may use this flag to maintain compatibility with the Windows and % +CIFS FILE_ATTRIBUTE_READONLY attribute. % .It Dv UF_APPEND % The file may only be appended to. This was already totally disordered. It even has UF's before SF's, although SF's sort before UF's both lexically and logically, though not in sys/stat.h or in bits. The disorder here was apparently copied from the implementation (sys/stat.h, which is in the order of the bits, which is historical for binary compatibility) and not cleaned up like chflags(1) and strttoflags(3). % .It Dv UF_NOUNLINK % The file may not be renamed or deleted. % .It Dv UF_OPAQUE % The directory is opaque when viewed through a union stack. % +.It Dv UF_SYSTEM % +The file has the Windows and CIFS FILE_ATTRIBUTE_SYSTEM attribute. % +Filesystems in FreeBSD may store and display this flag, but do not provide % +any special handling when it is set. More disordered than before... % +.It Dv UF_HIDDEN % +The file may be hidden from directory listings at the application's % +discretion. % +The file has the Windows FILE_ATTRIBUTE_HIDDEN attribute. Not just Windows... % .It Dv SF_ARCHIVED % -The file may be archived. % +The file has been archived. % +This flag means the opposite of the Windows and CIFS FILE_ATTRIBUTE_ARCHIVE % +attribute. % +That attribute means that the file should be archived, whereas % +.Dv SF_ARCHIVED % +means that the file has been archived. WinXP uses the poor wording "avialable for archiving". This is better. % +Filesystems in FreeBSD may or may not have special handling for this flag. % +For instance, ZFS tracks changes to files and will clear this bit when a % +file is updated. % +UFS only stores the flag, and relies on the application to change it when % +needed. I think that is useless, since changing it is needed whenever the file changes, and applications can do that (short of running as daemons and watching for changes). WinXP seems to not set the flag when attributes change. I think that is a bug, and FreeBSD msdosfs does set it when attributes are changed (except for the SF_ARCHIVED attribute itself, and atimes when the atimes are set automatically). The FreeBSD behaviour is like setting the ctime for any change and is bug for bug compatible in doing this even for null changes. FreeBSD doesn't have many attributes to set, but you just added some more. I think this should be controlled by a mount option so that bug for bug compatibility with WinDOS is possible. % .It Dv SF_IMMUTABLE % The file may not be changed. % +This flag also indicates that the Windows ans CIFS FILE_ATTRIBUTE_READONLY % +attribute is set. s/ans/and/ You only mentioned UF_IMMUTABLE in your general description. Mapping READONLY to both gives even more confusing semantics, and mapping it to SF_IMMUTABLE seems too strict since for example it prevents using FreeBSD to manage the flag at high securelevels. % --- //depot/users/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.370 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c 2013-03-06 14:49:47.179130318 -0700 % @@ -345,8 +345,17 @@ % vap->va_birthtime.tv_nsec = 0; % } % vap->va_flags = 0; % + /* % + * The DOS Archive attribute means that a file needs to be % + * archived. The BSD SF_ARCHIVED attribute means that a file has % + * been archived. Thus the inversion here. % + */ No need to document it again. It goes without saying that ARCHIVE != ARCHIVED. % if ((dep->de_Attributes & ATTR_ARCHIVE) == 0) % vap->va_flags |= SF_ARCHIVED; % + if (dep->de_Attributes & ATTR_HIDDEN) % + vap->va_flags |= UF_HIDDEN; % + if (dep->de_Attributes & ATTR_SYSTEM) % + vap->va_flags |= UF_SYSTEM; % vap->va_gen = 0; % vap->va_blocksize = pmp->pm_bpcluster; % vap->va_bytes = % @@ -420,12 +429,21 @@ % if (error) % return (error); % } The permissions check before this is delicate and was broken and is more broken now. It is still short-circuit to handle setting the single flag that used to be supported, and is slightly broken for that: - unprivileged user asks to set ARCHIVE by passing !SF_ARCHIVED. We allow that, although this may toggle the flag and normal semantics for SF flags is to not allow toggling. - unprivileged user asks to clear ARCHIVE by passing SF_ARCHIVED. We don't allow that. But we should allow preserving ARCHIVE if it is already clear. The bug wasn't very important when only 1 flag was supported. Now it prevents unprivileged users managing the new UF flags if ARCHIVE is clear. Fortunately, this is the unusual case. Anyway, unprivileged users can set ARCHIVE by doing some other operation. Even the chflags() operation should set ARCHIVE and thus allow further chflags()'s that now keep ARCHIVE set. Except it is very confusing if a chflags() asks for ARCHIVE to be clear. This request might be just to try to preserve the current setting and not want it if other things are changed, or it might be to purposely clear it. Changing it from set to clear should still be privileged. See the more complicated permissions check in ffs. It would be safest to duplicate most of it, to get different permissions checking for the SF and UF flags. Then decide if we want to keep allowing setting ARCHIVE without privilege. % - if (vap->va_flags & ~SF_ARCHIVED) % + if (vap->va_flags & ~(SF_ARCHIVED|UF_HIDDEN|UF_SYSTEM)) Style bugs (missing spaces around binary operator '|'). These style bugs are common in atribute handling for other fs's but were not here. % return EOPNOTSUPP; % if (vap->va_flags & SF_ARCHIVED) % dep->de_Attributes &= ~ATTR_ARCHIVE; % else if (!(dep->de_Attributes & ATTR_DIRECTORY)) % dep->de_Attributes |= ATTR_ARCHIVE; The comment before this says that we ignore attmps to set ATTR_ARCHIVED for directories. However, it is out of date. WinXP allows setting it and all the new flags for directories, and so do we. % + if (vap->va_flags & UF_HIDDEN) % + dep->de_Attributes |= ATTR_HIDDEN; % + else % + dep->de_Attributes &= ~ATTR_HIDDEN; % + if (vap->va_flags & UF_SYSTEM) % + dep->de_Attributes |= ATTR_SYSTEM; % + else % + dep->de_Attributes &= ~ATTR_SYSTEM; I thought you were adding ATTR_READONLY -> UF_IMMUTABLE here. The WinXP attrib command (at least on a FAT32 fs) doesn't allow setting or clearing ARCHIVE (even if it is already set or clear) if any of HIDDEN, READONLY or SYSTEM is already set and remains set after the command. Thus the HRS attributes act a bit like immutable flags, but subtly differently. (ffs has the quite different and worse behaviour of allowing chflags() irrespective of immutable flags being set before or after, provided there is enough privilege to change the immutable flags.) Anyway, they should all give some aspects of immutability. % + Style bug (extra blank line). % dep->de_flag |= DE_MODIFIED; % } % % --- //depot/users/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.563 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-06 15:05:45.936126193 -0700 % @@ -268,6 +268,22 @@ % #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ % #define UF_NOUNLINK 0x00000010 /* file may not be removed or renamed */ Old style bugs: inconsistent space instead of tab after #define for the 2 newer definitions. % /* % + * These two bits are defined in MacOS X. They are not currently used in % + * FreeBSD. % + */ % +#if 0 % +#define UF_COMPRESSED 0x00000020 /* file is compressed */ % +#define UF_TRACKED 0x00000040 /* renames and deletes are tracked */ % +#endif % + % +#define UF_SYSTEM 0x00000080 /* Windows system file bit */ % +#define UF_SPARSE 0x00000100 /* sparse file */ % +#define UF_OFFLINE 0x00000200 /* file is offline */ % +#define UF_REPARSE 0x00000400 /* Windows reparse point file bit */ % +/* This is the same as the MacOS X definition of UF_HIDDEN */ % +#define UF_HIDDEN 0x00008000 /* file is hidden */ New style bugs: random spaces/tabs after #define. 1 main comment not punctuated. % + % +/* % * Super-user changeable flags. % */ % #define SF_SETTABLE 0xffff0000 /* mask of superuser changeable flags */ % --- //depot/users/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-04 17:51:12.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-04 17:51:12.000000000 -0700 % --- /tmp/tmp.49594.581 2013-03-06 16:42:43.000000000 -0700 % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-06 15:06:27.004132409 -0700 % @@ -529,8 +529,9 @@ % } % if (vap->va_flags != VNOVAL) { % if ((vap->va_flags & ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | % - UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | % - SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) % + UF_OPAQUE | UF_NOUNLINK | UF_HIDDEN | UF_SYSTEM | % + UF_SPARSE | UF_OFFLINE | UF_REPARSE | SF_ARCHIVED | % + SF_IMMUTABLE | SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) % return (EOPNOTSUPP); % if (vp->v_mount->mnt_flag & MNT_RDONLY) % return (EROFS); The random order is harder to read with more flags. I think unsupported flags should just be unsupported -- most or all of the new flags and SF_ARCHIVED too. Most need system support to work. Without system support you can only copy them from other fs's and then lose them when utilities like tar don't unsderstand them. Bruce From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 16:06:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E82E7783 for ; Thu, 7 Mar 2013 16:06:08 +0000 (UTC) (envelope-from universite@ukr.net) Received: from ffe17.ukr.net (ffe17.ukr.net [195.214.192.83]) by mx1.freebsd.org (Postfix) with ESMTP id 95D752C9 for ; Thu, 7 Mar 2013 16:06:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Date:Message-Id:From:To:Subject:Content-Type:Content-Transfer-Encoding:MIME-Version; bh=yTDqn1eqBJSKF1sTI+6Luif4AD2qF3psWBnB8tAHQxA=; b=XXbxsl/I0pReN9XMpslvL1rno+MlrSZnSKlVLm3m4vpSk1J9dqsb6PFBGCMq/hmiDx6Fniwev84aVsKml2jehG26WMjdzTCXl69GvD7bUgX0P4zaHvNRLmgitGmkNzmEmsUxS2NmpDNzJybWF99rpOcUpCQS2t4wYY1RnJRzhbE=; Received: from mail by ffe17.ukr.net with local ID 1UDd5Y-000Lvd-Q8 for freebsd-fs@freebsd.org; Thu, 07 Mar 2013 17:50:36 +0200 MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: binary Content-Type: text/plain; charset="windows-1251" Subject: Argument list too long To: freebsd-fs@freebsd.org From: "Vladislav Prodan" X-Mailer: freemail.ukr.net 4.0 Message-Id: <82112.1362671436.13776555968178880512@ffe17.ukr.net> Date: Thu, 07 Mar 2013 17:50:36 +0200 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 16:06:09 -0000 Why 12K small files from one directory to cause problems? # ll | wc -l 11467 # grep X-PHP-Script * | more /sbin/grep: Argument list too long. # egrep X-PHP-Script *.ua | more /usr/sbin/egrep: Argument list too long. # cat *.ua | grep X-PHP-Script | more /sbin/cat: Argument list too long. -- Vladislav V. Prodan System & Network Administrator http://support.od.ua +380 67 4584408, +380 99 4060508 VVP88-RIPE From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 16:15:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1F416982 for ; Thu, 7 Mar 2013 16:15:49 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id B5752329 for ; Thu, 7 Mar 2013 16:15:48 +0000 (UTC) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 13E1C6A6006; Thu, 7 Mar 2013 17:15:47 +0100 (CET) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.5/8.14.5) with ESMTP id r27GFkTh046549; Thu, 7 Mar 2013 17:15:46 +0100 (CET) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.5/8.14.5/Submit) id r27GFkhb046195; Thu, 7 Mar 2013 17:15:46 +0100 (CET) (envelope-from lars) Date: Thu, 7 Mar 2013 17:15:46 +0100 From: Lars Engels To: Vladislav Prodan Subject: Re: Argument list too long Message-ID: <20130307161546.GV47829@e-new.0x20.net> References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="finhkSr2yyrWw3ow" Content-Disposition: inline In-Reply-To: <82112.1362671436.13776555968178880512@ffe17.ukr.net> X-Editor: VIM - Vi IMproved 7.3 X-Operation-System: FreeBSD 8.3-RELEASE-p4 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 16:15:49 -0000 --finhkSr2yyrWw3ow Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 07, 2013 at 05:50:36PM +0200, Vladislav Prodan wrote: > Why 12K small files from one directory to cause problems? >=20 > # ll | wc -l > 11467 >=20 > # grep X-PHP-Script * | more > /sbin/grep: Argument list too long. >=20 > # egrep X-PHP-Script *.ua | more > /usr/sbin/egrep: Argument list too long. >=20 > # cat *.ua | grep X-PHP-Script | more > /sbin/cat: Argument list too long. >=20 >=20 >=20 Your shell can't process that many arguments. Use this: grep -R "X-PHP-Script" . or if you don't want to descent into subdirectories: find . -type -f -name '*.ua' -maxdepth 1 -exec grep "X-PHP-Script" {} \+ --finhkSr2yyrWw3ow Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlE4vTIACgkQKc512sD3afjnvACeLHvYfj+ScPgn5mDUUpU9UcBi h2EAn2HctdmTK4zvz2QRS9zEAD2MaMoB =Omb/ -----END PGP SIGNATURE----- --finhkSr2yyrWw3ow-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 16:35:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1BAACBC9 for ; Thu, 7 Mar 2013 16:35:02 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (mail.distal.com [IPv6:2001:470:e24c:200::ae25]) by mx1.freebsd.org (Postfix) with ESMTP id DA1CB3FB for ; Thu, 7 Mar 2013 16:35:01 +0000 (UTC) Received: from zalamar.mm-corp.net (static-66-16-13-46.dsl.cavtel.net [66.16.13.46]) (authenticated bits=0) by mail.distal.com (8.14.3/8.14.3) with ESMTP id r27GYthh008702 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 7 Mar 2013 11:34:57 -0500 (EST) Subject: Re: Argument list too long Mime-Version: 1.0 (Apple Message framework v1283) From: Chris Ross In-Reply-To: <20130307161546.GV47829@e-new.0x20.net> Date: Thu, 7 Mar 2013 11:34:48 -0500 Message-Id: <0CABF9B0-5274-44B3-984E-54CCD5C2F687@distal.com> References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> <20130307161546.GV47829@e-new.0x20.net> To: Lars Engels X-Mailer: Apple Mail (2.1283) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 16:35:02 -0000 On Mar 7, 2013, at 11:15 AM, Lars Engels wrote: > On Thu, Mar 07, 2013 at 05:50:36PM +0200, Vladislav Prodan wrote: >> Why 12K small files from one directory to cause problems? >>=20 >> # ll | wc -l >> 11467 >>=20 >> # grep X-PHP-Script * | more >> /sbin/grep: Argument list too long. >>=20 >> # egrep X-PHP-Script *.ua | more >> /usr/sbin/egrep: Argument list too long. >>=20 >> # cat *.ua | grep X-PHP-Script | more >> /sbin/cat: Argument list too long. >>=20 >=20 > Your shell can't process that many arguments. Use this: >=20 > grep -R "X-PHP-Script" . >=20 > or if you don't want to descent into subdirectories: >=20 > find . -type -f -name '*.ua' -maxdepth 1 -exec grep "X-PHP-Script" {} = \+ Those solutions will work, but the problem is actually the inability = of the shell to start a process with that long of an argument list. = (ARG_MAX bytes). The shell can process them, just not hand them off to = another process via the argument vector. echo * | xargs grep X-PHP-Script | more will also work. (man xargs for more information) Sorry to be a bit pedantic. - Chris From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 19:08:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 399BE256 for ; Thu, 7 Mar 2013 19:08:40 +0000 (UTC) (envelope-from prvs=1778223946=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C9FF7D91 for ; Thu, 7 Mar 2013 19:08:39 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50002603136.msg for ; Thu, 07 Mar 2013 19:08:38 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 07 Mar 2013 19:08:38 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1778223946=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> From: "Steven Hartland" To: "Larry Rosenman" References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> Subject: Re: zfs send/recv invalid data Date: Thu, 7 Mar 2013 19:08:44 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 19:08:40 -0000 ----- Original Message ----- From: "Larry Rosenman" To: "Steven Hartland" Cc: "Martin Simmons" ; ; ; Sent: Thursday, March 07, 2013 1:14 AM Subject: Re: zfs send/recv invalid data > On 2013-03-06 17:48, Steven Hartland wrote: >> ----- Original Message ----- From: "Larry Rosenman" >> To: "Steven Hartland" >> Cc: "Martin Simmons" ; >> ; ; >> >> Sent: Wednesday, March 06, 2013 11:16 PM >> Subject: Re: zfs send/recv invalid data >> >> >>> On 2013-03-06 17:00, Steven Hartland wrote: >>>> ----- Original Message ----- From: "Larry Rosenman" >>>>> # cat send.stream | ssh root@home openssl md5 >>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> # uname -a >>>>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 >>>>> r247820: Mon Mar 4 18:08:11 CST 2013 >>>>> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER >>>>> amd64 >>>>> # >>>>> # exec csh >>>>> root@borg:/home/ler # pwd >>>>> /home/ler >>>>> root@borg:/home/ler # ssh tbh "cat >>>>> /home/ler/public_html/ZFS_RECV/send.stream | ssh >>>>> root@home.lerctr.org openssl md5" >>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> root@borg:/home/ler # uname -a >>>>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 >>>>> r247826: Mon Mar 4 19:59:08 CST 2013 >>>>> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >>>>> root@borg:/home/ler # >>>> When you run a command from ssh it will run as the target users >>>> shell >>>> so for csh if there is something bad in .cshrc or iirc .profile then >>>> you may get something output purely by logging in. >>>> So ensure you shell config's for the user on the target machine are >>>> clean. >>>> You can confirm this by comparing the output from the following >>>> two commands run from the sending machine:- >>>> md5 send.stream >>>> cat send.stream | ssh root@home.lerctr.org "cat" |md5 >>>> If these two commands produce different md5's then the login >>>> shell for root on home.lerctr.org is outputing something >>>> even when there is no terminal. >>>> You should be able to see what the shell is outputting using:- >>>> echo -n '' | ssh root@home.lerctr.org "cat" > 1.log >>>> After running this 1.log should be 0 bytes if its not your >>>> shell is definitely your problem and I'd start by looking >>>> at .cshrc and .profile for this user, or even the globals >>>> for these in /etc/ >>>> >>> I've done that, and get identical md5's..... >>> I'm at a loss. >> >> I believe what you ran was similar but not identical, if you >> check the quoting subtly different. >> >> I'm not sure that will have any effect but both of the above >> are worth running just to confirm. >> >> Regards >> Steve > $ ^DConnection to thebighonker.lerctr.org closed. > $ ssh root@tbh > Last login: Tue Mar 5 06:51:46 2013 from cpe-72-182-19-1 > FreeBSD 8.3-STABLE (THEBIGHONKER) #54 r247820: Mon Mar 4 18:08:11 CST > 2013 > > Welcome to FreeBSD! > > root@thebighonker:~ # pwd > /root > root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send > send-file-snap-list send.stream send.uname > root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send.stream > MD5 (/home/ler/public_html/ZFS_RECV/send.stream) = > 9cd1d73ea8411f1c222bc90e7bea3d33 > root@thebighonker:~ # cat /home/ler/public_html/ZFS_RECV/send.stream > |ssh root@home.lerctr.org "cat"|md5 > 9cd1d73ea8411f1c222bc90e7bea3d33 > root@thebighonker:~ # > > > looks like it's clean..... So to summarise, just to make sure I've got everything clear so far:- 1. sending the stream over ssh from the source -> target results in the same md5 (from the test above) 2. sending the stream over ssh from source -> target piped into zfs recv fails 3. scp'ing the stream file from the source to the target and then importing it via zfs recv works Yes / No? If so this is very strange indeed. I'd like to confirm that adding the zfs recv to the picture isn't changing the dynamic such that the stdin results are changing and thus eliminating ssh as the source of any corruption to do this can you test:- md5 send.stream cat send.stream | ssh root@home.lerctr.org "tee tea.stream | zfs recv " ssh root@home.lerctr.org "md5 tea.stream" Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Thu Mar 7 19:36:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B001AF39 for ; Thu, 7 Mar 2013 19:36:13 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id 72EF8EAE for ; Thu, 7 Mar 2013 19:36:13 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id va7so657289obc.41 for ; Thu, 07 Mar 2013 11:36:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=J9D66F5JczxnEXk2YAE95PNF5yb9u1HIzrIj+9PXERI=; b=Em72/BdDWfIoTYVjsdTGY5OSRYOwzugrt0QoBYUa8UeZwg6/hm22C3YFrPiA8F5oEw 3OaRyqZWIonkwjyty8Y84tHWCtr5uH2HpxhBBPvOfO3mUnwLnn05LMhAbKMB5Esr1jMH nRd2RH6uo0dXdOiomqVy9s7iEagJPMqxut9r4mUeh/uToGCBaiRNwdGG3vsZvNOTZ0IH UhJkYfvOC1r+1L4FnmlF57ci25YuuQA0Taa3PZOQxqx1h6icyy5TMi95JaB/mitKP5Es s3sS8BsyEpMkd9mHF9lQcw6vaq/T32By5l9bjav0VpLl1f6myOsy9mBIkdY7bY72/rQ2 C+QQ== MIME-Version: 1.0 X-Received: by 10.60.29.161 with SMTP id l1mr22818472oeh.111.1362684973060; Thu, 07 Mar 2013 11:36:13 -0800 (PST) Received: by 10.76.109.236 with HTTP; Thu, 7 Mar 2013 11:36:12 -0800 (PST) In-Reply-To: <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> Date: Thu, 7 Mar 2013 14:36:12 -0500 Message-ID: Subject: Re: zfs send/recv invalid data From: Ryan Stone To: Steven Hartland Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: tevans.uk@googlemail.com, "freebsd-fs@freebsd.org" , ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 19:36:13 -0000 If you were to try zfs send blahblah... | ssh server sh -c 'cat > /tmp/zfs.test' And then on a server did a zfs recv < /tmp/zfs.test, does that work? That would narrow the problem down to ssh mangling the data or zfs recv not properly reading from the pipe, I think. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 04:47:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8E41DF68 for ; Fri, 8 Mar 2013 04:47:22 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0DE2B7C8 for ; Fri, 8 Mar 2013 04:47:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=Lzb1avJP38Vc3hPSeahYVTeVs2qDM3YpaZkhzpX3Y+k=; b=CHOKyZcVhT6XBRSD/5myDPwf1I4mmCMRneo8SBeGjJpFqYU2i0tJQxJOnKfeQ9Wign6aUK2inhFeZSmQEc3QFJ6CXtwwkvZZ5/lqZtKIvMVvgobuaHTs92Q5vRMQch3nlyq8rdu7AIlZVFUNT+XrG3q1zI2otVuDc/vh80jELWQ=; Received: from localhost.lerctr.org ([127.0.0.1]:48716 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDpD7-000D6r-RM; Thu, 07 Mar 2013 22:47:15 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 07 Mar 2013 22:47:12 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 07 Mar 2013 22:47:12 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> Message-ID: <1c4687bbf52352119abbc7d12334cef1@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 04:47:22 -0000 On 2013-03-07 13:08, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" > To: "Steven Hartland" > Cc: "Martin Simmons" ; > ; ; > > Sent: Thursday, March 07, 2013 1:14 AM > Subject: Re: zfs send/recv invalid data > > >> On 2013-03-06 17:48, Steven Hartland wrote: >>> ----- Original Message ----- From: "Larry Rosenman" >>> To: "Steven Hartland" >>> Cc: "Martin Simmons" ; >>> ; ; >>> >>> Sent: Wednesday, March 06, 2013 11:16 PM >>> Subject: Re: zfs send/recv invalid data >>> >>> >>>> On 2013-03-06 17:00, Steven Hartland wrote: >>>>> ----- Original Message ----- From: "Larry Rosenman" >>>>> >>>>>> # cat send.stream | ssh root@home openssl md5 >>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>>> # uname -a >>>>>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 >>>>>> r247820: Mon Mar 4 18:08:11 CST 2013 >>>>>> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER >>>>>> amd64 >>>>>> # >>>>>> # exec csh >>>>>> root@borg:/home/ler # pwd >>>>>> /home/ler >>>>>> root@borg:/home/ler # ssh tbh "cat >>>>>> /home/ler/public_html/ZFS_RECV/send.stream | ssh >>>>>> root@home.lerctr.org openssl md5" >>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>>> root@borg:/home/ler # uname -a >>>>>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 >>>>>> r247826: Mon Mar 4 19:59:08 CST 2013 >>>>>> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >>>>>> root@borg:/home/ler # >>>>> When you run a command from ssh it will run as the target users >>>>> shell >>>>> so for csh if there is something bad in .cshrc or iirc .profile >>>>> then >>>>> you may get something output purely by logging in. >>>>> So ensure you shell config's for the user on the target machine >>>>> are >>>>> clean. >>>>> You can confirm this by comparing the output from the following >>>>> two commands run from the sending machine:- >>>>> md5 send.stream >>>>> cat send.stream | ssh root@home.lerctr.org "cat" |md5 >>>>> If these two commands produce different md5's then the login >>>>> shell for root on home.lerctr.org is outputing something >>>>> even when there is no terminal. >>>>> You should be able to see what the shell is outputting using:- >>>>> echo -n '' | ssh root@home.lerctr.org "cat" > 1.log >>>>> After running this 1.log should be 0 bytes if its not your >>>>> shell is definitely your problem and I'd start by looking >>>>> at .cshrc and .profile for this user, or even the globals >>>>> for these in /etc/ >>>>> >>>> I've done that, and get identical md5's..... >>>> I'm at a loss. >>> I believe what you ran was similar but not identical, if you >>> check the quoting subtly different. >>> I'm not sure that will have any effect but both of the above >>> are worth running just to confirm. >>> Regards >>> Steve >> $ ^DConnection to thebighonker.lerctr.org closed. >> $ ssh root@tbh >> Last login: Tue Mar 5 06:51:46 2013 from cpe-72-182-19-1 >> FreeBSD 8.3-STABLE (THEBIGHONKER) #54 r247820: Mon Mar 4 18:08:11 >> CST 2013 >> Welcome to FreeBSD! >> root@thebighonker:~ # pwd >> /root >> root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send >> send-file-snap-list send.stream send.uname >> root@thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send.stream >> MD5 (/home/ler/public_html/ZFS_RECV/send.stream) = >> 9cd1d73ea8411f1c222bc90e7bea3d33 >> root@thebighonker:~ # cat /home/ler/public_html/ZFS_RECV/send.stream >> |ssh root@home.lerctr.org "cat"|md5 >> 9cd1d73ea8411f1c222bc90e7bea3d33 >> root@thebighonker:~ # >> >> looks like it's clean..... > > So to summarise, just to make sure I've got everything clear so far:- > 1. sending the stream over ssh from the source -> target results in > the same md5 (from the test above) yes > 2. sending the stream over ssh from source -> target piped into > zfs recv fails yes, sometimes now (I need to see what happens from cron(8) tonight) > 3. scp'ing the stream file from the source to the target and then > importing it via zfs recv works > yes > Yes / No? > > If so this is very strange indeed. > > I'd like to confirm that adding the zfs recv to the picture isn't > changing > the dynamic such that the stdin results are changing and thus > eliminating > ssh as the source of any corruption to do this can you test:- > md5 send.stream > cat send.stream | ssh root@home.lerctr.org "tee tea.stream | zfs recv > " > ssh root@home.lerctr.org "md5 tea.stream" > Will try this with tonights stream(s) if it fails. appreciate..... > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 19:09:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5A03DB2B for ; Fri, 8 Mar 2013 19:09:25 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 93E695E6 for ; Fri, 8 Mar 2013 19:09:24 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id r28J9IhY036146; Fri, 8 Mar 2013 13:09:18 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id r28J9HXT036145; Fri, 8 Mar 2013 13:09:17 -0600 (CST) (envelope-from brooks) Date: Fri, 8 Mar 2013 13:09:17 -0600 From: Brooks Davis To: Lars Engels Subject: Re: Argument list too long Message-ID: <20130308190917.GA34838@lor.one-eyed-alien.net> References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> <20130307161546.GV47829@e-new.0x20.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WIyZ46R2i8wDzkSu" Content-Disposition: inline In-Reply-To: <20130307161546.GV47829@e-new.0x20.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 19:09:25 -0000 --WIyZ46R2i8wDzkSu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 07, 2013 at 05:15:46PM +0100, Lars Engels wrote: > On Thu, Mar 07, 2013 at 05:50:36PM +0200, Vladislav Prodan wrote: > > Why 12K small files from one directory to cause problems? > >=20 > > # ll | wc -l > > 11467 > >=20 > > # grep X-PHP-Script * | more > > /sbin/grep: Argument list too long. > >=20 > > # egrep X-PHP-Script *.ua | more > > /usr/sbin/egrep: Argument list too long. > >=20 > > # cat *.ua | grep X-PHP-Script | more > > /sbin/cat: Argument list too long. > >=20 > >=20 > >=20 >=20 > Your shell can't process that many arguments. Use this: >=20 > grep -R "X-PHP-Script" . >=20 > or if you don't want to descent into subdirectories: >=20 > find . -type -f -name '*.ua' -maxdepth 1 -exec grep "X-PHP-Script" {} \+ This won't include file names and is gratuitously inefficient starting one grep per file. The command you're looking for is: find . -type -f -name '*.ua' -maxdepth 1 -print0 | xargs -0 grep -H "X-PHP-= Script" The find -print0 | xargs -0 allows filenames to contain spaces. The grep -H is mostly theoretical in that the last grep invocation by xargs couple only include one file and thus wouldn't include the filename. -- Brooks --WIyZ46R2i8wDzkSu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFROjdcXY6L6fI4GtQRAmEDAKCtwDnf44zqZ/GMFR7WnbupmqsK/gCfYY8O Jj0AsMtFo0inkCqLURQVBT4= =zhxl -----END PGP SIGNATURE----- --WIyZ46R2i8wDzkSu-- From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 20:05:50 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CB479615; Fri, 8 Mar 2013 20:05:50 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id 803CF7DC; Fri, 8 Mar 2013 20:05:50 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.6/8.14.6) with ESMTP id r28K5fAF089779; Fri, 8 Mar 2013 13:05:41 -0700 (MST) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.6/8.14.6/Submit) with ESMTP id r28K5fvX089776; Fri, 8 Mar 2013 13:05:41 -0700 (MST) (envelope-from wblock@wonkity.com) Date: Fri, 8 Mar 2013 13:05:41 -0700 (MST) From: Warren Block To: Brooks Davis Subject: Re: Argument list too long In-Reply-To: <20130308190917.GA34838@lor.one-eyed-alien.net> Message-ID: References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> <20130307161546.GV47829@e-new.0x20.net> <20130308190917.GA34838@lor.one-eyed-alien.net> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (wonkity.com [127.0.0.1]); Fri, 08 Mar 2013 13:05:41 -0700 (MST) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 20:05:50 -0000 On Fri, 8 Mar 2013, Brooks Davis wrote: > On Thu, Mar 07, 2013 at 05:15:46PM +0100, Lars Engels wrote: >> On Thu, Mar 07, 2013 at 05:50:36PM +0200, Vladislav Prodan wrote: >>> Why 12K small files from one directory to cause problems? >>> >>> # ll | wc -l >>> 11467 >>> >>> # grep X-PHP-Script * | more >>> /sbin/grep: Argument list too long. >>> >>> # egrep X-PHP-Script *.ua | more >>> /usr/sbin/egrep: Argument list too long. >>> >>> # cat *.ua | grep X-PHP-Script | more >>> /sbin/cat: Argument list too long. >>> >>> >>> >> >> Your shell can't process that many arguments. Use this: >> >> grep -R "X-PHP-Script" . >> >> or if you don't want to descent into subdirectories: >> >> find . -type -f -name '*.ua' -maxdepth 1 -exec grep "X-PHP-Script" {} \+ There is a typo, should be "-type f". > This won't include file names and is gratuitously inefficient starting one > grep per file. But the final \+ means "{} is replaced with as many pathnames as possible for each invocation of utility. This behaviour is similar to that of xargs(1)." > The command you're looking for is: > > find . -type -f -name '*.ua' -maxdepth 1 -print0 | xargs -0 grep -H "X-PHP-Script" > > The find -print0 | xargs -0 allows filenames to contain spaces. The > grep -H is mostly theoretical in that the last grep invocation by xargs > couple only include one file and thus wouldn't include the filename. For fun, after running each of these several times to preload cache: find /usr/ports -type f -print0 | xargs -0 grep -H "X-PHP-Script" 40.98 seconds (average) find /usr/ports -type f -exec grep -H "X-PHP-Script" {} \+ 42.27 seconds (average) So they aren't too different in performance. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 21:40:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 14CEF381 for ; Fri, 8 Mar 2013 21:40:44 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id D318EB7C for ; Fri, 8 Mar 2013 21:40:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=TqDjB+GZjaRGzC42pY0v3mY27GDCcidvkxsfRxB1328=; b=kITFws4YZfhpW99lpLI+zGFCP9KUQ2kmYjKMCHTaIX8NHwFkWQ5Zwdzor+JghqJz1IsBeY76UIWTkOZ65ejDOXupqf6fNIivdoBXlga1b12PEnKXh8yJbDkHQFVVIwDkZnwggthRd+2bD89AM2hIxX34IBKG3Albxn/Ztfn+i4o=; Received: from localhost.lerctr.org ([127.0.0.1]:18104 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UE51o-000M3g-PS; Fri, 08 Mar 2013 15:40:37 -0600 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 08 Mar 2013 15:40:36 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 08 Mar 2013 15:40:36 -0600 From: Larry Rosenman To: Steven Hartland Subject: Re: zfs send/recv invalid data In-Reply-To: <1c4687bbf52352119abbc7d12334cef1@webmail.lerctr.org> References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> <1c4687bbf52352119abbc7d12334cef1@webmail.lerctr.org> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org, ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 21:40:44 -0000 On 2013-03-07 22:47, Larry Rosenman wrote: > Will try this with tonights stream(s) if it fails. > > appreciate..... [trimmed copied text] I worked fine last night from cron(8). Will keep an eye on it. I hate intermittent / unpredictable failures with a passion :) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 21:43:00 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B2F8C412; Fri, 8 Mar 2013 21:43:00 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ea0-x230.google.com (mail-ea0-x230.google.com [IPv6:2a00:1450:4013:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 1DA2FBDD; Fri, 8 Mar 2013 21:42:59 +0000 (UTC) Received: by mail-ea0-f176.google.com with SMTP id h10so394284eaj.21 for ; Fri, 08 Mar 2013 13:42:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=0J4rT2dEYWbhS/LXvzR+ZsbgYDeDcfxLZzZtFhKwdOM=; b=YCL1U0alTNahG+VaWqaz9XJx32Ae1YtK9aX1Rl2VtBR5c5nBOUiKZyrQ1N/VLsfRpf kSk50XKTE4gs1SJMmrTTRToMEUKAh8IFPaXs7WTV5SjSBk8ueM1WAd/tjFb6foBBvsao EYjlAuFm7Oup6RsxFSkPLzVTVi5JkcBd8JxRAl+BCjvT9vhUGrmgnrHnJatW5s1Q1VkQ 9FP7gyeRv8xdZ7ekkFTCytgA37bk7+Emfmnfx1ST46f1DWlPAPNnhz7eoec31bGUhkQQ mCMoSMRzRCCBrbPRHtb/jY6WV0KokhLhEAWiOxbAu+ccpT4YbMYDd0WazMIqz3rRevSu D0vw== MIME-Version: 1.0 X-Received: by 10.15.34.198 with SMTP id e46mr9917891eev.27.1362778979183; Fri, 08 Mar 2013 13:42:59 -0800 (PST) Sender: utisoft@gmail.com Received: by 10.14.124.7 with HTTP; Fri, 8 Mar 2013 13:42:58 -0800 (PST) Received: by 10.14.124.7 with HTTP; Fri, 8 Mar 2013 13:42:58 -0800 (PST) In-Reply-To: References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> <20130307161546.GV47829@e-new.0x20.net> <20130308190917.GA34838@lor.one-eyed-alien.net> Date: Fri, 8 Mar 2013 21:42:58 +0000 X-Google-Sender-Auth: ifVkpIn8vzp_yEXhapjIayVn0uA Message-ID: Subject: Re: Argument list too long From: Chris Rees To: Warren Block Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, Brooks Davis X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 21:43:00 -0000 On 8 Mar 2013 20:05, "Warren Block" wrote: > > On Fri, 8 Mar 2013, Brooks Davis wrote: > >> On Thu, Mar 07, 2013 at 05:15:46PM +0100, Lars Engels wrote: >>> >>> On Thu, Mar 07, 2013 at 05:50:36PM +0200, Vladislav Prodan wrote: >>>> >>>> Why 12K small files from one directory to cause problems? >>>> >>>> # ll | wc -l >>>> 11467 >>>> >>>> # grep X-PHP-Script * | more >>>> /sbin/grep: Argument list too long. >>>> >>>> # egrep X-PHP-Script *.ua | more >>>> /usr/sbin/egrep: Argument list too long. >>>> >>>> # cat *.ua | grep X-PHP-Script | more >>>> /sbin/cat: Argument list too long. >>>> >>>> >>>> >>> >>> Your shell can't process that many arguments. Use this: >>> >>> grep -R "X-PHP-Script" . >>> >>> or if you don't want to descent into subdirectories: >>> >>> find . -type -f -name '*.ua' -maxdepth 1 -exec grep "X-PHP-Script" {} \+ > > > There is a typo, should be "-type f". > > >> This won't include file names and is gratuitously inefficient starting one >> grep per file. > > > But the final \+ means "{} is replaced with as many pathnames as possible for each invocation of utility. This behaviour is similar to that of xargs(1)." > > >> The command you're looking for is: >> >> find . -type -f -name '*.ua' -maxdepth 1 -print0 | xargs -0 grep -H "X-PHP-Script" >> >> The find -print0 | xargs -0 allows filenames to contain spaces. The >> grep -H is mostly theoretical in that the last grep invocation by xargs >> couple only include one file and thus wouldn't include the filename. > > > For fun, after running each of these several times to preload cache: > > find /usr/ports -type f -print0 | xargs -0 grep -H "X-PHP-Script" > 40.98 seconds (average) > > find /usr/ports -type f -exec grep -H "X-PHP-Script" {} \+ > 42.27 seconds (average) > > So they aren't too different in performance. The \+ form I have also found impossible to achieve in csh, which is why I never recommend it. Chris (Rees not Ross :) From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 22:32:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 14E081AA for ; Fri, 8 Mar 2013 22:32:09 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-vc0-f179.google.com (mail-vc0-f179.google.com [209.85.220.179]) by mx1.freebsd.org (Postfix) with ESMTP id A6CF2DA7 for ; Fri, 8 Mar 2013 22:32:08 +0000 (UTC) Received: by mail-vc0-f179.google.com with SMTP id k1so1160572vck.38 for ; Fri, 08 Mar 2013 14:32:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=ubWN/P63Fd8Ejl1qUj+Pol0mukJK1C1tlQGrGd8IwCs=; b=k2Jl0oj/xu5MpLH6IEugc3eS63vp+bfxEQe+gSBMjXtpVLXNKCc7+/Dy0ezRi1Eu0H PIqlx0KZvh03KXsKc5uWW/ek/OOm7NT79rvEpk3mdpg6sKz4jGEw9hmu7+9EZBY87KbL GMD2wcwRVs8vg5TvBRfGc/odpKtT2Rol1KNLwEwEC/mzb0U9vlEqyuBht3PxI3cyQmpY e1S9KF8N2d8/e2Jh88KnyMuQRQyEQfLcvdZlWqvgPA0rvp/vnBCPjf4g7Kn/TsZbQwG0 PKpbfP/Q1o/xUyXZxjZZARfcqvFFfG/AbV3J9Ak1BN3TgyPy+gDB4h4qWEJDd9jc1ssW GTcA== MIME-Version: 1.0 X-Received: by 10.220.153.143 with SMTP id k15mr1637848vcw.33.1362781927707; Fri, 08 Mar 2013 14:32:07 -0800 (PST) Sender: artemb@gmail.com Received: by 10.220.108.3 with HTTP; Fri, 8 Mar 2013 14:32:07 -0800 (PST) In-Reply-To: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> <218B0537E987442EAB8027EA478F8BE9@multiplay.co.uk> <5378B8D5A65A4F19BE1CF6C25A9CFE22@multiplay.co.uk> <2DCDD0136DE7498E9DA95D5CDA3F60CC@multiplay.co.uk> <1c4687bbf52352119abbc7d12334cef1@webmail.lerctr.org> Date: Fri, 8 Mar 2013 14:32:07 -0800 X-Google-Sender-Auth: FjH5F2x4q2OIzroyOLETLX_85ys Message-ID: Subject: Re: zfs send/recv invalid data From: Artem Belevich To: Larry Rosenman Content-Type: text/plain; charset=ISO-8859-1 Cc: Tom Evans , freebsd-fs , ronald-freebsd8@klop.yi.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 22:32:09 -0000 On Fri, Mar 8, 2013 at 1:40 PM, Larry Rosenman wrote: > On 2013-03-07 22:47, Larry Rosenman wrote: >> >> Will try this with tonights stream(s) if it fails. >> >> appreciate..... > > [trimmed copied text] > > I worked fine last night from cron(8). > > Will keep an eye on it. > > I hate intermittent / unpredictable failures with a passion :) Do you use ECC memory? If not, it may be a good idea to test memory on both machines if you didn't do it already. If you do use ECC, grep your logs for 'MCA'. If there are correctable errors, they would typically be reported via machine check exceptions and logged. YMMV. I was unable to make my Asus board (P5BV/SAS) report anything at all even though in one experiment I had physically taped off one data bit on one DIMM -- system still boots and works, but I get no exceptions. Supermicro board I had at work (X8-something) did report correctable ECC errors when one DIMM went marginally bad. --Artem From owner-freebsd-fs@FreeBSD.ORG Fri Mar 8 23:21:58 2013 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8C80FC7B; Fri, 8 Mar 2013 23:21:58 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 5B64CF12; Fri, 8 Mar 2013 23:21:57 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id r28NLuMG049746; Fri, 8 Mar 2013 16:21:56 -0700 (MST) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id r28NLt1c049745; Fri, 8 Mar 2013 16:21:55 -0700 (MST) (envelope-from ken) Date: Fri, 8 Mar 2013 16:21:55 -0700 From: "Kenneth D. Merry" To: Bruce Evans Subject: Re: patches to add new stat(2) file flags Message-ID: <20130308232155.GA47062@nargothrond.kdm.org> References: <20130307000533.GA38950@nargothrond.kdm.org> <20130307222553.P981@besplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Q68bSM7Ycu6FN28Q" Content-Disposition: inline In-Reply-To: <20130307222553.P981@besplex.bde.org> User-Agent: Mutt/1.4.2i Cc: arch@FreeBSD.org, fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 23:21:58 -0000 --Q68bSM7Ycu6FN28Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, Mar 08, 2013 at 00:37:15 +1100, Bruce Evans wrote: > On Wed, 6 Mar 2013, Kenneth D. Merry wrote: > > >I have attached diffs against head for some additional stat(2) file flags. > > > >The primary purpose of these flags is to improve compatibility with CIFS, > >both from the client and the server side. > >... > > I missed looking at the diffs in my previous reply. > > % --- //depot/users/kenm/FreeBSD-test3/bin/chflags/chflags.1 2013-03-04 > 17:51:12.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 > 2013-03-04 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.86 2013-03-06 16:42:43.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 > 2013-03-06 14:47:25.987128763 -0700 > % @@ -117,6 +117,16 @@ > % set the user immutable flag (owner or super-user only) > % .It Cm uunlnk , uunlink > % set the user undeletable flag (owner or super-user only) > % +.It Cm system , usystem > % +set the Windows system flag (owner or super-user only) > > This begins unsorting of the list. Fixed. > It's not just a Windows flag, since it also works in DOS. Fixed. > "Owner or" is too strict for msdosfs, since files can only have a > single owner so it is controlling access using groups is needed. I > use owner root and group msdosfs for msdosfs mounts. This works for > normal operations like open/read/write, but fails for most attributes > including file flags. msdosfs doesn't support many attributes but > this change is supposed to add support for 3 new file flags so it would > be good if it didn't restrict the support to root. I wasn't trying to change the existing security model for msdosfs, but if you've got a suggested patch to fix it I can add that in. > % +.It Cm sparse , usparse > % +set the sparse file attribute (owner or super-user only) > % +.It Cm offline , uoffline > % +set the offline file attribute (owner or super-user only) > % +.It Cm reparse , ureparse > % +set the Windows reparse point file attribute (owner or super-user only) > % +.It Cm hidden , uhidden > % +set the hidden file attribute (owner or super-user only) > > The additions at the end are also internally unsorted. Fixed. > Previously only "opaque" and "nodump" were unsorted. They are UF flags > sorted with the SF flags, and "no" in "nodump" is not ignored for the > purposes of sorting. > > Not having "u" in the old and new UF flag names messes up the sort order > (unless you add futher confusion by ignoring "u" for the purposes of > sorting) and makes it harder to add SF variants of the flags. They're now sorted in alphabetical order. > % .El > % .Pp > % Putting the letters > % --- //depot/users/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > 2013-03-04 17:51:12.000000000 -0700 > % +++ > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > 2013-03-04 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.178 2013-03-06 16:42:43.000000000 -0700 > % +++ > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > 2013-03-06 15:09:32.842437917 -0700 > % @@ -68,7 +68,17 @@ > % { "nodump", 1, UF_NODUMP }, > % { "noopaque", 0, UF_OPAQUE }, > % { "nouunlnk", 0, UF_NOUNLINK }, > % - { "nouunlink", 0, UF_NOUNLINK } > % + { "nouunlink", 0, UF_NOUNLINK }, > % + { "nosystem", 0, UF_SYSTEM, }, > % + { "nousystem", 0, UF_SYSTEM, }, > % + { "nosparse", 0, UF_SPARSE, }, > % + { "nousparse", 0, UF_SPARSE, }, > % + { "nooffline", 0, UF_OFFLINE, }, > % + { "nouoffline", 0, UF_OFFLINE, }, > % + { "noreparse", 0, UF_REPARSE, }, > % + { "noureparse", 0, UF_REPARSE, }, > % + { "nohidden", 0, UF_HIDDEN, }, > % + { "nouhidden", 0, UF_HIDDEN, } > > This is totally disordered too. > > The old table was sorted except for "nosnapshot". Another bug is that > "nosnapshot" is supported here (so chflags(1) can show it), but is not > documented in chflags(1). Actually, I think the table was previously sorted by the stat(2) flag name, and UF_NOUNLINK appears to be the only one that was out of place. I have fixed that and my additions. > % }; > % #define nmappings (sizeof(mapping) / sizeof(mapping[0])) > % > % --- //depot/users/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 2013-03-04 > 17:51:12.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 > 2013-03-04 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.257 2013-03-06 16:42:43.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 > 2013-03-06 14:49:39.357125717 -0700 > % @@ -75,16 +75,49 @@ > % Do not dump the file. > % .It Dv UF_IMMUTABLE > % The file may not be changed. > % +Filesystems may use this flag to maintain compatibility with the Windows > and > % +CIFS FILE_ATTRIBUTE_READONLY attribute. > % .It Dv UF_APPEND > % The file may only be appended to. > > This was already totally disordered. It even has UF's before SF's, although > SF's sort before UF's both lexically and logically, though not in sys/stat.h > or in bits. > > The disorder here was apparently copied from the implementation > (sys/stat.h, which is in the order of the bits, which is historical > for binary compatibility) and not cleaned up like chflags(1) and > strttoflags(3). Fixed. > % .It Dv UF_NOUNLINK > % The file may not be renamed or deleted. > % .It Dv UF_OPAQUE > % The directory is opaque when viewed through a union stack. > % +.It Dv UF_SYSTEM > % +The file has the Windows and CIFS FILE_ATTRIBUTE_SYSTEM attribute. > % +Filesystems in FreeBSD may store and display this flag, but do not > provide > % +any special handling when it is set. > > More disordered than before... > > % +.It Dv UF_HIDDEN > % +The file may be hidden from directory listings at the application's > % +discretion. > % +The file has the Windows FILE_ATTRIBUTE_HIDDEN attribute. > > Not just Windows... Fixed. > % .It Dv SF_ARCHIVED > % -The file may be archived. > % +The file has been archived. > % +This flag means the opposite of the Windows and CIFS > FILE_ATTRIBUTE_ARCHIVE > % +attribute. > % +That attribute means that the file should be archived, whereas > % +.Dv SF_ARCHIVED > % +means that the file has been archived. > > WinXP uses the poor wording "avialable for archiving". This is better. > > % +Filesystems in FreeBSD may or may not have special handling for this > flag. > % +For instance, ZFS tracks changes to files and will clear this bit when a > % +file is updated. > % +UFS only stores the flag, and relies on the application to change it when > % +needed. > > I think that is useless, since changing it is needed whenever the file > changes, and applications can do that (short of running as daemons and > watching for changes). Do you mean applications can't do that or can? > WinXP seems to not set the flag when attributes change. I think that is > a bug, and FreeBSD msdosfs does set it when attributes are changed (except > for the SF_ARCHIVED attribute itself, and atimes when the atimes are set > automatically). The FreeBSD behaviour is like setting the ctime for any > change and is bug for bug compatible in doing this even for null changes. > FreeBSD doesn't have many attributes to set, but you just added some more. > I think this should be controlled by a mount option so that bug for bug > compatibility with WinDOS is possible. > > % .It Dv SF_IMMUTABLE > % The file may not be changed. > % +This flag also indicates that the Windows ans CIFS > FILE_ATTRIBUTE_READONLY > % +attribute is set. > > s/ans/and/ > > You only mentioned UF_IMMUTABLE in your general description. Mapping > READONLY to both gives even more confusing semantics, and mapping it to > SF_IMMUTABLE seems too strict since for example it prevents using FreeBSD > to manage the flag at high securelevels. I agree. That man page change was left over from an earlier version of the code. I took it out. > % --- //depot/users/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > 2013-03-04 17:51:12.000000000 -0700 > % +++ > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > 2013-03-04 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.370 2013-03-06 16:42:43.000000000 -0700 > % +++ > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > 2013-03-06 14:49:47.179130318 -0700 > % @@ -345,8 +345,17 @@ > % vap->va_birthtime.tv_nsec = 0; > % } > % vap->va_flags = 0; > % + /* > % + * The DOS Archive attribute means that a file needs to be > % + * archived. The BSD SF_ARCHIVED attribute means that a file has > % + * been archived. Thus the inversion here. > % + */ > > No need to document it again. It goes without saying that ARCHIVE > != ARCHIVED. I disagree. It wasn't immediately obvious to me that SF_ARCHIVED was generally used as the inverse of the DOS Archived bit until I started digging into this. If this helps anyone figure that out more quickly, it's useful. > % if ((dep->de_Attributes & ATTR_ARCHIVE) == 0) > % vap->va_flags |= SF_ARCHIVED; > % + if (dep->de_Attributes & ATTR_HIDDEN) > % + vap->va_flags |= UF_HIDDEN; > % + if (dep->de_Attributes & ATTR_SYSTEM) > % + vap->va_flags |= UF_SYSTEM; > % vap->va_gen = 0; > % vap->va_blocksize = pmp->pm_bpcluster; > % vap->va_bytes = > % @@ -420,12 +429,21 @@ > % if (error) > % return (error); > % } > > The permissions check before this is delicate and was broken and is > more broken now. It is still short-circuit to handle setting the > single flag that used to be supported, and is slightly broken for that: > - unprivileged user asks to set ARCHIVE by passing !SF_ARCHIVED. We > allow that, although this may toggle the flag and normal semantics > for SF flags is to not allow toggling. > - unprivileged user asks to clear ARCHIVE by passing SF_ARCHIVED. We > don't allow that. But we should allow preserving ARCHIVE if it is > already clear. > The bug wasn't very important when only 1 flag was supported. Now it > prevents unprivileged users managing the new UF flags if ARCHIVE is > clear. Fortunately, this is the unusual case. Anyway, unprivileged > users can set ARCHIVE by doing some other operation. Even the chflags() > operation should set ARCHIVE and thus allow further chflags()'s that now > keep ARCHIVE set. Except it is very confusing if a chflags() asks for > ARCHIVE to be clear. This request might be just to try to preserve > the current setting and not want it if other things are changed, or > it might be to purposely clear it. Changing it from set to clear should > still be privileged. I changed it to allow setting or clearing SF_ARCHIVED. Now I can set or clear the flag as non-root: [root@doc-sd /testpool]# mount_msdosfs -u operator -g operator /dev/md0 /mnt [root@doc-sd /testpool]# cd /mnt [root@doc-sd /mnt]# ls -lao total 20 drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. -rwxr-xr-x 1 operator operator - 0 Mar 8 16:54 foo [root@doc-sd /mnt]# su operator [operator@doc-sd /mnt]$ chflags arch foo [operator@doc-sd /mnt]$ ls -lao total 20 drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. -rwxr-xr-x 1 operator operator arch 0 Mar 8 16:54 foo [operator@doc-sd /mnt]$ chflags 0 foo [operator@doc-sd /mnt]$ ls -lao total 20 drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. -rwxr-xr-x 1 operator operator - 0 Mar 8 16:54 foo > See the more complicated permissions check in ffs. It would be safest > to duplicate most of it, to get different permissions checking for the > SF and UF flags. Then decide if we want to keep allowing setting > ARCHIVE without privilege. I think we should allow getting and setting SF_ARCHIVED without special privileges. Given how it is generally used, I don't think it should be restricted to the super-user. Can you provide some code demonstrating how the permissions code should be changed in msdosfs? I don't know that much about that sort of thing, so I'll probably spend an inordinate amount of time stumbling through it. > % - if (vap->va_flags & ~SF_ARCHIVED) > % + if (vap->va_flags & ~(SF_ARCHIVED|UF_HIDDEN|UF_SYSTEM)) > > Style bugs (missing spaces around binary operator '|'). These style > bugs are common in atribute handling for other fs's but were not here. Fixed. > % return EOPNOTSUPP; > % if (vap->va_flags & SF_ARCHIVED) > % dep->de_Attributes &= ~ATTR_ARCHIVE; > % else if (!(dep->de_Attributes & ATTR_DIRECTORY)) > % dep->de_Attributes |= ATTR_ARCHIVE; > > The comment before this says that we ignore attmps to set ATTR_ARCHIVED > for directories. However, it is out of date. WinXP allows setting it > and all the new flags for directories, and so do we. Do you mean we allow setting it in UFS, or where? Obviously the code above won't set it on a directory. > % + if (vap->va_flags & UF_HIDDEN) > % + dep->de_Attributes |= ATTR_HIDDEN; > % + else > % + dep->de_Attributes &= ~ATTR_HIDDEN; > % + if (vap->va_flags & UF_SYSTEM) > % + dep->de_Attributes |= ATTR_SYSTEM; > % + else > % + dep->de_Attributes &= ~ATTR_SYSTEM; > > I thought you were adding ATTR_READONLY -> UF_IMMUTABLE here. No, I said in the commit message that it could be extended to map ATTR_READONLY to UF_IMMUTABLE, not that I actually did that. I didn't make the change because of the existing code to map readonly to standard Unix permissions. > The WinXP attrib command (at least on a FAT32 fs) doesn't allow setting > or clearing ARCHIVE (even if it is already set or clear) if any of > HIDDEN, READONLY or SYSTEM is already set and remains set after the > command. Thus the HRS attributes act a bit like immutable flags, but > subtly differently. (ffs has the quite different and worse behaviour > of allowing chflags() irrespective of immutable flags being set before > or after, provided there is enough privilege to change the immutable > flags.) Anyway, they should all give some aspects of immutability. We could do that for msdosfs, but why add more things for the user to trip over given how the filesystem is typically used? Most people probably use it for USB thumb drives these days. Or perahps on a dual boot system to access their Windows partition. > % + > > Style bug (extra blank line). Fixed. > % dep->de_flag |= DE_MODIFIED; > % } > % > % --- //depot/users/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 > 17:51:12.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 > 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.563 2013-03-06 16:42:43.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-06 > 15:05:45.936126193 -0700 > % @@ -268,6 +268,22 @@ > % #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ > % #define UF_NOUNLINK 0x00000010 /* file may not be removed or > renamed */ > > Old style bugs: inconsistent space instead of tab after #define for the 2 > newer definitions. Fixed. > % /* > % + * These two bits are defined in MacOS X. They are not currently used in > % + * FreeBSD. > % + */ > % +#if 0 > % +#define UF_COMPRESSED 0x00000020 /* file is compressed */ > % +#define UF_TRACKED 0x00000040 /* renames and deletes are > tracked */ > % +#endif > % + > % +#define UF_SYSTEM 0x00000080 /* Windows system file bit */ > % +#define UF_SPARSE 0x00000100 /* sparse file */ > % +#define UF_OFFLINE 0x00000200 /* file is offline */ > % +#define UF_REPARSE 0x00000400 /* Windows reparse point file bit */ > % +/* This is the same as the MacOS X definition of UF_HIDDEN */ > % +#define UF_HIDDEN 0x00008000 /* file is hidden */ > > New style bugs: random spaces/tabs after #define. 1 main comment not > punctuated. Fixed. > % + > % +/* > % * Super-user changeable flags. > % */ > % #define SF_SETTABLE 0xffff0000 /* mask of superuser > changeable flags */ > % --- //depot/users/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c 2013-03-04 > 17:51:12.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c > 2013-03-04 17:51:12.000000000 -0700 > % --- /tmp/tmp.49594.581 2013-03-06 16:42:43.000000000 -0700 > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c > 2013-03-06 15:06:27.004132409 -0700 > % @@ -529,8 +529,9 @@ > % } > % if (vap->va_flags != VNOVAL) { > % if ((vap->va_flags & ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | > % - UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | > % - SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) > % + UF_OPAQUE | UF_NOUNLINK | UF_HIDDEN | UF_SYSTEM | > % + UF_SPARSE | UF_OFFLINE | UF_REPARSE | SF_ARCHIVED | > % + SF_IMMUTABLE | SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) > != 0) > % return (EOPNOTSUPP); > % if (vp->v_mount->mnt_flag & MNT_RDONLY) > % return (EROFS); > > The random order is harder to read with more flags. Fixed. > I think unsupported flags should just be unsupported -- most or all of the > new flags and SF_ARCHIVED too. Most need system support to work. Without > system support you can only copy them from other fs's and then lose them > when utilities like tar don't unsderstand them. The primary reason I added these flags to UFS is to support CIFS servers. I have modifications to Likewise to support storing DOS/Windows/CIFS attributes as file flags, and we could do the same for Samba. My eventual intent is to also add support to our NFS code to get/set the attributes that are allowed in the NFS 4.1 standard (hidden, system and archive are the ones I could find). So in this case, the filesystem role for some of these really is just to store it for someone else. That is the way that ZFS handles attributes like hidden and system. It doesn't do anything with them, they're just used by the Solaris CIFS server. Ken -- Kenneth Merry ken@FreeBSD.ORG --Q68bSM7Ycu6FN28Q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="file_flags_head.20130308.1.txt" *** src/bin/chflags/chflags.1.orig --- src/bin/chflags/chflags.1 *************** *** 101,120 **** .Bl -tag -offset indent -width ".Cm opaque" .It Cm arch , archived set the archived flag (super-user only) .It Cm opaque set the opaque flag (owner or super-user only) - .It Cm nodump - set the nodump flag (owner or super-user only) .It Cm sappnd , sappend set the system append-only flag (super-user only) .It Cm schg , schange , simmutable set the system immutable flag (super-user only) .It Cm sunlnk , sunlink set the system undeletable flag (super-user only) .It Cm uappnd , uappend set the user append-only flag (owner or super-user only) .It Cm uchg , uchange , uimmutable set the user immutable flag (owner or super-user only) .It Cm uunlnk , uunlink set the user undeletable flag (owner or super-user only) .El --- 101,132 ---- .Bl -tag -offset indent -width ".Cm opaque" .It Cm arch , archived set the archived flag (super-user only) + .It Cm nodump + set the nodump flag (owner or super-user only) .It Cm opaque set the opaque flag (owner or super-user only) .It Cm sappnd , sappend set the system append-only flag (super-user only) .It Cm schg , schange , simmutable set the system immutable flag (super-user only) + .It Cm snapshot + set the snapshot flag (most filesystems do not allow changing this flag) .It Cm sunlnk , sunlink set the system undeletable flag (super-user only) .It Cm uappnd , uappend set the user append-only flag (owner or super-user only) .It Cm uchg , uchange , uimmutable set the user immutable flag (owner or super-user only) + .It Cm uhidden , hidden + set the hidden file attribute (owner or super-user only) + .It Cm uoffline , offline + set the offline file attribute (owner or super-user only) + .It Cm usparse , sparse + set the sparse file attribute (owner or super-user only) + .It Cm usystem , system + set the DOS and Windows system flag (owner or super-user only) + .It Cm ureparse , reparse + set the Windows reparse point file attribute (owner or super-user only) .It Cm uunlnk , uunlink set the user undeletable flag (owner or super-user only) .El *** src/lib/libc/gen/strtofflags.c.orig --- src/lib/libc/gen/strtofflags.c *************** *** 62,74 **** #endif { "nouappnd", 0, UF_APPEND }, { "nouappend", 0, UF_APPEND }, { "nouchg", 0, UF_IMMUTABLE }, { "nouchange", 0, UF_IMMUTABLE }, { "nouimmutable", 0, UF_IMMUTABLE }, { "nodump", 1, UF_NODUMP }, { "noopaque", 0, UF_OPAQUE }, ! { "nouunlnk", 0, UF_NOUNLINK }, ! { "nouunlink", 0, UF_NOUNLINK } }; #define nmappings (sizeof(mapping) / sizeof(mapping[0])) --- 62,84 ---- #endif { "nouappnd", 0, UF_APPEND }, { "nouappend", 0, UF_APPEND }, + { "nohidden", 0, UF_HIDDEN, }, + { "nouhidden", 0, UF_HIDDEN, }, { "nouchg", 0, UF_IMMUTABLE }, { "nouchange", 0, UF_IMMUTABLE }, { "nouimmutable", 0, UF_IMMUTABLE }, { "nodump", 1, UF_NODUMP }, + { "nouunlnk", 0, UF_NOUNLINK }, + { "nouunlink", 0, UF_NOUNLINK }, + { "nooffline", 0, UF_OFFLINE, }, + { "nouoffline", 0, UF_OFFLINE, }, { "noopaque", 0, UF_OPAQUE }, ! { "noreparse", 0, UF_REPARSE, }, ! { "noureparse", 0, UF_REPARSE, }, ! { "nosparse", 0, UF_SPARSE, }, ! { "nousparse", 0, UF_SPARSE, }, ! { "nosystem", 0, UF_SYSTEM, }, ! { "nousystem", 0, UF_SYSTEM, } }; #define nmappings (sizeof(mapping) / sizeof(mapping[0])) *** src/lib/libc/sys/chflags.2.orig --- src/lib/libc/sys/chflags.2 *************** *** 71,96 **** the following values .Pp .Bl -tag -width ".Dv SF_IMMUTABLE" -compact -offset indent ! .It Dv UF_NODUMP ! Do not dump the file. ! .It Dv UF_IMMUTABLE ! The file may not be changed. ! .It Dv UF_APPEND The file may only be appended to. - .It Dv UF_NOUNLINK - The file may not be renamed or deleted. - .It Dv UF_OPAQUE - The directory is opaque when viewed through a union stack. .It Dv SF_ARCHIVED ! The file may be archived. .It Dv SF_IMMUTABLE The file may not be changed. - .It Dv SF_APPEND - The file may only be appended to. .It Dv SF_NOUNLINK The file may not be renamed or deleted. .It Dv SF_SNAPSHOT The file is a snapshot file. .El .Pp If one of --- 71,127 ---- the following values .Pp .Bl -tag -width ".Dv SF_IMMUTABLE" -compact -offset indent ! .It Dv SF_APPEND The file may only be appended to. .It Dv SF_ARCHIVED ! The file has been archived. ! This flag means the opposite of the Windows and CIFS FILE_ATTRIBUTE_ARCHIVE ! attribute. ! That attribute means that the file should be archived, whereas ! .Dv SF_ARCHIVED ! means that the file has been archived. ! Filesystems in FreeBSD may or may not have special handling for this flag. ! For instance, ZFS tracks changes to files and will clear this bit when a ! file is updated. ! UFS only stores the flag, and relies on the application to change it when ! needed. .It Dv SF_IMMUTABLE The file may not be changed. .It Dv SF_NOUNLINK The file may not be renamed or deleted. .It Dv SF_SNAPSHOT The file is a snapshot file. + .It Dv UF_APPEND + The file may only be appended to. + .It Dv UF_HIDDEN + The file may be hidden from directory listings at the application's + discretion. + The file has the DOS and Windows FILE_ATTRIBUTE_HIDDEN attribute. + .It Dv UF_IMMUTABLE + The file may not be changed. + Filesystems may use this flag to maintain compatibility with the Windows and + CIFS FILE_ATTRIBUTE_READONLY attribute. + .It Dv UF_NODUMP + Do not dump the file. + .It Dv UF_NOUNLINK + The file may not be renamed or deleted. + .It Dv UF_OFFLINE + The file is offline, or has the Windows and CIFS FILE_ATTRIBUTE_OFFLINE + attribute. + Filesystems in FreeBSD store and display this flag, but do not provide any + special handling when it is set. + .It Dv UF_OPAQUE + The directory is opaque when viewed through a union stack. + .It Dv UF_REPARSE + The file contains a Windows reparse point and has the Windows and CIFS + FILE_ATTRIBUTE_REPARSE_POINT attribute. + .It Dv UF_SPARSE + The file has the Windows FILE_ATTRIBUTE_SPARSE_FILE attribute. + This may also be used by a filesystem to indicate a sparse file. + .It Dv UF_SYSTEM + The file has the Windows and CIFS FILE_ATTRIBUTE_SYSTEM attribute. + Filesystems in FreeBSD may store and display this flag, but do not provide + any special handling when it is set. .El .Pp If one of *************** *** 121,126 **** --- 152,164 ---- .Xr init 8 for details.) .Pp + The implementation of all flags is filesystem-dependent. + See the description of the + .Dv SF_ARCHIVED + flag above for one example of the differences in behavior. + Care should be exercised when writing applications to account for + support or lack of support of these flags in various filesystems. + .Pp The .Dv SF_SNAPSHOT flag is maintained by the system and cannot be toggled. *** src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c.orig --- src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c *************** *** 6057,6079 **** XVA_SET_REQ(&xvap, XAT_APPENDONLY); XVA_SET_REQ(&xvap, XAT_NOUNLINK); XVA_SET_REQ(&xvap, XAT_NODUMP); error = zfs_getattr(ap->a_vp, (vattr_t *)&xvap, 0, ap->a_cred, NULL); if (error != 0) return (error); /* Convert ZFS xattr into chflags. */ ! #define FLAG_CHECK(fflag, xflag, xfield) do { \ ! if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) \ fflags |= (fflag); \ } while (0) FLAG_CHECK(SF_IMMUTABLE, XAT_IMMUTABLE, ! xvap.xva_xoptattrs.xoa_immutable); FLAG_CHECK(SF_APPEND, XAT_APPENDONLY, ! xvap.xva_xoptattrs.xoa_appendonly); FLAG_CHECK(SF_NOUNLINK, XAT_NOUNLINK, ! xvap.xva_xoptattrs.xoa_nounlink); FLAG_CHECK(UF_NODUMP, XAT_NODUMP, ! xvap.xva_xoptattrs.xoa_nodump); #undef FLAG_CHECK *vap = xvap.xva_vattr; vap->va_flags = fflags; --- 6057,6105 ---- XVA_SET_REQ(&xvap, XAT_APPENDONLY); XVA_SET_REQ(&xvap, XAT_NOUNLINK); XVA_SET_REQ(&xvap, XAT_NODUMP); + XVA_SET_REQ(&xvap, XAT_READONLY); + XVA_SET_REQ(&xvap, XAT_ARCHIVE); + XVA_SET_REQ(&xvap, XAT_SYSTEM); + XVA_SET_REQ(&xvap, XAT_HIDDEN); + XVA_SET_REQ(&xvap, XAT_REPARSE); + XVA_SET_REQ(&xvap, XAT_OFFLINE); + XVA_SET_REQ(&xvap, XAT_SPARSE); + error = zfs_getattr(ap->a_vp, (vattr_t *)&xvap, 0, ap->a_cred, NULL); if (error != 0) return (error); /* Convert ZFS xattr into chflags. */ ! #define FLAG_CHECK(fflag, xflag, xfield, invert) do { \ ! if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) { \ ! if (!invert) \ ! fflags |= (fflag); \ ! } else if (invert) \ fflags |= (fflag); \ } while (0) FLAG_CHECK(SF_IMMUTABLE, XAT_IMMUTABLE, ! xvap.xva_xoptattrs.xoa_immutable, 0); FLAG_CHECK(SF_APPEND, XAT_APPENDONLY, ! xvap.xva_xoptattrs.xoa_appendonly, 0); FLAG_CHECK(SF_NOUNLINK, XAT_NOUNLINK, ! xvap.xva_xoptattrs.xoa_nounlink, 0); ! FLAG_CHECK(SF_ARCHIVED, XAT_ARCHIVE, ! xvap.xva_xoptattrs.xoa_archive, 1); FLAG_CHECK(UF_NODUMP, XAT_NODUMP, ! xvap.xva_xoptattrs.xoa_nodump, 0); ! FLAG_CHECK(UF_IMMUTABLE, XAT_READONLY, ! xvap.xva_xoptattrs.xoa_readonly, 0); ! FLAG_CHECK(UF_SYSTEM, XAT_SYSTEM, ! xvap.xva_xoptattrs.xoa_system, 0); ! FLAG_CHECK(UF_HIDDEN, XAT_HIDDEN, ! xvap.xva_xoptattrs.xoa_hidden, 0); ! FLAG_CHECK(UF_REPARSE, XAT_REPARSE, ! xvap.xva_xoptattrs.xoa_reparse, 0); ! FLAG_CHECK(UF_OFFLINE, XAT_OFFLINE, ! xvap.xva_xoptattrs.xoa_offline, 0); ! FLAG_CHECK(UF_SPARSE, XAT_SPARSE, ! xvap.xva_xoptattrs.xoa_sparse, 0); ! #undef FLAG_CHECK *vap = xvap.xva_vattr; vap->va_flags = fflags; *************** *** 6111,6117 **** return (EOPNOTSUPP); fflags = vap->va_flags; ! if ((fflags & ~(SF_IMMUTABLE|SF_APPEND|SF_NOUNLINK|UF_NODUMP)) != 0) return (EOPNOTSUPP); /* * Unprivileged processes are not permitted to unset system --- 6137,6152 ---- return (EOPNOTSUPP); fflags = vap->va_flags; ! /* ! * XXX KDM ! * We need to figure out whether it makes sense to allow ! * UF_REPARSE through, since we don't really have other ! * facilities to handle reparse points and zfs_setattr() ! * doesn't currently allow setting that attribute anyway. ! */ ! if ((fflags & ~(SF_IMMUTABLE|SF_APPEND|SF_ARCHIVED|SF_NOUNLINK| ! UF_NODUMP|UF_IMMUTABLE|UF_SYSTEM|UF_HIDDEN|UF_REPARSE| ! UF_OFFLINE|UF_SPARSE)) != 0) return (EOPNOTSUPP); /* * Unprivileged processes are not permitted to unset system *************** *** 6148,6170 **** } } ! #define FLAG_CHANGE(fflag, zflag, xflag, xfield) do { \ ! if (((fflags & (fflag)) && !(zflags & (zflag))) || \ ! ((zflags & (zflag)) && !(fflags & (fflag)))) { \ ! XVA_SET_REQ(&xvap, (xflag)); \ ! (xfield) = ((fflags & (fflag)) != 0); \ } \ } while (0) /* Convert chflags into ZFS-type flags. */ /* XXX: what about SF_SETTABLE?. */ FLAG_CHANGE(SF_IMMUTABLE, ZFS_IMMUTABLE, XAT_IMMUTABLE, ! xvap.xva_xoptattrs.xoa_immutable); FLAG_CHANGE(SF_APPEND, ZFS_APPENDONLY, XAT_APPENDONLY, ! xvap.xva_xoptattrs.xoa_appendonly); FLAG_CHANGE(SF_NOUNLINK, ZFS_NOUNLINK, XAT_NOUNLINK, ! xvap.xva_xoptattrs.xoa_nounlink); FLAG_CHANGE(UF_NODUMP, ZFS_NODUMP, XAT_NODUMP, ! xvap.xva_xoptattrs.xoa_nodump); #undef FLAG_CHANGE } return (zfs_setattr(vp, (vattr_t *)&xvap, 0, cred, NULL)); --- 6183,6227 ---- } } ! #define FLAG_CHANGE(fflag, zflag, xflag, xfield, invert) do { \ ! if (!invert) { \ ! if (((fflags & (fflag)) && !(zflags & (zflag))) || \ ! ((zflags & (zflag)) && !(fflags & (fflag)))) { \ ! XVA_SET_REQ(&xvap, (xflag)); \ ! (xfield) = ((fflags & (fflag)) != 0); \ ! } \ ! } else { \ ! if (((fflags & (fflag)) && (zflags & (zflag))) || \ ! (!(zflags & (zflag)) && !(fflags & (fflag)))) { \ ! XVA_SET_REQ(&xvap, (xflag)); \ ! (xfield) = ((fflags & (fflag)) == 0); \ ! } \ } \ } while (0) /* Convert chflags into ZFS-type flags. */ /* XXX: what about SF_SETTABLE?. */ FLAG_CHANGE(SF_IMMUTABLE, ZFS_IMMUTABLE, XAT_IMMUTABLE, ! xvap.xva_xoptattrs.xoa_immutable, 0); FLAG_CHANGE(SF_APPEND, ZFS_APPENDONLY, XAT_APPENDONLY, ! xvap.xva_xoptattrs.xoa_appendonly, 0); FLAG_CHANGE(SF_NOUNLINK, ZFS_NOUNLINK, XAT_NOUNLINK, ! xvap.xva_xoptattrs.xoa_nounlink, 0); ! FLAG_CHANGE(SF_ARCHIVED, ZFS_ARCHIVE, XAT_ARCHIVE, ! xvap.xva_xoptattrs.xoa_archive, 1); FLAG_CHANGE(UF_NODUMP, ZFS_NODUMP, XAT_NODUMP, ! xvap.xva_xoptattrs.xoa_nodump, 0); ! FLAG_CHANGE(UF_IMMUTABLE, ZFS_READONLY, XAT_READONLY, ! xvap.xva_xoptattrs.xoa_readonly, 0); ! FLAG_CHANGE(UF_SYSTEM, ZFS_SYSTEM, XAT_SYSTEM, ! xvap.xva_xoptattrs.xoa_system, 0); ! FLAG_CHANGE(UF_HIDDEN, ZFS_HIDDEN, XAT_HIDDEN, ! xvap.xva_xoptattrs.xoa_hidden, 0); ! FLAG_CHANGE(UF_REPARSE, ZFS_REPARSE, XAT_REPARSE, ! xvap.xva_xoptattrs.xoa_hidden, 0); ! FLAG_CHANGE(UF_OFFLINE, ZFS_OFFLINE, XAT_OFFLINE, ! xvap.xva_xoptattrs.xoa_offline, 0); ! FLAG_CHANGE(UF_SPARSE, ZFS_SPARSE, XAT_SPARSE, ! xvap.xva_xoptattrs.xoa_sparse, 0); #undef FLAG_CHANGE } return (zfs_setattr(vp, (vattr_t *)&xvap, 0, cred, NULL)); *** src/sys/fs/msdosfs/msdosfs_vnops.c.orig --- src/sys/fs/msdosfs/msdosfs_vnops.c *************** *** 345,352 **** --- 345,361 ---- vap->va_birthtime.tv_nsec = 0; } vap->va_flags = 0; + /* + * The DOS Archive attribute means that a file needs to be + * archived. The BSD SF_ARCHIVED attribute means that a file has + * been archived. Thus the inversion here. + */ if ((dep->de_Attributes & ATTR_ARCHIVE) == 0) vap->va_flags |= SF_ARCHIVED; + if (dep->de_Attributes & ATTR_HIDDEN) + vap->va_flags |= UF_HIDDEN; + if (dep->de_Attributes & ATTR_SYSTEM) + vap->va_flags |= UF_SYSTEM; vap->va_gen = 0; vap->va_blocksize = pmp->pm_bpcluster; vap->va_bytes = *************** *** 415,431 **** * set ATTR_ARCHIVE for directories `cp -pr' from a more * sensible filesystem attempts it a lot. */ ! if (vap->va_flags & SF_SETTABLE) { error = priv_check_cred(cred, PRIV_VFS_SYSFLAGS, 0); if (error) return (error); } ! if (vap->va_flags & ~SF_ARCHIVED) return EOPNOTSUPP; if (vap->va_flags & SF_ARCHIVED) dep->de_Attributes &= ~ATTR_ARCHIVE; else if (!(dep->de_Attributes & ATTR_DIRECTORY)) dep->de_Attributes |= ATTR_ARCHIVE; dep->de_flag |= DE_MODIFIED; } --- 424,448 ---- * set ATTR_ARCHIVE for directories `cp -pr' from a more * sensible filesystem attempts it a lot. */ ! if (vap->va_flags & (SF_SETTABLE & ~(SF_ARCHIVED))) { error = priv_check_cred(cred, PRIV_VFS_SYSFLAGS, 0); if (error) return (error); } ! if (vap->va_flags & ~(SF_ARCHIVED | UF_HIDDEN | UF_SYSTEM)) return EOPNOTSUPP; if (vap->va_flags & SF_ARCHIVED) dep->de_Attributes &= ~ATTR_ARCHIVE; else if (!(dep->de_Attributes & ATTR_DIRECTORY)) dep->de_Attributes |= ATTR_ARCHIVE; + if (vap->va_flags & UF_HIDDEN) + dep->de_Attributes |= ATTR_HIDDEN; + else + dep->de_Attributes &= ~ATTR_HIDDEN; + if (vap->va_flags & UF_SYSTEM) + dep->de_Attributes |= ATTR_SYSTEM; + else + dep->de_Attributes &= ~ATTR_SYSTEM; dep->de_flag |= DE_MODIFIED; } *** src/sys/fs/smbfs/smbfs_node.c.orig --- src/sys/fs/smbfs/smbfs_node.c *************** *** 370,379 **** if (diff > 2) /* XXX should be configurable */ return ENOENT; va->va_type = vp->v_type; /* vnode type (for create) */ if (vp->v_type == VREG) { va->va_mode = smp->sm_file_mode; /* files access mode and type */ ! if (np->n_dosattr & SMB_FA_RDONLY) va->va_mode &= ~(S_IWUSR|S_IWGRP|S_IWOTH); } else if (vp->v_type == VDIR) { va->va_mode = smp->sm_dir_mode; /* files access mode and type */ } else --- 370,382 ---- if (diff > 2) /* XXX should be configurable */ return ENOENT; va->va_type = vp->v_type; /* vnode type (for create) */ + va->va_flags = 0; /* flags defined for file */ if (vp->v_type == VREG) { va->va_mode = smp->sm_file_mode; /* files access mode and type */ ! if (np->n_dosattr & SMB_FA_RDONLY) { va->va_mode &= ~(S_IWUSR|S_IWGRP|S_IWOTH); + va->va_flags |= UF_IMMUTABLE; + } } else if (vp->v_type == VDIR) { va->va_mode = smp->sm_dir_mode; /* files access mode and type */ } else *************** *** 390,396 **** va->va_mtime = np->n_mtime; va->va_atime = va->va_ctime = va->va_mtime; /* time file changed */ va->va_gen = VNOVAL; /* generation number of file */ ! va->va_flags = 0; /* flags defined for file */ va->va_rdev = NODEV; /* device the special file represents */ va->va_bytes = va->va_size; /* bytes of disk space held by file */ va->va_filerev = 0; /* file modification number */ --- 393,409 ---- va->va_mtime = np->n_mtime; va->va_atime = va->va_ctime = va->va_mtime; /* time file changed */ va->va_gen = VNOVAL; /* generation number of file */ ! if (np->n_dosattr & SMB_FA_HIDDEN) ! va->va_flags |= UF_HIDDEN; ! if (np->n_dosattr & SMB_FA_SYSTEM) ! va->va_flags |= UF_SYSTEM; ! /* ! * The meaning of the SF_ARCHIVED flag is the inverse of the CIFS ! * archive flag. SF_ARCHIVED means that the file has been archived. ! * SMB_FA_ARCHIVED means that the file needs to be archived. ! */ ! if ((vp->v_type != VDIR) && ((np->n_dosattr & SMB_FA_ARCHIVE) == 0)) ! va->va_flags |= SF_ARCHIVED; va->va_rdev = NODEV; /* device the special file represents */ va->va_bytes = va->va_size; /* bytes of disk space held by file */ va->va_filerev = 0; /* file modification number */ *** src/sys/fs/smbfs/smbfs_vnops.c.orig --- src/sys/fs/smbfs/smbfs_vnops.c *************** *** 305,320 **** int old_n_dosattr; SMBVDEBUG("\n"); - if (vap->va_flags != VNOVAL) - return EOPNOTSUPP; isreadonly = (vp->v_mount->mnt_flag & MNT_RDONLY); /* * Disallow write attempts if the filesystem is mounted read-only. */ if ((vap->va_uid != (uid_t)VNOVAL || vap->va_gid != (gid_t)VNOVAL || vap->va_atime.tv_sec != VNOVAL || vap->va_mtime.tv_sec != VNOVAL || ! vap->va_mode != (mode_t)VNOVAL) && isreadonly) return EROFS; scred = smbfs_malloc_scred(); smb_makescred(scred, td, ap->a_cred); if (vap->va_size != VNOVAL) { --- 305,334 ---- int old_n_dosattr; SMBVDEBUG("\n"); isreadonly = (vp->v_mount->mnt_flag & MNT_RDONLY); /* * Disallow write attempts if the filesystem is mounted read-only. */ if ((vap->va_uid != (uid_t)VNOVAL || vap->va_gid != (gid_t)VNOVAL || vap->va_atime.tv_sec != VNOVAL || vap->va_mtime.tv_sec != VNOVAL || ! vap->va_mode != (mode_t)VNOVAL || vap->va_flags != VNOVAL) && ! isreadonly) return EROFS; + + /* + * We only support setting five flags. Don't allow setting others. + * + * We map both SF_IMMUTABLE and UF_IMMUTABLE to SMB_FA_RDONLY for + * setting attributes. This is compatible with the MacOS X version + * of this code. SMB_FA_RDONLY translates only to UF_IMMUTABLE + * when getting attributes. + */ + if (vap->va_flags != VNOVAL) { + if (vap->va_flags & ~(UF_IMMUTABLE|UF_HIDDEN|UF_SYSTEM| + SF_ARCHIVED|SF_IMMUTABLE)) + return EINVAL; + } + scred = smbfs_malloc_scred(); smb_makescred(scred, td, ap->a_cred); if (vap->va_size != VNOVAL) { *************** *** 353,364 **** goto out; } } ! if (vap->va_mode != (mode_t)VNOVAL) { old_n_dosattr = np->n_dosattr; ! if (vap->va_mode & S_IWUSR) ! np->n_dosattr &= ~SMB_FA_RDONLY; ! else ! np->n_dosattr |= SMB_FA_RDONLY; if (np->n_dosattr != old_n_dosattr) { error = smbfs_smb_setpattr(np, np->n_dosattr, NULL, scred); if (error) --- 367,413 ---- goto out; } } ! if ((vap->va_flags != VNOVAL) || (vap->va_mode != (mode_t)VNOVAL)) { old_n_dosattr = np->n_dosattr; ! ! if (vap->va_mode != (mode_t)VNOVAL) { ! if (vap->va_mode & S_IWUSR) ! np->n_dosattr &= ~SMB_FA_RDONLY; ! else ! np->n_dosattr |= SMB_FA_RDONLY; ! } ! ! if (vap->va_flags != VNOVAL) { ! if (vap->va_flags & UF_HIDDEN) ! np->n_dosattr |= SMB_FA_HIDDEN; ! else ! np->n_dosattr &= ~SMB_FA_HIDDEN; ! ! if (vap->va_flags & UF_SYSTEM) ! np->n_dosattr |= SMB_FA_SYSTEM; ! else ! np->n_dosattr &= ~SMB_FA_SYSTEM; ! ! if (vap->va_flags & SF_ARCHIVED) ! np->n_dosattr &= ~SMB_FA_ARCHIVE; ! else ! np->n_dosattr |= SMB_FA_ARCHIVE; ! ! /* ! * We only support setting the immutable / readonly ! * bit for regular files. According to comments in ! * the MacOS X version of this code, supporting the ! * readonly bit on directories doesn't do the same ! * thing in Windows as in Unix. ! */ ! if (vp->v_type == VREG) { ! if (vap->va_flags & (UF_IMMUTABLE|SF_IMMUTABLE)) ! np->n_dosattr |= SMB_FA_RDONLY; ! else ! np->n_dosattr &= ~SMB_FA_RDONLY; ! } ! } ! if (np->n_dosattr != old_n_dosattr) { error = smbfs_smb_setpattr(np, np->n_dosattr, NULL, scred); if (error) *** src/sys/sys/stat.h.orig --- src/sys/sys/stat.h *************** *** 265,272 **** #define UF_NODUMP 0x00000001 /* do not dump file */ #define UF_IMMUTABLE 0x00000002 /* file may not be changed */ #define UF_APPEND 0x00000004 /* writes to file may only append */ ! #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ ! #define UF_NOUNLINK 0x00000010 /* file may not be removed or renamed */ /* * Super-user changeable flags. */ --- 265,288 ---- #define UF_NODUMP 0x00000001 /* do not dump file */ #define UF_IMMUTABLE 0x00000002 /* file may not be changed */ #define UF_APPEND 0x00000004 /* writes to file may only append */ ! #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ ! #define UF_NOUNLINK 0x00000010 /* file may not be removed or renamed */ ! /* ! * These two bits are defined in MacOS X. They are not currently used in ! * FreeBSD. ! */ ! #if 0 ! #define UF_COMPRESSED 0x00000020 /* file is compressed */ ! #define UF_TRACKED 0x00000040 /* renames and deletes are tracked */ ! #endif ! ! #define UF_SYSTEM 0x00000080 /* Windows system file bit */ ! #define UF_SPARSE 0x00000100 /* sparse file */ ! #define UF_OFFLINE 0x00000200 /* file is offline */ ! #define UF_REPARSE 0x00000400 /* Windows reparse point file bit */ ! /* This is the same as the MacOS X definition of UF_HIDDEN. */ ! #define UF_HIDDEN 0x00008000 /* file is hidden */ ! /* * Super-user changeable flags. */ *** src/sys/ufs/ufs/ufs_vnops.c.orig --- src/sys/ufs/ufs/ufs_vnops.c *************** *** 528,536 **** return (EINVAL); } if (vap->va_flags != VNOVAL) { ! if ((vap->va_flags & ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | ! UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | ! SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) return (EOPNOTSUPP); if (vp->v_mount->mnt_flag & MNT_RDONLY) return (EROFS); --- 528,537 ---- return (EINVAL); } if (vap->va_flags != VNOVAL) { ! if ((vap->va_flags & ~(SF_APPEND | SF_ARCHIVED | SF_IMMUTABLE | ! SF_NOUNLINK | SF_SNAPSHOT | UF_APPEND | UF_HIDDEN | ! UF_IMMUTABLE | UF_NODUMP | UF_NOUNLINK | UF_OFFLINE | ! UF_OPAQUE | UF_REPARSE | UF_SPARSE | UF_SYSTEM)) != 0) return (EOPNOTSUPP); if (vp->v_mount->mnt_flag & MNT_RDONLY) return (EROFS); --Q68bSM7Ycu6FN28Q-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 00:12:53 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 82CC47D7; Sat, 9 Mar 2013 00:12:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id C7BC715D; Sat, 9 Mar 2013 00:12:52 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEANx9OlGDaFvO/2dsb2JhbAA9BoglvDaBc3SCLAEBAQMBAQEBIAQnIAsFFhgCAg0ZAikBCRgBDQYIBwQBHASHbAYMqWGSNYEjjDgMcQEzB4ItgRMDiHGKHYEHgj6BHo9UgyhPgQU1 X-IronPort-AV: E=Sophos;i="4.84,810,1355115600"; d="scan'208";a="17831967" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 08 Mar 2013 19:11:44 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 7CBF8B402B; Fri, 8 Mar 2013 19:11:44 -0500 (EST) Date: Fri, 8 Mar 2013 19:11:44 -0500 (EST) From: Rick Macklem To: "Kenneth D. Merry" Message-ID: <37540270.3721223.1362787904494.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20130308232155.GA47062@nargothrond.kdm.org> Subject: Re: patches to add new stat(2) file flags MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: arch@FreeBSD.org, fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 00:12:53 -0000 Kenneth D. Merry wrote: > On Fri, Mar 08, 2013 at 00:37:15 +1100, Bruce Evans wrote: > > On Wed, 6 Mar 2013, Kenneth D. Merry wrote: > > > > >I have attached diffs against head for some additional stat(2) file > > >flags. > > > > > >The primary purpose of these flags is to improve compatibility with > > >CIFS, > > >both from the client and the server side. > > >... > > > > I missed looking at the diffs in my previous reply. > > > > % --- //depot/users/kenm/FreeBSD-test3/bin/chflags/chflags.1 > > 2013-03-04 > > 17:51:12.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 > > 2013-03-04 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.86 2013-03-06 16:42:43.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/bin/chflags/chflags.1 > > 2013-03-06 14:47:25.987128763 -0700 > > % @@ -117,6 +117,16 @@ > > % set the user immutable flag (owner or super-user only) > > % .It Cm uunlnk , uunlink > > % set the user undeletable flag (owner or super-user only) > > % +.It Cm system , usystem > > % +set the Windows system flag (owner or super-user only) > > > > This begins unsorting of the list. > > Fixed. > > > It's not just a Windows flag, since it also works in DOS. > > Fixed. > > > "Owner or" is too strict for msdosfs, since files can only have a > > single owner so it is controlling access using groups is needed. I > > use owner root and group msdosfs for msdosfs mounts. This works for > > normal operations like open/read/write, but fails for most > > attributes > > including file flags. msdosfs doesn't support many attributes but > > this change is supposed to add support for 3 new file flags so it > > would > > be good if it didn't restrict the support to root. > > I wasn't trying to change the existing security model for msdosfs, but > if > you've got a suggested patch to fix it I can add that in. > > > % +.It Cm sparse , usparse > > % +set the sparse file attribute (owner or super-user only) > > % +.It Cm offline , uoffline > > % +set the offline file attribute (owner or super-user only) > > % +.It Cm reparse , ureparse > > % +set the Windows reparse point file attribute (owner or super-user > > only) > > % +.It Cm hidden , uhidden > > % +set the hidden file attribute (owner or super-user only) > > > > The additions at the end are also internally unsorted. > > Fixed. > > > Previously only "opaque" and "nodump" were unsorted. They are UF > > flags > > sorted with the SF flags, and "no" in "nodump" is not ignored for > > the > > purposes of sorting. > > > > Not having "u" in the old and new UF flag names messes up the sort > > order > > (unless you add futher confusion by ignoring "u" for the purposes of > > sorting) and makes it harder to add SF variants of the flags. > > They're now sorted in alphabetical order. > > > % .El > > % .Pp > > % Putting the letters > > % --- //depot/users/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > > 2013-03-04 17:51:12.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > > 2013-03-04 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.178 2013-03-06 16:42:43.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/gen/strtofflags.c > > 2013-03-06 15:09:32.842437917 -0700 > > % @@ -68,7 +68,17 @@ > > % { "nodump", 1, UF_NODUMP }, > > % { "noopaque", 0, UF_OPAQUE }, > > % { "nouunlnk", 0, UF_NOUNLINK }, > > % - { "nouunlink", 0, UF_NOUNLINK } > > % + { "nouunlink", 0, UF_NOUNLINK }, > > % + { "nosystem", 0, UF_SYSTEM, }, > > % + { "nousystem", 0, UF_SYSTEM, }, > > % + { "nosparse", 0, UF_SPARSE, }, > > % + { "nousparse", 0, UF_SPARSE, }, > > % + { "nooffline", 0, UF_OFFLINE, }, > > % + { "nouoffline", 0, UF_OFFLINE, }, > > % + { "noreparse", 0, UF_REPARSE, }, > > % + { "noureparse", 0, UF_REPARSE, }, > > % + { "nohidden", 0, UF_HIDDEN, }, > > % + { "nouhidden", 0, UF_HIDDEN, } > > > > This is totally disordered too. > > > > The old table was sorted except for "nosnapshot". Another bug is > > that > > "nosnapshot" is supported here (so chflags(1) can show it), but is > > not > > documented in chflags(1). > > Actually, I think the table was previously sorted by the stat(2) flag > name, > and UF_NOUNLINK appears to be the only one that was out of place. I > have > fixed that and my additions. > > > % }; > > % #define nmappings (sizeof(mapping) / sizeof(mapping[0])) > > % > > % --- //depot/users/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 > > 2013-03-04 > > 17:51:12.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 > > 2013-03-04 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.257 2013-03-06 16:42:43.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/lib/libc/sys/chflags.2 > > 2013-03-06 14:49:39.357125717 -0700 > > % @@ -75,16 +75,49 @@ > > % Do not dump the file. > > % .It Dv UF_IMMUTABLE > > % The file may not be changed. > > % +Filesystems may use this flag to maintain compatibility with the > > Windows > > and > > % +CIFS FILE_ATTRIBUTE_READONLY attribute. > > % .It Dv UF_APPEND > > % The file may only be appended to. > > > > This was already totally disordered. It even has UF's before SF's, > > although > > SF's sort before UF's both lexically and logically, though not in > > sys/stat.h > > or in bits. > > > > The disorder here was apparently copied from the implementation > > (sys/stat.h, which is in the order of the bits, which is historical > > for binary compatibility) and not cleaned up like chflags(1) and > > strttoflags(3). > > Fixed. > > > % .It Dv UF_NOUNLINK > > % The file may not be renamed or deleted. > > % .It Dv UF_OPAQUE > > % The directory is opaque when viewed through a union stack. > > % +.It Dv UF_SYSTEM > > % +The file has the Windows and CIFS FILE_ATTRIBUTE_SYSTEM > > attribute. > > % +Filesystems in FreeBSD may store and display this flag, but do > > not > > provide > > % +any special handling when it is set. > > > > More disordered than before... > > > > % +.It Dv UF_HIDDEN > > % +The file may be hidden from directory listings at the > > application's > > % +discretion. > > % +The file has the Windows FILE_ATTRIBUTE_HIDDEN attribute. > > > > Not just Windows... > > Fixed. > > > % .It Dv SF_ARCHIVED > > % -The file may be archived. > > % +The file has been archived. > > % +This flag means the opposite of the Windows and CIFS > > FILE_ATTRIBUTE_ARCHIVE > > % +attribute. > > % +That attribute means that the file should be archived, whereas > > % +.Dv SF_ARCHIVED > > % +means that the file has been archived. > > > > WinXP uses the poor wording "avialable for archiving". This is > > better. > > > > % +Filesystems in FreeBSD may or may not have special handling for > > this > > flag. > > % +For instance, ZFS tracks changes to files and will clear this bit > > when a > > % +file is updated. > > % +UFS only stores the flag, and relies on the application to change > > it when > > % +needed. > > > > I think that is useless, since changing it is needed whenever the > > file > > changes, and applications can do that (short of running as daemons > > and > > watching for changes). > > Do you mean applications can't do that or can? > > > WinXP seems to not set the flag when attributes change. I think that > > is > > a bug, and FreeBSD msdosfs does set it when attributes are changed > > (except > > for the SF_ARCHIVED attribute itself, and atimes when the atimes are > > set > > automatically). The FreeBSD behaviour is like setting the ctime for > > any > > change and is bug for bug compatible in doing this even for null > > changes. > > FreeBSD doesn't have many attributes to set, but you just added some > > more. > > I think this should be controlled by a mount option so that bug for > > bug > > compatibility with WinDOS is possible. > > > > % .It Dv SF_IMMUTABLE > > % The file may not be changed. > > % +This flag also indicates that the Windows ans CIFS > > FILE_ATTRIBUTE_READONLY > > % +attribute is set. > > > > s/ans/and/ > > > > You only mentioned UF_IMMUTABLE in your general description. Mapping > > READONLY to both gives even more confusing semantics, and mapping it > > to > > SF_IMMUTABLE seems too strict since for example it prevents using > > FreeBSD > > to manage the flag at high securelevels. > > I agree. That man page change was left over from an earlier version of > the > code. I took it out. > > > % --- > > //depot/users/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > > 2013-03-04 17:51:12.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > > 2013-03-04 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.370 2013-03-06 16:42:43.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/fs/msdosfs/msdosfs_vnops.c > > 2013-03-06 14:49:47.179130318 -0700 > > % @@ -345,8 +345,17 @@ > > % vap->va_birthtime.tv_nsec = 0; > > % } > > % vap->va_flags = 0; > > % + /* > > % + * The DOS Archive attribute means that a file needs to be > > % + * archived. The BSD SF_ARCHIVED attribute means that a file has > > % + * been archived. Thus the inversion here. > > % + */ > > > > No need to document it again. It goes without saying that ARCHIVE > > != ARCHIVED. > > I disagree. It wasn't immediately obvious to me that SF_ARCHIVED was > generally used as the inverse of the DOS Archived bit until I started > digging into this. If this helps anyone figure that out more quickly, > it's > useful. > > > % if ((dep->de_Attributes & ATTR_ARCHIVE) == 0) > > % vap->va_flags |= SF_ARCHIVED; > > % + if (dep->de_Attributes & ATTR_HIDDEN) > > % + vap->va_flags |= UF_HIDDEN; > > % + if (dep->de_Attributes & ATTR_SYSTEM) > > % + vap->va_flags |= UF_SYSTEM; > > % vap->va_gen = 0; > > % vap->va_blocksize = pmp->pm_bpcluster; > > % vap->va_bytes = > > % @@ -420,12 +429,21 @@ > > % if (error) > > % return (error); > > % } > > > > The permissions check before this is delicate and was broken and is > > more broken now. It is still short-circuit to handle setting the > > single flag that used to be supported, and is slightly broken for > > that: > > - unprivileged user asks to set ARCHIVE by passing !SF_ARCHIVED. We > > allow that, although this may toggle the flag and normal semantics > > for SF flags is to not allow toggling. > > - unprivileged user asks to clear ARCHIVE by passing SF_ARCHIVED. We > > don't allow that. But we should allow preserving ARCHIVE if it is > > already clear. > > The bug wasn't very important when only 1 flag was supported. Now it > > prevents unprivileged users managing the new UF flags if ARCHIVE is > > clear. Fortunately, this is the unusual case. Anyway, unprivileged > > users can set ARCHIVE by doing some other operation. Even the > > chflags() > > operation should set ARCHIVE and thus allow further chflags()'s that > > now > > keep ARCHIVE set. Except it is very confusing if a chflags() asks > > for > > ARCHIVE to be clear. This request might be just to try to preserve > > the current setting and not want it if other things are changed, or > > it might be to purposely clear it. Changing it from set to clear > > should > > still be privileged. > > I changed it to allow setting or clearing SF_ARCHIVED. Now I can set > or > clear the flag as non-root: > > [root@doc-sd /testpool]# mount_msdosfs -u operator -g operator > /dev/md0 /mnt > [root@doc-sd /testpool]# cd /mnt > [root@doc-sd /mnt]# ls -lao > total 20 > drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . > drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. > -rwxr-xr-x 1 operator operator - 0 Mar 8 16:54 foo > [root@doc-sd /mnt]# su operator > [operator@doc-sd /mnt]$ chflags arch foo > [operator@doc-sd /mnt]$ ls -lao > total 20 > drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . > drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. > -rwxr-xr-x 1 operator operator arch 0 Mar 8 16:54 foo > [operator@doc-sd /mnt]$ chflags 0 foo > [operator@doc-sd /mnt]$ ls -lao > total 20 > drwxr-xr-x 1 operator operator arch 16384 Jan 1 1980 . > drwxr-xr-x 27 root wheel - 1024 Mar 8 17:05 .. > -rwxr-xr-x 1 operator operator - 0 Mar 8 16:54 foo > > > See the more complicated permissions check in ffs. It would be > > safest > > to duplicate most of it, to get different permissions checking for > > the > > SF and UF flags. Then decide if we want to keep allowing setting > > ARCHIVE without privilege. > > I think we should allow getting and setting SF_ARCHIVED without > special > privileges. Given how it is generally used, I don't think it should be > restricted to the super-user. > > Can you provide some code demonstrating how the permissions code > should > be changed in msdosfs? I don't know that much about that sort of > thing, > so I'll probably spend an inordinate amount of time stumbling > through it. > > > % - if (vap->va_flags & ~SF_ARCHIVED) > > % + if (vap->va_flags & ~(SF_ARCHIVED|UF_HIDDEN|UF_SYSTEM)) > > > > Style bugs (missing spaces around binary operator '|'). These style > > bugs are common in atribute handling for other fs's but were not > > here. > > Fixed. > > > % return EOPNOTSUPP; > > % if (vap->va_flags & SF_ARCHIVED) > > % dep->de_Attributes &= ~ATTR_ARCHIVE; > > % else if (!(dep->de_Attributes & ATTR_DIRECTORY)) > > % dep->de_Attributes |= ATTR_ARCHIVE; > > > > The comment before this says that we ignore attmps to set > > ATTR_ARCHIVED > > for directories. However, it is out of date. WinXP allows setting it > > and all the new flags for directories, and so do we. > > Do you mean we allow setting it in UFS, or where? Obviously the code > above > won't set it on a directory. > > > % + if (vap->va_flags & UF_HIDDEN) > > % + dep->de_Attributes |= ATTR_HIDDEN; > > % + else > > % + dep->de_Attributes &= ~ATTR_HIDDEN; > > % + if (vap->va_flags & UF_SYSTEM) > > % + dep->de_Attributes |= ATTR_SYSTEM; > > % + else > > % + dep->de_Attributes &= ~ATTR_SYSTEM; > > > > I thought you were adding ATTR_READONLY -> UF_IMMUTABLE here. > > No, I said in the commit message that it could be extended to map > ATTR_READONLY to UF_IMMUTABLE, not that I actually did that. > > I didn't make the change because of the existing code to map readonly > to > standard Unix permissions. > > > The WinXP attrib command (at least on a FAT32 fs) doesn't allow > > setting > > or clearing ARCHIVE (even if it is already set or clear) if any of > > HIDDEN, READONLY or SYSTEM is already set and remains set after the > > command. Thus the HRS attributes act a bit like immutable flags, but > > subtly differently. (ffs has the quite different and worse behaviour > > of allowing chflags() irrespective of immutable flags being set > > before > > or after, provided there is enough privilege to change the immutable > > flags.) Anyway, they should all give some aspects of immutability. > > We could do that for msdosfs, but why add more things for the user to > trip > over given how the filesystem is typically used? Most people probably > use it for USB thumb drives these days. Or perahps on a dual boot > system > to access their Windows partition. > > > % + > > > > Style bug (extra blank line). > > Fixed. > > > % dep->de_flag |= DE_MODIFIED; > > % } > > % > > % --- //depot/users/kenm/FreeBSD-test3/sys/sys/stat.h 2013-03-04 > > 17:51:12.000000000 -0700 > > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h > > 2013-03-04 > > 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.563 2013-03-06 16:42:43.000000000 -0700 > > % +++ /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/sys/stat.h > > 2013-03-06 > > 15:05:45.936126193 -0700 > > % @@ -268,6 +268,22 @@ > > % #define UF_OPAQUE 0x00000008 /* directory is opaque wrt. union */ > > % #define UF_NOUNLINK 0x00000010 /* file may not be removed or > > renamed */ > > > > Old style bugs: inconsistent space instead of tab after #define for > > the 2 > > newer definitions. > > Fixed. > > > % /* > > % + * These two bits are defined in MacOS X. They are not currently > > used in > > % + * FreeBSD. > > % + */ > > % +#if 0 > > % +#define UF_COMPRESSED 0x00000020 /* file is compressed */ > > % +#define UF_TRACKED 0x00000040 /* renames and deletes are > > tracked */ > > % +#endif > > % + > > % +#define UF_SYSTEM 0x00000080 /* Windows system file bit */ > > % +#define UF_SPARSE 0x00000100 /* sparse file */ > > % +#define UF_OFFLINE 0x00000200 /* file is offline */ > > % +#define UF_REPARSE 0x00000400 /* Windows reparse point file bit > > */ > > % +/* This is the same as the MacOS X definition of UF_HIDDEN */ > > % +#define UF_HIDDEN 0x00008000 /* file is hidden */ > > > > New style bugs: random spaces/tabs after #define. 1 main comment not > > punctuated. > > Fixed. > > > % + > > % +/* > > % * Super-user changeable flags. > > % */ > > % #define SF_SETTABLE 0xffff0000 /* mask of superuser > > changeable flags */ > > % --- //depot/users/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c > > 2013-03-04 > > 17:51:12.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c > > 2013-03-04 17:51:12.000000000 -0700 > > % --- /tmp/tmp.49594.581 2013-03-06 16:42:43.000000000 -0700 > > % +++ > > /usr/home/kenm/perforce4/kenm/FreeBSD-test3/sys/ufs/ufs/ufs_vnops.c > > 2013-03-06 15:06:27.004132409 -0700 > > % @@ -529,8 +529,9 @@ > > % } > > % if (vap->va_flags != VNOVAL) { > > % if ((vap->va_flags & ~(UF_NODUMP | UF_IMMUTABLE | UF_APPEND | > > % - UF_OPAQUE | UF_NOUNLINK | SF_ARCHIVED | SF_IMMUTABLE | > > % - SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) != 0) > > % + UF_OPAQUE | UF_NOUNLINK | UF_HIDDEN | UF_SYSTEM | > > % + UF_SPARSE | UF_OFFLINE | UF_REPARSE | SF_ARCHIVED | > > % + SF_IMMUTABLE | SF_APPEND | SF_NOUNLINK | SF_SNAPSHOT)) > > != 0) > > % return (EOPNOTSUPP); > > % if (vp->v_mount->mnt_flag & MNT_RDONLY) > > % return (EROFS); > > > > The random order is harder to read with more flags. > > Fixed. > > > I think unsupported flags should just be unsupported -- most or all > > of the > > new flags and SF_ARCHIVED too. Most need system support to work. > > Without > > system support you can only copy them from other fs's and then lose > > them > > when utilities like tar don't unsderstand them. > > The primary reason I added these flags to UFS is to support CIFS > servers. > I have modifications to Likewise to support storing DOS/Windows/CIFS > attributes as file flags, and we could do the same for Samba. > > My eventual intent is to also add support to our NFS code to get/set > the > attributes that are allowed in the NFS 4.1 standard (hidden, system > and > archive are the ones I could find). So in this case, the filesystem > role > for some of these really is just to store it for someone else. > Just fyi, they are supported by NFSv4.0 as well, so there is no need to wait for the NFSv4.1 server code (which I have just started to work on). If you'd like, I can easily do this in April (or later), if/when your patch is committed. Have fun with it, rick > That is the way that ZFS handles attributes like hidden and system. It > doesn't do anything with them, they're just used by the Solaris CIFS > server. > > Ken > -- > Kenneth Merry > ken@FreeBSD.ORG > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to > "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 01:41:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id ED96C9D5; Sat, 9 Mar 2013 01:41:41 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id 7254237F; Sat, 9 Mar 2013 01:41:41 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.6/8.14.6) with ESMTP id r291feQT091412; Fri, 8 Mar 2013 18:41:40 -0700 (MST) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.6/8.14.6/Submit) with ESMTP id r291few6091409; Fri, 8 Mar 2013 18:41:40 -0700 (MST) (envelope-from wblock@wonkity.com) Date: Fri, 8 Mar 2013 18:41:40 -0700 (MST) From: Warren Block To: Chris Rees Subject: Re: Argument list too long In-Reply-To: Message-ID: References: <82112.1362671436.13776555968178880512@ffe17.ukr.net> <20130307161546.GV47829@e-new.0x20.net> <20130308190917.GA34838@lor.one-eyed-alien.net> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (wonkity.com [127.0.0.1]); Fri, 08 Mar 2013 18:41:40 -0700 (MST) Cc: freebsd-fs@freebsd.org, Brooks Davis X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 01:41:42 -0000 On Fri, 8 Mar 2013, Chris Rees wrote: > > For fun, after running each of these several times to preload cache: > > > > find /usr/ports -type f -print0 | xargs -0 grep -H "X-PHP-Script" > > 40.98 seconds (average) > > > > find /usr/ports -type f -exec grep -H "X-PHP-Script" {} \+ > > 42.27 seconds (average) > > > > So they aren't too different in performance. > > The \+ form I have also found impossible to achieve in csh, which is why I never recommend it. Do you mean in scripted form? It works in interactive csh, that's what I used for this test. From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 01:52:46 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B2E9AABF; Sat, 9 Mar 2013 01:52:46 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5D8EB3E9; Sat, 9 Mar 2013 01:52:46 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r291qj70010186; Fri, 8 Mar 2013 20:52:45 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r291qjvK010183; Fri, 8 Mar 2013 20:52:45 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20794.38381.221980.5038@hergotha.csail.mit.edu> Date: Fri, 8 Mar 2013 20:52:45 -0500 From: Garrett Wollman To: Rick Macklem Subject: Re: NFS DRC size In-Reply-To: <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> References: <20794.7012.265887.99878@hergotha.csail.mit.edu> <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Fri, 08 Mar 2013 20:52:45 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 01:52:46 -0000 < said: > The cached replies are copies of the mbuf list done via m_copym(). > As such, the clusters in these replies won't be free'd (ref cnt -> 0) > until the cache is trimmed (nfsrv_trimcache() gets called after the > TCP layer has received an ACK for receipt of the reply from the client). I wonder if this bit is even working at all. In my experience, the size of the DRC quickly grows under load up to the maximum (or actually, slightly beyond), and never drops much below that level. On my production server right now, "nfsstat -se" reports: Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 13036780 359901 1723623 3420 36397693 12385668 346590 109984 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 45173 16 116791 14192 1176 24 12876747 3398533 Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf 0 2703 14992 7502 1329196 0 1 1 Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock 263034 0 0 263019 0 0 545104 0 LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH 0 0 263012 0 0 23753375 0 1 Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create 2 263006 263033 0 0 0 Server: Retfailed Faults Clients 0 0 1 OpenOwner Opens LockOwner Locks Delegs 56 10 0 0 0 Server Cache Stats: Inprog Idem Non-idem Misses CacheSize TCPPeak 0 0 0 81714128 60997 61017 It's only been up for about the last 24 hours. Should I be setting the size limit to something truly outrageous, like 200,000? (I'd definitely need to deal with the mbuf cluster issue then!) The average request rate over this time is about 1000/s, but that includes several episodes of high-cpu spinning (which I resolved by increasing the DRC limit). Meanwhile, some relevant bits from sysctl: vfs.nfsd.udphighwater: 500 vfs.nfsd.tcphighwater: 61000 vfs.nfsd.minthreads: 16 vfs.nfsd.maxthreads: 64 vfs.nfsd.threads: 64 vfs.nfsd.request_space_used: 1416 vfs.nfsd.request_space_used_highest: 4284672 vfs.nfsd.request_space_high: 47185920 vfs.nfsd.request_space_low: 31457280 vfs.nfsd.request_space_throttled: 0 vfs.nfsd.request_space_throttle_count: 0 (I'd actually like to put maxthreads back up at 256, which is where I had it during testing, but I need to test that the jumbo-frames issue is fixed first. I did pre-production testing on a non-jumbo network.) -GAWollman From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 03:04:07 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9F8F3AFA; Sat, 9 Mar 2013 03:04:07 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (gw.catspoiler.org [75.1.14.242]) by mx1.freebsd.org (Postfix) with ESMTP id 7C4A5878; Sat, 9 Mar 2013 03:04:07 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id r2933qqJ032330; Fri, 8 Mar 2013 19:03:56 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201303090303.r2933qqJ032330@gw.catspoiler.org> Date: Fri, 8 Mar 2013 19:03:52 -0800 (PST) From: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! To: lev@FreeBSD.org In-Reply-To: <1402477662.20130306165337@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=iso-8859-5 Content-Transfer-Encoding: 8BIT Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 03:04:07 -0000 On 6 Mar, Lev Serebryakov wrote: > Hello, Don. > You wrote 6 марта 2013 г., 14:01:08: > >>> DL> With NCQ or TCQ, the drive can have a sizeable number of writes >>> DL> internally queued and it is free to reorder them as it pleases even with >>> DL> write caching disabled, but if write caching is disabled it has to delay >>> DL> the notification of their completion until the data is on the platters >>> DL> so that UFS+SU can enforce the proper dependency ordering. >>> But, again, performance would be terrible :( I've checked it. On >>> very sparse multi-threaded patterns (multiple torrents download on >>> fast channel in my simple home case, and, I think, things could be >>> worse in case of big file server in organization) and "simple" SATA >>> drives it significant worse in my experience :( > > DL> I'm surprised that a typical drive would have enough onboard cache for > DL> write caching to help signficantly in that situation. Is the torrent > It is 5x64MiB in my case, oh, effectively, 4x64MiB :) > Really, I could repeat experiment with some predictable and > repeatable benchmark. What in out ports could be used for > massively-parallel (16+ files) random (with blocks like 64KiB and > file sizes like 2+GiB) but "repeatable" benchmark? I don't happen to know of any benchmark software in ports for this, but I haven't really looked. > DL> software doing a lot of fsync() calls? Those would essentially turn > Nope. It trys to avoid fsync(), of course > > DL> Creating a file by writing it in random order is fairly expensive. Each > DL> time a new block is written by the application, UFS+SU has to first find > DL> a free block by searching the block bitmaps, mark that block as > DL> allocated, wait for that write of the bitmap block to complete, write > DL> the data to that block, wait for that to complete, and then write the > DL> block pointer to the inode or an indirect block. Because of the random > DL> write ordering, there is probably not enough locality to do coalesce > DL> multiple updates to the bitmap and indirect blocks into one write before > DL> the syncer interval expires. These operations all happen in the > DL> background after the write() call, but once you hit the I/O per second > DL> limit of the drive, eventually enough backlog builds to stall the > DL> application. Also, if another update needs to be done to a block that > DL> the syncer has queued for writing, that may also cause a stall until the > DL> write completes. If you hack the torrent software to create and > DL> pre-zero each file before it starts downloading it, then each bitmap and > DL> indirect block will probably only get written once during that operation > DL> and won't get written again during the actual download, and zeroing the > DL> data blocks will be sequential and fast. During the download, the only > DL> writes will be to the data blocks, so you might see something like a 3x > DL> performance improvement. > My client (transmission, from ports) is configured to do "real > preallocation" (not sparse one), but it doesn't help much. It surely > limited by disk I/O :( > But anyway, torrent client is bad benchmark if we start to speak > about some real experiments to decide what could be improved in > FFS/GEOM stack, as it is not very repeatable. I seem to recall that you mentioning that the raid5 geom layer is doing a lot of caching, presumably to coalesce writes. If this causes the responses to writes to be delayed too much, then the geom layer could end up starved for writes because the vfs.hirunningspace limit will be reached. If this happens, you'll see threads waiting on wdrain. You could also monitor vfs.runningbufspace to see how close it is getting to the limit. If this is the problem, you might want to try cranking up the value of vfs.hirunningspace to see if it helps. One thing that doesn't seem to fit this theory is that if the raid5 layer is doing a lot of caching to try to do write coalescing, then I wouldn't expect that the extra write completion latency caused by turning off write caching in the drives would make much of a difference. Another possibility is that you might be running into the 32 NCQ command limit when write caching is off. With write caching on, then you can probably shove a lot more write commands into the drive before being blocked. That might help the drive get a bit higher iops, but I wouldn't expect a big difference. It could also be that when you hit the limit that you end up blocking read commands from getting sent to the drives, which causes whatever is depending on the data to stall. The gstat command allows the queue length and number of reads and writes to be monitored, but I don't know of a way to monitor the number of read and write commands that the drive has in its internal queue. Something else to look at is what problems might the delayed write completion notifications from the drives cause in the raid5 layer itself. Could that be preventing the raid5 layer from sending other I/O commands to the drives? Between the time a write command has been sent to a drive and the drive reports the completion of the write, what happens if something wants to touch that buffer? What size writes does the application typically do? What is the UFS blocksize? What is the raid5 stripe size? With this access pattern, you may get poor results if the stripe size is much greater than the block and write sizes. From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 12:08:28 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0F228E58; Sat, 9 Mar 2013 12:08:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id C1A0ACD4; Sat, 9 Mar 2013 12:08:27 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:8571:2d32:217f:d124]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 53E394AC57; Sat, 9 Mar 2013 16:08:21 +0400 (MSK) Date: Sat, 9 Mar 2013 16:08:17 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <1809201254.20130309160817@serebryakov.spb.ru> To: Don Lewis Subject: Re: Unexpected SU+J inconsistency AGAIN -- please, don't shift topic to ZFS! In-Reply-To: <201303090303.r2933qqJ032330@gw.catspoiler.org> References: <1402477662.20130306165337@serebryakov.spb.ru> <201303090303.r2933qqJ032330@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-5 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@FreeBSD.org, ivoras@FreeBSD.org, freebsd-geom@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 12:08:28 -0000 Hello, Don. You wrote 9 =DC=D0=E0=E2=D0 2013 =D3., 7:03:52: >> But anyway, torrent client is bad benchmark if we start to speak >> about some real experiments to decide what could be improved in >> FFS/GEOM stack, as it is not very repeatable. DL> I seem to recall that you mentioning that the raid5 geom layer is doing DL> a lot of caching, presumably to coalesce writes. If this causes the DL> responses to writes to be delayed too much, then the geom layer could DL> end up starved for writes because the vfs.hirunningspace limit will be DL> reached. If this happens, you'll see threads waiting on wdrain. You DL> could also monitor vfs.runningbufspace to see how close it is getting to DL> the limit. If this is the problem, you might want to try cranking up Strangely enough, vfs.runningbufspace is always zero, even under load. My geom_raid5 is configured to dealy writes up to 15 seconds... DL> Something else to look at is what problems might the delayed write DL> completion notifications from the drives cause in the raid5 layer DL> itself. Could that be preventing the raid5 layer from sending other I/O DL> commands to the drives? Between the time a write command has been sent Nope. It should not. I'm not sure for 100%, as I picked up these sources from original author and sources are rather cryptic, but I could not see any throttling in it. DL> to a drive and the drive reports the completion of the write, what DL> happens if something wants to touch that buffer? DL> What size writes does the application typically do? What is the UFS 64K writes, 32K blocksize, 128K stripe size... Now I'm analyzing traces from this device to understand exact write patterns. DL> blocksize? What is the raid5 stripe size? With this access pattern, DL> you may get poor results if the stripe size is much greater than the DL> block and write sizes. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 15:44:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 44A23ED for ; Sat, 9 Mar 2013 15:44:01 +0000 (UTC) (envelope-from stephane.lapie@darkbsd.org) Received: from denrei.darkbsd.org (denrei.darkbsd.org [IPv6:2001:41d0:1:f442::1]) by mx1.freebsd.org (Postfix) with ESMTP id BE4DC9F7 for ; Sat, 9 Mar 2013 15:44:00 +0000 (UTC) Received: from denrei.darkbsd.org (localhost [127.0.0.1]) by denrei.darkbsd.org (Postfix) with ESMTP id 03D64E41 for ; Sat, 9 Mar 2013 16:43:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:content-type; s=selector1; bh=I1bgyoLGa+Z34u+HcILoE4Y3UpE=; b=tPbTYX9/77rdlaj84YEaeveHSOZN qO4xkUBVRBaflH+YdBE0Gz4Hh57zPDnNMgkrWe/orP+1ukE/+tkdVN/Vs1T2TrFY 0KNZaIvZJ2WXDa4pIAU0Z747rN4Qs7LajeqQRl77Afw7qW1LsK4HA23J6JZj1XrP nSPxjrR+glIq3F0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:content-type; q=dns; s= selector1; b=vkmq67yAYtJ4zpzbaFlRp169w6514VO1ki/apWOoPPo2CKVmO2w wKfIhT6x/mA0ImnjR1gC7/fVJ4H8x0YwXODzJ8lC0CAwKrpNSG0qStz72BW+jx11 tzyAPxCbM9/x6jXva4hVDQdDbMOHZrhUpKHcgPqMNeOO/ZLKx7LQTHb4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-type:content-type:subject:subject:mime-version :user-agent:from:from:date:date:message-id:received:received; s= selector1; t=1362843836; bh=mMfM2CIvmSLSEnrVGmDz9uR4+zFn7+YvwI2G h4vfnuI=; b=oxyv9P2prRU8qSjR5MdaiXcYWGf84Dn/j9XpDYW+MpnkhIAzlfxb pZ1KB5LlJNAjT+bEBAchyJHSkuNTLnJ90J+uRHhf6jEIEvlQxpiAIkXm5lBE0C56 rvJ6Yd7gb+6xotQ+LeMsPniFhDW08vMn6q+Yg80b8/YwCChoWaAIV3Q= X-Virus-Scanned: amavisd-new at darkbsd.org Received: from denrei.darkbsd.org ([127.0.0.1]) by denrei.darkbsd.org (denrei.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ABhpU41Avt95 for ; Sat, 9 Mar 2013 16:43:56 +0100 (CET) Received: from [IPv6:2001:470:24:42d::42] (archer.yomi.darkbsd.org [IPv6:2001:470:24:42d::42]) (Authenticated sender: darksoul@darkbsd.org) by denrei.darkbsd.org (Postfix) with ESMTPSA id 9AEC2E40 for ; Sat, 9 Mar 2013 16:43:55 +0100 (CET) Message-ID: <513B58B6.2090903@darkbsd.org> Date: Sun, 10 Mar 2013 00:43:50 +0900 From: Stephane LAPIE User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Panic loop on ZFS with 9.1-RELEASE X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig0782E335DDACC550DDA5AB15" X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 15:44:01 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0782E335DDACC550DDA5AB15 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello list, I currently am faced with a sudden death case I can't understand at all, and I would be very appreciating of any explanation or assistance :( Here is my current kernel version : FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59 JST 2013 darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-no-= za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN=20 amd64 (Kernel is basically a lightened GENERIC kernel without VESA options and unneeded controllers removed) The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrored transaction log Booting and trying to import the pool is met with : Solaris(panic): zfs: panic: allocating allocated segment(offset=3D2335563722752 size=3D1024) Booting single mode on my emergency flash card with a base OS and zpool import -o readonly=3Don is met with : panic: solaris assert: zio->io_type !=3D ZIO_TYPE_WRITE || spa_writeable(spa), file: /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs/..= /../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2461 I tried zpool import -F -f, zpool import -F -f -m after removing the mirrored transaction log devices, but after 40s of trying to import, it just blows up. I am currently running "zdb -emm" as per the procedure suggested here : http://simplex.swordsaint.net/?p=3D199 if only to get some debug informat= ion. Thanks in advance for your time. Cheers, --=20 Stephane LAPIE, EPITA SRS, Promo 2005 "Even when they have digital readouts, I can't understand them." --MegaTokyo --------------enig0782E335DDACC550DDA5AB15 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlE7WLYACgkQDJ4OK7D3FWRoTAEA37F5qwLiUYAkvXWWZdnZv9Ei P2fnY5rRRROUeI/8ZFABALOKW6jFin09l7RPxw68xjy6MTP447zIfgtmSdYx7kIn =ofzD -----END PGP SIGNATURE----- --------------enig0782E335DDACC550DDA5AB15-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 16:27:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 50F51FD2; Sat, 9 Mar 2013 16:27:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id B9078E61; Sat, 9 Mar 2013 16:27:34 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqEEAE5iO1GDaFvO/2dsb2JhbABDiCi8OIF1dIItAQEBAwEBAQEgBCcgCwUWGAICDRkCKQEJJgYIBwQBHASHbAYMqT2SC4EjjCkKBX00B4ItgRMDiHGLJYI+gR6PVYMoT30IFx4 X-IronPort-AV: E=Sophos;i="4.84,814,1355115600"; d="scan'208";a="17907963" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Mar 2013 11:27:32 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BBA7BB4036; Sat, 9 Mar 2013 11:27:32 -0500 (EST) Date: Sat, 9 Mar 2013 11:27:32 -0500 (EST) From: Rick Macklem To: Garrett Wollman Message-ID: <1639798917.3728142.1362846452693.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20794.38381.221980.5038@hergotha.csail.mit.edu> Subject: Re: NFS DRC size MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 16:27:35 -0000 Garrett Wollman wrote: > < said: > > > The cached replies are copies of the mbuf list done via m_copym(). > > As such, the clusters in these replies won't be free'd (ref cnt -> > > 0) > > until the cache is trimmed (nfsrv_trimcache() gets called after the > > TCP layer has received an ACK for receipt of the reply from the > > client). > > I wonder if this bit is even working at all. In my experience, the > size of the DRC quickly grows under load up to the maximum (or > actually, slightly beyond), and never drops much below that level. On > my production server right now, "nfsstat -se" reports: > Well, once you add the patches and turn vfs.nfsd.tcphighwater up, it will only trim the cache when that highwater mark is exceeded. When it does the trim, the size does drop for the simple testing I do with a single client. (I'll take another look at drc3.patch and see if I can spot anywhere this might be broken, although my hunch is that you have a lot of TCP connections and enough activity that it rapidly grows back up to the limit.) The fact that it trims down to around the highwater mark basically indicates this is working. If it wasn't throwing away replies where the receipt has been ack'd at the TCP level, the cache would grow very large, since they would only be discarded after a loonnngg timeout (12hours unless you've changes NFSRVCACHE_TCPTIMEOUT in sys/fs/nfs/nfs.h). > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 13036780 359901 1723623 3420 36397693 12385668 346590 109984 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 45173 16 116791 14192 1176 24 12876747 3398533 > Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf > 0 2703 14992 7502 1329196 0 1 1 > Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock > 263034 0 0 263019 0 0 545104 0 > LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH > 0 0 263012 0 0 23753375 0 1 > Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create > 2 263006 263033 0 0 0 > Server: > Retfailed Faults Clients > 0 0 1 > OpenOwner Opens LockOwner Locks Delegs > 56 10 0 0 0 > Server Cache Stats: > Inprog Idem Non-idem Misses CacheSize TCPPeak > 0 0 0 81714128 60997 61017 > > It's only been up for about the last 24 hours. Should I be setting > the size limit to something truly outrageous, like 200,000? (I'd > definitely need to deal with the mbuf cluster issue then!) The > average request rate over this time is about 1000/s, but that includes > several episodes of high-cpu spinning (which I resolved by increasing > the DRC limit). > It is the number of TCP connections from clients that determines how much gets cached, not the request rate. For TCP, a scheme like LRU doesn't work, because RPC retries (as opposed to TCP segment retransmits) only happen long after the initial RPC request. (Usually after a TCP connection has broken and the client has established a new connection, although some NFSv3 over TCP clients will retry an RPC after a long timeout.) The cache needs to hold the last N RPC replies for each TCP connection and discard them when further traffic on the TCP connection indicates that the connection is still working. (Some NFSv3 over TCP servers don't guarantee to generate a reply for an RPC when resource constrained, but the FreeBSD one always sends a reply, except for NFSv2, where it will close down the TCP connection when it has no choice. I doubt any client is doing NFSv2 over TCP, so I don't consider this relevent.) If the CPU is spinning in nfsrc_trimcache() a lot, increasing vfs.nfsd.tcphighwater should decrease that, but with an increase in mbuf cluster allocation. If there is a lot of contention for mutexes, increasing the size of the hash table might help. The drc3.patch bumped the hash table from 20->200, but that would still be about 300 entries per hash list and one mutex for those 300 entries, assuming the hash function is working well. Increasing it only adds list head pointers and mutexes. (It's NFSRVCACHE_HASHSIZE in sys/fs/nfs/nfsrvcache.h.) Unfortunately, increasing it requires a kernel rebuild/reboot. Maybe the patch for head should change the size of the hash table when vfs.nfsd.tcphighwater is set much larger? (Not quite trivial and will probably result in a short stall of the nfsd threads, since all the entries will need to be rehashed/moved to new lists, but could be worth the effort.) > Meanwhile, some relevant bits from sysctl: > > vfs.nfsd.udphighwater: 500 > vfs.nfsd.tcphighwater: 61000 > vfs.nfsd.minthreads: 16 > vfs.nfsd.maxthreads: 64 > vfs.nfsd.threads: 64 > vfs.nfsd.request_space_used: 1416 > vfs.nfsd.request_space_used_highest: 4284672 > vfs.nfsd.request_space_high: 47185920 > vfs.nfsd.request_space_low: 31457280 > vfs.nfsd.request_space_throttled: 0 > vfs.nfsd.request_space_throttle_count: 0 > > (I'd actually like to put maxthreads back up at 256, which is where I > had it during testing, but I need to test that the jumbo-frames issue > is fixed first. I did pre-production testing on a non-jumbo network.) > > -GAWollman > Well, the DRC will try to cache replies until the client's TCP layer acknowledges receipt of the reply. It is hard to say how many replies that is for a given TCP connection, since it is a function of the level of concurrently (# of nfsiod threads in the FreeBSD client) in the client. I'd guess it's somewhere between 1<->20? Multiply that by the number of TCP connections from all clients and you have about how big the server's DRC will be. (Some clients use a single TCP connection for the client whereas others use a separate TCP connection for each mount point.) When ivoras@ and I have a patch for head, it should probably allow the DRC to be disabled for TCP mounts (by setting vfs.nfsd.tcphighwater == -1?). I don't really like the idea, but I can see the argument that TCP maintains a reliable enough RPC transport that the DRC isn't needed. rick > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 17:15:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6C05BC60 for ; Sat, 9 Mar 2013 17:15:21 +0000 (UTC) (envelope-from stephane.lapie@darkbsd.org) Received: from denrei.darkbsd.org (denrei.darkbsd.org [IPv6:2001:41d0:1:f442::1]) by mx1.freebsd.org (Postfix) with ESMTP id CE58F113 for ; Sat, 9 Mar 2013 17:15:20 +0000 (UTC) Received: from denrei.darkbsd.org (localhost [127.0.0.1]) by denrei.darkbsd.org (Postfix) with ESMTP id 75AE4E88 for ; Sat, 9 Mar 2013 18:15:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; s=selector1; bh=YQ7by/HY1wjz0vFBDcOoHHLKzTk=; b=h XtIJmvkAdQ8NVA1VGw0RTs5wK60W500svJOQjffBTIbb7x2Az/QdJ0luwBTVcEZ6 bLDnQRjNh7zAy1m2GS76QOTQz5fwGOs8E6zx5k0Y++3D+fbMEBvOwFHiS8PEeu9o /rz9p/wBdxdJxed6gRM17rKP7lg6/SN9mHekipQDzU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; q=dns; s=selector1; b=R/IoL8TSv+G9IxLa0U2Do6v/uja 8L2qzuMTw8VQavjt5FYoPOOCsPHaP1txCREIlDm7Qpr8c7by7SVOuOSvwMdHpKtw j6Gao/ol983Kwqpz8s+hLDdZjcDt6NgP16kyUTX+pY0hnkc1Uqpq9F3sVYBk6oIm w3T6W65zvbARUKlc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-type:content-type:in-reply-to:references:subject:subject :mime-version:user-agent:from:from:date:date:message-id:received :received; s=selector1; t=1362849317; bh=V9trAQghPHqibB25khouo7e HxhJx8JjPoTEuVx/phko=; b=TtlB7++GWzF5jQMfwCYobOYor9Jhs1yXiA6fx0t 3B/oT0Btsxja+echg4Og3J7kX3QBj+1WqkA4svn/dQj5LGKkQexc2n1mufbalhYs SVh1N6CB3QUYjZHFKjc7Vt1TPZuq2N+PXgu935nmADbBuxsE34nUN8WKlUUHQV0Z wkGQ= X-Virus-Scanned: amavisd-new at darkbsd.org Received: from denrei.darkbsd.org ([127.0.0.1]) by denrei.darkbsd.org (denrei.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8Qs0FDBh8N7h for ; Sat, 9 Mar 2013 18:15:17 +0100 (CET) Received: from [IPv6:2001:470:24:42d::42] (archer.yomi.darkbsd.org [IPv6:2001:470:24:42d::42]) (Authenticated sender: darksoul@darkbsd.org) by denrei.darkbsd.org (Postfix) with ESMTPSA id 2BDF4E87 for ; Sat, 9 Mar 2013 18:15:15 +0100 (CET) Message-ID: <513B6E1E.6080805@darkbsd.org> Date: Sun, 10 Mar 2013 02:15:10 +0900 From: Stephane LAPIE User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Panic loop on ZFS with 9.1-RELEASE References: <513B58B6.2090903@darkbsd.org> In-Reply-To: <513B58B6.2090903@darkbsd.org> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig1A98E6B6132DA9A790EA46E6" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 17:15:21 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1A98E6B6132DA9A790EA46E6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Posting a quick update. I ran a "zdb -emm" command to figure out what was going on, and it blew up in my face with an abort trap here : - vdev 0 has 145 metaslabs, which are cleared without any problems. - vdev 1 has 145 metaslabs, but fails in the middle : metaslab 32 offset 20000000000 spacemap 289 free 1.64G segments 19509 maxsize 41.7M freepct 2% metaslab 33 offset 21000000000 spacemap 303 free 11.9G error: zfs: allocating allocated segment(offset=3D2335563722752 size=3D10= 24) Abort trap(core dumped) Converting offset 2335563722752 from earlier kernel panic messages gives : 21fca723000, which matches the broken metaslab found by zdb. Is there anything I can do at this point, using zdb? It just sounds surrealistic I have ONE broken metaslab (seemingly?) and that I can't recover anything... Cheers, On 03/10/2013 12:43 AM, Stephane LAPIE wrote: > Hello list, > > I currently am faced with a sudden death case I can't understand at all= , > and I would be very appreciating of any explanation or assistance :( > > Here is my current kernel version : > FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59 > JST 2013 > darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-n= o-za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN=20 > amd64 > (Kernel is basically a lightened GENERIC kernel without VESA options an= d > unneeded controllers removed) > > The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrored= > transaction log > > Booting and trying to import the pool is met with : > Solaris(panic): zfs: panic: allocating allocated > segment(offset=3D2335563722752 size=3D1024) > > Booting single mode on my emergency flash card with a base OS and zpool= > import -o readonly=3Don is met with : > panic: solaris assert: zio->io_type !=3D ZIO_TYPE_WRITE || > spa_writeable(spa), file: > /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs/= =2E./../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, > line: 2461 > > I tried zpool import -F -f, zpool import -F -f -m after removing the > mirrored transaction log devices, but after 40s of trying to import, it= > just blows up. > > I am currently running "zdb -emm" as per the procedure suggested here := > http://simplex.swordsaint.net/?p=3D199 if only to get some debug inform= ation. > > Thanks in advance for your time. > > Cheers, > > > --=20 > Stephane LAPIE, EPITA SRS, Promo 2005 > "Even when they have digital readouts, I can't understand them." > --MegaTokyo --------------enig1A98E6B6132DA9A790EA46E6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlE7bh4ACgkQDJ4OK7D3FWSh6AEAjXH57LB79EthHQDPtaJkbLKt VgyGWQoa2mAgw2qPMagBAMvBfS2roNM0mC8eQq/v4mejgGNi5KJ0IvmB2xUTyYzC =H/PJ -----END PGP SIGNATURE----- --------------enig1A98E6B6132DA9A790EA46E6-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 17:46:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C2F17ED for ; Sat, 9 Mar 2013 17:46:15 +0000 (UTC) (envelope-from stephane.lapie@darkbsd.org) Received: from denrei.darkbsd.org (denrei.darkbsd.org [IPv6:2001:41d0:1:f442::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8EF101FD for ; Sat, 9 Mar 2013 17:46:14 +0000 (UTC) Received: from denrei.darkbsd.org (localhost [127.0.0.1]) by denrei.darkbsd.org (Postfix) with ESMTP id 81550E99 for ; Sat, 9 Mar 2013 18:46:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; s=selector1; bh=/Xw46G3uBK1ZlYyx3tukTYCqCqM=; b=Q OaJntrjRHoZUYa/C+Vaslahc2LKC9mMVD/WQQRGd2jMOaF5uoguW+xcLlMy9/2yL Y2qqGF+teVX4ToH7D6HSkwKqExn6EN12iv/AlrDQS24/xwD1KBHoRhVehGQw/4Ew ig7VVJiq7G012RGfqpqAxgNnHfLDGcqX9eej2h5N4w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; q=dns; s=selector1; b=RxD3PDkIUKDJroZjZVymV1JJNSX TWRIDApk7kokjBxUg7iCm1njBUvXPywWjQyNkBIG+PM7HWMe/9KfQTxYGXoXLQpz JcUF3slxnFDiS03BBz3mhkWeSFogFaIzH6Js8a7en+yByg+aoRWFcBOrXNZornwP T7aed+Juf4mwO+oE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-type:content-type:in-reply-to:references:subject:subject :mime-version:user-agent:from:from:date:date:message-id:received :received; s=selector1; t=1362851164; bh=1+68QKk4ORGPwm10mM/fNOL BcMmtH3Sn4M51X8jCmNQ=; b=QsH7PxwWev0IZ4CZ9Tft04EE9LGo2D9LEFiE205 nSMS+h/cvXeFLzP9Ac5Rd/A6UcE1PsKazJRG0bs61VjoxyojbN0feXUvED72077T 9pCDAN8YNLI8YHca6mS8OzU9qaUlMfX9J/rlp5DpwtrEjrB8o3hJ8DLmarLivKDd IfC0= X-Virus-Scanned: amavisd-new at darkbsd.org Received: from denrei.darkbsd.org ([127.0.0.1]) by denrei.darkbsd.org (denrei.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 2mnBmjRnI4Yr for ; Sat, 9 Mar 2013 18:46:04 +0100 (CET) Received: from [IPv6:2001:470:24:42d::42] (archer.yomi.darkbsd.org [IPv6:2001:470:24:42d::42]) (Authenticated sender: darksoul@darkbsd.org) by denrei.darkbsd.org (Postfix) with ESMTPSA id 4E3EEE98 for ; Sat, 9 Mar 2013 18:46:02 +0100 (CET) Message-ID: <513B7555.1010701@darkbsd.org> Date: Sun, 10 Mar 2013 02:45:57 +0900 From: Stephane LAPIE User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Panic loop on ZFS with 9.1-RELEASE References: <513B58B6.2090903@darkbsd.org> <513B6E1E.6080805@darkbsd.org> In-Reply-To: <513B6E1E.6080805@darkbsd.org> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigBBCFCDBBE6DBB23636CFC8BA" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 17:46:15 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBBCFCDBBE6DBB23636CFC8BA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Pinpoint analysis of the zpool on the broken vdev gives the following information : # zdb -AAA -e -mm prana 1 33 Metaslabs: vdev 1 metaslabs 145 offset spacemap free =20 --------------- ------------------- --------------- -----------= -- metaslab 33 offset 21000000000 spacemap 303 free 11.= 9G WARNING: zfs: allocating allocated segment(offset=3D2335563722752 size=3D= 1024) Assertion failed: sm->sm_space =3D=3D space (0x2f927f400 =3D=3D 0x2f927f8= 00), file /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/cddl/lib/libzpool/= =2E./../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, line 353. pid 51 (zdb), uid 0: exited on signal 6 (core dumped) Abort trap (core dumped) Just in case, root vdev 1 is made of the following devices : children[1]: type: 'raidz' id: 1 guid: 1078755695237588414 nparity: 1 metaslab_array: 175 metaslab_shift: 36 ashift: 9 asize: 10001970626560 is_log: 0 children[0]: type: 'disk' id: 0 guid: 12900041001921590764 path: '/dev/da10' phys_path: '/dev/da10' whole_disk: 0 DTL: 4127 children[1]: type: 'disk' id: 1 guid: 7211789756938666186 path: '/dev/da3' phys_path: '/dev/da3' whole_disk: 1 DTL: 4119 children[2]: type: 'disk' id: 2 guid: 12094368820342087236 path: '/dev/da5' phys_path: '/dev/da5' whole_disk: 1 DTL: 212 children[3]: type: 'disk' id: 3 guid: 6868867539761908697 path: '/dev/da4' phys_path: '/dev/da4' whole_disk: 0 DTL: 4173 children[4]: type: 'disk' id: 4 guid: 3091570768700552191 path: '/dev/da6' phys_path: '/dev/da6' whole_disk: 0 DTL: 4182 At this point I am nearly considering ripping these out and zpool importing while ignoring missing devices... :/ On 03/10/2013 02:15 AM, Stephane LAPIE wrote: > Posting a quick update. > > I ran a "zdb -emm" command to figure out what was going on, and it blew= > up in my face with an abort trap here : > - vdev 0 has 145 metaslabs, which are cleared without any problems. > - vdev 1 has 145 metaslabs, but fails in the middle : > metaslab 32 offset 20000000000 spacemap 289 free 1.64G= > segments 19509 maxsize 41.7M freepct 2%= > metaslab 33 offset 21000000000 spacemap 303 free 11.9G= > error: zfs: allocating allocated segment(offset=3D2335563722752 size=3D= 1024) > Abort trap(core dumped) > > Converting offset 2335563722752 from earlier kernel panic messages give= s > : 21fca723000, which matches the broken metaslab found by zdb. > > Is there anything I can do at this point, using zdb? > It just sounds surrealistic I have ONE broken metaslab (seemingly?) and= > that I can't recover anything... > > Cheers, > > On 03/10/2013 12:43 AM, Stephane LAPIE wrote: >> Hello list, >> >> I currently am faced with a sudden death case I can't understand at al= l, >> and I would be very appreciating of any explanation or assistance :( >> >> Here is my current kernel version : >> FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59= >> JST 2013 >> darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-= no-za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN=20 >> amd64 >> (Kernel is basically a lightened GENERIC kernel without VESA options a= nd >> unneeded controllers removed) >> >> The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrore= d >> transaction log >> >> Booting and trying to import the pool is met with : >> Solaris(panic): zfs: panic: allocating allocated >> segment(offset=3D2335563722752 size=3D1024) >> >> Booting single mode on my emergency flash card with a base OS and zpoo= l >> import -o readonly=3Don is met with : >> panic: solaris assert: zio->io_type !=3D ZIO_TYPE_WRITE || >> spa_writeable(spa), file: >> /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs= /../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, >> line: 2461 >> >> I tried zpool import -F -f, zpool import -F -f -m after removing the >> mirrored transaction log devices, but after 40s of trying to import, i= t >> just blows up. >> >> I am currently running "zdb -emm" as per the procedure suggested here = : >> http://simplex.swordsaint.net/?p=3D199 if only to get some debug infor= mation. >> >> Thanks in advance for your time. >> >> Cheers, >> >> >> --=20 >> Stephane LAPIE, EPITA SRS, Promo 2005 >> "Even when they have digital readouts, I can't understand them." >> --MegaTokyo > > --=20 > Stephane LAPIE, EPITA SRS, Promo 2005 > "Even when they have digital readouts, I can't understand them." > --MegaTokyo --------------enigBBCFCDBBE6DBB23636CFC8BA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlE7dVUACgkQDJ4OK7D3FWQVbgD/ajAVkm/QkciZuNmMT11EkSDV 3semGzseAYsYopdpI40A/0u9MYMeUjVK1r8nUaNQXBkCQhxE760uIAgJMn+Hp3w4 =Z06k -----END PGP SIGNATURE----- --------------enigBBCFCDBBE6DBB23636CFC8BA-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 18:00:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C5033C08; Sat, 9 Mar 2013 18:00:05 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 66E9627C; Sat, 9 Mar 2013 18:00:05 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r29I04Sp062160; Sat, 9 Mar 2013 13:00:04 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r29I04gL062157; Sat, 9 Mar 2013 13:00:04 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20795.30884.330015.123616@hergotha.csail.mit.edu> Date: Sat, 9 Mar 2013 13:00:04 -0500 From: Garrett Wollman To: Rick Macklem Subject: Re: NFS DRC size In-Reply-To: <1639798917.3728142.1362846452693.JavaMail.root@erie.cs.uoguelph.ca> References: <20794.38381.221980.5038@hergotha.csail.mit.edu> <1639798917.3728142.1362846452693.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Sat, 09 Mar 2013 13:00:04 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 18:00:05 -0000 < said: > around the highwater mark basically indicates this is working. If it wasn't > throwing away replies where the receipt has been ack'd at the TCP > level, the cache would grow very large, since they would only be > discarded after a loonnngg timeout (12hours unless you've changes > NFSRVCACHE_TCPTIMEOUT in sys/fs/nfs/nfs.h). That seems unreasonably large. > Well, the DRC will try to cache replies until the client's TCP layer > acknowledges receipt of the reply. It is hard to say how many replies > that is for a given TCP connection, since it is a function of the level > of concurrently (# of nfsiod threads in the FreeBSD client) > in the client. I'd guess it's somewhere between 1<->20? Nearly all our clients are Linux, so it's likely to be whatever Debian does by default. > Multiply that by the number of TCP connections from all clients and > you have about how big the server's DRC will be. (Some clients use > a single TCP connection for the client whereas others use a separate > TCP connection for each mount point.) The Debian client appears to use a single TCP connection for everything. So if I want to support 2,000 clients each with 20 requests in flight, that would suggest that I need a DRC size of 40,000, which my experience shows is not sufficient with even a much smaller number of clients. -GAWollman From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 23:40:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 10A4F932 for ; Sat, 9 Mar 2013 23:40:06 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ia0-x230.google.com (mail-ia0-x230.google.com [IPv6:2607:f8b0:4001:c02::230]) by mx1.freebsd.org (Postfix) with ESMTP id DB580FC6 for ; Sat, 9 Mar 2013 23:40:05 +0000 (UTC) Received: by mail-ia0-f176.google.com with SMTP id i18so2550075iac.21 for ; Sat, 09 Mar 2013 15:40:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:date:x-google-sender-auth:message-id :subject:from:to:content-type; bh=tSbdhVN0Scv5h8SNgRfZGSe0bIcGsJb8ZiN013uaPSg=; b=V9/ZoCPKydSzPALcvboCu3LeFMmfRZl0sJVqT2JnXdp1SiouuHvGyVwTWnTVgocM6j nRmaOs0MstOTAkWEW889M7kw0Y5LCT0EyaDbSlYkBGv9G3xgeGmHV16zQFgUsie2vaoy rOuS5R4imBlnO2n1O7Awq0XFO9J1vKxxywznC5dwjRMZM3FpjYyem43Mdi5X1a33wJ6t lwyxgC6GCkyCJ3brHzpm/75aej8/efzOxbUrfvKDEqtjKOSDj3F67V1icRWC/2jzvlyO k1hAtKNxtYnkM7+dtp1jAVl6ojTjsD8ILu7Y6Y4pgyV74TLle6xMDOF+OqOtmIgGnByS Tvqw== MIME-Version: 1.0 X-Received: by 10.50.77.230 with SMTP id v6mr3647993igw.29.1362872405603; Sat, 09 Mar 2013 15:40:05 -0800 (PST) Sender: jdavidlists@gmail.com Received: by 10.42.153.133 with HTTP; Sat, 9 Mar 2013 15:40:05 -0800 (PST) Date: Sat, 9 Mar 2013 18:40:05 -0500 X-Google-Sender-Auth: DfqW6gRooUx9xhvb5eRHfuBcvls Message-ID: Subject: FreeBSD & no single point of failure file service From: J David To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 23:40:06 -0000 Hello, I would like to build a file server with no single point of failure, and I would like to use FreeBSD and ZFS to do it. The hardware configuration we're looking at would be two servers with 4x SAS connectors and two SAS JBOD shelves. Both servers would have dual connections to each shelf. The disks would be configured in mirrored pairs, with one disk from each pair in each shelf. One pair for ZIL, one or two pairs for L2ARC, and the rest for ZFS data. We would be shooting for an active/standby configuration where the standby system is booted up but doesn't touch the bus unless/until it detects CARP failover from the master via devd, then it does a zpool import. (Even so all TCP sessions for NFS and iSCSI will get reset, which seems unavoidable but recoverable.) This will be really expensive to test, so I would be very interested if anyone has feedback on how FreeBSD will handle this type of shared-SAS hardware configuration. Thanks for any advice!