From owner-freebsd-fs@freebsd.org Sun Dec 25 04:41:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82958C855AF for ; Sun, 25 Dec 2016 04:41:09 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09CF11E5B for ; Sun, 25 Dec 2016 04:41:08 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (pD9FE9508.dip0.t-ipconnect.de [217.254.149.8]) (authenticated bits=128) by land.berklix.org (8.15.2/8.15.2) with ESMTPA id uBP3V4iO085211 for ; Sun, 25 Dec 2016 03:31:04 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id uBP3V1rl070868 for ; Sun, 25 Dec 2016 04:31:01 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id uBP3Un70004592 for ; Sun, 25 Dec 2016 04:31:01 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201612250331.uBP3Un70004592@fire.js.berklix.net> To: freebsd-fs@freebsd.org Subject: when ufs is 99% full, current seems to limit creat to 28672 bytes From: "Julian H. Stacey" Organization: http://berklix.eu BSD Linux Unix Consultants, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ Date: Sun, 25 Dec 2016 04:30:49 +0100 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Dec 2016 04:41:09 -0000 A puzzle below, that I think I've solved, but comments welcome: When ufs is 99% full, current limits creat to 28672 bytes. On current, mount /dev/ada0s4f on /data (ufs, NFS exported, local, soft-updates) /bin/df /data Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ada0s4f 868629268 863272592 5356676 99% /data csh foreach i ( `cd /host/fire/0s4/ftp/pub/FreeBSD/mail/ports ; /bin/ls -1 [0-9]*`) # Above just a trick to generate 3000 numeric names. # ( testblock -v -b 512 tb.$i ) # My little C program # http://berklix.com/~jhs/src/bsd/jhs/bin/public/testblock/ # OR a simpler standard dd ... ( dd if=/dev/zero of=dd.$i ) end Produces loads of little files of size 28672 bytes = 0x7000 A foreground fsck of FS shows no errors. No ZFS in use. Where does all the extra space come from ? Why does it keep allowing more little files ? But not files bigger than 7K ? The last 5 Gig seems unusable space not in the meta space of normal free blocks for big files. Presumably instead in the payload of initial data space pre allocated along with each unused inode ? So 1K for inode & 7K data ? ffsinfo | more bsize int32_t 0x00008000 fsize int32_t 0x00001000 df -i /dev/ada0s4f Filesystem 1K-blocks Used Avail Capacity iused ifree %iused /dev/ada0s4f 868629268 863272592 5356676 99% 9595016 102683126 9% dc 102683126 28672 * p # 2 944 130 588 672 3G not 5.3G as above from df, but seems not too dissimilar. If this 0x7K phenomena is not documented in manuals somewhere ? it perhaps should be ? How I noticed: I wanted to soak up every last block on FS, (cos Ive been having block device errors, so I wanted to force a read write on all unused blocks to hopefully get IDE drive to re-allocate anything flakey.) ada0: 953869MB (1953525168 512 byte sectors) Tips very welcome on [usb scsi] Commands/Manuals/ ports/ to either: rescan partition (I could achive a crude reallocation of discovered bad blocks, using drivers inside disk or FreeBSD, by writing new test data on my partition with dd or my testblock -b 60k /dev/ada0s4f reformat disk. Maybe man fsck See Also should ref camcontrol ? Cheers, Julian -- Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#stolen_votes From owner-freebsd-fs@freebsd.org Sun Dec 25 10:11:54 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C296C90229 for ; Sun, 25 Dec 2016 10:11:54 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wj0-x244.google.com (mail-wj0-x244.google.com [IPv6:2a00:1450:400c:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D75BD1279 for ; Sun, 25 Dec 2016 10:11:53 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wj0-x244.google.com with SMTP id j10so45728022wjb.3 for ; Sun, 25 Dec 2016 02:11:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=9eoC4Jcfvv22zJV+YREk5iHOsdkTTnkH1GbgZAHN1Cc=; b=n4S0gGmndSjvmT8mivX/A5bMgtr0ycg0y2SFji0uxr3jwdldx0s5xgxUFM/bgUTTYW /pYlW8Xiauc6+VgWkCBBruBM9VYZ+fHh4tl2gzE2MQPn94DUGHE7FlkghJArJl3O0VL9 7zlbITePZP+TrO2iKg0CuuY2/YUoy+hXvkSIjCPzPqPU5UzQbY6FEfb76zdE/Qgv2+SC OEJiPVrJirBIU9G2tavWuorHkaYG3+QcuZ5uieSft7swCI2P+wnklR4eMOHM3jyjQ7mE vRi860wcDm60vytemA6jNkTZuptoUCpJfThCjG/g5feCuB3mRY6ivSLawsnDD0mR3OI/ Pkjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=9eoC4Jcfvv22zJV+YREk5iHOsdkTTnkH1GbgZAHN1Cc=; b=FM0TeyNSurtpm1tWDJtAJARY+IqzsDyTNTSgi2ED22ZE+yGwULMmQXIshXZ/7UJtL7 VszU953ny8MGn4IwJSbDeIy23e0mleNO9/krHZZW1DbepRD9MdXChu3KoL1BIXZoP3PB uMojoGtESh5GYrDl4mO957xPPKdpVrlTAMCRsjqLlDf7cuLg5jaeKX7inuo66jlsSa9E smGoeZKRwVcQacOOIgI/MnoOKbkDTGNRCwW534wvJtO72+xnUm/wYOJKfL+darHcC/AP jNcR876D+vejKtm8sWaQZAIwvRLz9qkS4fmnHXxfZhpRPD3UGPeLwYSEyLKroy0weiLP kXTw== X-Gm-Message-State: AIkVDXIjPY4djZcfdWDQj/GJLWvJA3/6OSE+P4jNQwmrCiQ7+zYAfGjG3ImI28ikAzw/tw== X-Received: by 10.194.5.165 with SMTP id t5mr21658587wjt.107.1482660711120; Sun, 25 Dec 2016 02:11:51 -0800 (PST) Received: from ernst.home (p578E2DF8.dip0.t-ipconnect.de. [87.142.45.248]) by smtp.gmail.com with ESMTPSA id jz1sm23960646wjc.38.2016.12.25.02.11.49 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 25 Dec 2016 02:11:50 -0800 (PST) Date: Sun, 25 Dec 2016 11:11:48 +0100 From: Gary Jennejohn To: freebsd-fs@freebsd.org Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes Message-ID: <20161225111148.6ee769b8@ernst.home> In-Reply-To: <201612250331.uBP3Un70004592@fire.js.berklix.net> References: <201612250331.uBP3Un70004592@fire.js.berklix.net> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Dec 2016 10:11:54 -0000 On Sun, 25 Dec 2016 04:30:49 +0100 "Julian H. Stacey" wrote: > A puzzle below, that I think I've solved, but comments welcome: > When ufs is 99% full, current limits creat to 28672 bytes. > > On current, > mount > /dev/ada0s4f on /data (ufs, NFS exported, local, soft-updates) > > /bin/df /data > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/ada0s4f 868629268 863272592 5356676 99% /data > > csh > foreach i ( `cd /host/fire/0s4/ftp/pub/FreeBSD/mail/ports ; /bin/ls -1 [0-9]*`) > # Above just a trick to generate 3000 numeric names. > # ( testblock -v -b 512 tb.$i ) # My little C program > # http://berklix.com/~jhs/src/bsd/jhs/bin/public/testblock/ > # OR a simpler standard dd ... > ( dd if=/dev/zero of=dd.$i ) > end > > Produces loads of little files of size 28672 bytes = 0x7000 > > A foreground fsck of FS shows no errors. No ZFS in use. Where > does all the extra space come from ? Why does it keep allowing > more little files ? But not files bigger than 7K ? > > The last 5 Gig seems unusable space not in the meta space of normal > free blocks for big files. Presumably instead in the payload of > initial data space pre allocated along with each unused inode ? So > 1K for inode & 7K data ? > > ffsinfo | more > bsize int32_t 0x00008000 > fsize int32_t 0x00001000 > > > df -i /dev/ada0s4f > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused > /dev/ada0s4f 868629268 863272592 5356676 99% 9595016 102683126 9% > > dc 102683126 28672 * p # 2 944 130 588 672 > 3G not 5.3G as above from df, but seems not too dissimilar. > > If this 0x7K phenomena is not documented in manuals somewhere ? it > perhaps should be ? > > How I noticed: > I wanted to soak up every last block on FS, (cos Ive been having > block device errors, so I wanted to force a read write on all > unused blocks to hopefully get IDE drive to re-allocate anything flakey.) > ada0: 953869MB (1953525168 512 byte sectors) > > Tips very welcome on [usb scsi] Commands/Manuals/ ports/ to either: > rescan partition > (I could achive a crude reallocation of discovered bad blocks, > using drivers inside disk or FreeBSD, by writing new test data > on my partition with dd or my testblock -b 60k /dev/ada0s4f > reformat disk. > Maybe man fsck See Also should ref camcontrol ? > I suspect this ia a result of how UFS is designed. Did you use the standard options for block and fragment size? How about inodes? Is the file system UFS1 or UFS2? UFS is a very compex bit of software and I imagine there are all kinds of interesting surprises when the file system is 99% full. Anyway, the newfs man page may provide some clues. Or look at Wikipedia, there's a UFS entry there, but it doesn't go into the gorey details. -- Gary Jennejohn From owner-freebsd-fs@freebsd.org Sun Dec 25 21:00:04 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AEE7C90AEC for ; Sun, 25 Dec 2016 21:00:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 516EB1DE0 for ; Sun, 25 Dec 2016 21:00:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBPL01jG000583 for ; Sun, 25 Dec 2016 21:00:04 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201612252100.uBPL01jG000583@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 25 Dec 2016 21:00:04 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Dec 2016 21:00:04 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 203419 | solaris assert: (dn->dn_phys->dn_nlevels == 0 && Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Dec 26 03:27:09 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12C3FC90B3D for ; Mon, 26 Dec 2016 03:27:09 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by mx1.freebsd.org (Postfix) with ESMTP id 9BCB6BE4 for ; Mon, 26 Dec 2016 03:27:08 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ppp118-210-147-181.bras1.adl6.internode.on.net (HELO leader.local) ([118.210.147.181]) by ipmail05.adl6.internode.on.net with ESMTP; 26 Dec 2016 13:51:59 +1030 Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes To: gljennjohn@gmail.com, freebsd-fs@freebsd.org References: <201612250331.uBP3Un70004592@fire.js.berklix.net> <20161225111148.6ee769b8@ernst.home> From: Shane Ambler Message-ID: <86167004-c918-f879-e8d9-9abd38e94499@ShaneWare.Biz> Date: Mon, 26 Dec 2016 13:51:56 +1030 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161225111148.6ee769b8@ernst.home> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 03:27:09 -0000 On 25/12/2016 20:41, Gary Jennejohn wrote: > On Sun, 25 Dec 2016 04:30:49 +0100 > "Julian H. Stacey" wrote: > >> A puzzle below, that I think I've solved, but comments welcome: >> When ufs is 99% full, current limits creat to 28672 bytes. >> >> Produces loads of little files of size 28672 bytes = 0x7000 >> >> A foreground fsck of FS shows no errors. No ZFS in use. Where >> does all the extra space come from ? Why does it keep allowing >> more little files ? But not files bigger than 7K ? > > I suspect this ia a result of how UFS is designed. Did you use the > standard options for block and fragment size? How about inodes? > Been a while since I looked into filesystem specs but as I recall - My guess is free block fragmentation - I believe on UFS each file is kept in a continuous strip of blocks on disk. If blocks are 8K and the system is unable to find more than one free block between existing files it would prevent a new file over 8k being created, even though there may be MB of free blocks available throughout the filesystem. At a certain fill factor moving files to create enough free space becomes too costly. There is also fragmentation, I understand that multiple (or just two) small files can be stored in one block, so a 3k file can be added to a block that already has a 3k file on it, but it may be unable to find space to create a new 12k file. -- FreeBSD - the place to B...Storing Data Shane Ambler From owner-freebsd-fs@freebsd.org Mon Dec 26 06:29:08 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 785B1C84DDC for ; Mon, 26 Dec 2016 06:29:08 +0000 (UTC) (envelope-from www-data@effectencompagnie.nl) Received: from smtp1.global-e.nl (smtp1.global-e.nl [77.94.242.6]) by mx1.freebsd.org (Postfix) with ESMTP id 3F6C53D5 for ; Mon, 26 Dec 2016 06:29:07 +0000 (UTC) (envelope-from www-data@effectencompagnie.nl) Received: from VI-VEC-WEB3 (mobile.effectencompagnie.nl [77.94.245.35]) by smtp1.global-e.nl (Postfix) with ESMTP id 886E22821D1 for ; Mon, 26 Dec 2016 07:29:00 +0100 (CET) Received: by VI-VEC-WEB3 (Postfix, from userid 33) id 13B6BE4F09; Mon, 26 Dec 2016 07:29:07 +0100 (CET) To: freebsd-fs@freebsd.org Subject: Notification status of your delivery (FedEx 0000310212) X-PHP-Originating-Script: 33:post.php(4) : regexp code(1) : eval()'d code(17) : eval()'d code Date: Mon, 26 Dec 2016 07:29:07 +0100 MIME-Version: 1.0 Message-ID: Reply-To: "FedEx Priority Solutions" From: "FedEx Priority Solutions" Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 06:29:08 -0000 Dear Customer, FedEx courier was unable to contact you for your parcel delivery. You can download the shipment label attached! Best regards, Lester Hurst, Parcels Delivery Agent. From owner-freebsd-fs@freebsd.org Mon Dec 26 09:11:28 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F94FC8F037 for ; Mon, 26 Dec 2016 09:11:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2DEB1CAC for ; Mon, 26 Dec 2016 09:11:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uBQ9BHWa053664 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 26 Dec 2016 11:11:17 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uBQ9BHWa053664 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uBQ9BH4M053663; Mon, 26 Dec 2016 11:11:17 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 26 Dec 2016 11:11:16 +0200 From: Konstantin Belousov To: Shane Ambler Cc: gljennjohn@gmail.com, freebsd-fs@freebsd.org Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes Message-ID: <20161226091116.GO94325@kib.kiev.ua> References: <201612250331.uBP3Un70004592@fire.js.berklix.net> <20161225111148.6ee769b8@ernst.home> <86167004-c918-f879-e8d9-9abd38e94499@ShaneWare.Biz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86167004-c918-f879-e8d9-9abd38e94499@ShaneWare.Biz> User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 09:11:28 -0000 On Mon, Dec 26, 2016 at 01:51:56PM +1030, Shane Ambler wrote: > On 25/12/2016 20:41, Gary Jennejohn wrote: > > On Sun, 25 Dec 2016 04:30:49 +0100 > > "Julian H. Stacey" wrote: > > > >> A puzzle below, that I think I've solved, but comments welcome: > >> When ufs is 99% full, current limits creat to 28672 bytes. > >> > > >> Produces loads of little files of size 28672 bytes = 0x7000 > >> > >> A foreground fsck of FS shows no errors. No ZFS in use. Where > >> does all the extra space come from ? Why does it keep allowing > >> more little files ? But not files bigger than 7K ? > > > > > I suspect this ia a result of how UFS is designed. Did you use the > > standard options for block and fragment size? How about inodes? > > > > Been a while since I looked into filesystem specs but as I recall - > > My guess is free block fragmentation - I believe on UFS each file is > kept in a continuous strip of blocks on disk. This is absolutely false statements. UFS does not require any continuosity in the blocks for file. It does make some measures to reduce fragmentation, in particular, so that single op could read or write MAXPHYS bytes of data in one io, but there is nothing which would prevent it from working on not adjacent blocks. > If blocks are 8K and the > system is unable to find more than one free block between existing files > it would prevent a new file over 8k being created, even though there may > be MB of free blocks available throughout the filesystem. At a certain > fill factor moving files to create enough free space becomes too costly. > > There is also fragmentation, I understand that multiple (or just two) > small files can be stored in one block, so a 3k file can be added to a > block that already has a 3k file on it, but it may be unable to find > space to create a new 12k file. dumpfs(8) allows to look at the UFS allocation bitmaps tracking free clusters and inodes in relatively human-friendly form. From owner-freebsd-fs@freebsd.org Mon Dec 26 17:21:20 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35B4BC92A55 for ; Mon, 26 Dec 2016 17:21:20 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC4521164 for ; Mon, 26 Dec 2016 17:21:18 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p5B2263FD.dip0.t-ipconnect.de [91.34.99.253]) (authenticated bits=128) by land.berklix.org (8.15.2/8.15.2) with ESMTPA id uBQHLGwL069537 for ; Mon, 26 Dec 2016 17:21:16 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id uBQHLBb5077399 for ; Mon, 26 Dec 2016 18:21:11 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id uBQHKxcJ056264 for ; Mon, 26 Dec 2016 18:21:11 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201612261721.uBQHKxcJ056264@fire.js.berklix.net> To: freebsd-fs@freebsd.org Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Sun, 25 Dec 2016 11:11:48 +0100." <20161225111148.6ee769b8@ernst.home> Date: Mon, 26 Dec 2016 18:20:59 +0100 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 17:21:20 -0000 Gary J wrote: > I suspect this ia a result of how UFS is designed. Yes. > Did you use the > standard options for block and fragment size? How about inodes? Yes, I created that partition years ago, I pretty much always just use newfs unless experimenting perhaps for a small partition on USB stick or CDROM (& then I'd normally delete), so it would have been a newfs almost guaranteed with no parameters, default. > Is the file system UFS1 or UFS2? UFS2 > UFS is a very compex bit of software and I imagine there are all > kinds of interesting surprises when the file system is 99% full. > > Anyway, the newfs man page may provide some clues. Or look at > Wikipedia, there's a UFS entry there, but it doesn't go into the gorey > details. I'll read https://en.wikipedia.org/wiki/Unix_File_System Konstantin B wrote > dumpfs(8) allows to look at .... Top of dumpfs: magic 19540119 (UFS2) time Sun Dec 25 03:19:13 2016 superblock location 65536 id [ 548daf7f b34ae147 ] ncg 1399 size 224197115 blocks 217157317 bsize 32768 shift 15 mask 0xffff8000 fsize 4096 shift 12 mask 0xfffff000 frag 8 shift 3 fsbtodb 3 minfree 0% optim space symlinklen 120 maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 nbfree 0 ndir 1555427 nifree 102683126 nffree 1339169 bpg 20035 fpg 160280 ipg 80256 unrefs 0 nindir 4096 inopb 128 maxfilesize 2252349704110079 sbsize 4096 cgsize 32768 csaddr 5056 cssize 24576 sblkno 24 cblkno 32 iblkno 40 dblkno 5056 cgrotor 622 fmod 0 ronly 0 clean 1 metaspace 6408 avgfpdir 64 avgfilesize 16384 flags soft-updates fsmnt /data volname swuid 0 providersize 224197115 Thanks All, Cheers, Julian -- Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#stolen_votes From owner-freebsd-fs@freebsd.org Mon Dec 26 23:13:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB265C9269C for ; Mon, 26 Dec 2016 23:13:34 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6E5F130D for ; Mon, 26 Dec 2016 23:13:34 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id uBQNDNrX030075; Mon, 26 Dec 2016 15:13:23 -0800 (PST) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201612262313.uBQNDNrX030075@chez.mckusick.com> From: Kirk McKusick To: "Julian H. Stacey" Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes cc: freebsd-fs@freebsd.org In-reply-to: <201612261721.uBQHKxcJ056264@fire.js.berklix.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <30073.1482794003.1@chez.mckusick.com> Content-Transfer-Encoding: quoted-printable Date: Mon, 26 Dec 2016 15:13:23 -0800 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 23:13:35 -0000 > To: freebsd-fs@freebsd.org > Subject: Re: when ufs is 99% full, current seems to limit creat to 28672= bytes > From: "Julian H. Stacey" > Date: Mon, 26 Dec 2016 18:20:59 +0100 > = > Gary J wrote: > = > > I suspect this ia a result of how UFS is designed. > = > Yes. > = > > Did you use the > > standard options for block and fragment size? How about inodes? > = > Yes, I created that partition years ago, I pretty much always just > use newfs unless experimenting perhaps for a small partition on USB > stick or CDROM (& then I'd normally delete), so it would have been > a newfs almost guaranteed with no parameters, default. > = > > Is the file system UFS1 or UFS2? > = > UFS2 > = > = > > UFS is a very compex bit of software and I imagine there are all > > kinds of interesting surprises when the file system is 99% full. > > = > > Anyway, the newfs man page may provide some clues. Or look at > > Wikipedia, there's a UFS entry there, but it doesn't go into the gorey > > details. > = > I'll read https://en.wikipedia.org/wiki/Unix_File_System > = > Konstantin B wrote > > dumpfs(8) allows to look at .... > = > Top of dumpfs: > magic 19540119 (UFS2) time Sun Dec 25 03:19:13 2016 > superblock location 65536 id [ 548daf7f b34ae147 ] > ncg 1399 size 224197115 blocks 217157317 > bsize 32768 shift 15 mask 0xffff8000 > fsize 4096 shift 12 mask 0xfffff000 > frag 8 shift 3 fsbtodb 3 > minfree 0% optim space symlinklen 120 > maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 > nbfree 0 ndir 1555427 nifree 102683126 nffree 133916= 9 > bpg 20035 fpg 160280 ipg 80256 unrefs 0 > nindir 4096 inopb 128 maxfilesize 2252349704110079 > sbsize 4096 cgsize 32768 csaddr 5056 cssize 24576 > sblkno 24 cblkno 32 iblkno 40 dblkno 5056 > cgrotor 622 fmod 0 ronly 0 clean 1 > metaspace 6408 avgfpdir 64 avgfilesize 16384 > flags soft-updates = > fsmnt /data > volname swuid 0 providersize 224197115 > = > Thanks All, > = > Cheers, > Julian > -- = > Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich > Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-pri= ntable. > http://berklix.eu/brexit/#stolen_votes You HAVE made a change to the filesystem parameters. You have set minfree to 0%. This means that you are not allowing UFS to keep any blocks in reserve. At this point you have no full sized blocks left in the filesystem (nbfree =3D=3D 0). Thus the only thing left in = the filesystem is fragments. A file in UFS is made up of zero or more full-sized blocks and at most one fragment. When you have used up all of the full-sized blocks you cannot create files larger than the largest available fragment. So if you continue with your experiment you will find that the biggest file you can create will eventually fall to 1K. So long as you keep minfree at 1% or higher, you will not hit this condition which is why tunefs warns you not to set minfree to absurdly low values. If you want to ensure that your filesystem can be filled to the last block, you should set the blocksize and the fragment size to the same value. This will ensure that your filesystem has only full-sized blocks and never creates fragments. Thus you will be able to allocate files until it is compelely full. Kirk McKusick From owner-freebsd-fs@freebsd.org Tue Dec 27 17:30:51 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41675C92B04 for ; Tue, 27 Dec 2016 17:30:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30C1C1660 for ; Tue, 27 Dec 2016 17:30:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBRHUp1O079578 for ; Tue, 27 Dec 2016 17:30:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212318] sysutils/fusefs-libs: fails to unmount due to a uniq dev node Date: Tue, 27 Dec 2016 17:30:51 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ben.rubson@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-ports-bugs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2016 17:30:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212318 --- Comment #10 from Ben RUBSON --- Yep I think so too. I also just saw that fuse_unmount_compat22 has gone from libfuse 3.0. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Dec 27 23:42:21 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6AEA4C9425A for ; Tue, 27 Dec 2016 23:42:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A4171812 for ; Tue, 27 Dec 2016 23:42:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBRNgLUk080558 for ; Tue, 27 Dec 2016 23:42:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 141897] [msdosfs] [panic] Kernel panic. msdofs: file name length 266 to large. Date: Tue, 27 Dec 2016 23:42:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jilles@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: DUPLICATE X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2016 23:42:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D141897 Jilles Tjoelker changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |DUPLICATE Status|In Progress |Closed CC| |jilles@FreeBSD.org --- Comment #2 from Jilles Tjoelker --- Fixed by using the 8.3 name if the converted long name is too long. *** This bug has been marked as a duplicate of bug 204643 *** --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Dec 27 23:42:23 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50192C94264 for ; Tue, 27 Dec 2016 23:42:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3BBFB181E for ; Tue, 27 Dec 2016 23:42:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBRNgLUo080558 for ; Tue, 27 Dec 2016 23:42:23 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204643] [msdosfs] [panic] Crash while accessing files with large, non-english names Date: Tue, 27 Dec 2016 23:42:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: patch, security X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jilles@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2016 23:42:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D204643 Jilles Tjoelker changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nsand@sura.ru --- Comment #8 from Jilles Tjoelker --- *** Bug 141897 has been marked as a duplicate of this bug. *** --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 28 07:03:37 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39EF6C94BCB for ; Wed, 28 Dec 2016 07:03:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 1A0361AC6 for ; Wed, 28 Dec 2016 07:03:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1959EC94BCA; Wed, 28 Dec 2016 07:03:37 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18FA0C94BC9 for ; Wed, 28 Dec 2016 07:03:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wj0-x243.google.com (mail-wj0-x243.google.com [IPv6:2a00:1450:400c:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A402A1AC4 for ; Wed, 28 Dec 2016 07:03:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wj0-x243.google.com with SMTP id kp2so53448817wjc.0 for ; Tue, 27 Dec 2016 23:03:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=pLSpKqPaCnvI39WITE7KgAj1RIQizxtB20s0WgdqifY=; b=RNJw5dJz1L5Qf05CooyGuBLQwKXv7YYmfv/UJOD8LTdM+iybK6B8lOUTXUz2NwcKFJ GDOHFQ7USIvIT17CU7zYwgHBokq2MYA/5YUymldYVlUhjnbcas5RjMMA8J4jHKZbxJy4 K34mCNrpJiOTX8m9OA6vW4GNQSaCL2NT6GCbjVwtcDgPrenEpLc2sE9/3pT/9ynoRncr T+DGtiTHF+APapqvz34qWd6XWUcVyvVDFVHrwInbvh6X/hDdpkrk4VQjkdAtWk4Mg9Dn LUlFX4Wsgii/bfTsd7hnBICFgFoHgMxuv05TzssTHeB/W8GuY7xWTm+MA4lBcuY3CC6c pong== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=pLSpKqPaCnvI39WITE7KgAj1RIQizxtB20s0WgdqifY=; b=VJMwmvxM+edX5nvWFfeZRDUI/rN1zWpgg0oZIPZkXH9hJmdc7A9BqymgYH9+EqodVS pNNwX17t81jAA8uw6vhe6nsoWf2Voc14+4dSseVEgGYGoMh3tjOwg9CH5qcduLcyqOhh q2ej5NFIi2ydFwCjxtJC84kzmmONKMwBMDWZ8UcfMb1KoKYQ4BRNsFj/xbtxxXfRLZ33 oVbYBcL0j0VPYQNCidr1v1oYmGxBnK6cEKVLtMcyyHpfCZoE5imStNBCV7j7tHyuXFHa l3rmeUyo6UP89X/+TtcEwDsWMHeudqKidfbpJ5AjPOtgBG86zWDoh9k8maCxXrbq3BN6 UFMA== X-Gm-Message-State: AIkVDXLdZxYnlAn72OMlw1natyTqyiEZmgYLm7vWuoYm0aNQVeuhy7iArIk0luoWx9xU5w== X-Received: by 10.194.140.234 with SMTP id rj10mr24526999wjb.39.1482908614669; Tue, 27 Dec 2016 23:03:34 -0800 (PST) Received: from ben.home (LFbn-1-7159-4.w90-116.abo.wanadoo.fr. [90.116.90.4]) by smtp.gmail.com with ESMTPSA id v3sm62884904wjp.13.2016.12.27.23.03.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 27 Dec 2016 23:03:34 -0800 (PST) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Force ZFS to always look for devices in a defined location Message-Id: <4E030EC6-39AC-484C-BA53-2C8CA0E79B84@gmail.com> Date: Wed, 28 Dec 2016 08:03:35 +0100 To: fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 07:03:37 -0000 Hello, My zpools are made of devices from /dev/label. Example : NAME STATE READ WRITE CKSUM my-zpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/slotone ONLINE 0 0 0 label/slottwo ONLINE 0 0 0 (...) For some reasons, it may happen that access to some devices fails and = devices (quickly) toggle between ONLINE and UNAVAIL/REMOVED state. Then, sometimes, ZFS re-use the device using the /dev/da path, instead = of waiting for the /dev/label path to be rediscovered by the system. It then leads to something like this, with is not nice, and not really = correct (as last sector of /dev/da contains the label) : NAME STATE READ WRITE CKSUM my-zpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/slotone ONLINE 0 0 0 da14 ONLINE 0 0 0 (...) My question is then, how to force ZFS to look for devices in a defined = location ? (/dev/label/ in this example) I tried to offline / online impacted devices (da*), but it does not = work, they do not come back with their /dev/label path. Zpool export / import works, but of course this is something we would = like to avoid. Many thanks ! Ben From owner-freebsd-fs@freebsd.org Wed Dec 28 09:30:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AD1EC94C58; Wed, 28 Dec 2016 09:30:25 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3266013D9; Wed, 28 Dec 2016 09:30:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA05639; Wed, 28 Dec 2016 11:30:15 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1cMAYp-000LnF-F1; Wed, 28 Dec 2016 11:30:15 +0200 To: "freebsd-arch@freebsd.org" , freebsd-fs From: Andriy Gapon Subject: INVARIANTS vs DIAGNOSTIC % lf_advlockasync Message-ID: <2225968a-7bce-b100-f3fa-a5e2eb8b9f47@FreeBSD.org> Date: Wed, 28 Dec 2016 11:28:54 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 09:30:25 -0000 I wonder if there are any guidelines on when to use INVARIANTS vs DIAGNOSTIC vs something else. Should the amount of output be taken into account? Or the performance impact? Or just the common sense? What I really mean is that if some sanity check could be rather expensive (e.g. it needs to iterate over a potentially long list), what option should be used to enabled it? I ask this question with one particular case in mind. lf_advlockasync() has a block of code under INVARIANTS with a loop over a list that has a nested loop over the same list for pair-wise checks. What's worse is that that code is executed with a lock held and that lock can potentially be highly contended (ls_lock). In our test environment we can observe the lock being held for as much as 125 milliseconds resulting in a huge backlog on the lock. (Even though the requested advisory locks are all shared locks and unlocks.) So, we would like to reduce the performance hit in that code, but still have the benefits of INVARIANTS enabled over all. Any suggestions are welcome. Thank you. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Dec 28 11:56:53 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7A7BC94DF4; Wed, 28 Dec 2016 11:56:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 58D7B1250; Wed, 28 Dec 2016 11:56:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uBSBukm6036421 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 28 Dec 2016 13:56:46 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uBSBukm6036421 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uBSBuk7e036420; Wed, 28 Dec 2016 13:56:46 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 28 Dec 2016 13:56:46 +0200 From: Konstantin Belousov To: Andriy Gapon Cc: "freebsd-arch@freebsd.org" , freebsd-fs Subject: Re: INVARIANTS vs DIAGNOSTIC % lf_advlockasync Message-ID: <20161228115646.GU94325@kib.kiev.ua> References: <2225968a-7bce-b100-f3fa-a5e2eb8b9f47@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2225968a-7bce-b100-f3fa-a5e2eb8b9f47@FreeBSD.org> User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 11:56:53 -0000 On Wed, Dec 28, 2016 at 11:28:54AM +0200, Andriy Gapon wrote: > > I wonder if there are any guidelines on when to use INVARIANTS vs DIAGNOSTIC vs > something else. Should the amount of output be taken into account? Or the > performance impact? Or just the common sense? > > What I really mean is that if some sanity check could be rather expensive (e.g. > it needs to iterate over a potentially long list), what option should be used to > enabled it? > > I ask this question with one particular case in mind. > lf_advlockasync() has a block of code under INVARIANTS with a loop over a list > that has a nested loop over the same list for pair-wise checks. > What's worse is that that code is executed with a lock held and that lock can > potentially be highly contended (ls_lock). > In our test environment we can observe the lock being held for as much as 125 > milliseconds resulting in a huge backlog on the lock. (Even though the > requested advisory locks are all shared locks and unlocks.) There are at least two blocks of code in kern_lockf.c that you might want to conditionally enable, one is lf_advlockasync(), another in the beginning of lf_add_edge(). > > So, we would like to reduce the performance hit in that code, but still have the > benefits of INVARIANTS enabled over all. > We have a precedent with DIAGNOSTIC enabling very heavy weight checks, see kern/subr_vmem.c enable_vmem_check sysctl. From owner-freebsd-fs@freebsd.org Wed Dec 28 15:42:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E16AC94C8D for ; Wed, 28 Dec 2016 15:42:57 +0000 (UTC) (envelope-from jrm@ftfl.ca) Received: from mail-qt0-x22c.google.com (mail-qt0-x22c.google.com [IPv6:2607:f8b0:400d:c0d::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B8081ECC for ; Wed, 28 Dec 2016 15:42:57 +0000 (UTC) (envelope-from jrm@ftfl.ca) Received: by mail-qt0-x22c.google.com with SMTP id c47so352549808qtc.2 for ; Wed, 28 Dec 2016 07:42:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ftfl.ca; s=google; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=Px73QtQ/oCqHwGQlBag8KNKrMlmO3afjNvmQ9OZIwcg=; b=A0nFs0cC8sccG04zhaPOgYNmUUYi8uImTZc+EX6j4MKCYpvMbGP4LNqHDSJfUcJe04 hluA0CMAcHitqyDSP601TnqZmk7WdQXykFehEm2l/j3olXXP6SbICkMBZWJvXA5+yhc0 eINIf/Wy8aR1ZxiH1w9ohjIU2v3U9IXPcpsb8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=Px73QtQ/oCqHwGQlBag8KNKrMlmO3afjNvmQ9OZIwcg=; b=rZhaQYSm0uWJ8esOKBMyZp/vhnLONrd93Q9Q15/48IBuVyXZf0I2M+SIIE49BSMIbq wHpmL/ZVNtvNHwQPoGmFd8AYNc3I6yj/gRTiA2Sgoe/PdsAzp/nMYO59uewCtN9kIbpX b+CvDOwG/gqHAjz5ABsGq2wDtkSDcKM7bxT9wqiKjotp5ZgukQkbuaTP0ObkGhzvH9Ah NymMob3JR4KTr6e5bZIba8yTN+h/Hb9n1/YoGhZxTCkcdwKFZKzOETwpnXI+LGAnVwi/ tcZqNIsgvzXtrpk1YQh32zj0JD4sSEItoU4ydvTBhbZFsvQATJN86lM3ss90Ion2PsmP O/7Q== X-Gm-Message-State: AIkVDXI+bgtotKWx/TPMF+jOxDxO0KP17awQuxKRw4t/xZ6/rdlwKyZdq3iMi2hgGLcmUA== X-Received: by 10.237.53.102 with SMTP id b35mr38298516qte.266.1482939776123; Wed, 28 Dec 2016 07:42:56 -0800 (PST) Received: from phe.ftfl.ca.ftfl.ca (hlfxns017vw-047055140230.dhcp-dynamic.FibreOp.ns.bellaliant.net. [47.55.140.230]) by smtp.gmail.com with ESMTPSA id h47sm30992855qtc.27.2016.12.28.07.42.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 28 Dec 2016 07:42:55 -0800 (PST) From: Joseph Mingrone To: freebsd-fs@freebsd.org Cc: Dmitry Marakasov , Steven Hartland Subject: Re: ZFS resilver from disk with bad sectors constantly restarts References: <20151030103614.GL57666@hades.panopticon> Date: Wed, 28 Dec 2016 11:42:53 -0400 In-Reply-To: <20151030103614.GL57666@hades.panopticon> (Dmitry Marakasov's message of "Fri, 30 Oct 2015 13:36:14 +0300") Message-ID: <86zijgf4aq.fsf@ftfl.ca> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 15:42:57 -0000 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Dmitry Marakasov writes: > I've just got a case where resilvering a new replacement disk in raidz2 > never finished. > The problem: one disk in raidz is failing by having a large number of > unreadable sectors. It's replaced with a spare. Resilver though is > constantly restarted with log full of read error from bad disk.=20 > It looks like this: > --- > pool: spool > state: ONLINE > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Wed Oct 28 05:26:28 2015 > 369G scanned out of 9,87T at 123M/s, 22h29m to go > 41,4G resilvered, 3,65% done > config: > NAME STATE READ WRITE CKSUM > spool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ada0 ONLINE 0 0 0 > ada1 ONLINE 0 0 0 > spare-2 ONLINE 0 0 733 > ada11 ONLINE 0 0 0 > ada2 ONLINE 0 0 0 (resilvering) > raidz1-1 ONLINE 0 0 0 > ada3 ONLINE 0 0 0 > ada4 ONLINE 0 0 0 > ada5 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > ada6 ONLINE 0 0 0 > ada7 ONLINE 0 0 0 > ada10 ONLINE 0 0 0 > spares > 588540573008830286 INUSE was /dev/ada2 > errors: No known data errors > --- > `resilver in progress since' date is constantly reset, so resilved > progress cannot pass beyond 5% or so. My guess is that it happens on > read errors on ada11. I think I've seen (resilvering) on ada11 line > couple of times. > In the end I've had to offline ada11 and after that resilver completed > in under 16 hours. However the situation doesn't seem normal, as I'd > prefer to not lose redundancy with offlining dying disk and still be > able to use it for resilvering (imagine there were bad sectors on ada0/1 > as well, but not intersecting with bad sectors on ada11), or at least > some more verbose indication of why the resilver is constantly restarted. > I should also note that's outdated FreeBSD 9.1, so maybe that problem > was fixed already. We have been dealing with, what seems to be, the same issue on 11.0-RELEASE= with a two-raidz1-vdev pool. You said that your issue was with a raidz2, but yo= ur zpool status output shows raidz1. The problem disk had checksum mismatches= and smart was reporting errors, but it was still online. The resilver would ma= ke it through many hours, but then restart. This loop went on for a few days. A= s in your case, after offlining the problem disk, the replacement finished. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJYY919XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ1NUIwOTNBNzI2QzM4ODU1NzEyMkJBRDUz NkE0MEM4M0IwRDZFRjlFAAoJEDakDIOw1u+eBWEQAJVFWKPF9qvJTBvbt0JSsKw7 tgFYsa/EMD59rJpyIW+ASpfLczgDBgrGZCm2LCQadbqjnCgjKhDv6/Uo4kviRoaq Xg7AWOsYAQ09B3aS9zDDySaBDu4oDFCCQFiWYqzh5xm9kVnhVvCPKTywFbFmiLq0 8WZa9fvD7pYpDFMgFLxM83Z/MWclN2Cyk+SX9g87hezcc2Sr6XRXpw2S+w/KdUIr kw1ZmuDh3nqNObjgkYAqxcNc0XLEr6J4HPEKKK76HXDp/WrQ7H0peE9NmkDYQd4W 2HUKE6xPXQQDYDCQCr5Ywg4N/VJovcOQXx7ulD2SxVzj6oRUu+8suMFubOA69dGN yNgpgqiRYOxXpsUzX8PDUglNRySLQ2ysh+NcVwQyYxfho19DtTbM0VLiFEIMkaBH qqk/0lkyr5vTRo2c/6ybghkjJsHzpP51v69120fdhR+h9UrunCo4dcjmLz4kgV6f KedkJpsxYmp7W5sC4t3QKRAy7UqQJBwzbPBOH/JVhUM2airlYxyW1VJAmMm2lQIT n8y+6oV9Ue1fuykNCQo2DC9wr2ZyVfft8DO8cuXEDdyDlWQjVmXS5x470RDY6laE 0uNHCOr2z2j6fsPqTrb3iFMuObdIxF3KZXngEVbCpe9KCenZg5PWzm/+zd7vHSuW vN/lTRHj9U3idaR6C9xo =YFhG -----END PGP SIGNATURE----- --=-=-=-- From owner-freebsd-fs@freebsd.org Wed Dec 28 17:17:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F533C9489E for ; Wed, 28 Dec 2016 17:17:01 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id E2A4A1012 for ; Wed, 28 Dec 2016 17:17:00 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id E1FF2C9489D; Wed, 28 Dec 2016 17:17:00 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E19F8C9489C for ; Wed, 28 Dec 2016 17:17:00 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qk0-x230.google.com (mail-qk0-x230.google.com [IPv6:2607:f8b0:400d:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9DCB01011 for ; Wed, 28 Dec 2016 17:17:00 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qk0-x230.google.com with SMTP id t184so251071092qkd.0 for ; Wed, 28 Dec 2016 09:17:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=uQGTQgqumHi1Iwddjs7TYn/WT080ABu8mUQbp7mm7+8=; b=UwbQzSxKHrIwX/0Q3hVNoXUd47ZA0MJE+Ho4WLw0Ps8KYz2y62PrxFVkFE8e+UmDx8 J5IjjVCwlTPnjDhHsKCi4mob5K6oOPd8xBIDqmbHw3me9HyS97IXI+7hl+yekuWuEa2g 086cylEL1wcDgOJe6UiJZkIEKhOUrSa7FUbRfJ7IM3HXg5A97KS6f2azf4desE5fg6YH VBoMYl2b5sJvFssKUq74uLSaKXJJdXqbxzySc1sbiIX2pyxqWUOJgGxcSBVy1k9XVNpm p7GMr5SgUVsK/0yWO7rfxCjTapolP85emfADR9+xy1X6cbcnmppA+8YymrnFXYLFUwL9 S30g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=uQGTQgqumHi1Iwddjs7TYn/WT080ABu8mUQbp7mm7+8=; b=prHZQdpesI4sOw/nYxYZx8hWFSnh8p/JlBtWpZaarxCkcwXJw+5gdNxuY1WnNS/RIT iDAA8NsUSgKopn77740TxTsubKqbjIKEo9/LJqkj9NOlD+Onjn/ROHuwMdhGidyd/E1L 5HXFuwFzDy2r6KymuAGRWDUWwbcJJpuFtZbRnmKEnlOjMlxUGbQYUO+U1eMC83o+sxx0 vlEbcuXcooeoFNeJOTL4SvswnE1bS7WpI2OyVnhgbWtfXqVfppYwoZ5ckFboCoPzzBeg y8CYA19VUYDyckSBURqW13+c6s2Q/5eA8Je0l/dw1etVIL+8hSujsBUiQ3OdbowRYOVK wNHw== X-Gm-Message-State: AIkVDXJ/V/oE0orkH6atJsdIWpygAFmFl4Itei7dsQx9NPJpDsKiaceAqIUgvSIA8ZYI2faN9m0D9GfxiGBaOA== X-Received: by 10.55.151.199 with SMTP id z190mr39555516qkd.166.1482945419846; Wed, 28 Dec 2016 09:16:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.165.161 with HTTP; Wed, 28 Dec 2016 09:16:59 -0800 (PST) Received: by 10.12.165.161 with HTTP; Wed, 28 Dec 2016 09:16:59 -0800 (PST) In-Reply-To: <4E030EC6-39AC-484C-BA53-2C8CA0E79B84@gmail.com> References: <4E030EC6-39AC-484C-BA53-2C8CA0E79B84@gmail.com> From: Freddie Cash Date: Wed, 28 Dec 2016 09:16:59 -0800 Message-ID: Subject: Re: Force ZFS to always look for devices in a defined location To: Ben RUBSON Cc: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 17:17:01 -0000 On Dec 27, 2016 11:03 PM, "Ben RUBSON" wrote: Hello, My zpools are made of devices from /dev/label. Example : NAME STATE READ WRITE CKSUM my-zpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/slotone ONLINE 0 0 0 label/slottwo ONLINE 0 0 0 (...) For some reasons, it may happen that access to some devices fails and devices (quickly) toggle between ONLINE and UNAVAIL/REMOVED state. Then, sometimes, ZFS re-use the device using the /dev/da path, instead of waiting for the /dev/label path to be rediscovered by the system. It then leads to something like this, with is not nice, and not really correct (as last sector of /dev/da contains the label) : NAME STATE READ WRITE CKSUM my-zpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/slotone ONLINE 0 0 0 da14 ONLINE 0 0 0 (...) My question is then, how to force ZFS to look for devices in a defined location ? (/dev/label/ in this example) I tried to offline / online impacted devices (da*), but it does not work, they do not come back with their /dev/label path. Zpool export / import works, but of course this is something we would like to avoid. Many thanks ! Ben Export the pool. Then use -d to import it using the labels: zpool import -d /dev/label my-zpool That will search the label directory for pool devices and use that path into the GEOM layer. And it will remember it for future imports. Aft least until things get confused again in the future. :) Cheers, Freddie From owner-freebsd-fs@freebsd.org Wed Dec 28 17:56:45 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B44CDC95585 for ; Wed, 28 Dec 2016 17:56:45 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9090E1E47 for ; Wed, 28 Dec 2016 17:56:45 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 8D108C95584; Wed, 28 Dec 2016 17:56:45 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CB6BC95583 for ; Wed, 28 Dec 2016 17:56:45 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 206691E43 for ; Wed, 28 Dec 2016 17:56:45 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id m203so62890651wma.3 for ; Wed, 28 Dec 2016 09:56:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:references:to:in-reply-to; bh=g684zOvavwLJHaOx2/ULtC76G/8pQYXDl4bb02rryRI=; b=dB0wgy12VEiUijs6To8WdRztQXChrEaS8b9tpGLVxfzxkC56N45PgZtdgo7hvAdj6s sHmTQ3VbBnehj4OqESy7d/lm2eUqXSsh24BHb+HY0ScUke/cMzzgWrZUPIMTrjh35RgH ju0m3TaMhmcK6cid3d+0C2KMD92sCHexR6Gux8xbbA6dIKLMIshIfrgFkfIrS86KaaKc ltDQ61re2yZzIMpFB2uuKUXwild8NmEWJmUTJ64zTzZZltvHtE67vrJayHFGoTlC6f+x 1kYO5W6zgnvRJE5cw7Ot2hh3r/HWBHsK6UquGFad5KlAJA6GZW6F0yw/nPvjvxIqWPTj NrDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to; bh=g684zOvavwLJHaOx2/ULtC76G/8pQYXDl4bb02rryRI=; b=EmfUUjVt+cD2JBkIuQ505kQwhKRAmu+jUS5fTtFvI1yOJ6BAMNHzKxKHEZw4QCV7f/ 7bRW9fzUmUgoefcMXj9r2NctqmZGWeYVqhhVjWwnQIY+3NynpYy+Snuuw4x1MYY/vap+ 9SvqQfDYs0WQPQkLE/qPs7QuvgmyoLtpYuvIsW6o4aFYcufaICh7B5gfuEbBGItfXTA7 9Qk9x+KZSJzbfbd1SFIEGKD1YWiavPq/YCkXBtLbhOZRLdPufqA4nV4US5U90e5iiKBu 743qYB+KH/KOq/y0M8T1b9EW1XKLW4KCUwkCx6oyxLcBmN5DNKQ1Byl9OSfeSX3HMlqB qoQg== X-Gm-Message-State: AIkVDXIlRBqkrEDqG5wgwbLTUVIDGP8WkKqXu4hlW/dCvJGJKtMuqjVFPCf9Pd4kOAB1Tw== X-Received: by 10.28.150.69 with SMTP id y66mr35915318wmd.107.1482947803392; Wed, 28 Dec 2016 09:56:43 -0800 (PST) Received: from ben.home (LFbn-1-7159-4.w90-116.abo.wanadoo.fr. [90.116.90.4]) by smtp.gmail.com with ESMTPSA id v2sm65337990wja.41.2016.12.28.09.56.42 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 28 Dec 2016 09:56:42 -0800 (PST) From: Ben RUBSON Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Force ZFS to always look for devices in a defined location Date: Wed, 28 Dec 2016 18:56:41 +0100 References: <4E030EC6-39AC-484C-BA53-2C8CA0E79B84@gmail.com> To: fs@freebsd.org In-Reply-To: X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2016 17:56:45 -0000 > On 28 Dec 2016, at 18:16, Freddie Cash wrote: >=20 > On Dec 27, 2016 11:03 PM, "Ben RUBSON" > wrote: > (...) > My question is then, how to force ZFS to look for devices in a defined = location ? > (/dev/label/ in this example) >=20 > Export the pool. >=20 > Then use -d to import it using the labels: >=20 > zpool import -d /dev/label my-zpool >=20 > That will search the label directory for pool devices and use that = path into the GEOM layer. And it will remember it for future imports. = Aft least until things get confused again in the future. :) Thank you Freddie for your feedback. Yes the question is then, will ZFS keep searching into /dev/label only, = even if devices toggle again ? I will test "import -d" and see what occurs. Thank you ! Ben From owner-freebsd-fs@freebsd.org Thu Dec 29 09:16:38 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82FAC96F88 for ; Thu, 29 Dec 2016 09:16:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D85631288 for ; Thu, 29 Dec 2016 09:16:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBT9GcV5023984 for ; Thu, 29 Dec 2016 09:16:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 215635] LOR in zfs Date: Thu, 29 Dec 2016 09:16:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2016 09:16:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215635 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Dec 29 09:16:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81AC6C96FED for ; Thu, 29 Dec 2016 09:16:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7161D1356 for ; Thu, 29 Dec 2016 09:16:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBT9Gv6T037765 for ; Thu, 29 Dec 2016 09:16:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 215634] zfs receive trips up and live-locks for non-incremental fs Date: Thu, 29 Dec 2016 09:16:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2016 09:16:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215634 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- CC|freebsd-amd64@FreeBSD.org | Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Dec 29 09:28:10 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C5B6C93500 for ; Thu, 29 Dec 2016 09:28:10 +0000 (UTC) (envelope-from admin@x171.save99off.com) Received: from x171.save99off.com (x171.save99off.com [43.251.229.171]) by mx1.freebsd.org (Postfix) with ESMTP id E2CF218D7 for ; Thu, 29 Dec 2016 09:28:09 +0000 (UTC) (envelope-from admin@x171.save99off.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=save99off; d=x171.save99off.com; h=MIME-Version:From:To:Date:Subject:Content-Type:Content-Transfer-Encoding; i=admin@x171.save99off.com; bh=f9JtERwQ0gCjSWBh970ExsA3CmM=; b=AKV8PM1K0o/pWdPV1c+4b3u1ClqOoJBPQYi75y4UqZyXFAJ5Nq/gbwTcVNga/JQOqb3OHfbZ4bKd uTAKlcFgMKV9Jsv8wdB029b+QzdYPDp6dxOXznBY7B/pMeHYL2OMgu4Is04DPIPoV2nakwYR/LI7 zvV9ovDzt9CxT7YwZIs= DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=save99off; d=x171.save99off.com; b=cw/pC18P7xOp803Smp94CWwv0wD00d9I0r9PWhz1JDte2zN6Zrx8XRosNKKuYQYAxQYKU8FPnCnl 38AbXfBdxpbOKyn/BVuzX4tnWHqe8tZgXxH+NXwAWTyZDNzhna7ECs/Dg1+5DCL0h7itAyJDiZS9 b4rEQjOJaiHqFp5fqss=; From: "UGG Australia Boots" To: freebsd-fs@freebsd.org Date: 29 Dec 2016 17:17:37 +0800 Subject: CYBER MONDAY PATCH MADNESS!!50% OFF OUR ENTIRE PATCH SHOP! win 92$ MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2016 09:28:10 -0000 From owner-freebsd-fs@freebsd.org Thu Dec 29 20:39:41 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46EBBC96D58 for ; Thu, 29 Dec 2016 20:39:41 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D25B710D3 for ; Thu, 29 Dec 2016 20:39:39 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p5083C20F.dip0.t-ipconnect.de [80.131.194.15]) (authenticated bits=128) by land.berklix.org (8.15.2/8.15.2) with ESMTPA id uBTKdbqU040927; Thu, 29 Dec 2016 20:39:37 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id uBTKdYQ2091622; Thu, 29 Dec 2016 21:39:35 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id uBTKdGR4033963; Thu, 29 Dec 2016 21:39:28 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201612292039.uBTKdGR4033963@fire.js.berklix.net> To: Kirk McKusick cc: freebsd-fs@freebsd.org Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Mon, 26 Dec 2016 15:13:23 -0800." <201612262313.uBQNDNrX030075@chez.mckusick.com> Date: Thu, 29 Dec 2016 21:39:16 +0100 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2016 20:39:41 -0000 Kirk McKusick wrote Mon, 26 Dec 2016 15:13:23 -0800 > > To: freebsd-fs@freebsd.org > > Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes > > From: "Julian H. Stacey" > > Date: Mon, 26 Dec 2016 18:20:59 +0100 > > > > Gary J wrote: > > > > > I suspect this ia a result of how UFS is designed. > > > > Yes. > > > > > Did you use the > > > standard options for block and fragment size? How about inodes? > > > > Yes, I created that partition years ago, I pretty much always just > > use newfs unless experimenting perhaps for a small partition on USB > > stick or CDROM (& then I'd normally delete), so it would have been > > a newfs almost guaranteed with no parameters, default. > > > > > Is the file system UFS1 or UFS2? > > > > UFS2 > > > > > > > UFS is a very compex bit of software and I imagine there are all > > > kinds of interesting surprises when the file system is 99% full. > > > > > > Anyway, the newfs man page may provide some clues. Or look at > > > Wikipedia, there's a UFS entry there, but it doesn't go into the gorey > > > details. > > > > I'll read https://en.wikipedia.org/wiki/Unix_File_System > > > > Konstantin B wrote > > > dumpfs(8) allows to look at .... > > > > Top of dumpfs: > > magic 19540119 (UFS2) time Sun Dec 25 03:19:13 2016 > > superblock location 65536 id [ 548daf7f b34ae147 ] > > ncg 1399 size 224197115 blocks 217157317 > > bsize 32768 shift 15 mask 0xffff8000 > > fsize 4096 shift 12 mask 0xfffff000 > > frag 8 shift 3 fsbtodb 3 > > minfree 0% optim space symlinklen 120 > > maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 > > nbfree 0 ndir 1555427 nifree 102683126 nffree 1339169 > > bpg 20035 fpg 160280 ipg 80256 unrefs 0 > > nindir 4096 inopb 128 maxfilesize 2252349704110079 > > sbsize 4096 cgsize 32768 csaddr 5056 cssize 24576 > > sblkno 24 cblkno 32 iblkno 40 dblkno 5056 > > cgrotor 622 fmod 0 ronly 0 clean 1 > > metaspace 6408 avgfpdir 64 avgfilesize 16384 > > flags soft-updates > > fsmnt /data > > volname swuid 0 providersize 224197115 > > > > Thanks All, > > > > Cheers, > > Julian > > -- > > Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich > > Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. > > http://berklix.eu/brexit/#stolen_votes > > You HAVE made a change to the filesystem parameters. You have set > minfree to 0%. Sorry for my delay replying. Thanks for yours. Yes. Crossed wires :-) I meant I ran plain newfs years back, but yes I did a few days back run tunefs -m 0 to try to write all last blocks, per my original post: jhs> How I noticed: jhs> I wanted to soak up every last block on FS, (cos Ive been having jhs> block device errors, so I wanted to force a read write on all jhs> unused blocks to hopefully get IDE drive to re-allocate anything flakey.) > This means that you are not allowing UFS to keep any > blocks in reserve. At this point you have no full sized blocks > left in the filesystem (nbfree == 0). Thus the only thing left in > the filesystem is fragments. A file in UFS is made up of zero or > more full-sized blocks and at most one fragment. When you have used > up all of the full-sized blocks you cannot create files larger than > the largest available fragment. So if you continue with your > experiment you will find that the biggest file you can create will > eventually fall to 1K. > > So long as you keep minfree at 1% or higher, you will not hit this > condition which is why tunefs warns you not to set minfree to > absurdly low values. If you want to ensure that your filesystem can > be filled to the last block, you should set the blocksize and the > fragment size to the same value. This will ensure that your filesystem > has only full-sized blocks and never creates fragments. Thus you > will be able to allocate files until it is compelely full. > > Kirk McKusick Interesting thanks :-) It was really a temporary cludge of mine to tickle for bad blocks, working from inside the file system. I'd better do it properly, unmount & experiment with eg camcontrol on the partition or whole disk, or search ports/ &/or write a little C prog that reads blocks from a file (aka dev name of partition or whole disk, stores, to memory, & writes back blocks) Cheers, Julian -- Julian Stacey, BSD Linux Unix Sys Eng Consultant Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#stolen_votes From owner-freebsd-fs@freebsd.org Thu Dec 29 21:53:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64907C96DC5 for ; Thu, 29 Dec 2016 21:53:02 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3135E12C0 for ; Thu, 29 Dec 2016 21:53:02 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.87_1 (FreeBSD)) (envelope-from ) id 1cMid9-0000bT-C2; Thu, 29 Dec 2016 21:52:59 +0000 Date: Thu, 29 Dec 2016 21:52:59 +0000 From: Gary Palmer To: "Julian H. Stacey" Cc: Kirk McKusick , freebsd-fs@freebsd.org Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes Message-ID: <20161229215258.GA2169@in-addr.com> References: <201612262313.uBQNDNrX030075@chez.mckusick.com> <201612292039.uBTKdGR4033963@fire.js.berklix.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201612292039.uBTKdGR4033963@fire.js.berklix.net> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2016 21:53:02 -0000 On Thu, Dec 29, 2016 at 09:39:16PM +0100, Julian H. Stacey wrote: > It was really a temporary cludge of mine to tickle for bad blocks, > working from inside the file system. I'd better do it properly, > unmount & experiment with eg camcontrol on the partition or whole > disk, or search ports/ &/or write a little C prog that reads blocks > from a file (aka dev name of partition or whole disk, stores, to > memory, & writes back blocks) You may want to try recoverdisk(1). I'm not sure, I've never used it to read and write to the same device Regards, Gary From owner-freebsd-fs@freebsd.org Fri Dec 30 02:42:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3F36C96169 for ; Fri, 30 Dec 2016 02:42:58 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-it0-x22e.google.com (mail-it0-x22e.google.com [IPv6:2607:f8b0:4001:c0b::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A121C1CFD for ; Fri, 30 Dec 2016 02:42:58 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: by mail-it0-x22e.google.com with SMTP id x2so224414078itf.1 for ; Thu, 29 Dec 2016 18:42:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=TKDlh92mkh7nqtVASjQSc7h/YQ4CYruXsdo69xQdqE4=; b=gAfZfyIL4dL53/IflpCUEWxqEh258IEtiwUSoqCR0A4D7latSI2YB/m172XasMYgll GHRL4VMjukIaIlP1zn4GF3D06qjrkE2o662MX2IJnp3B4x3BHnYNg9o3exv2ICEpWpbO k7evU6mxbfkjjuf/9c5ktJ6zvkNadpVxvbhU0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=TKDlh92mkh7nqtVASjQSc7h/YQ4CYruXsdo69xQdqE4=; b=XGZNcjSmuwfFEObaUTgECeNBMygohUqjg2V1/ltrMXTaOm71Gl4TX4WKzJWP6y0TwD 7EG3e98S/SSlEiPnua7YtzCvtAtdvHqFHwbYI30AI17dX5Lw5jTOPPFlLoQ1cXGD6IlJ Q1eOwaS4D/nFb+OTsGKQh7FgsEiMjt/AUGvzFkEtc62QghwnQiYmTMmQDJsmALk/PsPh gcRUstMRzw6I+7NMRmbdt5PCZTwSVES8YvVXv/EcMhrXl7SB8nwtI8aCC+gVy9LW7tQJ tFeib62FlI4kyjx+cZel1ibpyrkePk7QE1L7WXCZhZCVaO9QIgnrY+qg4gaiF7O1wi+V Uvzg== X-Gm-Message-State: AIkVDXJjIagmddFI/XM8aGuwO1LWsx/TZ2pL6+hrQp3XqRBwFCa1s2nsHnKAWmbgEgSXkQ== X-Received: by 10.36.123.82 with SMTP id q79mr37655143itc.25.1483065777744; Thu, 29 Dec 2016 18:42:57 -0800 (PST) Received: from unassigned.v6.your.org ([2001:4978:1:45:e95d:9cf9:2c69:b03a]) by smtp.gmail.com with ESMTPSA id l3sm10098731iof.33.2016.12.29.18.42.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Dec 2016 18:42:57 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: when ufs is 99% full, current seems to limit creat to 28672 bytes From: Kevin Day In-Reply-To: <201612292039.uBTKdGR4033963@fire.js.berklix.net> Date: Thu, 29 Dec 2016 20:42:56 -0600 Cc: "freebsd-fs@FreeBSD.org Filesystems" Content-Transfer-Encoding: quoted-printable Message-Id: References: <201612292039.uBTKdGR4033963@fire.js.berklix.net> To: "Julian H. Stacey" X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Dec 2016 02:42:58 -0000 >=20 > It was really a temporary cludge of mine to tickle for bad blocks, > working from inside the file system. I'd better do it properly, > unmount & experiment with eg camcontrol on the partition or whole > disk, or search ports/ &/or write a little C prog that reads blocks > from a file (aka dev name of partition or whole disk, stores, to > memory, & writes back blocks) >=20 If you don=E2=80=99t mind losing everything: dd if=3D/dev/zero of=3D/dev/daXXX (write blocks of zeros to drive, starting at the first sector and going = until the end) then dd of=3D/dev/null if=3D/dev/daXXX (read everything back and throw it away) If both complete without error, you=E2=80=99re good. Call newfs and = start over. If you have space somewhere else to fit the entire drive and don=E2=80=99t= want to lose everything: dd if=3D/dev/daXXX of=3D/path/to/giant/file (copy the entire drive to /path/to/giant/file) dd if=3D/path/to/giant/file of=3D/dev/daXXX (copy it back to the drive) If any of those do produce errors and you want dd to continue just = skipping the errored sector(s) add =E2=80=9Cconv=3Dsync,noerror=E2=80=9D = and instead of a bad sector being a fatal error, it=E2=80=99ll replace = it with all NULs and keep going. =E2=80=94 Kevin From owner-freebsd-fs@freebsd.org Sat Dec 31 10:26:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 822EEC96879 for ; Sat, 31 Dec 2016 10:26:13 +0000 (UTC) (envelope-from jpaetzel@fastmail.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5AFA319EA; Sat, 31 Dec 2016 10:26:12 +0000 (UTC) (envelope-from jpaetzel@fastmail.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 18C1E20A69; Sat, 31 Dec 2016 05:26:12 -0500 (EST) Received: from web1 ([10.202.2.211]) by compute4.internal (MEProxy); Sat, 31 Dec 2016 05:26:12 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=fastmail.net; h=cc :content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= mesmtp; bh=lKcEv4GFdjI57yd6G9D9Vv7H9M0=; b=IL0C6NyUgSO01OUyQw+YS g0FMCJHqelWzRJlmdIJ+Thk9JmR61BNr00CnBHyF8X0VHGBIPqrCrRdAAXQK9zJc cIJFkdM+s/wi3VGu6HB5A9GIg27yXetHYuoJuOleYyeHmvIdGiFhwAXT3qY4yBUb 8iQRj17xWmYkNpf2/p0VBc= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=smtpout; bh=lKcEv4GFdjI57yd6G9D9Vv7H9 M0=; b=W0xw8cpeoNsDG7GebtRbcCITC8caj841ljgYDMIxPzWlT+7PBN5TmWLRu QXoyw8/na6RcqfWrV25iAIBTfn1+3gNcLslOtHvLxbx6L9X/FGkc9X9XxtJUUr+U M9s3tghOshah0FhlQOXPcTReYhv7K6crmDaqY8bV/xdGfXu7NY= X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id EDEC9AA6C5; Sat, 31 Dec 2016 05:26:11 -0500 (EST) Message-Id: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> From: Josh Paetzel To: freebsd-fs@freebsd.org Cc: mckusick@freebsd.org, Rick Macklem MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-9c115fcf Date: Sat, 31 Dec 2016 04:26:11 -0600 Subject: NFS readdirplus on ZFS with > 1 billion files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 10:26:13 -0000 We've been chasing this bug for a very long time and finally managed to pin it down. When a ZFS dataset has more than 1 billion files on it and an NFS client does a readdirplus the file handles for files with high znode/inode numbers gets truncated due to a 64 -> 32 bit conversion. https://reviews.freebsd.org/D9009 This isn't a fix so much as a workaround. From a performance standpoint it's the same as if the client mounts with noreaddirplus; sometimes it's a win, sometimes it's a lose. CPU usage does go up on the server a bit. -- Thanks, Josh Paetzel From owner-freebsd-fs@freebsd.org Sat Dec 31 13:34:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3902C998D4 for ; Sat, 31 Dec 2016 13:34:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 357341E43; Sat, 31 Dec 2016 13:34:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uBVDXpWD093460 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 31 Dec 2016 15:33:52 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uBVDXpWD093460 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uBVDXoNd093458; Sat, 31 Dec 2016 15:33:50 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 31 Dec 2016 15:33:50 +0200 From: Konstantin Belousov To: Josh Paetzel Cc: freebsd-fs@freebsd.org, mckusick@freebsd.org Subject: Re: NFS readdirplus on ZFS with > 1 billion files Message-ID: <20161231133350.GU1923@kib.kiev.ua> References: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 13:34:01 -0000 On Sat, Dec 31, 2016 at 04:26:11AM -0600, Josh Paetzel wrote: > We've been chasing this bug for a very long time and finally managed to > pin it down. When a ZFS dataset has more than 1 billion files on it and > an NFS client does a readdirplus the file handles for files with high > znode/inode numbers gets truncated due to a 64 -> 32 bit conversion. > > https://reviews.freebsd.org/D9009 > > This isn't a fix so much as a workaround. From a performance standpoint > it's the same as if the client mounts with noreaddirplus; sometimes it's > a win, sometimes it's a lose. CPU usage does go up on the server a bit. > Can you point to the places in ZFS code where the truncation occur ? I have no idea about ZFS code, and my question is mainly is the truncation just occurs due to different types of ino_t and zfs node id, or some code actively does the range reduction. My question is in the context of the long-dragging ino64 work, which might be finished in some visible future. In particular, I am curious if just using the patched kernel fixes your issue. See https://github.com/FreeBSDFoundation/freebsd/tree/ino64 although I do not make any claim about the state of the code yet. Your patch, after a review, might be still useful for stable/10 and 11, since I do not think that ino64 has any bits which could be merged. From owner-freebsd-fs@freebsd.org Sat Dec 31 14:59:18 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE3BCC99E35 for ; Sat, 31 Dec 2016 14:59:18 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b01.edpnet.be (relay-b01.edpnet.be [212.71.1.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 812701C81 for ; Sat, 31 Dec 2016 14:59:18 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1483195173-0a7ff5320b678620001-dE2xID Received: from mordor.lan ([213.219.148.14]) by relay-b01.edpnet.be with ESMTP id ApuKWn9EeUmyR531 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 31 Dec 2016 15:39:34 +0100 (CET) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: UNKNOWN[213.219.148.14] X-Barracuda-Apparent-Source-IP: 213.219.148.14 Date: Sat, 31 Dec 2016 15:39:32 +0100 From: Julien Cigar To: freebsd-fs@FreeBSD.org Subject: nfsuserd + jails mbufs leak ? Message-ID: <20161231143932.GS15696@mordor.lan> X-ASG-Orig-Subj: nfsuserd + jails mbufs leak ? MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6pbY/KU4ayLo+qis" Content-Disposition: inline User-Agent: Mutt/1.7.1 (2016-10-04) X-Barracuda-Connect: UNKNOWN[213.219.148.14] X-Barracuda-Start-Time: 1483195173 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.221:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 1736 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0000 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.35471 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 14:59:18 -0000 --6pbY/KU4ayLo+qis Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I just upgraded a bunch of machines from 10.0 to 10.3. Those machines have a lot of jails with NFS shares (mounted on the HOST), for example: jcigar@HOST:~/ > mount|grep -i 'filer' filer.prod.lan:/pictures/collections on /usr/jails/www2/filer/pictures/collection (nfs, read-only, nfsv4acls) filer.prod.lan:/webapps on /usr/jails/www2/filer/webapps (nfs, nfsv4acls) filer.prod.lan:/documents on /usr/jails/www2/filer/documents (nfs, read-only, nfsv4acls) filer.prod.lan:/apache on /usr/jails/www2/filer/apache (nfs, nfsv4acls) filer.prod.lan:/geoserver on /usr/jails/java2/filer/geoserver (nfs, nfsv4acls) HOSTs and NFS server have vfs.nfsd.server_min_nfsvers=3D4 vfs.nfsd.server_max_nfsvers=3D4 (but it's probably not related to the problem) and nfsuserd is runnings on each HOST (with no special ${nfsuserd_flags}) On the hosts since I upgraded to 10.3 I'm seeing tons of: Dec 31 14:29:33 duvel nfsuserd:[675]: req from ip=3D0xc0a80a21 port=3D618 It is not clear to me yet why I'm getting this, but from what I=20 understand it's because requests are not coming from 127.0.0.1 but from the jail ip (192.168.10.x in my case).. Parallel to this I'm observing a constant increase of mbufs usage so that at some point in time I'm getting an exhaustion of mbufs: [zone: mbuf] kern.ipc.nmbufs limit reached =2E. and the HOST needs a reboot :( If I'm stopping nfsuserd I don't see any mbuf leakage anymore Any idea ? :) Thanks, Julien --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --6pbY/KU4ayLo+qis Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE7vn2l0to0nV7EWolsrs3EKIEI8AFAlhnwyEACgkQsrs3EKIE I8C2tQ//eLltljnBIZiCLHV5j99toF6UO/lmYINWxl5WVxG2UENSo0AVNHhW92b6 yZmwg/KG3EKoT8DShSe0LRg2LA2GSrDYk1x/s7szfdZo8Og5QtOvUaKGm6cpk/OK XP4gNXrTWCiQsmRxUkdtvjsjqnXQ2UOjZ5Bbt0Q2yvpiMiV1tdYLYLIyRn0fu6Ag 60zcGKKfP1utI+kpD+QcplUEhyxAuDU/Ze2sOaTFvtlnmFkV2Gyt0/L43TWkQbUU FMNiTreD4913U2hI5CSao7uDjtC6ZMQLp8vG+ktLfJllo7wU4DeIFo/T7rgdTO73 Rwg1Zc0JVGlvWuwgVHsB90QVMJ+NzsAxWTAvuelwxUeAzDzotm/fMvAdEmPUgODl GnDJh3qISmfOotyQti3GZE7dEhsAzD9ULBAxFBs8u1OEsgRL/MFXNO9bEC8A0RLs IhsvUEu8OkKickH/rWnuUD5+k60gJlj3DObniWDCRxCNR2iwCsmCBbl10uSfaI3C EYbE+8oU6YdntnR12Fx4BHSgxPqOI4DZEuNXEKGynR40GFJKa5PrONJnUPxRf/Z1 NNdBsuAcTjvTDNPZpaMhp47dCafud/zMlNr1nlEheI0/NnKrzCB5THMq5ITORUIP HSybirKOAktMzgUdGiIHBZvKV1zReNP3DjSbqvvzWxSza0ZFX/M= =vAfY -----END PGP SIGNATURE----- --6pbY/KU4ayLo+qis-- From owner-freebsd-fs@freebsd.org Sat Dec 31 18:08:48 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41E32C99819 for ; Sat, 31 Dec 2016 18:08:48 +0000 (UTC) (envelope-from jpaetzel@FreeBSD.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BF901269 for ; Sat, 31 Dec 2016 18:08:47 +0000 (UTC) (envelope-from jpaetzel@FreeBSD.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 5D9B920783; Sat, 31 Dec 2016 13:08:36 -0500 (EST) Received: from web1 ([10.202.2.211]) by compute2.internal (MEProxy); Sat, 31 Dec 2016 13:08:36 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=smtpout; bh=Lp Z4iosNWVMACeRMryjy0GdCOEw=; b=RrCnnEPPFr83vgc3hAK1Df+8MRqg77PzKm y6Z1Lhklm2GtXu6pGRAva/E19FJ1fVein+EyaHcUVs/ffI2/ERJ6jJ1QhMWdgJli ZpZbti0VkACYahCxjgo/cRNCFzSTYs0QdBORA57GceF+/ageWJaiwG4wkAn1bLyN ta04PWcrA= X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 3CA4EAA6C5; Sat, 31 Dec 2016 13:08:36 -0500 (EST) Message-Id: <1483207716.3465220.833841385.061386FF@webmail.messagingengine.com> From: Josh Paetzel To: Konstantin Belousov Cc: freebsd-fs@freebsd.org, Rick Macklem , ash@ixsystems.com MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-9c115fcf In-Reply-To: <20161231133350.GU1923@kib.kiev.ua> Subject: Re: NFS readdirplus on ZFS with > 1 billion files References: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> <20161231133350.GU1923@kib.kiev.ua> Date: Sat, 31 Dec 2016 12:08:36 -0600 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 18:08:48 -0000 On Sat, Dec 31, 2016, at 07:33 AM, Konstantin Belousov wrote: > On Sat, Dec 31, 2016 at 04:26:11AM -0600, Josh Paetzel wrote: > > We've been chasing this bug for a very long time and finally managed to > > pin it down. When a ZFS dataset has more than 1 billion files on it and > > an NFS client does a readdirplus the file handles for files with high > > znode/inode numbers gets truncated due to a 64 -> 32 bit conversion. > > > > https://reviews.freebsd.org/D9009 > > > > This isn't a fix so much as a workaround. From a performance standpoint > > it's the same as if the client mounts with noreaddirplus; sometimes it's > > a win, sometimes it's a lose. CPU usage does go up on the server a bit. > > > > Can you point to the places in ZFS code where the truncation occur ? > I have no idea about ZFS code, and my question is mainly is the > truncation > just occurs due to different types of ino_t and zfs node id, or some code > actively does the range reduction. > > My question is in the context of the long-dragging ino64 work, which > might > be finished in some visible future. In particular, I am curious if just > using the patched kernel fixes your issue. See > https://github.com/FreeBSDFoundation/freebsd/tree/ino64 > although I do not make any claim about the state of the code yet. > > Your patch, after a review, might be still useful for stable/10 and 11, > since I do not think that ino64 has any bits which could be merged. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" That's a great question and I will attempt to answer the best I can, however I am cc'ing Ash Gokhale and Rick Macklem here because they understand the issue better and might be able to provide a better answer. My understanding is the issue occurs here: http://fxr.watson.org/fxr/source/fs/nfsserver/nfs_nfsdport.c?v=FREEBSD10#L2090 This codepath casts dirent d->fileno from 32 to 64bits to stuff the nfs fileno, but the legacy struct dirent->d_fileno is still 32 bit. I'm not entirely sure this is a ZFS specific issue at all, I've never tried to put 1 billion files on a UFS filesystem to see what would happen. (I suspect this issue with the NFS server would be the least of your issues) I agree the correct solution is the ino64 work. I'm fine with this hack going directly in to 11-STABLE and 10-STABLE. (In fact I think that's the best solution) Another thing we kicked around was making this hack a sysctl, such that you could manually activate it if a filesystem went over the threshold for the bug to occur. No one is completely convinced we understand fully the performance implications of this patch. -- Thanks, Josh Paetzel From owner-freebsd-fs@freebsd.org Sat Dec 31 19:41:39 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92CBCC99399 for ; Sat, 31 Dec 2016 19:41:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22B121CBD; Sat, 31 Dec 2016 19:41:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uBVJfVRb083789 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 31 Dec 2016 21:41:31 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uBVJfVRb083789 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uBVJfVG7083788; Sat, 31 Dec 2016 21:41:31 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 31 Dec 2016 21:41:31 +0200 From: Konstantin Belousov To: Josh Paetzel Cc: freebsd-fs@freebsd.org, Rick Macklem , ash@ixsystems.com Subject: Re: NFS readdirplus on ZFS with > 1 billion files Message-ID: <20161231194131.GC1923@kib.kiev.ua> References: <1483179971.3381747.833629401.5EF242B8@webmail.messagingengine.com> <20161231133350.GU1923@kib.kiev.ua> <1483207716.3465220.833841385.061386FF@webmail.messagingengine.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1483207716.3465220.833841385.061386FF@webmail.messagingengine.com> User-Agent: Mutt/1.7.2 (2016-11-26) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 19:41:39 -0000 On Sat, Dec 31, 2016 at 12:08:36PM -0600, Josh Paetzel wrote: > > > On Sat, Dec 31, 2016, at 07:33 AM, Konstantin Belousov wrote: > > On Sat, Dec 31, 2016 at 04:26:11AM -0600, Josh Paetzel wrote: > > > We've been chasing this bug for a very long time and finally managed to > > > pin it down. When a ZFS dataset has more than 1 billion files on it and > > > an NFS client does a readdirplus the file handles for files with high > > > znode/inode numbers gets truncated due to a 64 -> 32 bit conversion. > > > > > > https://reviews.freebsd.org/D9009 > > > > > > This isn't a fix so much as a workaround. From a performance standpoint > > > it's the same as if the client mounts with noreaddirplus; sometimes it's > > > a win, sometimes it's a lose. CPU usage does go up on the server a bit. > > > > > > > Can you point to the places in ZFS code where the truncation occur ? > > I have no idea about ZFS code, and my question is mainly is the > > truncation > > just occurs due to different types of ino_t and zfs node id, or some code > > actively does the range reduction. > > > > My question is in the context of the long-dragging ino64 work, which > > might > > be finished in some visible future. In particular, I am curious if just > > using the patched kernel fixes your issue. See > > https://github.com/FreeBSDFoundation/freebsd/tree/ino64 > > although I do not make any claim about the state of the code yet. > > > > Your patch, after a review, might be still useful for stable/10 and 11, > > since I do not think that ino64 has any bits which could be merged. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > That's a great question and I will attempt to answer the best I can, > however I am cc'ing Ash Gokhale and Rick Macklem here because they > understand the issue better and might be able to provide a better > answer. > > My understanding is the issue occurs here: > > http://fxr.watson.org/fxr/source/fs/nfsserver/nfs_nfsdport.c?v=FREEBSD10#L2090 > > This codepath casts dirent d->fileno from 32 to 64bits to stuff the nfs > fileno, but the legacy struct dirent->d_fileno is still 32 bit. > > I'm not entirely sure this is a ZFS specific issue at all, I've never > tried to put 1 billion files on a UFS filesystem to see what would > happen. (I suspect this issue with the NFS server would be the least of > your issues) UFS2 inode number is 32bit. If by billion you mean 10^12, you cannot put that many files on UFS volume. > > I agree the correct solution is the ino64 work. I'm fine with this hack > going directly in to 11-STABLE and 10-STABLE. (In fact I think that's > the best solution) All commits should go into HEAD first. I doubt that ino64 could land into HEAD earlier than in a month (but >= 2-3 months is less strain in estimation, IMO). > > Another thing we kicked around was making this hack a sysctl, such that > you could manually activate it if a filesystem went over the threshold > for the bug to occur. No one is completely convinced we understand > fully the performance implications of this patch. From owner-freebsd-fs@freebsd.org Sat Dec 31 22:34:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80291C99EAB for ; Sat, 31 Dec 2016 22:34:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0073.outbound.protection.outlook.com [104.47.37.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 382CD109C for ; Sat, 31 Dec 2016 22:34:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.817.10; Sat, 31 Dec 2016 22:01:18 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0817.009; Sat, 31 Dec 2016 22:01:17 +0000 From: Rick Macklem To: Julien Cigar , "freebsd-fs@FreeBSD.org" Subject: Re: nfsuserd + jails mbufs leak ? Thread-Topic: nfsuserd + jails mbufs leak ? Thread-Index: AQHSY3aEeZOV3aKrOkKpZJz6LX9A86EimXdV Date: Sat, 31 Dec 2016 22:01:17 +0000 Message-ID: References: <20161231143932.GS15696@mordor.lan> In-Reply-To: <20161231143932.GS15696@mordor.lan> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-office365-filtering-correlation-id: 95c695a4-304b-4085-7ec9-08d431c88c16 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0189; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0189; 7:i1YLTMOUd33rNU7FXUDD4Bib4jDE4h5ZBVrSkWzbsrIwiglIVp/Wl96rsgnpYwYjogaK3gWsIU8A3Ud+z8NJw930JZlmFxOQ0Y1HDciQ3h5a1xqTCKyQ29Tinh0cC+tfWXeWDQ2YOtyI8gU+x5eJzXSCQr0VQkz4bRwW0nzCBoeWdZSGZZom2ZuMMY8lr3zVuDwPrVokqZXJu+zvkFSPsOHCIsWoCljOKiodYhIunXQdZVux+PsSfiNCHIkop5MMN9DOhVD9TlVHukK1vpO4/h0gh3w6yzdZShJ9WYh5N10MJPuk9HVpoLk8yxJY6+eFbYd3WutvKROs4gbpVOt2tabhHnXuy+E5XNTaBEsniwZ/QuJdIulODcflLkvuEHB8aHD2ibsj2/asWxBI2TmwogIqQpKR+tUaVUWZ3/CNJhf0LfapmYavwLCgvDlggq8jeAtp2hm4jfrHF1OhQSJiIg== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148); SRVR:YTXPR01MB0189; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0189; x-forefront-prvs: 0173C6D4D5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(39450400003)(199003)(24454002)(189002)(3660700001)(68736007)(81166006)(8936002)(3280700002)(81156014)(305945005)(74316002)(55016002)(106116001)(106356001)(105586002)(101416001)(9686002)(8676002)(2906002)(33656002)(2501003)(2900100001)(102836003)(2950100002)(97736004)(6436002)(189998001)(5001770100001)(107886002)(92566002)(74482002)(77096006)(7696004)(5660300001)(6506006)(229853002)(122556002)(54356999)(76176999)(50986999)(575784001)(86362001)(38730400001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0189; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Dec 2016 22:01:17.3472 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0189 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 22:34:03 -0000 Julien Cigar wrote: >I just upgraded a bunch of machines from 10.0 to 10.3. Those machines >have a lot of jails with NFS shares (mounted on the HOST), for example: [stuff snipped] >On the hosts since I upgraded to 10.3 I'm seeing tons of: >Dec 31 14:29:33 duvel nfsuserd:[675]: req from ip=3D0xc0a80a21 port=3D618 > >It is not clear to me yet why I'm getting this, but from what I >understand it's because requests are not coming from 127.0.0.1 but from >the jail ip (192.168.10.x in my case).. Parallel to this I'm observing a >constant increase of mbufs usage so that at some point in time I'm >getting an exhaustion of mbufs: Yes, nfsuserd only accepts upcalls from 127.0.0.1 (about the only time a reserved port# actually means anything). As such, it doesn't work when you have jails. I came up with a patch that switched nfsuserd to using AF_LOCAL, but the person testing this at the time had hangs, so I never committed it to head,= etc. (I don't know what the mbuf leak is, but since it is busted and useless, I = don't think fixing the leak is necessary.;-) Your choices are: 1 - As you mentioned, don't run nfsuserd. If you are using FreeBSD and/or L= inux clients, this should work once you set vfs.nfsd.enablestringtouid (or = whatever it is called) to 1. 2 - I had a patch that added an option to nfsuserd that allowed specificati= on of a different ip address (unfortunately, I don't have this old patch han= dy, but might be able to find it at the end of this week. Email if you want me= to find it.) rick