From owner-freebsd-fs@freebsd.org Sun Sep 2 21:00:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5D74FF97B4 for ; Sun, 2 Sep 2018 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 899FB870C1 for ; Sun, 2 Sep 2018 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 44A02FF97B1; Sun, 2 Sep 2018 21:00:06 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A73AFF97B0 for ; Sun, 2 Sep 2018 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D2F8870BD for ; Sun, 2 Sep 2018 21:00:05 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id C5F1C173C3 for ; Sun, 2 Sep 2018 21:00:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w82L04eZ088878 for ; Sun, 2 Sep 2018 21:00:04 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w82L04Wl088871 for fs@FreeBSD.org; Sun, 2 Sep 2018 21:00:04 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201809022100.w82L04Wl088871@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 2 Sep 2018 21:00:04 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Sep 2018 21:00:07 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Sep 3 12:41:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D4FDFE89A8 for ; Mon, 3 Sep 2018 12:41:47 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id F05CD83181 for ; Mon, 3 Sep 2018 12:41:46 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Mon, 03 Sep 2018 14:41:40 +0200 Authentication-Results: connect.ultra-secure.de; auth=pass (login); spf=none smtp.mailfrom=ultra-secure.de Received-SPF: None (connect.ultra-secure.de: domain of ultra-secure.de does not designate 127.0.0.10 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=127.0.0.10; helo=connect.ultra-secure.de; envelope-from= Received: from connect.ultra-secure.de (webmail [127.0.0.10]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id BE4B16C7-89D8-4C17-A64A-C7ACA9E1B16F.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-SHA verify=NO); Mon, 03 Sep 2018 14:41:33 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 03 Sep 2018 14:41:33 +0200 From: rainer@ultra-secure.de To: Yamagi Burmeister Cc: freebsd-fs@freebsd.org Subject: Re: ZFS (ARC) performance regression in r321610 In-Reply-To: <20180829131940.7ca6ccb23035eee6808119fb@yamagi.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <1c816966fbe698f0497df231139f9a05@ultra-secure.de> <20180829131940.7ca6ccb23035eee6808119fb@yamagi.org> Message-ID: <4e47ba5c44767368fc01b9692a154b0c@ultra-secure.de> X-Sender: rainer@ultra-secure.de User-Agent: Roundcube Webmail/1.2.0 X-Haraka-GeoIP: --, , NaNkm X-Haraka-GeoIP-Received: X-Haraka-p0f: os="undefined undefined" link_type="undefined" distance=undefined total_conn=undefined shared_ip=Y X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 689, bad: 0, connections: 695, history: 689, pass:all_good, relaying X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2018 12:41:47 -0000 Am 2018-08-29 13:19, schrieb Yamagi Burmeister: > Hi, > my patch was committed this morning. If you're on 12-CURRENT you can > just set vfs.zfs.abd_scatter_enabled to 0. On other branches the > easiest way to make the change permanent is to edit > sys/cddl/contrib/opensolaris/uts/common/fszfs/abd.c and change > zfs_abd_scatter_enabled (on 11.2 it's in line 141) from B_TRUE to > B_FALSE. Then rebuild the kernel, reboot and you're done. Hi, thanks a lot. It's just that none of my machines have sources. The last time I actually compiled a kernel was for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 IIRC. It's not that I don't know how to do it (I started with FreeBSD 2.2.x). The problem is also that I have to ask for downtime for each reboot or just restart of mysql on those servers and it can only be done after 10pm. And it takes the f'ing HP server 10m to be back at the login-prompt. But I'll look into it. The servers are on 11.2. Best Regards, Rainer From owner-freebsd-fs@freebsd.org Tue Sep 4 07:06:35 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86165FE4BFE for ; Tue, 4 Sep 2018 07:06:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-15.consmr.mail.bf2.yahoo.com (sonic316-15.consmr.mail.bf2.yahoo.com [74.6.130.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3109B8AB92 for ; Tue, 4 Sep 2018 07:06:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: MfHGptkVM1nyotgN_R0Q_cvqI1cRokLVXBu3ZUttTGn9TGpaaSfpMn9nq4mEkvN HX1bKeez8.D6cvMwKv8Mrn6S.3k6YYCP5YmBVmAdB1sK9MB59FvKMTLwi2inC6L9vCXgIh8MpGCJ WQnnBFM5EYn8CeuI_X3Bkm..ONxt8nl3xXhY9XGN2eRgJhaa8g0k4koDK3mM9YOcrbXe.cx03bzr H88NLthtAFC0xurEMXYmuDwxP.apCPLwRGRTl.cucC0.nrqd5WPgW2BDm3nmyCNTdO1y5II5jI6D d9kVVRefw21P72yygok2eopCQnv87DmaJ_FLfRidKp4cFn.YoY_Q.SbtjbtLZk2kQ1Emil.mYisV wI67xiNPYJn4ObT_PvU8VX0FCRXpWpo5QzywBsS0ilnXyky95O1U9n3NF6Wtb2HOvi5ltBVINWU1 minMGA3SWJsdpRPVbnGhmo0_chnxSNrGJ4z_bdisyN0Y0nSjWP4fngRKD1Eo1X8zJ0YxXPxJRa.g e9zIMPc.2ROq293bbpcE6T62g47LNRxDJz7myro3U3nZgLIsTpJeI76.eJfb4w6BotjPurr_cEka D_snHkuCjHM9dVL6GRcDT8br1uLC.kkENJmO1BohNeLf6.eeNnW1m2jLJapJfhnawN2xXbzn6usT BFN2Dyw4poEgJzbCh9V.yF0OBxHyDhfO.8ddt.rLf8c.FC591Jw_JBVmyTkGHcMKlmkGRE5eRrSN _dFJErLqmYkGkR2xBAGOTTqInVQ6qEu04jbP3xsJf_FKkiyw2t5wWKeTbWVMhRWyCLwotTogk1pl pYDTJFl2UVx7WAQsMFsJ8SOSl93z5NzchGlqqaGuZuMtVRGBWYfcyDpphQbX8Nf6_Mw5XkHyJeV1 Z6GrE7JUfay2jGvYUMm4nqV1wwV.KWWJGlzQJACrMF7dhy_bJ5r76F9wIgqjRC2RpsyVRdPgVcYI Y6A4mEDA7brEYrQ5pY7AUA0PKtXcChnjcYJSiTp8.sr0K6m0BMjUc6igHO4yK81U4DFSWrdC.5UM nnTxdzXqxa.fHALIF4nV2jw0PfM3n_a.0ThNy Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.bf2.yahoo.com with HTTP; Tue, 4 Sep 2018 07:06:28 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp411.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 6077b1e144452fda820690632017329e; Tue, 04 Sep 2018 07:06:23 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <392FF90D-6D90-457B-94D2-9C4A05B46309@yahoo.com> Date: Tue, 4 Sep 2018 00:06:21 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <3F4C9F6F-667A-4E71-A73F-31CF6A0E754A@yahoo.com> References: <201808240504.w7O54SPJ067702@chez.mckusick.com> <392FF90D-6D90-457B-94D2-9C4A05B46309@yahoo.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 07:06:35 -0000 [Test status update. I give some details of the test context and complications/limitations as well.] On 2018-Aug-23, at 10:58 PM, Mark Millard wrote: > On 2018-Aug-23, at 10:04 PM, Kirk McKusick = wrote: >=20 >> So are you in a position to try out the TRIM consolidation = implementation? >> In particular to see if it helps? >=20 > The Pine64+ 2GB that I have access to is in the middle of > a poudriere-devel bulk that probably has 24 hours or more > to go. That will delay anything that I might try. >=20 > I normally do not have the root file system and swap partition > run on a microsd card. So I do not have a history to compare > against. (I'd seen reports on the lists and just avoided the > issues up front.) I'm not sure of a good way to test at this > point. I might just end up gstat -pd monitoring a self-hosted > buildworld buildkernel or some such. Suggestions welcome. >=20 > Swap partitions do not get TRIMs and swap files are greatly > unreliable. (For the later see: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206048 > for what I'm referring to.) So the testing would likely be > just for the file system activity, with any swapping mixed > in the activity. >=20 > Once the poudriere-devel finishes it will take a bit to > set up a microsd card environment that is modern enough > to have the changes. So may be next week sometime? >=20 > Hopefully the: >=20 > # umount /mnt > pine64# tunefs -tenable /dev/mmcsd0s2a > tunefs: issue TRIM to the disk set >=20 > information will enable Bob P. to be able to do some > rpi3 and/or rpi2 testing. But I do not know if he has > a context were he can have some microsd card(s) that > he uses in an unmounted but accessible status. (I > have such a context.) >=20 >=20 > If FreeBSD could still boot such, I'd have tested > an e.MCC on an adapter card plugged into the microsd > slot. But when I took my large jump to head -r337400 > I discovered that combination no longer booted. So, > unless the status has changed after -r377400 when I > update, I can not test the e.MMC via sdcard-slot > context on the Pine64+ 2GB. I finally have access to a mmcsdxc card of about the same capacity as the on-an-adpater e.MMC that I used to use with the Pine64+ 2GB. I've copied things over, enabled trim on the (root) file system, and have "vfs.ffs.dotrimcons: 1". I did a fsck_ffs -E. The complete UFS (root) file system and the swap partition are on the microsdxc card, no other storage devices involved. (Swap not getting trim but the UFS file system getting it.) I'm doing an incremental -j4 buildworld buildkernel in this environment, jumping from a build of -r337400 to -r338450, but the Pine64+ 2GB is running -r338341. The build removes old files as it goes along. We will see how it goes. It is using "vm.pageout_oom_seq: 120". I'm using: LDFLAGS.lld +=3D -Wl,--no-threads I'll note that Ian Lepore reported in an exchange about RPI3 testing: QUOTE Trim consolidation at the ufs layer might not have as much effect on mmcsd as it does on other devices. The mmcsd driver has always consolidated adjacent trim requests itself even when they arrive in the IO queue as a large number of small BIO_DELETE commands. It assembles blocks of adjacent deletes until they reach the size of a full erase block, then issue one erase command (I've never been convinced that doing so is necessary, to me the sd spec implies you can delete individual sectors). END QUOTE I've no known, good means of isolating and comparing/contrasting the two consolidations (vs. the combination of both). As stands, at best I'll be able to report if I see any failures or evidence of any bottlenecks. (Similarly for a later poudriere-devel bulk.) The microsdcx card tested for this is from a: Sandisk Ultra 128GB Micro SDXC UHS-I Card with Adapter = SDSQUAR-128G-GN6MA It has the application "A1" rating. About 2/3 of the UFS partition is "Avail". About 5.5 GiByte of the microsdcx card is not in any partition (but that was dd'd from the e.MCC too and I do not know how to readily trim those areas). Other points of context: The Pine64+ 2GB is running a non-debug kernel (but with symbols), as is my norm unless I'm looking into a failure. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Tue Sep 4 20:08:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2E89FFA5E8; Tue, 4 Sep 2018 20:08:22 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6954D7F3FF; Tue, 4 Sep 2018 20:08:22 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:88d0:b252:4399:906d]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id DAC0DBB5; Tue, 4 Sep 2018 23:08:20 +0300 (MSK) Date: Tue, 4 Sep 2018 23:08:20 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <609400979.20180904230820@serebryakov.spb.ru> To: FreeBSD Current , freebsd-fs@FreeBSD.org Subject: newfs silently fails if random is not ready (?) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 20:08:22 -0000 Hello FreeBSD, I have problems with diskless install when kernel doesn't have tmpfs and random device takes long time to unlock. Looks like newfs used to create in-memory /etc (on /dev/md0) silently fail to create FS. I've added '-XL' options to mdmfs and it looks like this: da0: quirks=0x2 arc4random: no preloaded entropy cache arc4random: no preloaded entropy cache *** Figure out our NFS root path *** *** handle_remount /conf *** *** templates are base default *** *** handle_remount /conf/base/etc *** *** handle_remount /conf/base/var *** *** handle_remount /conf/default/etc *** *** handle_remount /conf/default/var *** *** create_md etc with size 20480 *** DEBUG: running: /sbin/mdconfig -a -t swap -s 10485760B DEBUG: running: /sbin/newfs -U /dev/md0 random: read_random_uio unblock wait random: read_random_uio unblock wait random: unblocking device. DEBUG: running: /sbin/mount /dev/md0 /etc mount: /dev/md0: No such file or directory mdmfs: mount exited with error code 1 "/dev/md0" is here, but it is empty, without any FS. I could run "/sbin/newfs -U /dev/md0" later by hands and it works. -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Tue Sep 4 20:46:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C39A5FFB512; Tue, 4 Sep 2018 20:46:07 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62AB180901; Tue, 4 Sep 2018 20:46:07 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f44.google.com with SMTP id h23-v6so6620215ita.5; Tue, 04 Sep 2018 13:46:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=UzeyLKnZo9N56ajtMvjP5A8BmYfJptEP9kxmENU0ajA=; b=bD3NvAPHo0dDVotaT+Epx/6sqnFwSjJg5xWDpWbqP/zih0HuYa97ZmSccFnterlABW GwC8goutPFENBfRPwn9gfie2HSvJMpW72OJ2GArpH5G0X7G4WCQhH422RriZqiT17MfZ ttjYYLhoAE2keg5RvPn9XO9uef2/JBG5byyxids97R7zvfJzM7sc9iE3HxrovF8ziEND KmhZBz14rpLVyS3oAG2TMuzkZFRHNJSmbBA3h9MYuKHW057jphEw2c8EwcIzr2ZgYIOS 63/eNWKLndhr0ZOcbPmPWFRPcDjeFTIZyyeA1RENEt8roBps2jcUd8wk8V+ijBVppH/r Pysg== X-Gm-Message-State: APzg51Ba1Nw7mrinp3UkIPAWyn3LDOVAeeXte1p7IzFw2u+x/vkw3FNP YUd7pYyf1QsbVShTNfPakonU3IAb X-Google-Smtp-Source: ANB0VdaIsCyEzY7Wo/aFxyfut9c6cFba1QbQ5dVW7ipzBy1zUSt4UArQ95aalmb1snLA24kIpBADfg== X-Received: by 2002:a24:17c7:: with SMTP id 190-v6mr8490839ith.99.1536093480490; Tue, 04 Sep 2018 13:38:00 -0700 (PDT) Received: from mail-io0-f181.google.com (mail-io0-f181.google.com. [209.85.223.181]) by smtp.gmail.com with ESMTPSA id d65-v6sm9847857iof.46.2018.09.04.13.38.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Sep 2018 13:38:00 -0700 (PDT) Received: by mail-io0-f181.google.com with SMTP id v14-v6so4175094iob.4; Tue, 04 Sep 2018 13:38:00 -0700 (PDT) X-Received: by 2002:a6b:97c6:: with SMTP id z189-v6mr24320774iod.120.1536093480015; Tue, 04 Sep 2018 13:38:00 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 13:37:59 -0700 (PDT) In-Reply-To: <609400979.20180904230820@serebryakov.spb.ru> References: <609400979.20180904230820@serebryakov.spb.ru> From: Conrad Meyer Date: Tue, 4 Sep 2018 13:37:59 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: lev@freebsd.org Cc: FreeBSD Current , freebsd-fs , Xin LI Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 20:46:08 -0000 Hi Lev, Newfs just uses arc4random(3) to generate its FSID and generation numbers. arc4random(3) is seeded from getentropy(3) -> getrandom(2) -> read_random_uio(9), which is what produces the "random: read_random_uio unblock wait" messages. Is newfs tripping on a raise()/abort() in arc4random(3) / getentropy(3)? Is your program that runs newfs checking for non-zero exit status? Do you see any newfs cores when this happens? Thanks, Conrad On Tue, Sep 4, 2018 at 1:08 PM, Lev Serebryakov wrote: > Hello FreeBSD, > > I have problems with diskless install when kernel doesn't have tmpfs and > random device takes long time to unlock. > > Looks like newfs used to create in-memory /etc (on /dev/md0) silently fail > to create FS. > > I've added '-XL' options to mdmfs and it looks like this: > > da0: quirks=0x2 > arc4random: no preloaded entropy cache > arc4random: no preloaded entropy cache > *** Figure out our NFS root path *** > *** handle_remount /conf *** > *** templates are base default *** > *** handle_remount /conf/base/etc *** > *** handle_remount /conf/base/var *** > *** handle_remount /conf/default/etc *** > *** handle_remount /conf/default/var *** > *** create_md etc with size 20480 *** > DEBUG: running: /sbin/mdconfig -a -t swap -s 10485760B > DEBUG: running: /sbin/newfs -U /dev/md0 > random: read_random_uio unblock wait > random: read_random_uio unblock wait > random: unblocking device. > DEBUG: running: /sbin/mount /dev/md0 /etc > mount: /dev/md0: No such file or directory > mdmfs: mount exited with error code 1 > > "/dev/md0" is here, but it is empty, without any FS. > > I could run "/sbin/newfs -U /dev/md0" later by hands and it works. > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Sep 4 20:55:16 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 584A0FFBA6D; Tue, 4 Sep 2018 20:55:16 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id E8BE481000; Tue, 4 Sep 2018 20:55:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:88d0:b252:4399:906d]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id CD42ABCC; Tue, 4 Sep 2018 23:55:14 +0300 (MSK) Date: Tue, 4 Sep 2018 23:55:14 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <1942661439.20180904235514@serebryakov.spb.ru> To: Conrad Meyer CC: FreeBSD Current , freebsd-fs , Xin LI Subject: Re: newfs silently fails if random is not ready (?) In-Reply-To: References: <609400979.20180904230820@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 20:55:16 -0000 Hello Conrad, Tuesday, September 4, 2018, 11:37:59 PM, you wrote: > Newfs just uses arc4random(3) to generate its FSID and generation > numbers. arc4random(3) is seeded from getentropy(3) -> getrandom(2) ->> read_random_uio(9), which is what produces the "random: > read_random_uio unblock wait" messages. > Is newfs tripping on a raise()/abort() in arc4random(3) / > getentropy(3)? Nope, it is silently does nothing > Is your program that runs newfs checking for non-zero > exit status? It is not "my" program, it is system mdmfs(8), and it checks exit statuses, as far as I can see from source code. > Do you see any newfs cores when this happens? It is very first step in diskless boot, so no cores are possible, but I don't see "core dumped" messages too. > Thanks, > Conrad > On Tue, Sep 4, 2018 at 1:08 PM, Lev Serebryakov wrote: >> Hello FreeBSD, >> >> I have problems with diskless install when kernel doesn't have tmpfs and >> random device takes long time to unlock. >> >> Looks like newfs used to create in-memory /etc (on /dev/md0) silently fail >> to create FS. >> >> I've added '-XL' options to mdmfs and it looks like this: >> >> da0: quirks=0x2 >> arc4random: no preloaded entropy cache >> arc4random: no preloaded entropy cache >> *** Figure out our NFS root path *** >> *** handle_remount /conf *** >> *** templates are base default *** >> *** handle_remount /conf/base/etc *** >> *** handle_remount /conf/base/var *** >> *** handle_remount /conf/default/etc *** >> *** handle_remount /conf/default/var *** >> *** create_md etc with size 20480 *** >> DEBUG: running: /sbin/mdconfig -a -t swap -s 10485760B >> DEBUG: running: /sbin/newfs -U /dev/md0 >> random: read_random_uio unblock wait >> random: read_random_uio unblock wait >> random: unblocking device. >> DEBUG: running: /sbin/mount /dev/md0 /etc >> mount: /dev/md0: No such file or directory >> mdmfs: mount exited with error code 1 >> >> "/dev/md0" is here, but it is empty, without any FS. >> >> I could run "/sbin/newfs -U /dev/md0" later by hands and it works. >> >> -- >> Best regards, >> Lev mailto:lev@FreeBSD.org >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Tue Sep 4 21:05:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF194FFBEF3; Tue, 4 Sep 2018 21:05:45 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5CF0D8165B; Tue, 4 Sep 2018 21:05:45 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f50.google.com with SMTP id h1-v6so6695456itj.4; Tue, 04 Sep 2018 14:05:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=PypNyxxUwRBszqw2G4nACmW0LX58vdtyH9xCNyrpAn4=; b=mwd8D1BF6su5Sy7q+3ULO2Av8C+LacdK0ed7P93WSykNnAEl6yh5/s9B7N2KUbOVCR FGoqvmLkcsQqhgk8zqWCPROMH1fdMuFGPaZr/D3npDQU3lqQN8fgBOlXlnh+ebhCB1+U PHJ+L1/XkkUvPqT9+l0g4TW8Cwa+tU17ZiEpo+FxVpZ/y/x3gxkfl7rwmocD4Dr6oN/M tt2wd4b1PYYUk5Lsekdsqw2fAMOHU2CoTlM/MNR/v8R1qtHE+drWE+COE0dInG5xgUqw /wbRUCda3b5zy9yh1rZ1hwHC7Dey5QQ7LjoMlc4ePVX190I93vGpPPX+tnirjj5Bp7dU TqBw== X-Gm-Message-State: APzg51CtgQXf1edBbGTK8Do+wMLzpGPKuFlMD4vwBX0fDz3AwYOg+gxV mlQeh32tP8xhdgo05W89HoaUebeL X-Google-Smtp-Source: ANB0VdbA5FbNlKJQ0Tvevvn5Zy4Fm4A+33qws1nCoON37HMnKZTewm98dO3ILFYxtDpkn7dTCy36Xg== X-Received: by 2002:a02:7123:: with SMTP id n35-v6mr24646358jac.91.1536095144576; Tue, 04 Sep 2018 14:05:44 -0700 (PDT) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com. [209.85.214.54]) by smtp.gmail.com with ESMTPSA id i139-v6sm14093349ioa.26.2018.09.04.14.05.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Sep 2018 14:05:44 -0700 (PDT) Received: by mail-it0-f54.google.com with SMTP id h20-v6so6703275itf.2; Tue, 04 Sep 2018 14:05:44 -0700 (PDT) X-Received: by 2002:a24:44d7:: with SMTP id o206-v6mr1654384ita.66.1536095144075; Tue, 04 Sep 2018 14:05:44 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 14:05:43 -0700 (PDT) In-Reply-To: <1942661439.20180904235514@serebryakov.spb.ru> References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> From: Conrad Meyer Date: Tue, 4 Sep 2018 14:05:43 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Xin LI Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 21:05:46 -0000 Hi Lev, On Tue, Sep 4, 2018 at 1:55 PM, Lev Serebryakov wrote: > Tuesday, September 4, 2018, 11:37:59 PM, you wrote: >> Is newfs tripping on a raise()/abort() in arc4random(3) / >> getentropy(3)? > Nope, it is silently does nothing I think it is tripping on raise/abort() in one of these routines, but nothing is printing that information. See below. >> Is your program that runs newfs checking for non-zero >> exit status? > It is not "my" program, it is system mdmfs(8), and it checks exit > statuses, as far as I can see from source code. Ah, thanks. I missed this. mdmfs(8) has a bug in its run() function. It treats programs that exit with a signal (KILL, ABRT, ILL, SEGV...) the same as programs that exit with success. This is a (major) problem and the reason raise/abort is not visible. Best, Conrad From owner-freebsd-fs@freebsd.org Tue Sep 4 21:10:36 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEC35FFC0F6; Tue, 4 Sep 2018 21:10:36 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 730CF818B2; Tue, 4 Sep 2018 21:10:36 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:88d0:b252:4399:906d]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 8361FBD8; Wed, 5 Sep 2018 00:10:35 +0300 (MSK) Date: Wed, 5 Sep 2018 00:10:35 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <774228883.20180905001035@serebryakov.spb.ru> To: Conrad Meyer CC: FreeBSD Current , freebsd-fs , Xin LI Subject: Re: newfs silently fails if random is not ready (?) In-Reply-To: References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 21:10:37 -0000 Hello Conrad, Wednesday, September 5, 2018, 12:05:43 AM, you wrote: > On Tue, Sep 4, 2018 at 1:55 PM, Lev Serebryakov wrote: >> Tuesday, September 4, 2018, 11:37:59 PM, you wrote: >>> Is newfs tripping on a raise()/abort() in arc4random(3) / >>> getentropy(3)? >> Nope, it is silently does nothing > I think it is tripping on raise/abort() in one of these routines, but > nothing is printing that information. See below. Maybe, it should be fixed? One second after that random is ready to use and newfs works as intended. It is, really, some race between early init script (rc.initdiskless) and kernel. I don't think, that newfs must be killed/aborted in such situation, it could wait! >>> Is your program that runs newfs checking for non-zero >>> exit status? >> It is not "my" program, it is system mdmfs(8), and it checks exit >> statuses, as far as I can see from source code. > Ah, thanks. I missed this. mdmfs(8) has a bug in its run() function. > It treats programs that exit with a signal (KILL, ABRT, ILL, SEGV...) > the same as programs that exit with success. This is a (major) > problem and the reason raise/abort is not visible. And it is another problem. But if it will be fixed, it doesn't help to init system properly in my case, only diagnostic will be better. -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Tue Sep 4 21:27:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5BB0FFFCACE; Tue, 4 Sep 2018 21:27:28 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F0F4B826C9; Tue, 4 Sep 2018 21:27:27 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f50.google.com with SMTP id j198-v6so15076975ita.0; Tue, 04 Sep 2018 14:27:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=tTJD9sq6mta4u4FctQgnsX6w6gCliPb2+jUnDaDLMl0=; b=PCTzf5rMtbBcz60ZzJru1qFgxU3zIdBv7dNNOlfVsy/Vej+Aq2DV4Z0qqIkp08qNF3 Wsk9ljbM/6U7e+Iux7OUWH2Ev+HfQyKJ0AyzIUMJSHCLHP+HvJvbF48hf8F0Z18NqpQp UsV8UmGetH5xymkLU4Wsf3K6Ux97+rzlaGufRgafRvn2J/n68MM4ab9xLU7EKCuQGxks K9wTeubbhdBuGjiPLh+y+vTKkFtyAQCJkmKsvXDhio2FcMBt7UEpNCN7oJPSHfwVTaG1 UlmtWzlGYuRNiA5fVOanANIIvFrsZz2hQno5nBS2oc57YtT36pQ0oBAw7Vlaz3bIpLcz aajw== X-Gm-Message-State: APzg51BURlV2/zTWMzgA9DDh04+bxYkv16gZhPiJoh3F5068tDvo08i9 lVJ0jiKKHD8Vliw1+u9w+dC3t3FD X-Google-Smtp-Source: ANB0VdbQ/+UN9kCfGoFC2hGturqx1RFfTPNuiEyQEWsdqq2RrmIpIoTpnhFb0AFruNo7mxLyRuJKVA== X-Received: by 2002:a24:2848:: with SMTP id h69-v6mr9444920ith.63.1536096447221; Tue, 04 Sep 2018 14:27:27 -0700 (PDT) Received: from mail-io0-f173.google.com (mail-io0-f173.google.com. [209.85.223.173]) by smtp.gmail.com with ESMTPSA id 68-v6sm215245itx.19.2018.09.04.14.27.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Sep 2018 14:27:27 -0700 (PDT) Received: by mail-io0-f173.google.com with SMTP id e12-v6so4240902iok.12; Tue, 04 Sep 2018 14:27:26 -0700 (PDT) X-Received: by 2002:a5e:9615:: with SMTP id a21-v6mr23473349ioq.53.1536096446818; Tue, 04 Sep 2018 14:27:26 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 14:27:26 -0700 (PDT) In-Reply-To: <774228883.20180905001035@serebryakov.spb.ru> References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> From: Conrad Meyer Date: Tue, 4 Sep 2018 14:27:26 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Xin LI Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Sep 2018 21:27:28 -0000 On Tue, Sep 4, 2018 at 2:10 PM, Lev Serebryakov wrote: > Wednesday, September 5, 2018, 12:05:43 AM, you wrote: >> I think it is tripping on raise/abort() in one of these routines, but >> nothing is printing that information. See below. > > Maybe, it should be fixed? Yes, it should. > One second after that random is ready to use and > newfs works as intended. It is, really, some race between early init script > (rc.initdiskless) and kernel. I don't think, that newfs must be > killed/aborted in such situation, it could wait! I agree. It looks like it is waiting, in fact, but then Something Bad Happens when the random device is unblocked. >> Ah, thanks. I missed this. mdmfs(8) has a bug in its run() function. >> It treats programs that exit with a signal (KILL, ABRT, ILL, SEGV...) >> the same as programs that exit with success. This is a (major) >> problem and the reason raise/abort is not visible. > > And it is another problem. But if it will be fixed, it doesn't help to init > system properly in my case, only diagnostic will be better. Yep. Best, Conrad From owner-freebsd-fs@freebsd.org Wed Sep 5 03:13:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9158DFE0BC6; Wed, 5 Sep 2018 03:13:45 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 323318E058; Wed, 5 Sep 2018 03:13:45 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f44.google.com with SMTP id 139-v6so7681258itf.0; Tue, 04 Sep 2018 20:13:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=afY1+KdvF0mIZrGupKYbJbDiULwpIROjdW15VeYOVZM=; b=tMApxOD1rLRqTzgBUFj3U5vjcboauemRlS1429JHYTN9sXobHj9xaY/d5OHHEu1Y8A wvovthN2uQeedNR9RTX69JIWJPt1FA3FI1APy4zl7oej4cEbD8ZaJDpxOLLUhfyGRb0G /fWntpmIFBU6K+144cG8dasT5uMSO3Y+hT+3kwcUuAxcA0vIpoHICuR3QRlDKsYAwgzD uFoiKbefriGj6JW+y1wD2EePVsyk3MkuVYiFlzX98ZhJFngUiO9NflC0S0xMUASeDtXG sIx7OlOSFFD430Iv8RY9N3Frr+DlFtlKz972URazKzEZk7iXUV/kaNMtetp1P84hiOto G1yw== X-Gm-Message-State: APzg51DqRJ3G8I+4yZ+f8DwW5NFuuT2SWxy4UD059jEIDDFF5d/1VP9F ih3IZtVemFT1vE7LeK1B08VlmgAV X-Google-Smtp-Source: ANB0VdZw1jGc5hMsLD9vnHaJ8zbV+tpRFEuyBIBum4IFMMaApmTbjIqnQbdm8lVhn3KaAvHRjJYYPw== X-Received: by 2002:a24:4a83:: with SMTP id k125-v6mr10011627itb.121.1536117223274; Tue, 04 Sep 2018 20:13:43 -0700 (PDT) Received: from mail-io0-f176.google.com (mail-io0-f176.google.com. [209.85.223.176]) by smtp.gmail.com with ESMTPSA id 137-v6sm476719itl.39.2018.09.04.20.13.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Sep 2018 20:13:42 -0700 (PDT) Received: by mail-io0-f176.google.com with SMTP id w11-v6so4828456iob.2; Tue, 04 Sep 2018 20:13:42 -0700 (PDT) X-Received: by 2002:a6b:254e:: with SMTP id l75-v6mr23753191iol.47.1536117222414; Tue, 04 Sep 2018 20:13:42 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 20:13:41 -0700 (PDT) In-Reply-To: <774228883.20180905001035@serebryakov.spb.ru> References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> From: Conrad Meyer Date: Tue, 4 Sep 2018 20:13:41 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Xin LI Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 03:13:45 -0000 Hi Lev, I took a first attempt at reproducing this problem on a fast desktop-class system. First steps, give us a way to revert back to unseeded status: --- a/sys/dev/random/fortuna.c +++ b/sys/dev/random/fortuna.c @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); #ifdef _KERNEL #include +#include #include #include #include @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) return; } + /* + * When set, pretend we do not have enough entropy to reseed yet. + */ + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { + if (RETURN_VALUE != 0) { + RANDOM_RESEED_UNLOCK(); + return; + } + }); + + #ifdef _KERNEL fortuna_state.fs_lasttime = now; #endif @@ -442,5 +454,11 @@ bool random_fortuna_seeded(void) { + /* When set, act as if we are not seeded. */ + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { + if (RETURN_VALUE != 0) + fortuna_state.fs_counter = UINT128_ZERO; + }); + return (!uint128_is_zero(fortuna_state.fs_counter)); } Second step, enable the failpoints and launch repro program: $ sudo sysctl debug.fail_point.random_fortuna_pre_read='return(1)' debug.fail_point.random_fortuna_pre_read: off -> return(1) $ sudo sysctl debug.fail_point.random_fortuna_seeded='return(1)' debug.fail_point.random_fortuna_seeded: off -> return(1) $ cat ./blocked_random_poc.c #include #include #include int main(int argc, char **argv) { printf("%x\n", arc4random()); return (0); } $ ./blocked_random_poc ... Third step, I looked at what that process was doing: Curiously, it is not in getrandom() at all, but instead the ARND sysctl fallback. I probably need to rebuild world (libc) to test this (new libc arc4random based on Chacha). $ procstat -kk 1196 PID TID COMM TDNAME KSTACK 1196 100435 blocked_random_poc - read_random+0x3d sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b amd64_syscall+0x940 fast_syscall_common+0x101 When I unblocked the failpoints, it completed successfully: $ sudo sysctl debug.fail_point.random_fortuna_pre_read='off' debug.fail_point.random_fortuna_pre_read: return(1) -> off $ sudo sysctl debug.fail_point.random_fortuna_seeded=off debug.fail_point.random_fortuna_seeded: return(1) -> off ... 9e5eb30f Best, Conrad From owner-freebsd-fs@freebsd.org Wed Sep 5 03:16:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C116FE0E3A for ; Wed, 5 Sep 2018 03:16:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-21.consmr.mail.gq1.yahoo.com (sonic304-21.consmr.mail.gq1.yahoo.com [98.137.68.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA6298E327 for ; Wed, 5 Sep 2018 03:16:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: .hf.p_4VM1mNlYQsyB9Us4lXagXaG6Z85YEsvYKIhatRod6i0hfvBg1r1FPVkJG Oxhl9MSor5_OJvu3MTlqCeLxStO8fGgwo66GXLDcs4PqYl5YBPU5zyLd7p0SSAGEAUyJEQ_mJ1Z2 uNJejIAyv_9Y5wh4OL8mLlw5IfkfvWQJ.Hq3W_xFgm1cOKHXYObyDquPlx2BNTrK7uUAdLFR7GY1 80JUgqZ9z9bq7pXhxNraDnYI5wrHKmKdFxU.bY2C2sqftv0uOoVxhOeiD6JSpS134ZVWuPyDSBTr wyMEq6fMLat9CWNIDsFeXcn3ndCo1vGLVVLN5nAzoZVPefdhCaTJRzYXCXYdkeGqfzNt4jFUzsUH VmmvTdoLbGHYuUhipOaKA4VN7Z15LTe8IU1Y4TbKrZ898TdpS9s6RKqmseCuHX2CazljMKybQGiE IaRFlQOja8sNKuSBT4l4kZshFg6cCxSG.J3t0nS7G50L0CMtO44kVfvww4UgxcUuH7TOcxPyH5hg PXiIY_aMXPptWDMMp1R.kA8fgwvTR52hHaCWXmlisamK_OwAI4OiBM7OW7bmB4_FvHkjHDUKitEu Dvd6_MhRH2cGn422L0UcHS1V8XnfYwQMPeh9GpxzDNvPhK0vXftr7xHiwpiTs2smRU5bGgFx5Yl1 5JrBVCOiI9ee9gnHNU8UF9_BMo9H9FuIzroZSVQp1w3UygpMCkbYXzOW8iP4kyh8I92PeOCs7zwK oS6Q9bAkAGT3JdhFVMH9cQ9y3411KnrwIkCEms8NNszG8g3oLhL.GnLWJshzyxg36J4QQQnyyARN CyGeThqlm9T5ue1ivFPKlVjKWaAS5GZHpnq2pYO7tEvhE581mc2GzKDGYWwXdO4DubNpRwIPH4zD 2FEoitCb2L9QJ9feUNRLSmE1G5NUnznsEIP_0.MqhbM.1nfoQx5cBqIInPDVQaTzH0P5gxIXYbkA LrybQL3rKNiL8dp9HPxY.BcPsG77Fn3IgF8zMWem4o32sRvW7LznYBG5yEv2Asxaav6Ct0EXz3Zo rIfU24G7O Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Wed, 5 Sep 2018 03:16:42 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp424.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID e2f508d1d64b6898544458fba911a409; Wed, 05 Sep 2018 03:16:38 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <3F4C9F6F-667A-4E71-A73F-31CF6A0E754A@yahoo.com> Date: Tue, 4 Sep 2018 20:16:37 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <373CBC85-7A7C-4EEE-9A50-6AEAFEC05F63@yahoo.com> References: <201808240504.w7O54SPJ067702@chez.mckusick.com> <392FF90D-6D90-457B-94D2-9C4A05B46309@yahoo.com> <3F4C9F6F-667A-4E71-A73F-31CF6A0E754A@yahoo.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 03:16:51 -0000 [Some results.] On 2018-Sep-4, at 12:06 AM, Mark Millard wrote: > [Test status update. I give some details of the > test context and complications/limitations as > well.] >=20 > On 2018-Aug-23, at 10:58 PM, Mark Millard = wrote: >=20 >> On 2018-Aug-23, at 10:04 PM, Kirk McKusick = wrote: >>=20 >>> So are you in a position to try out the TRIM consolidation = implementation? >>> In particular to see if it helps? >>=20 >> . . . >=20 >=20 > I finally have access to a mmcsdxc card of about the > same capacity as the on-an-adpater e.MMC that I used > to use with the Pine64+ 2GB. I've copied things over, > enabled trim on the (root) file system, and have > "vfs.ffs.dotrimcons: 1". I did a fsck_ffs -E. The > complete UFS (root) file system and the swap partition > are on the microsdxc card, no other storage devices > involved. (Swap not getting trim but the UFS file > system getting it.) >=20 > I'm doing an incremental -j4 buildworld buildkernel > in this environment, jumping from a build of -r337400 > to -r338450, but the Pine64+ 2GB is running -r338341. > The build removes old files as it goes along. We will > see how it goes. It is using "vm.pageout_oom_seq: 120". > I'm using: LDFLAGS.lld +=3D -Wl,--no-threads >=20 > I'll note that Ian Lepore reported in an exchange about > RPI3 testing: >=20 > QUOTE > Trim consolidation at the ufs layer might not have as much effect on > mmcsd as it does on other devices. The mmcsd driver has always > consolidated adjacent trim requests itself even when they arrive in = the > IO queue as a large number of small BIO_DELETE commands. It assembles > blocks of adjacent deletes until they reach the size of a full erase > block, then issue one erase command (I've never been convinced that > doing so is necessary, to me the sd spec implies you can delete > individual sectors). > END QUOTE >=20 > I've no known, good means of isolating and > comparing/contrasting the two consolidations (vs. the > combination of both). >=20 > As stands, at best I'll be able to report if I see any > failures or evidence of any bottlenecks. (Similarly for > a later poudriere-devel bulk.) >=20 > The microsdcx card tested for this is from a: >=20 > Sandisk Ultra 128GB Micro SDXC UHS-I Card with Adapter = SDSQUAR-128G-GN6MA >=20 > It has the application "A1" rating. About 2/3 of the UFS > partition is "Avail". About 5.5 GiByte of the microsdcx card > is not in any partition (but that was dd'd from the e.MCC > too and I do not know how to readily trim those areas). >=20 > Other points of context: >=20 > The Pine64+ 2GB is running a non-debug kernel (but with > symbols), as is my norm unless I'm looking into a > failure. Summary: So far, so good. The incremental -j4 buildworld buildkernel completed fine in somewhat under 13 hr 20 minutes. It included building the llvm related materials. The installkernel . . . installworld . . . reboot worked fine as well. I saw no evidence of bottlenecks or other issues. The context is now -r338450 based. The kernel has Mark Johnston's patch for reporting on long latencies in swapping I/O. They did not report anything. There were no console messages at all during the build, so nor reported I/O warnings or errors. What I saw via "gstat -pd" looked good to me, as did what I was via "top -CawSores". (An RPI3 has only 1 GiByte of RAM instead of 2 GiByte, and so has more I/O required and would be a different test than I ran.) The svnlite update for /usr/ports and the the poudriere bulk with PARALLEL_JOBS=3D2 and ALLOW_MAKE_JOBS=3Dyes has built 25 of 26 ports for its update, the last one being a devel/gcc8 full bootstrap that will likely continue for 10 to 11 hours more. After the pkg update and pkg upgrade I'll add some x11 related ports to the file that I use with -f and start another poudriere bulk, at least that is my current plan. Note: My poudriere-devel use is based on: # poudriere jail -l JAILNAME VERSION ARCH METHOD TIMESTAMP = PATH FBSDcortexA53jail 12.0-CURRENT arm64.aarch64 null 2017-10-20 00:19:15 = /usr/obj/DESTDIRs/clang-cortexA53-installworld-poud The PATH is to an installworld result from the same buildworld used for the Pine64+ 2GB's own installworld, but also with "distrib-dirs distribution DB_FROM_SRC=3D1". Also: # poudriere ports -l PORTSTREE METHOD TIMESTAMP PATH default null 2017-10-20 00:19:48 /usr/ports currently with: # svnlite info /usr/ports/ | grep "Re[plv]" Relative URL: ^/head Repository Root: svn://svn.freebsd.org/ports Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5 Revision: 478753 Last Changed Rev: 478753 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Wed Sep 5 04:46:08 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27D62FE2E6F; Wed, 5 Sep 2018 04:46:08 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f68.google.com (mail-it0-f68.google.com [209.85.214.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A351E7089D; Wed, 5 Sep 2018 04:46:07 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f68.google.com with SMTP id j81-v6so8213709ite.0; Tue, 04 Sep 2018 21:46:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=OoWhYAHctVTxe05zzMqXe8eCxjffx2Ls5rIvvSc2eng=; b=hXzoDPyYLAwbPUzu5PPw7FfDeqK8YXv8zDNu50JKc31Nube0R4otMYaTwP1QU9NOiQ c4osuVQsLq/DF2EeirZVD2X77YHVGTVngV716cPh4mhrEJPQFpsimVbbLTOL9zy4Mm7T xHM53vQO75WwGMcO0dQwebb34zPo00fOOsoWqHqunLBy2Y1kwmb5wCS2x7CpWGdpptfX CTRxfkBYNTlSMClMGlUDAzNeTzPX5lFvAMISyZlUqpfe4UJh7AfsSBwHKchla96CooeW Slse5+psZLZ/+G9VKw5CQ4ABk3sX+H/V/SBlN/C2TXb5vneg47BKv4lMiXr8rJ+jBvRM VVBA== X-Gm-Message-State: APzg51DLv7SA8nGS423TDpqEJRYMIYDw5AlRWfcrytn1aoozfH2MMUxA RiKVCmp0OfvmdjiK+YZnNxCjUoQR X-Google-Smtp-Source: ANB0VdanrQjIRa9uAPEQJKoWDImHXIScYfZByDgepI4dhy0Nmg2BwJOgkeupulboKIUGwWykq5bHGA== X-Received: by 2002:a02:39a:: with SMTP id e26-v6mr25422749jae.135.1536122349172; Tue, 04 Sep 2018 21:39:09 -0700 (PDT) Received: from mail-it0-f45.google.com (mail-it0-f45.google.com. [209.85.214.45]) by smtp.gmail.com with ESMTPSA id b129-v6sm639178ioa.75.2018.09.04.21.39.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Sep 2018 21:39:08 -0700 (PDT) Received: by mail-it0-f45.google.com with SMTP id p129-v6so7847513ite.3; Tue, 04 Sep 2018 21:39:08 -0700 (PDT) X-Received: by 2002:a24:41cd:: with SMTP id b74-v6mr2516930itd.85.1536122348375; Tue, 04 Sep 2018 21:39:08 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Tue, 4 Sep 2018 21:39:07 -0700 (PDT) In-Reply-To: References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> From: Conrad Meyer Date: Tue, 4 Sep 2018 21:39:07 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Xin LI Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 04:46:08 -0000 With current libc, I instead see: load: 0.10 cmd: blocked_random_poc 1668 [randseed] 1.27r 0.00u 0.00s 0% 2328k (SIGINFO) $ procstat -kk 1668 PID TID COMM TDNAME KSTACK 1668 100609 blocked_random_poc - mi_switch+0xd3 sleepq_catch_signals+0x386 sleepq_timedwait_sig+0x12 _sleep+0x272 read_random_uio+0xb3 sys_getrandom+0xa3 amd64_syscall+0x940 fast_syscall_common+0x101 and: $ truss ./blocked_random_poc ... getrandom(0x7fffffffd340,40,0) ERR#35 'Resource temporarily unavailable' thr_self(0x7fffffffd310) = 0 (0x0) thr_kill(100609,SIGKILL) = 0 (0x0) SIGNAL 9 (SIGKILL) code=SI_NOINFO So getrandom(2) (via READ_RANDOM_UIO) is returning a bogus EAGAIN after we have already slept until random was seeded. This bubbles up to getentropy(3) -> arc4random(3), which sees a surprising failure from getentropy(3) and raises KILL against the program. I believe the EWOULDBLOCK is just a boring leak of tsleep(9)'s timeout condition. This may be sufficient to fix the problem: --- a/sys/dev/random/randomdev.c +++ b/sys/dev/random/randomdev.c @@ -156,6 +156,10 @@ READ_RANDOM_UIO(struct uio *uio, bool nonblock) error = tsleep(&random_alg_context, PCATCH, "randseed", hz/10); if (error == ERESTART || error == EINTR) break; + /* Squash hz/10 timeout condition */ + if (error == EWOULDBLOCK) + error = 0; + KASSERT(error == 0, ("unexpected %d", error)); } if (error == 0) { read_rate_increment((uio->uio_resid + sizeof(uint32_t))/sizeof(uint32_t)); Best, Conrad On Tue, Sep 4, 2018 at 8:13 PM, Conrad Meyer wrote: > Hi Lev, > > I took a first attempt at reproducing this problem on a fast > desktop-class system. First steps, give us a way to revert back to > unseeded status: > > --- a/sys/dev/random/fortuna.c > +++ b/sys/dev/random/fortuna.c > @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); > > #ifdef _KERNEL > #include > +#include > #include > #include > #include > @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) > return; > } > > + /* > + * When set, pretend we do not have enough entropy to reseed yet. > + */ > + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { > + if (RETURN_VALUE != 0) { > + RANDOM_RESEED_UNLOCK(); > + return; > + } > + }); > + > + > #ifdef _KERNEL > fortuna_state.fs_lasttime = now; > #endif > @@ -442,5 +454,11 @@ bool > random_fortuna_seeded(void) > { > > + /* When set, act as if we are not seeded. */ > + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { > + if (RETURN_VALUE != 0) > + fortuna_state.fs_counter = UINT128_ZERO; > + }); > + > return (!uint128_is_zero(fortuna_state.fs_counter)); > } > > > Second step, enable the failpoints and launch repro program: > > $ sudo sysctl debug.fail_point.random_fortuna_pre_read='return(1)' > debug.fail_point.random_fortuna_pre_read: off -> return(1) > $ sudo sysctl debug.fail_point.random_fortuna_seeded='return(1)' > debug.fail_point.random_fortuna_seeded: off -> return(1) > > $ cat ./blocked_random_poc.c > #include > #include > #include > > int > main(int argc, char **argv) > { > printf("%x\n", arc4random()); > return (0); > } > > > $ ./blocked_random_poc > ... > > > Third step, I looked at what that process was doing: > > Curiously, it is not in getrandom() at all, but instead the ARND > sysctl fallback. I probably need to rebuild world (libc) to test this > (new libc arc4random based on Chacha). > > $ procstat -kk 1196 > PID TID COMM TDNAME KSTACK > 1196 100435 blocked_random_poc - read_random+0x3d > sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 > sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b > amd64_syscall+0x940 fast_syscall_common+0x101 > > > When I unblocked the failpoints, it completed successfully: > > $ sudo sysctl debug.fail_point.random_fortuna_pre_read='off' > debug.fail_point.random_fortuna_pre_read: return(1) -> off > $ sudo sysctl debug.fail_point.random_fortuna_seeded=off > debug.fail_point.random_fortuna_seeded: return(1) -> off > > ... > 9e5eb30f > > > Best, > Conrad From owner-freebsd-fs@freebsd.org Wed Sep 5 05:00:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C161FE338E; Wed, 5 Sep 2018 05:00:43 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12D8570F70; Wed, 5 Sep 2018 05:00:43 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from odin.corp.delphij.net (unknown [IPv6:2601:646:8882:37a:a954:32ec:5043:1ba5]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: delphij/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 5A45111168; Wed, 5 Sep 2018 05:00:42 +0000 (UTC) (envelope-from delphij@FreeBSD.org) To: cem@freebsd.org, Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Mark R V Murray , re@FreeBSD.org References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> From: Xin Li Openpgp: preference=signencrypt Autocrypt: addr=delphij@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFJNzwQBEACuPNSJjL/AD8oHFuG72vtx5P7Q6dpiEbFABgw/IohS65yDZDd3qFH9ssQv AsFafwB/ofsk6t7dx6zIC05dv5qjhGIOKSJxFC4U1HAot9+QpeUG+8boTKZiiycrMruItj2U JANlv+gN5h0mAsL5f9eNzhRM43kdjN8cQnBIujhO54Derjnrnqz6cQtoonV6SvvVJZUQGxHK 5R1XYJ6wiTuvoEuRYnNObJmPFWZyYOaGZz0qqD6Qe1BhkZuRzv2bZxwJc3Raap/GF6Pm9J/c hlYHUmm2QLaXvmoP8WNosNjla1fup0tgYQE+7MTtHFVxmVj9ZTihN3rEL5IkeEKjQAqcpe1n Db8X2o4K262LRpFl8WtVMW2TfN5Avpj+knZMl3tkYGvYK/nfadCr6Af4co9mkhX6QYgkerg2 mXEGaQzSD/omnsxHCfqMgdphaX3B3eoY2Fv36BMpjSdHmm0rmwqjqZaqlZn89vQ/I6ATvLyx JsdHwTbrj57audl/RKC+OpREOJPaVULp1L+9zdBXslILO8MJaT6YEw1T29bEj5jvLm03Y4rF u/YTruHcMPpsGbpJckDKiy6ISAbMtPvz7/KR91xPHS6KExGiIakIX9xpIXIDKgq+ecEWwkFK PogoKqO6K0/GYkTRoKdXGzsILvIurtbPqSFqWzbRIyNOa82jowARAQABzRZYaW4gTGkgPGRA ZGVscGhpai5uZXQ+wsF9BBMBCgAnBQJTQvBFAhsjBQkJZgGABQsJCAcDBRUKCQgLBRYCAwEA Ah4BAheAAAoJEJW2GBstM+nsha4P/2Roa/REjZLZlIG1TKOxEDqmwc3fynX4w2g7/FXA7f7Z YO5N4vnnnQdJbDZDt4TJtiP1NHHdheQ5+loJrrCXVlU31LuJv1ebM2Ajsuo/0l3tfulEf6Ki GoozmaNZAhwiGJkQVg9DSKsea5xIA31lPnFH4T0SKn8Q6F4HYienmJJtlKVTADvYXA+DRmv0 rNOyVe+V/AuTFuelKg3Ua5a+dY3oqtrQQvFS4n7iIrNjEMUBVx0XTrYLddnF+YjXDg5Phf0D pV/2yJOXiTGiZMK6i7vwHZkJvarACoTSrUrr6OBuZv5Gf87VgifZKLr2Fuf+FePiVCoZTQiL 0hPQyABMzeWa32P6BY2LBMMMFvFiyL5pN5k6nJ0nx4skl8UxZ5ay4yyVg2u3f4aI3+m0XlZ+ iixrjmCTGi1s+d/n6E3eFXdJUUbSOXLZaU4qrbXRzTYCZmZViryv7ibtOHXnG6oWy7BFEHuT rUW6OBvsQDTp5iQ6opENJ5/ZzSA3c5p1WS9Ezv4Bpdqcm7LTQX2j6kXikj8YqICtDF2rkKZ2 Ynjm9se9B0h/T1SOaSpbtRg05UKjsinDq2x8EeX21yFs3UyvwePLrGoNKL45EJM0xwxrnlfr M0ayKJNLoYysY78d54hg7XMmkQD/oZz9I+k4fN6CmZ2i5WGH2BgYs0313JMHxSg7zsFNBFJN zwQBEADPtS+nfTKM6PwgSWLDGVgUYQ/RLaKzCcpQAf4ryLBugXpx3s2BBT1bixX7CpsLXKQi +RRETgSFzDaBL9SEs2ZDV2YT+zGp08aijK/Yl9+RIeezAukI3c+XMHuo8ktUWJmo5/1DX07q G30ckG7uFuTnt31sFzwhh/ZeSuLFyel/fWF48KExLDIVa8DyEUJaYvE9Vfph4T/3LkKuzVTy +iwUBLiSLj5G5N70A+4usbL3eKyYrJqCSaLfrP99/nlgBhMAHVcKcv0uqSuiaH9OMqg1VjQs N8j6NDQug9QrbBTM6U7oZWF/AK+CdFoe+leq5MZfzwCevs0BQgxWm4SHMpXL2vtly67QSPMY dl96fOzw8YbKHv1o0ixhCvc37cI9oUVuSJLXKhEEAvWvLuusiuNeoz+6aPlELvD8h5txJqui tVOzctvJ7ktGZTNiz73tKYVdkKaQVyo8QJFLCNLnUulrQ5wXwteYPg6mrpBxu9VqgDrMp7eB T2kaZ4GRBoMWXXPYSIEe5PM5hhNCsSUfqrKj34UZPijPe+HiWoFJ4S5vIpzutiae11Ctki7u XzeLAhOJQB2raraIqDlFP9I9Zj9JOAZhmiKSEWKfOooCNxQYGiUdPrdYnAe+m7FXRomjF0OO gSepNIESt2gOEIbE5cMxQ0gAueNJc58eHCjWhsNJIwARAQABwsFlBBgBCgAPBQJSTc8EAhsM BQkJZgGAAAoJEJW2GBstM+nsh8EP/1sxZpkJelu+smmqaqdrGHlNrFVLOmeN5yr2IGHBUbmF htjr7fVoU8T0mUnlUU724aKPla4nWhMb4NMu+VxRRFGaT2TYpyR6VIxaStycyUdMGjdXV0Pz TGmxFXhNZXKEITXH9sIxuONBp1czl4AgwN7AAl1MKyV13AaLIyajs58mYmuXtyFn/O+4lxh5 nl2Fa3L9YkL9O7QU2p6WAnDky+L3PgUWp1AzJGfYlLZ8XXCi+KK+pnta+f9yKHt/Oqd/s7OC W4mXgFkBrfuSZZofa4eZckh5u0yBYW3OnEJhClgxRbuOhyYwqQr5oxPrQtjtbMiBzbrOkHhy NnrVCFd9EqlojREGDefHo3V+ZlUOc6OoN3CAYnNa2uLEOm5DCuqOE4z5atBCih5EyITPp7JP J2disEP6ddipcilqbnJdP+TyRQwSv5qRNy8cHahD1Cg9XJJHiC3qr+W3eOtqPkJxhU5biPEr 7dljaLS1Ij771brzqO/x5zW1L9py7muXzYBsW8+keKj8LOYs2242KgjI5Og9YhIJGBFBNddQ wxKBKQpytKQOiXwjhk4Nj77U796bsCd/jIS0r0ZUKBEptPyKso7ncfrm163aEmSaDUkiIjyp 9CEOVT87D+VAVh9PyLGP1niQzWEWFSK36tRGZlF0odP1ZB6wub9zq2DxFouSjHgH Organization: The FreeBSD Project Subject: Re: newfs silently fails if random is not ready (?) Message-ID: Date: Tue, 4 Sep 2018 22:00:34 -0700 User-Agent: Thunderbird MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 05:00:43 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK Content-Type: multipart/mixed; boundary="nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq"; protected-headers="v1" From: Xin Li To: cem@freebsd.org, Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Mark R V Murray , re@FreeBSD.org Message-ID: Subject: Re: newfs silently fails if random is not ready (?) References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> In-Reply-To: --nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 9/4/18 21:39, Conrad Meyer wrote: > With current libc, I instead see: >=20 > load: 0.10 cmd: blocked_random_poc 1668 [randseed] 1.27r 0.00u 0.00s > 0% 2328k (SIGINFO) >=20 > $ procstat -kk 1668 > PID TID COMM TDNAME KSTACK > 1668 100609 blocked_random_poc - mi_switch+0xd3 > sleepq_catch_signals+0x386 sleepq_timedwait_sig+0x12 _sleep+0x272 > read_random_uio+0xb3 sys_getrandom+0xa3 amd64_syscall+0x940 > fast_syscall_common+0x101 >=20 > and: >=20 > $ truss ./blocked_random_poc > ... > getrandom(0x7fffffffd340,40,0) ERR#35 'Resource > temporarily unavailable' > thr_self(0x7fffffffd310) =3D 0 (0x0) > thr_kill(100609,SIGKILL) =3D 0 (0x0) > SIGNAL 9 (SIGKILL) code=3DSI_NOINFO >=20 > So getrandom(2) (via READ_RANDOM_UIO) is returning a bogus EAGAIN > after we have already slept until random was seeded. This bubbles up > to getentropy(3) -> arc4random(3), which sees a surprising failure > from getentropy(3) and raises KILL against the program. >=20 > I believe the EWOULDBLOCK is just a boring leak of tsleep(9)'s timeout > condition. This may be sufficient to fix the problem: >=20 > --- a/sys/dev/random/randomdev.c > +++ b/sys/dev/random/randomdev.c > @@ -156,6 +156,10 @@ READ_RANDOM_UIO(struct uio *uio, bool nonblock) > error =3D tsleep(&random_alg_context, PCATCH, "randseed= ", hz/10); > if (error =3D=3D ERESTART || error =3D=3D EINTR) > break; > + /* Squash hz/10 timeout condition */ > + if (error =3D=3D EWOULDBLOCK) > + error =3D 0; > + KASSERT(error =3D=3D 0, ("unexpected %d", error)); > } > if (error =3D=3D 0) { > read_rate_increment((uio->uio_resid + > sizeof(uint32_t))/sizeof(uint32_t)); +markm, re I think the proposed change is reasonable (note that I think the same theory applies to the tsleep_sbt() case below as well, which should be handled similarly). > Best, > Conrad >=20 >=20 > On Tue, Sep 4, 2018 at 8:13 PM, Conrad Meyer wrote: >> Hi Lev, >> >> I took a first attempt at reproducing this problem on a fast >> desktop-class system. First steps, give us a way to revert back to >> unseeded status: >> >> --- a/sys/dev/random/fortuna.c >> +++ b/sys/dev/random/fortuna.c >> @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); >> >> #ifdef _KERNEL >> #include >> +#include >> #include >> #include >> #include >> @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) >> return; >> } >> >> + /* >> + * When set, pretend we do not have enough entropy to reseed y= et. >> + */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { >> + if (RETURN_VALUE !=3D 0) { >> + RANDOM_RESEED_UNLOCK(); >> + return; >> + } >> + }); >> + >> + >> #ifdef _KERNEL >> fortuna_state.fs_lasttime =3D now; >> #endif >> @@ -442,5 +454,11 @@ bool >> random_fortuna_seeded(void) >> { >> >> + /* When set, act as if we are not seeded. */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { >> + if (RETURN_VALUE !=3D 0) >> + fortuna_state.fs_counter =3D UINT128_ZERO; >> + }); >> + >> return (!uint128_is_zero(fortuna_state.fs_counter)); >> } >> >> >> Second step, enable the failpoints and launch repro program: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read=3D'return(1)' >> debug.fail_point.random_fortuna_pre_read: off -> return(1) >> $ sudo sysctl debug.fail_point.random_fortuna_seeded=3D'return(1)' >> debug.fail_point.random_fortuna_seeded: off -> return(1) >> >> $ cat ./blocked_random_poc.c >> #include >> #include >> #include >> >> int >> main(int argc, char **argv) >> { >> printf("%x\n", arc4random()); >> return (0); >> } >> >> >> $ ./blocked_random_poc >> ... >> >> >> Third step, I looked at what that process was doing: >> >> Curiously, it is not in getrandom() at all, but instead the ARND >> sysctl fallback. I probably need to rebuild world (libc) to test this= >> (new libc arc4random based on Chacha). >> >> $ procstat -kk 1196 >> PID TID COMM TDNAME KSTACK >> 1196 100435 blocked_random_poc - read_random+0x3d >> sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 >> sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b >> amd64_syscall+0x940 fast_syscall_common+0x101 >> >> >> When I unblocked the failpoints, it completed successfully: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read=3D'off' >> debug.fail_point.random_fortuna_pre_read: return(1) -> off >> $ sudo sysctl debug.fail_point.random_fortuna_seeded=3Doff >> debug.fail_point.random_fortuna_seeded: return(1) -> off >> >> ... >> 9e5eb30f >> >> >> Best, >> Conrad --nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq-- --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJbj2L3AAoJEJW2GBstM+nsxVwP/jIyR53g2isbfBVdaseuiiCs Ql9eS1x1xzpIxAHPAndb4bPdROmpZzIgeocZZ1wRM1h/A6Z3isS8AJmtww4D6+W7 Hwm+r1nyGDBv7wgUMqMavQs1JIMpimv/pDScbXD43chlB7n5p1BdSdAcQuu4d3Aq eVrm1eIaVzTldmA5TVS9lBtqkXI9RCx0fwDccDujPB2DNxZoHcp+1h7rNkL31yRg UzF8PtaMLgN1LeDT0BXtYsjtUCZgZtJSZ9PzZWFCjGYVitBYIHYrdrXKbLBjDE00 HEVD/Eyb9dhBhJqFQ9kIprcFJujoY9pAjaDL/qIA8ZPCvyUDt7hbIuaWjxZaC2ep RCAAB5btM9KTRpNAsqt0MhSJC+I/dFmWcgheG4+XOEMSUFlluoIfxVeTFDjgOzt8 OUjc2oLyn8uPCsJQg4q48WwrUGH4hDv8hccFJ1WH7rhfMcR8/51jQHvWt7ObKcx+ mHZUoYBgusePhD1OO/XasBcSwmABviXzpWk/Q6UaFbnFa8BX0uYwFH3dNEeIOmBO Y6ZkoWL2Bg3fdyTHYWe1pnysdQ2DowCxyS8RL1HQgoOAOULnkIHK6MKhNGYVJYn7 bK08/nczyKXz1a2vujPboYLwGfTcyYZb0tLwApRnvnU74jao+nuQO06oZ2TU17uB szrA1pupCukN9iedsogh =auFP -----END PGP SIGNATURE----- --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK-- From owner-freebsd-fs@freebsd.org Wed Sep 5 05:15:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 629F7FE3BA1 for ; Wed, 5 Sep 2018 05:15:42 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7E277196C for ; Wed, 5 Sep 2018 05:15:41 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w855KwxS083430; Tue, 4 Sep 2018 22:20:58 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201809050520.w855KwxS083430@chez.mckusick.com> From: Kirk McKusick To: Mark Millard Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: bob prohaska , FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <373CBC85-7A7C-4EEE-9A50-6AEAFEC05F63@yahoo.com> Comments: In-reply-to Mark Millard message dated "Tue, 04 Sep 2018 20:16:37 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <83428.1536124858.1@chez.mckusick.com> Date: Tue, 04 Sep 2018 22:20:58 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 05:15:42 -0000 Thanks for the report. Do you have a sense/measurement that the build times better/same/worse than without the consolidation? Kirk McKusick From owner-freebsd-fs@freebsd.org Wed Sep 5 07:37:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2096AFE6897 for ; Wed, 5 Sep 2018 07:37:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.gq1.yahoo.com (sonic309-22.consmr.mail.gq1.yahoo.com [98.137.65.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8779675974 for ; Wed, 5 Sep 2018 07:37:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: fNKKAh8VM1mw2arjdvo2rn2DN2VlUNkKdIfLlaVFxyL2hz0bizyQbmuWWqx3ZzS 9Mb75B1zIl_AIWIcobKWVpthcpj7j3Gt8uE3LLwWjvVkoYW6A3_PXhn41za1xVnpHhdgAguzCtkV ZkVMihgHJs2K0QTMc0s1Cf0eLJNHUOlnFdaMMnD1SmEiskUF.FP3jfWGQ8ImJ9WTNimDDiqLS0TR IQerNO0eoEshulN5GLg_CP77fCFpc.suf4GxOT5JrcMmJyC4lm599Ib3nIFHSAu6Dzjoe9HHcQjQ RbKxd.kSia3esV1H5nVsF6uySDIkHLdFVSQrk4247CBeqduJ9uztNPkRRAD5TxdNhTPr9X9wAU4x QWa7AHQ1mMxKlqDRkUqvuCnn6wzT_PSpV2i2x3QYHzggoM1fsJnqRIe9apts3zmSfaqxHI9jX5SP 5zaG7RD8paLqFz57ke6lRE02i3MC.x6LtdV0WNbK2_mKaEGaFNarnO829NrZavnLcXI7Af4rim8o JKROvT3UBErPre8.hDmkR21yDkSk2jG2c_n3GP22aasxgDo1IrlxUARt5h9Exzx83SmUW8JIn6DD llE2mZJnOiSk6mNDyBH6W8XMggfaeqcjmYRsShSukcQm7LdcNV0OJVZPiEh5hwlxNiwpK9nN5T4D 6SDNWz_AME2ydTCl_z6BH7eSFuD63_KCOibk.ZtoV43eAM7ler4ypJw9sXd8m_31t2Eiwd3ZvC_K PRw12Q16S5uPQvvWg8J6t6pGMJ2o64K94xhDP_rZoZsGjCIj6Le9dzoFy.OJbKStAqAXdDfcsIXD WpFyRqgURBFm0G1ip37kUc7Gz74svit6bNr0TagyQL3WIGSdlolZOT_Wc6VQUnbUMHBw0_HWhmPE cfABPrzXKVjgEsDMKsq1ZFcEcZRigsujrrQUDUny6ehnNHwdTu.V2Rt1NabTZsgCSAzT3Rt2dczc uMlW82yYAy3KKVKKz09jGuxdQ1GK_FF7br0oIf3iL_3_eEheIFLqDpUSVsa8fV1sNaMyyVrwnRNd Ax.JgymSB Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Wed, 5 Sep 2018 07:37:23 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp411.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 26d886d48418080755230e2691722203; Wed, 05 Sep 2018 07:06:55 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <201809050520.w855KwxS083430@chez.mckusick.com> Date: Wed, 5 Sep 2018 00:06:54 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: 7bit Message-Id: References: <201809050520.w855KwxS083430@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 07:37:25 -0000 On 2018-Sep-4, at 10:20 PM, Kirk McKusick wrote: > Thanks for the report. Do you have a sense/measurement that the > build times better/same/worse than without the consolidation? buildworld buildkernel is far from I/O bound overall (elapsed time) when built via clang when the llvm materials are being built in this context: CPU/RAM bound. (I watched gstat -pd and top -CawSores for evidence of this.) Also, swap/paging I/O would not involve consolidation. It would be odd for the new consolidation to make much of a difference for this activity. (gcc 4.2.1 based builds were more I/O bound in my experience.) But installworld is far more I/O bound and does take something like 11 or 12 minutes in this context. (installkernel is also more I/O bound but takes far less time.) I have a record of an updating installworld for clang-cortexA53-installworld-poud already with consolidation turned on . . . Start time: 2018-09-04:12:48:14 End time: 2018-09-04:12:59:54 (so 11 min 40 sec or so elapsed) typescript log size: 8698790 So after the poudriere/pkg activity I could repeat the "-j4 installworld distrib-dirs distribution DB_FROM_SRC=1 DESTDIR=/usr/obj/DESTDIRs/clang-cortexA53-installworld-poud" with the new consolidation turned off. (That would leave active the mmc/sd code's consolidation that Ian L. referenced.) I could also try yet again but with trim disabled for the file system, giving a 3rd contrasting case. Would these comparisons help? As for my poudriere-devel use, I previously used PARALLEL_JOBS=1 but did this activity with PARALLEL_JOBS=2 (both with ALLOW_MAKE_JOBS=yes ). PARALLEL_JOBS=2 makes comparing individual port-build times a problem because of competing use of the CPUs over the same time period. There also is the issue of how I/O bound each port's build is or is not for the context. Using "gstat -pd" and "top -CawSores" to watch devel/gcc8 build indicates I/O %busy is usually < 2%, even < 1%, and much of the time is 0.0% or 0.1%. (This is during a "prev-gcc" bootstrap stage, no longer using clang.) It would be odd for the new consolidation to make much of an overall difference here. A more I/O bound port build? Note: I mount with -o noatime in use. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Wed Sep 5 07:57:41 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D57B4FE6F75; Wed, 5 Sep 2018 07:57:40 +0000 (UTC) (envelope-from markm@FreeBSD.org) Received: from gromit.grondar.org (grandfather.grondar.org [IPv6:2a01:348:0:15:5d59:5c20:0:2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66C897620B; Wed, 5 Sep 2018 07:57:40 +0000 (UTC) (envelope-from markm@FreeBSD.org) Received: from graveyard.grondar.org ([88.96.155.33] helo=gronkulator.grondar.org) by gromit.grondar.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fxSh0-00064g-M2; Wed, 05 Sep 2018 08:57:38 +0100 Content-Type: multipart/signed; boundary="Apple-Mail=_BFFD48EF-4EA4-4154-B13C-6C02B2D6201C"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: newfs silently fails if random is not ready (?) From: Mark R V Murray In-Reply-To: Date: Wed, 5 Sep 2018 08:57:37 +0100 Cc: Lev Serebryakov , FreeBSD Current , freebsd-fs , Xin LI Message-Id: <4637985A-28EF-4A6B-B8A6-764A86005E6B@FreeBSD.org> References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> To: cem@freebsd.org X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 07:57:41 -0000 --Apple-Mail=_BFFD48EF-4EA4-4154-B13C-6C02B2D6201C Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Nice catch! Thanks :-) M > On 5 Sep 2018, at 04:13, Conrad Meyer wrote: > > Hi Lev, > > I took a first attempt at reproducing this problem on a fast > desktop-class system. First steps, give us a way to revert back to > unseeded status: > > --- a/sys/dev/random/fortuna.c > +++ b/sys/dev/random/fortuna.c > @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); > > #ifdef _KERNEL > #include > +#include > #include > #include > #include > @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) > return; > } > > + /* > + * When set, pretend we do not have enough entropy to reseed yet. > + */ > + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { > + if (RETURN_VALUE != 0) { > + RANDOM_RESEED_UNLOCK(); > + return; > + } > + }); > + > + > #ifdef _KERNEL > fortuna_state.fs_lasttime = now; > #endif > @@ -442,5 +454,11 @@ bool > random_fortuna_seeded(void) > { > > + /* When set, act as if we are not seeded. */ > + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { > + if (RETURN_VALUE != 0) > + fortuna_state.fs_counter = UINT128_ZERO; > + }); > + > return (!uint128_is_zero(fortuna_state.fs_counter)); > } > > > Second step, enable the failpoints and launch repro program: > > $ sudo sysctl debug.fail_point.random_fortuna_pre_read='return(1)' > debug.fail_point.random_fortuna_pre_read: off -> return(1) > $ sudo sysctl debug.fail_point.random_fortuna_seeded='return(1)' > debug.fail_point.random_fortuna_seeded: off -> return(1) > > $ cat ./blocked_random_poc.c > #include > #include > #include > > int > main(int argc, char **argv) > { > printf("%x\n", arc4random()); > return (0); > } > > > $ ./blocked_random_poc > ... > > > Third step, I looked at what that process was doing: > > Curiously, it is not in getrandom() at all, but instead the ARND > sysctl fallback. I probably need to rebuild world (libc) to test this > (new libc arc4random based on Chacha). > > $ procstat -kk 1196 > PID TID COMM TDNAME KSTACK > 1196 100435 blocked_random_poc - read_random+0x3d > sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 > sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b > amd64_syscall+0x940 fast_syscall_common+0x101 > > > When I unblocked the failpoints, it completed successfully: > > $ sudo sysctl debug.fail_point.random_fortuna_pre_read='off' > debug.fail_point.random_fortuna_pre_read: return(1) -> off > $ sudo sysctl debug.fail_point.random_fortuna_seeded=off > debug.fail_point.random_fortuna_seeded: return(1) -> off > > ... > 9e5eb30f > > > Best, > Conrad > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- Mark R V Murray --Apple-Mail=_BFFD48EF-4EA4-4154-B13C-6C02B2D6201C Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 Comment: GPGTools - http://gpgtools.org iQEzBAEBCgAdFiEEyzPHvybPbOpU9MCxQlsJDh9CUqAFAluPjHEACgkQQlsJDh9C UqBdHwgAg+abT1fvBHrfDw1OvTYLA//1b3KRVkdYTUjMrdzm8g68y5ZThM9L14U3 yDeGszrDCIdnVJ9bBxGjxDeqSPY/3m+0SY2qdCT9Ly3r3t4o08WKbLqXjhooaVQE D5Ag72Q2ehWsR+/squ/Z6+3PQWkgWRE/RxTbwjOOJdZoBdJdArV/wSwOTTmcKEwG kZtxcthHbptf1RGeL+3vlVCXR4L5OoJhTym/DIdkE5rQek6cU+16nUwxNZ1NUxwf EZ07pB6pdQCfwwrh23823/zIW9CXeAxzuAf3U4M1v2EiMcgsO1TmzEVWShygNR64 X3YtB2gi3UkuXWa4TXjA77vzz5M+GA== =mSJR -----END PGP SIGNATURE----- --Apple-Mail=_BFFD48EF-4EA4-4154-B13C-6C02B2D6201C-- From owner-freebsd-fs@freebsd.org Wed Sep 5 08:49:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 529F8FE9125 for ; Wed, 5 Sep 2018 08:49:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E266677FDC for ; Wed, 5 Sep 2018 08:49:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A7446FE9118; Wed, 5 Sep 2018 08:49:42 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 960C3FE9117 for ; Wed, 5 Sep 2018 08:49:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3889D77FD9 for ; Wed, 5 Sep 2018 08:49:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 6B8D716FAF for ; Wed, 5 Sep 2018 08:49:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w858nfYm092915 for ; Wed, 5 Sep 2018 08:49:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w858nfWw092914 for fs@FreeBSD.org; Wed, 5 Sep 2018 08:49:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 228750] panic on zfs mirror removal Date: Wed, 05 Sep 2018 08:49:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: crest@rlwinm.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 08:49:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228750 crest@rlwinm.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |crest@rlwinm.de --- Comment #3 from crest@rlwinm.de --- I just ran into the same problem on FreeBSD 11.2-p2. Please put at least a warning in if the feature is not (yet) usable. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Sep 5 11:42:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5DF3FECE13; Wed, 5 Sep 2018 11:42:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 896CA7D333; Wed, 5 Sep 2018 11:42:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [94.19.235.70]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 6C690CB0; Wed, 5 Sep 2018 14:42:21 +0300 (MSK) Date: Wed, 5 Sep 2018 14:42:19 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <532863197.20180905144219@serebryakov.spb.ru> To: Conrad Meyer CC: freebsd-fs , FreeBSD Current , Xin LI Subject: Re: newfs silently fails if random is not ready (?) In-Reply-To: References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 11:42:29 -0000 Hello Conrad, Wednesday, September 5, 2018, 7:39:07 AM, you wrote: > I believe the EWOULDBLOCK is just a boring leak of tsleep(9)'s timeout > condition. This may be sufficient to fix the problem: > --- a/sys/dev/random/randomdev.c > +++ b/sys/dev/random/randomdev.c > @@ -156,6 +156,10 @@ READ_RANDOM_UIO(struct uio *uio, bool nonblock) > error = tsleep(&random_alg_context, PCATCH, "randseed", hz/10); > if (error == ERESTART || error == EINTR) > break; > + /* Squash hz/10 timeout condition */ > + if (error == EWOULDBLOCK) > + error = 0; > + KASSERT(error == 0, ("unexpected %d", error)); > } > if (error == 0) { > read_rate_increment((uio->uio_resid + > sizeof(uint32_t))/sizeof(uint32_t)); Fantastic! Thanks! -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Wed Sep 5 14:05:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5182FF1130; Wed, 5 Sep 2018 14:05:09 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1761E825EE; Wed, 5 Sep 2018 14:05:09 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: by mail-lf1-x134.google.com with SMTP id h64-v6so6086792lfi.10; Wed, 05 Sep 2018 07:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=SYiJ4UFamiWU0/ZAzUyHRat24KWTcNJKf+KMjcoJU3k=; b=EOacOsNI2wJJ5cOqGHdbg4wD93WtHDGWAetSfCk4C2gwZOBziixzhYT8B3tu2m2WAT kmxRLgcSG/GnCVrEs9CxC8aaLku3aoCWhR+uBx9QkWnthz6cpdIA7+k6Pvi4rUvA0ZvL uV51LBH7L5ZieM3c7HoobuqB+bT1wk4P2jlizn11Giyr8oy9S0Klj9xLqn1Z/NpGyfjK MBbn7K1omnlAbB4GAsADLWj/pcniBRIhgwcgRvxbNdtk1jXROOigdbGzmtW5BaCTjZyc unN0AQkUELJiC21GWuBKqzSoWW+rslt6U6X/2ggrUpmFxuxEwcuzL8yqok/RQFL2TU3v M2hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=SYiJ4UFamiWU0/ZAzUyHRat24KWTcNJKf+KMjcoJU3k=; b=MmZw9OY97jVXiWoVaAWoeRuATGDdYmwlwz/2XnEvi2I6MqbqJMU6+Qv6NbHBxad6bt BBH3lEPhJtdgGSd5yStumgK+HlY4aDam40ut1oTw79rW6dS55srCfCQDkChT9O6sO6QJ E3UbbFQ5jsHqv8NbDR6C2dHvXt+SxAiaUgZb0IaQonNI9fr5xhpPcqqXOFGs0wPMWLQ6 sO4Ild5U7o+fakDtBzDmQEeJuLpxGjLyRaIBeh2NgdpqiGLdcg48vgekxPTw9feeT1pl sZR9E03HghA/kNssSAasPqTZhwws2poqaAurwIq6iuToD/6g6GWzVdLKG9JNF+Aff7hD D9dQ== X-Gm-Message-State: APzg51BeGT6idtImQFPsesPbz7ktaQ8Yu0AGuOgWVi3rIyTBu7xkv5/r oWU7LZeDuT3wAiGSFRtdpTe9hk/H39sMg5cnGmGVhc8h X-Google-Smtp-Source: ANB0Vdbb7e7nhRo22QxFjbaSeEZBXlkKfHQ9EQXAY8qbbj0Xr3AiEtDiBmwD6Hcf5QDesM21XAsHnoUI9KuKkn90M/Q= X-Received: by 2002:a19:a915:: with SMTP id s21-v6mr24625868lfe.92.1536156307583; Wed, 05 Sep 2018 07:05:07 -0700 (PDT) MIME-Version: 1.0 From: Subbsd Date: Wed, 5 Sep 2018 17:04:56 +0300 Message-ID: Subject: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 To: freebsd-current Current , freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 14:05:09 -0000 Hi, I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 to latest revision (r338466 the moment) and related to ARC. I can not say which revision was before except that the newver.sh pointed to ALPHA3. Problems are observed if you try to limit ARC. In my case: vfs.zfs.arc_max="128M" I know that this is very small. However, for two years with this there were no problems. When i send SIGINFO to process which is currently working with ZFS, i see "arc_reclaim_waiters_cv": e.g when i type: /bin/csh I have time (~5 seconds) to press several times 'ctrl+t' before csh is executed: load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s 0% 3512k load: 0.70 cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s 0% 3512k load: 0.73 cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s 0% 4156k same story with find or any other commans: load: 0.34 cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00s 0% 2676k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00s 0% 2680k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00s 0% 2684k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00s 0% 2704k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00s 0% 2716k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00s 0% 2760k this problem goes away after increasing vfs.zfs.arc_max From owner-freebsd-fs@freebsd.org Wed Sep 5 14:50:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D0C3FF27E4; Wed, 5 Sep 2018 14:50:53 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 841588440D; Wed, 5 Sep 2018 14:50:52 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf1-x434.google.com with SMTP id k19-v6so3621572pfi.1; Wed, 05 Sep 2018 07:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Oi8f3AoRcxGoLa8AqgNxVvYRZW3eQDYx2AwcFskK5QE=; b=BBOlbdn+k12VLuGu8/txuh3S21OTbNKoNm31YXKz0imPp9mZ+HlpSO67h3crbmMAv5 nRYRH8kj94BoRjnq7orNaS3dhP3HYAGpUVgQvhBwGDzKxjeaQ4363rKBOOJlmD3Csee+ tP8NSOIvTC9KNzfGV8tIQiutBtLsgzdiCFWA1W3LzXt5nd5VuKPJb3RHFtmHc6XuGmZP USJtBZGrgga6b3a0G3qM0QWkukPR0mYpAn2ZKtPUmDzDPPAUsR8DrN3tdkQQKUiBO1eX KQyGRA6ZxUBQwDD6winKqXmLnmd7/p8ZO43A2IEOqs/FWaf6W8HDmLRwfhcRGR8mUlaG 1xNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=Oi8f3AoRcxGoLa8AqgNxVvYRZW3eQDYx2AwcFskK5QE=; b=QEZYA42TAAEKe54LzzLzMRcU4OxilGR7GMtqX/6DQ4RBMaC1ueovDLHlIDiFq1Fcs8 6R4TUkbIaww46ZQNdNUNbckFjtu6/lAajzRoiOC4WVMCYGH4oeHlGx2VWR4jcqlY4RB8 xXi0N7SXXEwYtBDkTQZLBZGkRVxUXxfP5vhh5A/UZpzS+xzjZlkf6xnvGay3R+kYIPMY Hf4St5kl6bcOxhBItP8kgx0Nh7ldXGWCHlrwDKUZ6k1bPDg50oVGZZ6OmWXACD3EqzwE iBAllamgzldGFlVA1kkLYZNKBP5yEiYFj9Z967oNtZPsUutp/Cya0nUYYYSS7KcIYFe3 WIJA== X-Gm-Message-State: APzg51DGy41e0S0Ryt88XGrXX4UgZyVRhSGVOhxx8U+EUVGyegDCQ8dh jFyRRqgiYodGu/S2WDqCmTazv/iy X-Google-Smtp-Source: ANB0VdYKprZurHWGHd/5aa7x0zJDpPnx2dAwXbCPzMAxAz3WxnYvIVdmA6CgPF+WqaiNf02DWsHJlQ== X-Received: by 2002:a62:2483:: with SMTP id k3-v6mr41373058pfk.195.1536159051560; Wed, 05 Sep 2018 07:50:51 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id d22-v6sm8055727pfm.48.2018.09.05.07.50.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Sep 2018 07:50:50 -0700 (PDT) Sender: Mark Johnston Date: Wed, 5 Sep 2018 10:50:45 -0400 From: Mark Johnston To: Subbsd Cc: freebsd-current Current , freebsd-fs@freebsd.org Subject: Re: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 Message-ID: <20180905145045.GA37268@raichu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 14:50:53 -0000 On Wed, Sep 05, 2018 at 05:04:56PM +0300, Subbsd wrote: > Hi, > > I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 > to latest revision (r338466 the moment) and related to ARC. > > I can not say which revision was before except that the newver.sh > pointed to ALPHA3. Could you please try reverting r338416 to see if that fixes the problem? > Problems are observed if you try to limit ARC. In my case: > > vfs.zfs.arc_max="128M" > > I know that this is very small. However, for two years with this there > were no problems. Two years on -CURRENT? How often did you update? From owner-freebsd-fs@freebsd.org Wed Sep 5 15:40:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FD84FF433A; Wed, 5 Sep 2018 15:40:20 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15C3B86658; Wed, 5 Sep 2018 15:40:20 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f50.google.com with SMTP id d10-v6so10490158itj.5; Wed, 05 Sep 2018 08:40:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=kmLFZdjiIxRY/wZ5aRNWlLrgbE5CuZc0Mn8ocFx3My4=; b=hnI5G8VCvyRCTj3EUv/cGkTCJZk23zPVFZ3t0yx7Q/s3AzMZNaD/Tv21AiiYzefXiz radSli4iVY1KblhtwnkbNGP/bCiWwVgKtjbLN/lGAeaIPL5E1yE4e7Q8kY1NmBah/gsr +5LKxtzbHAx4uLb5P8Cfp30h5ZIhcCckxMC947P5IRJjEvaqdEkKCUtEu0YigX8bbrIP Wve7UJq0i4CQjnd47PdZlBX0Eqd2LOF4stAfQ1hTLDdw/i+dH5PiMDLWOvTCUD4/3brU w4AVk/PPVaocgnieN/R96saP6MPIOdVAkUSWLVTKqV2i7TrEyNcJsX5RmIcYLXEM0Snf fgtA== X-Gm-Message-State: APzg51BxfT0uKvYhvhysY+RNexbW0sQ/tZah0XeEzOVOhyl1md+6XdFu 7RL1c4v7KJes6g/vBiQUiPV/VIVL X-Google-Smtp-Source: ANB0VdYyEgl0cj3dy83KGbIad7mVKPg57gBJT93Qk4sobJN8KvlNgCf/Fk0+DDy65TkbxeRf6wAhlg== X-Received: by 2002:a02:cd0:: with SMTP id 77-v6mr26904077jan.67.1536162019146; Wed, 05 Sep 2018 08:40:19 -0700 (PDT) Received: from mail-it0-f42.google.com (mail-it0-f42.google.com. [209.85.214.42]) by smtp.gmail.com with ESMTPSA id g198-v6sm7093744itg.4.2018.09.05.08.40.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Sep 2018 08:40:18 -0700 (PDT) Received: by mail-it0-f42.google.com with SMTP id 139-v6so10042996itf.0; Wed, 05 Sep 2018 08:40:18 -0700 (PDT) X-Received: by 2002:a24:db09:: with SMTP id c9-v6mr834254itg.92.1536162018500; Wed, 05 Sep 2018 08:40:18 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:9542:0:0:0:0:0 with HTTP; Wed, 5 Sep 2018 08:40:17 -0700 (PDT) In-Reply-To: <4637985A-28EF-4A6B-B8A6-764A86005E6B@FreeBSD.org> References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> <4637985A-28EF-4A6B-B8A6-764A86005E6B@FreeBSD.org> From: Conrad Meyer Date: Wed, 5 Sep 2018 08:40:17 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: newfs silently fails if random is not ready (?) To: FreeBSD Current Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 15:40:20 -0000 Differential up here: https://reviews.freebsd.org/D17049 for any lurkers I didn't manage to tag in the review. Best, Conrad On Wed, Sep 5, 2018 at 12:57 AM, Mark R V Murray wrote: > Nice catch! Thanks :-) > > M > > >> On 5 Sep 2018, at 04:13, Conrad Meyer wrote: >> >> Hi Lev, >> >> I took a first attempt at reproducing this problem on a fast >> desktop-class system. First steps, give us a way to revert back to >> unseeded status: >> >> --- a/sys/dev/random/fortuna.c >> +++ b/sys/dev/random/fortuna.c >> @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); >> >> #ifdef _KERNEL >> #include >> +#include >> #include >> #include >> #include >> @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) >> return; >> } >> >> + /* >> + * When set, pretend we do not have enough entropy to reseed yet. >> + */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { >> + if (RETURN_VALUE != 0) { >> + RANDOM_RESEED_UNLOCK(); >> + return; >> + } >> + }); >> + >> + >> #ifdef _KERNEL >> fortuna_state.fs_lasttime = now; >> #endif >> @@ -442,5 +454,11 @@ bool >> random_fortuna_seeded(void) >> { >> >> + /* When set, act as if we are not seeded. */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { >> + if (RETURN_VALUE != 0) >> + fortuna_state.fs_counter = UINT128_ZERO; >> + }); >> + >> return (!uint128_is_zero(fortuna_state.fs_counter)); >> } >> >> >> Second step, enable the failpoints and launch repro program: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read='return(1)' >> debug.fail_point.random_fortuna_pre_read: off -> return(1) >> $ sudo sysctl debug.fail_point.random_fortuna_seeded='return(1)' >> debug.fail_point.random_fortuna_seeded: off -> return(1) >> >> $ cat ./blocked_random_poc.c >> #include >> #include >> #include >> >> int >> main(int argc, char **argv) >> { >> printf("%x\n", arc4random()); >> return (0); >> } >> >> >> $ ./blocked_random_poc >> ... >> >> >> Third step, I looked at what that process was doing: >> >> Curiously, it is not in getrandom() at all, but instead the ARND >> sysctl fallback. I probably need to rebuild world (libc) to test this >> (new libc arc4random based on Chacha). >> >> $ procstat -kk 1196 >> PID TID COMM TDNAME KSTACK >> 1196 100435 blocked_random_poc - read_random+0x3d >> sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 >> sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b >> amd64_syscall+0x940 fast_syscall_common+0x101 >> >> >> When I unblocked the failpoints, it completed successfully: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read='off' >> debug.fail_point.random_fortuna_pre_read: return(1) -> off >> $ sudo sysctl debug.fail_point.random_fortuna_seeded=off >> debug.fail_point.random_fortuna_seeded: return(1) -> off >> >> ... >> 9e5eb30f >> >> >> Best, >> Conrad >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> > > -- > Mark R V Murray > From owner-freebsd-fs@freebsd.org Wed Sep 5 16:53:37 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09609FF6C73 for ; Wed, 5 Sep 2018 16:53:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.ne1.yahoo.com (sonic305-21.consmr.mail.ne1.yahoo.com [66.163.185.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9BA888A367 for ; Wed, 5 Sep 2018 16:53:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 3GrAbD0VM1mxm.lvdnOloMUpwgx9S9h8BOcbgHb2QjHwBl8UuqhfkY7EI64IzW_ p1Zij5ocs083IABlW8Ng1eIQfREcm3FrKyKTAI9HuHLkXpFnmsnr2QoReCa3qOH_o7rn.1wqjIvz C0SRF4BY7ffpScGSH_kpzGo82rdRmqFEv7irkXiR1UZ22oViSdPm9iTVhFgbcv3Y46CPjJk2zbZn jqYE8P6RYO6zuv1jAWxkIxQOSlCVS0ZT0IUgVTfuUPXh6DoJgcFWOhLSPgPHRNy0PdG.aOqre7hg rh0lWZrdq.0TXdKgZ9_v1ZALVsTZiXPdXjpV8zn5PnMtUWX52iOOwB9C3vvfIj71NZeELimWguuI EGxsNPzMI6APZX1FI522rWkGVjIYj1ekYk0LhzB3k4deJ02hh_xln065_IxkyUl0e2zEQPG37L0T 98svV1lmZZmCqIjYSX5WrtiAOaIDrkh049MtSvv0ONvkHsPRWX6MYMZrB3uST8fNC4H6RptEoTh9 YgGR.bQu5Ei7S4zjVXM.rs0w7nSKP8RQ3gsNzSMUiQqU7vfwPvlCT5.0aatFfa7UeLry2bvnI2CE _2XcwaVBi.byooWsFl.a4BkRfS6i_V9y0YO8LWrZl.e1K6Fpc7NQxF6IYGZvi4F_Zfb36PETpswe Wbsd7g0SQxf98cizZtojhXvYiDUcgf3cJnG.3jySMhLn9Sz2902aIjWfXe7vjAmfgpZw_olqEcjf 3QQGReWfSiH3Ha0FLVVE4S3kizoHnkdgBfKN5Ji_7vJ1EnJHmVqpo1mWR.t2Jux.OlTMmZgLq87M 4EjDuhJ5Of1WMI1aNwDY0UW_YsaY1uYRmFupNf8z_hBK1KpXaM9DOtSf3pDPq6Lu4HhsT04u8iSB ZINmqJYbHQYPdD9XnNQ5ILNwjnF2TMH6mO8oFui19Dc4y6EEvtriI82JW4H24_XpH4zxnonu_eeo v7rZaq_sDlDAFnezwYIbDkZ2xWAhDOzypzNHq83YWW5Dlg9viOeKFDqhFbskh4JhyoHQI_Coc9kf HyKrT5sDkw6vq7kY9lxa8g1MG7se_OEZktxnx23EHOH7u4417c2WILpYeU70- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.ne1.yahoo.com with HTTP; Wed, 5 Sep 2018 16:53:29 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp423.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 77394b93c1f28596724bb7d745c47a10; Wed, 05 Sep 2018 16:53:28 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: Date: Wed, 5 Sep 2018 09:53:27 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: References: <201809050520.w855KwxS083430@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 16:53:37 -0000 [I got some results but also provide notes about context limitations for what can be tested.] On 2018-Sep-5, at 12:06 AM, Mark Millard wrote: > On 2018-Sep-4, at 10:20 PM, Kirk McKusick = wrote: >=20 >> Thanks for the report. Do you have a sense/measurement that the >> build times better/same/worse than without the consolidation? >=20 >=20 > buildworld buildkernel is far from I/O bound overall (elapsed time) > when built via clang when the llvm materials are being built in this > context: CPU/RAM bound. (I watched gstat -pd and top -CawSores for > evidence of this.) Also, swap/paging I/O would not involve > consolidation. It would be odd for the new consolidation to make > much of a difference for this activity. (gcc 4.2.1 based builds were > more I/O bound in my experience.) >=20 > But installworld is far more I/O bound and does take something > like 11 or 12 minutes in this context. (installkernel is also > more I/O bound but takes far less time.) >=20 > I have a record of an updating installworld for > clang-cortexA53-installworld-poud already with consolidation > turned on . . . >=20 > Start time: 2018-09-04:12:48:14 > End time: 2018-09-04:12:59:54 (so 11 min 40 sec or so elapsed) > typescript log size: 8698790 >=20 > So after the poudriere/pkg activity I could repeat the > "-j4 installworld distrib-dirs distribution DB_FROM_SRC=3D1 > DESTDIR=3D/usr/obj/DESTDIRs/clang-cortexA53-installworld-poud" > with the new consolidation turned off. (That would leave > active the mmc/sd code's consolidation that Ian L. > referenced.) >=20 > I could also try yet again but with trim disabled for the file > system, giving a 3rd contrasting case. >=20 > Would these comparisons help? >=20 >=20 > As for my poudriere-devel use, I previously used PARALLEL_JOBS=3D1 but > did this activity with PARALLEL_JOBS=3D2 (both with = ALLOW_MAKE_JOBS=3Dyes ). > PARALLEL_JOBS=3D2 makes comparing individual port-build times a = problem > because of competing use of the CPUs over the same time period. There > also is the issue of how I/O bound each port's build is or is not > for the context. >=20 > Using "gstat -pd" and "top -CawSores" to watch devel/gcc8 build > indicates I/O %busy is usually < 2%, even < 1%, and much of the time > is 0.0% or 0.1%. (This is during a "prev-gcc" bootstrap stage, no > longer using clang.) It would be odd for the new consolidation to > make much of an overall difference here. A more I/O bound port build? >=20 >=20 > Note: I mount with -o noatime in use. There is the limitation of the Pine64+ 2GB to at most 50 Mhz High Speed mode because of limiting to 3.3V: it would take 1.8V for SDR50 or DDR50 or SDR104 for an sdcard. This may make the measurements not as interesting. If FreeBSD gains e.MMC DDR52 support(via an adapter to sdcard) at 3.3V for the Pine64+ 2GB (operating at 50 MHz, say) I could then test that. (Modern Linux has such support for the Pine64+ 2GB.) The below close figures may be specific to the SANDISK Ultra 128 GB's with the application A1 class. I do not have other sdcard alternatives around to test. This may mix with the above "only HS mode" issue. For: # sysctl vfs.ffs.dotrimcons=3D0 vfs.ffs.dotrimcons: 1 -> 0 reinstalling from the same buildworld I got: Start time: 2018-09-05:08:32:34 End time: 2018-09-05:08:44:40 (so 12 min 6 sec or so elapsed) typescript log size: 8682920 instead of the prior 11 min 40 sec or so elapsed (but that was for an upgrade instead of the same buildworld content). Retrying with: # sysctl vfs.ffs.dotrimcons=3D1 = = vfs.ffs.dotrimcons: 0 = -> 1 reinstalling from the same buildworld again I got: Start time: 2018-09-05:08:54:10 End time: 2018-09-05:09:06:10 (so 12 min 0 sec or so elapsed) typescript log size: 8694405 On this scale of difference re-running multiple times under the same setting could be called for to observe the variability. Let me know if you want such. I do not know if you want trim-disabled figures for this context or not. For reference for a fairly modern Linux and the Pine64+ 2GB . . . The linux is from: https://dl.armbian.com/pine64/Ubuntu_bionic_dev_nightly.7z which is: nightly mainline kernel master branch 4.17.y or 4.18.y It shows as: # uname -ap Linux pine64 4.18.0-rc4-sunxi64 #220 SMP Sun Jul 15 14:16:31 UTC 2018 = aarch64 aarch64 aarch64 GNU/Linux For the sdcard from the SDSQXBG-128G-GN6MA: # cat /sys/kernel/debug/mmc0/ios clock: 50000000 Hz actual clock: 50000000 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 2 (4 bits) timing spec: 2 (sd high-speed) signal voltage: 0 (3.30 V) driver type: 0 (driver type B) # hdparm -t /dev/mmcblk0 /dev/mmcblk0: Timing buffered disk reads: 70 MB in 3.07 seconds =3D 22.80 MB/sec (as proof of performance basically matching HS). So basically matching what FreeBSD does. For reference the e.MMC on an adapter used in the Pine64+ 2GB for this Linux: # cat /sys/kernel/debug/mmc0/ios clock: 52000000 Hz actual clock: 50000000 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 2 (4 bits) timing spec: 8 (mmc DDR52) signal voltage: 0 (3.30 V) driver type: 0 (driver type B) # hdparm -t /dev/mmcblk0 /dev/mmcblk0: Timing buffered disk reads: 134 MB in 3.04 seconds =3D 44.03 MB/sec (as proof of performance basically matching mmc DDR52 being in use). FreeBSD's head does not support this. In fact it fails to boot instead of using some slower mmc mode with 3.3V. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Wed Sep 5 22:02:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2E28FFE668 for ; Wed, 5 Sep 2018 22:02:05 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1511D75B35 for ; Wed, 5 Sep 2018 22:02:04 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w85M7GS2000773; Wed, 5 Sep 2018 15:07:16 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201809052207.w85M7GS2000773@chez.mckusick.com> From: Kirk McKusick To: Mark Millard Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: bob prohaska , FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: Comments: In-reply-to Mark Millard message dated "Wed, 05 Sep 2018 00:06:54 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <771.1536185236.1@chez.mckusick.com> Date: Wed, 05 Sep 2018 15:07:16 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 22:02:05 -0000 > From: Mark Millard > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > Date: Wed, 5 Sep 2018 00:06:54 -0700 > Cc: bob prohaska , > FreeBSD Filesystems > To: Kirk McKusick > > On 2018-Sep-4, at 10:20 PM, Kirk McKusick wrote: > >> Thanks for the report. Do you have a sense/measurement that the >> build times better/same/worse than without the consolidation? > > ... > > But installworld is far more I/O bound and does take something > like 11 or 12 minutes in this context. (installkernel is also > more I/O bound but takes far less time.) > > I have a record of an updating installworld for > clang-cortexA53-installworld-poud already with consolidation > turned on . . . > > Start time: 2018-09-04:12:48:14 > End time: 2018-09-04:12:59:54 (so 11 min 40 sec or so elapsed) > typescript log size: 8698790 > > So after the poudriere/pkg activity I could repeat the > "-j4 installworld distrib-dirs distribution DB_FROM_SRC=1 > DESTDIR=/usr/obj/DESTDIRs/clang-cortexA53-installworld-poud" > with the new consolidation turned off. (That would leave > active the mmc/sd code's consolidation that Ian L. > referenced.) > > I could also try yet again but with trim disabled for the file > system, giving a 3rd contrasting case. > > Would these comparisons help? > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) The above three comparisons on the (relatively) I/O bound installworld would be useful. Even if the TRIM consolidation does not help, it is useful to know that it does not hurt. And running without TRIM would also be a useful data point to measure the cost/savings of doing TRIM. It is a bit tricky to just turn off TRIM and then measure it, because in the immediate aftermath of its having been used, it will leave behind a legacy of easier to use flash areas yet not have the cost of keeping them cleaned up. So, it might initially look like enabling TRIM is a bad idea. Thus you would have to run many installworlds without TRIM enabled to see what the long-term result of not using TRIM turns out to be. Kirk McKusick From owner-freebsd-fs@freebsd.org Wed Sep 5 23:38:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB832FCD304 for ; Wed, 5 Sep 2018 23:38:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-22.consmr.mail.gq1.yahoo.com (sonic312-22.consmr.mail.gq1.yahoo.com [98.137.69.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12FBC78E3F for ; Wed, 5 Sep 2018 23:38:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 9jrPf7AVM1mETGgJIHl7wt7ogPJBrn8PTkTMHVpiBVO77hgwfFISMZogRIxzck6 P81x.U7pqZ2HmOeBWxS6XX2M.K9r38T4qwedx7FZp8.VOOno0ko4nBkzv15X9jvBi_VuBIPRWuLu G30ICRKboTuMc1R.HZjCpV1y1howvMeF3MhMvWUNNIRCO7sxlUSSrtWhE4Fe..BGZlXXNwQgxvyp iMowNRpQ.O8N7q9MZ8omFuId8qX3IblxZyySXrYoLnOW3ljaJ_HETkHBB5BjnX0GbbtdD_d_DWQ6 kzfncjU_zaLq6.b9YXnVb6t1cDdG4n8LSHhGIG2fRZj_30MDXikH10Zv9nVMH4iqbh3v4KuQy5gn WLJOYz52CPjMVx6MslH6mKoHHYy_4y1CYIHpB1pg2D81vBtNt8CDwcDh2_GX9OwgnKuZo66Dlzor 0.B6gZWh5QXq2zxmkbymJAhm647gEV8N_114yl5KkZ3tsjvbYvDsLLDY_vu50R9KxaLomr4UnPI2 iPEvBmmGqHZYFbd9PQqA.KN7t4jTBiHxrBpYqMLwShFSn8z6ByEJYecIC1L6tmNt2AlTux_0AP6J HAoEMZoNmOjMy3VYITzguVM5hnCL7UtY8XCY4SHIsMkgthXCFljHkgcMR_Qt6SREUvpDG2gKhlej rOT.qOfMRP8A34o0diZHV3rxHjpg6uPNxQJwaSeaDTjlwSH2VTKh5M6sPCDoJrrfz.y3ST_CjT4. bX4W8HdYPHqswyUs.b2nKRWiSwfCCXyfyMD_N8vaY3bOO_0WUdmIe.oZn0dvzsEIB2cA6rkwqwru FEd6LayUReVRqjo3lVzInLILbyplPdPD0RVEbdLE7yRBLiVkrXY.iglNZjIK9m5pAyj74yk8ClGg iWmaRbKWkeG6GImb_mG4nE0nTcjOj_Xs5KVhgzHiePjMBCRuBI9IU6aW83varzdXWcVxLcEDhd6y cs_coupI3N_88m22Z36C0q0OkHXjmdQA6FKNkl6NPM3_M9S_22Tt9rm1VF.tckw7ikNkt_12z84i kNGmWla2t Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Wed, 5 Sep 2018 23:38:40 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp421.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID e65da061a3dcc1aa8e91df7133740b56; Wed, 05 Sep 2018 23:38:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <201809052207.w85M7GS2000773@chez.mckusick.com> Date: Wed, 5 Sep 2018 16:38:35 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <932BF7CA-9F5F-4541-8F45-68B68A45DA7E@yahoo.com> References: <201809052207.w85M7GS2000773@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 23:38:48 -0000 On 2018-Sep-5, at 3:07 PM, Kirk McKusick = wrote: >> From: Mark Millard >> Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems >> Date: Wed, 5 Sep 2018 00:06:54 -0700 >> Cc: bob prohaska , >> FreeBSD Filesystems >> To: Kirk McKusick >>=20 >> On 2018-Sep-4, at 10:20 PM, Kirk McKusick = wrote: >>=20 >>> Thanks for the report. Do you have a sense/measurement that the >>> build times better/same/worse than without the consolidation? >>=20 >> ... >>=20 >> But installworld is far more I/O bound and does take something >> like 11 or 12 minutes in this context. (installkernel is also >> more I/O bound but takes far less time.) >>=20 >> I have a record of an updating installworld for >> clang-cortexA53-installworld-poud already with consolidation >> turned on . . . >>=20 >> Start time: 2018-09-04:12:48:14 >> End time: 2018-09-04:12:59:54 (so 11 min 40 sec or so elapsed) >> typescript log size: 8698790 >>=20 >> So after the poudriere/pkg activity I could repeat the >> "-j4 installworld distrib-dirs distribution DB_FROM_SRC=3D1 >> DESTDIR=3D/usr/obj/DESTDIRs/clang-cortexA53-installworld-poud" >> with the new consolidation turned off. (That would leave >> active the mmc/sd code's consolidation that Ian L. >> referenced.) >>=20 >> I could also try yet again but with trim disabled for the file >> system, giving a 3rd contrasting case. >>=20 >> Would these comparisons help? >>=20 >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >> ( dsl-only.net went >> away in early 2018-Mar) >=20 > The above three comparisons on the (relatively) I/O bound installworld > would be useful. Even if the TRIM consolidation does not help, it is > useful to know that it does not hurt. And running without TRIM would > also be a useful data point to measure the cost/savings of doing TRIM. I provided some initial figures for consolidation enabled vs. not in another message. It also noted other limitations of the testing via a Pine64+ 2GB. I've not done anything for "without trim". > It is a bit tricky to just turn off TRIM and then measure it, because > in the immediate aftermath of its having been used, it will leave > behind a legacy of easier to use flash areas yet not have the cost > of keeping them cleaned up. So, it might initially look like enabling > TRIM is a bad idea. Thus you would have to run many installworlds > without TRIM enabled to see what the long-term result of not using > TRIM turns out to be. It does not help that the ufs on /dev/mmcsd0 has over 60 GiBytes free and clang-cortexA53-installworld-poud is more like 3 GiBytes. There is also about 5.5 GiBytes outside any partition that I do not know the detailed status of for how I copied the e.MMC to the microssdxc card. I do have another microsdxc card of the same type that I could put into a USB device (so no TRIM) and dd a copy over to, disable TRIM on the new file system, and then boot from. But more might be required to avoid apparently "clean" blocks on the new microsdxc card. (For example, avoid having areas of all-zeros?) Any suggestions for what you would like to have set up for a TRIM disabled test? Do the Pine64+ 2GB limitation limit the value of the tests? The pine64+ 2GB is now doing a poudriere bulk that targets some x11 related ports --leading to 225 ports being attempted for the bulk run. It is going to be a while before it finishes. Almost certainly not today. (FYI: Consolidation is enabled.) This will use some part of that 60 GiBytes but likely not all that much of it. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Thu Sep 6 00:12:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8C49FCE981 for ; Thu, 6 Sep 2018 00:12:22 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 66CEC7B021 for ; Thu, 6 Sep 2018 00:12:22 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w860CIXQ003236 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 5 Sep 2018 17:12:19 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w860CIYM003235; Wed, 5 Sep 2018 17:12:18 -0700 (PDT) (envelope-from fbsd) Date: Wed, 5 Sep 2018 17:12:17 -0700 From: bob prohaska To: Kirk McKusick Cc: Mark Millard , FreeBSD Filesystems , bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems Message-ID: <20180906001217.GB818@www.zefox.net> References: <201809052207.w85M7GS2000773@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201809052207.w85M7GS2000773@chez.mckusick.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 00:12:23 -0000 On Wed, Sep 05, 2018 at 03:07:16PM -0700, Kirk McKusick wrote: > > It is a bit tricky to just turn off TRIM and then measure it, because > in the immediate aftermath of its having been used, it will leave > behind a legacy of easier to use flash areas yet not have the cost > of keeping them cleaned up. So, it might initially look like enabling > TRIM is a bad idea. Thus you would have to run many installworlds > without TRIM enabled to see what the long-term result of not using > TRIM turns out to be. > Just for fun I ran a (somewhat absurd) -j4 buildworld on RPI3 using 6 GB of swap, three on USB and three on microSD, just to see if anything interesting (bad) happened. The process took about 24 hours, and the oversuppy of swap didn't cause any obvious problems. Next I turned on TRIM and re-ran the buildworld script. There were no obvious problems, but the process took about an extra hour. Since /var, /tmp and /usr were all on USB there was no hope TRIM could be any help on the busy filesystems. TRIM was enabled on microSD, but it had little to do. There does seem to be a modest penalty for using TRIM when it can't help much. Is there any hope of implementing something like TRIM for USB on the Pi? It appears that congestion on USB is a serious bottlneck from time to time just for traffic with /tmp and /usr. Adding swap to the mix makes it worse. Log files are at http://www.zefox.net/~fbsd/rpi3/swaptests/r338342/3gbsd_3gbusb/ in case they're of interest. Thanks for reading! bob prohaska From owner-freebsd-fs@freebsd.org Thu Sep 6 08:11:44 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 506FAFF3180 for ; Thu, 6 Sep 2018 08:11:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-21.consmr.mail.gq1.yahoo.com (sonic301-21.consmr.mail.gq1.yahoo.com [98.137.64.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D032F8C33A for ; Thu, 6 Sep 2018 08:11:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: XRcrgKQVM1mSWPBGlNVW429e7n3UiCsXGzwYeoGMYyWhgUwESPFoe72B5WjZ8KL laLOaL9H0kTvTs4a2ZVsRj5vFORo09j4mQjOWQ_bkF7AFcVXnQ1joV2W127qJXHR4XZqQUC0S9RQ 50nDvf6YaEGFbQlvkYaajb5GmS9yrzx9i2ARPKxnKuge7noNIMfI40PvQR49IhZ3f6xRjO7YtCb1 EbkGNRTteVM_TPW9zw58k3BekvYPK4xBRMpF7.rCNQMfO7yv6w3bNjPXs04DcGgvhPt0N4t3oGmr XE.DBaDixyMWEteiWJ1KhwdJb6zJ1LUpaX9bz248_G1jTwbGJ9yyTnAl9l61ScYjQPuiNoR6RV3j RB5nRMcF4CcJ9jpiE3Fje1aYhCEA8dJvSyUEo9s3rOlWon3X5REgk4N4j8Ht3pCCBw3p.EcvHuaY rZj_BLwhpJO0bd6SUBp8uF08VIpl48t4uTbCG.EEIIxHbKKbzfVr9g8HcadWXlCZ5wX3rQD9r8it cn8yWwuU0sJ4UrAlBH.0MWzm6_7vjYGf.IHgGQ7jshuC9Mc2Dsh0V9VAz55SDi7.vWJY9.2FDxal 1AHdyBxSEWU9wHn4YgX2si1JxCmHY64cRK9RwxGQ.JKWibW0Hm.3j2uc6YJsD6YpNeORIhuWeCjY nDk3Avm43pJ8gBjOWc82aWbHO5q4L9xYP5twcps7km6WUwDcJzmbN99Okj11YeqDcITnA3NwNR2F aO49tMeVMYIoVZZ0kmehKIy.5SfYhBwD7GsNZHPscOzA.S6E24LGcPAPH2gYSsbUB4NZDpUNm_VW hpVFknI97BVwdiC2Mahc0ZrumWm2p5fz38dcvZZNfrgkX9aBU7Ue7DPZksf0YpbW_ew6Ugt.ib82 D6qhgOWHv2bw84v1PFvBI.neq.3T3J5kM6PEl_fPPzGKVLUplUzGv2B7cwprhJzk9CRYj1wp0u7l ozG31aJtyn56rpLdajMTgewNDe3WRPIsvTym6xDQzg1SxCOqH9EovP96zfF6vY4kM8CmutH2e6h6 Khwv47j9_ Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 Sep 2018 08:11:35 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp430.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 26140bc27fd95585bacf261bc215ab57; Thu, 06 Sep 2018 08:11:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <20180906001217.GB818@www.zefox.net> Date: Thu, 6 Sep 2018 01:11:30 -0700 Cc: FreeBSD Filesystems , bob prohaska Content-Transfer-Encoding: quoted-printable Message-Id: <97765759-B81A-44F0-98A0-F7B9D05F7432@yahoo.com> References: <201809052207.w85M7GS2000773@chez.mckusick.com> <20180906001217.GB818@www.zefox.net> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 08:11:44 -0000 [Just correcting a bad claim about the timing of the two buildworld runs that were referenced. It could be misleading to Kirk for his requested information.] On 2018-Sep-5, at 5:12 PM, bob prohaska wrote: > On Wed, Sep 05, 2018 at 03:07:16PM -0700, Kirk McKusick wrote: >>=20 >> It is a bit tricky to just turn off TRIM and then measure it, because >> in the immediate aftermath of its having been used, it will leave >> behind a legacy of easier to use flash areas yet not have the cost >> of keeping them cleaned up. So, it might initially look like enabling >> TRIM is a bad idea. Thus you would have to run many installworlds >> without TRIM enabled to see what the long-term result of not using >> TRIM turns out to be. >>=20 >=20 > Just for fun I ran a (somewhat absurd) -j4 buildworld on RPI3 using 6 = GB of swap, > three on USB and three on microSD, just to see if anything interesting = (bad) happened.=20 > The process took about 24 hours, and the oversuppy of swap didn't = cause any obvious=20 > problems. Next I turned on TRIM and re-ran the buildworld script. =20 >=20 > There were no obvious problems, but the process took about an extra = hour. buildworld did not take an hour longer for one vs. the other based on the timestamps in the log files: trim off: World build started on Sun Sep 2 20:28:12 PDT 2018 . . . World build completed on Mon Sep 3 21:35:47 PDT 2018 So somewhat over 25 hours 7 minutes. trim on: World build started on Tue Sep 4 00:02:36 PDT 2018 . . . World build completed on Wed Sep 5 01:12:47 PDT 2018 So somewhat over 25 hours 10 minutes. I get an under 5 minute difference from those timestamps. > Since /var, /tmp and /usr were all on USB there was no hope TRIM could = be > any help on the busy filesystems. TRIM was enabled on microSD, but it = had > little to do. There does seem to be a modest penalty for using TRIM = when > it can't help much. >=20 > Is there any hope of implementing something like TRIM for USB on the = Pi? > It appears that congestion on USB is a serious bottlneck from time to = time > just for traffic with /tmp and /usr. Adding swap to the mix makes it = worse. >=20 > Log files are at > http://www.zefox.net/~fbsd/rpi3/swaptests/r338342/3gbsd_3gbusb/ > in case they're of interest. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Thu Sep 6 10:17:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A43E4FF5BE6 for ; Thu, 6 Sep 2018 10:17:45 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b01.edpnet.be (relay-b01.edpnet.be [212.71.1.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A3638FB26 for ; Thu, 6 Sep 2018 10:17:44 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1536229053-0a7ff512e0a56e10001-3nHGF7 Received: from mordor.lan (212.71.13.182.adsl.dyn.edpnet.net [212.71.13.182]) by relay-b01.edpnet.be with ESMTP id HEmBMFCiki81BGuC (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 06 Sep 2018 12:17:34 +0200 (CEST) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: 212.71.13.182.adsl.dyn.edpnet.net[212.71.13.182] X-Barracuda-Apparent-Source-IP: 212.71.13.182 Date: Thu, 6 Sep 2018 12:17:33 +0200 From: Julien Cigar To: freebsd-fs@freebsd.org Subject: zpool upgrade with efi partition Message-ID: <20180906101733.GC11886@mordor.lan> X-ASG-Orig-Subj: zpool upgrade with efi partition MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="DSayHWYpDlRfCAAQ" Content-Disposition: inline User-Agent: Mutt/1.9.2 (2017-12-15) X-Barracuda-Connect: 212.71.13.182.adsl.dyn.edpnet.net[212.71.13.182] X-Barracuda-Start-Time: 1536229053 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.221:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-Scan-Msg-Size: 598 X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5045 1.0000 0.7500 X-Barracuda-Spam-Score: 1.25 X-Barracuda-Spam-Status: No, SCORE=1.25 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=7.0 tests=BSF_SC0_MV0713, BSF_SC0_MV0713_2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.56983 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.30 BSF_SC0_MV0713_2 BSF_SC0_MV0713_2 0.20 BSF_SC0_MV0713 Custom rule MV0713 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 10:17:45 -0000 --DSayHWYpDlRfCAAQ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I just run a $> zpool upgrade -a on a 11.2-RELEASE-p1 machine with an=20 UEFI partition scheme (1) and, just to be sure, I don't need any gpart bootcode afterwards, right ? (From what I read this is all handled by=20 the UEFI booting image) Thanks ! (1) https://gist.github.com/silenius/800bb95298eaf28ef38f91e82e2ed7f0 --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --DSayHWYpDlRfCAAQ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE7vn2l0to0nV7EWolsrs3EKIEI8AFAluQ/roACgkQsrs3EKIE I8AGSA//cuBW4/NZUTLreJmGAubcQ6IzNHHSHRAk2oMKB7DNplrbdAuxygOU47Dt bsyIomKOk5XSCyfyNWkJPK9yuk1BAoJJeG34nO1bGsFoP7iwuDQmm7vEOpwF7Ym4 fNGbEHrtOFrO9RZMr8s+cXUVFlmvxebX6dTgmuGBKUnsfrq2ure7Reg4S3vY7M/o dhC/qe5EkGlWt7HPTyM6xJyLAEi7HLLjVLqNKeyUq89E/e3AGU4gd5kzt0M8OUDp mK1njY9oeqM3Gs9O515RVGIJ6ztwl0CLHTvrOSH3+W53+p+uN3MBxchmBD+bFwUn TkM8pDiGI2PqM8IxPTYOKgNdfo/OjoX2+PJwrU5aLi+jZRTNh0wK37OY21AG6F3U zCq5WAVyZOWyCKaMQtiNaq0W3R3ggmlvCnP9G1ci0ZNRsBg4bKP2oxYNvtczx1FL NwTsCf6diA+exds9olPCjBHFtb/iXXAJGRV+5kDJppqrhQtsi5lMIW5spZZxhOTQ 9ed+6mZWxkZF1K468u5w46A7Z1mS4FYxE51ec0ilp0HUAeRMHh8iJ+ctxlL0ooaZ IbscIZkT9JW88tUua1dRyWZSSm0smZfO/94usVOck5dh63oCh6Xod/uDVZm1RIOy DjqQV4ZtNMX5hK0757LWp10s4OuOq36QW2l+xwOfJvinFyFkQNQ= =JnFi -----END PGP SIGNATURE----- --DSayHWYpDlRfCAAQ-- From owner-freebsd-fs@freebsd.org Thu Sep 6 18:26:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 805E4FCF6AD for ; Thu, 6 Sep 2018 18:26:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0741F840D1 for ; Thu, 6 Sep 2018 18:26:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id BD3C1FCF6AC; Thu, 6 Sep 2018 18:26:18 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B563FCF6AB for ; Thu, 6 Sep 2018 18:26:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21506840CA for ; Thu, 6 Sep 2018 18:26:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5508B8B0 for ; Thu, 6 Sep 2018 18:26:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w86IQHWN023246 for ; Thu, 6 Sep 2018 18:26:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w86IQHGt023245 for fs@FreeBSD.org; Thu, 6 Sep 2018 18:26:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 227784] zfs: Fatal trap 9: general protection fault while in kernel mode on shutdown Date: Thu, 06 Sep 2018 18:26:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: inextricable.nadir@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 18:26:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227784 Conor changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |inextricable.nadir@gmail.co | |m --- Comment #11 from Conor --- I've also seen this issue on FreeBSD 11.2-RELEASE-p2, not CURRENT in somewh= at reproducible fashion; the panic seems to happen under heavy ZFS load reading and writing via NFS _eventually_ without fail. The manner which I trigger it each time is running a bittorrent client which is writing to the NFS share = from another machine. The stack trace is always the same. Problem Machine: FreeBSD nas 11.2-RELEASE-p2 FreeBSD 11.2-RELEASE-p2 #0: Tue Aug 14 21:45:40= UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC a= md64 Entry in /var/log/messages on reboot: Sep 4 23:08:35 nas kernel: Fatal trap 9: general protection fault while in kernel mode Sep 4 23:08:35 nas kernel: cpuid =3D 0; apic id =3D 00 Sep 4 23:08:35 nas kernel: instruction pointer =3D 0x20:0xffffffff80f7572e Sep 4 23:08:35 nas kernel: stack pointer =3D 0x28:0xfffffe023273a9b0 Sep 4 23:08:35 nas kernel: frame pointer =3D 0x28:0xfffffe023273a9b0 Sep 4 23:08:35 nas kernel: code segment =3D base 0x0, limit 0xfffff, type 0x 1b Sep 4 23:08:35 nas kernel: =3D DPL 0, pres 1, long 1, def32 0, gran 1 Sep 4 23:08:35 nas kernel: processor eflags =3D interrupt enabled, resu= me, IOPL =3D 0 Sep 4 23:08:35 nas kernel: current process =3D 0 (zio_read_int= r_0_4) Sep 4 23:08:35 nas kernel: trap number =3D 9 Sep 4 23:08:35 nas kernel: panic: general protection fault Sep 4 23:08:35 nas kernel: cpuid =3D 0 Sep 4 23:08:35 nas kernel: KDB: stack backtrace: Sep 4 23:08:35 nas kernel: #0 0xffffffff80b3d567 at kdb_backtrace+0x67 Sep 4 23:08:35 nas kernel: #1 0xffffffff80af6b07 at vpanic+0x177 Sep 4 23:08:35 nas kernel: #2 0xffffffff80af6983 at panic+0x43 Sep 4 23:08:35 nas kernel: #3 0xffffffff80f77fcf at trap_fatal+0x35f Sep 4 23:08:35 nas kernel: #4 0xffffffff80f7758e at trap+0x5e Sep 4 23:08:35 nas kernel: #5 0xffffffff80f57dac at calltrap+0x8 Sep 4 23:08:35 nas kernel: #6 0xffffffff8228a6b6 at abd_copy_off+0x156 Sep 4 23:08:35 nas kernel: #7 0xffffffff82307ffa at vdev_queue_agg_io_done+0x6a Sep 4 23:08:35 nas kernel: #8 0xffffffff8232ecfe at zio_done+0x90e Sep 4 23:08:35 nas kernel: #9 0xffffffff8232a74c at zio_execute+0xac Sep 4 23:08:35 nas kernel: #10 0xffffffff80b4ed74 at taskqueue_run_locked+0x154 Sep 4 23:08:35 nas kernel: #11 0xffffffff80b4fed8 at taskqueue_thread_loop+0x98 Sep 4 23:08:35 nas kernel: #12 0xffffffff80aba083 at fork_exit+0x83 zpool status: root@nas:/var/log # zpool status pool: zroot state: ONLINE scan: scrub repaired 24K in 0h7m with 0 errors on Tue Sep 4 23:21:17 2018 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 ada2p3 ONLINE 0 0 0 ada3p3 ONLINE 0 0 0 ada4p3 ONLINE 0 0 0 ada5p3 ONLINE 0 0 0 ada6p3 ONLINE 0 0 0 ada7p3 ONLINE 0 0 0 errors: No known data errors I can collect a crash dump if it will be useful. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Sep 6 18:27:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAF09FCF795 for ; Thu, 6 Sep 2018 18:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 589F884192 for ; Thu, 6 Sep 2018 18:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1D735FCF794; Thu, 6 Sep 2018 18:27:49 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BF98FCF793 for ; Thu, 6 Sep 2018 18:27:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EF9E8418E for ; Thu, 6 Sep 2018 18:27:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id AEFD38BA for ; Thu, 6 Sep 2018 18:27:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w86IRlCj024938 for ; Thu, 6 Sep 2018 18:27:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w86IRlPl024935 for fs@FreeBSD.org; Thu, 6 Sep 2018 18:27:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 227784] zfs: Fatal trap 9: general protection fault while in kernel mode on shutdown Date: Thu, 06 Sep 2018 18:27:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic, regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 18:27:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227784 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |regression CC| |markj@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Sep 6 23:13:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97301FE8B49 for ; Thu, 6 Sep 2018 23:13:45 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1857170C13 for ; Thu, 6 Sep 2018 23:13:44 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w86NJ13q027079; Thu, 6 Sep 2018 16:19:01 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201809062319.w86NJ13q027079@chez.mckusick.com> From: Kirk McKusick To: bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: Mark Millard , FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <20180906001217.GB818@www.zefox.net> Comments: In-reply-to bob prohaska message dated "Wed, 05 Sep 2018 17:12:17 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <27077.1536275941.1@chez.mckusick.com> Date: Thu, 06 Sep 2018 16:19:01 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 23:13:45 -0000 > Date: Wed, 5 Sep 2018 17:12:17 -0700 > From: bob prohaska > To: Kirk McKusick > Cc: Mark Millard , > FreeBSD Filesystems , > bob prohaska > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > > Just for fun I ran a (somewhat absurd) -j4 buildworld on RPI3 using > 6 GB of swap, three on USB and three on microSD, just to see if > anything interesting (bad) happened. The process took about 24 > hours, and the oversuppy of swap didn't cause any obvious problems. > Next I turned on TRIM and re-ran the buildworld script. > > There were no obvious problems, but the process took about an extra > hour. Since /var, /tmp and /usr were all on USB there was no hope > TRIM could be any help on the busy filesystems. TRIM was enabled > on microSD, but it had little to do. There does seem to be a modest > penalty for using TRIM when it can't help much. > > Is there any hope of implementing something like TRIM for USB on > the Pi? It appears that congestion on USB is a serious bottlneck > from time to time just for traffic with /tmp and /usr. Adding swap > to the mix makes it worse. > > Log files are at > http://www.zefox.net/~fbsd/rpi3/swaptests/r338342/3gbsd_3gbusb/ > in case they're of interest. > > Thanks for reading! > > bob prohaska Thanks for the additional information. My guess is that there will not be any move to pass TRIM through to USB devices as there is no standard way to do it, and in cases where it has been sort of hacked in, there has been no perceived benefit and in some cases severe performance penalties. Kirk McKusick From owner-freebsd-fs@freebsd.org Thu Sep 6 23:25:34 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A382BFE8F3E for ; Thu, 6 Sep 2018 23:25:34 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1CE6A71025 for ; Thu, 6 Sep 2018 23:25:33 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w86NUphJ027262; Thu, 6 Sep 2018 16:30:51 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201809062330.w86NUphJ027262@chez.mckusick.com> From: Kirk McKusick To: Mark Millard Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: bob prohaska , FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <932BF7CA-9F5F-4541-8F45-68B68A45DA7E@yahoo.com> Comments: In-reply-to Mark Millard message dated "Wed, 05 Sep 2018 16:38:35 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <27260.1536276651.1@chez.mckusick.com> Date: Thu, 06 Sep 2018 16:30:51 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 23:25:34 -0000 > To: Kirk McKusick > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > Date: Wed, 5 Sep 2018 16:38:35 -0700 > Cc: bob prohaska , > FreeBSD Filesystems > > On 2018-Sep-5, at 3:07 PM, Kirk McKusick wrote: > >> The above three comparisons on the (relatively) I/O bound installworld >> would be useful. Even if the TRIM consolidation does not help, it is >> useful to know that it does not hurt. And running without TRIM would >> also be a useful data point to measure the cost/savings of doing TRIM. > > I provided some initial figures for consolidation enabled vs. not in > another message. It also noted other limitations of the testing via > a Pine64+ 2GB. > > I've not done anything for "without trim". > >> It is a bit tricky to just turn off TRIM and then measure it, because >> in the immediate aftermath of its having been used, it will leave >> behind a legacy of easier to use flash areas yet not have the cost >> of keeping them cleaned up. So, it might initially look like enabling >> TRIM is a bad idea. Thus you would have to run many installworlds >> without TRIM enabled to see what the long-term result of not using >> TRIM turns out to be. > > It does not help that the ufs on /dev/mmcsd0 has over 60 GiBytes free > and clang-cortexA53-installworld-poud is more like 3 GiBytes. There is > also about 5.5 GiBytes outside any partition that I do not know > the detailed status of for how I copied the e.MMC to the microssdxc > card. > > I do have another microsdxc card of the same type that I could > put into a USB device (so no TRIM) and dd a copy over to, disable > TRIM on the new file system, and then boot from. But more might be > required to avoid apparently "clean" blocks on the new microsdxc > card. (For example, avoid having areas of all-zeros?) > > Any suggestions for what you would like to have set up for a > TRIM disabled test? Do the Pine64+ 2GB limitation limit the > value of the tests? > > > The pine64+ 2GB is now doing a poudriere bulk that targets some > x11 related ports --leading to 225 ports being attempted for > the bulk run. It is going to be a while before it finishes. > Almost certainly not today. (FYI: Consolidation is enabled.) > This will use some part of that 60 GiBytes but likely not > all that much of it. > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) I concur that there is no experiment "without TRIM" that is likely to produce any meaningful result. Your tests with TRIM have been most helpful in helping to show that the new TRIM code is not hazardous to the stability of the kernel. Kirk McKusick From owner-freebsd-fs@freebsd.org Thu Sep 6 23:44:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F182FE9519 for ; Thu, 6 Sep 2018 23:44:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A846171888 for ; Thu, 6 Sep 2018 23:44:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: mmpgdHMVM1ldF_A87tosTWnc0MnNvfgXBi8qA1up2h3wDSQ1UwrrRjLLRHNgR9M xyUBAg90lgHkSnlONLCVyNsaYLqMZu0.n5RNTwv5n6lB9r4cDl9LU4CHL758Gop4Qbw2aKpz2Omo RXQOWG0S53fcRWZ_NnbhdcB6JLC4gDis0gEL.YbXfMvB0hDZZnrazIkHLlJKyLEbftb2aonotuKr CIUsJWl36lCHyal0EeXit44EibY2qxbutOAd1i03eCDoec.upttz5TjHK3rXkp5wciHilSDYeVgQ 8tPjF91ykyD_hucRMy_spSbcpD3rHV2tMhUgl3TtoWOlhjBSXphpCPI.pTC0OakIWdBQ3u2Hbqi5 vaVwa4wOtLLRUPPJJf4TtmztDncU0YPtQctW__eGlokhZZvV0aERokWNz4k4JTHFHLu5r0LHFq.d lkvyImI.vKaPsxuFDMdpvt6yi.xvSAU3dKBCk1HAEZLYEv87dYQr852alYj95j0P5.q0SCBVcEpE J7t5CFvzx3d5vmdQkguSbIRWyA_wORtqPSTrsPaM4EJSKhgZyYAWt96KxghXoCWlN1M3Pk9vCRDL 5qJLFqBaBFMlcMBBMS4vy924BCeuLQr7ZzsqP.SDbhZs4WDAjcmfBGWjFDeVeau5NfTvUnaj9VdX wjSvoecRzx8NCkc5dWP0qJ31PHNcNJJ5npqXhk.E9aoRI6bf.v7NcvZg6XI7kL1QiGcHehPiD41S EuhSDN6EDAjJTTsSlF_NKOiAuRdOgqyZVEFDhrvCvIBKKqLWjP7Mj91CHnOI.9m22nWHkOHC9reM DGpDK5RP7QT6OI2TmsjmsZEbP8dgPrMaBe6dkrJ3ComS4YGjCkJSRm0n1V4Na.4fo_ylppQaj0iN hV2wf0IFF8NT0AC857PpZfWouoE66UKt4DBNRDZYlscUwQDYAgR4Rk7OacIUO_b3nybkFCYgjaOu 65cpDSyXOAGOZ.hwU0TY69Se8vP0xQ_KLQ.ImuWaqf4u8i34qdPAuLXdh7Lju4O7jIOPoRMI9gKl .bc8fDH2w.1A- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 Sep 2018 23:44:51 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp424.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 529dcc53c196658130e128db86fa8579; Thu, 06 Sep 2018 23:34:43 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <201809062330.w86NUphJ027262@chez.mckusick.com> Date: Thu, 6 Sep 2018 16:34:42 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: 7bit Message-Id: References: <201809062330.w86NUphJ027262@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 23:44:53 -0000 On 2018-Sep-6, at 4:30 PM, Kirk McKusick wrote: >> To: Kirk McKusick >> Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems >> Date: Wed, 5 Sep 2018 16:38:35 -0700 >> Cc: bob prohaska , >> FreeBSD Filesystems >> >> On 2018-Sep-5, at 3:07 PM, Kirk McKusick wrote: >> >>> The above three comparisons on the (relatively) I/O bound installworld >>> would be useful. Even if the TRIM consolidation does not help, it is >>> useful to know that it does not hurt. And running without TRIM would >>> also be a useful data point to measure the cost/savings of doing TRIM. >> >> I provided some initial figures for consolidation enabled vs. not in >> another message. It also noted other limitations of the testing via >> a Pine64+ 2GB. >> >> I've not done anything for "without trim". >> >>> It is a bit tricky to just turn off TRIM and then measure it, because >>> in the immediate aftermath of its having been used, it will leave >>> behind a legacy of easier to use flash areas yet not have the cost >>> of keeping them cleaned up. So, it might initially look like enabling >>> TRIM is a bad idea. Thus you would have to run many installworlds >>> without TRIM enabled to see what the long-term result of not using >>> TRIM turns out to be. >> >> It does not help that the ufs on /dev/mmcsd0 has over 60 GiBytes free >> and clang-cortexA53-installworld-poud is more like 3 GiBytes. There is >> also about 5.5 GiBytes outside any partition that I do not know >> the detailed status of for how I copied the e.MMC to the microssdxc >> card. >> >> I do have another microsdxc card of the same type that I could >> put into a USB device (so no TRIM) and dd a copy over to, disable >> TRIM on the new file system, and then boot from. But more might be >> required to avoid apparently "clean" blocks on the new microsdxc >> card. (For example, avoid having areas of all-zeros?) >> >> Any suggestions for what you would like to have set up for a >> TRIM disabled test? Do the Pine64+ 2GB limitation limit the >> value of the tests? >> >> >> The pine64+ 2GB is now doing a poudriere bulk that targets some >> x11 related ports --leading to 225 ports being attempted for >> the bulk run. It is going to be a while before it finishes. >> Almost certainly not today. (FYI: Consolidation is enabled.) >> This will use some part of that 60 GiBytes but likely not >> all that much of it. >> >> . . . A little while ago the x11 related builds completed and I installed 4 such ports explicitly (and 218 implicitly). No evidence of file system problems or other such. (X11 operation is untested.) > I concur that there is no experiment "without TRIM" that is likely to > produce any meaningful result. Okay. > Your tests with TRIM have been most helpful in helping to show that the > new TRIM code is not hazardous to the stability of the kernel. > Glad to help. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Fri Sep 7 04:22:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BB8EFEFB80 for ; Fri, 7 Sep 2018 04:22:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0834279D5E for ; Fri, 7 Sep 2018 04:22:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C1E4BFEFB7E; Fri, 7 Sep 2018 04:22:19 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0904FEFB7C for ; Fri, 7 Sep 2018 04:22:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4093E79D55 for ; Fri, 7 Sep 2018 04:22:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8F8DD6029 for ; Fri, 7 Sep 2018 04:22:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w874MId7082322 for ; Fri, 7 Sep 2018 04:22:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w874MInF082321 for fs@FreeBSD.org; Fri, 7 Sep 2018 04:22:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231211] [zfs] possible deadlock triggered by zfs test suite Date: Fri, 07 Sep 2018 04:22:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 04:22:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231211 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Sep 7 04:22:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11C15FEFBB6 for ; Fri, 7 Sep 2018 04:22:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A28ED79DD1 for ; Fri, 7 Sep 2018 04:22:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 68083FEFBB3; Fri, 7 Sep 2018 04:22:47 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56AECFEFBB2 for ; Fri, 7 Sep 2018 04:22:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC30F79DC9 for ; Fri, 7 Sep 2018 04:22:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 2E45B6039 for ; Fri, 7 Sep 2018 04:22:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w874MkXS083086 for ; Fri, 7 Sep 2018 04:22:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w874Mkqb083085 for fs@FreeBSD.org; Fri, 7 Sep 2018 04:22:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231212] `zfs destroy` a jail running buildworld causes "ranlib: fatal: Failed to open 'libXXX.a'" Date: Fri, 07 Sep 2018 04:22:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 04:22:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231212 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Sep 7 08:55:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 404A0FF4F9D for ; Fri, 7 Sep 2018 08:55:33 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (mail1.yamagi.org [IPv6:2001:19f0:5001:17bd::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5EF98104A for ; Fri, 7 Sep 2018 08:55:32 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [212.48.125.108] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1fyCY1-0009cJ-C6 for freebsd-fs@freebsd.org; Fri, 07 Sep 2018 10:55:30 +0200 Date: Fri, 7 Sep 2018 10:55:14 +0200 From: Yamagi Burmeister To: freebsd-fs@freebsd.org Subject: Re: ZFS (ARC) performance regression in r321610 Message-Id: <20180907105514.b2601aeaeb8e4f1898322525@yamagi.org> In-Reply-To: <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA256"; boundary="Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 08:55:33 -0000 --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I just want a give a short conclusion for this problem. On Mon, 27 Aug 2018 17:19:10 +0300 Lev Serebryakov wrote: > On 27.08.2018 16:47, Yamagi Burmeister wrote: >=20 > > With this program I was able to bisect the source and identify commit > > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culprit: > > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repeat > tests to be sure, that this code is a problem. After about 2 weeks with "zfs_abd_scatter_enabled" set to 0 (in form of r338365 merged back to 11.2-RELEASE and vfs.zfs.abd_scatter_enabled set to 0) I can confirm that switching of scattered allocations helps, my performance problems are gone. The system is running fine and zfs-stats looks good. To be clear: Switching off scattered allocations has a very positive effect in this and only this workload. On our other ~100 hosts - some of them MySQL installations with other workload patterns - the scattered allocations are a benefit or at least don't hurt. Regards, Yamagi --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluSPPMACgkQ6xRy5x1Q JRUhFA/7Bx8s5WBs9Kui1PEvUqoNNjZIY2M8i6K3wuvSgMoIQRt8RIS6f9RjTVAv Gw0XikJZ+mQV3aXNuvev309IJTOK4lp76QqQWxsgO1165wy9qsXF9V0AXEO9YCmq Am0rgF+GwHwqeqZt7M2jGRer3NGZZlax+bTsjmWkY20k2RPFR1KNU06NB/TEkHh4 KiwFWX4Dl/wTs7ev1oQuBqlDTtCX7XywlPRvxjbDG+UFyz/EQxPexbthm3WtBOiX tNXSaFyMgPFflUbWjm+KxEtlOtpKv7k1ukUHdU7Tvf5VKOMMO2iMVMgz5VTc9FAp v9+/WBo8/BcAFfcLS9IU0pHgidpoj1yzmFY2OAuDcMUjvjxJmAsO9wHztVjHmisW dcTENsOX6MCrmHoXi5vP7pvMoKqkjIe8l1vO8feONFy3Tz9JfYiTToR1yziGOynQ minPTjwGdRAhvmrEDyAwVjOyvX5lxPVHHWKvQPiqXpqeGG5Xj5BG7APi3uQHhsKq pjGHSlnfkMjQFQb0J4UDVS/V9UH7xYE40qwg42Xb5gCnsPNxxaFuQOaAgrgPeHS9 jjGHgXxa3fYUte/FxgUV8ba7Vk9AVBYWRsa2m+zXAJr9sizf8pGunPnMalN9aNTx cvGY31H7Ro6MUEbUEa2yobC0mTsnbKAytXb/U9yz+KM9ahcjhds= =p3+5 -----END PGP SIGNATURE----- --Signature=_Fri__7_Sep_2018_10_55_14_+0200_MH/WoywG1byHdYqT-- From owner-freebsd-fs@freebsd.org Fri Sep 7 08:59:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 963FBFF5072 for ; Fri, 7 Sep 2018 08:59:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 33537811AB for ; Fri, 7 Sep 2018 08:59:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E48A5FF5071; Fri, 7 Sep 2018 08:59:49 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3127FF5070 for ; Fri, 7 Sep 2018 08:59:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5EAA6811A3 for ; Fri, 7 Sep 2018 08:59:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 89F80107B8 for ; Fri, 7 Sep 2018 08:59:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w878xmR4006207 for ; Fri, 7 Sep 2018 08:59:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w878xmaA006206 for fs@FreeBSD.org; Fri, 7 Sep 2018 08:59:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Fri, 07 Sep 2018 08:59:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 08:59:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 --- Comment #10 from Harald Schmalzbauer --- (In reply to Harald Schmalzbauer from comment #9) Just a short note: Made some progress with the test setups, but hardware was in bad condition; replacement parts expected to arrive on monday, so I'll hopefully be able to run tests with 11-current (pre- r299448 was mentioned, will check) the week= end after this, but then I should be able to cover various kinds of interoperability tests and simply run things :-) [offtopic: and re-use the setup to track down the Server2016 WindowsServerBackup failure with Samba 4= .x ]. -harry --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Sep 8 08:36:41 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BB97FECE6C for ; Sat, 8 Sep 2018 08:36:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-22.consmr.mail.gq1.yahoo.com (sonic317-22.consmr.mail.gq1.yahoo.com [98.137.66.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D144E8BE1B for ; Sat, 8 Sep 2018 08:36:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 5a72AzQVM1mY1EHh47knjCkETWPxu4vojb.IdwOblaYEBoqRvE24OZW9d3OhUnK q5yw0f4iKGvFPhW49zOe_7OCsxv24xZl0jLUEE3F1P9irfveYxgDGSQPWVuS_LSSi33F1PmvYNrz UoNvmAJCGYNN90u4WEz4o8KQlHD.AtGKLNkt0Qv3CUYOgvBJDX3YQT_o_alSuqQ9gABKrK2GXAWJ ydFoci707ktKjZPEM3PXAGp0kd85Eqait7LTi5E.TJnXUy1yAyQw4EMR7v6Tf_giy0WOdb3FEqO. TiizWx7b.p7pp89c5uIGlKW0HglHCjpDGtlfHdl3uI4H_PaBf8y3tE1HLKdT1RFHcCmpRaMrwAch c.Neuosa7p3fPmEdvX3re0mnP0uFGYLcYQ84xy75BusEpo5weJI4Tb7PC6d9HsIVMp5vptbGz5hY oCuDtfi1YF5.AtW9LIGgmYUFsfYRCW88TckSPYxoZwddXKEiyxa14oGxBPuhGmJPz1zPqAuFv_qZ 8CzJD8jXVAl65ojCtak9UfuDcrrdgKCLa_qasDF2DNK3enL5.fi3JHNtmi40n6e0pwll0pD_yNtW 2M9adOR25doP__OYhwEi9aJzrvKbFZNgWxEMJzxUsGVhXXUMkRazz0NxR_xqZE0IVUCVV6v7b42. QzGycvZzNANVEdjhBe8lascMEvpr8Jyf7Igne0nCr2bcRC.as291Zgzw94Px3yWF7Vqacm7IObcd 31rCM47RT.mgpG4Aim0394jFP7evz_bZZ1V0qfXC2lphsa3pMPaAIwoM1IZumhoVwiH6KIPlVsuK QGJv2_rRKy6uRwDIn6YxBo9J1E5pYNBnZAWsC2WNwKkiIoygEYJeq2Ew7_OVvCDaJwx09YLFxwmT LCFI39DLrhFapa8uonLMYuYIY7_xT04SatpcjbPwhFKEO9ZZLhQ44H4UR5U1eDlkyPjBVjgDe0zA A2t3rHNQIH_Vi.R6G4ptI1saU2EanCTXmtzvVExEo1fvlFgrSrktNcAKf1GIQJSbOkKDmgQ.hoyu b.mM23kQs Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Sat, 8 Sep 2018 08:36:32 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp432.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 4d42ee13eaaca40bf2a446cbb7ff4597; Sat, 08 Sep 2018 08:36:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: Date: Sat, 8 Sep 2018 01:36:29 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: 7bit Message-Id: References: <201809062330.w86NUphJ027262@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Sep 2018 08:36:41 -0000 Just an FYI. I figured out a hack that allows the e.MMC on a microsd card adapter to be used to boot and operate the Pine64+ 2GB, in DDR52 mode at that. (And so shows what was missing in the FreeBSD operation even if the code change is not the proper form of an official fix.) But in the process I discovered that FreeBSD is using (for e.MMC in this tested context): mmc0: REQUEST: CMD38 arg 0x1 flags 0x1d That "0x1" means: TRIM that forces reads of zeros: the true Data Removal command for mmc protocol, a form of erase. 0x3 would be DISCARD, the "performance command" that does not guarantee what would be read afterwards: no erase required. TRIM is older (added in e/MCC 4.4). DISCARD is newer (added in e.MMC 4.5). Each is mandatory when the version has the function at all. Does FreeBSD have a policy of preferring erasure when there is also the option to not require it? === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Sat Sep 8 09:32:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72614FEE4F4 for ; Sat, 8 Sep 2018 09:32:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-20.consmr.mail.gq1.yahoo.com (sonic309-20.consmr.mail.gq1.yahoo.com [98.137.65.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E8F388D96F for ; Sat, 8 Sep 2018 09:32:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: vdV_uYsVM1no.QFATa2_NoSRJq2y3iEu3Gg.Bjce94SOfG7bDSfuaMirnegNEw0 nWa5rdR3oK21GxqKEJ7dpLWIUq2dq3FQRMs0XPEpXq5vShETl2Lmv_PewNo7gp0OdcDXvJ3E1suC ebqkiZMbmio5HYIk6c0j6250ubB42iECx3476fJkAra.aIbI.vr9fKh727FABqfsuqpMSBhmWqW_ tUjtdbi2Xerd7QIU5r3Ma5XNeFD.TKhKDmrsO9T6Q_MjnzD.d6YgMwEonSpILhhnN1kjaKXKe3AR A98khwSIB2ry0l2_AP_aBNoaw.iQwoe7sjS9ig42n6Pl9hu88ffXeGYb.rPRKLU7uyU.hh4jVkqi NHUqtjbKLh72EesCeUbx4ZnADgInAbme_5ts0BdchktDU41pVhfR1WWnQu2wgYGcHuvU6_O8wto9 SyD.0aHzCpmJyJv8_WQQG6wLHEozpYVUG2pjz6haHD6InpadP7Ot1VgWTlJfD.9mF.CySVPCV7c. Vfta5VYNSNcKYDCtCHSsXx5hNZmPzr8nIQTZK.AoeYgWKag8gVU6cnb4kco6aR1.7eRTkc.1.yQb gFzIoVfwnF3esRlMrl2rYP9STNmKtPZ6UdWlz7JtBnzIxBgSf63RknuPHV0OCTbjHrA7.dKgA.rj IXaLXjmpmZAKIKW9NZ03C9O.em_SJM.QkrFlnPt26_lOXmzHtdw1eYpBQv6pvxoitFCgTbjxtASi .Ilwg89eq6xEC.V5Q5EfAQGhDUFGtxJ.T7Ynzb8qy3qy1R4nzI2Vg6NUpQGk_QVEEH3Mri_NuxC0 3H9tf97rRbCbtQMjG70n1oGAQXXNiBCyP56ZlNi3lorckdmQNGqgWnS9Qx2q2KMFZSIOummfGFrc pPWqORpxNcSAG79xv2N09hJbhrkkYGq1JmyfgeiB1JR7VEGZtMxlsdgxEmNJNj.6r3LfYS2TvjLX uIK4a7aQ75l70KllAw2hf5V3MPsGbE1ZchjPcpUYUvnWGIVk0jUvIZAXUpWkyjE2r.y0wqqsmE1j zu4V717aS Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sat, 8 Sep 2018 09:32:39 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp401.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID f010c9f6994ccd3735c83186d7b3f674; Sat, 08 Sep 2018 09:32:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: Date: Sat, 8 Sep 2018 02:32:35 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: 7bit Message-Id: References: <201809062330.w86NUphJ027262@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Sep 2018 09:32:46 -0000 On 2018-Sep-8, at 1:36 AM, Mark Millard wrote: > Just an FYI. > > I figured out a hack that allows the e.MMC on a microsd card > adapter to be used to boot and operate the Pine64+ 2GB, in > DDR52 mode at that. (And so shows what was missing in the > FreeBSD operation even if the code change is not the > proper form of an official fix.) > > But in the process I discovered that FreeBSD is using > (for e.MMC in this tested context): > > mmc0: REQUEST: CMD38 arg 0x1 flags 0x1d > > That "0x1" means: TRIM that forces reads of zeros: the true > Data Removal command for mmc protocol, a form of erase. 0x3 > would be DISCARD, the "performance command" that does not > guarantee what would be read afterwards: no erase required. > > TRIM is older (added in e/MCC 4.4). DISCARD is newer (added > in e.MMC 4.5). Each is mandatory when the version has the > function at all. > > Does FreeBSD have a policy of preferring erasure when > there is also the option to not require it? I looked up published SD card material and that command set encodes differently for the argument: 0x1 is DISCARD (also not requiring an erase) 0x2 is FULE when start LBA=LBA0 and end LBA=MaxLBA. Otherwise it is ERASE. (FreeBSD's mmcsd_delete uses 0x0 for ERASE selection.) (I ignore handling violations of start LBA <= end LBA <= Max LBA or out of order command sequences. FULE: Full User Area Logical Erase.) It looks like FreeBSD uses mmcsd_delete for both SD and e.MMC and it is coded for SD to be more optimal (DISCARD), not differentiating e.MMC from SD for argument values to use. In e.MMC 4.5+ 0x2 is a "discard enable" bit in the argument, and never a FULE or other such. In e.MMC the 0x0 case for CMD38's argument is for performing an erase on erase group(s) instead of sector(s). (I'm not describing Secure Request or Force Garbage Collect being 0x1 for e.MMC: apparently unused by mmcsd_delete .) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)