From owner-freebsd-stable@freebsd.org Tue Feb 23 03:25:45 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 54BC855AEAB for ; Tue, 23 Feb 2021 03:25:45 +0000 (UTC) (envelope-from chris.anderson@gmail.com) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dl4FN1mmpz3pWs; Tue, 23 Feb 2021 03:25:44 +0000 (UTC) (envelope-from chris.anderson@gmail.com) Received: by mail-wr1-f48.google.com with SMTP id l30so5436788wrb.12; Mon, 22 Feb 2021 19:25:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ReGpGrabUPBotjjj1qjfMgD4EW8riAFQafKPnibGXWI=; b=HNjUmo+onPc3MOyy48xaVXTj6Wam5Z61lYpkHthEnvVLFJnQNPwgna0+eHW1to+ogF 2l7aEM8vN2orQWla32DBGo2BTDiObaUkxSS8spbCSuxelE3zUdXJcmLuBrNvEmPOTMPL rFw0dR4W/zGepgQGFTt3o7c/0VnQgZHH34JbA+ePlmiQ3s6lDKef5YWgobd5Lsy9f025 HH2kQ4IJHJXBMjifQ7/xS8wT0KI+Ib/whcWl9bDuUIyC/3A3IqqFKmsaDYYSybyj/7lw 3suq3pH3dgKVKYPsDfE7toq2Gf65PJ5Fx7PKPjctkGt02pfcRB7lMWkNb/q+6I41GlmB 4ogA== X-Gm-Message-State: AOAM531S7NwTXC/esD0TfIgjzMeKKsa/senEB3QOxzUvRYpDg93Ttn0c jr6RpvQl+sHxrqiA1N00YydTwuG1jmbODNta5lr5v+V6AB8= X-Google-Smtp-Source: ABdhPJz+yc1MABKFBpZzhFti7etIbfI67EI616iVm8oQ48B3tIT4zKIAT08llbk78Ejwll52BNvlbZRiBnauLDeJqNo= X-Received: by 2002:adf:f584:: with SMTP id f4mr4727045wro.311.1614050742462; Mon, 22 Feb 2021 19:25:42 -0800 (PST) MIME-Version: 1.0 References: <48b78acb-7667-7829-8dd0-e753b7ac3336@FreeBSD.org> In-Reply-To: From: Chris Anderson Date: Mon, 22 Feb 2021 21:25:30 -0600 Message-ID: Subject: Re: lots of "no such file or directory" errors in zfs filesystem To: Andriy Gapon Cc: freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 4Dl4FN1mmpz3pWs X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=pobox.com (policy=none); spf=pass (mx1.freebsd.org: domain of chrisanderson@gmail.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=chrisanderson@gmail.com X-Spamd-Result: default: False [-2.89 / 15.00]; FROM_NEQ_ENVFROM(0.00)[cva@pobox.com,chrisanderson@gmail.com]; MAILMAN_DEST(0.00)[freebsd-stable]; DMARC_POLICY_SOFTFAIL(0.10)[pobox.com : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; SPAMHAUS_ZRD(0.00)[209.85.221.48:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.221.48:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.48:from]; NEURAL_HAM_SHORT(-0.99)[-0.994]; FORGED_SENDER(0.30)[cva@pobox.com,chrisanderson@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.48:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 03:25:45 -0000 On Mon, Feb 22, 2021 at 9:13 AM Andriy Gapon wrote: > On 22/02/2021 16:20, Chris Anderson wrote: > > On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon > > wrote: > > > > On 22/02/2021 09:31, Chris Anderson wrote: > > > None of these files are especially important to me, however I was > wondering > > > if there would be any benefit to the community from trying to > debug this > > > issue further to understand what might be going wrong. > > > > Yes. > > > > > > Could you offer any guidance about what kind of debugging information I > could > > collect that would be of use? > > You can start with picking a single file that demonstrates the problem. > Then, > ls -li the-file > zdb -dddd file's-filesystem file's-inode-number > The filesystem can be found out from df output, the inode number is in ls > -li > output -- if the command prints anything at all. > If it does not, then do ls -lid on the file's directory and then zdb -dddd > for > the directory's inode number. In the output there should be the file name > and > its number (I think that it's in hex, but not sure). > so I can't ls -i the file since that triggers the no such file warning. if I run zdb -dddd on the inode of a directory which contains one of those missing files, I can get the inode of the file from that, but I don't get anything particularly interesting in the output. most of the files that are missing are in directories with a large number of files (the largest has 180k) but I managed to find a directory which had a single file entry that is missing: Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=46916371L/46916371P fill=908537 cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09 Object lvl iblk dblk dsize dnsize lsize %full type 38268 1 128K 1K 0 512 1K 100.00 ZFS directory 264 bonus ZFS znode dnode flags: USED_BYTES USERUSED_ACCOUNTED dnode maxblkid: 0 uid 1001 gid 1001 atime Sun Aug 6 02:00:41 2017 mtime Wed Apr 15 12:12:42 2020 ctime Wed Apr 15 12:12:42 2020 crtime Sat Aug 5 15:10:07 2017 gen 23881023 mode 40755 size 3 parent 38176 links 2 pflags 40800000144 xattr 0 rdev 0x0000000000000000 microzap: 1024 bytes, 1 entries hash_test.go = 38274 (type: Regular File) # zdb -dddd tank/home/cva 38274 Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=46916371L/46916371P fill=908537 cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09 Object lvl iblk dblk dsize dnsize lsize %full type zdb: dmu_bonus_hold(38274) failed, errno 2 > > -- > Andriy Gapon >