From nobody Wed May 17 07:55:05 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QLll63Cm3z4C2GF for ; Wed, 17 May 2023 07:55:14 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Received: from mail.optiplex-networks.com (mail.optiplex-networks.com [212.159.80.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QLll50syZz3qXZ for ; Wed, 17 May 2023 07:55:13 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=optiplex-networks.com header.s=AE93A2AC-7F67-11EA-90AE-8A1FE64F6997 header.b="yzLfend/"; spf=pass (mx1.freebsd.org: domain of kayasaman@optiplex-networks.com designates 212.159.80.20 as permitted sender) smtp.mailfrom=kayasaman@optiplex-networks.com Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id 0352015C2C76 for ; Wed, 17 May 2023 08:55:06 +0100 (BST) Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id PvWdmz7Rf_Sl for ; Wed, 17 May 2023 08:55:05 +0100 (BST) Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id 819AA15C2EC6 for ; Wed, 17 May 2023 08:55:05 +0100 (BST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.optiplex-networks.com 819AA15C2EC6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=optiplex-networks.com; s=AE93A2AC-7F67-11EA-90AE-8A1FE64F6997; t=1684310105; bh=XhCjGLnVW9fIZBRwoktaoAjLg/Q70puwrW74gT2xJPI=; h=Message-ID:Date:MIME-Version:To:From; b=yzLfend/sXSsxxmTnePjda4XVd45oRM+54HtVp8QmmvBmdjqA8azdd/69uam6C3DJ rjx2KNYCTJm0PiQUzXfQgY3gEqz/YObh58VpO6pZB2cActFUVsfhrd9sqtWIoMEikC l8vC+Zcl+5Kj4XikvpjSlySz5q4FjDX5PsbTg4UIFcXi1pYEQvNzRMls0UG2ZnTAPB qrTR7Zxalm1cS/vid8CIOfQGFtENqaiDwlnOU0/V8Jdxnu7xvFT+YzgxG4cjo8b5sU K2Yl+YX5EfX2DJjoh4kqg7L10vAOTQBRJiafWTk2uprBY5keGrj/D4u2ctbnQudpmw 1uzpOS9TbeSdg== X-Virus-Scanned: amavisd-new at mail.optiplex-networks.com Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id cUNZOfMLnBLc for ; Wed, 17 May 2023 08:55:05 +0100 (BST) Received: from [192.168.20.23] (unknown [192.168.20.23]) by mail.optiplex-networks.com (Postfix) with ESMTPSA id 6B7BD15C2C76 for ; Wed, 17 May 2023 08:55:05 +0100 (BST) Message-ID: <1e30ac66-a339-ce08-75ac-8e566f4d2278@optiplex-networks.com> Date: Wed, 17 May 2023 08:55:05 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: Tool to compare directories and delete duplicate files from one directory Content-Language: en-US To: questions@freebsd.org References: <9887a438-95e7-87cc-a162-4ad7a70d744f@optiplex-networks.com> <7747f587-f33e-f39c-ac97-fe4fe19e0b76@optiplex-networks.com> <7c2429c5-55d0-1649-a442-ce543f2d46c2@holgerdanske.com> <6a0aba81-485a-8985-d20d-6da58e9b5580@optiplex-networks.com> <347612746.1721811.1683912265841@fidget.co-bxl> <08804029-03de-e856-568b-74494dfc81cf@holgerdansk e.com> <126434505.494354.1684104532813@ichabod.co-bxl> <818813a2-8ab0-df5 4-3c59-0e1ba9ce743d@holgerdanske.com> <941908372.622746.1684189567246@ichabod.co-bxl> From: Kaya Saman In-Reply-To: <941908372.622746.1684189567246@ichabod.co-bxl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4QLll50syZz3qXZ X-Spamd-Bar: / X-Spamd-Result: default: False [-0.50 / 15.00]; R_DKIM_ALLOW(-0.20)[optiplex-networks.com:s=AE93A2AC-7F67-11EA-90AE-8A1FE64F6997]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; DKIM_TRACE(0.00)[optiplex-networks.com:+]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; ASN(0.00)[asn:6871, ipnet:212.159.64.0/18, country:GB]; FROM_HAS_DN(0.00)[]; local_wl_ip(0.00)[212.159.80.20] X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_ip X-ThisMailContainsUnwantedMimeParts: N On 5/15/23 23:26, Sysadmin Lists wrote: >> ---------------------------------------- >> From: David Christensen >> Date: May 15, 2023, 1:43:38 AM >> To: >> Subject: Re: Tool to compare directories and delete duplicate files from one directory >> >> >> I looks like your script only finds duplicates when the subpath is >> identical (?): >> > Yeah. Wasn't that the original problem description? I went off the example > given by Paul earlier in this thread, and it looked like only files with > matching subpaths were being considered (because the OP accidentally rsync'd > files from a source to a bunch of destination dirs). > Glad to see this thread has turned into an interesting discussion.... Just as the OP :-) I will clarify.... There was no accidental rsync in place. Due to lack of storage my files where basically all over the place on different zpools. The problem is that most of those were on iscsi drives (all running Freebsd), so I needed to get them in a single place. Of course as the files where all over things became a mess. I bought a few new drives and created a new zpool just for this case. So virtually I had to sync the multiple directories to a single destination. *but* of course I didn't use the --remove-source-files option as I didn't want things to be destructive. But then I needed the extra space too and that's where this post came from. Regards, Kaya