From owner-freebsd-current@freebsd.org Sat Jan 2 17:12:39 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 792794D7968 for ; Sat, 2 Jan 2021 17:12:39 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f41.google.com (mail-oo1-f41.google.com [209.85.161.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D7T3W2vdFz4gH5; Sat, 2 Jan 2021 17:12:39 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f41.google.com with SMTP id o5so5339641oop.12; Sat, 02 Jan 2021 09:12:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kg4KkpsHB4pjw0aYOg+SmkCLG7KJqhEU4Cje7WQd0s0=; b=EDCgZ2AdTWEH95MJZXyXHY8dbWKa8A4VM9AUoYCGMvKzYLHgYHRqSeOaAkxEUmgdiE YSZ6Lwh241S4ucjM2YK+fmZTbu/zcUj8bODlSTumZXZmHVIMT+O4EPPmzPYC3wY+P0RL bj90JQO/Il0SkVyaTQN3YBTUzVeSa/vJp9eBoDoEqr4CAmz+DYu5XeDrU2LzZQtvlb3T VsUnunEDRtAOiCKJ/RJb/osnLz/0wEoQwV7nHt7fivvkufV4YxOFqGERkVoDOzZSykT5 pSHmeX6yZGaKoy0KaWDNpxqj3gPNLGVhe2zNBN5Oe6ERar1CjtmJ4Aei8HSXSzd6slrP 40Jw== X-Gm-Message-State: AOAM533cttC9bytqhiYBkkVYS3GYFmz38A8a5wB6c6+203dMKMqUIkNr WqTUOP5wkAlKgUuULLMAkqCibSd45ksFAraHP9g= X-Google-Smtp-Source: ABdhPJxA+1N2yB0aEF+SkBX8caG+g7pv7gdZ9UaI383VsSPGL2ljk7JRUSG/IZL61PKCtYFP6m00yRt315s5ApLw8Ps= X-Received: by 2002:a05:6820:34b:: with SMTP id m11mr34815832ooe.74.1609607558176; Sat, 02 Jan 2021 09:12:38 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Sat, 2 Jan 2021 10:12:27 -0700 Message-ID: Subject: Re: cp(1) of large files is causing 100% CPU utilization and poor transfer To: Rick Macklem Cc: Matthias Apitz , FreeBSD CURRENT , Konstantin Belousov , Kirk McKusick X-Rspamd-Queue-Id: 4D7T3W2vdFz4gH5 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jan 2021 17:12:39 -0000 It seems pretty clear to me, based on my experiments, that FIOSEEKHOLE is O(n) in filesize on UFS (though not on ZFS). And vn_generic_copy_file_range calls VOP_IOCTL(..., FIOSEEKHOLE) once for every block it copies, where blocks can range from 4KB to 1 MB. So I think there are three required actions: 1) Fix vn_generic_copy_file_range to remember the hole locations across different iterations of its loop. 2) Increase the block size that cp uses with copy_file_range. Right now it's 2MB, but it should be much larger. 3) Optionally, improve UFS's FIOSEEKHOLE performance. But it probably won't be necessary if we fix 1 and 2. -Alan On Sat, Jan 2, 2021 at 10:06 AM Rick Macklem wrote: > Just fyi, I've reproduced the problem. > All I did was create a 20Gbyte file > on UFS on a slow (4Gbyte or RAM, > slow spinning disk) laptop. > (The UFS file system is just what the installer creates these days.) > > cp still hasn't finished and is definitely > taking a looott longer than dd did. > > I'll start drilling down later to-day. > > I'll admit doing lots of testing of copy_file_range(2) > with large sparse files, but I may have missed testing > a large non-sparse file. > > rick > ps: I've added Kostik and Kirk to the cc. > > > ________________________________________ > From: owner-freebsd-current@freebsd.org > on behalf of Alan Somers > Sent: Saturday, January 2, 2021 11:30 AM > To: Matthias Apitz; FreeBSD CURRENT > Subject: Re: cp(1) of large files is causing 100% CPU utilization and poo= r > transfer > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca > > > On Sat, Jan 2, 2021 at 9:12 AM Matthias Apitz wrote: > > > El d=C3=ADa s=C3=A1bado, enero 02, 2021 a las 09:06:24a. m. -0700, Alan= Somers > > escribi=C3=B3: > > > > > > As I said, it can be reproduced using only the local file system. > This > > > > was setup recently on a SSD: > > > > > > > > # dmesg | grep ada0 > > > > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > > > > ada0: ACS-2 ATA SATA 3.x device > > > > ada0: Serial Number F995890846 > > > > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes) > > > > ada0: Command Queueing enabled > > > > ada0: 488386MB (1000215216 512 byte sectors) > > > > > > > > and by this procedure: > > > > > > > > # gpart create -s gpt ada0 > > > > # gpart add -t freebsd-boot -s 512k -a4k -l ssdboot ada0 > > > > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i1 ada0 > > > > # gpart add -t freebsd-ufs -l ssdrootfs -b 1m -s 2g ada0 > > > > # gpart add -t freebsd-ufs -l ssdvarfs -a 1m -s 2g ada0 > > > > # gpart add -t freebsd-ufs -l ssdusrfs -a 1m ada0 > > > > # newfs -U -t /dev/gpt/ssdrootfs > > > > # newfs -U -t /dev/gpt/ssdvarfs > > > > # newfs -U -t /dev/gpt/ssdusrfs > > > > > > > > # gpart show -l ada0 > > > > =3D> 40 1000215136 ada0 GPT (477G) > > > > 40 1024 1 ssdboot (512K) > > > > 1064 984 - free - (492K) > > > > 2048 4194304 2 ssdrootfs (2.0G) > > > > 4196352 4194304 3 ssdvarfs (2.0G) > > > > 8390656 16777216 4 ssdswap (8.0G) > > > > 25167872 975046656 5 ssdusrfs (465G) > > > > 1000214528 648 - free - (324K) > > > > > > > > # mount -t ufs > > > > /dev/gpt/ssdrootfs on / (ufs, local, soft-updates) > > > > /dev/gpt/ssdvarfs on /var (ufs, local, soft-updates) > > > > /dev/gpt/ssdusrfs on /usr (ufs, local, soft-updates) > > > > > > > > When I run in the /usr fs the command > > > > > > > > # cp -p guru-20210102.tar.gz xxx > > > > > > > > it copies around 168M per minute. > > > > > > > > > Is that copying from /usr to /usr, or from /usr to /var or /? > > > > # cd /home/backups > > # cp -p guru-20210102.tar.gz xxx > > > > i.e. from /usr to /usr. > > > > matthias > > > > Ok, let's narrow this down. Could you please run the command with the > attached D script ? > sudo dtrace -s copy_file_range.d -c "cp -p guru-20210102.tar.gz xxx" > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " >