Date: Sat, 2 Jan 2021 10:12:27 -0700 From: Alan Somers <asomers@freebsd.org> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Matthias Apitz <guru@unixarea.de>, FreeBSD CURRENT <freebsd-current@freebsd.org>, Konstantin Belousov <kib@freebsd.org>, Kirk McKusick <mckusick@mckusick.com> Subject: Re: cp(1) of large files is causing 100% CPU utilization and poor transfer Message-ID: <CAOtMX2hwDSt2EXX%2BS%2BL9pj9pJqsp8BQqa8tXtMHj_jeH7AjBYg@mail.gmail.com> In-Reply-To: <YQXPR0101MB0968022D574AA673BB80DEC6DDD40@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> References: <X/CKQFbpbWDdLXvw@c720-r368166.fritz.box> <CAOtMX2gd6vaBF=6Z6stefGRN8A7S4Gtf4drO-YgAbd=KXPwKNg@mail.gmail.com> <X/CY/kuKUJHUVEbB@c720-r368166.fritz.box> <CAOtMX2hFupzf-MD84eo_-n9OzfYX6b6tWRsHPECZGKaq5QCUVw@mail.gmail.com> <X/CbUu4tVQG81ItJ@c720-r368166.fritz.box> <CAOtMX2iRS6XVLkABSMQdcDCUHRXhEHEjyzuOqkMHudK=he33GA@mail.gmail.com> <YQXPR0101MB0968022D574AA673BB80DEC6DDD40@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
It seems pretty clear to me, based on my experiments, that FIOSEEKHOLE is O(n) in filesize on UFS (though not on ZFS). And vn_generic_copy_file_range calls VOP_IOCTL(..., FIOSEEKHOLE) once for every block it copies, where blocks can range from 4KB to 1 MB. So I think there are three required actions: 1) Fix vn_generic_copy_file_range to remember the hole locations across different iterations of its loop. 2) Increase the block size that cp uses with copy_file_range. Right now it's 2MB, but it should be much larger. 3) Optionally, improve UFS's FIOSEEKHOLE performance. But it probably won't be necessary if we fix 1 and 2. -Alan On Sat, Jan 2, 2021 at 10:06 AM Rick Macklem <rmacklem@uoguelph.ca> wrote: > Just fyi, I've reproduced the problem. > All I did was create a 20Gbyte file > on UFS on a slow (4Gbyte or RAM, > slow spinning disk) laptop. > (The UFS file system is just what the installer creates these days.) > > cp still hasn't finished and is definitely > taking a looott longer than dd did. > > I'll start drilling down later to-day. > > I'll admit doing lots of testing of copy_file_range(2) > with large sparse files, but I may have missed testing > a large non-sparse file. > > rick > ps: I've added Kostik and Kirk to the cc. > > > ________________________________________ > From: owner-freebsd-current@freebsd.org <owner-freebsd-current@freebsd.or= g> > on behalf of Alan Somers <asomers@freebsd.org> > Sent: Saturday, January 2, 2021 11:30 AM > To: Matthias Apitz; FreeBSD CURRENT > Subject: Re: cp(1) of large files is causing 100% CPU utilization and poo= r > transfer > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca > > > On Sat, Jan 2, 2021 at 9:12 AM Matthias Apitz <guru@unixarea.de> wrote: > > > El d=C3=ADa s=C3=A1bado, enero 02, 2021 a las 09:06:24a. m. -0700, Alan= Somers > > escribi=C3=B3: > > > > > > As I said, it can be reproduced using only the local file system. > This > > > > was setup recently on a SSD: > > > > > > > > # dmesg | grep ada0 > > > > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > > > > ada0: <TS512GMTS430S R0906A> ACS-2 ATA SATA 3.x device > > > > ada0: Serial Number F995890846 > > > > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes) > > > > ada0: Command Queueing enabled > > > > ada0: 488386MB (1000215216 512 byte sectors) > > > > > > > > and by this procedure: > > > > > > > > # gpart create -s gpt ada0 > > > > # gpart add -t freebsd-boot -s 512k -a4k -l ssdboot ada0 > > > > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i1 ada0 > > > > # gpart add -t freebsd-ufs -l ssdrootfs -b 1m -s 2g ada0 > > > > # gpart add -t freebsd-ufs -l ssdvarfs -a 1m -s 2g ada0 > > > > # gpart add -t freebsd-ufs -l ssdusrfs -a 1m ada0 > > > > # newfs -U -t /dev/gpt/ssdrootfs > > > > # newfs -U -t /dev/gpt/ssdvarfs > > > > # newfs -U -t /dev/gpt/ssdusrfs > > > > > > > > # gpart show -l ada0 > > > > =3D> 40 1000215136 ada0 GPT (477G) > > > > 40 1024 1 ssdboot (512K) > > > > 1064 984 - free - (492K) > > > > 2048 4194304 2 ssdrootfs (2.0G) > > > > 4196352 4194304 3 ssdvarfs (2.0G) > > > > 8390656 16777216 4 ssdswap (8.0G) > > > > 25167872 975046656 5 ssdusrfs (465G) > > > > 1000214528 648 - free - (324K) > > > > > > > > # mount -t ufs > > > > /dev/gpt/ssdrootfs on / (ufs, local, soft-updates) > > > > /dev/gpt/ssdvarfs on /var (ufs, local, soft-updates) > > > > /dev/gpt/ssdusrfs on /usr (ufs, local, soft-updates) > > > > > > > > When I run in the /usr fs the command > > > > > > > > # cp -p guru-20210102.tar.gz xxx > > > > > > > > it copies around 168M per minute. > > > > > > > > > Is that copying from /usr to /usr, or from /usr to /var or /? > > > > # cd /home/backups > > # cp -p guru-20210102.tar.gz xxx > > > > i.e. from /usr to /usr. > > > > matthias > > > > Ok, let's narrow this down. Could you please run the command with the > attached D script ? > sudo dtrace -s copy_file_range.d -c "cp -p guru-20210102.tar.gz xxx" > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hwDSt2EXX%2BS%2BL9pj9pJqsp8BQqa8tXtMHj_jeH7AjBYg>