Date: Fri, 5 Mar 2021 13:11:06 -0800 From: Mark Millard <marklmi@yahoo.com> To: chris@cretaforce.gr, freebsd-stable@freebsd.org Subject: Re: Filesystem operations slower in 13.0 than 12.2 Message-ID: <204B3B0C-E695-4CCF-90DE-14A841BA6180@yahoo.com> In-Reply-To: <D3F6AD61-FE78-4042-BD17-48272B3D2780@yahoo.com> References: <D3F6AD61-FE78-4042-BD17-48272B3D2780@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2021-Mar-4, at 14:16, Mark Millard <marklmi at yahoo.com> wrote:
> Christos Chatzaras chris at cretaforce.gr wrote on
> Thu Mar 4 21:41:01 UTC 2021 :
>=20
>=20
>> After finding slow filesystem operations with 13.0-BETA2 I did more =
tests.
>>=20
>> All tests done with same hardware (Seagate ST4000NM0245 4TB HDD - 2 =
disks with RAID-1 using gmirror).
>>=20
>> Filesystem mounted with noatime.
>>=20
>> Command used:
>>=20
>> /usr/bin/time -l portsnap extract
>>=20
>> but similar differences I see with "/usr/bin/time -l rm -fr =
/usr/ports"
>=20
> I doubt that "rm -fr" gets large differences of the
> type:
>=20
> (from 12.2p4:)
> 0 messages sent
> 0 messages received
> vs. (13.0-BETA4 and 14.0-CURRENT:)
> 4412 messages sent
> 2536379 messages received
The more I think above the above figures, the more
it seems like 12.2 probably just does not track
messsages sent and received, especially given the
lack of huge "voluntary context switches" differences
vs. 13.0-BETA4 and 14.0-CURRENT. (I expect the message
sends/receives to context switch, but I might be
wrong.)
> In other words, large variations in Inter-Process-Communiciation
> counts, especially "received".
>=20
> It is not obvious that the "portsnap extract" issue
> is dominated by file system I/O vs IPC issues.
>=20
> portsanp is a script and does something that looks
> like the following, with the "while read" happening
> over 29000 times:
>=20
> . . . | while read FILE HASH; do
> echo ${PORTSDIR}/${FILE}
> if ! [ -s "${WORKDIR}/files/${HASH}.gz" ]; then
> echo "files/${HASH}.gz not found -- snapshot =
corrupt."
> return 1
> fi
> case ${FILE} in
> */)
> rm -rf ${PORTSDIR}/${FILE%/}
> mkdir -p ${PORTSDIR}/${FILE}
> tar -xz --numeric-owner -f =
${WORKDIR}/files/${HASH}.gz \
> -C ${PORTSDIR}/${FILE}
> ;;
> *)
> rm -f ${PORTSDIR}/${FILE}
> tar -xz --numeric-owner -f =
${WORKDIR}/files/${HASH}.gz \
> -C ${PORTSDIR} ${FILE}
> ;;
> esac
> done; then
>=20
> I expect that the "tar -xz . . . *.gz" sort of commands
> also involve internal IPC use. (It looked like the
> portsnap script has not changed noticeably since
> something like late 2016.)
I wonder if the large user and/or sys differences between
12.2 and 13.0-BETA4 might be in process creation given the
over 29000 repititions of the loop and the number of
processes created per loop iteration.
The block input and output figures make no clear
difference that I can tell:
29 block input operations
2783 block output operations
vs.
716 block input operations
868 block output operations
There is also:
11821398 page reclaims
vs.
12288156 page reclaims
but none of that suggests that scale of differences in:
98.18 real 35.31 user 59.31 sys
vs.
163.81 real 71.93 user 107.32 sys
So it might be that "time -l" just does not report
on what makes up much of the difference.
Given the scale of the differences, I'd not expect
the variations in the likes of "involuntary context
switches" or the like to explain much of the
observed differences.
(I avoid 14.0-CURRENT for this because of its debug
build status that was reported. I avoid 13.0-BETA2
because of know block input/output operation count
issues.)
> (13.0-BETA2 showed a large "voluntary context switches"
> difference as well, but I ignore that middle step in
> the version sequence here.)
>=20
> So I expect publishing the "rm -fr /usr/ports" figures
> from "time -l" would be appropriate. I do not know if
> the reports should be via separate topic or not but I
> doubt the figures with large differences will be the
> same for most-modern vs. older: I do not expect notable
> IPC from "rm -fr".
>=20
>> ----------
>>=20
>> FreeBSD 12.2p4=20
>>=20
>> 98.18 real 35.31 user 59.31 sys
>> 49064 maximum resident set size
>> 21 average shared memory size
>> 3 average unshared data size
>> 86 average unshared stack size
>> 11821398 page reclaims
>> 0 page faults
>> 0 swaps
>> 29 block input operations
>> 2783 block output operations
>> 0 messages sent
>> 0 messages received
>> 0 signals received
>> 354648 voluntary context switches
>> 322 involuntary context switches
>>=20
>> ----------
>>=20
>> FreeBSD 13.0-BETA2 (2021-02-12)
>>=20
>> 497.88 real 76.06 user 120.03 sys
>> 49032 maximum resident set size
>> 22 average shared memory size
>> 3 average unshared data size
>> 91 average unshared stack size
>> 12288156 page reclaims
>> 23 page faults
>> 0 swaps
>> 29890 block input operations
>> 621229 block output operations
>> 4412 messages sent
>> 2536379 messages received
>> 0 signals received
>> 1004790 voluntary context switches
>> 251 involuntary context switches
>>=20
>> --------------
>>=20
>> FreeBSD 13.0-BETA4 (2021-02-26)
>>=20
>> 163.81 real 71.93 user 107.32 sys
>> 49032 maximum resident set size
>> 21 average shared memory size
>> 3 average unshared data size
>> 89 average unshared stack size
>> 12288156 page reclaims
>> 5 page faults
>> 0 swaps
>> 716 block input operations
>> 868 block output operations
>> 4412 messages sent
>> 2536379 messages received
>> 0 signals received
>> 355244 voluntary context switches
>> 277 involuntary context switches
>>=20
>> ----------
>>=20
>> FreeBSD 14-CURRENT (2021-03-04)
>>=20
>> 255.43 real 74.94 user 148.90 sys
>> 49032 maximum resident set size
>> 23 average shared memory size
>> 3 average unshared data size
>> 96 average unshared stack size
>> 12288156 page reclaims
>> 23 page faults
>> 0 swaps
>> 31207 block input operations
>> 175 block output operations
>> 4412 messages sent
>> 2536379 messages received
>> 0 signals received
>> 385527 voluntary context switches
>> 369 involuntary context switches
>>=20
>> ----------
>>=20
>> Differences between 13.0 and 14-CURRENT maybe related to debugging =
features.
>>=20
>> But 13.0-BETA4 is slower than 12.2. Does someone have more =
information about this?
>=20
> Again, I expect that the "time -l" figures may point in
> different directions for "portsnap extract" vs.
> "rm -fr /usr/ports" in your context. The question may
> need to be split because the answers may be different.
>=20
While I still think explicit "rm -fr" figures would be
good to show, I no longer read so much into the
messages sent and received figure differences.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?204B3B0C-E695-4CCF-90DE-14A841BA6180>