Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 5 Mar 2021 13:11:06 -0800
From:      Mark Millard <marklmi@yahoo.com>
To:        chris@cretaforce.gr, freebsd-stable@freebsd.org
Subject:   Re: Filesystem operations slower in 13.0 than 12.2
Message-ID:  <204B3B0C-E695-4CCF-90DE-14A841BA6180@yahoo.com>
In-Reply-To: <D3F6AD61-FE78-4042-BD17-48272B3D2780@yahoo.com>
References:  <D3F6AD61-FE78-4042-BD17-48272B3D2780@yahoo.com>

next in thread | previous in thread | raw e-mail | index | archive | help


On 2021-Mar-4, at 14:16, Mark Millard <marklmi at yahoo.com> wrote:

> Christos Chatzaras chris at cretaforce.gr wrote on
> Thu Mar 4 21:41:01 UTC 2021 :
>=20
>=20
>> After finding slow filesystem operations with 13.0-BETA2 I did more =
tests.
>>=20
>> All tests done with same hardware (Seagate ST4000NM0245 4TB HDD - 2 =
disks with RAID-1 using gmirror).
>>=20
>> Filesystem mounted with noatime.
>>=20
>> Command used:
>>=20
>> /usr/bin/time -l portsnap extract
>>=20
>> but similar differences I see with "/usr/bin/time -l rm -fr =
/usr/ports"
>=20
> I doubt that "rm -fr" gets large differences of the
> type:
>=20
> (from 12.2p4:)
>         0  messages sent
>         0  messages received
> vs. (13.0-BETA4 and 14.0-CURRENT:)
>      4412  messages sent
>   2536379  messages received

The more I think above the above figures, the more
it seems like 12.2 probably just does not track
messsages sent and received, especially given the
lack of huge "voluntary context switches" differences
vs. 13.0-BETA4 and 14.0-CURRENT. (I expect the message
sends/receives to context switch, but I might be
wrong.)

> In other words, large variations in Inter-Process-Communiciation
> counts, especially "received".
>=20
> It is not obvious that the "portsnap extract" issue
> is dominated by file system I/O vs IPC issues.
>=20
> portsanp is a script and does something that looks
> like the following, with the "while read" happening
> over 29000 times:
>=20
> . . . | while read FILE HASH; do
>                echo ${PORTSDIR}/${FILE}
>                if ! [ -s "${WORKDIR}/files/${HASH}.gz" ]; then
>                        echo "files/${HASH}.gz not found -- snapshot =
corrupt."
>                        return 1
>                fi
>                case ${FILE} in
>                */)
>                        rm -rf ${PORTSDIR}/${FILE%/}
>                        mkdir -p ${PORTSDIR}/${FILE}
>                        tar -xz --numeric-owner -f =
${WORKDIR}/files/${HASH}.gz \
>                            -C ${PORTSDIR}/${FILE}
>                        ;;
>                *)
>                        rm -f ${PORTSDIR}/${FILE}
>                        tar -xz --numeric-owner -f =
${WORKDIR}/files/${HASH}.gz \
>                            -C ${PORTSDIR} ${FILE}
>                        ;;
>                esac
>        done; then
>=20
> I expect that the "tar -xz . . . *.gz" sort of commands
> also involve internal IPC use. (It looked like the
> portsnap script has not changed noticeably since
> something like late 2016.)

I wonder if the large user and/or sys differences between
12.2 and 13.0-BETA4 might be in process creation given the
over 29000 repititions of the loop and the number of
processes created per loop iteration.

The block input and output figures make no clear
difference that I can tell:

        29  block input operations
      2783  block output operations
vs.
       716  block input operations
       868  block output operations

There is also:

  11821398  page reclaims
vs.
  12288156  page reclaims

but none of that suggests that scale of differences in:

       98.18 real        35.31 user        59.31 sys
vs.
      163.81 real        71.93 user       107.32 sys

So it might be that "time -l" just does not report
on what makes up much of the difference.

Given the scale of the differences, I'd not expect
the variations in the likes of "involuntary context
switches" or the like to explain much of the
observed differences.

(I avoid 14.0-CURRENT for this because of its debug
build status that was reported. I avoid 13.0-BETA2
because of know block input/output operation count
issues.)

> (13.0-BETA2 showed a large "voluntary context switches"
> difference as well, but I ignore that middle step in
> the version sequence here.)
>=20
> So I expect publishing the "rm -fr /usr/ports" figures
> from "time -l" would be appropriate. I do not know if
> the reports should be via separate topic or not but I
> doubt the figures with large differences will be the
> same for most-modern vs. older: I do not expect notable
> IPC from "rm -fr".
>=20
>> ----------
>>=20
>> FreeBSD 12.2p4=20
>>=20
>>       98.18 real        35.31 user        59.31 sys
>>     49064  maximum resident set size
>>        21  average shared memory size
>>         3  average unshared data size
>>        86  average unshared stack size
>>  11821398  page reclaims
>>         0  page faults
>>         0  swaps
>>        29  block input operations
>>      2783  block output operations
>>         0  messages sent
>>         0  messages received
>>         0  signals received
>>    354648  voluntary context switches
>>       322  involuntary context switches
>>=20
>> ----------
>>=20
>> FreeBSD 13.0-BETA2 (2021-02-12)
>>=20
>>      497.88 real        76.06 user       120.03 sys
>>     49032  maximum resident set size
>>        22  average shared memory size
>>         3  average unshared data size
>>        91  average unshared stack size
>>  12288156  page reclaims
>>        23  page faults
>>         0  swaps
>>     29890  block input operations
>>    621229  block output operations
>>      4412  messages sent
>>   2536379  messages received
>>         0  signals received
>>   1004790  voluntary context switches
>>       251  involuntary context switches
>>=20
>> --------------
>>=20
>> FreeBSD 13.0-BETA4 (2021-02-26)
>>=20
>>      163.81 real        71.93 user       107.32 sys
>>     49032  maximum resident set size
>>        21  average shared memory size
>>         3  average unshared data size
>>        89  average unshared stack size
>>  12288156  page reclaims
>>         5  page faults
>>         0  swaps
>>       716  block input operations
>>       868  block output operations
>>      4412  messages sent
>>   2536379  messages received
>>         0  signals received
>>    355244  voluntary context switches
>>       277  involuntary context switches
>>=20
>> ----------
>>=20
>> FreeBSD 14-CURRENT (2021-03-04)
>>=20
>>      255.43 real        74.94 user       148.90 sys
>>     49032  maximum resident set size
>>        23  average shared memory size
>>         3  average unshared data size
>>        96  average unshared stack size
>>  12288156  page reclaims
>>        23  page faults
>>         0  swaps
>>     31207  block input operations
>>       175  block output operations
>>      4412  messages sent
>>   2536379  messages received
>>         0  signals received
>>    385527  voluntary context switches
>>       369  involuntary context switches
>>=20
>> ----------
>>=20
>> Differences between 13.0 and 14-CURRENT maybe related to debugging =
features.
>>=20
>> But 13.0-BETA4 is slower than 12.2. Does someone have more =
information about this?
>=20
> Again, I expect that the "time -l" figures may point in
> different directions for "portsnap extract" vs.
> "rm -fr /usr/ports" in your context. The question may
> need to be split because the answers may be different.
>=20

While I still think explicit "rm -fr" figures would be
good to show, I no longer read so much into the
messages sent and received figure differences.


=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?204B3B0C-E695-4CCF-90DE-14A841BA6180>