Date: Wed, 4 Oct 2017 22:13:28 -0700 From: javocado <javocado@gmail.com> To: Freddie Cash <fjwcash@gmail.com> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: lockup during zfs destroy Message-ID: <CAP1HOmRqbAqstzKMtJxP_g4DJxXXC%2B_WRF-Wnf1VbYE1FhROcw@mail.gmail.com> In-Reply-To: <CAOjFWZ7ohEkTvK-jRUOjFmTaaOOViJUrtQWKR8oJyo-CV=%2Bk6Q@mail.gmail.com> References: <CAP1HOmQtU14X1EvwYMHQmOru9S4uyXep=n0pU4PL5z-%2BQnX02A@mail.gmail.com> <CAOjFWZ54hB_jRaSQ8NX=s214Km9o%2BN=qvnQehJykZbY_QJGESA@mail.gmail.com> <CAOjFWZ7ohEkTvK-jRUOjFmTaaOOViJUrtQWKR8oJyo-CV=%2Bk6Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Setting vfs.zfs.free_max_blocks to 20k has not helped unfortunately. I was
able to get a small amount of debug out though. Any thoughts on how I can:
- get more detailed debug on the progress of this operation, or whether
progress is being made at all each time I reboot and start over after a
freeze
- configure my way out of this issue?
# dtrace -q -n 'zfs-dbgmsg{printf("%s\n", stringof(arg0))}'
txg 34628587 open pool version 28; software version 5000/5; uts host
10.3-RELEASE 1003000 amd64
txg 34628587 destroy begin tank/temp (id 3680)
txg 34628588 destroy tank/temp (id 3680)
On Wed, Oct 4, 2017 at 10:58 AM, Freddie Cash <fjwcash@gmail.com> wrote:
> On Wed, Oct 4, 2017 at 9:27 AM, Freddie Cash <fjwcash@gmail.com> wrote:
>
>> On Wed, Oct 4, 2017 at 9:15 AM, javocado <javocado@gmail.com> wrote:
>>
>>> I am trying to destroy a dense, large filesystem and it's not going wel=
l.
>>>
>>> Details:
>>> - zpool is a raidz3 with 3 x 12 drive vdevs.
>>> - target filesystem to be destroyed is ~2T with ~63M inodes.
>>> - OS: FreeBSD 10.3amd with 192 GB of RAM.
>>> - 120 GB of swap (90GB recently added as swap-on-disk)
>>>
>>
>> =E2=80=8BDo you have dedupe enabled on any filesystems in the pool? Or =
was it
>> enabled at any point in the past?
>>
>> This is a common occurrence when destroying large filesystems or lots of
>> filesystems/snapshots on pools that have/had dedupe enabled and there's =
not
>> enough RAM/L2ARC to contain the DDT. The system runs out of usable wire=
d
>> memory=E2=80=8B and locks up. Adding more RAM and/or being patient with=
the
>> boot-wait-lockup-repeat cycle will (usually) eventually allow it to fini=
sh
>> the destroy.
>>
>> There was a loader.conf tunable (or sysctl) added in the 10.x series tha=
t
>> mitigates this by limiting the number of delete operations that occur in=
a
>> transaction group, but I forget the details on it.
>>
>> Not sure if this affects pools that never had dedupe enabled or not.
>>
>> (We used to suffer through this at least once a year until we enabled a
>> delete-oldest-snapshot-before-running-backups process to limit the
>> number of snapshots.)=E2=80=8B
>>
>
> =E2=80=8BFound it. You can set vfs.zfs.free_max_blocks in /etc/sysctl.co=
nf. That
> will limit the number to-be-freed blocks in a single transaction group.
> You can play with that number until you find a value that won't run the
> system out of kernel memory trying to free all those blocks in a single
> transaction.
>
> On our problem server, running dedupe with only 64 GB of RAM for a 53 TB
> pool, we set it to 200,000 blocks:
>
> =E2=80=8Bvfs.zfs.free_max_blocks=3D200000
>
> --
> Freddie Cash
> fjwcash@gmail.com
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAP1HOmRqbAqstzKMtJxP_g4DJxXXC%2B_WRF-Wnf1VbYE1FhROcw>