Date: Fri, 21 Oct 2016 05:21:45 +0100 From: Steven Hartland <killing@multiplay.co.uk> To: Jonathan Chen <jonc@chen.org.nz> Cc: freebsd-stable@freebsd.org Subject: Re: zfs, a directory that used to hold lot of files and listing pause Message-ID: <fcb9bee8-575c-b04b-1c2a-98f30ee3a8e8@multiplay.co.uk> In-Reply-To: <CAJuc1zPs0YHDQtw9=6sUKZ_6e=RZ0Qf814xdEy-wde0H3rcRGg@mail.gmail.com> References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> <nub8aq$203d$1@oper.dinoex.de> <CAJuc1zPJWAQbqQGLJdD_Zm_J8Z1tk_J9Tat7=%2BR90q37Ud54qA@mail.gmail.com> <92607b3b-a2d1-2391-5bd6-9781f426d7a6@multiplay.co.uk> <CAJuc1zNDDKX%2BqopGkNw=B5_mQgv5%2Bn9N2Bhh4i=qBWrXXaCtxA@mail.gmail.com> <1b187ab5-cf67-bf66-ab5f-8ec5ae29c140@multiplay.co.uk> <CAJuc1zPs0YHDQtw9=6sUKZ_6e=RZ0Qf814xdEy-wde0H3rcRGg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In your case there your vdev (ada0) is saturated with writes from postgres.
You should consider more / faster disks.
You might also want to consider enabling lz4 compression on the PG
volume as its works well in IO bound situations.
On 21/10/2016 01:54, Jonathan Chen wrote:
> On 21 October 2016 at 12:56, Steven Hartland <killing@multiplay.co.uk> wrote:
> [...]
>> When you see the stalling what does gstat -pd and top -SHz show?
> On my dev box:
>
> 1:38pm# uname -a
> FreeBSD irontree 10.3-STABLE FreeBSD 10.3-STABLE #0 r307401: Mon Oct
> 17 10:17:22 NZDT 2016 root@irontree:/usr/obj/usr/src/sys/GENERIC
> amd64
> 1:49pm# gstat -pd
> dT: 1.004s w: 1.000s
> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
> ms/d %busy Name
> 0 0 0 0 0.0 0 0 0.0 0 0
> 0.0 0.0| cd0
> 18 618 1 128 41.4 606 52854 17.2 0 0
> 0.0 100.5| ada0
> ^C
> 1:49pm# top -SHz
> last pid: 83284; load averages: 0.89, 0.68, 0.46
> up
> 4+03:11:32 13:49:05
> 565 processes: 9 running, 517 sleeping, 17 zombie, 22 waiting
> CPU: 3.7% user, 0.0% nice, 1.9% system, 0.0% interrupt, 94.3% idle
> Mem: 543M Active, 2153M Inact, 11G Wired, 10M Cache, 2132M Free
> ARC: 7249M Total, 1325M MFU, 4534M MRU, 906M Anon, 223M Header, 261M Other
> Swap: 32G Total, 201M Used, 32G Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 83149 postgres 38 0 2197M 528M zio->i 5 1:13 23.19% postgres
> 83148 jonc 22 0 36028K 13476K select 2 0:11 3.86% pg_restore
> 852 postgres 20 0 2181M 2051M select 5 0:27 0.68% postgres
> 0 root -15 - 0K 4240K - 6 0:50 0.49%
> kernel{zio_write_issue_}
> 0 root -15 - 0K 4240K - 6 0:50 0.39%
> kernel{zio_write_issue_}
> 0 root -15 - 0K 4240K - 6 0:50 0.39%
> kernel{zio_write_issue_}
> 0 root -15 - 0K 4240K - 7 0:50 0.39%
> kernel{zio_write_issue_}
> 0 root -15 - 0K 4240K - 7 0:50 0.39%
> kernel{zio_write_issue_}
> 0 root -15 - 0K 4240K - 7 0:50 0.29%
> kernel{zio_write_issue_}
> 3 root -8 - 0K 112K zio->i 6 1:50 0.20%
> zfskern{txg_thread_enter}
> 12 root -88 - 0K 352K WAIT 0 1:07 0.20%
> intr{irq268: ahci0}
> 0 root -16 - 0K 4240K - 4 0:29 0.20%
> kernel{zio_write_intr_4}
> 0 root -16 - 0K 4240K - 7 0:29 0.10%
> kernel{zio_write_intr_6}
> 0 root -16 - 0K 4240K - 0 0:29 0.10%
> kernel{zio_write_intr_1}
> 0 root -16 - 0K 4240K - 5 0:29 0.10%
> kernel{zio_write_intr_2}
> 0 root -16 - 0K 4240K - 1 0:29 0.10%
> kernel{zio_write_intr_5}
> ...
>
> Taking another look at the internal dir structure for postgres, I'm
> not too sure whether this is related to the original poster's problem
> though.
>
> Cheers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fcb9bee8-575c-b04b-1c2a-98f30ee3a8e8>