From owner-freebsd-stable@freebsd.org Fri Oct 21 04:21:21 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2227FC1A7BA for ; Fri, 21 Oct 2016 04:21:21 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B82063B for ; Fri, 21 Oct 2016 04:21:20 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-lf0-x22b.google.com with SMTP id b81so118009725lfe.1 for ; Thu, 20 Oct 2016 21:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=nOZlYr2hvEi0k4W3P+b27OAlmal8HSOKHvz6DK+I79I=; b=Uty3ZArEwRE9CTMLUaiRTjy1daVkfuk1oe6v/AkiWp425RJaGZzhOE2Kcc3nRiodOj p1psbRmX6/NU04T3IsHHybs1/mLBdTSJd94GWfnOvS0Q/qVfA4L8a9nWEPwSOTj1pm+9 zV5BxWXZK7VMdZ+A7I2cgH64yNACzFOGUzWywdgpEMPS/Gu8ho7S6AnG2S4Xm3EPhBLb sT3Q+fwN/x9f+B0+Z9D99Cm2hYGZyhmK+dl1nupgc3673da4qPLCHyXjMDxyXKfcDp1/ 74A1/zboGT+A7Ux/qIPmwh13fhqADUdqIL2mMcMQQAVm+BHn20tk99RZzXsFMU00CmSl uMjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=nOZlYr2hvEi0k4W3P+b27OAlmal8HSOKHvz6DK+I79I=; b=l3v1QGHBglL32hN+Q3Xtc/XlAk6nFeV3GjBlLW5fVzGjJFe3OZueVctPKYPvD81sYA SyACZjjuO7wK9gEl4B1/3KC2BvF2Ywz+dxjl7WU9pIDPJoa1WlaML8q0VwCmJgs4f51+ t3dtIKyN6BSo71R8f2wvYqfyc2SXIyhHS99+Fpo85rmggfAkx3gyLshx7al5pKMKEQiJ Bp6qyFypzTxk8ldDrH1npyRppSdWCGelkR1NFGuMSiIRbwmkuc1svK2E9rw11bXkg5Wy QrBSXeeRJGYoaUeVR5VfXo9chZQrOlh0uczR5ZKpHWYjFfGiapbaP7+XK8D1/yxaFRxm 3jUA== X-Gm-Message-State: AA6/9RmebIec8PR190Txd7MCLQ/YMduUDraa5mdpavEmRF8+ZGFkmVpcDk/8VMHBGfCqfjbj X-Received: by 10.28.142.82 with SMTP id q79mr1016713wmd.20.1477023676395; Thu, 20 Oct 2016 21:21:16 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id q135sm1997724wmd.16.2016.10.20.21.21.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Oct 2016 21:21:15 -0700 (PDT) Subject: Re: zfs, a directory that used to hold lot of files and listing pause To: Jonathan Chen References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> <92607b3b-a2d1-2391-5bd6-9781f426d7a6@multiplay.co.uk> <1b187ab5-cf67-bf66-ab5f-8ec5ae29c140@multiplay.co.uk> Cc: freebsd-stable@freebsd.org From: Steven Hartland Message-ID: Date: Fri, 21 Oct 2016 05:21:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Oct 2016 04:21:21 -0000 In your case there your vdev (ada0) is saturated with writes from postgres. You should consider more / faster disks. You might also want to consider enabling lz4 compression on the PG volume as its works well in IO bound situations. On 21/10/2016 01:54, Jonathan Chen wrote: > On 21 October 2016 at 12:56, Steven Hartland wrote: > [...] >> When you see the stalling what does gstat -pd and top -SHz show? > On my dev box: > > 1:38pm# uname -a > FreeBSD irontree 10.3-STABLE FreeBSD 10.3-STABLE #0 r307401: Mon Oct > 17 10:17:22 NZDT 2016 root@irontree:/usr/obj/usr/src/sys/GENERIC > amd64 > 1:49pm# gstat -pd > dT: 1.004s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 0 0 0 0 0.0 0 0 0.0 0 0 > 0.0 0.0| cd0 > 18 618 1 128 41.4 606 52854 17.2 0 0 > 0.0 100.5| ada0 > ^C > 1:49pm# top -SHz > last pid: 83284; load averages: 0.89, 0.68, 0.46 > up > 4+03:11:32 13:49:05 > 565 processes: 9 running, 517 sleeping, 17 zombie, 22 waiting > CPU: 3.7% user, 0.0% nice, 1.9% system, 0.0% interrupt, 94.3% idle > Mem: 543M Active, 2153M Inact, 11G Wired, 10M Cache, 2132M Free > ARC: 7249M Total, 1325M MFU, 4534M MRU, 906M Anon, 223M Header, 261M Other > Swap: 32G Total, 201M Used, 32G Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 83149 postgres 38 0 2197M 528M zio->i 5 1:13 23.19% postgres > 83148 jonc 22 0 36028K 13476K select 2 0:11 3.86% pg_restore > 852 postgres 20 0 2181M 2051M select 5 0:27 0.68% postgres > 0 root -15 - 0K 4240K - 6 0:50 0.49% > kernel{zio_write_issue_} > 0 root -15 - 0K 4240K - 6 0:50 0.39% > kernel{zio_write_issue_} > 0 root -15 - 0K 4240K - 6 0:50 0.39% > kernel{zio_write_issue_} > 0 root -15 - 0K 4240K - 7 0:50 0.39% > kernel{zio_write_issue_} > 0 root -15 - 0K 4240K - 7 0:50 0.39% > kernel{zio_write_issue_} > 0 root -15 - 0K 4240K - 7 0:50 0.29% > kernel{zio_write_issue_} > 3 root -8 - 0K 112K zio->i 6 1:50 0.20% > zfskern{txg_thread_enter} > 12 root -88 - 0K 352K WAIT 0 1:07 0.20% > intr{irq268: ahci0} > 0 root -16 - 0K 4240K - 4 0:29 0.20% > kernel{zio_write_intr_4} > 0 root -16 - 0K 4240K - 7 0:29 0.10% > kernel{zio_write_intr_6} > 0 root -16 - 0K 4240K - 0 0:29 0.10% > kernel{zio_write_intr_1} > 0 root -16 - 0K 4240K - 5 0:29 0.10% > kernel{zio_write_intr_2} > 0 root -16 - 0K 4240K - 1 0:29 0.10% > kernel{zio_write_intr_5} > ... > > Taking another look at the internal dir structure for postgres, I'm > not too sure whether this is related to the original poster's problem > though. > > Cheers.