From owner-freebsd-stable@freebsd.org Fri Oct 21 00:55:01 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A40CC19B78 for ; Fri, 21 Oct 2016 00:55:01 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 04DF8363 for ; Fri, 21 Oct 2016 00:55:01 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: by mail-it0-x229.google.com with SMTP id 4so206684715itv.0 for ; Thu, 20 Oct 2016 17:55:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chen-org-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MUKTpbGALqtjtL6af361fPTs/BIUAOIiS9FCaqWOOdk=; b=PdmO//UuYiVoKuuff1pLvjbL1kB89IyG/muhwFUrCMgJiyB3BxsF0497ztIytZvegn 1mSrKF/F5fmD4XDhrV5W5P2EZbhXM2pG9imGXbyayGIVWE5P4VWoJDgTLqe/A0Ks6tdg I+ijpwjhRZMtXV+EBdYRJDbUBZQ3qdqYM46QNmtldqGL2tYOskQeqPiMcbd+n1mmmzL2 4tyG1CtMtoSV4ogCjk23TKWAsLBmb8bBJCuUq4XXs9vm/cxfZhMfH2VWlDAtvfiCXW8n osmY/cIJL9AJ93ZcdOgpWeJcCby01RfBtlSUPzfksJFSqlyfzryU5HxV78Te6GQN/np0 JxsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MUKTpbGALqtjtL6af361fPTs/BIUAOIiS9FCaqWOOdk=; b=KhKek/gWilP2JSKhZFwNYDS1Qekg/A4/Z4r8MC3oqLEKl4gJDyf60u5f2ugSZlbAOF bM5pNO1cWtsxznOQvm92RoHwPbe5nCwpOkOkmb/Te7bw5kDn8mHXaY6/M61TfsYnPc+I x9IMvPz/mE9JN9Vj4/eGInhhhgDKMu6wWbcwFPDObQP1bFq9p4adEl2TkVy1brfomIim gSl3v6zyqZxBSfd8grIwqes2xMQvQ9CgRDQiqCeiF8zq2Y+hlgfG+v2/rWxMs0B+KM0W vMhcfOQz84lLl9V2+awvM9OAqPcIx9iicwLEoNfZQdFqwHTLC7hMpR8dvB1dP/vQnUF0 h0zw== X-Gm-Message-State: AA6/9RnmmgByOjTiEuV1SbTHTyms43iVHRsM9ltdAd5BVbUqjGJkB3RR8J2DXPQY9qQgcEgOte6cR9OHcgxH8Q== X-Received: by 10.36.44.80 with SMTP id i77mr3677718iti.119.1477011298481; Thu, 20 Oct 2016 17:54:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.38.87 with HTTP; Thu, 20 Oct 2016 17:54:57 -0700 (PDT) X-Originating-IP: [203.99.129.1] In-Reply-To: <1b187ab5-cf67-bf66-ab5f-8ec5ae29c140@multiplay.co.uk> References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> <92607b3b-a2d1-2391-5bd6-9781f426d7a6@multiplay.co.uk> <1b187ab5-cf67-bf66-ab5f-8ec5ae29c140@multiplay.co.uk> From: Jonathan Chen Date: Fri, 21 Oct 2016 13:54:57 +1300 Message-ID: Subject: Re: zfs, a directory that used to hold lot of files and listing pause To: Steven Hartland Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Oct 2016 00:55:01 -0000 On 21 October 2016 at 12:56, Steven Hartland wrote: [...] > When you see the stalling what does gstat -pd and top -SHz show? On my dev box: 1:38pm# uname -a FreeBSD irontree 10.3-STABLE FreeBSD 10.3-STABLE #0 r307401: Mon Oct 17 10:17:22 NZDT 2016 root@irontree:/usr/obj/usr/src/sys/GENERIC amd64 1:49pm# gstat -pd dT: 1.004s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| cd0 18 618 1 128 41.4 606 52854 17.2 0 0 0.0 100.5| ada0 ^C 1:49pm# top -SHz last pid: 83284; load averages: 0.89, 0.68, 0.46 up 4+03:11:32 13:49:05 565 processes: 9 running, 517 sleeping, 17 zombie, 22 waiting CPU: 3.7% user, 0.0% nice, 1.9% system, 0.0% interrupt, 94.3% idle Mem: 543M Active, 2153M Inact, 11G Wired, 10M Cache, 2132M Free ARC: 7249M Total, 1325M MFU, 4534M MRU, 906M Anon, 223M Header, 261M Other Swap: 32G Total, 201M Used, 32G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 83149 postgres 38 0 2197M 528M zio->i 5 1:13 23.19% postgres 83148 jonc 22 0 36028K 13476K select 2 0:11 3.86% pg_restore 852 postgres 20 0 2181M 2051M select 5 0:27 0.68% postgres 0 root -15 - 0K 4240K - 6 0:50 0.49% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 6 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 6 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.29% kernel{zio_write_issue_} 3 root -8 - 0K 112K zio->i 6 1:50 0.20% zfskern{txg_thread_enter} 12 root -88 - 0K 352K WAIT 0 1:07 0.20% intr{irq268: ahci0} 0 root -16 - 0K 4240K - 4 0:29 0.20% kernel{zio_write_intr_4} 0 root -16 - 0K 4240K - 7 0:29 0.10% kernel{zio_write_intr_6} 0 root -16 - 0K 4240K - 0 0:29 0.10% kernel{zio_write_intr_1} 0 root -16 - 0K 4240K - 5 0:29 0.10% kernel{zio_write_intr_2} 0 root -16 - 0K 4240K - 1 0:29 0.10% kernel{zio_write_intr_5} ... Taking another look at the internal dir structure for postgres, I'm not too sure whether this is related to the original poster's problem though. Cheers. -- Jonathan Chen