From owner-freebsd-stable@freebsd.org Fri Oct 21 10:02:30 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC10CC1A64F for ; Fri, 21 Oct 2016 10:02:30 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-qk0-x22a.google.com (mail-qk0-x22a.google.com [IPv6:2607:f8b0:400d:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A43AE2D for ; Fri, 21 Oct 2016 10:02:30 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-qk0-x22a.google.com with SMTP id f128so136668526qkb.1 for ; Fri, 21 Oct 2016 03:02:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=09G4124X6kZpe2hIpYKMaBi3cxzY033SjX2MKU2LqF0=; b=X8w9XuNX3Eqmo860o/JY0CvTA+jIR4cyUCtClh982DKte02yWE8j110Q6/mKigM6Ds 1v9EWFN8mTZRQIPQ2xh0/+BSz5QxNhYMjf7JJPSJAGgGyvmr3207xED639vqg668XUoR C0uhQJQYihcJsWxILNdEzMU6i5OmExycatbcdkOZS5gTU9fnVBAMrhKglZmpAl4hiwgK 8KB+FWrs5vc9iNW5T3Y1gomoLSU/2srC9lXjDElSAqsYA+zafOcoB04gHgqQaPiHL69L FSxmQ+SexaXJjIlQ9vkZ3eaJa0hWbEIfN99wxaVlI2JCmBVo8Q/LKdGLu8akLT72XZCv UHZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=09G4124X6kZpe2hIpYKMaBi3cxzY033SjX2MKU2LqF0=; b=FkW+pJddUjg3rvJA7fr7UobdHCjyE5ima58EEz2D6ufTmTiZmazU6jssfMgXd76Bwu cxUOw4bnxFLV2CfLj3LhQ9PPIVdBXK2PYgz7GBoRJxm0dvBqu1vxWsFZIgOpTXVoMaep vHeOQ1Vr3lQ9Ae7blxsv2GhoEChXRtVtlbwlj91ltMRsCF09wONbG1D8wtmvorBHAA97 ztgHqiWRPrUxfN8zDKOMQywBm0N7BGI0oF4JwsjfUDJHaM0ZkYQmjwPWeUjAVKBQSlIw wWrwi8i6WFd9Yey3lJjk6X5yJia41dtLItWt0IP8/o3XoMerrG/Eq34hCbF06/aR01pP bfmQ== X-Gm-Message-State: ABUngvc8rVoIvf+9smg+VSs+lylwFXVxHTXl4VhQ0nzN+I7xpENL16lmi/BAhtRRnOGRGnrg X-Received: by 10.194.238.162 with SMTP id vl2mr19326wjc.39.1477044148897; Fri, 21 Oct 2016 03:02:28 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id pe5sm2041771wjb.15.2016.10.21.03.02.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Oct 2016 03:02:27 -0700 (PDT) Subject: Re: zfs, a directory that used to hold lot of files and listing pause To: freebsd-stable@freebsd.org References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> <40fa9fd6-15aa-d8f7-b958-8783e763e6bc@multiplay.co.uk> <577ab7b2-46c1-5cf0-b6ad-50895978d957@norma.perm.ru> <38a84fce-fd97-a6a1-5820-89e578d76f0b@multiplay.co.uk> <91f34889-7f4f-661d-c88f-8034402c39cd@norma.perm.ru> From: Steven Hartland Message-ID: <19a45e94-65e6-53a9-202d-c048055460d1@multiplay.co.uk> Date: Fri, 21 Oct 2016 11:02:57 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <91f34889-7f4f-661d-c88f-8034402c39cd@norma.perm.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Oct 2016 10:02:30 -0000 On 21/10/2016 10:04, Eugene M. Zheganin wrote: > Hi. > > On 21.10.2016 9:22, Steven Hartland wrote: >> On 21/10/2016 04:52, Eugene M. Zheganin wrote: >>> Hi. >>> >>> On 20.10.2016 21:17, Steven Hartland wrote: >>>> Do you have atime enabled for the relevant volume? >>> I do. >>>> >>>> If so disable it and see if that helps: >>>> zfs set atime=off >>>> >>> Nah, it doesn't help at all. >> As per with Jonathon what does gstat -pd and top -SHz show? > > gstat (while ls'ing): > > dT: 1.005s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 1 49 49 2948 13.5 0 0 0.0 0 0 0.0 > 65.0| ada0 > 0 32 32 1798 11.1 0 0 0.0 0 0 0.0 > 35.3| ada1 > Averagely busy then on rust. > gstat (while idling): > > dT: 1.003s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 > 0.0| ada0 > 0 2 2 255 0.8 0 0 0.0 0 0 0.0 > 0.1| ada1 > > top -SHz output doesn't really differ while ls'ing or idling: > > last pid: 12351; load averages: 0.46, 0.49, > 0.46 up 39+14:41:02 14:03:05 > 376 processes: 3 running, 354 sleeping, 19 waiting > CPU: 5.8% user, 0.0% nice, 16.3% system, 0.0% interrupt, 77.9% idle > Mem: 21M Active, 646M Inact, 931M Wired, 2311M Free > ARC: 73M Total, 3396K MFU, 21M MRU, 545K Anon, 1292K Header, 47M Other > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 600 root 39 0 27564K 5072K nanslp 1 295.0H 24.56% monit > 0 root -17 0 0K 2608K - 1 75:24 0.00% > kernel{zio_write_issue} > 767 freeswitch 20 0 139M 31668K uwait 0 48:29 0.00% > freeswitch{freeswitch} > 683 asterisk 20 0 806M 483M uwait 0 41:09 0.00% > asterisk{asterisk} > 0 root -8 0 0K 2608K - 0 37:43 0.00% > kernel{metaslab_group_t} > [... others lines are just 0% ...] This looks like you only have ~4Gb ram which is pretty low for ZFS I suspect vfs.zfs.prefetch_disable will be 1, which will crash the performance. Regards Steve