From owner-freebsd-stable@freebsd.org Thu Oct 20 15:34:58 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2DDAC1ABE1 for ; Thu, 20 Oct 2016 15:34:58 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4581911E for ; Thu, 20 Oct 2016 15:34:57 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:27:b4d9:28e5:c182:e05b:6a1e] ([IPv6:2a02:2698:27:b4d9:28e5:c182:e05b:6a1e]) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id u9KFYsr1024981 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 20 Oct 2016 20:34:54 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1476977694; bh=lLUzJ2vf4omuiBb6nMiVtcSvVZ/Me9NjEWYQnmWkR4E=; h=Subject:To:References:From:Date:In-Reply-To; b=VSxHadqb6t9ZSvZejLMxtYelQqLxs9en5sXHtw7Mh1Plkh3JmSaqAiwulIjJiuqqU Phf6odPUwjsTA6g22wpLaFIzp0UaOqEdeOYZ5eEcqPo1LlyfF6V//uxjD8BwcWxxD8 A1QagnbjFt+6b5DOScjemhCZSIwz0fbg5N047q1Y= Subject: Re: zfs, a directory that used to hold lot of files and listing pause To: freebsd-stable@freebsd.org References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> From: "Eugene M. Zheganin" Message-ID: Date: Thu, 20 Oct 2016 20:34:59 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Oct 2016 15:34:59 -0000 Hi. On 20.10.2016 18:54, Nicolas Gilles wrote: > Looks like it's not taking up any processing time, so my guess is > the lag probably comes from stalled I/O ... bad disk? Well, I cannot rule this out completely, but first time I've seen this lag on this particular server about two months ago, and I guess two months is enough time for zfs on a redundant pool to ger errors, but as you can see: ]# zpool status pool: zroot state: ONLINE status: One or more devices are configured to use a non-native block size. Expect reduced performance. action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool. scan: resilvered 5.74G in 0h31m with 0 errors on Wed Jun 8 11:54:14 2016 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zroot0 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/zroot1 ONLINE 0 0 0 errors: No known data errors there's none. Yup, disks have different sector size, but this issue happened with one particular directory, not all of them. So I guess this is irrelevant. > Does a second "ls" immediately returned (ie. metadata has been > cached) ? Nope. Although the lag varies slightly: 4.79s real 0.00s user 0.02s sys 5.51s real 0.00s user 0.02s sys 4.78s real 0.00s user 0.02s sys 6.88s real 0.00s user 0.02s sys Thanks. Eugene.