From owner-freebsd-fs@freebsd.org Wed Aug 2 12:59:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11699DD06A3; Wed, 2 Aug 2017 12:59:45 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 470C56408F; Wed, 2 Aug 2017 12:59:43 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1dcszy-0007ZC-55; Wed, 02 Aug 2017 14:43:38 +0200 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Eugene M. Zheganin" Cc: freebsd-stable Subject: Re: some general zfs tuning (for iSCSI) References: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> Date: Wed, 02 Aug 2017 14:43:31 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: bdb49c4ff80bd276e321aade33e76e02752072e2 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: ++ X-Spam-Score: 2.0 X-Spam-Status: No, score=2.0 required=5.0 tests=ALL_TRUSTED, BAYES_95 autolearn=disabled version=3.4.0 X-Scan-Signature: 788438cbfdc4dc137ce560360a3a99c7 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 12:59:45 -0000 On Fri, 28 Jul 2017 12:56:11 +0200, Eugene M. Zheganin wrote: > Hi, > > > I'm using several FreeBSD zfs installations as the iSCSI production > systems, they basically consist of an LSI HBA, and a JBOD with a bunch > of SSD disks (12-24, Intel, Toshiba or Sandisk (avoid Sandisks btw)). > And I observe a problem very often: gstat shows 20-30% of disk load, but > the system reacts very slowly: cloning a dataset takes 10 seconds, > similar operations aren't lightspeeding too. To my knowledge, until the > disks are 90-100% busy, this shouldn't happen. My systems are equipped > with 32-64 gigs of RAM, and the only tuning I use is limiting the ARC > size (in a very tender manner - at least to 16 gigs) and playing with > TRIM. The number of datasets is high enough - hundreds of clones, dozens > of snapshots, most of teh data ovjects are zvols. Pools aren't > overfilled, most are filled up to 60-70% (no questions about low space > pools, but even in this case the situation is clearer - %busy goes up in > the sky). > > So, my question is - is there some obvious zfs tuning not mentioned in > the Handbook ? On the other side - handbook isn't much clear on how to > tune zfs, it's written mostly in the manner of "these are sysctl iods > you can play with". Of course I have seen several ZFS tuning guides. > Like Opensolaris one, but they are mostly file- and > application-specific. Is there some special approach to tune ZFS in the > environment with loads of disks ? I don't know.... like tuning the vdev > cache or something simllar. ? > > > Thanks. > > Eugene. What version of FreeBSD are you running? What is the system doing during all this? How are your pools setup (raidz1/2/3, mirror, 3mirror)? How is your iSCSI configured and what are the clients doing with it? Is the data distributed evenly on all disks? Do the clients write a lot of sync data? I think this kind of information helps people helping you. Regards, Ronald.