From owner-freebsd-fs@freebsd.org Fri Jul 28 10:56:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1094DC3072; Fri, 28 Jul 2017 10:56:19 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 313B780B43; Fri, 28 Jul 2017 10:56:15 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net206-94.perm.ertelecom.ru [46.146.206.94] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id v6SAuBTj049223 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 28 Jul 2017 15:56:12 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: freebsd-fs@FreeBSD.org Cc: freebsd-stable From: "Eugene M. Zheganin" Subject: some general zfs tuning (for iSCSI) Message-ID: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> Date: Fri, 28 Jul 2017 15:56:11 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [1.50 / 25.00] RBL_SPAMHAUS_PBL(2.00)[94.206.146.46.zen.spamhaus.org : 127.0.0.10] HFILTER_HOSTNAME_UNKNOWN(2.50)[] BAYES_HAM(-3.00)[99.99%] DMARC_NA(0.00)[norma.perm.ru] MIME_GOOD(-0.10)[text/plain] R_DKIM_NA(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] MID_RHS_MATCH_FROM(0.00)[] RECEIVED_SPAMHAUS(0.00)[94.206.146.46.zen.spamhaus.org] TO_DN_SOME(0.00)[] RCPT_COUNT_2(0.00)[] TO_MATCH_ENVRCPT_ALL(0.00)[] FROM_HAS_DN(0.00)[] FROM_EQ_ENVFROM(0.00)[] RCVD_COUNT_1(0.00)[] ONCE_RECEIVED(0.10)[] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 0.69 X-Rspamd-Queue-ID: v6SAuBTj049223 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jul 2017 10:56:19 -0000 Hi, I'm using several FreeBSD zfs installations as the iSCSI production systems, they basically consist of an LSI HBA, and a JBOD with a bunch of SSD disks (12-24, Intel, Toshiba or Sandisk (avoid Sandisks btw)). And I observe a problem very often: gstat shows 20-30% of disk load, but the system reacts very slowly: cloning a dataset takes 10 seconds, similar operations aren't lightspeeding too. To my knowledge, until the disks are 90-100% busy, this shouldn't happen. My systems are equipped with 32-64 gigs of RAM, and the only tuning I use is limiting the ARC size (in a very tender manner - at least to 16 gigs) and playing with TRIM. The number of datasets is high enough - hundreds of clones, dozens of snapshots, most of teh data ovjects are zvols. Pools aren't overfilled, most are filled up to 60-70% (no questions about low space pools, but even in this case the situation is clearer - %busy goes up in the sky). So, my question is - is there some obvious zfs tuning not mentioned in the Handbook ? On the other side - handbook isn't much clear on how to tune zfs, it's written mostly in the manner of "these are sysctl iods you can play with". Of course I have seen several ZFS tuning guides. Like Opensolaris one, but they are mostly file- and application-specific. Is there some special approach to tune ZFS in the environment with loads of disks ? I don't know.... like tuning the vdev cache or something simllar. ? Thanks. Eugene.