From owner-freebsd-fs@freebsd.org Sat Jan 21 08:50:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A05DACB77FA for ; Sat, 21 Jan 2017 08:50:14 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8FA4F1288 for ; Sat, 21 Jan 2017 08:50:14 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 8C0E6CB77F7; Sat, 21 Jan 2017 08:50:14 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BBC2CB77F6 for ; Sat, 21 Jan 2017 08:50:14 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mail.sub.ru (mail.sub.ru [88.212.205.2]) by mx1.freebsd.org (Postfix) with SMTP id CF3CE1286 for ; Sat, 21 Jan 2017 08:50:13 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: (qmail 72750 invoked from network); 21 Jan 2017 11:42:50 +0300 Received: from ppp109-252-90-166.pppoe.spdop.ru (ppp109-252-90-166.pppoe.spdop.ru [109.252.90.166]) by mail.sub.ru ([88.212.205.2]) with ESMTP via TCP; 31 Dec 1969 23:59:59 -0000 To: fs@freebsd.org From: Alex Povolotsky Subject: Poor ZFS performance Message-ID: <083391de-d153-e0f6-c453-63d95d3e1f55@webmail.sub.ru> Date: Sat, 21 Jan 2017 11:42:35 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Antivirus: avast! (VPS 170120-2, 20.01.2017), Outbound message X-Antivirus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jan 2017 08:50:14 -0000 Hello I'm writing lots of (tens of millions) relatively small files, hashing them out in three-level directory, 100 entries per level. Speed is quite unsatisfying, about 2 millions files per day writing in 20 processes, that is, about one file per second for process. CPU is not loaded (la 1.34, 0.4%sys, 0.2% intr, 5.7% user) Disk is used (sysat gives 0.45 MB/s, 89% busy) atime is off, even sync is disabled. FreeBSD insta1.inf.sudo.su 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz (2400.14-MHz K8-class CPU) Origin="GenuineIntel" Id=0x106a5 Family=0x6 Model=0x1a Stepping=5 Features=0xbfebfbff Features2=0x9ce3bd AMD Features=0x28100800 AMD Features2=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID TSC: P-state invariant, performance statistics real memory = 34363932672 (32772 MB) avail memory = 33271255040 (31729 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <080312 APIC1521> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads No L2ARC. ARC Size: 60.73% 18.29 GiB Target Size: (Adaptive) 60.84% 18.32 GiB Min Size (Hard Limit): 12.50% 3.76 GiB Max Size (High Water): 8:1 30.12 GiB ARC Size Breakdown: Recently Used Cache Size: 93.56% 17.14 GiB Frequently Used Cache Size: 6.44% 1.18 GiB ARC Efficiency: 1.60b Cache Hit Ratio: 65.87% 1.05b Cache Miss Ratio: 34.13% 544.82m Actual Hit Ratio: 62.58% 999.04m Data Demand Efficiency: 99.88% 283.23m Data Prefetch Efficiency: 31.76% 109.26m (I've disabled prefetch yesterday with no visible result) Attempt to restart program with only two processes yields 10 files per second in several minutes falling to 5, to 3 and so on. Zpool status for the pool in question pool: Z840TZPB state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM Z840TZPB ONLINE 0 0 0 gpt/Z840TZPB ONLINE 0 0 0 errors: No known data errors smartctl === START OF INFORMATION SECTION === Model Family: Seagate Archive HDD Device Model: ST8000AS0002-1NA17Z Serial Number: Z840TZPB LU WWN Device Id: 5 000c50 093201557 Firmware Version: AR17 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5980 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Jan 21 11:41:30 2017 MSK SMART support is: Available - device has SMART capability. SMART support is: Enabled disk is fresh (420 hours), no errors. What can I do to improve writing performance? Alex