Date: Sat, 21 Jan 2017 11:49:50 +0300 From: Alex Povolotsky <tarkhil@webmail.sub.ru> To: freebsd-fs@freebsd.org Subject: Poor ZFS performance Message-ID: <595d8117-e2f2-fa4f-a45e-3a9fb93d0687@webmail.sub.ru>
next in thread | raw e-mail | index | archive | help
Hello
I'm writing lots of (tens of millions) relatively small files, hashing
them out in three-level directory, 100 entries per level.
Speed is quite unsatisfying, about 2 millions files per day writing in
20 processes, that is, about one file per second for process.
CPU is not loaded (la 1.34, 0.4%sys, 0.2% intr, 5.7% user)
Disk is used (sysat gives 0.45 MB/s, 89% busy)
atime is off, even sync is disabled.
FreeBSD insta1.inf.sudo.su 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0
r306420: Thu Sep 29 01:43:23 UTC 2016
root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz (2400.14-MHz
K8-class CPU)
Origin="GenuineIntel" Id=0x106a5 Family=0x6 Model=0x1a Stepping=5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
TSC: P-state invariant, performance statistics
real memory = 34363932672 (32772 MB)
avail memory = 33271255040 (31729 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <080312 APIC1521>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
No L2ARC.
ARC Size: 60.73% 18.29 GiB
Target Size: (Adaptive) 60.84% 18.32 GiB
Min Size (Hard Limit): 12.50% 3.76 GiB
Max Size (High Water): 8:1 30.12 GiB
ARC Size Breakdown:
Recently Used Cache Size: 93.56% 17.14 GiB
Frequently Used Cache Size: 6.44% 1.18 GiB
ARC Efficiency: 1.60b
Cache Hit Ratio: 65.87% 1.05b
Cache Miss Ratio: 34.13% 544.82m
Actual Hit Ratio: 62.58% 999.04m
Data Demand Efficiency: 99.88% 283.23m
Data Prefetch Efficiency: 31.76% 109.26m
(I've disabled prefetch yesterday with no visible result)
Attempt to restart program with only two processes yields 10 files per
second in several minutes falling to 5, to 3 and so on.
Zpool status for the pool in question
pool: Z840TZPB
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
Z840TZPB ONLINE 0 0 0
gpt/Z840TZPB ONLINE 0 0 0
errors: No known data errors
smartctl
=== START OF INFORMATION SECTION ===
Model Family: Seagate Archive HDD
Device Model: ST8000AS0002-1NA17Z
Serial Number: Z840TZPB
LU WWN Device Id: 5 000c50 093201557
Firmware Version: AR17
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Jan 21 11:41:30 2017 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
disk is fresh (420 hours), no errors.
What can I do to improve writing performance?
Alex
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?595d8117-e2f2-fa4f-a45e-3a9fb93d0687>