Date: Tue, 20 Jun 2017 20:26:06 +0000 From: "Caza, Aaron" <Aaron.Caza@ca.weatherford.com> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, Steven Hartland <killing@multiplay.co.uk> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Message-ID: <c4c6d9a56a6648189011a28ab1a90f59@DM2PR58MB013.032d.mgd.msft.net>
next in thread | raw e-mail | index | archive | help
> -----Original Message-----
> From: Steven Hartland [mailto:killing@multiplay.co.uk]
> Sent: Tuesday, June 20, 2017 1:53 PM
> To: Caza, Aaron; freebsd-fs@freebsd.org
> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs
>
>
>
> On 20/06/2017 17:58, Caza, Aaron wrote:
> >
> >> Can you show the output from gstat -pd during this DD please.
> > dT: 1.001s w: 1.000s
> > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps =
ms/d %busy Name
> > 0 4318 4318 34865 0.0 0 0 0.0 0 0 =
0.0 14.2| ada0
> > 0 4402 4402 35213 0.0 0 0 0.0 0 0 =
0.0 14.4| ada1
> >
> > dT: 1.002s w: 1.000s
> > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps =
ms/d %busy Name
> > 1 4249 4249 34136 0.0 0 0 0.0 0 0 =
0.0 14.1| ada0
> > 0 4393 4393 35287 0.0 0 0 0.0 0 0 =
0.0 14.5| ada1
> You %busy is very low, so sounds like the bottleneck isn't in raw disk pe=
rformance but somewhere else.
>
> Might be interesting to see if anything stands out in top -Sz and then pr=
ess h for threads.
>
I rebooted the system to disable Trim so currently not degraded.
> You also mention ada, could you share:
> sysctl kern.cam
>
kern.cam.enc.emulate_array_devices: 1
kern.cam.sa.allow_io_split: 0
kern.cam.da.default_softtimeout: 0
kern.cam.da.send_ordered: 1
kern.cam.da.default_timeout: 60
kern.cam.da.retry_count: 4
kern.cam.da.poll_period: 3
kern.cam.ada.1.sort_io_queue: 0
kern.cam.ada.1.max_seq_zones: 0
kern.cam.ada.1.optimal_nonseq_zones: 0
kern.cam.ada.1.optimal_seq_zones: 0
kern.cam.ada.1.zone_support: None
kern.cam.ada.1.zone_mode: Not Zoned
kern.cam.ada.1.rotating: 0
kern.cam.ada.1.unmapped_io: 1
kern.cam.ada.1.write_cache: -1
kern.cam.ada.1.read_ahead: -1
kern.cam.ada.1.delete_method: DSM_TRIM
kern.cam.ada.0.sort_io_queue: 0
kern.cam.ada.0.max_seq_zones: 0
kern.cam.ada.0.optimal_nonseq_zones: 0
kern.cam.ada.0.optimal_seq_zones: 0
kern.cam.ada.0.zone_support: None
kern.cam.ada.0.zone_mode: Not Zoned
kern.cam.ada.0.rotating: 0
kern.cam.ada.0.unmapped_io: 1
kern.cam.ada.0.write_cache: -1
kern.cam.ada.0.read_ahead: -1
kern.cam.ada.0.delete_method: DSM_TRIM
kern.cam.ada.write_cache: 1
kern.cam.ada.read_ahead: 1
kern.cam.ada.spindown_suspend: 1
kern.cam.ada.spindown_shutdown: 1
kern.cam.ada.send_ordered: 1
kern.cam.ada.default_timeout: 30
kern.cam.ada.retry_count: 4
kern.cam.cd.timeout: 30000
kern.cam.cd.retry_count: 4
kern.cam.cd.poll_period: 3
kern.cam.scsi_delay: 5000
kern.cam.cam_srch_hi: 0
kern.cam.pmp.hide_special: 1
kern.cam.pmp.default_timeout: 30
kern.cam.pmp.retry_count: 1
kern.cam.debug_delay: 0
kern.cam.dflags: 0
kern.cam.num_doneqs: 2
kern.cam.xpt_generation: 13
kern.cam.boot_delay: 0
kern.cam.sort_io_queues: 1
> And:
> sysctl vfs.zfs
>
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 0
vfs.zfs.vol.immediate_write_sz: 32768
vfs.zfs.vol.unmap_enabled: 1
vfs.zfs.vol.recursive: 0
vfs.zfs.vol.mode: 1
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.version.ioctl: 7
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.immediate_write_sz: 32768
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.dva_throttle_enabled: 1
vfs.zfs.zio.exclude_metadata: 0
vfs.zfs.zio.use_uma: 1
vfs.zfs.zil_slog_limit: 786432
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.min_auto_ashift: 12
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.queue_depth_pct: 1000
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_write_min_active: 1
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.async_read_min_active: 1
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_write_min_active: 10
vfs.zfs.vdev.sync_read_max_active: 10
vfs.zfs.vdev.sync_read_min_active: 10
vfs.zfs.vdev.max_active: 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent: 60
vfs.zfs.vdev.async_write_active_min_dirty_percent: 30
vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1
vfs.zfs.vdev.mirror.non_rotating_inc: 0
vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576
vfs.zfs.vdev.mirror.rotating_seek_inc: 5
vfs.zfs.vdev.mirror.rotating_inc: 0
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.metaslabs_per_vdev: 200
vfs.zfs.txg.timeout: 5
vfs.zfs.space_map_blksz: 4096
vfs.zfs.spa_min_slop: 134217728
vfs.zfs.spa_slop_shift: 5
vfs.zfs.spa_asize_inflation: 24
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_synctime_ms: 1000000
vfs.zfs.debug_flags: 0
vfs.zfs.debugflags: 0
vfs.zfs.recover: 0
vfs.zfs.spa_load_verify_data: 1
vfs.zfs.spa_load_verify_metadata: 1
vfs.zfs.spa_load_verify_maxinflight: 10000
vfs.zfs.ccw_retry_interval: 300
vfs.zfs.check_hostid: 1
vfs.zfs.mg_fragmentation_threshold: 85
vfs.zfs.mg_noalloc_threshold: 0
vfs.zfs.condense_pct: 200
vfs.zfs.metaslab.bias_enabled: 1
vfs.zfs.metaslab.lba_weighting_enabled: 1
vfs.zfs.metaslab.fragmentation_factor_enabled: 1
vfs.zfs.metaslab.preload_enabled: 1
vfs.zfs.metaslab.preload_limit: 3
vfs.zfs.metaslab.unload_delay: 8
vfs.zfs.metaslab.load_pct: 50
vfs.zfs.metaslab.min_alloc_size: 33554432
vfs.zfs.metaslab.df_free_pct: 4
vfs.zfs.metaslab.df_alloc_threshold: 131072
vfs.zfs.metaslab.debug_unload: 0
vfs.zfs.metaslab.debug_load: 0
vfs.zfs.metaslab.fragmentation_threshold: 70
vfs.zfs.metaslab.gang_bang: 16777217
vfs.zfs.free_bpobj_enabled: 1
vfs.zfs.free_max_blocks: 18446744073709551615
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 4
vfs.zfs.resilver_delay: 2
vfs.zfs.top_maxinflight: 32
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.max_idistance: 67108864
vfs.zfs.zfetch.max_distance: 8388608
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 0
vfs.zfs.delay_scale: 500000
vfs.zfs.delay_min_dirty_percent: 60
vfs.zfs.dirty_data_sync: 67108864
vfs.zfs.dirty_data_max_percent: 10
vfs.zfs.dirty_data_max_max: 4294967296
vfs.zfs.dirty_data_max: 853010841
vfs.zfs.max_recordsize: 1048576
vfs.zfs.send_holes_without_birth_time: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.dedup.prefetch: 1
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_esize: 393216
vfs.zfs.mfu_ghost_metadata_esize: 224032256
vfs.zfs.mfu_ghost_size: 224425472
vfs.zfs.mfu_data_esize: 0
vfs.zfs.mfu_metadata_esize: 2973696
vfs.zfs.mfu_size: 8696320
vfs.zfs.mru_ghost_data_esize: 261177344
vfs.zfs.mru_ghost_metadata_esize: 0
vfs.zfs.mru_ghost_size: 261177344
vfs.zfs.mru_data_esize: 6841171968
vfs.zfs.mru_metadata_esize: 9506816
vfs.zfs.mru_size: 6954673152
vfs.zfs.anon_data_esize: 0
vfs.zfs.anon_metadata_esize: 0
vfs.zfs.anon_size: 22016
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 1803964416
vfs.zfs.arc_free_target: 14080
vfs.zfs.compressed_arc_enabled: 1
vfs.zfs.arc_shrink_shift: 7
vfs.zfs.arc_average_blocksize: 8192
vfs.zfs.arc_min: 901982208
vfs.zfs.arc_max: 7215857664
> Finally when its performing well can you repeat the gstat -pd
>
dT: 1.001s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d=
%busy Name
3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0=
90.7| ada0
3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0=
92.0| ada1
dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d=
%busy Name
3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0=
91.6| ada0
3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0=
93.0| ada1
test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec)
Note that this is with vfs.zfs.arc_min and vfs.zfs.arc_max unconstrained. =
In previous testing on FreeBSD 10.3 Stable, I had tried a setting of "4000M=
" for vfs.zfs.arc_min and vfs.zfs.arc_max and had still experienced the per=
formance degradation though that was using the GEOM ELI layering so this la=
test testing using ashift instead on FreeBSD 11.1 Beta 2 may exhibit a diff=
erent behavior.
--
Aaron
This message may contain confidential and privileged information. If it has=
been sent to you in error, please reply to advise the sender of the error =
and then immediately delete it. If you are not the intended recipient, do n=
ot read, copy, disclose or otherwise use this message. The sender disclaims=
any liability for such unauthorized use. PLEASE NOTE that all incoming e-m=
ails sent to Weatherford e-mail accounts will be archived and may be scanne=
d by us and/or by external service providers to detect and prevent threats =
to our systems, investigate illegal or inappropriate behavior, and/or elimi=
nate unsolicited promotional e-mails (spam). This process could result in d=
eletion of a legitimate e-mail before it is read by its intended recipient =
at our organization. Moreover, based on the scanning results, the full text=
of e-mails and attachments may be made available to Weatherford security a=
nd other personnel for review and appropriate action. If you have any conce=
rns about this process, please contact us at dataprivacy@weatherford.com.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c4c6d9a56a6648189011a28ab1a90f59>