Date: Sat, 23 Nov 2013 03:26:13 -0500 From: Eitan Adler <lists@eitanadler.com> To: "freebsd-fs@freebsd.org" <fs@freebsd.org> Subject: ZFS (or something) is absurdly slow Message-ID: <CAF6rxgmepSN9pFPH%2BQiLaNqhzXxkXwu=59zvfD-6gGEMg9zh1g@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Every so often I see absurdly slow tasks stuck on "zio->io_cv".
For example a recent "git checkout file.c" did not complete for many minutes
load: 0.65 cmd: git 74577 [zio->io_cv] 435.58r 0.20u 2.54s 0% 71488k
I have seen "ls ~" take tends of minutes to complete. Even "ls
/var/empty" can take just as long. This length of time is variable
but is usually much longer than expected.
Does anyone have any suggestions for helping to figure out what is
taking a long time?
I have two disks: a tiny 16GB SSD I use for boot and the root FS. I
also have a 1TB HDD I use for my ~ and other data:
=> 34 31277165 ada0 GPT (15G)
34 222 1 freebsd-boot (111K)
256 31276943 2 freebsd-zfs (15G)
=> 34 1953525101 ada1 GPT (932G)
34 94 - free - (47K)
128 33554432 1 freebsd-swap (16G)
33554560 1919970560 2 freebsd-zfs (916G)
1953525120 15 - free - (7.5K)
[10014 eitan@gravity (100%) ~ !2!]%time zfs list
NAME USED AVAIL REFER MOUNTPOINT
zdata 278G 620G 31K none
zdata/cache 3.08G 620G 3.08G /cache
zdata/compresseddisk 106G 620G 106G /root/compresseddisk
zdata/debug 711M 620G 711M /usr/lib/debug
zdata/distfiles 15.2G 620G 15.2G /data/distfiles
zdata/home 124G 620G 71.2G /home
zdata/home/eitan 52.9G 620G 34K none
zdata/home/eitan/svn 52.9G 620G 51.3G /home/eitan/svn
zdata/home/eitan/svn/ports_master 1.56G 620G 1.56G
/home/eitan/svn/fbsd/ports
zdata/local 3.24G 620G 3.24G /usr/local
zdata/obj 11.5G 620G 11.5G /usr/obj
zdata/ports 4.21G 620G 4.21G /usr/ports
zdata/poudriere 4.91G 620G 31K none
zdata/poudriere/data 1.10G 620G 1.10G
/usr/local/poudriere/data
zdata/poudriere/jails 3.81G 620G 31K none
zdata/poudriere/jails/83amd64 795M 620G 795M
/usr/local/poudriere/jails/83amd64
zdata/poudriere/jails/91amd64 1.03G 620G 1.03G
/usr/local/poudriere/jails/91amd64
zdata/poudriere/jails/91i386 961M 620G 961M
/usr/local/poudriere/jails/91i386
zdata/poudriere/jails/92amd64 1.07G 620G 1.07G
/usr/local/poudriere/jails/92amd64
zdata/src 578M 620G 578M /usr/src
zdata/work 4.88G 620G 4.88G /work
zroot 3.53G 11.1G 2.23G /
zroot/tmp 154M 11.1G 154M none
zroot/usr 715M 11.1G 715M /usr
zroot/var 455M 11.1G 80.9M /var
zroot/var/db 373M 11.1G 373M /var/db
zroot/var/empty 31K 11.1G 31K /var/empty
zroot/var/tmp 1022K 11.1G 1022K /var/tmp
zfs list 0.02s user 0.01s system 0% cpu 9.076 total
[10009 eitan@gravity (100%) ~ !2!]%time zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zdata 912G 278G 634G 30% 1.00x ONLINE -
zexternal - - - - - FAULTED -
zroot 14.9G 3.53G 11.3G 23% 1.00x ONLINE -
zpool list 0.00s user 0.01s system 0% cpu 2.192 total
[10019 eitan@gravity (100%) ~ ]%uname -a
FreeBSD gravity.local 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r258140M:
Thu Nov 14 17:04:27 EST 2013
eitan@gravity.local:/usr/obj/usr/src/sys/EADLER amd64
FWIW, this kernel is compiled with INVARIANTS but without WITNESS.
MALLOC_PRODUCTION is enabled.
--
Eitan Adler
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAF6rxgmepSN9pFPH%2BQiLaNqhzXxkXwu=59zvfD-6gGEMg9zh1g>