From owner-freebsd-fs@FreeBSD.ORG Sat Nov 23 08:26:45 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1507AB4E for ; Sat, 23 Nov 2013 08:26:45 +0000 (UTC) Received: from mail-qa0-x22d.google.com (mail-qa0-x22d.google.com [IPv6:2607:f8b0:400d:c00::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CCF9C2C13 for ; Sat, 23 Nov 2013 08:26:44 +0000 (UTC) Received: by mail-qa0-f45.google.com with SMTP id o15so1229544qap.4 for ; Sat, 23 Nov 2013 00:26:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:from:date:message-id:subject:to:content-type; bh=iS2OUWPO0uZQFlKGdvl/t5EBy8eSsXKjbGA6FiET2so=; b=irDNbAw34YRi/loknH6mbIJZD2z0DAfC5dwDIGCYIBdrrQJfbwDPBEx0YINlBjF84B mwyCYlbLGQZizWFELT8a8UWSsL7mUDXjONmQvg0bfJ3I9gYoUE+qAAoxY5UKFVerHIvC aWJDzwe3ZEfZ6fDhFSrQZ5vO/cXJAw3VPjh5I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=iS2OUWPO0uZQFlKGdvl/t5EBy8eSsXKjbGA6FiET2so=; b=alxeUMOoWv7HTB9MDml6/pcW0qPJdlnXD+3CXR77gz/dClrwDuYNyi+vDfBCam7x5G MKN9TlCDFrFyYYAeSbKdT10OTnYnYGK0ieVepLrJGygsqQWtlWSiBeR5iGTodMe+fb2C mTBO42+dN/V39ajnaXLNEV160U97Guwpof1o+5F4HrQ7XtP0tUNKV4znrf9lzA5lsz44 V/RuL5HmOML1qtQjMRpQk5yFJsRK5ur/aKrak/Ibmo1s6LIAW6w/vSfVdml6cTxc1AhO 9S+8Kfz+ft2pODZJ8luTp/ge7IDmZ9SG4DeKr++rHFANAyGpFJZyvR4RY8sgeq2zYQxP u6RQ== X-Gm-Message-State: ALoCoQmEfAEq0mzjKOXRcbNNE/WGMrISOl+qMdNL4Uc7imEI9K/RqYtQ8mUH8ND2sKEJupRNgauE X-Received: by 10.224.69.132 with SMTP id z4mr28490516qai.78.1385195203838; Sat, 23 Nov 2013 00:26:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.96.63.101 with HTTP; Sat, 23 Nov 2013 00:26:13 -0800 (PST) From: Eitan Adler Date: Sat, 23 Nov 2013 03:26:13 -0500 Message-ID: Subject: ZFS (or something) is absurdly slow To: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Nov 2013 08:26:45 -0000 Every so often I see absurdly slow tasks stuck on "zio->io_cv". For example a recent "git checkout file.c" did not complete for many minutes load: 0.65 cmd: git 74577 [zio->io_cv] 435.58r 0.20u 2.54s 0% 71488k I have seen "ls ~" take tends of minutes to complete. Even "ls /var/empty" can take just as long. This length of time is variable but is usually much longer than expected. Does anyone have any suggestions for helping to figure out what is taking a long time? I have two disks: a tiny 16GB SSD I use for boot and the root FS. I also have a 1TB HDD I use for my ~ and other data: => 34 31277165 ada0 GPT (15G) 34 222 1 freebsd-boot (111K) 256 31276943 2 freebsd-zfs (15G) => 34 1953525101 ada1 GPT (932G) 34 94 - free - (47K) 128 33554432 1 freebsd-swap (16G) 33554560 1919970560 2 freebsd-zfs (916G) 1953525120 15 - free - (7.5K) [10014 eitan@gravity (100%) ~ !2!]%time zfs list NAME USED AVAIL REFER MOUNTPOINT zdata 278G 620G 31K none zdata/cache 3.08G 620G 3.08G /cache zdata/compresseddisk 106G 620G 106G /root/compresseddisk zdata/debug 711M 620G 711M /usr/lib/debug zdata/distfiles 15.2G 620G 15.2G /data/distfiles zdata/home 124G 620G 71.2G /home zdata/home/eitan 52.9G 620G 34K none zdata/home/eitan/svn 52.9G 620G 51.3G /home/eitan/svn zdata/home/eitan/svn/ports_master 1.56G 620G 1.56G /home/eitan/svn/fbsd/ports zdata/local 3.24G 620G 3.24G /usr/local zdata/obj 11.5G 620G 11.5G /usr/obj zdata/ports 4.21G 620G 4.21G /usr/ports zdata/poudriere 4.91G 620G 31K none zdata/poudriere/data 1.10G 620G 1.10G /usr/local/poudriere/data zdata/poudriere/jails 3.81G 620G 31K none zdata/poudriere/jails/83amd64 795M 620G 795M /usr/local/poudriere/jails/83amd64 zdata/poudriere/jails/91amd64 1.03G 620G 1.03G /usr/local/poudriere/jails/91amd64 zdata/poudriere/jails/91i386 961M 620G 961M /usr/local/poudriere/jails/91i386 zdata/poudriere/jails/92amd64 1.07G 620G 1.07G /usr/local/poudriere/jails/92amd64 zdata/src 578M 620G 578M /usr/src zdata/work 4.88G 620G 4.88G /work zroot 3.53G 11.1G 2.23G / zroot/tmp 154M 11.1G 154M none zroot/usr 715M 11.1G 715M /usr zroot/var 455M 11.1G 80.9M /var zroot/var/db 373M 11.1G 373M /var/db zroot/var/empty 31K 11.1G 31K /var/empty zroot/var/tmp 1022K 11.1G 1022K /var/tmp zfs list 0.02s user 0.01s system 0% cpu 9.076 total [10009 eitan@gravity (100%) ~ !2!]%time zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zdata 912G 278G 634G 30% 1.00x ONLINE - zexternal - - - - - FAULTED - zroot 14.9G 3.53G 11.3G 23% 1.00x ONLINE - zpool list 0.00s user 0.01s system 0% cpu 2.192 total [10019 eitan@gravity (100%) ~ ]%uname -a FreeBSD gravity.local 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r258140M: Thu Nov 14 17:04:27 EST 2013 eitan@gravity.local:/usr/obj/usr/src/sys/EADLER amd64 FWIW, this kernel is compiled with INVARIANTS but without WITNESS. MALLOC_PRODUCTION is enabled. -- Eitan Adler