From owner-freebsd-fs@FreeBSD.ORG Mon Mar 7 15:43:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDB841065674 for ; Mon, 7 Mar 2011 15:43:06 +0000 (UTC) (envelope-from cheeky.m@live.com) Received: from bay0-omc4-s4.bay0.hotmail.com (bay0-omc4-s4.bay0.hotmail.com [65.54.190.206]) by mx1.freebsd.org (Postfix) with ESMTP id D7F528FC13 for ; Mon, 7 Mar 2011 15:43:06 +0000 (UTC) Received: from BAY147-W47 ([65.54.190.199]) by bay0-omc4-s4.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Mar 2011 07:43:06 -0800 Message-ID: X-Originating-IP: [209.6.82.6] From: Roger Hammerstein To: Date: Mon, 7 Mar 2011 10:43:05 -0500 Importance: Normal In-Reply-To: References: , <20110302200310.GA86404@alchemy.franken.de>, <20110306152245.GA25023@alchemy.franken.de>, MIME-Version: 1.0 X-OriginalArrivalTime: 07 Mar 2011 15:43:06.0393 (UTC) FILETIME=[5A2CD890:01CBDCDE] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: FW: sparc64 hang with zfs v28 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 15:43:07 -0000 forwarding over here. Anyone using zfs on sparc64 ?=20 Anyone have guesses why zfs/zpool eat 99% of the cpu ?=20 Thanks! From: cheeky.m@live.com CC: freebsd-sparc64@freebsd.org Subject: RE: sparc64 hang with zfs v28 Date: Sun=2C 6 Mar 2011 23:27:42 -0500 > FYI=2C kernel modules generally should work again with r219340=2C I haven= 't > tested ZFS though. Thanks!=20 I cvsuppedd and rebuilt kernel. falcon# uname -a FreeBSD falcon 9.0-CURRENT FreeBSD 9.0-CURRENT #3: Sun Mar 6 18:55:14 EST = 2011 root@falcon:/usr/obj/usr/src/sys/GENERIC sparc64 falcon#=20 I did a kldload zfs and it loaded ok. falcon# kldstat Id Refs Address Size Name 1 9 0xc0000000 e42878 kernel 2 1 0xc14a2000 32e000 zfs.ko 3 1 0xc17d0000 104000 opensolaris.ko falcon#=20 But a 'zpool status' or 'zfs list' will cause a zfs or zpool process to eat 99% of a cpu and essentially hang the shell i ran zfs/zpool in. falcon# zfs list ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is pres= ent=3B to enable=2C add "vfs.zfs.prefetch_disable=3D0" to /boot/loader= .conf. ZFS filesystem version 5 ZFS storage pool version 28 [Hang here] last pid: 1012=3B load averages: 0.79=2C 0.30=2C 0.16 = up 0+00:13:58 20:58:43 23 processes: 2 running=2C 21 sleeping CPU: 0.0% user=2C 0.0% nice=2C 52.5% system=2C 0.0% interrupt=2C 47.5% i= dle Mem: 16M Active=2C 11M Inact=2C 46M Wired=2C 64K Cache=2C 12M Buf=2C 1915M = Free Swap: 4055M Total=2C 4055M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND = =20 1006 root 1 53 0 21672K 2904K CPU1 1 0:05 99.47% zfs 998 root 1 40 0 41776K 6376K select 0 0:01 0.00% sshd 994 root 1 16 0 11880K 3536K pause 0 0:01 0.00% csh 795 root 1 40 0 16720K 3968K select 0 0:00 0.00% ntpd 1001 root 1 16 0 11880K 3464K pause 0 0:00 0.00% csh 975 root 1 8 0 25168K 2672K wait 1 0:00 0.00% login stays at 99%. truss -p 1006 doesn't "attach"=2C it just hangs. ctrl-t on the zfs list shell: oad: 0.95 cmd: zfs 1006 [running] 182.26r 0.00u 4.66s 99% 2872k load: 0.95 cmd: zfs 1006 [running] 183.30r 0.00u 4.66s 99% 2872k load: 0.95 cmd: zfs 1006 [running] 183.76r 0.00u 4.66s 99% 2872k load: 0.95 cmd: zfs 1006 [running] 184.08r 0.00u 4.66s 99% 2872k load: 0.95 cmd: zfs 1006 [running] 184.36r 0.00u 4.66s 99% 2872k A second time with zpool status:: last pid: 1224=3B load averages: 0.98=2C 0.55=2C 0.24 = up 0+02:07:39 23:12:33 26 processes: 2 running=2C 24 sleeping CPU: 0.0% user=2C 0.0% nice=2C 50.2% system=2C 0.4% interrupt=2C 49.4% i= dle Mem: 18M Active=2C 13M Inact=2C 46M Wired=2C 64K Cache=2C 12M Buf=2C 1911M = Free Swap: 4055M Total=2C 4055M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND = =20 1200 root 1 62 0 22704K 2920K CPU1 1 0:00 99.02% zpool 793 root 1 40 0 16720K 3968K select 0 0:02 0.00% ntpd 1180 root 1 16 0 11880K 3536K pause 1 0:01 0.00% csh 1184 root 1 40 0 41776K 6376K select 0 0:01 0.00% sshd 1201 root 1 40 0 41776K 6376K select 0 0:01 0.00% sshd falcon# truss -p 1200 truss: can not attach to target process: Device busy falcon# truss -p 1200 truss: can not attach to target process: Device busy falcon#=20 ctrl-t on the zpool status command: load: 0.62 cmd: zpool 1200 [running] 54.30r 0.00u 0.07s 83% 2888k load: 0.99 cmd: zpool 1200 [running] 271.73r 0.00u 0.07s 99% 2888k load: 0.99 cmd: zpool 1200 [running] 272.37r 0.00u 0.07s 99% 2888k load: 0.99 cmd: zpool 1200 [running] 272.75r 0.00u 0.07s 99% 2888k load: 0.99 cmd: zpool 1200 [running] 273.38r 0.00u 0.07s 99% 2888k truss -f zpool status:: 1014: sigprocmask(SIG_SETMASK=2C0x0=2C0x0) =3D 0 (0x0) 1014: sigprocmask(SIG_BLOCK=2CSIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALR= M|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGX= CPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2=2C0x0) =3D 0= (0x0) 1014: sigprocmask(SIG_SETMASK=2C0x0=2C0x0) =3D 0 (0x0) 1014: sigprocmask(SIG_BLOCK=2CSIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALR= M|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGX= CPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2=2C0x0) =3D 0= (0x0) 1014: sigprocmask(SIG_SETMASK=2C0x0=2C0x0) =3D 0 (0x0) 1014: modfind(0x40d3f140=2C0x9a0=2C0xc78=2C0x10a=2C0x1027e8=2C0x7fdffffe8d= 0) =3D 303 (0x12f) 1014: open("/dev/zfs"=2CO_RDWR=2C06170) =3D 3 (0x3) 1014: open("/dev/zero"=2CO_RDONLY=2C0666) =3D 4 (0x4) 1014: open("/etc/zfs/exports"=2CO_RDONLY=2C0666) ERR#2 'No such file or= directory' 1014: __sysctl(0x7fdffff8de8=2C0x2=2C0x7fdffff8eb0=2C0x7fdffff8f18=2C0x40d= 3f118=2C0x13) =3D 0 (0x0) 1014: __sysctl(0x7fdffff8eb0=2C0x4=2C0x40e4d084=2C0x7fdffff8fe0=2C0x0=2C0x= 0) =3D 0 (0x0) [hang] ctrl-t=20 load: 0.31 cmd: zpool 1014 [running] 12.47r 0.00u 0.07s 44% 2912k 1014 root 1 54 0 22704K 2944K CPU0 0 0:00 98.47% zpool falcon# truss -p 1014 truss: can not attach to target process: Device busy iostat -x 1 shows no reads and no writes to any disks There's a 2-disk zfs mirror attached to this ultra60 from a freebsd-8 insta= ll=2C but I don't know why that would cause a problem with the latest zfs v28. I can successfully read the labels on those two mirror disks with zdb -l /d= ev/da[36] =