From owner-freebsd-fs@FreeBSD.ORG Sun Aug 17 19:38:58 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2044A1065672 for ; Sun, 17 Aug 2008 19:38:58 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) by mx1.freebsd.org (Postfix) with ESMTP id EE7488FC0A for ; Sun, 17 Aug 2008 19:38:57 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from 76-205-169-61.lightspeed.austtx.sbcglobal.net ([76.205.169.61]:57925 helo=borg) by thebighonker.lerctr.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KUnsg-000Irp-1b; Sun, 17 Aug 2008 14:25:39 -0500 Date: Sun, 17 Aug 2008 14:25:27 -0500 (CDT) From: Larry Rosenman Sender: ler@borg To: Pawel Jakub Dawidek In-Reply-To: <20080727125413.GG1345@garage.freebsd.pl> Message-ID: <20080817142354.X2181@borg> References: <20080727125413.GG1345@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.3 (--) X-LERCTR-Spam-Score: -2.3 (--) X-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 X-LERCTR-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 DomainKey-Status: no signature Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Aug 2008 19:38:58 -0000 On Sun, 27 Jul 2008, Pawel Jakub Dawidek wrote: > Hi. > > http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 > > The patch above contains the most recent ZFS version that could be found > in OpenSolaris as of today. Apart for large amount of new functionality, > I belive there are many stability (and also performance) improvements > compared to the version from the base system. > > Check out OpenSolaris website to find out the differences between base > system version and patch version. > > Please test, test, test. If I get enough positive feedback, I may be > able to squeeze it into 7.1-RELEASE, but this might be hard. > > If you have any questions, please use mailing lists > (freebsd-fs@FreeBSD.org would be the best). > > Thank you in advance! > > I upgraded from 7-STABLE to 8-CURRENT yesterday with no issues, so I figured I'd try these. No issues, and now happily running zpool version 11 zfs version 3 on all my fs's (except root :) ) Thank You pjd! (amd64, 4g real, 6 SATA 400g disks). -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Sun Aug 17 22:14:03 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8799E106564A; Sun, 17 Aug 2008 22:14:03 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) by mx1.freebsd.org (Postfix) with ESMTP id 581938FC12; Sun, 17 Aug 2008 22:14:03 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from 76-205-169-61.lightspeed.austtx.sbcglobal.net ([76.205.169.61]:43741 helo=borg) by thebighonker.lerctr.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KUqVc-000MR9-8j; Sun, 17 Aug 2008 17:14:02 -0500 Date: Sun, 17 Aug 2008 17:13:57 -0500 (CDT) From: Larry Rosenman Sender: ler@borg To: Pawel Jakub Dawidek In-Reply-To: <20080817142354.X2181@borg> Message-ID: <20080817171314.M3100@borg> References: <20080727125413.GG1345@garage.freebsd.pl> <20080817142354.X2181@borg> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.3 (--) X-LERCTR-Spam-Score: -2.3 (--) X-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 X-LERCTR-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 DomainKey-Status: no signature Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Aug 2008 22:14:03 -0000 On Sun, 17 Aug 2008, Larry Rosenman wrote: > On Sun, 27 Jul 2008, Pawel Jakub Dawidek wrote: > >> Hi. >> >> http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 >> >> The patch above contains the most recent ZFS version that could be found >> in OpenSolaris as of today. Apart for large amount of new functionality, >> I belive there are many stability (and also performance) improvements >> compared to the version from the base system. >> >> Check out OpenSolaris website to find out the differences between base >> system version and patch version. >> >> Please test, test, test. If I get enough positive feedback, I may be >> able to squeeze it into 7.1-RELEASE, but this might be hard. >> >> If you have any questions, please use mailing lists >> (freebsd-fs@FreeBSD.org would be the best). >> >> Thank you in advance! >> >> > I upgraded from 7-STABLE to 8-CURRENT yesterday with no issues, so I figured > I'd try these. > > No issues, and now happily running zpool version 11 zfs version 3 on all my > fs's (except root :) ) > One comment, when I issue zfs commands against filesystems I get the following: WARNING pid 2412 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2412 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2479 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2479 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2479 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2479 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2494 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2494 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2494 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 WARNING pid 2494 (zfs): ioctl sign-extension ioctl ffffffffcc285a12 > Thank You pjd! > > (amd64, 4g real, 6 SATA 400g disks). > > > -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 From owner-freebsd-fs@FreeBSD.ORG Mon Aug 18 11:06:49 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 910FC1065681 for ; Mon, 18 Aug 2008 11:06:49 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 89EB28FC33 for ; Mon, 18 Aug 2008 11:06:49 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m7IB6nML079794 for ; Mon, 18 Aug 2008 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m7IB6mQc079790 for freebsd-fs@FreeBSD.org; Mon, 18 Aug 2008 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 18 Aug 2008 11:06:48 GMT Message-Id: <200808181106.m7IB6mQc079790@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Aug 2008 11:06:49 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t 7 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o bin/118249 fs mv(1): moving a directory changes its mtime o kern/124621 fs [ext3] Cannot mount ext2fs partition o kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file 9 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Aug 18 19:29:15 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60C101065673 for ; Mon, 18 Aug 2008 19:29:15 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 05CC18FC23 for ; Mon, 18 Aug 2008 19:29:14 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 57744 invoked by uid 89); 18 Aug 2008 19:29:12 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 18 Aug 2008 19:29:12 -0000 Date: Mon, 18 Aug 2008 15:29:12 -0400 (EDT) From: Weldon S Godfrey 3 To: freebsd-fs@freebsd.org, pjd@FreeBSD.org In-Reply-To: <20080806101621.H24586@emmett.excelsus.com> Message-ID: <20080814091337.Y94482@emmett.excelsus.com> References: <20080806101621.H24586@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Aug 2008 19:29:15 -0000 Update on what else I have tried (all yeild same results, same backtraces, no indication in logs/console of why it is panicing other than page fault: (FYI--I have tried to load 8-CURRENT, but it panics during install on the Dell 2950-3 I am using, I see a patch for a newer port of zfs, that looks like for 8, is there a patch for 7.0-RELEASE?) I have tried breaking it into two smaller < 2TB filesystems and performed same test on one, still I tried disabling swap all together (although I wasn't swapping) I upped number of nfs daemons from 12 to 100 I turned on zfs debugging and WITNESS to see if anything would show, like locking issues (nothing shows) I ran loops every 3s to monitor max vnodes, kmem, and arc during testes and up until the panic nothing was climbing I turned off ZIL and disabled prefetch, the problem still occurs I didn't get a panic in these situations: I created a zfs mirror filesystem of only two drives (one on each chasis) and performed the test I took one drive, created a UFS filesystem and performed the test. If memory serves me right, sometime around Aug 6, Weldon S Godfrey 3 told me: > > Hello, > > Please forgive me, I didn't really see this discussed in the archives but I am > wondering if anyone has seen this issue. I can replicate this issue under > FreeBSD amd64 7.0-RELEASE and the latest -STABLE (RELENG_7). I do not > replicate any problems running 9 instances of postmark on the machine > directly, so the issue appears to be isolated with NFS. > > There are backtraces and more information in ticket kern/124280 > > I am experiencing random kernel panics while running postmark benchmark from 9 > NFS clients (clients on RedHat) to a 3TB ZFS filesystem exported with NFS. > The panics happen as soon as 5 mins from starting the benchmark or may take > hours before it panics and reboots. It doesn't correspond to a time a cron > job is going on. I am using the following settings in postmark: > > set number 20000 > set transactions 10000000 > set subdirectories 1000 > set size 10000 15000 > set report verbose > set location /var/mail/store1/X (where X is a number 1-9 so each is operating > in its own tree) > > The problem happens if I run 1 postmark on 9 NFS clients at the same time > (each client is its own server) or if I run 9 postmarks on one NFS client. > > commands used to create filesystem: > zpool create tank mirror da0 da12 mirror da1 da13 mirror da2 da14 mirror da3 > da15\ > mirror da4 da16 mirror da5 da17 mirror da6 da18 mirror da7 da19 mirror da8 > da20 \ > mirror da9 da21 mirror da10 da22 spare da11 da23 > zfs set atime=off tank > zfs create tank/mail > zfs set mountpoint=/var/mail tank/mail > zfs set sharenfs="-maproot=root -network 192.168.2.0 -mask 255.255.255.0" > tank/mail > > I am using a 3ware 9690 SAS controller. I have 2 IBM EXP3000 enclosures, each > drive is shown as single disk by the controller. > > > this is my loader.conf: > vm.kmem_size_max="1073741824" > vm.kmem_size="1073741824" > kern.maxvnodes="800000" > vfs.zfs.prefetch_disable="1" > vfs.zfs.cache_flush_disable="1" > > (I should note that kern.maxnodes in loader.conf does not appear to do > anything, after boot, it is shown to be at 100000 with sysctl. It does change > to 800000 if I manually set it with sysctl. However it appears my vnode usage > sits at around 25-26K and is near that within 5s of the panic. > > The server has 16GB of RAM, and 2 quad core XEON processors. > > This server is only a NFS fileserver. The only non-default daemon running is > sshd. It is running the GENERIC kernel, right now, unmodified. > > I am using two NICs. NFS is exported only on the secondary NIC. Each NIC is > in it's own subnet. > > > nothing in /var/log/messages near time of panic except: > Aug 6 08:45:30 store1 savecore: reboot after panic: page fault > Aug 6 08:45:30 store1 savecore: writing core to vmcore.2 > > I can provide cores if needed. > > Thank you for your time! > > Weldon > > > > kgdb with backtrace: > > store1# kgdb kernel.debug /var/crash/vmcore.2 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > cpuid = 5; apic id = 05 > fault virtual address = 0xdc > fault code = supervisor read data, page not present > instruction pointer = 0x8:0xffffffff8063b3d8 > stack pointer = 0x10:0xffffffffdfbc5720 > frame pointer = 0x10:0xffffff00543ed000 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 839 (nfsd) > trap number = 12 > panic: page fault > cpuid = 5 > Uptime: 18m53s > Physical memory: 16366 MB > Dumping 1991 MB: 1976 1960 1944 1928 1912 1896 1880 1864 1848 1832 1816 1800 > 1784 1768 1752 1736 1720 1704 1688 1672 1656 1640 1624 1608 1592 1576 1560 > 1544 1528 1512 1496 1480 1464 1448 1432 1416 1400 1384 1368 1352 1336 1320 > 1304 1288 1272 1256 1240 1224 1208 1192 1176 1160 1144 1128 1112 1096 1080 > 1064 1048 1032 1016 1000 984 968 952 936 920 904 888 872 856 840 824 808 792 > 776 760 744 728 712 696 680 664 648 632 616 600 584 568 552 536 520 504 488 > 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 232 216 200 184 > 168 152 136 120 104 88 72 56 40 24 8 > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /boot/kernel/zfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > #0 doadump () at pcpu.h:194 > 194 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); > (kgdb) backtrace > #0 doadump () at pcpu.h:194 > #1 0x0000000000000004 in ?? () > #2 0xffffffff804a7049 in boot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:418 > #3 0xffffffff804a744d in panic (fmt=0x104
) at > /usr/src/sys/kern/kern_shutdown.c:572 > #4 0xffffffff807780e4 in trap_fatal (frame=0xffffff000bce26c0, > eva=18446742974395967712) > at /usr/src/sys/amd64/amd64/trap.c:724 > #5 0xffffffff807784b5 in trap_pfault (frame=0xffffffffdfbc5670, usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:641 > #6 0xffffffff80778de8 in trap (frame=0xffffffffdfbc5670) at > /usr/src/sys/amd64/amd64/trap.c:410 > #7 0xffffffff8075e7ce in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:169 > #8 0xffffffff8063b3d8 in nfsrv_access (vp=0xffffff00207d7dc8, flags=128, > cred=0xffffff00403d4800, rdonly=0, > td=0xffffff000bce26c0, override=0) at > /usr/src/sys/nfsserver/nfs_serv.c:4284 > #9 0xffffffff8063c4f1 in nfsrv3_access (nfsd=0xffffff00543ed000, > slp=0xffffff0006396d00, td=0xffffff000bce26c0, > mrq=0xffffffffdfbc5af0) at /usr/src/sys/nfsserver/nfs_serv.c:234 > #10 0xffffffff8064cd1d in nfssvc (td=Variable "td" is not available. > ) at /usr/src/sys/nfsserver/nfs_syscalls.c:456 > #11 0xffffffff80778737 in syscall (frame=0xffffffffdfbc5c70) at > /usr/src/sys/amd64/amd64/trap.c:852 > #12 0xffffffff8075e9db in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:290 > #13 0x0000000800687acc in ?? () > Previous frame inner to this frame (corrupt stack?) > From owner-freebsd-fs@FreeBSD.ORG Mon Aug 18 20:24:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4759D1065685 for ; Mon, 18 Aug 2008 20:24:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from gigi.cs.uoguelph.ca (gigi.cs.uoguelph.ca [131.104.94.210]) by mx1.freebsd.org (Postfix) with ESMTP id D0EFB8FC08 for ; Mon, 18 Aug 2008 20:24:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by gigi.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m7IKO1xB028160; Mon, 18 Aug 2008 16:24:01 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m7IKZmP09424; Mon, 18 Aug 2008 16:35:48 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Mon, 18 Aug 2008 16:35:48 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Doug Rabson In-Reply-To: Message-ID: References: <86myk06e18.fsf@ds4.des.no> <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.210 Cc: freebsd-fs@freebsd.org, =?utf-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Aug 2008 20:24:05 -0000 On Fri, 8 Aug 2008, Doug Rabson wrote: > > Don't use static linking? > Just to let everyone know, with help from Doug I have gotten a gssd.c to work with the libraries in FreeBSD-CURRENT and it has been uploaded to the Perforce server. I don't know exactly why it would crash in the gss_acquire_cred() call when dynaically linked, but it no longer does. (I changed to specifying the Kerberos mechanism explicitly instead of letting the library function work through the mechanism list, which might explain it. Anyhow, I'm a happy camper now, rick From owner-freebsd-fs@FreeBSD.ORG Wed Aug 20 16:19:30 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A72B6106568B for ; Wed, 20 Aug 2008 16:19:30 +0000 (UTC) (envelope-from caelian@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.173]) by mx1.freebsd.org (Postfix) with ESMTP id 332F18FC28 for ; Wed, 20 Aug 2008 16:19:29 +0000 (UTC) (envelope-from caelian@gmail.com) Received: by ug-out-1314.google.com with SMTP id o4so750472uge.39 for ; Wed, 20 Aug 2008 09:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=h7AXGVxbAdYwYrU59hSm9D0ECCrfc5odc6B5YCR5svk=; b=MvZzpuGK7Z3ihOdszvZmQQXHZNbLoU4fh/5TABfxhzY0dlGGXPTx+UyHqNuiaOuTiq hvwtThAHJAfbOcdG4Y9MyESRDQOhNaqtU8Vac9jgnG8mwkNeYVkByfud9pNERPWoX0On QqvaL6jkewt0VOSp7zB559I44vhEBUkZQSSII= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=w03MThA4fSjEafbx0lF9NqOTtoYLAqqbGTVI4UrRgZhct7g97Fswxjcs7o39hltbql YqDnN/f6yG2DmPyFRX5OgCJbw6x9EgM3hb4e9yCHGBym6xCqzDwa8nyW5boqBBqibGzq fKWj02h4mtVHacWyznkMKcC9mM39BxG0VV2E0= Received: by 10.210.71.12 with SMTP id t12mr327047eba.36.1219249168703; Wed, 20 Aug 2008 09:19:28 -0700 (PDT) Received: by 10.210.44.20 with HTTP; Wed, 20 Aug 2008 09:19:28 -0700 (PDT) Message-ID: Date: Wed, 20 Aug 2008 18:19:28 +0200 From: "Pascal Hofstee" To: "Pawel Jakub Dawidek" In-Reply-To: <20080727125413.GG1345@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080727125413.GG1345@garage.freebsd.pl> Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Aug 2008 16:19:30 -0000 On Sun, Jul 27, 2008 at 2:54 PM, Pawel Jakub Dawidek wrote: > Hi. > > http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 Just thought i'd give a slight HEADS UP for those of us running with this ZFS patch. With the recent VIMAGE code having hit the CURRENT source tree, the above patch fails to apply in a single (very minor way) in sys/kern/kern_jail.c contents of sys/kern/kern_jail.c.rej *************** *** 34,39 **** #include #include #include #include #include --- 34,40 ---- #include #include #include + #include #include #include The above rejection is caused by the inclusion of the header immediately after the inclusion in a fresh CURRENT source tree. So when you next update your CURRENT tree ... keep in mind that you will need to manually apply this part of the zfs-patchset (until somebody is kind enough to provide an updated patchset). -- Pascal Hofstee From owner-freebsd-fs@FreeBSD.ORG Wed Aug 20 18:43:39 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA6D91065677 for ; Wed, 20 Aug 2008 18:43:39 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 589C28FC17 for ; Wed, 20 Aug 2008 18:43:33 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 54050 invoked by uid 89); 20 Aug 2008 18:43:31 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 20 Aug 2008 18:43:31 -0000 Date: Wed, 20 Aug 2008 14:43:31 -0400 (EDT) From: Weldon S Godfrey 3 To: Pawel Jakub Dawidek In-Reply-To: Message-ID: <20080820143851.D76650@emmett.excelsus.com> References: <20080727125413.GG1345@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Aug 2008 18:43:39 -0000 I installed the latest from 8-HEAD from cvs today, and applied the ZFS patches in http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2. It compiled fine and came up fine. However, every zfs command yeilds an out of memory error: store1# zpool list internal error: out of memory store1# zpool destroy store1-1 internal error: out of memory store1# zpool status internal error: out of memory nothing mounts, I can't create since it says the devices are in use. loader.conf: vm.kmem_size_max="16106127360" vm.kmem_size="1073741824" kern.maxvnodes="800000" vfs.zfs.debug="1" #vfs.zfs.zil_disable="1" vfs.zfs.prefetch_disable="1" I verified settings took with sysctl -a. I 1st tried with zfs zil disabled. From owner-freebsd-fs@FreeBSD.ORG Wed Aug 20 19:10:43 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2197C106567B; Wed, 20 Aug 2008 19:10:43 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) by mx1.freebsd.org (Postfix) with ESMTP id F24048FC36; Wed, 20 Aug 2008 19:10:42 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from 64.3.1.253.ptr.us.xo.net ([64.3.1.253]:53245 helo=LROSENMAN) by thebighonker.lerctr.org with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KVt4r-0003ZJ-1X; Wed, 20 Aug 2008 14:10:42 -0500 From: "Larry Rosenman" To: "'Weldon S Godfrey 3'" , "'Pawel Jakub Dawidek'" References: <20080727125413.GG1345@garage.freebsd.pl> <20080820143851.D76650@emmett.excelsus.com> In-Reply-To: <20080820143851.D76650@emmett.excelsus.com> Date: Wed, 20 Aug 2008 14:10:30 -0500 Message-ID: <019b01c902f8$6ffaeb70$4ff0c250$@org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AckC9K+23e/9s+tiRhuIFxRpaxOL9AAA7RcA Content-Language: en-us X-Spam-Score: -2.3 (--) X-LERCTR-Spam-Score: -2.3 (--) X-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 X-LERCTR-Spam-Report: SpamScore (-2.3/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TVD_RCVD_IP=1.931, TW_ZF=0.077 DomainKey-Status: no signature Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: RE: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Aug 2008 19:10:43 -0000 did you install the new userland as well? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On Behalf Of Weldon S Godfrey 3 Sent: Wednesday, August 20, 2008 1:44 PM To: Pawel Jakub Dawidek Cc: freebsd-fs@freebsd.org; freebsd-current@freebsd.org Subject: Re: ZFS patches. I installed the latest from 8-HEAD from cvs today, and applied the ZFS patches in http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2. It compiled fine and came up fine. However, every zfs command yeilds an out of memory error: store1# zpool list internal error: out of memory store1# zpool destroy store1-1 internal error: out of memory store1# zpool status internal error: out of memory nothing mounts, I can't create since it says the devices are in use. loader.conf: vm.kmem_size_max="16106127360" vm.kmem_size="1073741824" kern.maxvnodes="800000" vfs.zfs.debug="1" #vfs.zfs.zil_disable="1" vfs.zfs.prefetch_disable="1" I verified settings took with sysctl -a. I 1st tried with zfs zil disabled. _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Aug 20 19:23:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23DBC1065672 for ; Wed, 20 Aug 2008 19:23:13 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id CE3478FC1B for ; Wed, 20 Aug 2008 19:23:12 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 59018 invoked by uid 89); 20 Aug 2008 19:23:10 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 20 Aug 2008 19:23:10 -0000 Date: Wed, 20 Aug 2008 15:23:10 -0400 (EDT) From: Weldon S Godfrey 3 To: Larry Rosenman In-Reply-To: <019b01c902f8$6ffaeb70$4ff0c250$@org> Message-ID: <20080820152239.R76650@emmett.excelsus.com> References: <20080727125413.GG1345@garage.freebsd.pl> <20080820143851.D76650@emmett.excelsus.com> <019b01c902f8$6ffaeb70$4ff0c250$@org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, 'Pawel Jakub Dawidek' Subject: RE: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Aug 2008 19:23:13 -0000 No, but that is a good point (sorry, my thick head thought it was only a kernel mod)...I'll do that. Thanks If memory serves me right, sometime around 2:10pm, Larry Rosenman told me: > did you install the new userland as well? > > > -- > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 512-248-2683 E-Mail: ler@lerctr.org > US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 > > > -----Original Message----- > From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On > Behalf Of Weldon S Godfrey 3 > Sent: Wednesday, August 20, 2008 1:44 PM > To: Pawel Jakub Dawidek > Cc: freebsd-fs@freebsd.org; freebsd-current@freebsd.org > Subject: Re: ZFS patches. > > > I installed the latest from 8-HEAD from cvs today, and applied the ZFS > patches in http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2. > > It compiled fine and came up fine. However, every zfs command yeilds an > out of memory error: > > store1# zpool list > internal error: out of memory > store1# zpool destroy store1-1 > internal error: out of memory > store1# zpool status > internal error: out of memory > > nothing mounts, I can't create since it says the devices are in use. > > loader.conf: > vm.kmem_size_max="16106127360" > vm.kmem_size="1073741824" > kern.maxvnodes="800000" > vfs.zfs.debug="1" > #vfs.zfs.zil_disable="1" > vfs.zfs.prefetch_disable="1" > > I verified settings took with sysctl -a. I 1st tried with zfs zil > disabled. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 12:54:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 788A51065677 for ; Thu, 21 Aug 2008 12:54:06 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from mail.yellowspace.net (mail.yellowspace.net [80.190.200.164]) by mx1.freebsd.org (Postfix) with ESMTP id EC65A8FC12 for ; Thu, 21 Aug 2008 12:54:05 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from five.intranet ([88.217.64.49]) (AUTH: LOGIN lopez.on.the.lists@yellowspace.net) by mail.yellowspace.net with esmtp; Thu, 21 Aug 2008 14:44:01 +0200 id 00278E68.0000000048AD6311.00017FB1 Message-Id: From: Lorenzo Perone To: freebsd-fs@FreeBSD.org In-Reply-To: <20080727125413.GG1345@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Thu, 21 Aug 2008 14:44:00 +0200 References: <20080727125413.GG1345@garage.freebsd.pl> X-Mailer: Apple Mail (2.926) Cc: Subject: Re: ZFS patches X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 12:54:06 -0000 Hi, Just let me intro this mail with a "Sorry for asking..." as I know the efforts already ongoing ar huge and I do respect this! But, here it is: any chances to see these patches on 7-STABLE anytime... soon? I think there would be many more testers available (me included) than for HEAD. In my case, for example, all I could afford now is to set up a complete-test-only box with the HEAD code, which in turn wouldn't be a real test case as it would be "just" a test box for zfs. Whereas I could afford to test it in much more "real life" situation with 7-STABLE. My guess is that this would be the case for many others. The problem about HEAD is that there would be too many spots with potential problems (which ports work, which don't, scripts that might make 7-bound assumptions, etc..) so that I can't afford that for anything below "test only" boxes.. Just experienced a deadlock again on 7-STABLE with zfs, that's why I'm refreshing this... Kudos && Regards, Lorenzo From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 18:15:31 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E36C1065676 for ; Thu, 21 Aug 2008 18:15:31 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from mail-gx0-f17.google.com (mail-gx0-f17.google.com [209.85.217.17]) by mx1.freebsd.org (Postfix) with ESMTP id 3E25B8FC1C for ; Thu, 21 Aug 2008 18:15:31 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by gxk10 with SMTP id 10so542691gxk.19 for ; Thu, 21 Aug 2008 11:15:30 -0700 (PDT) Received: by 10.142.164.10 with SMTP id m10mr35602wfe.60.1219342529251; Thu, 21 Aug 2008 11:15:29 -0700 (PDT) Received: from matt.spry.com ( [207.178.4.6]) by mx.google.com with ESMTPS id 31sm389563wff.16.2008.08.21.11.15.23 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 21 Aug 2008 11:15:24 -0700 (PDT) Message-Id: <9EA26FF4-3B5D-4C41-8A9D-50F752159566@corp.spry.com> From: Matt Simerson To: freebsd-fs@FreeBSD.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Date: Thu, 21 Aug 2008 11:15:20 -0700 References: <20080727125413.GG1345@garage.freebsd.pl> X-Mailer: Apple Mail (2.928.1) Cc: Subject: Re: ZFS patches X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 18:15:31 -0000 It's still a bit too early for me to make any announcement about ZFS and stability on HEAD but I was having deadlocks on 7.0 every other day under my workload. I took the plunge and upgraded both my servers (which are now in production, BTW) to HEAD. I have one running HEAD without the latest patches and one with HEAD + patch and have not experienced a deadlock since the upgrade. FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Fri Aug 15 16:42:36 PDT 2008 root@back01.int.spry.com:/usr/obj/usr/src/ sys/BACK01 amd64 FreeBSD back02.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed Aug 13 13:57:19 PDT 2008 root@back02.int.spry.com:/usr/obj/usr/src/ sys/BACK02-HEAD amd64 It turns out that I disliked the known instability of ZFS and 7-STABLE than the unknown risks associated with HEAD. As always, YMMMV but since ZFS is still experimental, odds are good you'll have a better experience if you are willing to upgrade to -HEAD. Matt $ cat /boot/loader.conf vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" vfs.zfs.prefetch_disable=1 On Aug 21, 2008, at 5:44 AM, Lorenzo Perone wrote: > Hi, > > Just let me intro this mail with a "Sorry for asking..." > as I know the efforts already ongoing ar huge and I do > respect this! > > But, here it is: any chances to see these patches on > 7-STABLE anytime... soon? > > I think there would be many more testers available (me included) > than for HEAD. In my case, for example, all I could afford now > is to set up a complete-test-only box with the HEAD code, which in > turn wouldn't be a real test case as it would be "just" a test box > for zfs. > > Whereas I could afford to test it in much more "real life" > situation with 7-STABLE. > My guess is that this would be the case for many others. > > The problem about HEAD is that there would be too many > spots with potential problems (which ports work, which don't, > scripts that might make 7-bound assumptions, etc..) > so that I can't afford that for anything below "test only" boxes.. > > Just experienced a deadlock again on 7-STABLE with zfs, that's > why I'm refreshing this... > > Kudos && Regards, > > Lorenzo > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 19:35:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A09081065678 for ; Thu, 21 Aug 2008 19:35:07 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 4B4778FC0A for ; Thu, 21 Aug 2008 19:35:06 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 95462 invoked by uid 89); 21 Aug 2008 19:35:04 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 21 Aug 2008 19:35:04 -0000 Date: Thu, 21 Aug 2008 15:35:04 -0400 (EDT) From: Weldon S Godfrey 3 To: freebsd-fs@freebsd.org, pjd@FreeBSD.org In-Reply-To: <20080814091337.Y94482@emmett.excelsus.com> Message-ID: <20080821153107.W76650@emmett.excelsus.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 19:35:07 -0000 Looks like the bug with NFS and ZFS still exists. Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran the benchmarks again this morning and after about an hour, it paniced with the same message about page fault with nfsd. It dropped to debugger on shutdown, it didn't do a savecore, dumpdev is set to AUTO. I will be more than happy to provide anything to assist in debugging it. Thanks! Weldon If memory serves me right, sometime around Monday, Weldon S Godfrey 3 told me: > > Update on what else I have tried (all yeild same results, same backtraces, no > indication in logs/console of why it is panicing other than page fault: > (FYI--I have tried to load 8-CURRENT, but it panics during install on the Dell > 2950-3 I am using, I see a patch for a newer port of zfs, that looks like for > 8, is there a patch for 7.0-RELEASE?) > > > I have tried breaking it into two smaller < 2TB filesystems and performed same > test on one, still > > I tried disabling swap all together (although I wasn't swapping) > > I upped number of nfs daemons from 12 to 100 > > I turned on zfs debugging and WITNESS to see if anything would show, like > locking issues (nothing shows) > > I ran loops every 3s to monitor max vnodes, kmem, and arc during testes and up > until the panic nothing was climbing > > I turned off ZIL and disabled prefetch, the problem still occurs > > > > > I didn't get a panic in these situations: > > I created a zfs mirror filesystem of only two drives (one on each chasis) and > performed the test > > I took one drive, created a UFS filesystem and performed the test. > > > > > If memory serves me right, sometime around Aug 6, Weldon S Godfrey 3 told me: > >> >> Hello, >> >> Please forgive me, I didn't really see this discussed in the archives but I >> am wondering if anyone has seen this issue. I can replicate this issue >> under FreeBSD amd64 7.0-RELEASE and the latest -STABLE (RELENG_7). I do not >> replicate any problems running 9 instances of postmark on the machine >> directly, so the issue appears to be isolated with NFS. >> >> There are backtraces and more information in ticket kern/124280 >> >> I am experiencing random kernel panics while running postmark benchmark from >> 9 NFS clients (clients on RedHat) to a 3TB ZFS filesystem exported with NFS. >> The panics happen as soon as 5 mins from starting the benchmark or may take >> hours before it panics and reboots. It doesn't correspond to a time a cron >> job is going on. I am using the following settings in postmark: >> >> set number 20000 >> set transactions 10000000 >> set subdirectories 1000 >> set size 10000 15000 >> set report verbose >> set location /var/mail/store1/X (where X is a number 1-9 so each is >> operating in its own tree) >> >> The problem happens if I run 1 postmark on 9 NFS clients at the same time >> (each client is its own server) or if I run 9 postmarks on one NFS client. >> >> commands used to create filesystem: >> zpool create tank mirror da0 da12 mirror da1 da13 mirror da2 da14 mirror da3 >> da15\ >> mirror da4 da16 mirror da5 da17 mirror da6 da18 mirror da7 da19 mirror da8 >> da20 \ >> mirror da9 da21 mirror da10 da22 spare da11 da23 >> zfs set atime=off tank >> zfs create tank/mail >> zfs set mountpoint=/var/mail tank/mail >> zfs set sharenfs="-maproot=root -network 192.168.2.0 -mask 255.255.255.0" >> tank/mail >> >> I am using a 3ware 9690 SAS controller. I have 2 IBM EXP3000 enclosures, >> each drive is shown as single disk by the controller. >> >> >> this is my loader.conf: >> vm.kmem_size_max="1073741824" >> vm.kmem_size="1073741824" >> kern.maxvnodes="800000" >> vfs.zfs.prefetch_disable="1" >> vfs.zfs.cache_flush_disable="1" >> >> (I should note that kern.maxnodes in loader.conf does not appear to do >> anything, after boot, it is shown to be at 100000 with sysctl. It does >> change to 800000 if I manually set it with sysctl. However it appears my >> vnode usage sits at around 25-26K and is near that within 5s of the panic. >> >> The server has 16GB of RAM, and 2 quad core XEON processors. >> >> This server is only a NFS fileserver. The only non-default daemon running >> is sshd. It is running the GENERIC kernel, right now, unmodified. >> >> I am using two NICs. NFS is exported only on the secondary NIC. Each NIC >> is in it's own subnet. >> >> >> nothing in /var/log/messages near time of panic except: >> Aug 6 08:45:30 store1 savecore: reboot after panic: page fault >> Aug 6 08:45:30 store1 savecore: writing core to vmcore.2 >> >> I can provide cores if needed. >> >> Thank you for your time! >> >> Weldon >> >> >> >> kgdb with backtrace: >> >> store1# kgdb kernel.debug /var/crash/vmcore.2 >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 5; apic id = 05 >> fault virtual address = 0xdc >> fault code = supervisor read data, page not present >> instruction pointer = 0x8:0xffffffff8063b3d8 >> stack pointer = 0x10:0xffffffffdfbc5720 >> frame pointer = 0x10:0xffffff00543ed000 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 839 (nfsd) >> trap number = 12 >> panic: page fault >> cpuid = 5 >> Uptime: 18m53s >> Physical memory: 16366 MB >> Dumping 1991 MB: 1976 1960 1944 1928 1912 1896 1880 1864 1848 1832 1816 1800 >> 1784 1768 1752 1736 1720 1704 1688 1672 1656 1640 1624 1608 1592 1576 1560 >> 1544 1528 1512 1496 1480 1464 1448 1432 1416 1400 1384 1368 1352 1336 1320 >> 1304 1288 1272 1256 1240 1224 1208 1192 1176 1160 1144 1128 1112 1096 1080 >> 1064 1048 1032 1016 1000 984 968 952 936 920 904 888 872 856 840 824 808 792 >> 776 760 744 728 712 696 680 664 648 632 616 600 584 568 552 536 520 504 488 >> 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 232 216 200 184 >> 168 152 136 120 104 88 72 56 40 24 8 >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /boot/kernel/zfs.ko.symbols...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> #0 doadump () at pcpu.h:194 >> 194 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); >> (kgdb) backtrace >> #0 doadump () at pcpu.h:194 >> #1 0x0000000000000004 in ?? () >> #2 0xffffffff804a7049 in boot (howto=260) at >> /usr/src/sys/kern/kern_shutdown.c:418 >> #3 0xffffffff804a744d in panic (fmt=0x104
) at >> /usr/src/sys/kern/kern_shutdown.c:572 >> #4 0xffffffff807780e4 in trap_fatal (frame=0xffffff000bce26c0, >> eva=18446742974395967712) >> at /usr/src/sys/amd64/amd64/trap.c:724 >> #5 0xffffffff807784b5 in trap_pfault (frame=0xffffffffdfbc5670, usermode=0) >> at /usr/src/sys/amd64/amd64/trap.c:641 >> #6 0xffffffff80778de8 in trap (frame=0xffffffffdfbc5670) at >> /usr/src/sys/amd64/amd64/trap.c:410 >> #7 0xffffffff8075e7ce in calltrap () at >> /usr/src/sys/amd64/amd64/exception.S:169 >> #8 0xffffffff8063b3d8 in nfsrv_access (vp=0xffffff00207d7dc8, flags=128, >> cred=0xffffff00403d4800, rdonly=0, >> td=0xffffff000bce26c0, override=0) at >> /usr/src/sys/nfsserver/nfs_serv.c:4284 >> #9 0xffffffff8063c4f1 in nfsrv3_access (nfsd=0xffffff00543ed000, >> slp=0xffffff0006396d00, td=0xffffff000bce26c0, >> mrq=0xffffffffdfbc5af0) at /usr/src/sys/nfsserver/nfs_serv.c:234 >> #10 0xffffffff8064cd1d in nfssvc (td=Variable "td" is not available. >> ) at /usr/src/sys/nfsserver/nfs_syscalls.c:456 >> #11 0xffffffff80778737 in syscall (frame=0xffffffffdfbc5c70) at >> /usr/src/sys/amd64/amd64/trap.c:852 >> #12 0xffffffff8075e9db in Xfast_syscall () at >> /usr/src/sys/amd64/amd64/exception.S:290 >> #13 0x0000000800687acc in ?? () >> Previous frame inner to this frame (corrupt stack?) >> > From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 19:47:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E270B106566B; Thu, 21 Aug 2008 19:47:42 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id CE8B98FC1A; Thu, 21 Aug 2008 19:47:42 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id B57EC1CC0C3; Thu, 21 Aug 2008 12:47:42 -0700 (PDT) Date: Thu, 21 Aug 2008 12:47:42 -0700 From: Jeremy Chadwick To: Weldon S Godfrey 3 Message-ID: <20080821194742.GA19362@eos.sc1.parodius.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080821153107.W76650@emmett.excelsus.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org, pjd@FreeBSD.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 19:47:43 -0000 On Thu, Aug 21, 2008 at 03:35:04PM -0400, Weldon S Godfrey 3 wrote: > Looks like the bug with NFS and ZFS still exists. > > Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran > the benchmarks again this morning and after about an hour, it paniced > with the same message about page fault with nfsd. It dropped to debugger > on shutdown, it didn't do a savecore, dumpdev is set to AUTO. Specifically regarding the debugger/didn't run savecore/dumpdev statement: What exactly did you type once at the debugger prompt? It matters. There's also this, which I reported nearly a year ago: http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/118255 I haven't been able to reproduce my above PR on RELENG_7, but I'm unaware of anything that might have changed in RELENG_7 that fixes this problem. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 19:50:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B48C71065674 for ; Thu, 21 Aug 2008 19:50:42 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206045140.chello.pl [87.206.45.140]) by mx1.freebsd.org (Postfix) with ESMTP id F26418FC1A for ; Thu, 21 Aug 2008 19:50:41 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 9CC4F45CA6; Thu, 21 Aug 2008 21:50:39 +0200 (CEST) Received: from localhost (chello087206045140.chello.pl [87.206.45.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 5BC96456B1; Thu, 21 Aug 2008 21:50:33 +0200 (CEST) Date: Thu, 21 Aug 2008 21:50:43 +0200 From: Pawel Jakub Dawidek To: Weldon S Godfrey 3 Message-ID: <20080821195043.GA1585@garage.freebsd.pl> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline In-Reply-To: <20080821153107.W76650@emmett.excelsus.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 19:50:42 -0000 --huq684BweRXVnRxX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 21, 2008 at 03:35:04PM -0400, Weldon S Godfrey 3 wrote: >=20 > Looks like the bug with NFS and ZFS still exists. >=20 > Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran= =20 > the benchmarks again this morning and after about an hour, it paniced wit= h=20 > the same message about page fault with nfsd. It dropped to debugger on= =20 > shutdown, it didn't do a savecore, dumpdev is set to AUTO. >=20 > I will be more than happy to provide anything to assist in debugging it. [...] > (kgdb) backtrace > #0 doadump () at pcpu.h:194 > #1 0x0000000000000004 in ?? () > #2 0xffffffff804a7049 in boot (howto=3D260) at /usr/src/sys/kern/kern_sh= utdown.c:418 > #3 0xffffffff804a744d in panic (fmt=3D0x104
) at /usr/src/sys/kern/kern_shutdown.c:572 > #4 0xffffffff807780e4 in trap_fatal (frame=3D0xffffff000bce26c0, eva=3D1= 8446742974395967712) at /usr/src/sys/amd64/amd64/trap.c:724 > #5 0xffffffff807784b5 in trap_pfault (frame=3D0xffffffffdfbc5670, usermo= de=3D0) at /usr/src/sys/amd64/amd64/trap.c:641 > #6 0xffffffff80778de8 in trap (frame=3D0xffffffffdfbc5670) at /usr/src/s= ys/amd64/amd64/trap.c:410 > #7 0xffffffff8075e7ce in calltrap () at /usr/src/sys/amd64/amd64/excepti= on.S:169 > #8 0xffffffff8063b3d8 in nfsrv_access (vp=3D0xffffff00207d7dc8, flags=3D= 128, cred=3D0xffffff00403d4800, rdonly=3D0, td=3D0xffffff000bce26c0, overri= de=3D0) at /usr/src/sys/nfsserver/nfs_serv.c:4284 > #9 0xffffffff8063c4f1 in nfsrv3_access (nfsd=3D0xffffff00543ed000, slp= =3D0xffffff0006396d00, td=3D0xffffff000bce26c0, mrq=3D0xffffffffdfbc5af0) a= t /usr/src/sys/nfsserver/nfs_serv.c:234 > #10 0xffffffff8064cd1d in nfssvc (td=3DVariable "td" is not available.) a= t /usr/src/sys/nfsserver/nfs_syscalls.c:456 > #11 0xffffffff80778737 in syscall (frame=3D0xffffffffdfbc5c70) at /usr/sr= c/sys/amd64/amd64/trap.c:852 > #12 0xffffffff8075e9db in Xfast_syscall () at /usr/src/sys/amd64/amd64/ex= ception.S:290 Can you tell me how exactly line 4284 of sys/nfsserver/nfs_serv.c looks in your source? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --huq684BweRXVnRxX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIrccTForvXbEpPzQRAmWgAJ0d2v0XAB0q/OWyQcGLK9ynZ0JyrwCfV9ww oirZw4ffN1oI7kH8YovhKCQ= =AXkH -----END PGP SIGNATURE----- --huq684BweRXVnRxX-- From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 20:16:58 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B94FD106567D for ; Thu, 21 Aug 2008 20:16:58 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2C6938FC1D for ; Thu, 21 Aug 2008 20:16:57 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.2/8.14.2) with ESMTP id m7LKGqv9019593; Thu, 21 Aug 2008 22:16:52 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.2/8.14.2/Submit) id m7LKGpkC019592; Thu, 21 Aug 2008 22:16:51 +0200 (CEST) (envelope-from olli) Date: Thu, 21 Aug 2008 22:16:51 +0200 (CEST) Message-Id: <200808212016.m7LKGpkC019592@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, matt@corp.spry.com In-Reply-To: <9EA26FF4-3B5D-4C41-8A9D-50F752159566@corp.spry.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.3-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Thu, 21 Aug 2008 22:16:52 +0200 (CEST) Cc: Subject: Re: ZFS patches X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, matt@corp.spry.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 20:16:58 -0000 Matt Simerson wrote: > [...] > FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Fri > Aug 15 16:42:36 PDT 2008 root@back01.int.spry.com:/usr/obj/usr/src/ > sys/BACK01 amd64 > > FreeBSD back02.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed > Aug 13 13:57:19 PDT 2008 root@back02.int.spry.com:/usr/obj/usr/src/ > sys/BACK02-HEAD amd64 > [...] > $ cat /boot/loader.conf > vm.kmem_size="1536M" > vm.kmem_size_max="1536M" I think those two lines can be removed, thanks to Alan Cox' recent improvents of the kmem addressing on amd64. > vfs.zfs.arc_min="16M" > vfs.zfs.arc_max="64M" > vfs.zfs.prefetch_disable=1 Those are probably OK. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Documentation is like sex; when it's good, it's very, very good, and when it's bad, it's better than nothing." -- Dick Brandon From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 20:36:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C80671065684 for ; Thu, 21 Aug 2008 20:36:02 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id B21598FC44 for ; Thu, 21 Aug 2008 20:36:02 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 9DAB61CC0C2; Thu, 21 Aug 2008 13:36:02 -0700 (PDT) Date: Thu, 21 Aug 2008 13:36:02 -0700 From: Jeremy Chadwick To: Weldon S Godfrey 3 Message-ID: <20080821203602.GA22354@eos.sc1.parodius.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> <20080821194742.GA19362@eos.sc1.parodius.com> <20080821155118.C76650@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080821155118.C76650@emmett.excelsus.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 20:36:02 -0000 On Thu, Aug 21, 2008 at 03:55:09PM -0400, Weldon S Godfrey 3 wrote: > To be hostest, I told it to reboot. Sorry, I am not familiar with the > debugger and I didn't see (but I often overlook) anything to would > initiate a savecore when I typed help. (although I could have tried go, > next, or whatever command would force it to step ahead, I didn't try > that) If you know the command, I can repeat the test tomorrow and type > the right thing. I do have cores from the crashes with 7.0. First, please do not remove the mailing list from the CC list; I've re-added it. People need to know what you've said. :-) If my memory serves me correctly, the problem is that you typed "reboot" and not "panic". I think this causes the machine to simply reboot without dumping memory contents to swap, thus savecore won't find any panic image in swap when the machine restarts. Others should be able to help you through using the kernel debugger. > If memory serves me right, sometime around 12:47pm, Jeremy Chadwick told me: > >> On Thu, Aug 21, 2008 at 03:35:04PM -0400, Weldon S Godfrey 3 wrote: >>> Looks like the bug with NFS and ZFS still exists. >>> >>> Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran >>> the benchmarks again this morning and after about an hour, it paniced >>> with the same message about page fault with nfsd. It dropped to debugger >>> on shutdown, it didn't do a savecore, dumpdev is set to AUTO. >> >> Specifically regarding the debugger/didn't run savecore/dumpdev >> statement: >> >> What exactly did you type once at the debugger prompt? It matters. >> >> There's also this, which I reported nearly a year ago: >> http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/118255 >> >> I haven't been able to reproduce my above PR on RELENG_7, but I'm >> unaware of anything that might have changed in RELENG_7 that fixes this >> problem. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 21 20:39:38 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2BDB1065678 for ; Thu, 21 Aug 2008 20:39:38 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 395B68FC0C for ; Thu, 21 Aug 2008 20:39:37 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 2844 invoked by uid 89); 21 Aug 2008 20:39:35 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 21 Aug 2008 20:39:35 -0000 Date: Thu, 21 Aug 2008 16:39:35 -0400 (EDT) From: Weldon S Godfrey 3 To: Jeremy Chadwick In-Reply-To: <20080821203602.GA22354@eos.sc1.parodius.com> Message-ID: <20080821163736.X76650@emmett.excelsus.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> <20080821194742.GA19362@eos.sc1.parodius.com> <20080821155118.C76650@emmett.excelsus.com> <20080821203602.GA22354@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2008 20:39:38 -0000 Thank would make sense, since panic dumps (I stupidly thought it already panic). I'll repeat tomorrow and do that. If anyone wants me to do anything else during this process, let me know. thanks! Weldon If memory serves me right, sometime around 1:36pm, Jeremy Chadwick told me: > On Thu, Aug 21, 2008 at 03:55:09PM -0400, Weldon S Godfrey 3 wrote: >> To be hostest, I told it to reboot. Sorry, I am not familiar with the >> debugger and I didn't see (but I often overlook) anything to would >> initiate a savecore when I typed help. (although I could have tried go, >> next, or whatever command would force it to step ahead, I didn't try >> that) If you know the command, I can repeat the test tomorrow and type >> the right thing. I do have cores from the crashes with 7.0. > > First, please do not remove the mailing list from the CC list; I've > re-added it. People need to know what you've said. :-) > > If my memory serves me correctly, the problem is that you typed "reboot" > and not "panic". I think this causes the machine to simply reboot > without dumping memory contents to swap, thus savecore won't find any > panic image in swap when the machine restarts. > > Others should be able to help you through using the kernel debugger. > From owner-freebsd-fs@FreeBSD.ORG Fri Aug 22 16:02:49 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B482F1065678 for ; Fri, 22 Aug 2008 16:02:49 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 683B38FC12 for ; Fri, 22 Aug 2008 16:02:49 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 7258 invoked by uid 89); 22 Aug 2008 16:02:47 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 22 Aug 2008 16:02:47 -0000 Date: Fri, 22 Aug 2008 12:02:47 -0400 (EDT) From: Weldon S Godfrey 3 To: Jeremy Chadwick In-Reply-To: <20080821194742.GA19362@eos.sc1.parodius.com> Message-ID: <20080822115932.M76650@emmett.excelsus.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> <20080821194742.GA19362@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, pjd@FreeBSD.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Aug 2008 16:02:49 -0000 Ok, I tried panic, it gave a page of the typical panic page that this crash generates under 7.0. I rebooted and no core, so I am missing a step. Sorry for being clueless here. Since the panic didn't reboot, I did a bt, it said process it was at process 1001 access.nfsrv and access.nfs3srv (sorry, I know that isn't quite right, I meant to write it down, it was definately something with access and nfsrv) Thanks, Weldon If memory serves me right, sometime around Yesterday, Jeremy Chadwick told me: > On Thu, Aug 21, 2008 at 03:35:04PM -0400, Weldon S Godfrey 3 wrote: >> Looks like the bug with NFS and ZFS still exists. >> >> Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran >> the benchmarks again this morning and after about an hour, it paniced >> with the same message about page fault with nfsd. It dropped to debugger >> on shutdown, it didn't do a savecore, dumpdev is set to AUTO. > > Specifically regarding the debugger/didn't run savecore/dumpdev > statement: > > What exactly did you type once at the debugger prompt? It matters. > > There's also this, which I reported nearly a year ago: > http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/118255 > > I haven't been able to reproduce my above PR on RELENG_7, but I'm > unaware of anything that might have changed in RELENG_7 that fixes this > problem. > > -- > | Jeremy Chadwick jdc at parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP: 4BD6C0CB | > > > From owner-freebsd-fs@FreeBSD.ORG Fri Aug 22 17:44:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EBA1106571C; Fri, 22 Aug 2008 17:44:11 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 6D6D68FC20; Fri, 22 Aug 2008 17:44:11 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 51E5B1CC0BF; Fri, 22 Aug 2008 10:44:11 -0700 (PDT) Date: Fri, 22 Aug 2008 10:44:11 -0700 From: Jeremy Chadwick To: Weldon S Godfrey 3 Message-ID: <20080822174411.GA89610@eos.sc1.parodius.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> <20080821194742.GA19362@eos.sc1.parodius.com> <20080822115932.M76650@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080822115932.M76650@emmett.excelsus.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org, pjd@FreeBSD.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Aug 2008 17:44:11 -0000 On Fri, Aug 22, 2008 at 12:02:47PM -0400, Weldon S Godfrey 3 wrote: > Ok, I tried panic, it gave a page of the typical panic page that this > crash generates under 7.0. I rebooted and no core, so I am missing a > step. Sorry for being clueless here. Then you're probably being bit by what's listed in the below PR. Supposedly you can do "panic", it should dump memory contents to swap, then upon rebooting go into single-user mode, "mount -a", then run savecore. A real PITA, I know, but supposedly it works. I can't help with the cause of the actual panic, however; it's outside of my skillset. > Since the panic didn't reboot, I did a bt, it said process it was at > process 1001 access.nfsrv and access.nfs3srv (sorry, I know that isn't > quite right, I meant to write it down, it was definately something with > access and nfsrv) > > Thanks, > > Weldon > > > If memory serves me right, sometime around Yesterday, Jeremy Chadwick told me: > >> On Thu, Aug 21, 2008 at 03:35:04PM -0400, Weldon S Godfrey 3 wrote: >>> Looks like the bug with NFS and ZFS still exists. >>> >>> Well, I got the lastest 8-HEAD on with the most recent ZFS patch and ran >>> the benchmarks again this morning and after about an hour, it paniced >>> with the same message about page fault with nfsd. It dropped to debugger >>> on shutdown, it didn't do a savecore, dumpdev is set to AUTO. >> >> Specifically regarding the debugger/didn't run savecore/dumpdev >> statement: >> >> What exactly did you type once at the debugger prompt? It matters. >> >> There's also this, which I reported nearly a year ago: >> http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/118255 >> >> I haven't been able to reproduce my above PR on RELENG_7, but I'm >> unaware of anything that might have changed in RELENG_7 that fixes this >> problem. >> >> -- >> | Jeremy Chadwick jdc at parodius.com | >> | Parodius Networking http://www.parodius.com/ | >> | UNIX Systems Administrator Mountain View, CA, USA | >> | Making life hard for others since 1977. PGP: 4BD6C0CB | -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Aug 22 18:29:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B879106568C for ; Fri, 22 Aug 2008 18:29:02 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 090DF8FC19 for ; Fri, 22 Aug 2008 18:29:01 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 20657 invoked by uid 89); 22 Aug 2008 18:28:59 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 22 Aug 2008 18:28:59 -0000 Date: Fri, 22 Aug 2008 14:28:57 -0400 (EDT) From: Weldon S Godfrey 3 To: Jeremy Chadwick In-Reply-To: <20080822174411.GA89610@eos.sc1.parodius.com> Message-ID: <20080822142834.J76650@emmett.excelsus.com> References: <20080806101621.H24586@emmett.excelsus.com> <20080814091337.Y94482@emmett.excelsus.com> <20080821153107.W76650@emmett.excelsus.com> <20080821194742.GA19362@eos.sc1.parodius.com> <20080822115932.M76650@emmett.excelsus.com> <20080822174411.GA89610@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Aug 2008 18:29:02 -0000 Thanks, I'll give that a try. If memory serves me right, sometime around 10:44am, Jeremy Chadwick told me: > On Fri, Aug 22, 2008 at 12:02:47PM -0400, Weldon S Godfrey 3 wrote: >> Ok, I tried panic, it gave a page of the typical panic page that this >> crash generates under 7.0. I rebooted and no core, so I am missing a >> step. Sorry for being clueless here. > > Then you're probably being bit by what's listed in the below PR. > Supposedly you can do "panic", it should dump memory contents to swap, > then upon rebooting go into single-user mode, "mount -a", then run > savecore. A real PITA, I know, but supposedly it works. > > I can't help with the cause of the actual panic, however; it's outside > of my skillset. > From owner-freebsd-fs@FreeBSD.ORG Fri Aug 22 23:28:33 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65A64106564A for ; Fri, 22 Aug 2008 23:28:33 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 4ADDB8FC19 for ; Fri, 22 Aug 2008 23:28:33 +0000 (UTC) (envelope-from randy@psg.com) Received: from ip192.186.dsl-acs2.seawa0.iinet.com ([209.20.186.192] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KWg3U-000GBQ-OE for freebsd-fs@freebsd.org; Fri, 22 Aug 2008 23:28:33 +0000 Message-ID: <48AF4BA0.5040208@psg.com> Date: Fri, 22 Aug 2008 16:28:32 -0700 From: Randy Bush User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: freebsd-fs@freebsd.org X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: zfs bringing a new drive online X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Aug 2008 23:28:33 -0000 one of the drives in my pool got funky. i put it offline 2008-07-26.16:23:44 zpool offline -t tank ad6s1 2008-07-26.16:24:28 zpool offline tank ad6s1 and then replaced it. i rebooted so the hw and driver would be happy about the drive, and then # zpool online tank ad6s1 Bringing device ad6s1 online # zpool status -x pool: tank state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Fri Aug 22 23:14:44 2008 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad4s2 ONLINE 0 0 0 ad8s2 ONLINE 0 0 0 ad6s1 UNAVAIL 0 0 0 cannot open ad10s1 ONLINE 0 0 0 errors: No known data errors smartctl seems to like the spindle # smartctl -a /dev/ad6 smartctl version 5.38 [amd64-portbld-freebsd8.0] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 family Device Model: ST3320620AS Serial Number: 6QF3RPZC Firmware Version: 3.AAE User Capacity: 320,072,933,376 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri Aug 22 23:25:37 2008 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 430) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 115) minutes. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 105 100 006 Pre-fail Always - 145017563 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 1 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 279592 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 4 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 054 054 045 Old_age Always - 46 (Lifetime Min/Max 37/46) 194 Temperature_Celsius 0x0022 046 046 000 Old_age Always - 46 (0 37 0 0) 195 Hardware_ECC_Recovered 0x001a 063 060 000 Old_age Always - 159460935 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 253 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 4 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. clearly, i am not understanding something randy From owner-freebsd-fs@FreeBSD.ORG Sat Aug 23 11:41:36 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8106106567F for ; Sat, 23 Aug 2008 11:41:36 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 673F48FC16 for ; Sat, 23 Aug 2008 11:41:36 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id A8ECC207F; Sat, 23 Aug 2008 13:41:35 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 84FB0844A8; Sat, 23 Aug 2008 13:41:35 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Lorenzo Perone References: <20080727125413.GG1345@garage.freebsd.pl> Date: Sat, 23 Aug 2008 13:41:35 +0200 In-Reply-To: (Lorenzo Perone's message of "Thu, 21 Aug 2008 14:44:00 +0200") Message-ID: <86wsi8dkls.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS patches X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Aug 2008 11:41:36 -0000 Lorenzo Perone writes: > But, here it is: any chances to see these patches on 7-STABLE > anytime... soon? They're not even in HEAD yet, and the 7.1 release cycle starts in a few days, so no. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Sat Aug 23 11:45:44 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A85B1065671 for ; Sat, 23 Aug 2008 11:45:44 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 5B6738FC34 for ; Sat, 23 Aug 2008 11:45:44 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 9E2B12090; Sat, 23 Aug 2008 13:45:43 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 7D155844A8; Sat, 23 Aug 2008 13:45:43 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Randy Bush References: <48AF4BA0.5040208@psg.com> Date: Sat, 23 Aug 2008 13:45:43 +0200 In-Reply-To: <48AF4BA0.5040208@psg.com> (Randy Bush's message of "Fri, 22 Aug 2008 16:28:32 -0700") Message-ID: <86skswdkew.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: zfs bringing a new drive online X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Aug 2008 11:45:44 -0000 Randy Bush writes: > one of the drives in my pool got funky. i put it offline > > 2008-07-26.16:23:44 zpool offline -t tank ad6s1 > 2008-07-26.16:24:28 zpool offline tank ad6s1 > > and then replaced it. i rebooted so the hw and driver would be happy > about the drive, and then > > # zpool online tank ad6s1 > Bringing device ad6s1 online The correct command is 'zpool replace tank ad6s1', as explained in the fine manual. BTW, it is generally a bad idea to feed ZFS slices instead of whole disks. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Sat Aug 23 15:36:17 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 823A5106568D for ; Sat, 23 Aug 2008 15:36:17 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 66BAC8FC40 for ; Sat, 23 Aug 2008 15:36:17 +0000 (UTC) (envelope-from randy@psg.com) Received: from ip192.186.dsl-acs2.seawa0.iinet.com ([209.20.186.192] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KWvA0-000IZY-Kc; Sat, 23 Aug 2008 15:36:16 +0000 Message-ID: <48B02E6F.6070008@psg.com> Date: Sat, 23 Aug 2008 08:36:15 -0700 From: Randy Bush User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <48AF4BA0.5040208@psg.com> <86skswdkew.fsf@ds4.des.no> In-Reply-To: <86skswdkew.fsf@ds4.des.no> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs bringing a new drive online X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Aug 2008 15:36:17 -0000 Dag-Erling Smørgrav wrote: > Randy Bush writes: >> one of the drives in my pool got funky. i put it offline >> >> 2008-07-26.16:23:44 zpool offline -t tank ad6s1 >> 2008-07-26.16:24:28 zpool offline tank ad6s1 >> >> and then replaced it. i rebooted so the hw and driver would be happy >> about the drive, and then >> >> # zpool online tank ad6s1 >> Bringing device ad6s1 online > > The correct command is 'zpool replace tank ad6s1', as explained in the > fine manual. thanks. read and reread man pages, not wikis. > BTW, it is generally a bad idea to feed ZFS slices instead of whole > disks. i have four drives, o two with gmirrored boot/root slices and the second slice for zfs o two zpooled disks so slicing was the mode of the day. maybe i need to rethink this? randy