From owner-freebsd-fs@FreeBSD.ORG Sun Sep 28 09:00:52 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C3B66BA for ; Sun, 28 Sep 2014 09:00:52 +0000 (UTC) Received: from mail-yk0-x236.google.com (mail-yk0-x236.google.com [IPv6:2607:f8b0:4002:c07::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D46407D8 for ; Sun, 28 Sep 2014 09:00:51 +0000 (UTC) Received: by mail-yk0-f182.google.com with SMTP id 20so4877945yks.41 for ; Sun, 28 Sep 2014 02:00:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=U7sRVNkSVLEH9UPz1H9G8beLGoZodiUYTuBEaO1rkMU=; b=S+NpQ+6An9fnT/oXHN3XDZVwuRln2JBX3zUGoPcFLw2Se2ruUAvvzF5n2IYCN5YD9M m/K6dYo3QyCDsZREQBvaPc4nXRBsPRdQxKgkivnYMzrP4HaiDZHhKyCghEa4RGy6drvc 5XCdHA8AV9PhfNTIte+BS6TSF2gAjuPw8PPUk66yEKlff1aEoNfWt4AIiF/JIo6F9iSd XDUFhZvGrAvs1DxcYW+Ckdt9ZV6sH0/TWaaG7kTw++w1cVtnWXQoK+KTNcqX+vRfWYrx Lf4XcHAxBp+Kgp8u5LpCrExCjjE4sUJAUTXamtxARvb8TBEAXDb8Q8Tj6olB7A3cggSE LEBw== MIME-Version: 1.0 X-Received: by 10.236.149.79 with SMTP id w55mr3189557yhj.107.1411894850949; Sun, 28 Sep 2014 02:00:50 -0700 (PDT) Received: by 10.170.150.85 with HTTP; Sun, 28 Sep 2014 02:00:50 -0700 (PDT) In-Reply-To: References: Date: Sun, 28 Sep 2014 05:00:50 -0400 Message-ID: Subject: Re: Unexpected zfs ARC behavior From: FF To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2014 09:00:52 -0000 I'm not sure, but this may be addressed by : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191510. On the same system, on a brand new SSD (Intel 320 series, supports TRIM, 973 hours of total run time and no SMARTCTL errors), the l2arc is accruing some errors and I can't tell if they are inter-related or not. sysctl -a |grep l2_ kstat.zfs.misc.arcstats.evict_l2_cached: 8717673244160 kstat.zfs.misc.arcstats.evict_l2_eligible: 2307662193664 kstat.zfs.misc.arcstats.evict_l2_ineligible: 3432677600768 kstat.zfs.misc.arcstats.l2_hits: 11990587 kstat.zfs.misc.arcstats.l2_misses: 66503164 kstat.zfs.misc.arcstats.l2_feeds: 3721611 kstat.zfs.misc.arcstats.l2_rw_clash: 153 kstat.zfs.misc.arcstats.l2_read_bytes: 1152794672128 kstat.zfs.misc.arcstats.l2_write_bytes: 27096209368064 kstat.zfs.misc.arcstats.l2_writes_sent: 1741090 kstat.zfs.misc.arcstats.l2_writes_done: 1741090 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 2479 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 1149 kstat.zfs.misc.arcstats.l2_evict_reading: 137 kstat.zfs.misc.arcstats.l2_free_on_write: 1000507 kstat.zfs.misc.arcstats.l2_abort_lowmem: 225 kstat.zfs.misc.arcstats.l2_cksum_bad: 8 kstat.zfs.misc.arcstats.l2_io_error: 6 kstat.zfs.misc.arcstats.l2_size: 35261600768 kstat.zfs.misc.arcstats.l2_asize: 33760987648 kstat.zfs.misc.arcstats.l2_hdr_size: 58288928 kstat.zfs.misc.arcstats.l2_compress_successes: 31952645 kstat.zfs.misc.arcstats.l2_compress_zeros: 903 kstat.zfs.misc.arcstats.l2_compress_failures: 436119 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 2481253 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 61954692 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 26664326 kstat.zfs.misc.arcstats.l2_write_in_l2: 19967700467 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 94722 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 11359575294 kstat.zfs.misc.arcstats.l2_write_full: 730445 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 3721611 kstat.zfs.misc.arcstats.l2_write_pios: 1741090 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 3464594001047552 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 229359623 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 65007815 Anyway, thanks in advance for any help offered. Best. On Sat, Sep 27, 2014 at 2:24 PM, FF wrote: > > Hi, forwarding from -questions. > > It looks like the image didn't make it across, so here it is: > > http://snag.gy/Ghb7X.jpg > > Thanks in advance for any pointers or suggestions, or confirmations that > somehow this behavior is normal. > > -- > > > So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with > the load steadily increasing over the month (migrated from other > servers)... the ARC size has unexpectedly dropped (please see attached > graphic, if it makes through the mailing list server). The system peaks > around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a > very low hit rate and prefetch has been turned off to increase the L2ARC > efficiency... but none of that should really matter as far as I can tell > since the L1ARC should try to use all the memory it can. > >> >> Some tuning of zfs sysctls (mostly write_boost and write_max) to increase >> them, but this has since been backed off to default. >> >> vmstat -m reports that 24G is dedicated to opensolaris: >> solaris 1113433 24565820K - 6185559839 >> 16,32,64,128,256,512,1024,2048,4096 >> >> And top reports memory free: >> >> last pid: 32075; load averages: 0.16, 0.13, >> 0.14 up 36+21:44:59 09:16:49 >> 24 processes: 1 running, 23 sleeping >> CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle >> Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M >> Free >> ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other >> >> zfs-stats -a : >> >> ------------------------------------------------------------------------ >> ZFS Subsystem Report Sat Sep 27 09:17:07 2014 >> ------------------------------------------------------------------------ >> >> System Information: >> >> Kernel Version: 902001 (osreldate) >> Hardware Platform: amd64 >> Processor Architecture: amd64 >> >> ZFS Storage pool Version: 5000 >> ZFS Filesystem Version: 5 >> >> FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root >> 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 >> >> ------------------------------------------------------------------------ >> >> System Memory: >> >> 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact >> 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache >> 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap >> >> Real Installed: 32.00 GiB >> Real Available: 99.63% 31.88 GiB >> Real Managed: 97.01% 30.93 GiB >> >> Logical Total: 32.00 GiB >> Logical Used: 83.03% 26.57 GiB >> Logical Free: 16.97% 5.43 GiB >> >> Kernel Memory: 23.54 GiB >> Data: 99.90% 23.52 GiB >> Text: 0.10% 23.13 MiB >> >> Kernel Memory Map: 29.76 GiB >> Size: 76.40% 22.74 GiB >> Free: 23.60% 7.02 GiB >> >> ------------------------------------------------------------------------ >> >> ARC Summary: (HEALTHY) >> Memory Throttle Count: 0 >> >> ARC Misc: >> Deleted: 90.09m >> Recycle Misses: 2.44m >> Mutex Misses: 794.67k >> Evict Skips: 17.90m >> >> ARC Size: 44.78% 13.40 GiB >> Target Size: (Adaptive) 44.78% 13.40 GiB >> Min Size (Hard Limit): 12.50% 3.74 GiB >> Max Size (High Water): 8:1 29.93 GiB >> >> ARC Size Breakdown: >> Recently Used Cache Size: 86.16% 11.55 GiB >> Frequently Used Cache Size: 13.84% 1.85 GiB >> >> ARC Hash Breakdown: >> Elements Max: 786.71k >> Elements Current: 87.05% 684.85k >> Collisions: 153.35m >> Chain Max: 16 >> Chains: 194.92k >> >> ------------------------------------------------------------------------ >> >> ARC Efficiency: 506.24m >> Cache Hit Ratio: 87.56% 443.25m >> Cache Miss Ratio: 12.44% 62.99m >> Actual Hit Ratio: 80.06% 405.29m >> >> Data Demand Efficiency: 93.92% 372.74m >> Data Prefetch Efficiency: 49.49% 69.76m >> >> CACHE HITS BY CACHE LIST: >> Anonymously Used: 6.10% 27.05m >> Most Recently Used: 31.37% 139.05m >> Most Frequently Used: 60.07% 266.24m >> Most Recently Used Ghost: 0.80% 3.56m >> Most Frequently Used Ghost: 1.66% 7.35m >> >> CACHE HITS BY DATA TYPE: >> Demand Data: 78.98% 350.09m >> Prefetch Data: 7.79% 34.53m >> Demand Metadata: 12.15% 53.86m >> Prefetch Metadata: 1.08% 4.77m >> >> CACHE MISSES BY DATA TYPE: >> Demand Data: 35.95% 22.65m >> Prefetch Data: 55.93% 35.23m >> Demand Metadata: 4.49% 2.83m >> Prefetch Metadata: 3.63% 2.29m >> >> ------------------------------------------------------------------------ >> >> >> Any suggestions? Is this expected or acceptable behavior? >> >> Thanks in advance, >> >> -- >> FF >> > > > > -- > FF > -- FF From owner-freebsd-fs@FreeBSD.ORG Sun Sep 28 14:26:36 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15F06A20 for ; Sun, 28 Sep 2014 14:26:36 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 980D685F for ; Sun, 28 Sep 2014 14:26:35 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 27CE720E7088F; Sun, 28 Sep 2014 14:26:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id E32DF20E7088B; Sun, 28 Sep 2014 14:26:29 +0000 (UTC) Message-ID: <5B80B5A2CE304E5BB2BCD3BE5735D0D1@multiplay.co.uk> From: "Steven Hartland" To: "FF" , References: Subject: Re: Unexpected zfs ARC behavior Date: Sun, 28 Sep 2014 15:26:21 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2014 14:26:36 -0000 I wouldn't expect that to be related tbh. ----- Original Message ----- From: "FF" To: Sent: Sunday, September 28, 2014 10:00 AM Subject: Re: Unexpected zfs ARC behavior > I'm not sure, but this may be addressed by : > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191510. > > On the same system, on a brand new SSD (Intel 320 series, supports TRIM, > 973 hours of total run time and no SMARTCTL errors), the l2arc is accruing > some errors and I can't tell if they are inter-related or not. > > sysctl -a |grep l2_ > kstat.zfs.misc.arcstats.evict_l2_cached: 8717673244160 > kstat.zfs.misc.arcstats.evict_l2_eligible: 2307662193664 > kstat.zfs.misc.arcstats.evict_l2_ineligible: 3432677600768 > kstat.zfs.misc.arcstats.l2_hits: 11990587 > kstat.zfs.misc.arcstats.l2_misses: 66503164 > kstat.zfs.misc.arcstats.l2_feeds: 3721611 > kstat.zfs.misc.arcstats.l2_rw_clash: 153 > kstat.zfs.misc.arcstats.l2_read_bytes: 1152794672128 > kstat.zfs.misc.arcstats.l2_write_bytes: 27096209368064 > kstat.zfs.misc.arcstats.l2_writes_sent: 1741090 > kstat.zfs.misc.arcstats.l2_writes_done: 1741090 > kstat.zfs.misc.arcstats.l2_writes_error: 0 > kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 2479 > kstat.zfs.misc.arcstats.l2_evict_lock_retry: 1149 > kstat.zfs.misc.arcstats.l2_evict_reading: 137 > kstat.zfs.misc.arcstats.l2_free_on_write: 1000507 > kstat.zfs.misc.arcstats.l2_abort_lowmem: 225 > kstat.zfs.misc.arcstats.l2_cksum_bad: 8 > kstat.zfs.misc.arcstats.l2_io_error: 6 > kstat.zfs.misc.arcstats.l2_size: 35261600768 > kstat.zfs.misc.arcstats.l2_asize: 33760987648 > kstat.zfs.misc.arcstats.l2_hdr_size: 58288928 > kstat.zfs.misc.arcstats.l2_compress_successes: 31952645 > kstat.zfs.misc.arcstats.l2_compress_zeros: 903 > kstat.zfs.misc.arcstats.l2_compress_failures: 436119 > kstat.zfs.misc.arcstats.l2_write_trylock_fail: 2481253 > kstat.zfs.misc.arcstats.l2_write_passed_headroom: 61954692 > kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 26664326 > kstat.zfs.misc.arcstats.l2_write_in_l2: 19967700467 > kstat.zfs.misc.arcstats.l2_write_io_in_progress: 94722 > kstat.zfs.misc.arcstats.l2_write_not_cacheable: 11359575294 > kstat.zfs.misc.arcstats.l2_write_full: 730445 > kstat.zfs.misc.arcstats.l2_write_buffer_iter: 3721611 > kstat.zfs.misc.arcstats.l2_write_pios: 1741090 > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 3464594001047552 > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 229359623 > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 65007815 > > > Anyway, thanks in advance for any help offered. > > Best. > > > On Sat, Sep 27, 2014 at 2:24 PM, FF wrote: > >> >> Hi, forwarding from -questions. >> >> It looks like the image didn't make it across, so here it is: >> >> http://snag.gy/Ghb7X.jpg >> >> Thanks in advance for any pointers or suggestions, or confirmations that >> somehow this behavior is normal. >> >> -- >> >> >> So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with >> the load steadily increasing over the month (migrated from other >> servers)... the ARC size has unexpectedly dropped (please see attached >> graphic, if it makes through the mailing list server). The system peaks >> around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a >> very low hit rate and prefetch has been turned off to increase the L2ARC >> efficiency... but none of that should really matter as far as I can tell >> since the L1ARC should try to use all the memory it can. >> >>> >>> Some tuning of zfs sysctls (mostly write_boost and write_max) to increase >>> them, but this has since been backed off to default. >>> >>> vmstat -m reports that 24G is dedicated to opensolaris: >>> solaris 1113433 24565820K - 6185559839 >>> 16,32,64,128,256,512,1024,2048,4096 >>> >>> And top reports memory free: >>> >>> last pid: 32075; load averages: 0.16, 0.13, >>> 0.14 up 36+21:44:59 09:16:49 >>> 24 processes: 1 running, 23 sleeping >>> CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle >>> Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M >>> Free >>> ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other >>> >>> zfs-stats -a : >>> >>> ------------------------------------------------------------------------ >>> ZFS Subsystem Report Sat Sep 27 09:17:07 2014 >>> ------------------------------------------------------------------------ >>> >>> System Information: >>> >>> Kernel Version: 902001 (osreldate) >>> Hardware Platform: amd64 >>> Processor Architecture: amd64 >>> >>> ZFS Storage pool Version: 5000 >>> ZFS Filesystem Version: 5 >>> >>> FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root >>> 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 >>> >>> ------------------------------------------------------------------------ >>> >>> System Memory: >>> >>> 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact >>> 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache >>> 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap >>> >>> Real Installed: 32.00 GiB >>> Real Available: 99.63% 31.88 GiB >>> Real Managed: 97.01% 30.93 GiB >>> >>> Logical Total: 32.00 GiB >>> Logical Used: 83.03% 26.57 GiB >>> Logical Free: 16.97% 5.43 GiB >>> >>> Kernel Memory: 23.54 GiB >>> Data: 99.90% 23.52 GiB >>> Text: 0.10% 23.13 MiB >>> >>> Kernel Memory Map: 29.76 GiB >>> Size: 76.40% 22.74 GiB >>> Free: 23.60% 7.02 GiB >>> >>> ------------------------------------------------------------------------ >>> >>> ARC Summary: (HEALTHY) >>> Memory Throttle Count: 0 >>> >>> ARC Misc: >>> Deleted: 90.09m >>> Recycle Misses: 2.44m >>> Mutex Misses: 794.67k >>> Evict Skips: 17.90m >>> >>> ARC Size: 44.78% 13.40 GiB >>> Target Size: (Adaptive) 44.78% 13.40 GiB >>> Min Size (Hard Limit): 12.50% 3.74 GiB >>> Max Size (High Water): 8:1 29.93 GiB >>> >>> ARC Size Breakdown: >>> Recently Used Cache Size: 86.16% 11.55 GiB >>> Frequently Used Cache Size: 13.84% 1.85 GiB >>> >>> ARC Hash Breakdown: >>> Elements Max: 786.71k >>> Elements Current: 87.05% 684.85k >>> Collisions: 153.35m >>> Chain Max: 16 >>> Chains: 194.92k >>> >>> ------------------------------------------------------------------------ >>> >>> ARC Efficiency: 506.24m >>> Cache Hit Ratio: 87.56% 443.25m >>> Cache Miss Ratio: 12.44% 62.99m >>> Actual Hit Ratio: 80.06% 405.29m >>> >>> Data Demand Efficiency: 93.92% 372.74m >>> Data Prefetch Efficiency: 49.49% 69.76m >>> >>> CACHE HITS BY CACHE LIST: >>> Anonymously Used: 6.10% 27.05m >>> Most Recently Used: 31.37% 139.05m >>> Most Frequently Used: 60.07% 266.24m >>> Most Recently Used Ghost: 0.80% 3.56m >>> Most Frequently Used Ghost: 1.66% 7.35m >>> >>> CACHE HITS BY DATA TYPE: >>> Demand Data: 78.98% 350.09m >>> Prefetch Data: 7.79% 34.53m >>> Demand Metadata: 12.15% 53.86m >>> Prefetch Metadata: 1.08% 4.77m >>> >>> CACHE MISSES BY DATA TYPE: >>> Demand Data: 35.95% 22.65m >>> Prefetch Data: 55.93% 35.23m >>> Demand Metadata: 4.49% 2.83m >>> Prefetch Metadata: 3.63% 2.29m >>> >>> ------------------------------------------------------------------------ >>> >>> >>> Any suggestions? Is this expected or acceptable behavior? >>> >>> Thanks in advance, >>> >>> -- >>> FF >>> >> >> >> >> -- >> FF >> > > > > -- > FF > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Sun Sep 28 18:44:16 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5197DB8C for ; Sun, 28 Sep 2014 18:44:16 +0000 (UTC) Received: from mail-yh0-x236.google.com (mail-yh0-x236.google.com [IPv6:2607:f8b0:4002:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 123FE845 for ; Sun, 28 Sep 2014 18:44:16 +0000 (UTC) Received: by mail-yh0-f54.google.com with SMTP id f10so1963830yha.41 for ; Sun, 28 Sep 2014 11:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7/FGs3OTnapw5zEYXskIqhpUxg01Z7OyooQcUKS2oYI=; b=e/oPf4hzEeyZYaQ4GrYPT6x15S96XURc2roWRgeqHdnDiENYe61q42WO3YMt8hYLcy FevqNh3AMl6ocMl/dS3JtoUG7+d8oinaG4JODEkbMjeaeOHb0X1RQdRTNwMZp1sWkHi1 6jcS+k/Uel5+tSJTn6YMXUa1l8J8M8RxW9dCpe4CRKycreitAH9iR8ykV1aL/9A6ExB8 3ElX4GXiz3AMXJqebzgmoFarjEGK3JwweWzckQLyfj/LLKtsEQbm7Aucpkzn6MTxwctS TWb95suRljnSOSrVfX+U7pXep/a47nBi96G+g++h06A6UwTzUQPacTNfvn9DI5TcifO3 wkWQ== MIME-Version: 1.0 X-Received: by 10.236.160.100 with SMTP id t64mr47369466yhk.7.1411929855130; Sun, 28 Sep 2014 11:44:15 -0700 (PDT) Received: by 10.170.150.213 with HTTP; Sun, 28 Sep 2014 11:44:15 -0700 (PDT) In-Reply-To: <5B80B5A2CE304E5BB2BCD3BE5735D0D1@multiplay.co.uk> References: <5B80B5A2CE304E5BB2BCD3BE5735D0D1@multiplay.co.uk> Date: Sun, 28 Sep 2014 14:44:15 -0400 Message-ID: Subject: Re: Unexpected zfs ARC behavior From: FF To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2014 18:44:16 -0000 Ok. Thanks! I guess we'll test the patch for the ARC sizing on another machine and see how it goes. I turned of LZ4 compression on the file store thinking that might have some impact on the L2ARC errors I was accruing. In about 12 hours I've only added 1 each to chksum and io error and they seem to correspond to a time when the L2ARC was 100%. Best. On Sun, Sep 28, 2014 at 10:26 AM, Steven Hartland wrote: > I wouldn't expect that to be related tbh. > > ----- Original Message ----- From: "FF" > To: > Sent: Sunday, September 28, 2014 10:00 AM > Subject: Re: Unexpected zfs ARC behavior > > > I'm not sure, but this may be addressed by : >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191510. >> >> On the same system, on a brand new SSD (Intel 320 series, supports TRIM, >> 973 hours of total run time and no SMARTCTL errors), the l2arc is accruing >> some errors and I can't tell if they are inter-related or not. >> >> sysctl -a |grep l2_ >> kstat.zfs.misc.arcstats.evict_l2_cached: 8717673244160 >> kstat.zfs.misc.arcstats.evict_l2_eligible: 2307662193664 >> kstat.zfs.misc.arcstats.evict_l2_ineligible: 3432677600768 >> kstat.zfs.misc.arcstats.l2_hits: 11990587 >> kstat.zfs.misc.arcstats.l2_misses: 66503164 >> kstat.zfs.misc.arcstats.l2_feeds: 3721611 >> kstat.zfs.misc.arcstats.l2_rw_clash: 153 >> kstat.zfs.misc.arcstats.l2_read_bytes: 1152794672128 >> kstat.zfs.misc.arcstats.l2_write_bytes: 27096209368064 >> kstat.zfs.misc.arcstats.l2_writes_sent: 1741090 >> kstat.zfs.misc.arcstats.l2_writes_done: 1741090 >> kstat.zfs.misc.arcstats.l2_writes_error: 0 >> kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 2479 >> kstat.zfs.misc.arcstats.l2_evict_lock_retry: 1149 >> kstat.zfs.misc.arcstats.l2_evict_reading: 137 >> kstat.zfs.misc.arcstats.l2_free_on_write: 1000507 >> kstat.zfs.misc.arcstats.l2_abort_lowmem: 225 >> kstat.zfs.misc.arcstats.l2_cksum_bad: 8 >> kstat.zfs.misc.arcstats.l2_io_error: 6 >> kstat.zfs.misc.arcstats.l2_size: 35261600768 >> kstat.zfs.misc.arcstats.l2_asize: 33760987648 >> kstat.zfs.misc.arcstats.l2_hdr_size: 58288928 >> kstat.zfs.misc.arcstats.l2_compress_successes: 31952645 >> kstat.zfs.misc.arcstats.l2_compress_zeros: 903 >> kstat.zfs.misc.arcstats.l2_compress_failures: 436119 >> kstat.zfs.misc.arcstats.l2_write_trylock_fail: 2481253 >> kstat.zfs.misc.arcstats.l2_write_passed_headroom: 61954692 >> kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 26664326 >> kstat.zfs.misc.arcstats.l2_write_in_l2: 19967700467 >> kstat.zfs.misc.arcstats.l2_write_io_in_progress: 94722 >> kstat.zfs.misc.arcstats.l2_write_not_cacheable: 11359575294 >> kstat.zfs.misc.arcstats.l2_write_full: 730445 >> kstat.zfs.misc.arcstats.l2_write_buffer_iter: 3721611 >> kstat.zfs.misc.arcstats.l2_write_pios: 1741090 >> kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 3464594001047552 >> kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 229359623 >> kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 65007815 >> >> >> Anyway, thanks in advance for any help offered. >> >> Best. >> >> >> On Sat, Sep 27, 2014 at 2:24 PM, FF wrote: >> >> >>> Hi, forwarding from -questions. >>> >>> It looks like the image didn't make it across, so here it is: >>> >>> http://snag.gy/Ghb7X.jpg >>> >>> Thanks in advance for any pointers or suggestions, or confirmations that >>> somehow this behavior is normal. >>> >>> -- >>> >>> >>> So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with >>> the load steadily increasing over the month (migrated from other >>> servers)... the ARC size has unexpectedly dropped (please see attached >>> graphic, if it makes through the mailing list server). The system peaks >>> around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has >>> a >>> very low hit rate and prefetch has been turned off to increase the L2ARC >>> efficiency... but none of that should really matter as far as I can tell >>> since the L1ARC should try to use all the memory it can. >>> >>> >>>> Some tuning of zfs sysctls (mostly write_boost and write_max) to >>>> increase >>>> them, but this has since been backed off to default. >>>> >>>> vmstat -m reports that 24G is dedicated to opensolaris: >>>> solaris 1113433 24565820K - 6185559839 >>>> 16,32,64,128,256,512,1024,2048,4096 >>>> >>>> And top reports memory free: >>>> >>>> last pid: 32075; load averages: 0.16, 0.13, >>>> 0.14 up 36+21:44:59 09:16:49 >>>> 24 processes: 1 running, 23 sleeping >>>> CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle >>>> Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M >>>> Free >>>> ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other >>>> >>>> zfs-stats -a : >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> ZFS Subsystem Report Sat Sep 27 09:17:07 2014 >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> System Information: >>>> >>>> Kernel Version: 902001 (osreldate) >>>> Hardware Platform: amd64 >>>> Processor Architecture: amd64 >>>> >>>> ZFS Storage pool Version: 5000 >>>> ZFS Filesystem Version: 5 >>>> >>>> FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root >>>> 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> System Memory: >>>> >>>> 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact >>>> 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache >>>> 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap >>>> >>>> Real Installed: 32.00 GiB >>>> Real Available: 99.63% 31.88 GiB >>>> Real Managed: 97.01% 30.93 GiB >>>> >>>> Logical Total: 32.00 GiB >>>> Logical Used: 83.03% 26.57 GiB >>>> Logical Free: 16.97% 5.43 GiB >>>> >>>> Kernel Memory: 23.54 GiB >>>> Data: 99.90% 23.52 GiB >>>> Text: 0.10% 23.13 MiB >>>> >>>> Kernel Memory Map: 29.76 GiB >>>> Size: 76.40% 22.74 GiB >>>> Free: 23.60% 7.02 GiB >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> ARC Summary: (HEALTHY) >>>> Memory Throttle Count: 0 >>>> >>>> ARC Misc: >>>> Deleted: 90.09m >>>> Recycle Misses: 2.44m >>>> Mutex Misses: 794.67k >>>> Evict Skips: 17.90m >>>> >>>> ARC Size: 44.78% 13.40 GiB >>>> Target Size: (Adaptive) 44.78% 13.40 GiB >>>> Min Size (Hard Limit): 12.50% 3.74 GiB >>>> Max Size (High Water): 8:1 29.93 GiB >>>> >>>> ARC Size Breakdown: >>>> Recently Used Cache Size: 86.16% 11.55 GiB >>>> Frequently Used Cache Size: 13.84% 1.85 GiB >>>> >>>> ARC Hash Breakdown: >>>> Elements Max: 786.71k >>>> Elements Current: 87.05% 684.85k >>>> Collisions: 153.35m >>>> Chain Max: 16 >>>> Chains: 194.92k >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> ARC Efficiency: 506.24m >>>> Cache Hit Ratio: 87.56% 443.25m >>>> Cache Miss Ratio: 12.44% 62.99m >>>> Actual Hit Ratio: 80.06% 405.29m >>>> >>>> Data Demand Efficiency: 93.92% 372.74m >>>> Data Prefetch Efficiency: 49.49% 69.76m >>>> >>>> CACHE HITS BY CACHE LIST: >>>> Anonymously Used: 6.10% 27.05m >>>> Most Recently Used: 31.37% 139.05m >>>> Most Frequently Used: 60.07% 266.24m >>>> Most Recently Used Ghost: 0.80% 3.56m >>>> Most Frequently Used Ghost: 1.66% 7.35m >>>> >>>> CACHE HITS BY DATA TYPE: >>>> Demand Data: 78.98% 350.09m >>>> Prefetch Data: 7.79% 34.53m >>>> Demand Metadata: 12.15% 53.86m >>>> Prefetch Metadata: 1.08% 4.77m >>>> >>>> CACHE MISSES BY DATA TYPE: >>>> Demand Data: 35.95% 22.65m >>>> Prefetch Data: 55.93% 35.23m >>>> Demand Metadata: 4.49% 2.83m >>>> Prefetch Metadata: 3.63% 2.29m >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> >>>> Any suggestions? Is this expected or acceptable behavior? >>>> >>>> Thanks in advance, >>>> >>>> -- >>>> FF >>>> >>>> >>> >>> >>> -- >>> FF >>> >>> >> >> >> -- >> FF >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> -- FF From owner-freebsd-fs@FreeBSD.ORG Sun Sep 28 21:13:32 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 714CCEA2 for ; Sun, 28 Sep 2014 21:13:32 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4934D90C for ; Sun, 28 Sep 2014 21:13:32 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8SLDWsC053572 for ; Sun, 28 Sep 2014 21:13:32 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201409282113.s8SLDWsC053572@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 28 Sep 2014 21:13:32 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2014 21:13:32 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ----------------+-----------+------------------------------------------------- Needs MFC | 133174 | [msdosfs] [patch] msdosfs must support multibyt Needs MFC | 136470 | [nfs] Cannot mount / in read-only, over NFS Needs MFC | 139651 | [nfs] mount(8): read-only remount of NFS volume Needs MFC | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non 4 problems total for which you should take action. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 29 08:00:12 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA1A5304 for ; Mon, 29 Sep 2014 08:00:12 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ACC16F8A for ; Mon, 29 Sep 2014 08:00:12 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8T80CtK040658 for ; Mon, 29 Sep 2014 08:00:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Message-Id: <201409290800.s8T80CtK040658@kenobi.freebsd.org> From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [FreeBSD Bugzilla] Commit Needs MFC MIME-Version: 1.0 X-Bugzilla-Type: whine X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated Date: Mon, 29 Sep 2014 08:00:12 +0000 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2014 08:00:12 -0000 Hi, You have a bug in the "Needs MFC" state which has not been touched in 7 or more days. This email serves as a reminder that you may want to MFC this bug or marked it as completed. In the event you have a longer MFC timeout you may update this bug with a comment and I won't remind you again for 7 days. This reminder is only sent on Mondays. Please file a bug about concerns you may have. This search was scheduled by eadler@FreeBSD.org. (4 bugs) Bug 133174: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=133174 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [msdosfs] [patch] msdosfs must support multibyte international characters in file names Bug 136470: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=136470 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] Cannot mount / in read-only, over NFS Bug 139651: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=139651 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] mount(8): read-only remount of NFS volume does not work Bug 144447: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=144447 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [zfs] sharenfs fsunshare() & fsshare_main() non functional From owner-freebsd-fs@FreeBSD.ORG Mon Sep 29 22:46:45 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 45A9CF3E for ; Mon, 29 Sep 2014 22:46:45 +0000 (UTC) Received: from mail.pingpong.net (mail2.pingpong.net [79.136.116.202]) by mx1.freebsd.org (Postfix) with ESMTP id 07D23278 for ; Mon, 29 Sep 2014 22:46:44 +0000 (UTC) Received: from [10.0.1.45] (h-85-24-195-48.na.cust.bahnhof.se [85.24.195.48]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.pingpong.net (Postfix) with ESMTPSA id 8177F2F6D for ; Tue, 30 Sep 2014 00:46:43 +0200 (CEST) From: Palle Girgensohn Content-Type: multipart/signed; boundary="Apple-Mail=_576F6548-A2C2-4A50-B049-12C224CE8AF1"; protocol="application/pgp-signature"; micalg=pgp-sha1 Subject: [zfs] [panic] [reproducable] zfs/space_map.c: solaris assert: sm->sm_space + size <= sm->sm_size Message-Id: Date: Tue, 30 Sep 2014 00:46:42 +0200 To: freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2014 22:46:45 -0000 --Apple-Mail=_576F6548-A2C2-4A50-B049-12C224CE8AF1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I get a panic when using bonnie++ (or just mildly stressing the fs some = other way) on a zfs pool. the panic is in space_map.c as per subject. Most references to this kind of errors suggest there is something broken = with the pool itself. Question is, is there a way to fix it? Are there = any alternatives to recreating the pool and restore from backups? Any = ideas how this could have happened? And how can I get my sysadmins to = regain trust in the ZFS file system after this? See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D193875 Regards, Palle --Apple-Mail=_576F6548-A2C2-4A50-B049-12C224CE8AF1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJUKeFSAAoJEIhV+7FrxBJD3hgIAIBLO8p8CE8fD9/JUvKz+ch4 3VNU4ftkn/8TTK1SrYcMPBH6IrhuzhXVHuiYu85oHQM33YDsmF0joB/cNDZmm0th OKfIJz5rhp3NuK9AkgPuvKKRJiqjdixesjt4dwjFP/mb88RCCwNnFJWINShkjy2D 2kG4205ohZesv9WSRZZ0gTiksEi+jhYcbM927s6XlteNCQ4asr2S2IbhzoQ8YRmy L+Wz/JK35DPvy230KJpxMeHNWHKFbPyr7nNR8tHjnB7A6/9FUSlkIAPsTdgmtt+E ot3qIDIPjOIIfcmWEkbILuNSGFt4l8DPC05K2PPm7qUnZrH0Hmka7AxyP9BZ1Yk= =iTK6 -----END PGP SIGNATURE----- --Apple-Mail=_576F6548-A2C2-4A50-B049-12C224CE8AF1-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 01:41:42 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E0A2534A for ; Wed, 1 Oct 2014 01:41:42 +0000 (UTC) Received: from mail-pd0-x232.google.com (mail-pd0-x232.google.com [IPv6:2607:f8b0:400e:c02::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2771303 for ; Wed, 1 Oct 2014 01:41:42 +0000 (UTC) Received: by mail-pd0-f178.google.com with SMTP id y10so111300pdj.37 for ; Tue, 30 Sep 2014 18:41:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=djggLmVS2yJbZeGaYycqIsWs1xukPMOFWiVIgL75cks=; b=0pbkHb32wmO/c4oZJcz5GoFPvrCZsm9kQ8sxLKus4R4tny1eJ5kN7Ttg4n5xrvdBKh kqiCgtxRjBveF5JRTCFfi0lhvVUW4Cui5xcHYQTWJ981eJNkZ5hes3MiQj9HCW2QCs/c yyC1V6phaTzlf8rGNLGLbxahIzZ62CnhvDiHcnwVy5wJRO04YT33D4umLyy/FeOVmgBx Mju2pjsfuYlizsF8zb1T4l6M3m86s3bJNS2BHn6odoEOSggPqzLOHyKwhYm5gAWJSnoS Sj6qwKihipChuBAnYfa11GStjQNn9yUsF4eWEC8bI1Sktfkbu8c6KtmmdQXeD7KtT31x BClQ== X-Received: by 10.69.26.33 with SMTP id iv1mr74818547pbd.62.1412127702294; Tue, 30 Sep 2014 18:41:42 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id ks2sm16342045pdb.24.2014.09.30.18.41.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 30 Sep 2014 18:41:41 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. From: John Terrell In-Reply-To: Date: Tue, 30 Sep 2014 18:41:39 -0700 Message-Id: <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> To: Will Andrews X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 01:41:43 -0000 Sorry for the tardy response (busy at work. :)). Here=92s the command = and output (run from the SmartOS box): ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | zfs = receive -v -F zones/tank=20 The output (after running for a long time xferring data): receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 received 1.35GB stream in 15 seconds (91.9MB/sec) receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 Read from remote host nas00: Connection reset by peer cannot receive new filesystem stream: invalid backup stream Is it possible the command is simply timing out for some reason and = closing the connection? On Sep 26, 2014, at 3:04 PM, Will Andrews wrote: > What do the zfs commands print? The -v option prints some metadata = that might help diagnose the problem. >=20 > --Will.=20 >=20 > On Monday, September 22, 2014, John Terrell = wrote: > Hello all, I'm trying to replicate one of the ZFS filesystems that = lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The = transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >=20 > "cannot receive incremental stream: invalid backup stream" >=20 > The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >=20 > ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v = tank >=20 > I've also tried to use mbuffer as the transport interface (removing = SSH from the equation) and it has the same result. >=20 > Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS = implementations? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 02:39:57 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BAD24CEF for ; Wed, 1 Oct 2014 02:39:57 +0000 (UTC) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B3649D8 for ; Wed, 1 Oct 2014 02:39:57 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id fb1so273697pad.39 for ; Tue, 30 Sep 2014 19:39:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=kONf6AQg3ph40II3DBA+d5s7gp0yQIlXR9rna/+ScbY=; b=QMmnyb1yDoPfhbh7Y8m8tks5oto710s/t32Kb55SWrbyrrovRJtPqBjryP73c8e+Io xl38P51MhMG9meK+MzZi77AZ33tzDWfeMDB8n7uV/o32UNdqksrtG3e2Y0jKm/5d3/uP ewcz/x2mcr+ccCeQLcM53h4Z8I/ZaqVF4lgpG+U2i6gFlXledXuMmpZdqOvRIpUL01sm Jtr4/9qL76+C6mbONO9sXccCNV+JHpmBqv4Rx0ZtBLhIPAcIR/tMlJk2ywWnvv6+zeFI GEj6RgZ9NgCoOr2LSzu0AVFRGXwqwCtnOTAJhAHUATVgsaRAdyzhseYH4y1wZbMGOJxF FXbA== X-Received: by 10.68.228.37 with SMTP id sf5mr28250441pbc.154.1412131196972; Tue, 30 Sep 2014 19:39:56 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id b4sm16436528pdh.2.2014.09.30.19.39.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 30 Sep 2014 19:39:55 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. From: John Terrell In-Reply-To: <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> Date: Tue, 30 Sep 2014 19:39:54 -0700 Message-Id: <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> To: Will Andrews X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 02:39:57 -0000 By the way, here=92s another report that sounds similar to mine (though = on Linux): = http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway His repro doesn=92t even use ssh (appears to be completely local). On Sep 30, 2014, at 6:41 PM, John Terrell = wrote: > Sorry for the tardy response (busy at work. :)). Here=92s the = command and output (run from the SmartOS box): >=20 > ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | zfs = receive -v -F zones/tank=20 >=20 > The output (after running for a long time xferring data): >=20 > receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 > received 1.35GB stream in 15 seconds (91.9MB/sec) > receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 > Read from remote host nas00: Connection reset by peer > cannot receive new filesystem stream: invalid backup stream >=20 >=20 > Is it possible the command is simply timing out for some reason and = closing the connection? >=20 >=20 > On Sep 26, 2014, at 3:04 PM, Will Andrews wrote: >=20 >> What do the zfs commands print? The -v option prints some metadata = that might help diagnose the problem. >>=20 >> --Will.=20 >>=20 >> On Monday, September 22, 2014, John Terrell = wrote: >> Hello all, I'm trying to replicate one of the ZFS filesystems that = lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The = transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>=20 >> "cannot receive incremental stream: invalid backup stream" >>=20 >> The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >>=20 >> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v = tank >>=20 >> I've also tried to use mbuffer as the transport interface (removing = SSH from the equation) and it has the same result. >>=20 >> Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS = implementations? >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 02:51:07 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EDD3EF47 for ; Wed, 1 Oct 2014 02:51:06 +0000 (UTC) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BCA2BB7D for ; Wed, 1 Oct 2014 02:51:06 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id h15so200522igd.4 for ; Tue, 30 Sep 2014 19:51:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=HaeNxBvyvByqieSlybbebCdT0H2NgRb4u+MR5ZCGKo8=; b=g7imlcXrp1GMbBycIzG89QvyicmvGreCc8PA+li7FlnWqGrWaJ3Ec8nx43L0ZR4Foi NHzLWbDXqq03DTegl8hc5+tr6eMWsoB5x4uGwCrkgSl25SpVAwVDHtKsqbJ6x0bFCxRk Dn5kh7769UzkgAF9ve6s1Aj/mvTlRjPkhaQd1tICNE5tVPjM8nW0IrErR+fFb3HNVG+O alJtmbnmSRbssQCR4pKhHe26cZXndvuDZxNsx8M36jWRrUTXS7V6UAvCFD7zemTUraAB 3rK0aqauyWCEvUuiXA0+fOyBk46TWfJe6/kx62K6Wxo7sq2909cGo4y3DncMb6suo7J6 7/EA== MIME-Version: 1.0 X-Received: by 10.42.227.10 with SMTP id iy10mr61497848icb.3.1412131866123; Tue, 30 Sep 2014 19:51:06 -0700 (PDT) Received: by 10.64.74.195 with HTTP; Tue, 30 Sep 2014 19:51:06 -0700 (PDT) In-Reply-To: <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> Date: Tue, 30 Sep 2014 22:51:06 -0400 Message-ID: Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. From: Rich To: John Terrell Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 02:51:07 -0000 It's possible that your feature flags and the receiving end's aren't compatible - can you show us a zfs get all tank on both sides? - Rich On Tue, Sep 30, 2014 at 10:39 PM, John Terrell wrote: > By the way, here's another report that sounds similar to mine (though on Linux): > > http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway > > His repro doesn't even use ssh (appears to be completely local). > > > On Sep 30, 2014, at 6:41 PM, John Terrell wrote: > >> Sorry for the tardy response (busy at work. :)). Here's the command and output (run from the SmartOS box): >> >> ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | zfs receive -v -F zones/tank >> >> The output (after running for a long time xferring data): >> >> receiving full stream of tank@daily20140920 into zones/tank@daily20140920 >> received 1.35GB stream in 15 seconds (91.9MB/sec) >> receiving full stream of tank/photography@daily20140920 into zones/tank/photography@daily20140920 >> Read from remote host nas00: Connection reset by peer >> cannot receive new filesystem stream: invalid backup stream >> >> >> Is it possible the command is simply timing out for some reason and closing the connection? >> >> >> On Sep 26, 2014, at 3:04 PM, Will Andrews wrote: >> >>> What do the zfs commands print? The -v option prints some metadata that might help diagnose the problem. >>> >>> --Will. >>> >>> On Monday, September 22, 2014, John Terrell wrote: >>> Hello all, I'm trying to replicate one of the ZFS filesystems that lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>> >>> "cannot receive incremental stream: invalid backup stream" >>> >>> The stream being sent is not incremental so I'm not sure what the receiver is trying to do. Here's the command I'm using to transfer (executed on the SmartOS machine): >>> >>> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v tank >>> >>> I've also tried to use mbuffer as the transport interface (removing SSH from the equation) and it has the same result. >>> >>> Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS implementations? >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 03:02:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B24E3203 for ; Wed, 1 Oct 2014 03:02:10 +0000 (UTC) Received: from mail-pd0-x22e.google.com (mail-pd0-x22e.google.com [IPv6:2607:f8b0:400e:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81DA3D15 for ; Wed, 1 Oct 2014 03:02:10 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id y13so226245pdi.33 for ; Tue, 30 Sep 2014 20:02:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=ReHO2zwr6jE40e/UsGoDan4F1RxnRYVrDoxS9UrV5dU=; b=ijCdnS2eajKR/ME/63yxaRTBT5OqmML1imMzmi0eTN1kpcDcvAB8SSfxjr/KT378Bt jr5rZKP+zA3k1UbaelpXp8ZdrcyIWlVUPZJwU37g7pCJncWxFw1nPhL8U27mebROEekj 7Xo9xSYh8KsMJwE5g+nO5Tc76ztShzgaAV1W11TQ+jLexY2JSagvARBV/U9/CrglDeFc ZxuOlziLGZ/882MZ4MAtmu1DAgd7WAJNrXm1MFArZxjYjkK6xdoNEnEJhMPvHtUFPL3u 2n64vvFGPVo1pWIKa3vfmIu0hx/d9PZSVdskaCY0X3y07sc17SRo5hH08zixaOVkJi5n Lz1A== X-Received: by 10.68.194.66 with SMTP id hu2mr3158522pbc.19.1412132529997; Tue, 30 Sep 2014 20:02:09 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id n3sm7214838pda.7.2014.09.30.20.02.08 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 30 Sep 2014 20:02:09 -0700 (PDT) From: John Terrell Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Date: Tue, 30 Sep 2014 20:02:07 -0700 References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> To: "freebsd-fs@freebsd.org" In-Reply-To: X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 03:02:10 -0000 Sure. On the SmartOS (receiving) side: [root@smartos /]# zpool get all zones | grep feature@ zones feature@async_destroy enabled = local zones feature@empty_bpobj active = local zones feature@lz4_compress active = local zones feature@multi_vdev_crash_dump enabled = local zones feature@spacemap_histogram active = local zones feature@enabled_txg active = local zones feature@hole_birth active = local zones feature@extensible_dataset enabled = local zones feature@embedded_data active = local zones feature@bookmarks enabled = local zones feature@filesystem_limits enabled = local =85on the FreeBSD 10 (sending) side: [root@freebsd /]# zpool get all tank | grep feature@ tank feature@async_destroy enabled = local tank feature@empty_bpobj active = local tank feature@lz4_compress active = local tank feature@multi_vdev_crash_dump enabled = local So, the FreeBSD side is a subset of the SmartOS side. On Sep 30, 2014, at 7:51 PM, Rich wrote: > It's possible that your feature flags and the receiving end's aren't > compatible - can you show us a zfs get all tank on both sides? >=20 > - Rich >=20 > On Tue, Sep 30, 2014 at 10:39 PM, John Terrell = wrote: >> By the way, here's another report that sounds similar to mine (though = on Linux): >>=20 >> = http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway >>=20 >> His repro doesn't even use ssh (appears to be completely local). >>=20 >>=20 >> On Sep 30, 2014, at 6:41 PM, John Terrell = wrote: >>=20 >>> Sorry for the tardy response (busy at work. :)). Here's the = command and output (run from the SmartOS box): >>>=20 >>> ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | = zfs receive -v -F zones/tank >>>=20 >>> The output (after running for a long time xferring data): >>>=20 >>> receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 >>> received 1.35GB stream in 15 seconds (91.9MB/sec) >>> receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 >>> Read from remote host nas00: Connection reset by peer >>> cannot receive new filesystem stream: invalid backup stream >>>=20 >>>=20 >>> Is it possible the command is simply timing out for some reason and = closing the connection? >>>=20 >>>=20 >>> On Sep 26, 2014, at 3:04 PM, Will Andrews wrote: >>>=20 >>>> What do the zfs commands print? The -v option prints some metadata = that might help diagnose the problem. >>>>=20 >>>> --Will. >>>>=20 >>>> On Monday, September 22, 2014, John Terrell = wrote: >>>> Hello all, I'm trying to replicate one of the ZFS filesystems that = lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The = transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>>>=20 >>>> "cannot receive incremental stream: invalid backup stream" >>>>=20 >>>> The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >>>>=20 >>>> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v = tank >>>>=20 >>>> I've also tried to use mbuffer as the transport interface (removing = SSH from the equation) and it has the same result. >>>>=20 >>>> Is the possibly an incompatibility between FreeBSD10 and SmartOS = ZFS implementations? >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>=20 >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 08:55:24 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D5E6584F for ; Wed, 1 Oct 2014 08:55:24 +0000 (UTC) Received: from mailsec012.isp.belgacom.be (mailsec012.isp.belgacom.be [195.238.6.252]) by mx1.freebsd.org (Postfix) with ESMTP id 6E491FE6 for ; Wed, 1 Oct 2014 08:55:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skynet.be; i=@skynet.be; q=dns/txt; s=securemail; t=1412153724; x=1443689724; h=message-id:date:from:mime-version:to:subject: content-transfer-encoding; bh=ZgBd2wZxrnSXHqnabrpy+c/aKTEsJtOhdS1D55uxs5k=; b=ulQAa8oGmQPDQUerbx53ty6BmKbZKms9vVrYnvBbnytb82OkdWIUCs+7 tMK+t7YM/ItYhzxpZIdtUe9S9Fnt8w==; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ai1ZAEbAK1RQydAo/2dsb2JhbABggw6BDQcYgSOMLXOTK5hKl0kEBCRrFwF7hGBBAR0WCBADAgECAScNJAYCAQEXiCcBmTOlS5BDhDUBBJ0zjEWJLoNlOy+CSgEBAQ Received: from 40.208-201-80.adsl-dyn.isp.belgacom.be (HELO Acer-JFB.site) ([80.201.208.40]) by relay.skynet.be with ESMTP/TLS/DHE-RSA-AES128-SHA; 01 Oct 2014 10:54:13 +0200 Message-ID: <542BC135.1070906@Skynet.be> Date: Wed, 01 Oct 2014 10:54:13 +0200 From: JF-Bogaerts User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 To: freebsd-fs@FreeBSD.org Subject: HAST with broken HDD MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 08:55:24 -0000 Hello, I'm preparing a HA NAS solution using HAST. I'm wondering what will happen if one of disks of the primary node will fail or become erratic. Thx, Jean-François Bogaerts __________________________________________________________________ Rue d'Orp 23 4280 Wansin Belgium Mob: +32 475 45 61 38 mailto: [1]JF-Bogaerts@skynet.be __________________________________________________________________ References 1. mailto:JF-Bogaerts@skynet.be From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 09:21:35 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9217CC1 for ; Wed, 1 Oct 2014 09:21:35 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 40C083F9 for ; Wed, 1 Oct 2014 09:21:35 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3j7BmY2DXfzZB for ; Wed, 1 Oct 2014 11:21:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla2; t=1412155288; x=1414747289; bh=bFW WdnOVSJWzpDgMIjZL4aUmxvFms+C9SrM5IuETyr8=; b=fPjkzVN+cJMJYO4PsPF D9rd6aAYzvyb/P7XYDwtf70uxF6/PbfWXI6xpLLBH8CqSV9bIMWisarDtQ8BbC4R Xs2U6nOS8ukN9PIVW5FskcCgCZ/dAwAHmNpduzExRIqqzGxS2CFZVyGKHKPl59ge wKRez7EOJuUs4Hf0pp5iUVPw= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id XQIidiAgH2rR for ; Wed, 1 Oct 2014 11:21:28 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP for ; Wed, 1 Oct 2014 11:21:28 +0200 (CEST) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3j7BmS3hDyz1CR for ; Wed, 1 Oct 2014 11:21:28 +0200 (CEST) Received: from sleepy.ijs.si ([2001:1470:ff80:e001::1:1]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Wed, 01 Oct 2014 11:21:28 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 01 Oct 2014 11:21:28 +0200 From: Mark Martinec To: freebsd-fs@freebsd.org Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Organization: J. Stefan Institute In-Reply-To: References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> Message-ID: <7c3182852401d5cc2e98eea2787d6892@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.0.2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 09:21:35 -0000 2014-09-23, John Terrell wrote: >> Hello all, I'm trying to replicate one of the ZFS filesystems that >> lives >> on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. >> The transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >> >> "cannot receive incremental stream: invalid backup stream" 2014-09-30, John Terrell wrote: >> Here's the command and output (run from the SmartOS box): > > ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | zfs > receive -v -F zones/tank > > The output (after running for a long time xferring data): Try logging to the sending host and run 'zfs send' there interactively from a command line, noting any potential error messages on stdout or syslog. See also: http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html Mark From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 10:36:28 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B74449EA; Wed, 1 Oct 2014 10:36:28 +0000 (UTC) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.redhat.com", Issuer "DigiCert SHA2 Extended Validation Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89C2AD98; Wed, 1 Oct 2014 10:36:28 +0000 (UTC) Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s91AXW14024180 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 1 Oct 2014 06:33:32 -0400 Received: from [10.36.6.150] (vpn1-6-150.ams2.redhat.com [10.36.6.150]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s91AXI9O007878 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 1 Oct 2014 06:33:21 -0400 Subject: Re: FreeBSD support being added to GlusterFS Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=windows-1252 From: Justin Clift In-Reply-To: Date: Wed, 1 Oct 2014 11:33:18 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <73B63510-384B-43F3-AC10-78F429F4A5FF@gluster.org> References: <0F20AEEC-6244-42BC-815C-1440BBBDE664@mail.turbofuzz.com> <20140629203746.GI34108@ivaldir.etoilebsd.net> <1A58F492-946F-46D4-A19E-2734F368CDAC@mail.turbofuzz.com> <0ABAE2AC-BF1B-4125-ACA9-C6177D013E25@mail.turbofuzz.com> <20140706230910.GA8523@ivaldir.etoilebsd.net> <2F416D06-0A98-4E66-902C-ED0690A4B1C0@ixsystems.com> <20140825211459.GB65120@ivaldir.etoilebsd.net> To: Jordan Hubbard X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 10:36:28 -0000 On 25/08/2014, at 11:24 PM, Jordan Hubbard wrote: >> On Aug 25, 2014, at 2:14 PM, Baptiste Daroussin = wrote: >>=20 >> I do not plan to maintain the port I'm just trying to help :) I would = prefer one >> of your resident committer you has FreeNAS vendor will probably have = more >> feedbacks from glusterfs user and probably have more opportunities to = test it. >> I have no usage for glusterfs but deeply support having glusterfs is = FreeBSD :) >=20 > Fair enough! I will attempt to make those arrangements (just as soon = as everyone gets back from VMWorld and/or China :-) ). In the = meantime, here=92s a patch to bring the port up to date: >=20 > I=92ve already committed it to the FreeNAS repo and will be building a = nightly with this version of glusterfs in it shortly! As a data point, GlusterFS 3.6.0 beta3 has been released now too. :) = http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.6.0beta3.tar= .gz Everyone that has time, please test/try it out and report back any bugs. = :D Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 11:04:39 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5B7D017C for ; Wed, 1 Oct 2014 11:04:39 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) by mx1.freebsd.org (Postfix) with ESMTP id 177AD110 for ; Wed, 1 Oct 2014 11:04:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id CEC6C4C4C8ED; Wed, 1 Oct 2014 12:55:46 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oRXHmBwNsu+d; Wed, 1 Oct 2014 12:55:44 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id 6083E4C4CCA2; Wed, 1 Oct 2014 12:55:44 +0200 (CEST) Message-ID: <542BDDB3.8080805@internetx.com> Date: Wed, 01 Oct 2014 12:55:47 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: JF-Bogaerts , freebsd-fs@FreeBSD.org Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> In-Reply-To: <542BC135.1070906@Skynet.be> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 11:04:39 -0000 Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > Hello, > I'm preparing a HA NAS solution using HAST. > I'm wondering what will happen if one of disks of the primary node will > fail or become erratic. > > Thx, > Jean-François Bogaerts nothing. if you are using zfs on top of hast zfs wont even take notice about the disk failure. as long as the write operation was sucessfull on one of the 2 nodes, hast doesnt notify the ontop layers about io errors. interesting concept, took me some time to deal with this. > __________________________________________________________________ > > Rue d'Orp 23 > 4280 Wansin > Belgium > Mob: +32 475 45 61 38 > mailto: [1]JF-Bogaerts@skynet.be > __________________________________________________________________ > > References > > 1. mailto:JF-Bogaerts@skynet.be > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 12:28:35 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2F0EE1D for ; Wed, 1 Oct 2014 12:28:35 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EDD4CE7 for ; Wed, 1 Oct 2014 12:28:35 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id ho1so373864wib.16 for ; Wed, 01 Oct 2014 05:28:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7kFE+OdD1nynd/1qaHraaRtq1VLFKNCczJa7tjATB6E=; b=r21VonK6QOhFWCfi3zsgNTpHl4H+iBfNNPnyuWkwO83BSnheaGFcRFy5TsAyIa3o1R 9CvJ0X5Wtd0Klv9w5qj1CGOaWcl9s8C7ynRW8zW4iDNltUUCJICLDOIVWTsg8IyJpMyw GNesuewSZ4JgfM44B/1W5+/K0W15CuMA40qKUNM2Bnsce9op07468xa/X74k/N7NqXo8 NQSssvLi9RpJt7QDPqEzWTjhSUto1NfZSfbr8lkkYyPK8alniJ8O3Kd1fqsgo99ZGZLH 9BbexyKX2uNp47XefBZuatU+sDRNDgKwldQKReR+yzOh1BNlMtkVAn2I7HZcXxSC1qQp qIbA== MIME-Version: 1.0 X-Received: by 10.194.189.82 with SMTP id gg18mr2573323wjc.2.1412166513730; Wed, 01 Oct 2014 05:28:33 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 05:28:33 -0700 (PDT) In-Reply-To: <542BDDB3.8080805@internetx.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> Date: Wed, 1 Oct 2014 15:28:33 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: jg@internetx.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 12:28:36 -0000 On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter < jg@internetx.com> wrote: > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > Hello, > > I'm preparing a HA NAS solution using HAST. > > I'm wondering what will happen if one of disks of the primary node > will > > fail or become erratic. > > > > Thx, > > Jean-Fran=C3=A7ois Bogaerts > > nothing. if you are using zfs on top of hast zfs wont even take notice > about the disk failure. > > as long as the write operation was sucessfull on one of the 2 nodes, > hast doesnt notify the ontop layers about io errors. > > interesting concept, took me some time to deal with this. > > > Are you saying that the pool will appear to be optimal even with a bad drive? From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 12:34:38 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F1388D9 for ; Wed, 1 Oct 2014 12:34:38 +0000 (UTC) Received: from mail.iXsystems.com (mail.ixsystems.com [12.229.62.4]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D627EDC1 for ; Wed, 1 Oct 2014 12:34:38 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id 35F627F9C6; Wed, 1 Oct 2014 05:34:38 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 26226-01; Wed, 1 Oct 2014 05:34:37 -0700 (PDT) Received: from [10.8.0.38] (unknown [10.8.0.38]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 878F57F9BF; Wed, 1 Oct 2014 05:34:36 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1988\)) Subject: Re: HAST with broken HDD From: Jordan Hubbard In-Reply-To: Date: Wed, 1 Oct 2014 15:34:33 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <15E8A3A7-2CFA-43F0-B9DB-1B0DBAB5304C@mail.turbofuzz.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> To: George Kontostanos X-Mailer: Apple Mail (2.1988) Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 12:34:39 -0000 > On Oct 1, 2014, at 3:28 PM, George Kontostanos = wrote: > Are you saying that the pool will appear to be optimal even with a bad = drive? Yes. HAST means that ZFS won=E2=80=99t *see* a bad drive. It will just = continue to see =E2=80=9Ca drive=E2=80=9D even though one half of the = HAST pair has died. - Jordan From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 12:44:34 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E5EF6C7 for ; Wed, 1 Oct 2014 12:44:34 +0000 (UTC) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06BFEEFF for ; Wed, 1 Oct 2014 12:44:33 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id cc10so385948wib.0 for ; Wed, 01 Oct 2014 05:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9P8ORLK9mEaO68RMelVaiO/+BmOrVuOLaQV5sJJV23o=; b=D53es3vAkASWscHCcVeACEvNAvimpr0LC22wVxMbDbasaTDYNWLwhEisBRa4o/clGW OeWa9+lTErcjHExl75+dpWQPNZTGnIMwHTVkedQfACuIh3M6VsnfAeoiLmLYcI2WWCuE gvSXWvt47BHAr+I0POokaosse8IBTUcuGNGEEQLtOrfKtragfdYx7ZPOYCgBhc8icAO6 TxiwLWkBNhpVqy38qyHVvJVERkFRVQdqdqslU3IGRFCwqRIh5KOFSsanzOPRsddRxe6Q NMwCL7J+B31SaEGq3/aeOKAWOCxZbgJtNzHODJtMhiYvB9Hw0bY3NCGfWhRCWvRaYf6l Qx7Q== MIME-Version: 1.0 X-Received: by 10.180.97.199 with SMTP id ec7mr14152654wib.29.1412167472301; Wed, 01 Oct 2014 05:44:32 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 05:44:32 -0700 (PDT) In-Reply-To: <15E8A3A7-2CFA-43F0-B9DB-1B0DBAB5304C@mail.turbofuzz.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <15E8A3A7-2CFA-43F0-B9DB-1B0DBAB5304C@mail.turbofuzz.com> Date: Wed, 1 Oct 2014 15:44:32 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 12:44:34 -0000 On Wed, Oct 1, 2014 at 3:34 PM, Jordan Hubbard wrote: > > > On Oct 1, 2014, at 3:28 PM, George Kontostanos > wrote: > > Are you saying that the pool will appear to be optimal even with a bad > drive? > > Yes. HAST means that ZFS won=E2=80=99t *see* a bad drive. It will just = continue > to see =E2=80=9Ca drive=E2=80=9D even though one half of the HAST pair ha= s died. > > - Jordan > > > Given the fact that the pool is available only on the active server, I find it interesting that any write operation to that pool will not produce a checksum error. I am going to certainly try and create the problem on a test environment that I currently have with ZFS and HAST root@hast1:~ # zpool status pool: tank state: ONLINE scan: scrub repaired 0 in 0h1m with 0 errors on Tue Sep 30 21:55:16 2014 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 hast/disk1 ONLINE 0 0 0 hast/disk2 ONLINE 0 0 0 errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 12:48:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 142489E0 for ; Wed, 1 Oct 2014 12:48:00 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C09E7F58 for ; Wed, 1 Oct 2014 12:47:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id 7B7EF4C4CD22; Wed, 1 Oct 2014 14:47:56 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iL7QOKwttdwh; Wed, 1 Oct 2014 14:47:54 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id 110C24C4CDB2; Wed, 1 Oct 2014 14:47:54 +0200 (CEST) Message-ID: <542BF7FC.3070401@internetx.com> Date: Wed, 01 Oct 2014 14:47:56 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: George Kontostanos Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 12:48:00 -0000 Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > wrote: > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > Hello, > > I'm preparing a HA NAS solution using HAST. > > I'm wondering what will happen if one of disks of the primary node will > > fail or become erratic. > > > > Thx, > > Jean-François Bogaerts > > nothing. if you are using zfs on top of hast zfs wont even take notice > about the disk failure. > > as long as the write operation was sucessfull on one of the 2 nodes, > hast doesnt notify the ontop layers about io errors. > > interesting concept, took me some time to deal with this. > > > Are you saying that the pool will appear to be optimal even with a bad > drive? > > right, dont try to compare it with drbd the concept differs in basic things completly. check https://wiki.freebsd.org/HAST From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 12:49:25 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AEAAA7B for ; Wed, 1 Oct 2014 12:49:25 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) by mx1.freebsd.org (Postfix) with ESMTP id 34E2AF6E for ; Wed, 1 Oct 2014 12:49:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id 38A824C4CDB2; Wed, 1 Oct 2014 14:49:23 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sSa2zOi-9eVq; Wed, 1 Oct 2014 14:49:20 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id B1A904C4CD22; Wed, 1 Oct 2014 14:49:20 +0200 (CEST) Message-ID: <542BF853.3040604@internetx.com> Date: Wed, 01 Oct 2014 14:49:23 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: George Kontostanos Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 12:49:25 -0000 Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > wrote: > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > Hello, > > I'm preparing a HA NAS solution using HAST. > > I'm wondering what will happen if one of disks of the primary node will > > fail or become erratic. > > > > Thx, > > Jean-François Bogaerts > > nothing. if you are using zfs on top of hast zfs wont even take notice > about the disk failure. > > as long as the write operation was sucessfull on one of the 2 nodes, > hast doesnt notify the ontop layers about io errors. > > interesting concept, took me some time to deal with this. > > > Are you saying that the pool will appear to be optimal even with a bad > drive? > > https://forums.freebsd.org/viewtopic.php?&t=24786 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:06:43 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17D11E3D for ; Wed, 1 Oct 2014 13:06:43 +0000 (UTC) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3DE4216 for ; Wed, 1 Oct 2014 13:06:42 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id d1so531816wiv.6 for ; Wed, 01 Oct 2014 06:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=75LBMN1Hv/mum65TDy0CUAVnKyklV1wEhb9Jh2LNcIo=; b=hGd0MITVxgwkIRz4tGODGWYWJ/fc/SqDWRipmP9zLCrEINScS8ErnDT4FeoxFnswTo qUrZMqNXOrQuhMM/FVgFX6lboFgXfe3I/fz1N/pDbTfBy/Qj+qwsU5L45N+FFMxn9b1D BA1Pc5bxGLv2p5kdNd6fVLAaiysWSqZcKlKsRzw1CzfU6HqfsyW8J+6jE2goBbg9g+5Y gD/TEuGRPdXNr4cqCgEarRUM7eAwACnNB/SHlYcjuIP8gaQu56a66gStV87YxiP2TuNV 6XYyRRKWfzpEmONrTUDUIeFtwhF59HluY3kqc0ICZQET5+g53s+7iHWJCBo+K2/705UL zbLA== MIME-Version: 1.0 X-Received: by 10.194.76.97 with SMTP id j1mr60472076wjw.40.1412168800795; Wed, 01 Oct 2014 06:06:40 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 06:06:40 -0700 (PDT) In-Reply-To: <542BF853.3040604@internetx.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> Date: Wed, 1 Oct 2014 16:06:40 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: jg@internetx.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:06:43 -0000 On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinter < jg@internetx.com> wrote: > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > > wrote: > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > Hello, > > > I'm preparing a HA NAS solution using HAST. > > > I'm wondering what will happen if one of disks of the primary > node will > > > fail or become erratic. > > > > > > Thx, > > > Jean-Fran=C3=A7ois Bogaerts > > > > nothing. if you are using zfs on top of hast zfs wont even take > notice > > about the disk failure. > > > > as long as the write operation was sucessfull on one of the 2 nodes= , > > hast doesnt notify the ontop layers about io errors. > > > > interesting concept, took me some time to deal with this. > > > > > > Are you saying that the pool will appear to be optimal even with a bad > > drive? > > > > > > https://forums.freebsd.org/viewtopic.php?&t=3D24786 > It appears that this is actually the case. And it is very disturbing, meaning that a drive failure goes unnoticed. In my case I completely removed the second disk on the primary node and a zpool status showed absolutely no problem. Scrubbing the pool began resilvering which indicates that there is actually something wrong! pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 16K in 0h2m with 7 errors on Wed Oct 1 16:00:47 201= 4 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 7 mirror-0 ONLINE 0 0 40 hast/disk1 ONLINE 0 0 40 hast/disk2 ONLINE 0 0 40 Unfortunately, in this case there was data loss and hastctl status does not report the missing disk! Name Status Role Components disk1 complete primary /dev/ada1 hast2 disk2 complete primary /dev/ada2 hast2 --=20 George Kontostanos --- http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:14:55 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 14B3030A for ; Wed, 1 Oct 2014 13:14:55 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4590363 for ; Wed, 1 Oct 2014 13:14:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id C72F61472004; Wed, 1 Oct 2014 15:14:51 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 179Y5PePp39x; Wed, 1 Oct 2014 15:14:45 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id A4AA51472001; Wed, 1 Oct 2014 15:14:44 +0200 (CEST) Message-ID: <542BFE46.2020106@internetx.com> Date: Wed, 01 Oct 2014 15:14:46 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: JF-Bogaerts , freebsd-fs@FreeBSD.org Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> In-Reply-To: <542BC135.1070906@Skynet.be> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:14:55 -0000 Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > Hello, > I'm preparing a HA NAS solution using HAST. > I'm wondering what will happen if one of disks of the primary node will > fail or become erratic. > > Thx, > Jean-François Bogaerts > __________________________________________________________________ > did you consider a more "classic" setup with crosscabled jbods, this kind of HA will be much more reliable > Rue d'Orp 23 > 4280 Wansin > Belgium > Mob: +32 475 45 61 38 > mailto: [1]JF-Bogaerts@skynet.be > __________________________________________________________________ > > References > > 1. mailto:JF-Bogaerts@skynet.be > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:29:05 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10C4253F for ; Wed, 1 Oct 2014 13:29:05 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) by mx1.freebsd.org (Postfix) with ESMTP id 8D6B669A for ; Wed, 1 Oct 2014 13:29:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id 1B4561472007; Wed, 1 Oct 2014 15:29:02 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j3G2TTR76mKL; Wed, 1 Oct 2014 15:28:59 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id 7B30F4C4C9B0; Wed, 1 Oct 2014 15:28:59 +0200 (CEST) Message-ID: <542C019E.2080702@internetx.com> Date: Wed, 01 Oct 2014 15:29:02 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: George Kontostanos Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, JF-Bogaerts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:29:05 -0000 Am 01.10.2014 um 15:06 schrieb George Kontostanos: > > > On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinter > > wrote: > > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > > >> wrote: > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > Hello, > > > I'm preparing a HA NAS solution using HAST. > > > I'm wondering what will happen if one of disks of the > primary node will > > > fail or become erratic. > > > > > > Thx, > > > Jean-François Bogaerts > > > > nothing. if you are using zfs on top of hast zfs wont even > take notice > > about the disk failure. > > > > as long as the write operation was sucessfull on one of the 2 > nodes, > > hast doesnt notify the ontop layers about io errors. > > > > interesting concept, took me some time to deal with this. > > > > > > Are you saying that the pool will appear to be optimal even with a bad > > drive? > > > > > > https://forums.freebsd.org/viewtopic.php?&t=24786 > > > > It appears that this is actually the case. And it is very disturbing, > meaning that a drive failure goes unnoticed. In my case I completely > removed the second disk on the primary node and a zpool status showed > absolutely no problem. Scrubbing the pool began resilvering which > indicates that there is actually something wrong! right. lets go further and think how zfs works regarding direct hardware / disk access. theres a layer between which always says ey, everthing is fine. no more need for pool scrubbing, since hastd wont tell if anything is wrong :D > > pool: tank > > state: ONLINE > > status: One or more devices has experienced an error resulting in data > > corruption. Applications may be affected. > > action: Restore the file in question if possible. Otherwise restore the > > entire pool from backup. > > see: http://illumos.org/msg/ZFS-8000-8A > > scan: scrub repaired 16K in 0h2m with 7 errors on Wed Oct 1 16:00:47 2014 > > config: > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 7 > > mirror-0 ONLINE 0 0 40 > > hast/disk1 ONLINE 0 0 40 > > hast/disk2 ONLINE 0 0 40 > > > Unfortunately, in this case there was data loss and hastctl status does > not report the missing disk! > > NameStatusRoleComponents > > disk1complete primary /dev/ada1hast2 > > disk2complete primary /dev/ada2hast2 > > > -- > George Kontostanos > --- > http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:49:06 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38C15B29 for ; Wed, 1 Oct 2014 13:49:06 +0000 (UTC) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B40A4903 for ; Wed, 1 Oct 2014 13:49:05 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id y10so517302wgg.15 for ; Wed, 01 Oct 2014 06:49:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wpPSLOAY4SXyGk3rEet1hcIMpJu7vSh9aYL3mQpmCjs=; b=q+3GheLJoecchWVSTj/8rmScUKAQpXsk9efqax16tM9pMkcRirB2tshxYmG18qjpZp nAvBf0nEAVlk1dxX+seoe/LYvA7ehZKTrItQZ7nL5gl/PSJHshVkgdqU6bppNPTwvVwf dXYkq0twJ+VRSanyZp11T+/tzIdEudd9DwuQ0FBYVqnScGfV9w8vsW75jUN993gVsGXq yXbReVs+F9i9KiOEkQaqKfZk7oVesugps9tPgvufUAJEEeiHlMLgcY02xf9KGj5CF9Pr 4tLAWzH3PX/y4EwEyKUCoNlVfwM5G3eANCkNFzB8vWE1jNFZhu/uOKLuN+mhF2jky412 lJdg== MIME-Version: 1.0 X-Received: by 10.194.76.97 with SMTP id j1mr60843523wjw.40.1412171343916; Wed, 01 Oct 2014 06:49:03 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 06:49:03 -0700 (PDT) In-Reply-To: <542C019E.2080702@internetx.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> Date: Wed, 1 Oct 2014 16:49:03 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: jg@internetx.com, freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:49:06 -0000 On Wed, Oct 1, 2014 at 4:29 PM, InterNetX - Juergen Gotteswinter < jg@internetx.com> wrote: > Am 01.10.2014 um 15:06 schrieb George Kontostanos: > > > > > > On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinter > > > wrote: > > > > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > > > > >> wrote: > > > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > > Hello, > > > > I'm preparing a HA NAS solution using HAST. > > > > I'm wondering what will happen if one of disks of the > > primary node will > > > > fail or become erratic. > > > > > > > > Thx, > > > > Jean-Fran=C3=A7ois Bogaerts > > > > > > nothing. if you are using zfs on top of hast zfs wont even > > take notice > > > about the disk failure. > > > > > > as long as the write operation was sucessfull on one of the 2 > > nodes, > > > hast doesnt notify the ontop layers about io errors. > > > > > > interesting concept, took me some time to deal with this. > > > > > > > > > Are you saying that the pool will appear to be optimal even with = a > bad > > > drive? > > > > > > > > > > https://forums.freebsd.org/viewtopic.php?&t=3D24786 > > > > > > > > It appears that this is actually the case. And it is very disturbing, > > meaning that a drive failure goes unnoticed. In my case I completely > > removed the second disk on the primary node and a zpool status showed > > absolutely no problem. Scrubbing the pool began resilvering which > > indicates that there is actually something wrong! > > > right. lets go further and think how zfs works regarding direct hardware > / disk access. theres a layer between which always says ey, everthing is > fine. no more need for pool scrubbing, since hastd wont tell if anything > is wrong :D > > Correct, ZFS needs direct access and any layer in between might end up a disaster!!! Which means that practically HAST should only be used in UFS environments backed by a hardware controller. In that case, HAST will not notice again anything (unless you loose the controller) but at least you will know that you need to replace a disk, by monitoring the controller status. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:52:21 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 22A3AD1E for ; Wed, 1 Oct 2014 13:52:21 +0000 (UTC) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9E5229CA for ; Wed, 1 Oct 2014 13:52:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id B77651472001; Wed, 1 Oct 2014 15:52:16 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4+81oNmjKsP0; Wed, 1 Oct 2014 15:52:14 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id 179534C4C9BA; Wed, 1 Oct 2014 15:52:14 +0200 (CEST) Message-ID: <542C0710.3020402@internetx.com> Date: Wed, 01 Oct 2014 15:52:16 +0200 From: InterNetX - Juergen Gotteswinter Reply-To: jg@internetx.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: George Kontostanos , freebsd-fs@freebsd.org Subject: Re: HAST with broken HDD References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:52:21 -0000 Am 01.10.2014 um 15:49 schrieb George Kontostanos: > On Wed, Oct 1, 2014 at 4:29 PM, InterNetX - Juergen Gotteswinter > > wrote: > > Am 01.10.2014 um 15:06 schrieb George Kontostanos: > > > > > > On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinter > > >> wrote: > > > > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen Gotteswinter > > > > > > >>> wrote: > > > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > > Hello, > > > > I'm preparing a HA NAS solution using HAST. > > > > I'm wondering what will happen if one of disks of the > > primary node will > > > > fail or become erratic. > > > > > > > > Thx, > > > > Jean-François Bogaerts > > > > > > nothing. if you are using zfs on top of hast zfs wont even > > take notice > > > about the disk failure. > > > > > > as long as the write operation was sucessfull on one of the 2 > > nodes, > > > hast doesnt notify the ontop layers about io errors. > > > > > > interesting concept, took me some time to deal with this. > > > > > > > > > Are you saying that the pool will appear to be optimal even with a bad > > > drive? > > > > > > > > > > https://forums.freebsd.org/viewtopic.php?&t=24786 > > > > > > > > It appears that this is actually the case. And it is very disturbing, > > meaning that a drive failure goes unnoticed. In my case I completely > > removed the second disk on the primary node and a zpool status showed > > absolutely no problem. Scrubbing the pool began resilvering which > > indicates that there is actually something wrong! > > > right. lets go further and think how zfs works regarding direct hardware > / disk access. theres a layer between which always says ey, everthing is > fine. no more need for pool scrubbing, since hastd wont tell if anything > is wrong :D > > > Correct, ZFS needs direct access and any layer in between might end up a > disaster!!! > > Which means that practically HAST should only be used in UFS > environments backed by a hardware controller. In that case, HAST will > not notice again anything (unless you loose the controller) but at least > you will know that you need to replace a disk, by monitoring the > controller status. > imho this should be included at least as a notice/warning in the hastd manpage, afaik theres no real warning about such problems with the hastd/zfs combo. but lots of howtos are out there describing exactly such setups. sad, since the comparable piece on linux - drbd - is handling io errors fine. the upper layers get notified like it should be imho From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 13:57:26 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A6AFE42 for ; Wed, 1 Oct 2014 13:57:26 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18DCBA13 for ; Wed, 1 Oct 2014 13:57:26 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id bj1so324841pad.15 for ; Wed, 01 Oct 2014 06:57:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=FG32i/v81sqjEf6yNO9uRXRDP7awwnN7GHREPlvN14o=; b=KQtsK1ytMAtZESYRHxw4U0E4pzNAfDfyfaA97An+S6WYqlBDXI0wGQiCBPwRxekOKM kctEAZ1fZ227qidDzPu7DSwndatqTaU3MAI8/PRcRjTC+AknIKCgFa3D3lnyYyyGeEeP rZPAi2KTkGEAoMrNGhNENRBYsU3Utt1HKOW+tNpwf+fKrL+geDY23I9haSICIjEfpYWJ A8WlrZU13ACjAXI7EPS3hAFt+bVIwLfhjIMDSdifG2ZfoVPXwPu9AmiaS3Ftp/cn5nEs g47BQ6fVcU/TP/cz80m9FFYC6UbqjH9F9VsX53KtiGe2KqNpvGfPmjulnOEOI+sN1jja bbuQ== X-Received: by 10.66.220.230 with SMTP id pz6mr77419409pac.145.1412171845599; Wed, 01 Oct 2014 06:57:25 -0700 (PDT) Received: from [10.0.180.25] (50-46-123-160.evrt.wa.frontiernet.net. [50.46.123.160]) by mx.google.com with ESMTPSA id i10sm1153489pdn.26.2014.10.01.06.57.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 01 Oct 2014 06:57:24 -0700 (PDT) From: John Terrell Message-Id: <2CAB03AD-B755-4F85-A6CA-E30B76517269@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Date: Wed, 1 Oct 2014 06:57:22 -0700 References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> To: "freebsd-fs@freebsd.org" In-Reply-To: X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 13:57:26 -0000 Eureka! I think I=92ve found the issue and it=92s nothing to do with = ZFS - it=92s an infrastructure problem. I was running another test = last night and just happened to be watching about two hours into the = test when the Mac I use to connect to the network suddenly (and = unexpectedly) lost connectivity for about five seconds. When it came = back, the SSH connection was intact but the ZFS transfer in progress = immediately failed. I=92m not sure if this is a DHCP lease = re-aquisition or what but to see if this is indeed the problem, I gave = the Mac a manual IP address and started the transfer over night. As of = now, it=92s been running for about 7 hours without issue. I=92m not = quite ready to call it solved until this transfer completes successfully = but this is the furthest it=92s gotten. On Sep 30, 2014, at 8:02 PM, John Terrell = wrote: > Sure. On the SmartOS (receiving) side: >=20 > [root@smartos /]# zpool get all zones | grep feature@ > zones feature@async_destroy enabled = local > zones feature@empty_bpobj active = local > zones feature@lz4_compress active = local > zones feature@multi_vdev_crash_dump enabled = local > zones feature@spacemap_histogram active = local > zones feature@enabled_txg active = local > zones feature@hole_birth active = local > zones feature@extensible_dataset enabled = local > zones feature@embedded_data active = local > zones feature@bookmarks enabled = local > zones feature@filesystem_limits enabled = local >=20 > =85on the FreeBSD 10 (sending) side: >=20 > [root@freebsd /]# zpool get all tank | grep feature@ > tank feature@async_destroy enabled = local > tank feature@empty_bpobj active = local > tank feature@lz4_compress active = local > tank feature@multi_vdev_crash_dump enabled = local >=20 >=20 > So, the FreeBSD side is a subset of the SmartOS side. >=20 >=20 > On Sep 30, 2014, at 7:51 PM, Rich wrote: >=20 >> It's possible that your feature flags and the receiving end's aren't >> compatible - can you show us a zfs get all tank on both sides? >>=20 >> - Rich >>=20 >> On Tue, Sep 30, 2014 at 10:39 PM, John Terrell = wrote: >>> By the way, here's another report that sounds similar to mine = (though on Linux): >>>=20 >>> = http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway >>>=20 >>> His repro doesn't even use ssh (appears to be completely local). >>>=20 >>>=20 >>> On Sep 30, 2014, at 6:41 PM, John Terrell = wrote: >>>=20 >>>> Sorry for the tardy response (busy at work. :)). Here's the = command and output (run from the SmartOS box): >>>>=20 >>>> ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | = zfs receive -v -F zones/tank >>>>=20 >>>> The output (after running for a long time xferring data): >>>>=20 >>>> receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 >>>> received 1.35GB stream in 15 seconds (91.9MB/sec) >>>> receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 >>>> Read from remote host nas00: Connection reset by peer >>>> cannot receive new filesystem stream: invalid backup stream >>>>=20 >>>>=20 >>>> Is it possible the command is simply timing out for some reason and = closing the connection? >>>>=20 >>>>=20 >>>> On Sep 26, 2014, at 3:04 PM, Will Andrews = wrote: >>>>=20 >>>>> What do the zfs commands print? The -v option prints some metadata = that might help diagnose the problem. >>>>>=20 >>>>> --Will. >>>>>=20 >>>>> On Monday, September 22, 2014, John Terrell = wrote: >>>>> Hello all, I'm trying to replicate one of the ZFS filesystems that = lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The = transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>>>>=20 >>>>> "cannot receive incremental stream: invalid backup stream" >>>>>=20 >>>>> The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >>>>>=20 >>>>> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v = tank >>>>>=20 >>>>> I've also tried to use mbuffer as the transport interface = (removing SSH from the equation) and it has the same result. >>>>>=20 >>>>> Is the possibly an incompatibility between FreeBSD10 and SmartOS = ZFS implementations? >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 14:00:47 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB66E299 for ; Wed, 1 Oct 2014 14:00:46 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 691A1A9E for ; Wed, 1 Oct 2014 14:00:46 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id cc10so661981wib.13 for ; Wed, 01 Oct 2014 07:00:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cGOO6F8Rx55sCgOxCARfy6C151r9MOiqbpdi+HMoIJY=; b=JBvEeVyZ53/hi5JQxzPMu0MsNZQPC5E1cNQJNS/gLUXKINgncvUJVaycMmbobp8i0T ymX6FHcrkZnLou0tYCc/h2E1Tt8C1pRIMtFWAeWxwQgANz7ExNqqCvPZkFkLhkyaEZQS oOESMAITXNZhM9W4fGIX/lxN49OuAwqqfgH1mCfyw0N+d4NKGH85WBYMDNdykiRs5JJc dsCaiFIWpZabMliJGi1NnwK1Uv7+QGOHCnPVUsckzDwRfR6XdYCD1ptDtLvrqu3jj3iK HgHICZCniOe/WnchADZeCRSZinMR5CWfn1V/aY2gZyWRxEJIL2FTgA0pOE0OkB1FYUPf Kifw== MIME-Version: 1.0 X-Received: by 10.180.97.199 with SMTP id ec7mr14789757wib.29.1412172044638; Wed, 01 Oct 2014 07:00:44 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 07:00:44 -0700 (PDT) In-Reply-To: <542C0710.3020402@internetx.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> <542C0710.3020402@internetx.com> Date: Wed, 1 Oct 2014 17:00:44 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: jg@internetx.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 14:00:47 -0000 On Wed, Oct 1, 2014 at 4:52 PM, InterNetX - Juergen Gotteswinter < jg@internetx.com> wrote: > Am 01.10.2014 um 15:49 schrieb George Kontostanos: > > On Wed, Oct 1, 2014 at 4:29 PM, InterNetX - Juergen Gotteswinter > > > wrote: > > > > Am 01.10.2014 um 15:06 schrieb George Kontostanos: > > > > > > > > > On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinter > > > jg@internetx.com > > >> wrote: > > > > > > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen > Gotteswinter > > > > jg@internetx.com > > > > > > jg@internetx.com > > >>> wrote: > > > > > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > > > Hello, > > > > > I'm preparing a HA NAS solution using HAST. > > > > > I'm wondering what will happen if one of disks of > the > > > primary node will > > > > > fail or become erratic. > > > > > > > > > > Thx, > > > > > Jean-Fran=C3=A7ois Bogaerts > > > > > > > > nothing. if you are using zfs on top of hast zfs wont > even > > > take notice > > > > about the disk failure. > > > > > > > > as long as the write operation was sucessfull on one of > the 2 > > > nodes, > > > > hast doesnt notify the ontop layers about io errors. > > > > > > > > interesting concept, took me some time to deal with thi= s. > > > > > > > > > > > > Are you saying that the pool will appear to be optimal even > with a bad > > > > drive? > > > > > > > > > > > > > > https://forums.freebsd.org/viewtopic.php?&t=3D24786 > > > > > > > > > > > > It appears that this is actually the case. And it is very > disturbing, > > > meaning that a drive failure goes unnoticed. In my case I > completely > > > removed the second disk on the primary node and a zpool status > showed > > > absolutely no problem. Scrubbing the pool began resilvering which > > > indicates that there is actually something wrong! > > > > > > right. lets go further and think how zfs works regarding direct > hardware > > / disk access. theres a layer between which always says ey, > everthing is > > fine. no more need for pool scrubbing, since hastd wont tell if > anything > > is wrong :D > > > > > > Correct, ZFS needs direct access and any layer in between might end up = a > > disaster!!! > > > > Which means that practically HAST should only be used in UFS > > environments backed by a hardware controller. In that case, HAST will > > not notice again anything (unless you loose the controller) but at leas= t > > you will know that you need to replace a disk, by monitoring the > > controller status. > > > > imho this should be included at least as a notice/warning in the hastd > manpage, afaik theres no real warning about such problems with the > hastd/zfs combo. but lots of howtos are out there describing exactly > such setups. > > Yes, it should. I have actually written a guide like that when HAST was a= t its early stages. I had never tested it though for flaws. This thread started ringing some bells! > sad, since the comparable piece on linux - drbd - is handling io errors > fine. the upper layers get notified like it should be imho > > My next lab environment will be to try a DRBD similar set up. Although some tests we performed last year with ZFS on linux were not that promising. --=20 George Kontostanos --- http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 16:31:52 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 99457922 for ; Wed, 1 Oct 2014 16:31:52 +0000 (UTC) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A4B613C for ; Wed, 1 Oct 2014 16:31:51 +0000 (UTC) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.14.7/8.14.7) with ESMTP id s91FqEgc052930 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 1 Oct 2014 16:52:19 +0100 (BST) (envelope-from matt.churchyard@userve.net) Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.516.32; Wed, 1 Oct 2014 16:51:44 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0516.029; Wed, 1 Oct 2014 16:51:43 +0100 From: Matt Churchyard To: "freebsd-fs@freebsd.org" Subject: RE: HAST with broken HDD Thread-Topic: HAST with broken HDD Thread-Index: AQHP3VWBNn9KDQ+WSU+h7BxOi55P05wbARSAgAAZ64CAAAXSgIAABNQAgAAGQACAACeA2IAADLbQ Date: Wed, 1 Oct 2014 15:51:43 +0000 Message-ID: <97aab72e19d640ebb65c754c858043cc@SERVER.ad.usd-group.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> <542C0710.3020402@internetx.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 16:31:52 -0000 T24gV2VkLCBPY3QgMSwgMjAxNCBhdCA0OjUyIFBNLCBJbnRlck5ldFggLSBKdWVyZ2VuIEdvdHRl c3dpbnRlciA8IGpnQGludGVybmV0eC5jb20+IHdyb3RlOg0KDQo+IEFtIDAxLjEwLjIwMTQgdW0g MTU6NDkgc2NocmllYiBHZW9yZ2UgS29udG9zdGFub3M6DQo+ID4gT24gV2VkLCBPY3QgMSwgMjAx NCBhdCA0OjI5IFBNLCBJbnRlck5ldFggLSBKdWVyZ2VuIEdvdHRlc3dpbnRlciANCj4gPiA8amdA aW50ZXJuZXR4LmNvbSA8bWFpbHRvOmpnQGludGVybmV0eC5jb20+PiB3cm90ZToNCj4gPg0KPiA+ ICAgICBBbSAwMS4xMC4yMDE0IHVtIDE1OjA2IHNjaHJpZWIgR2VvcmdlIEtvbnRvc3Rhbm9zOg0K PiA+ICAgICA+DQo+ID4gICAgID4NCj4gPiAgICAgPiBPbiBXZWQsIE9jdCAxLCAyMDE0IGF0IDM6 NDkgUE0sIEludGVyTmV0WCAtIEp1ZXJnZW4gR290dGVzd2ludGVyDQo+ID4gICAgID4gPGpnQGlu dGVybmV0eC5jb20gPG1haWx0bzpqZ0BpbnRlcm5ldHguY29tPiA8bWFpbHRvOg0KPiBqZ0BpbnRl cm5ldHguY29tDQo+ID4gICAgIDxtYWlsdG86amdAaW50ZXJuZXR4LmNvbT4+PiB3cm90ZToNCj4g PiAgICAgPg0KPiA+ICAgICA+ICAgICBBbSAwMS4xMC4yMDE0IHVtIDE0OjI4IHNjaHJpZWIgR2Vv cmdlIEtvbnRvc3Rhbm9zOg0KPiA+ICAgICA+ICAgICA+DQo+ID4gICAgID4gICAgID4gT24gV2Vk LCBPY3QgMSwgMjAxNCBhdCAxOjU1IFBNLCBJbnRlck5ldFggLSBKdWVyZ2VuDQo+IEdvdHRlc3dp bnRlcg0KPiA+ICAgICA+ICAgICA+IDxqZ0BpbnRlcm5ldHguY29tIDxtYWlsdG86amdAaW50ZXJu ZXR4LmNvbT4gPG1haWx0bzoNCj4gamdAaW50ZXJuZXR4LmNvbQ0KPiA+ICAgICA8bWFpbHRvOmpn QGludGVybmV0eC5jb20+Pg0KPiA+ICAgICA+ICAgICA8bWFpbHRvOmpnQGludGVybmV0eC5jb20g PG1haWx0bzpqZ0BpbnRlcm5ldHguY29tPiA8bWFpbHRvOg0KPiBqZ0BpbnRlcm5ldHguY29tDQo+ ID4gICAgIDxtYWlsdG86amdAaW50ZXJuZXR4LmNvbT4+Pj4gd3JvdGU6DQo+ID4gICAgID4gICAg ID4NCj4gPiAgICAgPiAgICAgPiAgICAgQW0gMDEuMTAuMjAxNCB1bSAxMDo1NCBzY2hyaWViIEpG LUJvZ2FlcnRzOg0KPiA+ICAgICA+ICAgICA+ICAgICA+ICAgIEhlbGxvLA0KPiA+ICAgICA+ICAg ICA+ICAgICA+ICAgIEknbSBwcmVwYXJpbmcgYSBIQSBOQVMgc29sdXRpb24gdXNpbmcgSEFTVC4N Cj4gPiAgICAgPiAgICAgPiAgICAgPiAgICBJJ20gd29uZGVyaW5nIHdoYXQgd2lsbCBoYXBwZW4g aWYgb25lIG9mIGRpc2tzIG9mDQo+IHRoZQ0KPiA+ICAgICA+ICAgICBwcmltYXJ5IG5vZGUgd2ls bA0KPiA+ICAgICA+ICAgICA+ICAgICA+ICAgIGZhaWwgb3IgYmVjb21lIGVycmF0aWMuDQo+ID4g ICAgID4gICAgID4gICAgID4NCj4gPiAgICAgPiAgICAgPiAgICAgPiAgICBUaHgsDQo+ID4gICAg ID4gICAgID4gICAgID4gICAgSmVhbi1GcmFuw6dvaXMgQm9nYWVydHMNCj4gPiAgICAgPiAgICAg Pg0KPiA+ICAgICA+ICAgICA+ICAgICBub3RoaW5nLiBpZiB5b3UgYXJlIHVzaW5nIHpmcyBvbiB0 b3Agb2YgaGFzdCB6ZnMgd29udA0KPiBldmVuDQo+ID4gICAgID4gICAgIHRha2Ugbm90aWNlDQo+ ID4gICAgID4gICAgID4gICAgIGFib3V0IHRoZSBkaXNrIGZhaWx1cmUuDQo+ID4gICAgID4gICAg ID4NCj4gPiAgICAgPiAgICAgPiAgICAgYXMgbG9uZyBhcyB0aGUgd3JpdGUgb3BlcmF0aW9uIHdh cyBzdWNlc3NmdWxsIG9uIG9uZSBvZg0KPiB0aGUgMg0KPiA+ICAgICA+ICAgICBub2RlcywNCj4g PiAgICAgPiAgICAgPiAgICAgaGFzdCBkb2VzbnQgbm90aWZ5IHRoZSBvbnRvcCBsYXllcnMgYWJv dXQgaW8gZXJyb3JzLg0KPiA+ICAgICA+ICAgICA+DQo+ID4gICAgID4gICAgID4gICAgIGludGVy ZXN0aW5nIGNvbmNlcHQsIHRvb2sgbWUgc29tZSB0aW1lIHRvIGRlYWwgd2l0aCB0aGlzLg0KPiA+ ICAgICA+ICAgICA+DQo+ID4gICAgID4gICAgID4NCj4gPiAgICAgPiAgICAgPiBBcmUgeW91IHNh eWluZyB0aGF0IHRoZSBwb29sIHdpbGwgYXBwZWFyIHRvIGJlIG9wdGltYWwgZXZlbg0KPiB3aXRo IGEgYmFkDQo+ID4gICAgID4gICAgID4gZHJpdmU/DQo+ID4gICAgID4gICAgID4NCj4gPiAgICAg PiAgICAgPg0KPiA+ICAgICA+DQo+ID4gICAgID4gICAgIGh0dHBzOi8vZm9ydW1zLmZyZWVic2Qu b3JnL3ZpZXd0b3BpYy5waHA/JnQ9MjQ3ODYNCj4gPiAgICAgPg0KPiA+ICAgICA+DQo+ID4gICAg ID4NCj4gPiAgICAgPiBJdCBhcHBlYXJzIHRoYXQgdGhpcyBpcyBhY3R1YWxseSB0aGUgY2FzZS4g QW5kIGl0IGlzIHZlcnkNCj4gZGlzdHVyYmluZywNCj4gPiAgICAgPiBtZWFuaW5nIHRoYXQgYSBk cml2ZSBmYWlsdXJlIGdvZXMgdW5ub3RpY2VkLiBJbiBteSBjYXNlIEkNCj4gY29tcGxldGVseQ0K PiA+ICAgICA+IHJlbW92ZWQgdGhlIHNlY29uZCBkaXNrIG9uIHRoZSBwcmltYXJ5IG5vZGUgYW5k IGEgenBvb2wgc3RhdHVzDQo+IHNob3dlZA0KPiA+ICAgICA+IGFic29sdXRlbHkgbm8gcHJvYmxl bS4gU2NydWJiaW5nIHRoZSBwb29sIGJlZ2FuIHJlc2lsdmVyaW5nIHdoaWNoDQo+ID4gICAgID4g aW5kaWNhdGVzIHRoYXQgdGhlcmUgaXMgYWN0dWFsbHkgc29tZXRoaW5nIHdyb25nIQ0KPiA+DQo+ ID4NCj4gPiAgICAgcmlnaHQuIGxldHMgZ28gZnVydGhlciBhbmQgdGhpbmsgaG93IHpmcyB3b3Jr cyByZWdhcmRpbmcgZGlyZWN0DQo+IGhhcmR3YXJlDQo+ID4gICAgIC8gZGlzayBhY2Nlc3MuIHRo ZXJlcyBhIGxheWVyIGJldHdlZW4gd2hpY2ggYWx3YXlzIHNheXMgZXksDQo+IGV2ZXJ0aGluZyBp cw0KPiA+ICAgICBmaW5lLiBubyBtb3JlIG5lZWQgZm9yIHBvb2wgc2NydWJiaW5nLCBzaW5jZSBo YXN0ZCB3b250IHRlbGwgaWYNCj4gYW55dGhpbmcNCj4gPiAgICAgaXMgd3JvbmcgOkQNCj4gPg0K PiA+DQo+ID4gQ29ycmVjdCwgWkZTIG5lZWRzIGRpcmVjdCBhY2Nlc3MgYW5kIGFueSBsYXllciBp biBiZXR3ZWVuIG1pZ2h0IGVuZCANCj4gPiB1cCBhIGRpc2FzdGVyISEhDQo+ID4NCj4gPiBXaGlj aCBtZWFucyB0aGF0IHByYWN0aWNhbGx5IEhBU1Qgc2hvdWxkIG9ubHkgYmUgdXNlZCBpbiBVRlMg DQo+ID4gZW52aXJvbm1lbnRzIGJhY2tlZCBieSBhIGhhcmR3YXJlIGNvbnRyb2xsZXIuIEluIHRo YXQgY2FzZSwgSEFTVCANCj4gPiB3aWxsIG5vdCBub3RpY2UgYWdhaW4gYW55dGhpbmcgKHVubGVz cyB5b3UgbG9vc2UgdGhlIGNvbnRyb2xsZXIpIGJ1dCANCj4gPiBhdCBsZWFzdCB5b3Ugd2lsbCBr bm93IHRoYXQgeW91IG5lZWQgdG8gcmVwbGFjZSBhIGRpc2ssIGJ5IA0KPiA+IG1vbml0b3Jpbmcg dGhlIGNvbnRyb2xsZXIgc3RhdHVzLg0KPiA+DQo+DQo+IGltaG8gdGhpcyBzaG91bGQgYmUgaW5j bHVkZWQgYXQgbGVhc3QgYXMgYSBub3RpY2Uvd2FybmluZyBpbiB0aGUgaGFzdGQgDQo+IG1hbnBh Z2UsIGFmYWlrIHRoZXJlcyBubyByZWFsIHdhcm5pbmcgYWJvdXQgc3VjaCBwcm9ibGVtcyB3aXRo IHRoZSANCj4gaGFzdGQvemZzIGNvbWJvLiBidXQgbG90cyBvZiBob3d0b3MgYXJlIG91dCB0aGVy ZSBkZXNjcmliaW5nIGV4YWN0bHkgDQo+IHN1Y2ggc2V0dXBzLg0KPg0KPiBZZXMsIGl0IHNob3Vs ZC4gSSBoYXZlIGFjdHVhbGx5IHdyaXR0ZW4gYSBndWlkZSBsaWtlIHRoYXQgd2hlbiBIQVNUIA0K PiB3YXMgYXQNCml0cyBlYXJseSBzdGFnZXMuIEkgaGFkIG5ldmVyIHRlc3RlZCBpdCB0aG91Z2gg Zm9yIGZsYXdzLiBUaGlzIHRocmVhZCBzdGFydGVkIHJpbmdpbmcgc29tZSBiZWxscyENCg0KDQoN Cj4gc2FkLCBzaW5jZSB0aGUgY29tcGFyYWJsZSBwaWVjZSBvbiBsaW51eCAtIGRyYmQgLSBpcyBo YW5kbGluZyBpbyANCj4gZXJyb3JzIGZpbmUuIHRoZSB1cHBlciBsYXllcnMgZ2V0IG5vdGlmaWVk IGxpa2UgaXQgc2hvdWxkIGJlIGltaG8NCj4NCj4gTXkgbmV4dCBsYWIgZW52aXJvbm1lbnQgd2ls bCBiZSB0byB0cnkgYSBEUkJEIHNpbWlsYXIgc2V0IHVwLiBBbHRob3VnaA0Kc29tZSB0ZXN0cyB3 ZSBwZXJmb3JtZWQgbGFzdCB5ZWFyIHdpdGggWkZTIG9uIGxpbnV4IHdlcmUgbm90IHRoYXQgcHJv bWlzaW5nLg0KDQpGcm9tIHdoYXQgSSBjYW4gc2VlIEhBU1QgaXMgd29ya2luZywgYXQgbGVhc3Qg aW4gY29uY2VwdCwgYXMgaXQgc2hvdWxkLg0KDQpJZiB5b3UgaW5zdGFsbCBhbnkgZmlsZXN5c3Rl bSBvbiB0b3Agb2YgYSBSQUlEIG1pcnJvciwgZWl0aGVyIGRpc2sgY2FuIGZhaWwgYW5kIHRoZSBm aWxlc3lzdGVtIGFib3ZlIHNob3VsZCBqdXN0IGNvbnRpbnVlIG9uIGFzIGlmIG5vdGhpbmcgaGFw cGVuZWQuIEl0J3MgdXAgdG8gdGhlIFJBSUQgbGF5ZXIgdG8gbm90aWZ5IHlvdSBvZiB0aGUgcHJv YmxlbS4NCg0KSEFTVCBpcyBiYXNpY2FsbHkgIlJBSUQxLW92ZXItbmV0d29yayIsIHNvIGlmIGEg ZGlzayBmYWlscywgaXQgc2hvdWxkIGp1c3QgaGFuZGxlIHJlYWQvd3JpdGVzIHVzaW5nIHRoZSBv dGhlciBkaXNrLCBhbmQgdGhlIGZpbGVzeXN0ZW0gb24gdG9wLCBiZSBpdCBVRlMvWkZTL3doYXRl dmVyLCBzaG91bGQganVzdCBjYXJyeSBvbiBhcyBub3JtYWwgKHdoaWNoIGlzIHdoYXQgaGFzIGJl ZW4gb2JzZXJ2ZWQpLiBPZiBjb3Vyc2UsIEhBU1QgKG9yIHRoZSBPUykgc2hvdWxkIG5vdGlmeSB5 b3Ugb2YgdGhlIGRpc2sgZXJyb3IgdGhvdWdoIChwcm9iYWJseSB0aHJvdWdoIGRldmQpIHNvIHlv dSBjYW4gZG8gc29tZXRoaW5nIGFib3V0IGl0LiBNYXliZSBpdCBhbHJlYWR5IGV4aXN0cywgYnV0 IEhBU1Qgc2hvdWxkIGJlIGFibGUgdG8gcHJvdmlkZSBvdmVyYWxsIHN0YXR1cyBpbmZvcm1hdGlv biBhbmQgcmFpc2UgZXZlbnRzIGp1c3QgbGlrZSBaRlMgb3IgYW55IFJBSUQgc3Vic3lzdGVtIHdv dWxkLiBZb3UgYWxzbyBvZiBjb3Vyc2Ugc2hvdWxkbid0IGdldCBzY3J1YiBlcnJvcnMgYW5kIGNv cnJ1cHRpb24gbGlrZSB0aGF0IHNlZW4gaW4gdGhlIG9yaWdpbmFsIHBvc3QgZWl0aGVyIGp1c3Qg YmVjYXVzZSBvbmUgaGFsZiBvZiB0aGUgSEFTVCBtaXJyb3IgaGFzIGdvbmUuDQoNClBlcnNvbmFs bHkgSSd2ZSBub3QgYmVlbiBicmF2ZSBlbm91Z2ggdG8gdXNlIEhBU1QgeWV0LiBJdCBzZWVtcyB0 byBtZSBsaWtlIHRoZXJlJ3MgdG9vIG1hbnkgcG9zc2liaWxpdGllcyBmb3Igc2l0dWF0aW9ucyB3 aGVyZSB0aGluZ3MgY2FuIGdvIHdyb25nLiBPbmUgb2YgdGhlc2UgdGhhdCBoYXMgYmVlbiBkaXNj dXNzZWQgb24gdGhlIGZvcnVtcyBpcyB0aGF0IGEgWkZTIHNjcnViIHdpbGwgb25seSByZWFkIGRh dGEgZnJvbSB0aGUgbG9jYWwgZGlzay4gWW91IGNvdWxkIGhhcHBpbHkgcnVuIGEgc2VydmljZSBm cm9tIHRoZSBtYXN0ZXIgc2VydmVyIGZvciB5ZWFycywgc2NydWJiaW5nIHJlZ3VsYXJseSwgbmV2 ZXIga25vd2luZyB0aGF0IHlvdXIgZGF0YSBtYXkgYmUgY29ycnVwdCBvbiB0aGUgc2Vjb25kIEhB U1Qgbm9kZS4gT25lICdzb2x1dGlvbicgbWVudGlvbmVkIGZvciB0aGlzIHdvdWxkIGJlIHRvIHJl Z3VsYXJseSBzd2l0Y2ggdGhlIG1hc3Rlci9zbGF2ZSBub2RlcywgcnVubmluZyBzY3J1YnMgb24g ZWFjaCBvbmUgd2hpbGUgdGhleSBhcmUgbWFzdGVyLg0KDQotLQ0KTWF0dA0K From owner-freebsd-fs@FreeBSD.ORG Wed Oct 1 21:33:55 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E275BF6D for ; Wed, 1 Oct 2014 21:33:55 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6872FD9F for ; Wed, 1 Oct 2014 21:33:55 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id k14so1797401wgh.5 for ; Wed, 01 Oct 2014 14:33:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=36RcDHyNiGciDFRiSqTJrNyXE9b51ZxYMbIT7gIWNpM=; b=I1rX3xMD/5OFyBxXvViVO3zzDqzZQ1wREPUmJnBGmB0Ak7BTeZLig7fXWkqgz/y91T HaJ8mJGHMra/Y3ms3nKt+nUZrKReF6h+HC1nW/W6QbpPBDer333meY5y00JPJvtPEPXi 8q+O5QLwfB+8wB/+9v+r0Al0D7EavnqOvGb9VK7lCwoMlV1WRD803hcIGnkqP+2pz/pj DeTNYFoxP+SBCZs8A6kW0X2XNjXbkbdCgMTGFWBJKKgixyYec1ymUZZhjs996xlMklAC llBmH4lur4MquFtDPU5Lr+i//SKNNFAlfOqUNOU7E2CCI8zJZVI850FHMwSHZ8qb2pn7 QONw== MIME-Version: 1.0 X-Received: by 10.180.97.199 with SMTP id ec7mr18189995wib.29.1412199233556; Wed, 01 Oct 2014 14:33:53 -0700 (PDT) Received: by 10.27.137.130 with HTTP; Wed, 1 Oct 2014 14:33:53 -0700 (PDT) In-Reply-To: <97aab72e19d640ebb65c754c858043cc@SERVER.ad.usd-group.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> <542C0710.3020402@internetx.com> <97aab72e19d640ebb65c754c858043cc@SERVER.ad.usd-group.com> Date: Thu, 2 Oct 2014 00:33:53 +0300 Message-ID: Subject: Re: HAST with broken HDD From: George Kontostanos To: Matt Churchyard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 21:33:56 -0000 On Wed, Oct 1, 2014 at 6:51 PM, Matt Churchyard wrote: > On Wed, Oct 1, 2014 at 4:52 PM, InterNetX - Juergen Gotteswinter < > jg@internetx.com> wrote: > > > Am 01.10.2014 um 15:49 schrieb George Kontostanos: > > > On Wed, Oct 1, 2014 at 4:29 PM, InterNetX - Juergen Gotteswinter > > > > wrote: > > > > > > Am 01.10.2014 um 15:06 schrieb George Kontostanos: > > > > > > > > > > > > On Wed, Oct 1, 2014 at 3:49 PM, InterNetX - Juergen Gotteswinte= r > > > > > jg@internetx.com > > > >> wrote: > > > > > > > > Am 01.10.2014 um 14:28 schrieb George Kontostanos: > > > > > > > > > > On Wed, Oct 1, 2014 at 1:55 PM, InterNetX - Juergen > > Gotteswinter > > > > > > jg@internetx.com > > > > > > > > > jg@internetx.com > > > >>> wrote: > > > > > > > > > > Am 01.10.2014 um 10:54 schrieb JF-Bogaerts: > > > > > > Hello, > > > > > > I'm preparing a HA NAS solution using HAST. > > > > > > I'm wondering what will happen if one of disks o= f > > the > > > > primary node will > > > > > > fail or become erratic. > > > > > > > > > > > > Thx, > > > > > > Jean-Fran=C3=A7ois Bogaerts > > > > > > > > > > nothing. if you are using zfs on top of hast zfs wont > > even > > > > take notice > > > > > about the disk failure. > > > > > > > > > > as long as the write operation was sucessfull on one = of > > the 2 > > > > nodes, > > > > > hast doesnt notify the ontop layers about io errors. > > > > > > > > > > interesting concept, took me some time to deal with > this. > > > > > > > > > > > > > > > Are you saying that the pool will appear to be optimal ev= en > > with a bad > > > > > drive? > > > > > > > > > > > > > > > > > > https://forums.freebsd.org/viewtopic.php?&t=3D24786 > > > > > > > > > > > > > > > > It appears that this is actually the case. And it is very > > disturbing, > > > > meaning that a drive failure goes unnoticed. In my case I > > completely > > > > removed the second disk on the primary node and a zpool status > > showed > > > > absolutely no problem. Scrubbing the pool began resilvering whi= ch > > > > indicates that there is actually something wrong! > > > > > > > > > right. lets go further and think how zfs works regarding direct > > hardware > > > / disk access. theres a layer between which always says ey, > > everthing is > > > fine. no more need for pool scrubbing, since hastd wont tell if > > anything > > > is wrong :D > > > > > > > > > Correct, ZFS needs direct access and any layer in between might end > > > up a disaster!!! > > > > > > Which means that practically HAST should only be used in UFS > > > environments backed by a hardware controller. In that case, HAST > > > will not notice again anything (unless you loose the controller) but > > > at least you will know that you need to replace a disk, by > > > monitoring the controller status. > > > > > > > imho this should be included at least as a notice/warning in the hastd > > manpage, afaik theres no real warning about such problems with the > > hastd/zfs combo. but lots of howtos are out there describing exactly > > such setups. > > > > Yes, it should. I have actually written a guide like that when HAST > > was at > its early stages. I had never tested it though for flaws. This thread > started ringing some bells! > > > > > sad, since the comparable piece on linux - drbd - is handling io > > errors fine. the upper layers get notified like it should be imho > > > > My next lab environment will be to try a DRBD similar set up. Although > some tests we performed last year with ZFS on linux were not that > promising. > > From what I can see HAST is working, at least in concept, as it should. > > If you install any filesystem on top of a RAID mirror, either disk can > fail and the filesystem above should just continue on as if nothing > happened. It's up to the RAID layer to notify you of the problem. > > HAST is basically "RAID1-over-network", so if a disk fails, it should jus= t > handle read/writes using the other disk, and the filesystem on top, be it > UFS/ZFS/whatever, should just carry on as normal (which is what has been > observed). Of course, HAST (or the OS) should notify you of the disk erro= r > though (probably through devd) so you can do something about it. Maybe it > already exists, but HAST should be able to provide overall status > information and raise events just like ZFS or any RAID subsystem would. Y= ou > also of course shouldn't get scrub errors and corruption like that seen i= n > the original post either just because one half of the HAST mirror has gon= e. > > Personally I've not been brave enough to use HAST yet. It seems to me lik= e > there's too many possibilities for situations where things can go wrong. > One of these that has been discussed on the forums is that a ZFS scrub wi= ll > only read data from the local disk. You could happily run a service from > the master server for years, scrubbing regularly, never knowing that your > data may be corrupt on the second HAST node. One 'solution' mentioned for > this would be to regularly switch the master/slave nodes, running scrubs = on > each one while they are master. > > -- > Matt > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > I believe that HAST began as FreeBSD's answer to DRBD. Given the fact that FreeBSD had (still has) the competitive advantage of ZFS then maybe it should have been designed with that in mind. ZFS is not meant to work within a middle layer. Instead it needs direct access to the disks. So, only the fact that we are using a technology that creates another layer is a good stopper. There are of course other ways to achieve redundancy and avoid going over the network. But in some cases, where you need 2 storages located in to different DC's, HAST might have been a good choice. Of course, like you mentioned before. In order for this to work we would need for HAST to monitor the health of every resource. That could be from devd or from another HAST daemon or a combination. The administrator should be able to easily get warnings about faulty components. We would also need to use a fence device instead of relying on VRRP. Anyway, thats food for though :) --=20 George Kontostanos --- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 2 22:37:52 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 72154542 for ; Thu, 2 Oct 2014 22:37:52 +0000 (UTC) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF3E1C88 for ; Thu, 2 Oct 2014 22:37:51 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id gq15so73417lab.12 for ; Thu, 02 Oct 2014 15:37:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Mg3fzYca7V2k76uIsBm+JvDWOzeznNYFIDsWTzSMlCE=; b=wvTf3yrdGtkS4zNuXfaTUDwYfLTzf7SlQ9dIQbfKP2lRLYaKdRi8DHG6Tbos4r1OZT 63ckFNFjAA1WDOrwEYg7Fa+dJMN0P4KGqPoAFKO3V6s98+xZAwxzrqC6/dmHcZEN3EzG w+3oudxr2FLShS1n0lE0vrSaxpidoZ0Gt1H43UKVUnEWD6dWpJJY4HraiySGThDcRu4i 4uJhB1oolF5SdJzbvJ8JZFX9AWhFK2dg+VpW6z8NunfxkToua6EbtCwkh91vd2sS/cOh YNInb0upbLKqPMPlycuxotFBUnoKB4WP5Ti3xwJDmeufPCzmL8hTQYUxhkTNp/vNTVl6 jpiA== MIME-Version: 1.0 X-Received: by 10.153.11.132 with SMTP id ei4mr1766493lad.24.1412289469514; Thu, 02 Oct 2014 15:37:49 -0700 (PDT) Received: by 10.114.161.134 with HTTP; Thu, 2 Oct 2014 15:37:49 -0700 (PDT) Date: Thu, 2 Oct 2014 15:37:49 -0700 Message-ID: Subject: Mirrored SSDs for ZIL/SLOG - safety, flushing, capacitors From: javocado To: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Oct 2014 22:37:52 -0000 I'm setting up a SLOG to improve performance on my ZFS-based fileserver. I plan to use 2 mirrored SSD's to create the log device. Before I continue, I have some specific questions how to approach which hardware to use. Primarily, I'm concerned with what damage/corruption may occur when there's a power loss. 1. if some data is not successfully written to the SSD-SLOG device as a result of a power loss, is this a huge deal? Or is it just a loss of some non-fatal meta data? to help avoid any loss: 2. should I only use SSDs with capacitors? does anyone recommend any? I'm seeing good things about the Intel s3500 3. if only one of the SSDs has a capacitor (I plan to use 2 different SSDs to avoid simultaneous/similar failure rates) what might that do to the SLOG mirror after a power loss? Will it know which SSD had the capacitor and thus slightly-newer data to remirror from? 4. flushing: does anyone know if (and which) SSDs are respecting and flushing data to media when told? Thanks! From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 05:13:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05BEB48F for ; Fri, 3 Oct 2014 05:13:37 +0000 (UTC) Received: from mail-pa0-x231.google.com (mail-pa0-x231.google.com [IPv6:2607:f8b0:400e:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6B8E686 for ; Fri, 3 Oct 2014 05:13:36 +0000 (UTC) Received: by mail-pa0-f49.google.com with SMTP id hz1so936901pad.8 for ; Thu, 02 Oct 2014 22:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=7ixCAG5tLo8cXRtabfO3MKyHnN2YwaCWez3lA1VMMz4=; b=jUKqrzDYFRom0kON1y0iXjTvIgDNMRaMJmKl0Llm9BgjMenjiwRX8cTPpmBhqSXxZR GB4IrY1DoSdJ+BM817LD978jHzQ5ePYZWqe8r065XdqervK1XY2o76TqVQREz+foTaRe r93FB16cij4MyEbpdKErrHO7y+sCrTMrOONZez+5BLqP65pAW8eSSa+TOgUsjK0YICLG jHg9+QE1XQfojHfSs9VjIHMIl+FrP0pikrWLsHohPJry4jzDCCWdIprr4wJ6IiPy9BHm LhBdSLyS8HaIzvnrVcoLjxS4kCybaGfx2yYGbZVhILz+Bg4cCPDesNnvBenfv6DhET+s wZtA== X-Received: by 10.67.23.13 with SMTP id hw13mr4581277pad.69.1412313216189; Thu, 02 Oct 2014 22:13:36 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id v11sm5598300pas.24.2014.10.02.22.13.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 02 Oct 2014 22:13:35 -0700 (PDT) From: John Terrell Message-Id: <4526E82D-8AEF-4A3F-B628-C26D044E331E@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Date: Thu, 2 Oct 2014 22:13:33 -0700 References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> <2CAB03AD-B755-4F85-A6CA-E30B76517269@gmail.com> To: "freebsd-fs@freebsd.org" In-Reply-To: <2CAB03AD-B755-4F85-A6CA-E30B76517269@gmail.com> X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 05:13:37 -0000 Ok, I=92ve confirmed this isn=92t a true ZFS send/receive issue - it is, = in fact, an environment issue here. There=92s a switch in my = environment that a few times a day goes put to lunch (disconnects) for = 5-10 seconds. This disconnection was enough to trip up the = send/receive process but not enough to disrupt the SSH session that = initiated the transfer. =20 Thanks for all of the help and sorry to randomize the discussion. On Oct 1, 2014, at 6:57 AM, John Terrell wrote: > Eureka! I think I=92ve found the issue and it=92s nothing to do with = ZFS - it=92s an infrastructure problem. I was running another test = last night and just happened to be watching about two hours into the = test when the Mac I use to connect to the network suddenly (and = unexpectedly) lost connectivity for about five seconds. When it came = back, the SSH connection was intact but the ZFS transfer in progress = immediately failed. I=92m not sure if this is a DHCP lease = re-aquisition or what but to see if this is indeed the problem, I gave = the Mac a manual IP address and started the transfer over night. As of = now, it=92s been running for about 7 hours without issue. I=92m not = quite ready to call it solved until this transfer completes successfully = but this is the furthest it=92s gotten. >=20 >=20 > On Sep 30, 2014, at 8:02 PM, John Terrell = wrote: >=20 >> Sure. On the SmartOS (receiving) side: >>=20 >> [root@smartos /]# zpool get all zones | grep feature@ >> zones feature@async_destroy enabled = local >> zones feature@empty_bpobj active = local >> zones feature@lz4_compress active = local >> zones feature@multi_vdev_crash_dump enabled = local >> zones feature@spacemap_histogram active = local >> zones feature@enabled_txg active = local >> zones feature@hole_birth active = local >> zones feature@extensible_dataset enabled = local >> zones feature@embedded_data active = local >> zones feature@bookmarks enabled = local >> zones feature@filesystem_limits enabled = local >>=20 >> =85on the FreeBSD 10 (sending) side: >>=20 >> [root@freebsd /]# zpool get all tank | grep feature@ >> tank feature@async_destroy enabled = local >> tank feature@empty_bpobj active = local >> tank feature@lz4_compress active = local >> tank feature@multi_vdev_crash_dump enabled = local >>=20 >>=20 >> So, the FreeBSD side is a subset of the SmartOS side. >>=20 >>=20 >> On Sep 30, 2014, at 7:51 PM, Rich wrote: >>=20 >>> It's possible that your feature flags and the receiving end's aren't >>> compatible - can you show us a zfs get all tank on both sides? >>>=20 >>> - Rich >>>=20 >>> On Tue, Sep 30, 2014 at 10:39 PM, John Terrell = wrote: >>>> By the way, here's another report that sounds similar to mine = (though on Linux): >>>>=20 >>>> = http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway >>>>=20 >>>> His repro doesn't even use ssh (appears to be completely local). >>>>=20 >>>>=20 >>>> On Sep 30, 2014, at 6:41 PM, John Terrell = wrote: >>>>=20 >>>>> Sorry for the tardy response (busy at work. :)). Here's the = command and output (run from the SmartOS box): >>>>>=20 >>>>> ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | = zfs receive -v -F zones/tank >>>>>=20 >>>>> The output (after running for a long time xferring data): >>>>>=20 >>>>> receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 >>>>> received 1.35GB stream in 15 seconds (91.9MB/sec) >>>>> receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 >>>>> Read from remote host nas00: Connection reset by peer >>>>> cannot receive new filesystem stream: invalid backup stream >>>>>=20 >>>>>=20 >>>>> Is it possible the command is simply timing out for some reason = and closing the connection? >>>>>=20 >>>>>=20 >>>>> On Sep 26, 2014, at 3:04 PM, Will Andrews = wrote: >>>>>=20 >>>>>> What do the zfs commands print? The -v option prints some = metadata that might help diagnose the problem. >>>>>>=20 >>>>>> --Will. >>>>>>=20 >>>>>> On Monday, September 22, 2014, John Terrell = wrote: >>>>>> Hello all, I'm trying to replicate one of the ZFS filesystems = that lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. = The transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>>>>>=20 >>>>>> "cannot receive incremental stream: invalid backup stream" >>>>>>=20 >>>>>> The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >>>>>>=20 >>>>>> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive = -v tank >>>>>>=20 >>>>>> I've also tried to use mbuffer as the transport interface = (removing SSH from the equation) and it has the same result. >>>>>>=20 >>>>>> Is the possibly an incompatibility between FreeBSD10 and SmartOS = ZFS implementations? >>>>>> _______________________________________________ >>>>>> freebsd-fs@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >=20 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 11:59:50 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C048CE0 for ; Fri, 3 Oct 2014 11:59:50 +0000 (UTC) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4AF40146 for ; Fri, 3 Oct 2014 11:59:50 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.9/8.14.9) with ESMTP id s93BxmDd036487 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 3 Oct 2014 05:59:49 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.9/8.14.9/Submit) with ESMTP id s93BxmN5036482; Fri, 3 Oct 2014 05:59:48 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Fri, 3 Oct 2014 05:59:48 -0600 (MDT) From: Warren Block To: javocado Subject: Re: Mirrored SSDs for ZIL/SLOG - safety, flushing, capacitors In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (BSF 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Fri, 03 Oct 2014 05:59:49 -0600 (MDT) Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 11:59:50 -0000 On Thu, 2 Oct 2014, javocado wrote: > I'm setting up a SLOG to improve performance on my ZFS-based fileserver. I > plan to use 2 mirrored SSD's to create the log device. Before I continue, > I have some specific questions how to approach which hardware to use. > > Primarily, I'm concerned with what damage/corruption may occur when there's > a power loss. Do you have a good UPS? > 2. should I only use SSDs with capacitors? does anyone recommend any? I'm > seeing good things about the Intel s3500 I think the Crucial M550 SSDs have additional backup capacitors also. However, I'm still skeptical about the value of a small SSD power backup versus a true UPS for the entire system. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 14:34:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B56F6D4 for ; Fri, 3 Oct 2014 14:34:37 +0000 (UTC) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F375A651 for ; Fri, 3 Oct 2014 14:34:36 +0000 (UTC) Received: from gjp by mail.in-addr.com with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Xa3wI-0004uE-Op; Fri, 03 Oct 2014 15:34:34 +0100 Date: Fri, 3 Oct 2014 15:34:34 +0100 From: Gary Palmer To: Warren Block Subject: Re: Mirrored SSDs for ZIL/SLOG - safety, flushing, capacitors Message-ID: <20141003143434.GB41347@in-addr.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gjp@in-addr.com X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 14:34:37 -0000 On Fri, Oct 03, 2014 at 05:59:48AM -0600, Warren Block wrote: > On Thu, 2 Oct 2014, javocado wrote: > > > I'm setting up a SLOG to improve performance on my ZFS-based fileserver. I > > plan to use 2 mirrored SSD's to create the log device. Before I continue, > > I have some specific questions how to approach which hardware to use. > > > > Primarily, I'm concerned with what damage/corruption may occur when there's > > a power loss. > > Do you have a good UPS? > > > 2. should I only use SSDs with capacitors? does anyone recommend any? I'm > > seeing good things about the Intel s3500 > > I think the Crucial M550 SSDs have additional backup capacitors also. > However, I'm still skeptical about the value of a small SSD power backup > versus a true UPS for the entire system. The point of the supercap is all SSDs (even the ones without proper power protection) have a small amount of DRAM which they use for write optimisation. The cap is just there to let the SSD dump the DRAM to flash if the power dies. Even with a good UPS you should still get SSDs with power protection in case of PSU failures or people knocking out power cords or something. Regards, Gary From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 17:54:46 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A6703D3; Fri, 3 Oct 2014 17:54:46 +0000 (UTC) Received: from mail-lb0-x22e.google.com (mail-lb0-x22e.google.com [IPv6:2a00:1450:4010:c04::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84B59EBD; Fri, 3 Oct 2014 17:54:45 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id p9so1444170lbv.33 for ; Fri, 03 Oct 2014 10:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=m0UBa/fFCjwQWBiIUMUk/UZVON2J+MYYiA6O4yM9X+c=; b=yxNIwewZhE5qH3BlHsjaq8RFRtdLzpA2L56nboc+Sm6LbG1F3UEMzsxsefDSE3VYfz PTEhZOegePRYb6TUErP3Jp9nu3ZhkG5JDcPOXjtKI5Rhhw49nmepWzVWSa1/V0326AJD su5/xEWG147rmu0qJIU3feFAPd8ybH3rdqo8HgsBn3EVt30AJYYlOYe25mBSnB9qfDA3 H654ao4ylqTBHcbfJk55MtIyjWOHRyvgaV/tVuz1086Wa58+JwMeItNy4toC7APtNlWn Oh2xtkrvohp0KOb49/vR6vVnIkx9THwPeeT2a7rJleYtyaMKKBXgqUNRNVZhqPb51wZg /ZxQ== X-Received: by 10.152.206.35 with SMTP id ll3mr7585332lac.88.1412358883470; Fri, 03 Oct 2014 10:54:43 -0700 (PDT) Received: from localhost ([91.245.78.254]) by mx.google.com with ESMTPSA id l13sm2901099lbh.32.2014.10.03.10.54.42 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Oct 2014 10:54:42 -0700 (PDT) Date: Fri, 3 Oct 2014 20:54:40 +0300 From: Mikolaj Golub To: Matt Churchyard Subject: Re: HAST with broken HDD Message-ID: <20141003175439.GA7664@gmail.com> References: <542BC135.1070906@Skynet.be> <542BDDB3.8080805@internetx.com> <542BF853.3040604@internetx.com> <542C019E.2080702@internetx.com> <542C0710.3020402@internetx.com> <97aab72e19d640ebb65c754c858043cc@SERVER.ad.usd-group.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <97aab72e19d640ebb65c754c858043cc@SERVER.ad.usd-group.com> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 17:54:46 -0000 On Wed, Oct 01, 2014 at 03:51:43PM +0000, Matt Churchyard wrote: > HAST is basically "RAID1-over-network", so if a disk fails, it > should just handle read/writes using the other disk, and the > filesystem on top, be it UFS/ZFS/whatever, should just carry on as > normal (which is what has been observed). Of course, HAST (or the > OS) should notify you of the disk error though (probably through > devd) so you can do something about it. Maybe it already exists, but > HAST should be able to provide overall status information and raise > events just like ZFS or any RAID subsystem would. You also of course > shouldn't get scrub errors and corruption like that seen in the > original post either just because one half of the HAST mirror has > gone. Disk errors are recorded to syslog. Also error counters are displayed in `hastctl list' output. There is snmp_hast(3) in base -- a module for bsnmp to retrieve this statistics via snmp protocol (traps are not supported though). For notifications, the hastd can be configured to execute an arbitrary command on various HAST events (see description for `exec' in hast.conf(5)). Unfortunately, it does not have hooks for I/O error events currently. It might be worth adding though. The problem with this that it may generate to many events, so some throttling is needed. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 21:50:13 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 829E0DC0 for ; Fri, 3 Oct 2014 21:50:13 +0000 (UTC) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2001:470:8b2d:1e1c:21b:21ff:feb8:d7b0]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "khavrinen.csail.mit.edu", Issuer "Client CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 468E5B9C for ; Fri, 3 Oct 2014 21:50:13 +0000 (UTC) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s93LoBfA025619 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA) for ; Fri, 3 Oct 2014 17:50:11 -0400 (EDT) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.14.7/8.14.7/Submit) id s93LoBAu025616; Fri, 3 Oct 2014 17:50:11 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21551.6675.236161.476053@khavrinen.csail.mit.edu> Date: Fri, 3 Oct 2014 17:50:11 -0400 From: Garrett Wollman To: freebsd-fs@freebsd.org Subject: 9.3 NFS client bug? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (khavrinen.csail.mit.edu [127.0.0.1]); Fri, 03 Oct 2014 17:50:11 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 21:50:13 -0000 I've been doing some NFS testing, and found some issues with the NFS client in 9.3. This is pretty easily reproducible with bonnie++, but I haven't tried to reduce the size of the workload to make it easier to test or trace. ##### NFSv3 mount, 9.3 client from 9.3 server # bonnie++ -s 0 -n 1024:65536:0:1023:16384 -u 4294967294 -D Using uid:-2, gid:65533. Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...Bonnie: drastic I/O error (rmdir): Directory not empty Cleaning up test directory after error. Identical behavior is observed for a 9.3 client talking to a 9.1 server, and for NFSv4 mounts, but NOT for an Ubuntu 14.04 client talking to the 9.3 server using either NFSv3 or NFSv4. I haven't tried using a 9.2 or 9.1 client (yes, I still have some machines running 9.1 here) to see if the issue is a recently introduced one, nor have I traced the client or server code to see what's actually going on. Has anyone seen this before? -GAWollman From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 22:10:24 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8BE5348E for ; Fri, 3 Oct 2014 22:10:24 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 52FBDDE7 for ; Fri, 3 Oct 2014 22:10:23 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqwEAJ4dL1SDaFve/2dsb2JhbABgDoNTWASCfsgECoZ5VAKBJAF7hAMBAQEDAQEBASArIAsFFhgCAg0ZAikBCSYGCAcEARwEiBUIDakblgoBF4EsjjEBARs0B4J3gVQFljCCQYFJhGyUGIFxGYEZXCEvB4EIOYECAQEB X-IronPort-AV: E=Sophos;i="5.04,650,1406606400"; d="scan'208";a="159433771" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 03 Oct 2014 18:10:16 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id D5608B409E; Fri, 3 Oct 2014 18:10:16 -0400 (EDT) Date: Fri, 3 Oct 2014 18:10:16 -0400 (EDT) From: Rick Macklem To: Garrett Wollman Message-ID: <1265026957.57440085.1412374216862.JavaMail.root@uoguelph.ca> In-Reply-To: <21551.6675.236161.476053@khavrinen.csail.mit.edu> Subject: Re: 9.3 NFS client bug? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 22:10:24 -0000 Garrett Wollman wrote: > I've been doing some NFS testing, and found some issues with the NFS > client in 9.3. This is pretty easily reproducible with bonnie++, but > I haven't tried to reduce the size of the workload to make it easier > to test or trace. > > ##### NFSv3 mount, 9.3 client from 9.3 server > # bonnie++ -s 0 -n 1024:65536:0:1023:16384 -u 4294967294 -D > Using uid:-2, gid:65533. > Create files in sequential order...done. > Stat files in sequential order...done. > Delete files in sequential order...Bonnie: drastic I/O error (rmdir): > Directory not empty > Cleaning up test directory after error. > > Identical behavior is observed for a 9.3 client talking to a 9.1 > server, and for NFSv4 mounts, but NOT for an Ubuntu 14.04 client > talking to the 9.3 server using either NFSv3 or NFSv4. > A couple of things to try: 1 - I suspect you are using TCP, but you might want to check this. 2 - Try an "oldnfs" mount and see if the old client exhibits the same behaviour. 3 - If you could disable the "Cleaning up test..." after the failure, it would be nice to check and see if "ls" on the directory shows any files. Then, unmount/remount it and do the "ls" again. 4 - Capture the packets for wireshark. If you can just capture the "Delete files..." it would make the packet capture a lot smaller. rick ps: There is one difference between the old and new NFS client's readdir code that I am aware of. It was inherited from OpenBSD and I left it in, but it can be safely removed for FreeBSD. I have no reason to believe it would cause a problem, but I can easily generate a patch for you to test, if the "oldnfs" mount doesn't exhibit the problem. > I haven't tried using a 9.2 or 9.1 client (yes, I still have some > machines running 9.1 here) to see if the issue is a recently > introduced one, nor have I traced the client or server code to see > what's actually going on. > > Has anyone seen this before? > > -GAWollman > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Sat Oct 4 21:20:28 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1EA94CEE; Sat, 4 Oct 2014 21:20:28 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id C7A4B7DE; Sat, 4 Oct 2014 21:20:27 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAGJjMFSDaFve/2dsb2JhbABfhD2Cfs96gR4Be4QtVjUCDRkCX4hRqX6VMAEXgSyOTBk0gj0PMhKBQgWqEIkug38hgTZBgQIBAQE X-IronPort-AV: E=Sophos;i="5.04,654,1406606400"; d="scan'208";a="158384895" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 04 Oct 2014 17:20:19 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B2B70AE923; Sat, 4 Oct 2014 17:20:19 -0400 (EDT) Date: Sat, 4 Oct 2014 17:20:19 -0400 (EDT) From: Rick Macklem To: FreeBSD Filesystems Message-ID: <1692537569.57810384.1412457619725.JavaMail.root@uoguelph.ca> Subject: VFS_QUOTACTL() not implemented for ZFS MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Oct 2014 21:20:28 -0000 Hi, One of the changes in r272467 is enabling support for the NFSv4 quota attributes by default. (Prior to r272467, they were only enabled when a kernel was built with "options QUOTA".) araujo@ noted that this was because quotas are enabled by default for ZFS. Not being a ZFS guy, I took a look and it appears that VFS_QUOTACTL() isn't implemented for ZFS. This brings me to the question of what should NFSv4 do? Is there a way for the NFSv4 server to acquire/set quotas for ZFS? Or put another way, can zfs_quotactl() be implemented? If not, I suspect that NFSv4 shouldn't support quota attributes for ZFS volumes. I guess the NFSv4 server can determine this by seeing if VFS_QUOTACTL() returns EOPNOTSUPP. So, does anyone have thoughts on how NFSv4 should handle ZFS quotas? Thanks in advance for any comments, rick