From owner-freebsd-fs@FreeBSD.ORG Sun Oct 20 03:28:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 15E6129F; Sun, 20 Oct 2013 03:28:49 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CFEA821AB; Sun, 20 Oct 2013 03:28:48 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id u16so9435627iet.32 for ; Sat, 19 Oct 2013 20:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=VddClJMAVwl0gn8GrejKNWNkM/WZUeDaTKl0UZKFyX4=; b=kJlckGoEU1KhOYZjmTBj7lLOsCIbzxT7h/WD7yxihn6LJw8jSUY0O+4HwSKOc5AXSI JxtLBZUMR8lH+9Z2YEswJShOEU1xRIWFxbWmsMU96DFqsyBHf3grMmpWdpf3ZY/1UTHy 6hIlLnwnoSnabE1TohCj8Bi4eo8aoCuIFyGGUITuyt88Fr5QuZCTmc8jRN7xybhfxYRF vF+7w5gfrFltZ7tJdYE3g5xhq8EPePBj5tuPyI02UBpMhFptawfDifGYfbA1ZnLqBpI6 FXTp7tS+3Rg1tEoPyI89Gg+Bsoyx9u4O6vNyXokiZKMIq0x3La1l4dWny//1FDlChfFw 6D9Q== MIME-Version: 1.0 X-Received: by 10.50.73.66 with SMTP id j2mr4947074igv.1.1382239727719; Sat, 19 Oct 2013 20:28:47 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Sat, 19 Oct 2013 20:28:47 -0700 (PDT) In-Reply-To: <20131017070949.GA3865@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> Date: Sat, 19 Oct 2013 23:28:47 -0400 X-Google-Sender-Auth: QSaTKPjLLlRT7S2IhJntiUGoU40 Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Oct 2013 03:28:49 -0000 On Thu, Oct 17, 2013 at 3:09 AM, Konstantin Belousov wrote: > As I said you before, obtain the core and get backtrace from kgdb. Getting these machines to dump is not as easy as you may imagine. :( But it finally was accomplished. Albeit it was a different panic: panic: mutex not owned at /usr/src/sys/vm/vm_page.c:520 cpuid = 1 #0 doadump (textdump=1) at pcpu.h:249 #1 0x80ab0f98 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449 #2 0x80ab1448 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:637 #3 0x80a9cad8 in _mtx_assert (m=0x0, what=, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:741 #4 0x80d364fd in vm_page_busy (m=0x85198548) at /usr/src/sys/vm/vm_page.c:520 #5 0x80d23723 in vm_fault_hold (vaddr=3, fault_type=7 '\a', fault_flags=, m_hold=) at /usr/src/sys/vm/vm_fault.c:406 #6 0x80d23382 in vm_fault (map=0x81feb000, vaddr=, fault_type=0 '\0', fault_flags=) at /usr/src/sys/vm/vm_fault.c:229 #7 0x80f12aae in trap_pfault (frame=0x876dfb04, usermode=0, eva=) at /usr/src/sys/i386/i386/trap.c:934 #8 0x80f1213c in trap (frame=0x876dfb04) at /usr/src/sys/i386/i386/trap.c:555 #9 0x80efb43c in calltrap () at /usr/src/sys/i386/i386/exception.s:170 #10 0x80a98bfe in free (addr=0x8a3a2c00, mtp=0x818dc100) at /usr/src/sys/kern/kern_malloc.c:565 #11 0x818dae49 in zfs_kmem_free (buf=0x8a3a2c00, size=320) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:112 #12 0x8177f329 in dbuf_clear (db=0x8c322660) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1524 #13 0x8177f2a0 in dbuf_evict (db=0x8c322660) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:246 #14 0x81798ed2 in dnode_destroy (dn=0x8ab7c214) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:465 #15 0x817995ef in dnode_buf_pageout (db=0x8b9035d8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:998 #16 0x8177f2ed in dbuf_clear (db=0x8b9035d8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:217 #17 0x81784361 in dbuf_do_evict (private=0x9241bcd0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:246 #18 0x81775b92 in arc_do_user_evicts () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2306 #19 0x81779d48 in arc_reclaim_thread (dummy=0x64) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2554 #20 0x80a7febc in fork_exit (callout=0x81779a20 , arg=0x8ab71d50, frame=0x8a3a2c00) at /usr/src/sys/kern/kern_fork.c:992 #21 0x80efb4e4 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279 If you need additional commands run or source verified, just describe what you want. If the 4mb panic can be dumped, I will try to report that as well when it happens. Thanks! From owner-freebsd-fs@FreeBSD.ORG Sun Oct 20 13:02:26 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BDF41BF6; Sun, 20 Oct 2013 13:02:26 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 83A192798; Sun, 20 Oct 2013 13:02:26 +0000 (UTC) Received: by mail-ie0-f174.google.com with SMTP id qd12so9613867ieb.5 for ; Sun, 20 Oct 2013 06:02:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ewKGh6jLGYVzaVJ9gy6u6+KEmYulMqr2U+5vFwPQwTw=; b=xyO+rfm3wm0jt+yuNL1uiIPU+b6hsu7WbtFX7dtJspPqtkZ0yE6NJqE8dq/+mFAX+8 T/T1JSZpfl19SfpbHwfwA1SDxvCtK0e3LVp8oOsmKTvWxNwfPCChuideaTUvjZoFMSMd hLCV2TWdoYGtZecZTzDqETuJelwyw+wUb+mm3kfYdhW5HDurlQMQU8yZ5WkqBzpJ8/y8 riPqnOyYPJ3A4p6C7dJjojwMPt/Qdih4Hrf3iFxobbb0mdI98ZkgLjLihj9FG+RDhiu+ smuU9BkaicWW8N0LMFMHr6q44p3uRMCamH5WDSSWZKg7hWmyStR9RIPyPFGh1dYpaWf+ 3pvw== MIME-Version: 1.0 X-Received: by 10.50.1.78 with SMTP id 14mr5982942igk.37.1382274145894; Sun, 20 Oct 2013 06:02:25 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Sun, 20 Oct 2013 06:02:25 -0700 (PDT) In-Reply-To: References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> Date: Sun, 20 Oct 2013 09:02:25 -0400 X-Google-Sender-Auth: -T5U-rNEe3vtvACsX_uPgaPOzDs Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Oct 2013 13:02:26 -0000 The "attempted pmap_enter on 4MB page" panic occurred again. It says the vmcore "isn't a dump file" for some reason but the core.txt file contains this, which looks right and has source line numbers: #0 doadump (textdump=1) at pcpu.h:249 249 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:249 #1 0x80ab0f98 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449 #2 0x80ab1448 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:637 #3 0x80f0b5bc in pmap_enter (pmap=0x81487c78, va=2296995840, m=, prot=0 '\0') at /usr/src/sys/i386/i386/pmap.c:3532 #4 0x80d24f01 in vm_fault_hold (vaddr=808648704, fault_type=7 '\a', fault_flags=, m_hold=) at /usr/src/sys/vm/vm_fault.c:935 #5 0x80d23382 in vm_fault (map=0x81feb000, vaddr=, fault_type=0 '\0', fault_flags=) at /usr/src/sys/vm/vm_fault.c:229 #6 0x80f12aae in trap_pfault (frame=0x8774d9f4, usermode=0, eva=) at /usr/src/sys/i386/i386/trap.c:934 #7 0x80f1213c in trap (frame=0x8774d9f4) at /usr/src/sys/i386/i386/trap.c:555 #8 0x80efb43c in calltrap () at /usr/src/sys/i386/i386/exception.s:170 #9 0x81759322 in avl_last (tree=0x88c5c4e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:194 #10 0x817bed4b in metaslab_df_alloc (sm=0x885c4b80, size=2560) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:493 #11 0x817d8b83 in space_map_alloc (sm=, size=2560) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c:420 #12 0x817bd9d0 in metaslab_alloc (spa=0x30430000, mc=0x20, psize=3825868744137834496, bp=0x3769, txg=4344284791469325298, hintbp=0x7ae28e0, flags=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1478 #13 0x8180676f in zio_dva_allocate (zio=0x883ca8b8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2338 #14 0x81803c46 in zio_execute (zio=0x883ca8b8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1305 #15 0x81765380 in taskq_run_safe (arg=0x883cab70, pending=1) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:150 #16 0x80af4ca1 in taskqueue_run_locked (queue=0x883b05c0) at /usr/src/sys/kern/subr_taskqueue.c:312 #17 0x80af54a9 in taskqueue_thread_loop (arg=0x882e7170) at /usr/src/sys/kern/subr_taskqueue.c:501 #18 0x80a7febc in fork_exit (callout=0x80af5420 , arg=0x88e96c44, frame=0x10) at /usr/src/sys/kern/kern_fork.c:992 #19 0x80efb4e4 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279 Hopefully this is helpful. Thanks! From owner-freebsd-fs@FreeBSD.ORG Sun Oct 20 22:03:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F41596BA for ; Sun, 20 Oct 2013 22:03:26 +0000 (UTC) (envelope-from public@macfreek.nl) Received: from govert.macfreek.nl (govert.macfreek.nl [IPv6:2a01:238:43ed:a300:ea00:54c2:fb7a:7e66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B207A23A9 for ; Sun, 20 Oct 2013 22:03:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by govert.macfreek.nl (Postfix) with ESMTP id 971CD64A0B4 for ; Mon, 21 Oct 2013 00:03:23 +0200 (CEST) Received: from govert.macfreek.nl ([127.0.0.1]) by localhost (govert.macfreek.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yUGOxChRzo7I for ; Mon, 21 Oct 2013 00:03:23 +0200 (CEST) Received: from 77-175-100-208.ftth.ispfabriek.nl (77-175-100-208.FTTH.ispfabriek.nl [77.175.100.208]) by govert.macfreek.nl (Postfix) with ESMTPSA id 877D564A0B3 for ; Mon, 21 Oct 2013 00:03:22 +0200 (CEST) Message-ID: <52645329.3000806@macfreek.nl> Date: Mon, 21 Oct 2013 00:03:21 +0200 From: Freek Dijkstra User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: ZFS: unsupported ZFS version 5000 (should be 28) Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Oct 2013 22:03:27 -0000 Hi, I just installed FreeBSD 9.2 on my host. It has two mirrored SSD disks, running ZFS. All seemed to worked fine, except that the device does not boot. Right after booting, it shows the error: ZFS: unsupported ZFS version 5000 (should be 28) ZFS: unsupported ZFS version 5000 (should be 28) gptzfsboot: No ZFS pools located, can't boot Previously I used FreeBSD 9.0 and had no problems after a fresh install. Am I correct that the above error means that the ZFS parition uses zpool version 5000 (the 'features-flag' version), while the bootloader only understands zpool version 28, and hence can't boot. I did install the bootcode using # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 It almost seems that the bootloader contained on the memdisk is out of date. I once again booted from a 9.2 memdisk, executed this command, and rebooted. It gave the same error. I also tried a 9.0 memdisk and 10.0-BETA1 memdisk, but got the same error again. Two questions: * Is the problem indeed an old version of the bootloader? * Is there a recent version of the bootloader and where exactly can I donwload and install it? FYI, this is how I installed FreeBSD 9.2. I used Downloads/FreeBSD-9.2-RELEASE-i386-memstick.img and at the partitioning stage, dropped to the shell and executed these commands: # gpart create -s gpt ada0 # gpart add -b 34 -s 94 -t freebsd-boot ada0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 # gpart add -t freebsd-zfs -l ssd0 ada0 # gpart create -s gpt ada1 # gpart add -b 34 -s 94 -t freebsd-boot ada1 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 # gpart add -t freebsd-zfs -l ssd1 ada1 # zpool create -R /mnt zroot mirror /dev/gpt/ssd0 /dev/gpt/ssd1 # zfs set mountpoint=/ zroot # zfs set cachefile=none zroot # zpool set bootfs=zroot zroot Last, I created /boot/loader.conf to contain: zfs_load="YES" vfs.root.mountfrom="zfs:zroot" I am aware that this topic was previously discussed, but that discussion focussed on how to downgrade the zpool, while I prefer to upgrade the bootloader, if at all possible, Regards, Freek Dijkstra From owner-freebsd-fs@FreeBSD.ORG Mon Oct 21 00:21:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 95746A9C for ; Mon, 21 Oct 2013 00:21:11 +0000 (UTC) (envelope-from public@macfreek.nl) Received: from govert.macfreek.nl (govert.macfreek.nl [IPv6:2a01:238:43ed:a300:ea00:54c2:fb7a:7e66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 50B942B9E for ; Mon, 21 Oct 2013 00:21:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by govert.macfreek.nl (Postfix) with ESMTP id 6FD6A64A0B4 for ; Mon, 21 Oct 2013 02:21:08 +0200 (CEST) Received: from govert.macfreek.nl ([127.0.0.1]) by localhost (govert.macfreek.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z6CJQbw7NJ8P for ; Mon, 21 Oct 2013 02:21:07 +0200 (CEST) Received: from 77-175-100-208.ftth.ispfabriek.nl (77-175-100-208.FTTH.ispfabriek.nl [77.175.100.208]) by govert.macfreek.nl (Postfix) with ESMTPSA id 4C14A64A0B3 for ; Mon, 21 Oct 2013 02:21:07 +0200 (CEST) Message-ID: <52647372.7000203@macfreek.nl> Date: Mon, 21 Oct 2013 02:21:06 +0200 From: Freek Dijkstra User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) References: <52645329.3000806@macfreek.nl> In-Reply-To: <52645329.3000806@macfreek.nl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 00:21:11 -0000 On 21-10-2013 00:03, Freek Dijkstra wrote: > ZFS: unsupported ZFS version 5000 (should be 28) > gptzfsboot: No ZFS pools located, can't boot [...] > # gpart create -s gpt ada0 > # gpart add -b 34 -s 94 -t freebsd-boot ada0 > # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 I tried a full install again, and it seems that 94 blocks was too small for freebsd-boot. I just tried again, and things seem to work now. I am curious if this indeed was the cause. Is there a way to determine the required size of a bootloader? If this was indeed the cause (I need some sleep, so am not ruling anything out :) ), I guess it is useful to add a warning message to gpart. Regards, Freek From owner-freebsd-fs@FreeBSD.ORG Mon Oct 21 00:51:30 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1F0CB230 for ; Mon, 21 Oct 2013 00:51:30 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qe0-x231.google.com (mail-qe0-x231.google.com [IPv6:2607:f8b0:400d:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D7CC52D70 for ; Mon, 21 Oct 2013 00:51:29 +0000 (UTC) Received: by mail-qe0-f49.google.com with SMTP id a11so3377523qen.36 for ; Sun, 20 Oct 2013 17:51:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=GkqSxXr79Zf+uiKVvaWzZTFs0qM1buOlDck7opcYeKY=; b=YwgFdK1esqfIFvNDIIgm+sKf0lcO7S1Zj8/QeeplMr+KMjRUvol16Tgvw5TNgaBCiY iTPcoofPXODT5WGl5RNiMWxzRN4efmCP+4NiPig5num+bqNNNH2dXwaKKmbWeml4+ozk sfU9J+WBE0fQtEpejw+0KAkqr0v3kfcbRKL0LGzUSaT806sDGI4g4I9fgB+HLyYSiIJf CigtnyKWCiX3PRCRmt8ZIrv05VGSOE5Yao0ygoslqcEW+iOGLxN9LGmAe5qrYqUIcO6B mDxZ9dGxkWK3OZQmOrb92hTIeDF/YCgtqmF35AciMbxjxtEn/OFmeDsySw/LNZkImq3N 28eQ== MIME-Version: 1.0 X-Received: by 10.49.71.239 with SMTP id y15mr19313517qeu.14.1382316689051; Sun, 20 Oct 2013 17:51:29 -0700 (PDT) Received: by 10.96.180.233 with HTTP; Sun, 20 Oct 2013 17:51:28 -0700 (PDT) In-Reply-To: <52647372.7000203@macfreek.nl> References: <52645329.3000806@macfreek.nl> <52647372.7000203@macfreek.nl> Date: Mon, 21 Oct 2013 03:51:28 +0300 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Kimmo Paasiala To: Freek Dijkstra Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 00:51:30 -0000 On Mon, Oct 21, 2013 at 3:21 AM, Freek Dijkstra wrote: > On 21-10-2013 00:03, Freek Dijkstra wrote: > >> ZFS: unsupported ZFS version 5000 (should be 28) >> gptzfsboot: No ZFS pools located, can't boot > > [...] > >> # gpart create -s gpt ada0 >> # gpart add -b 34 -s 94 -t freebsd-boot ada0 >> # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 > > I tried a full install again, and it seems that 94 blocks was too small > for freebsd-boot. I just tried again, and things seem to work now. > > I am curious if this indeed was the cause. > Is there a way to determine the required size of a bootloader? > > If this was indeed the cause (I need some sleep, so am not ruling > anything out :) ), I guess it is useful to add a warning message to gpart. > > Regards, > Freek > You can look at the size of the /boot/gptzfsboot file to see how much space it will require. To be on the safe side you can always use a 256KB (512 blocks) partition or even larger for the boot loader. This allows room in case the loader suddenly grows in size in later versions. -Kimmo From owner-freebsd-fs@FreeBSD.ORG Mon Oct 21 08:34:31 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 50923F7F for ; Mon, 21 Oct 2013 08:34:31 +0000 (UTC) (envelope-from public@macfreek.nl) Received: from govert.macfreek.nl (govert.macfreek.nl [IPv6:2a01:238:43ed:a300:ea00:54c2:fb7a:7e66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1097024D9 for ; Mon, 21 Oct 2013 08:34:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by govert.macfreek.nl (Postfix) with ESMTP id F086C64A0BE; Mon, 21 Oct 2013 10:34:27 +0200 (CEST) Received: from govert.macfreek.nl ([127.0.0.1]) by localhost (govert.macfreek.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fe4W9I-khuAz; Mon, 21 Oct 2013 10:34:27 +0200 (CEST) Received: from saramac0003.local (unknown [145.100.42.205]) by govert.macfreek.nl (Postfix) with ESMTPSA id 1048D64A0BD; Mon, 21 Oct 2013 10:34:26 +0200 (CEST) Message-ID: <5264E710.9090209@macfreek.nl> Date: Mon, 21 Oct 2013 10:34:24 +0200 From: Freek Dijkstra User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: freebsd-fs Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) References: <52645329.3000806@macfreek.nl> <52647372.7000203@macfreek.nl> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 08:34:31 -0000 On 21-10-2013 02:51, Kimmo Paasiala wrote: > On Mon, Oct 21, 2013 at 3:21 AM, Freek Dijkstra wrote: >> On 21-10-2013 00:03, Freek Dijkstra wrote: >> >>> ZFS: unsupported ZFS version 5000 (should be 28) >>> gptzfsboot: No ZFS pools located, can't boot >> >> [...] >> >>> # gpart create -s gpt ada0 >>> # gpart add -b 34 -s 94 -t freebsd-boot ada0 >>> # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 >> >> I tried a full install again, and it seems that 94 blocks was too small >> for freebsd-boot. I just tried again, and things seem to work now. >> >> I am curious if this indeed was the cause. >> Is there a way to determine the required size of a bootloader? >> >> If this was indeed the cause (I need some sleep, so am not ruling >> anything out :) ), I guess it is useful to add a warning message to gpart. >> >> Regards, >> Freek >> > > You can look at the size of the /boot/gptzfsboot file to see how much > space it will require. To be on the safe side you can always use a > 256KB (512 blocks) partition or even larger for the boot loader. This > allows room in case the loader suddenly grows in size in later > versions. Thanks, your reply is much appreciated. I'll check later if indeed was the cause, but will use a different machines for that. (Now that this host boots fine, I prefer to get it up and running asap). Note that the FreeBSD handbook says the max size of the boot partition is 512kByte, so I'm going for that. Just a last check (in case anyone knows): Is this error: ZFS: unsupported ZFS version 5000 (should be 28) printed by the code present on the boot partition, or is it printed by /boot/loader on the regular disk. I presume the first. Freek From owner-freebsd-fs@FreeBSD.ORG Mon Oct 21 11:06:47 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3F30A9A3 for ; Mon, 21 Oct 2013 11:06:47 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2BD2B2E0F for ; Mon, 21 Oct 2013 11:06:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9LB6lgO018626 for ; Mon, 21 Oct 2013 11:06:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9LB6kw5018624 for freebsd-fs@FreeBSD.org; Mon, 21 Oct 2013 11:06:46 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 21 Oct 2013 11:06:46 GMT Message-Id: <201310211106.r9LB6kw5018624@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 11:06:47 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/182570 fs [zfs] [patch] ZFS panic in receive o kern/182536 fs [zfs] zfs deadlock o kern/181966 fs [zfs] Kernel panic in ZFS I/O: solaris assert: BP_EQUA o kern/181834 fs [nfs] amd mounting NFS directories can drive a dead-lo o kern/181565 fs [swap] Problem with vnode-backed swap space. o kern/181377 fs [zfs] zfs recv causes an inconsistant pool o kern/181281 fs [msdosfs] stack trace after successfull 'umount /mnt' o kern/181082 fs [fuse] [ntfs] Write to mounted NTFS filesystem using F o kern/180979 fs [netsmb][patch]: Fix large files handling o kern/180876 fs [zfs] [hast] ZFS with trim,bio_flush or bio_delete loc o kern/180678 fs [NFS] succesfully exported filesystems being reported o kern/180438 fs [smbfs] [patch] mount_smbfs fails on arm because of wr p kern/180236 fs [zfs] [nullfs] Leakage free space using ZFS with nullf o kern/178854 fs [ufs] FreeBSD kernel crash in UFS s kern/178467 fs [zfs] [request] Optimized Checksum Code for ZFS o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178387 fs [zfs] [patch] sparse files performance improvements o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/178238 fs [nullfs] nullfs don't release i-nodes on unlink. f kern/178231 fs [nfs] 8.3 nfsv4 client reports "nfsv4 client/server pr o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175449 fs [unionfs] unionfs and devfs misbehaviour o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/145750 fs [unionfs] [hang] unionfs locks the machine s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/137588 fs [unionfs] [lor] LOR nfs/ufs/nfs o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126973 fs [unionfs] [hang] System hang with unionfs and init chr o kern/126553 fs [unionfs] unionfs move directory problem 2 (files appe o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o bin/123574 fs [unionfs] df(1) -t option destroys info for unionfs (a o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o kern/121385 fs [unionfs] unionfs cross mount -> kernel panic o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/67326 fs [msdosfs] crash after attempt to mount write protected o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t o kern/9619 fs [nfs] Restarting mountd kills existing mounts 336 problems total. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 12:47:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3D03F4F8; Tue, 22 Oct 2013 12:47:01 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F2E112897; Tue, 22 Oct 2013 12:47:00 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id ar20so1602485iec.12 for ; Tue, 22 Oct 2013 05:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=UOusFWkUYC9/FWlzc3HJtodeGAW8fSQiaBaGyvH4gHw=; b=i3rQedzXXOEQ6x8Xw+S3I9+u/q97tp+FjnCpTnklf7R6ZX6gT/EsbsQupyo5Zg5b/o dhDPeqoeMOKCBWCnSNuIN2V0tcZEgvx7YBX3NV+qmZJnfZiTqZrlw4liQIP5UlscjPo5 fRfW/hGFHlt3mJK4Yva15goVXdj2PJdqbcJOIfIGMKbb40fvboL3Rk4SfkDTaqWqskaL Zkuew3XBTNxnGw8nxGwqSTFAlSU6JguX2VFBwuoUIykADDrXSYWAaf7e9PrmvZN7ZeB1 BNptYgf29Cfp2FbDAW73A3zyvIYCTyGHf4pqT2DurZBe3vS0M3awwubNik19Ps7Wfaw+ cJ2w== MIME-Version: 1.0 X-Received: by 10.50.117.40 with SMTP id kb8mr13064597igb.60.1382446020011; Tue, 22 Oct 2013 05:47:00 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Tue, 22 Oct 2013 05:46:59 -0700 (PDT) In-Reply-To: References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> Date: Tue, 22 Oct 2013 08:46:59 -0400 X-Google-Sender-Auth: wuW2TkyNh-kvhpoBAWLk3DHyjGM Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 12:47:01 -0000 On Sun, Oct 20, 2013 at 9:02 AM, J David wrote: > The "attempted pmap_enter on 4MB page" panic occurred again. Since a machine that crashes 3-4x a day is not useful, we have captured a few viable core images for this panic and have now set it to vm.pmap.pg_ps_enabled=0 in the hopes of stopping the crashes. If there is anything else we can do to aid the debugging, please let us know. Thanks! From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 17:09:00 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EAB4C858; Tue, 22 Oct 2013 17:09:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 0796A28B0; Tue, 22 Oct 2013 17:08:59 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA28756; Tue, 22 Oct 2013 20:08:49 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VYfRp-000NTd-6R; Tue, 22 Oct 2013 20:08:49 +0300 Message-ID: <5266B0E9.3040206@FreeBSD.org> Date: Tue, 22 Oct 2013 20:07:53 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: J David , Konstantin Belousov Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-fs@freebsd.org" , alc@FreeBSD.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 17:09:01 -0000 on 22/10/2013 15:46 J David said the following: > On Sun, Oct 20, 2013 at 9:02 AM, J David wrote: >> The "attempted pmap_enter on 4MB page" panic occurred again. > > Since a machine that crashes 3-4x a day is not useful, we have > captured a few viable core images for this panic and have now set it > to vm.pmap.pg_ps_enabled=0 in the hopes of stopping the crashes. > > If there is anything else we can do to aid the debugging, please let us know. Both panics you posted actually happened because of bad pointer dereference in ZFS. It appears that for some reason trap_pfault does not just call panic right there as it should, but instead attempts to "fault in" a corresponding page. All in all, these look like a memory corruption of some sort (or use after free). -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 17:36:09 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 371A6D64; Tue, 22 Oct 2013 17:36:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AD09B2A68; Tue, 22 Oct 2013 17:36:08 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9MHZvip051732; Tue, 22 Oct 2013 20:35:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9MHZvip051732 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9MHZviZ051731; Tue, 22 Oct 2013 20:35:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 22 Oct 2013 20:35:57 +0300 From: Konstantin Belousov To: Andriy Gapon Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131022173557.GB5208@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> <5266B0E9.3040206@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="nVMJ2NtxeReIH9PS" Content-Disposition: inline In-Reply-To: <5266B0E9.3040206@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-fs@freebsd.org" , alc@FreeBSD.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 17:36:09 -0000 --nVMJ2NtxeReIH9PS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 22, 2013 at 08:07:53PM +0300, Andriy Gapon wrote: > on 22/10/2013 15:46 J David said the following: > > On Sun, Oct 20, 2013 at 9:02 AM, J David wrot= e: > >> The "attempted pmap_enter on 4MB page" panic occurred again. > >=20 > > Since a machine that crashes 3-4x a day is not useful, we have > > captured a few viable core images for this panic and have now set it > > to vm.pmap.pg_ps_enabled=3D0 in the hopes of stopping the crashes. > >=20 > > If there is anything else we can do to aid the debugging, please let us= know. >=20 > Both panics you posted actually happened because of bad pointer > dereference in ZFS. > > It appears that for some reason trap_pfault does not just call panic > right there as it should, but instead attempts to "fault in" a > corresponding page. > > All in all, these look like a memory corruption of some sort (or use > after free). I just do not have time left for this right now. One thing that catched my eye initially are very strange pointers in the backtrace, like 'zio=3D0x883ca8b8'. The standard i386 address space map splits user/kernel mode at 3GB, so 0x883ca8b8 is definitely usermode, unless non-standard kernel config is used. This is why I asked for the gdb backtrace. why=20 --nVMJ2NtxeReIH9PS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSZrd8AAoJEJDCuSvBvK1BAi0P+gLUZtsigFQDptTNHDIglciM c53rszC89ItPH2tNlVrcz1MMkSGVTKDn92T/SV/js2Emq0TiVOgg1X3FNTxqxAvj wn4djXaonr/IxYnD4CaAyHSnCOQX5X+5j4waJD2dgKGpipAgQ0zk+pQkwFIhc65o Rx78sOJcT9gbmmloGEwmWC1gRiaaZe8xhF8UJWXSqLTPFXb6PzevGJiXreiKMZuA D3QKPE3ENWiUpUbILN8WHqDdZNXRGOHJvZrf4qHbUuFkpINqmHXqvElDFZvkcIK/ 3Yxzhwr8cmw9Os0wtwSBIqpx6FY2H62ClU34UGS1dMy8ql3t1/kuuHKHtGjBtk6i aBSChubFM8kKmt2qKKNcQ5QH4lgu58HChpWwpxV2uMuaTfSgJY3ZjLALdhnsMOao TfhyKZqd/H3FX2OxAJEQ1j73+/sxR80dvqG9YzB8G4nlaUTlIXatDuJFgvRoUV5y ae9gqnusecJWT6SQV2DpNfJSa7n5JBWWAdlTxIToactH4EeALB3jTmSwvY0CLEvb WaJnmfZSo9TTzFASk58XbtnUVP9np08fDh3k9pa0pPQ1W72p5M/UmzAu+LS2RUXx gVeKw7KECpDqS4puYGnO/ZSYqGkyeLLmQYuDtLOtlkx6CbZjOTLCtoAbhNYB4TCf W+aE9GQluVGF1gbC+hiH =S/ws -----END PGP SIGNATURE----- --nVMJ2NtxeReIH9PS-- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 19:42:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E3F7A636; Tue, 22 Oct 2013 19:42:08 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 932A322C6; Tue, 22 Oct 2013 19:42:08 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id pa12so5409986veb.6 for ; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Ot0BI8ZRT49L47kI9cMvxwUR/6atwDAxmJtjyfSH+dw=; b=rKLaMGttT/q/277VrNfFCJ08dURuMAevE/L5E3uVGD6Ny9qEmbJ18/d8qqd/7x/x/6 j0jAehxS2bMNTfVl6RCVHP5ViPsi4Bo6oaxeE0VCckkVs4xqSxXNSVZ3kT8u9STIRPT/ 0vnob+rKwq/xsU5qg8sXm7e9hFK8t8Py2lOG8u0zKv/CWiQwl8vhSihLcKE1JVJnZdOx cGOhOALCLlwbn+9nFisn5ux6GeZGZPq8sT0yrBMOv0z43HC9j16BpycbeqjOCnS27oD9 eSKeTCnNxvsdjTXzgSsL31AAxysY/WbV8vN2uh2/7aV0X8GqO3JjqdyQhJ7XU5nwRLzm 0fow== MIME-Version: 1.0 X-Received: by 10.58.207.15 with SMTP id ls15mr5917081vec.17.1382470927625; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) Received: by 10.220.30.130 with HTTP; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) Date: Tue, 22 Oct 2013 15:42:07 -0400 Message-ID: Subject: istgt causes massive jumbo nmbclusters loss From: Zaphod Beeblebrox To: FreeBSD Net , freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 19:42:09 -0000 I have a server FreeBSD virtual.accountingreality.com 9.2-STABLE FreeBSD 9.2-STABLE #13 r256549M: Tue Oct 15 16:29:48 EDT 2013 root@virtual.accountingreality.com:/usr/obj/usr/src/sys/VRA amd64 That has an em0 with jumbo packets enabled: em0: flags=8843 metric 0 mtu 9014 It has (among other things): ZFS, NFS, iSCSI (via istgt) and Samba. Every day or two, it looses it's ability to talk to the network. ifconfig down/up on em0 gives the message about not being able to allocate the receive buffers... With everything running, but with specifically iSCSI not used, everything seems good. When I start hitting istgt, I see the denied stat for 9k mbufs rise very rapidly (this amount only took a few seconds): [1:47:347]root@virtual:/usr/local/etc/iet> netstat -m 1313/877/2190 mbufs in use (current/cache/total) 20/584/604/523514 mbuf clusters in use (current/cache/total/max) 20/364 mbuf+clusters out of packet secondary zone in use (current/cache) 239/359/598/261756 4k (page size) jumbo clusters in use (current/cache/total/max) 1023/376/1399/77557 9k jumbo clusters in use (current/cache/total/max) 0/0/0/43626 16k jumbo clusters in use (current/cache/total/max) 10531K/6207K/16738K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/50199/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ... the denied number rises... and somewhere in the millions or more the machine stops --- but even with the large number of denied 9k clusters, the "9k jumbo clusters in use" line will always indicate some available. ... so is this a tuning or a bug issue? I've tried ietd --- basically it doesn't want to work with a zfs zvol, it seems (refuses to use it). From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 20:01:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C99FAF3E; Tue, 22 Oct 2013 20:01:02 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 833EF23E1; Tue, 22 Oct 2013 20:01:02 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id u16so2699758iet.32 for ; Tue, 22 Oct 2013 13:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Vh4biNud3Ob8HSuN7e3AveiPwI3SoQUNfsmlxw/J224=; b=ubQNqOybeYlo6xc09d19B523JryrmfSBMgGtRHDTHOPqQZCcXXstXkt4ZpgfqSHQT0 9OCPN2TSvnyszC1wRvfoNZ964pUPeCu6TQMrvpTpQHAmz+0iCoGfJZNwHiK9fNt2PTzq dzJaJ8r4ALH/+qTpoLfbJ7Hdh7xRiGV0Zt/cwUAzq92sk7QukJut7FFMGAkxUhpgYkev zyonzL/VEyiN6aSfXuc9ZqzEJcCKbkGGRebg3Bz+G+iT5dnogYCAv5TN8TK9OuDqtqYc eBodlmn1FZugr6gMmE2Mv5fp9Gqvm+2KP4hVvO4QhBNO0H8yLzVtrqGiVjeAdcLCewiS 9xxw== MIME-Version: 1.0 X-Received: by 10.42.126.18 with SMTP id c18mr2545737ics.46.1382472061355; Tue, 22 Oct 2013 13:01:01 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Tue, 22 Oct 2013 13:01:01 -0700 (PDT) In-Reply-To: <20131022173557.GB5208@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> <20131017070949.GA3865@kib.kiev.ua> <5266B0E9.3040206@FreeBSD.org> <20131022173557.GB5208@kib.kiev.ua> Date: Tue, 22 Oct 2013 16:01:01 -0400 X-Google-Sender-Auth: wFNG_kPwDp5FPgTAdZ5DN4HOecg Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Andriy Gapon , Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 20:01:02 -0000 On Tue, Oct 22, 2013 at 1:35 PM, Konstantin Belousov wrote: > One thing that catched my eye initially are very strange pointers in the > backtrace, like 'zio=0x883ca8b8'. The standard i386 address space map > splits user/kernel mode at 3GB, so 0x883ca8b8 is definitely usermode, > unless non-standard kernel config is used. The current kernel config for that machine is: options SW_WATCHDOG options KVA_PAGES=512 nodevice plip options DDB options BREAK_TO_DEBUGGER options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC It has been tried with and without KVA_PAGES=512. Doesn't seem to make a difference from a crashing-all-the-time standpoint. > This is why I asked for the gdb backtrace. That *was* a kgdb backtrace. Is there something else needed? Thanks! From owner-freebsd-fs@FreeBSD.ORG Wed Oct 23 16:26:18 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7E58E6CC; Wed, 23 Oct 2013 16:26:18 +0000 (UTC) (envelope-from Albert.Shih@obspm.fr) Received: from smtp-int-m.obspm.fr (smtp-int-m.obspm.fr [145.238.187.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 137462B0D; Wed, 23 Oct 2013 16:26:17 +0000 (UTC) Received: from pcjas.obspm.fr (pcjas.obspm.fr [145.238.184.233]) by smtp-int-m.obspm.fr (8.14.3/8.14.3/SIO Observatoire de Paris - 07/2009) with ESMTP id r9NGPjLF004026 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 23 Oct 2013 18:25:47 +0200 Date: Wed, 23 Oct 2013 18:25:45 +0200 From: Albert Shih To: pgsql-admin@postgresql.org, freebsd-fs@freebsd.org, freebsd-jail@freebsd.org Subject: ZFS-FreeBSD + postgresql performance Message-ID: <20131023162545.GA19794@pcjas.obspm.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.5.21 (2010-09-15) X-Miltered: at smtp-int-m.obspm.fr with ID 5267F889.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 5267F889.000/145.238.184.233/pcjas.obspm.fr/pcjas.obspm.fr/ X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 16:26:18 -0000 Hi I would like to known if someone here have in production a FreeBSD server with postgresql and the FS for the data of postgresql is a ZFS pool. I'm going to buy some server with 96Go of Ram and a jbod of 12 disks (4To each) The purpose is to have everything on this zfs pool (except the system who still on classic raid). So to have many jail (~20-30) running apache/mysql/etc. one postgresql server with all data on the zfs. each jail use his own zfs partition. So I can use zfs send/received to have a mirror of everything in a other server. My question is about the performance, I known ZFS eat all memory he can have (or almost), so what append when we run database like postgresql and jail ? (it's also the reason of 96 Go ram). Sorry for cross-posting but it's about 3 differents things.... Regards. JAS -- Albert SHIH DIO bâtiment 15 Observatoire de Paris 5 Place Jules Janssen 92195 Meudon Cedex France Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71 xmpp: jas@obspm.fr Heure local/Local time: mer 23 oct 2013 18:17:24 CEST From owner-freebsd-fs@FreeBSD.ORG Wed Oct 23 17:14:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4498CA30; Wed, 23 Oct 2013 17:14:33 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 188F62E8C; Wed, 23 Oct 2013 17:14:32 +0000 (UTC) Received: from compute6.internal (compute6.nyi.mail.srv.osa [10.202.2.46]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id CFB7F20CAC; Wed, 23 Oct 2013 13:14:25 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute6.internal (MEProxy); Wed, 23 Oct 2013 13:14:25 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:in-reply-to:references :subject:date; s=smtpout; bh=wnKDUyOj9OT9emJ7KZqCK5FfF0I=; b=OV9 wehDboy2zmVghFgYXrI79/EzO+2wul5psyc6AM2dpRjMGPRPWXTESb49QWfCsTbE u3S6o9APXm/eCE6ZYvQimpMZaPOTd9yZv8BL5EFjdzNJ47ui4Y93uUK05X97nXSK W5XhnJ0YXtmmxMyjgcy6fSM+6+PQ9sjHtU0JQCL0= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id 81DE1112CAE; Wed, 23 Oct 2013 13:14:25 -0400 (EDT) Message-Id: <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> X-Sasl-Enc: ZB3phMnykfvYtnsScG5xuDSYrImvYv+PKGgSX4PlbR7F 1382548465 From: Mark Felder To: Albert Shih , pgsql-admin@postgresql.org, freebsd-fs@freebsd.org, freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-25c9bdb0 In-Reply-To: <20131023162545.GA19794@pcjas.obspm.fr> References: <20131023162545.GA19794@pcjas.obspm.fr> Subject: Re: ZFS-FreeBSD + postgresql performance Date: Wed, 23 Oct 2013 12:14:25 -0500 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 17:14:33 -0000 On Wed, Oct 23, 2013, at 11:25, Albert Shih wrote: > Hi > > I would like to known if someone here have in production a FreeBSD server > with postgresql and the FS for the data of postgresql is a ZFS pool. > > I'm going to buy some server with 96Go of Ram and a jbod of 12 disks (4To > each) > > The purpose is to have everything on this zfs pool (except the system who > still on classic raid). So to have > > many jail (~20-30) running apache/mysql/etc. > one postgresql server with all data on the zfs. > > each jail use his own zfs partition. So I can use zfs send/received to > have > a mirror of everything in a other server. > > My question is about the performance, I known ZFS eat all memory he can > have (or almost), so what append when we run database like postgresql and > jail ? (it's also the reason of 96 Go ram). > > Sorry for cross-posting but it's about 3 differents things.... > To my understanding the solution is to change the primarycache to "metadata" for any ZFS filesystem that you do not want ZFS to heavily cache in memory. Example: # zfs set primarycache=metadata tank/usr/local/pgsql Now for my ZFS filesystem "tank/usr/local/pgsql" where the postgres data directory exists we will tell ZFS to only cache metadata there. Postgres will do its own caching of the data. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 23 20:58:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 80D26DFA for ; Wed, 23 Oct 2013 20:58:02 +0000 (UTC) (envelope-from schmiedgen@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1BF362004 for ; Wed, 23 Oct 2013 20:58:01 +0000 (UTC) Received: from pepe.smoke.local ([94.222.160.176]) by mail.gmx.com (mrgmx102) with ESMTPSA (Nemesis) id 0Mcxtm-1VI6qb3W7e-00IB65 for ; Wed, 23 Oct 2013 22:58:00 +0200 Message-ID: <52683855.6060609@gmx.net> Date: Wed, 23 Oct 2013 22:57:57 +0200 From: Michael Schmiedgen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: Mark Felder , Albert Shih , pgsql-admin@postgresql.org, freebsd-fs@freebsd.org, freebsd-jail@freebsd.org Subject: Re: ZFS-FreeBSD + postgresql performance References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> In-Reply-To: <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:oafUfVLPHm0lDULXur9DXxtH/oI+cPC6qw3rbDCdCEo/Gb5+gLV 8n1/XqrAxRNlIZe0uqEkgVK+esyr2JUw5NFZitdAD73vyFaQs+gH/hJcZ3xzQp/VX0MTTiF WdiiXXBnKd4EMhIpWoVWeYHAIhqJ6WOioALexjn7Qq9oC47QnUM3tNho5PIMvhW/GyiEiaQ WnY99IePcA5b3ievpZAZg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 20:58:02 -0000 On 10/23/13 19:14, Mark Felder wrote: > On Wed, Oct 23, 2013, at 11:25, Albert Shih wrote: >> My question is about the performance, I known ZFS eat all memory he can >> have (or almost), so what append when we run database like postgresql and >> jail ? (it's also the reason of 96 Go ram). >> >> Sorry for cross-posting but it's about 3 differents things.... >> > > To my understanding the solution is to change the primarycache to > "metadata" for any ZFS filesystem that you do not want ZFS to heavily > cache in memory. > > Example: > > # zfs set primarycache=metadata tank/usr/local/pgsql In addition to this it is recommended to set the recordsize to the fixed value of 8k on dataset creation. Michael From owner-freebsd-fs@FreeBSD.ORG Thu Oct 24 06:46:37 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 60365EEF; Thu, 24 Oct 2013 06:46:37 +0000 (UTC) (envelope-from Ivailo.TANUSHEFF@raiffeisen.bg) Received: from mg2.raiffeisen.bg (mg2.raiffeisen.bg [194.48.206.15]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E70B62DE5; Thu, 24 Oct 2013 06:46:36 +0000 (UTC) X-WSS-ID: 0MV5TZU-03-ETV-02 X-M-MSG: Received: from bgrbbl2.rbblan.internal (unknown [10.3.66.167]) by mg2.raiffeisen.bg (Postfix) with ESMTP id 207726425A0; Thu, 24 Oct 2013 09:28:42 +0300 (EEST) In-Reply-To: <52683855.6060609@gmx.net> To: Michael Schmiedgen Subject: Re: ZFS-FreeBSD + postgresql performance From: Ivailo.TANUSHEFF@raiffeisen.bg Message-ID: Date: Thu, 24 Oct 2013 09:28:36 +0300 References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> MIME-Version: 1.0 X-KeepSent: 0B1DBA74:B7B14575-C2257C0E:001EC20A; name=$KeepSent; type=4 X-Mailer: Raiffeisenbank (Bulgaria) X-Disclaimed: 31295 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: owner-freebsd-fs@freebsd.org, freebsd-fs@freebsd.org, pgsql-admin@postgresql.org, freebsd-jail@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 06:46:37 -0000 b3duZXItZnJlZWJzZC1mc0BmcmVlYnNkLm9yZyB3cm90ZSBvbiAyMy4xMC4yMDEzIDIzOjU3OjU3 Og0KDQo+IEZyb206DQo+IA0KPiBNaWNoYWVsIFNjaG1pZWRnZW4gPHNjaG1pZWRnZW5AZ214Lm5l dD4NCj4gDQo+IFRvOg0KPiANCj4gTWFyayBGZWxkZXIgPGZlbGRARnJlZUJTRC5vcmc+LCBBbGJl cnQgU2hpaCA8QWxiZXJ0LlNoaWhAb2JzcG0uZnI+LCANCj4gcGdzcWwtYWRtaW5AcG9zdGdyZXNx bC5vcmcsIGZyZWVic2QtZnNAZnJlZWJzZC5vcmcsIA0KZnJlZWJzZC1qYWlsQGZyZWVic2Qub3Jn LCANCj4gDQo+IERhdGU6DQo+IA0KPiAyMy4xMC4yMDEzIDIzOjU4DQo+IA0KPiBTdWJqZWN0Og0K PiANCj4gUmU6IFpGUy1GcmVlQlNEICsgcG9zdGdyZXNxbCBwZXJmb3JtYW5jZQ0KPiANCj4gU2Vu dCBieToNCj4gDQo+IG93bmVyLWZyZWVic2QtZnNAZnJlZWJzZC5vcmcNCj4gDQo+IE9uIDEwLzIz LzEzIDE5OjE0LCBNYXJrIEZlbGRlciB3cm90ZToNCj4gPiBPbiBXZWQsIE9jdCAyMywgMjAxMywg YXQgMTE6MjUsIEFsYmVydCBTaGloIHdyb3RlOg0KPiA+PiBNeSBxdWVzdGlvbiBpcyBhYm91dCB0 aGUgcGVyZm9ybWFuY2UsIEkga25vd24gWkZTIGVhdCBhbGwgbWVtb3J5IGhlIA0KY2FuDQo+ID4+ IGhhdmUgKG9yIGFsbW9zdCksIHNvIHdoYXQgYXBwZW5kIHdoZW4gd2UgcnVuIGRhdGFiYXNlIGxp a2UgcG9zdGdyZXNxbCANCmFuZA0KPiA+PiBqYWlsID8gKGl0J3MgYWxzbyB0aGUgcmVhc29uIG9m IDk2IEdvIHJhbSkuDQo+ID4+DQo+ID4+IFNvcnJ5IGZvciBjcm9zcy1wb3N0aW5nIGJ1dCBpdCdz IGFib3V0IDMgZGlmZmVyZW50cyB0aGluZ3MuLi4uDQo+ID4+DQo+ID4NCj4gPiBUbyBteSB1bmRl cnN0YW5kaW5nIHRoZSBzb2x1dGlvbiBpcyB0byBjaGFuZ2UgdGhlIHByaW1hcnljYWNoZSB0bw0K PiA+ICJtZXRhZGF0YSIgZm9yIGFueSBaRlMgZmlsZXN5c3RlbSB0aGF0IHlvdSBkbyBub3Qgd2Fu dCBaRlMgdG8gaGVhdmlseQ0KPiA+IGNhY2hlIGluIG1lbW9yeS4NCj4gPg0KPiA+IEV4YW1wbGU6 DQo+ID4NCj4gPiAjIHpmcyBzZXQgcHJpbWFyeWNhY2hlPW1ldGFkYXRhIHRhbmsvdXNyL2xvY2Fs L3Bnc3FsDQo+IA0KPiBJbiBhZGRpdGlvbiB0byB0aGlzIGl0IGlzIHJlY29tbWVuZGVkIHRvIHNl dCB0aGUgcmVjb3Jkc2l6ZQ0KPiB0byB0aGUgZml4ZWQgdmFsdWUgb2YgOGsgb24gZGF0YXNldCBj cmVhdGlvbi4NCj4gDQo+IE1pY2hhZWwNCj4gDQoNCkkgd291bGQgYWxzbyByZWNvbW1lbmQgdG8g dXNlIDRLIHNlY3RvciBzaXplIHVzaW5nIGdub3AgYW5kIHpwb29sIA0KZXhwb3J0L2ltcG9ydC4N CkkgYWdyZWUgdGhhdCB1c2luZyBmaXhlZCByZWNvcmRzaXplIGlzIGEgbXVzdCB3aGVuIHlvdSBo YXZlIGRhdGFiYXNlcyBhbmQgDQp5b3UgaGF2ZSBEQiBvbiBlYWNoIGphaWwuDQpBbHNvIHRoZXJl IGFyZSBhIGJ1bmNoIG9mIHN5c2N0bCBwYXJhbWV0ZXJzIHlvdSBtYXkgd2FudCB0byBwbGF5IHdp dGgsIA0KbW9zdGx5IGFib3V0IHR1bmluZyB0aGUgbWVtb3J5IHVzYWdlLg0KDQpBbHNvIGl0IG1p Z2h0IGJlIGFuIGlkZWEgdG8gdXNlIHBhcnRpdGlvbnMgb24gZGlza3MgYW5kOg0KMS4gVXNlIGZp cnN0IDEtMkcgb24gZWFjaCBkaXNrIGZvciBzd2FwIChqdXN0IGluIGNhc2UpLg0KMi4gVXNlIHRo ZSByZXN0IG9mIHRoZSBkaXNrIGZvciA0SyBhbGlnbmVkIGFuZCA0SyBzZWN0b3Igc2l6ZSBwYXJ0 aXRpb24sIA0KbGFiZWxsZWQgaW4gc29tZSB3YXkgKEkgdXNlIGRpc2sgSUQgYXMgbGFiZWwsIGZv ciBlYXN5IGlkZW50aWZpY2F0aW9uKS4NCg0KQmVzdCByZWdhcmRzLA0KSXZhaWxvIFRhbnVzaGVm Zg0KDQpESVNDTEFJTUVSOiBUaGUgY29udGVudHMgb2YgdGhpcyBlLW1haWwgYXJlIGludGVuZGVk IGZvciB0aGUgbmFtZWQgYWRkcmVzc2VlIG9ubHkuIEl0IGNvbnRhaW5zIGluZm9ybWF0aW9uIHRo YXQgbWF5IGJlIHByaXZpbGVnZWQgYW5kL29yIGNvbmZpZGVudGlhbC4gVW5sZXNzIHlvdSBhcmUg dGhlIG5hbWVkIGFkZHJlc3NlZSBvciBhbiBhdXRob3JpemVkIGRlc2lnbmVlLCB5b3UgbWF5IG5v dCBjb3B5IG9yIHVzZSBpdCwgb3IgZGlzY2xvc2UgaXQgdG8gYW55b25lIGVsc2UuIElmIHlvdSBy ZWNlaXZlZCBpdCBpbiBlcnJvciBwbGVhc2Ugbm90aWZ5IHVzIGltbWVkaWF0ZWx5IGFuZCB0aGVu IGRlc3Ryb3kgaXQuIE1lc3NhZ2VzIHRyYW5zcG9ydGVkIG92ZXIgdGhlIHB1YmxpYyBJbnRlcm5l dCBhcmUgc3VzY2VwdGlibGUgdG8gYWx0ZXJhdGlvbi4gUmFpZmZlaXNlbmJhbmsgKEJ1bGdhcmlh KSBFQUQsIFJhaWZmZWlzZW4gQXNzZXQgTWFuYWdlbWVudCAoQnVsZ2FyaWEpIEVBRCwgUmFpZmZl aXNlbiBJbnN1cmFuY2UgQnJva2VyIEVPT0QsIFJhaWZmZWlzZW4gU2VydmljZXMgRUFELCBSYWlm ZmVpc2VuIExlYXNpbmcgQnVsZ2FyaWEgT09ELCBSYWlmZmVpc2VuIEF1dG8gTGVhc2luZyBCdWxn YXJpYSBFT09EIGFuZCBSYWlmZmVpc2VuIFJlYWwgRXN0YXRlIEVPT0Qgc2hhbGwgbm90IGJlIGxp YWJsZSBmb3IgdGhlIG1lc3NhZ2UgaWYgYWx0ZXJlZCwgY2hhbmdlZCBvciBmYWxzaWZpZWQuICDv 5/Lh7un+5e7p5SDu4SDv9Ofv9+/y7u/z9PThOiDz38Tf0tbBzsnF1M8gzsEg1M/XwSDFzMXL1NLP zs7PINPfz8Ldxc7JxSDFINDSxcTOwdrOwd7Fzs8gxcTJztPU18XOzyDawSDQz9PP3sXOydEg0M/M 1d7B1MXMLiDz38/C3cXOycXUzyDT38Tf0tbBIMnOxs/SzcHDydEsIMvP0dTPIM3P1sUgxMEgwt/E xSDQ0sXEzsHazsHexc7BINrBIM/H0sHOyd7Fzs8g0M/M2tfBzsUgyS/JzMkg0M/XxdLJ1MXMzsEu IPcg08zV3sHKIN7FIM7FINPUxSDVy8Hawc7J0SDQz8zV3sHUxcwgzsEg1M/XwSDT38/C3cXOycUg yczJIM7Fx8/XIM/Uz9LJ2snSwc4g0NLFxNPUwdfJ1MXMLCDO0c3B1MUg0NLB188gxMEgx88gy8/Q ydLB1MUgyczJIMna0M/M2tfB1MUsIMvBy9TPIMkgxMEg0sHay9LJ18HUxSDT38Tf0tbBzsnF1M8g zdUg0NLFxCDU0sXUySDMycPBLiDhy88g09TFIMfPINDPzNXeyczJINDPIMfSxdvLwSwgzc/M0SDE wSDV18XEz83J1MUg0M/EwdTFzNEgzsXawcLB187PIMkgxMEgydrU0snF1MUg1M/XwSDT38/C3cXO ycUgz9Qg08nT1MXNwdTBLiDlzMXL1NLPzs7J1MUg09/Pwt3FzsnRLCDSwdrQ0s/T1NLBztHXwc7J INDPIOnO1MXSzsXUIM3Px8HUIMTBIMLfxMHUINDSz83FztHOyS4g8sHKxsHK2sXOwsHOyyAo4t/M x8HSydEpIOXh5Cwg8sHKxsHK2sXOIOHTxdQg7cXOycTWzd/O1CAo4t/Mx8HSydEpIOXh5Cwg8sHK xsHK2sXOIPrB09TSwcjP18HUxczFziDi0s/LxdIg5e/v5Cwg8sHKxsHK2sXOIPPf0tfJ08naIOXh 5Cwg8sHKxsHK2sXOIOzJ2snOxyDi38zHwdLJ0SDv7+QsIPLBysbBytrFziDh1dTPIOzJ2snOxyDi 38zHwdLJ0SDl7+/kIMkg8sHKxsHK2sXOIOnNz9TJIOXv7+QgzsUgzs/T0dQgz9THz9fP0s7P09Qg 2sEgzc/EycbJw8nSwc7JLCDQ0s/Nxc7FzskgyczJIMbBzNvJxsnDydLBzskg09/Pwt3FzsnRLiAg From owner-freebsd-fs@FreeBSD.ORG Thu Oct 24 08:21:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 81336762 for ; Thu, 24 Oct 2013 08:21:25 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews06.kpnxchange.com (cpsmtpb-ews06.kpnxchange.com [213.75.39.9]) by mx1.freebsd.org (Postfix) with ESMTP id 1AE1424E7 for ; Thu, 24 Oct 2013 08:21:24 +0000 (UTC) Received: from cpsps-ews19.kpnxchange.com ([10.94.84.185]) by cpsmtpb-ews06.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Thu, 24 Oct 2013 10:20:14 +0200 Received: from CPSMTPM-TLF102.kpnxchange.com ([195.121.3.5]) by cpsps-ews19.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Thu, 24 Oct 2013 10:20:14 +0200 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF102.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Thu, 24 Oct 2013 10:20:14 +0200 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 2E878329E for ; Thu, 24 Oct 2013 10:20:14 +0200 (CEST) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: ZFS-FreeBSD + postgresql performance References: <20131023162545.GA19794@pcjas.obspm.fr> Date: Thu, 24 Oct 2013 10:20:13 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <20131023162545.GA19794@pcjas.obspm.fr> User-Agent: Opera Mail/12.16 (FreeBSD) X-OriginalArrivalTime: 24 Oct 2013 08:20:14.0543 (UTC) FILETIME=[DD81CDF0:01CED091] X-RcptDomain: freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 08:21:25 -0000 On Wed, 23 Oct 2013 18:25:45 +0200, Albert Shih wrote: > Hi > > I would like to known if someone here have in production a FreeBSD server > with postgresql and the FS for the data of postgresql is a ZFS pool. > > I'm going to buy some server with 96Go of Ram and a jbod of 12 disks (4To > each) > > The purpose is to have everything on this zfs pool (except the system who > still on classic raid). So to have > > many jail (~20-30) running apache/mysql/etc. > one postgresql server with all data on the zfs. > > each jail use his own zfs partition. So I can use zfs send/received to > have > a mirror of everything in a other server. > > My question is about the performance, I known ZFS eat all memory he can > have (or almost), so what append when we run database like postgresql and > jail ? (it's also the reason of 96 Go ram). > > Sorry for cross-posting but it's about 3 differents things.... > > Regards. > > JAS Databases do a lot of sync writes, so it might be good to use a fast ZIL device (SSD). Ronald. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 24 12:25:28 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9C7F8E6 for ; Thu, 24 Oct 2013 12:25:28 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-qa0-x22a.google.com (mail-qa0-x22a.google.com [IPv6:2607:f8b0:400d:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5EB682A6F for ; Thu, 24 Oct 2013 12:25:28 +0000 (UTC) Received: by mail-qa0-f42.google.com with SMTP id w8so4956113qac.8 for ; Thu, 24 Oct 2013 05:25:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=eGZnpZq0SfVchBO1nSgwH66egoWwzyL6+UmooANussM=; b=B7Fg3bgk4feQ42WiIYutcGh/5LYKB34JCrXwVHAi120japcj4S+l4Tae48NiE+jq/k 0Ly/1JsuUnhW0v7DgIP7eaC41giftR7YXGKXmkNL89H7xfSF4lqNDD+1cx8FSFT4vi9i PgQHALHUde2Qkjqm2akICcY2cTaSIH0SNml9QierEcyXsqzJbVOTlmhr+/wWYxCHLEYh 4bwvvy8UB819ssQzCGJKtJXLJGAeiPtkYBD5g5ZDdwpasy6+jSvA3AhbNEZI9dVMFu36 SjC1iy9Y77amHIDoGaDrEGWquIumVFa6oxBfD2qqYOEIkIkIgND21eqJZO6wn0VgAU32 s9RQ== MIME-Version: 1.0 X-Received: by 10.229.12.197 with SMTP id y5mr3035686qcy.8.1382617527486; Thu, 24 Oct 2013 05:25:27 -0700 (PDT) Received: by 10.224.55.77 with HTTP; Thu, 24 Oct 2013 05:25:27 -0700 (PDT) In-Reply-To: References: <20131023162545.GA19794@pcjas.obspm.fr> Date: Thu, 24 Oct 2013 13:25:27 +0100 Message-ID: Subject: Re: ZFS-FreeBSD + postgresql performance From: krad To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 12:25:28 -0000 It is also generally better to allocate the ram to the db rather than zfs. This reduces context switches, and the db can cache more intelligently that the file system. However your mileage with this may vary depending on the db in question. On a system with such a large memory pool limiting zrc to 8 gb, maybe a good thing to do, especially if you can hook up a large l2arc. You say also say each jail has its own partition, by this i presume you mean zfs dataset. You may want to consider having separate pools of each db/jail, as this would limit spindle contention, but could produce a complicated system. If your dbs are doing lots of writes, consider implementing raid 10 on the pools level as its much faster for writes. As you have a lots of dbs, and potentially a lot of spindles, you could mix and match the above ideas depending on the dbs profile. Have say a raidz6 pool for you lower traffic mostly read dbs, a raid 10 pool for your heavy write dbs, and a dedicated pool for you low latency db On 24 October 2013 09:20, Ronald Klop wrote: > On Wed, 23 Oct 2013 18:25:45 +0200, Albert Shih > wrote: > > Hi >> >> I would like to known if someone here have in production a FreeBSD server >> with postgresql and the FS for the data of postgresql is a ZFS pool. >> >> I'm going to buy some server with 96Go of Ram and a jbod of 12 disks (4To >> each) >> >> The purpose is to have everything on this zfs pool (except the system who >> still on classic raid). So to have >> >> many jail (~20-30) running apache/mysql/etc. >> one postgresql server with all data on the zfs. >> >> each jail use his own zfs partition. So I can use zfs send/received to >> have >> a mirror of everything in a other server. >> >> My question is about the performance, I known ZFS eat all memory he can >> have (or almost), so what append when we run database like postgresql and >> jail ? (it's also the reason of 96 Go ram). >> >> Sorry for cross-posting but it's about 3 differents things.... >> >> Regards. >> >> JAS >> > > Databases do a lot of sync writes, so it might be good to use a fast ZIL > device (SSD). > > Ronald. > > ______________________________**_________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org > " > From owner-freebsd-fs@FreeBSD.ORG Thu Oct 24 12:50:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CAE0DA25 for ; Thu, 24 Oct 2013 12:50:35 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 38E262C10 for ; Thu, 24 Oct 2013 12:50:34 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r9OCeuBR075443 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 24 Oct 2013 15:40:57 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <52691558.5060306@digsys.bg> Date: Thu, 24 Oct 2013 15:40:56 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS-FreeBSD + postgresql performance References: <20131023162545.GA19794@pcjas.obspm.fr> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 12:50:35 -0000 On 24.10.13 15:25, krad wrote: > It is also generally better to allocate the ram to the db rather than zfs. > This reduces context switches, and the db can cache more intelligently that > the file system. However your mileage with this may vary depending on the > db in question. On a system with such a large memory pool limiting zrc to 8 > gb, maybe a good thing to do, especially if you can hook up a large l2arc. If you have large L2ARC, that will also require large ARC. Indeed, the theory is that most memory should be allocated to the PostgreSQL shared memory pool. However, my practical experience demonstrates exactly the opposite, especially when the server runs mixed load. Limiting ARC has always resulted in worse performance. In short: you need to run extensive tests. It used to be much worse, but current FreeBSD is behaving quite well without artificially limiting the ARC. > If your dbs are doing lots of writes, consider implementing raid 10 on the > pools level as its much faster for writes. By all means, if you will run PostgreSQL on the zpool avoid any other redundancy than MIRROR. You will be better having one separate MIRROR (or multiple mirror vdevs) pool for the DB and another for other data, any time. > As you have a lots of dbs, and potentially a lot of spindles, you could mix > and match the above ideas depending on the dbs profile. Have say a raidz6 > pool for you lower traffic mostly read dbs, a raid 10 pool for your heavy > write dbs, and a dedicated pool for you low latency db I have actually experimented with this, in using 15k drives for the "fast" databases and 10k drives for the "slow" databases. At the end, I discovered it was better idea to combine all the drives in a single zpool with multiple vdevs, than to do such separation -- as multiple vdevs are always going to provide you with more performance. With PostgreSQL, you want to have dedicated ZIL device. Any current SSD will do, but consider your workload -- you might be interested in an enterprise grade (more write endurance) SSD, or -- you might do what I do -- consider SSDs used as ZIL and L1ARC as consumables -- when they wear out, just put a new one. Their only role is to increase performance, no data is dependent on their long life... I also experimented with SSD-only zpools for PostrgeSQL. What to say.. for reasonably sized databases, this setup beats anything else hands down. Just make sure you have enough redundancy and backups, as any SSD can fail at any time for no obvious reason (disks too). Daniel From owner-freebsd-fs@FreeBSD.ORG Thu Oct 24 17:27:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C97BBBC7 for ; Thu, 24 Oct 2013 17:27:06 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 945102D70 for ; Thu, 24 Oct 2013 17:27:06 +0000 (UTC) Received: by mail-ob0-f179.google.com with SMTP id uy5so2635446obc.38 for ; Thu, 24 Oct 2013 10:27:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=iZoKHeuMbT4J3Bml5xFLRB8hTI1SgBCi+g1V4se5rO8=; b=Bkw6thqLJGOjAa3IVMXK9ikiiytee59BCAjaSzZ2vr2hygHeRJyKhQrnxGRiT3IXRj Ytoj9N/AIYy/+IQCD6LBh/BhKD6jgrb+fzh4kZjj712y7xjOhganIyBspDebVh5wcK4G SHZzPVMUhw8own1QjqmrGImsM+4oMsJnVBM7OeeaibdreDQn0seTEfIIlwsxq1T1Imum HgMtSpO9k1QSmJEl3WSzPUFE/ELyj1B7IIWYjkFVeN2PulEIyAgkA4zxu2V47vpe0xqz QjYYmMuw4k4Z/g1RPXQ6q98YcQQ4Y2bwnJtA085syFA5arDfcXg7ZRYHAfJ6Ml4w27sL TivQ== MIME-Version: 1.0 X-Received: by 10.60.63.36 with SMTP id d4mr2983743oes.29.1382635625254; Thu, 24 Oct 2013 10:27:05 -0700 (PDT) Received: by 10.76.132.9 with HTTP; Thu, 24 Oct 2013 10:27:05 -0700 (PDT) In-Reply-To: <52691558.5060306@digsys.bg> References: <20131023162545.GA19794@pcjas.obspm.fr> <52691558.5060306@digsys.bg> Date: Thu, 24 Oct 2013 10:27:05 -0700 Message-ID: Subject: Re: ZFS-FreeBSD + postgresql performance From: Freddie Cash To: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 17:27:06 -0000 On Thu, Oct 24, 2013 at 5:40 AM, Daniel Kalchev wrote: > > On 24.10.13 15:25, krad wrote: > >> If your dbs are doing lots of writes, consider implementing raid 10 on t= he >> > pools level as its much faster for writes. >> > > By all means, if you will run PostgreSQL on the zpool avoid any other > redundancy than MIRROR. You will be better having one separate MIRROR (or > multiple mirror vdevs) pool for the DB and another for other data, any ti= me. > > As you have a lots of dbs, and potentially a lot of spindles, you could >> mix >> =E2=80=8B a=E2=80=8B >> nd match the above ideas depending on the dbs profile. Have say a raidz6 >> =E2=80=8B =E2=80=8B >> pool for you lower traffic mostly read dbs, a raid 10 pool for your heav= y >> =E2=80=8B =E2=80=8B >> write dbs, and a dedicated pool for you low latency db >> > > I have actually experimented with this, in using 15k drives for the "fast= " > databases and 10k drives for the "slow" databases. At the end, I discover= ed > it was better idea to combine all the drives in a single zpool with > multiple vdevs, than to do such separation -- as multiple vdevs are alway= s > going to provide you with more performance. > =E2=80=8BThere was a discussion recently on the zfs-discuss mailing list co= vering this. There were no hard numbers thrown around, but the general consensus is that a single large pool per system, comprised of many vdevs, will almost always give better performance than a system with multiple separate pools. There are, of course, exceptions, but in general, you want to limit the number of individual pools per system. For example, a pool for the OS running on some small HDs; a pool of mirror vdevs using fast disk for things needed a lot of performance; and a pool of large/slow disks in raidz2/3 for things that need a lot of storage. =E2=80=8B =E2=80=8BOr, just go with a lot of raidz2/3 vdevs with a bunch of L2ARC in = front of it. Performance might be good enough for your needs. Or, just go with a lot of mirror vdevs if you need the raw throughput (maybe 3-way mirrors if you're paranoid). --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 15:43:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D02B9246 for ; Fri, 25 Oct 2013 15:43:09 +0000 (UTC) (envelope-from redlamb@redlamb.net) Received: from ironhide.redlamb.net (ironhide.redlamb.net [216.17.90.180]) by mx1.freebsd.org (Postfix) with ESMTP id AC3962ECF for ; Fri, 25 Oct 2013 15:43:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ironhide.redlamb.net (Postfix) with ESMTP id D41E1C7ABC for ; Fri, 25 Oct 2013 10:37:46 -0500 (CDT) Received: from ironhide.redlamb.net ([127.0.0.1]) by localhost (ironhide.redlamb.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 91415-09 for ; Fri, 25 Oct 2013 10:37:46 -0500 (CDT) Received: by ironhide.redlamb.net (Postfix, from userid 80) id 478D9C7AD1; Fri, 25 Oct 2013 10:37:46 -0500 (CDT) Received: from 173-165-231-73-minnesota.hfc.comcastbusiness.net (173-165-231-73-minnesota.hfc.comcastbusiness.net [173.165.231.73]) by imp.redlamb.net (Horde MIME library) with HTTP; Fri, 25 Oct 2013 10:37:46 -0500 Message-ID: <20131025103746.g0u1s5xlgc0k8c48@imp.redlamb.net> Date: Fri, 25 Oct 2013 10:37:46 -0500 From: Pete Erickson To: FreeBSD Filesystems Subject: missing files on ext2fs with 9-STABLE MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.2) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 15:43:09 -0000 I'm missing files on an ext2 filesystem in 9-STABLE. However, remounting the drive in Ubuntu (12.04.3) the files are back. The missing files range from a few KB to several GB so it's not based on the file sizes. While I've found an ugly way past this, I'm trying to understand why I'm having the problem in the first place. The ext2 fs was created in Ubuntu (12.04.3) in order to backup approx 1.4 TB from a hardware raid ext4 fs and copy it back once I repurpose the drives as ZFS in FreeBSD. The ext2 fs is on a gpt-labeled 3.0TB drive and I used rsync to copy the data. Unfortunately, I'm not getting any error messages so I have nothing to provide. FYI, my temp fix is mounting the raw ext2 fs to a ubuntu VM in VirtualBox and then rsync'ing it the ZFS fs via the virtual network. While it's ugly, it appears that it'll work. However, I'm hesitant to destroy the ext4 fs until I'm sure that I have the data backed up. Any help understanding this would be appreciated. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 15:53:52 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2CE458AE for ; Fri, 25 Oct 2013 15:53:52 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from nm4-vm3.bullet.mail.sg3.yahoo.com (nm4-vm3.bullet.mail.sg3.yahoo.com [106.10.148.130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 670712F73 for ; Fri, 25 Oct 2013 15:53:51 +0000 (UTC) Received: from [106.10.166.117] by nm4.bullet.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:43 -0000 Received: from [106.10.151.138] by tm6.bullet.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:43 -0000 Received: from [127.0.0.1] by omp1006.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:43 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 103156.90240.bm@omp1006.mail.sg3.yahoo.com Received: (qmail 14394 invoked by uid 60001); 25 Oct 2013 15:53:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1382716422; bh=rOunw42EwGWlsKNt/t4UuPfDgQwOxQNSdtxU6YTYDfo=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=v30trmo0cshy5r/n7Fpk/FBT/r4Eo0enyjIReA2jIrjlWJmey2a/H2jbWLvS5zrUuDbC9i0ZRlzOqjm9vlIEjJMcjDVcp+qbz0sLOSX2I6/DyWWrCWXxCEsUL0J27HX73tZtfS7GaKdZY9E/H2rAKX0lnvaHIfIjBqcS1VxcrTI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=VI/e5PhO/3YJpA3EFPOtNpGEunl8J5Ej3nTBYZPOBIBhtYlACIhhnu5F8j9p1NeOkI1ufF5kDdhKD/co2qfXiY5zMmySHHU8hFeV3nK+2LtzkR0taUvIxFcPS2e6PR5bX11wHAu9FzKGJAwzQGrJumILLs648E/JCa7GJVhBMi8=; X-YMail-OSG: x7TykDAVM1lr0gHra6Mhqd7bGnqiFrsnOfXFcXMUxxothsy 5q.6Iz9dhLxQFZ8QVjeYkqTltcW_sbxpJseJbb0I5bCTf050s0tmlgUOuHgv PMxp_Nc8KhScAAdPZObHi8_hoH9ZYB4y4S8smlKbssrkx.AfD45gpaFTaMHz e0N9jFCc2S1mqlB38cSHNXjYcrfzIFHsaKcLvZQXzSuk3slB2KzzsoNiXTKf LamAzrY.joW9mhoY1lcBepvWKUIrORApq2.HwdM7Vb.U_5rC_AYrReo3QsFr rqNRwoxEZsCV0H433gaktvvRdK56lsipfNLAWz8hapu6HiqAYZZosSGbNFtQ xONk77KymhDCnkbPGLnwFDY7WGzXkD2FJpw2uM_X_QtgQ7IxaT6KA2PSSS_9 vwVNLxH2h2ZjF7ZC.ODBoxjCW76IKvnUCwuRJdYoS74eh3nzVEJj1G.Ri.KW vQyI43zHqEo7Mr4IdydLfWF0R2ytfCGm.7Ef3Fxlb21Cl.snvjVJ4arklw1h giBLUX7N0fF6E.KAr8dPTMxFzQp184NLLe8.hWTPZhCNfUbndIYNvZCDyzQF Luwhn3W91iqhaMA-- Received: from [61.15.240.133] by web193504.mail.sg3.yahoo.com via HTTP; Fri, 25 Oct 2013 23:53:42 SGT X-Rocket-MIMEInfo: 002.001, PiBJIHdvdWxkIGFsc28gcmVjb21tZW5kIHRvIHVzZSA0SyBzZWN0b3Igc2l6ZSB1c2luZwpnbm9wIGFuZCB6cG9vbCBleHBvcnQvaW1wb3J0LgoKQWdyZWUsIGluIG15IHNtYWxsIHNldHVwLCA0SyBzZWN0b3JzIGFuZCBhbGlnbmVkIHBhcnRpdGlvbiBjYW4gYmUgc2V2ZW4gb3IgZWlnaHQgdGltZXMgZmFzdGVyIHRoYW4gNTEya2Igc2VjdG9ycyB3aGVuIHRoZSBoYXJkZGlzayBpcyBpbiBhZHZhbmNlIGZvcm1hdCAoNGtiIHBlciBzZWN0b3JzKQoKQWxzbyBzdWdnZXN0IHRvIGhhdmUgdHdvIHNlcGFydGUgZmEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> Message-ID: <1382716422.13782.YahooMailNeo@web193504.mail.sg3.yahoo.com> Date: Fri, 25 Oct 2013 23:53:42 +0800 (SGT) From: Patrick Dung Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance To: "Ivailo.TANUSHEFF@raiffeisen.bg" , Michael Schmiedgen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "owner-freebsd-fs@freebsd.org" , "freebsd-fs@freebsd.org" , "pgsql-admin@postgresql.org" , "freebsd-jail@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Patrick Dung List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 15:53:52 -0000 > I would also recommend to use 4K sector size using=0Agnop and zpool expor= t/import.=0A=0AAgree, in my small setup, 4K sectors and aligned partition c= an be seven or eight times faster than 512kb sectors when the harddisk is i= n advance format (4kb per sectors)=0A=0AAlso suggest to have two separte fa= st SSD (in mirror mode) for the ZIL.=0AThis is to avoid ZFS file system fra= gmentation when the ZIL is stored intent with the ZFS.=0A=0APatrick=0A=0A= =0A=0A=0AOn Thursday, October 24, 2013 10:18 PM, "Ivailo.TANUSHEFF@raiffeis= en.bg" wrote:=0A =0Aowner-freebsd-fs@freeb= sd.org wrote on 23.10.2013 23:57:57:=0A=0A> From:=0A> =0A> Michael Schmiedg= en =0A> =0A> To:=0A> =0A> Mark Felder , Albert Shih , =0A> pgsql-admin@postgresql.org, fre= ebsd-fs@freebsd.org, freebsd-jail@freebsd.org, =0A> =0A> Date:=0A> =0A> 23.= 10.2013 23:58=0A> =0A> Subject:=0A> =0A> Re: ZFS-FreeBSD + postgresql perfo= rmance=0A> =0A> Sent by:=0A> =0A> owner-freebsd-fs@freebsd.org=0A> =0A> On = 10/23/13 19:14, Mark Felder wrote:=0A> > On Wed, Oct 23, 2013, at 11:25, Al= bert Shih wrote:=0A> >> My question is about the performance, I known ZFS e= at all=0Amemory he can=0A> >> have (or almost), so what append when we run = database like=0Apostgresql and=0A> >> jail ? (it's also the reason of 96 Go= ram).=0A> >>=0A> >> Sorry for cross-posting but it's about 3 differents th= ings....=0A> >>=0A> >=0A> > To my understanding the solution is to change t= he primarycache=0Ato=0A> > "metadata" for any ZFS filesystem that you do no= t want=0AZFS to heavily=0A> > cache in memory.=0A> >=0A> > Example:=0A> >= =0A> > # zfs set primarycache=3Dmetadata tank/usr/local/pgsql=0A> =0A> In a= ddition to this it is recommended to set the recordsize=0A> to the fixed va= lue of 8k on dataset creation.=0A> =0A> Michael=0A> =0A=0AI would also reco= mmend to use 4K sector size using=0Agnop and zpool export/import.=0AI agree= that using fixed recordsize is a must when=0Ayou have databases and you ha= ve DB on each jail.=0AAlso there are a bunch of sysctl parameters you may= =0Awant to play with, mostly about tuning the memory usage.=0A=0AAlso it mi= ght be an idea to use partitions on disks=0Aand:=0A1. Use first 1-2G on eac= h disk for swap (just in case).=0A2. Use the rest of the disk for 4K aligne= d and 4K=0Asector size partition, labelled in some way (I use disk ID as la= bel, for=0Aeasy identification).=0A=0ABest regards,=0AIvailo Tanusheff=0A= =0A=0A=0ADISCLAIMER: The contents of this e-mail are intended for the named= addressee only. It contains information that may be privileged and/or conf= idential. Unless you are the named addressee or an authorized designee, you= may not copy or use it, or disclose it to anyone else. If you received it = in error please notify us immediately and then destroy it. Messages transpo= rted over the public Internet are susceptible to alteration. Raiffeisenbank= (Bulgaria) EAD, Raiffeisen Asset Management (Bulgaria) EAD, Raiffeisen Ins= urance Broker EOOD, Raiffeisen Services EAD, Raiffeisen Leasing Bulgaria OO= D, Raiffeisen Auto Leasing Bulgaria EOOD and Raiffeisen Real Estate EOOD sh= all not be liable for the message if altered, changed or falsified. =0A=0A= =D0=9E=D0=93=D0=A0=D0=90=D0=9D=D0=98=D0=A7=D0=95=D0=9D=D0=98=D0=95 =D0=9D= =D0=90 =D0=9E=D0=A2=D0=93=D0=9E=D0=92=D0=9E=D0=A0=D0=9D=D0=9E=D0=A1=D0=A2= =D0=A2=D0=90: =D0=A1=D1=8A=D0=B4=D1=8A=D1=80=D0=B6=D0=B0=D0=BD=D0=B8=D0=B5= =D1=82=D0=BE =D0=BD=D0=B0 =D1=82=D0=BE=D0=B2=D0=B0 =D0=B5=D0=BB=D0=B5=D0=BA= =D1=82=D1=80=D0=BE=D0=BD=D0=BD=D0=BE =D1=81=D1=8A=D0=BE=D0=B1=D1=89=D0=B5= =D0=BD=D0=B8=D0=B5 =D0=B5 =D0=BF=D1=80=D0=B5=D0=B4=D0=BD=D0=B0=D0=B7=D0=BD= =D0=B0=D1=87=D0=B5=D0=BD=D0=BE =D0=B5=D0=B4=D0=B8=D0=BD=D1=81=D1=82=D0=B2= =D0=B5=D0=BD=D0=BE =D0=B7=D0=B0 =D0=BF=D0=BE=D1=81=D0=BE=D1=87=D0=B5=D0=BD= =D0=B8=D1=8F =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B0=D1=82=D0=B5=D0=BB. =D0=A1= =D1=8A=D0=BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D0=B5=D1=82=D0=BE =D1=81=D1=8A= =D0=B4=D1=8A=D1=80=D0=B6=D0=B0 =D0=B8=D0=BD=D1=84=D0=BE=D1=80=D0=BC=D0=B0= =D1=86=D0=B8=D1=8F, =D0=BA=D0=BE=D1=8F=D1=82=D0=BE =D0=BC=D0=BE=D0=B6=D0=B5= =D0=B4=D0=B0 =D0=B1=D1=8A=D0=B4=D0=B5 =D0=BF=D1=80=D0=B5=D0=B4=D0=BD=D0=B0= =D0=B7=D0=BD=D0=B0=D1=87=D0=B5=D0=BD=D0=B0 =D0=B7=D0=B0 =D0=BE=D0=B3=D1=80= =D0=B0=D0=BD=D0=B8=D1=87=D0=B5=D0=BD=D0=BE =D0=BF=D0=BE=D0=BB=D0=B7=D0=B2= =D0=B0=D0=BD=D0=B5 =D0=B8/=D0=B8=D0=BB=D0=B8 =D0=BF=D0=BE=D0=B2=D0=B5=D1=80= =D0=B8=D1=82=D0=B5=D0=BB=D0=BD=D0=B0. =D0=92 =D1=81=D0=BB=D1=83=D1=87=D0=B0= =D0=B9 =D1=87=D0=B5 =D0=BD=D0=B5 =D1=81=D1=82=D0=B5 =D1=83=D0=BA=D0=B0=D0= =B7=D0=B0=D0=BD=D0=B8=D1=8F =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B0=D1=82=D0= =B5=D0=BB =D0=BD=D0=B0 =D1=82=D0=BE=D0=B2=D0=B0 =D1=81=D1=8A=D0=BE=D0=B1=D1= =89=D0=B5=D0=BD=D0=B8=D0=B5 =D0=B8=D0=BB=D0=B8 =D0=BD=D0=B5=D0=B3=D0=BE=D0= =B2 =D0=BE=D1=82=D0=BE=D1=80=D0=B8=D0=B7=D0=B8=D1=80=D0=B0=D0=BD =D0=BF=D1= =80=D0=B5=D0=B4=D1=81=D1=82=D0=B0=D0=B2=D0=B8=D1=82=D0=B5=D0=BB, =D0=BD=D1= =8F=D0=BC=D0=B0=D1=82=D0=B5 =D0=BF=D1=80=D0=B0=D0=B2=D0=BE =D0=B4=D0=B0 =D0= =B3=D0=BE =D0=BA=D0=BE=D0=BF=D0=B8=D1=80=D0=B0=D1=82=D0=B5 =D0=B8=D0=BB=D0= =B8 =D0=B8=D0=B7=D0=BF=D0=BE=D0=BB=D0=B7=D0=B2=D0=B0=D1=82=D0=B5, =D0=BA=D0= =B0=D0=BA=D1=82=D0=BE =D0=B8 =D0=B4=D0=B0 =D1=80=D0=B0=D0=B7=D0=BA=D1=80=D0= =B8=D0=B2=D0=B0=D1=82=D0=B5 =D1=81=D1=8A=D0=B4=D1=8A=D1=80=D0=B6=D0=B0=D0= =BD=D0=B8=D0=B5=D1=82=D0=BE =D0=BC=D1=83 =D0=BF=D1=80=D0=B5=D0=B4 =D1=82=D1= =80=D0=B5=D1=82=D0=B8 =D0=BB=D0=B8=D1=86=D0=B0. =D0=90=D0=BA=D0=BE =D1=81= =D1=82=D0=B5 =D0=B3=D0=BE =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B8=D0=BB=D0=B8 = =D0=BF=D0=BE =D0=B3=D1=80=D0=B5=D1=88=D0=BA=D0=B0, =D0=BC=D0=BE=D0=BB=D1=8F= =D0=B4=D0=B0 =D1=83=D0=B2=D0=B5=D0=B4=D0=BE=D0=BC=D0=B8=D1=82=D0=B5 =D0=BF= =D0=BE=D0=B4=D0=B0=D1=82=D0=B5=D0=BB=D1=8F =D0=BD=D0=B5=D0=B7=D0=B0=D0=B1= =D0=B0=D0=B2=D0=BD=D0=BE =D0=B8 =D0=B4=D0=B0 =D0=B8=D0=B7=D1=82=D1=80=D0=B8= =D0=B5=D1=82=D0=B5 =D1=82=D0=BE=D0=B2=D0=B0 =D1=81=D1=8A=D0=BE=D0=B1=D1=89= =D0=B5=D0=BD=D0=B8=D0=B5 =D0=BE=D1=82 =D1=81=D0=B8=D1=81=D1=82=D0=B5=D0=BC=D0=B0=D1=82=D0=B0. =D0=95=D0=BB=D0=B5= =D0=BA=D1=82=D1=80=D0=BE=D0=BD=D0=BD=D0=B8=D1=82=D0=B5 =D1=81=D1=8A=D0=BE= =D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D1=8F, =D1=80=D0=B0=D0=B7=D0=BF=D1=80=D0=BE= =D1=81=D1=82=D1=80=D0=B0=D0=BD=D1=8F=D0=B2=D0=B0=D0=BD=D0=B8 =D0=BF=D0=BE = =D0=98=D0=BD=D1=82=D0=B5=D1=80=D0=BD=D0=B5=D1=82 =D0=BC=D0=BE=D0=B3=D0=B0= =D1=82 =D0=B4=D0=B0 =D0=B1=D1=8A=D0=B4=D0=B0=D1=82 =D0=BF=D1=80=D0=BE=D0=BC= =D0=B5=D0=BD=D1=8F=D0=BD=D0=B8. =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7= =D0=B5=D0=BD=D0=B1=D0=B0=D0=BD=D0=BA (=D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80= =D0=B8=D1=8F) =D0=95=D0=90=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0= =B7=D0=B5=D0=BD =D0=90=D1=81=D0=B5=D1=82 =D0=9C=D0=B5=D0=BD=D0=B8=D0=B4=D0= =B6=D0=BC=D1=8A=D0=BD=D1=82 (=D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80=D0=B8=D1= =8F) =D0=95=D0=90=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5= =D0=BD =D0=97=D0=B0=D1=81=D1=82=D1=80=D0=B0=D1=85=D0=BE=D0=B2=D0=B0=D1=82= =D0=B5=D0=BB=D0=B5=D0=BD =D0=91=D1=80=D0=BE=D0=BA=D0=B5=D1=80 =D0=95=D0=9E= =D0=9E=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=A1= =D1=8A=D1=80=D0=B2=D0=B8=D1=81=D0=B8=D0=B7 =D0=95=D0=90=D0=94, =D0=A0=D0=B0= =D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=9B=D0=B8=D0=B7=D0=B8=D0=BD= =D0=B3 =D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80=D0=B8=D1=8F =D0=9E=D0=9E=D0=94,= =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=90=D1=83=D1=82= =D0=BE =D0=9B=D0=B8=D0=B7=D0=B8=D0=BD=D0=B3 =D0=91=D1=8A=D0=BB=D0=B3=D0=B0= =D1=80=D0=B8=D1=8F =D0=95=D0=9E=D0=9E=D0=94 =D0=B8 =D0=A0=D0=B0=D0=B9=D1=84= =D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=98=D0=BC=D0=BE=D1=82=D0=B8 =D0=95=D0=9E= =D0=9E=D0=94 =D0=BD=D0=B5 =D0=BD=D0=BE=D1=81=D1=8F=D1=82 =D0=BE=D1=82=D0=B3= =D0=BE=D0=B2=D0=BE=D1=80=D0=BD=D0=BE=D1=81=D1=82 =D0=B7=D0=B0 =D0=BC=D0=BE= =D0=B4=D0=B8=D1=84=D0=B8=D1=86=D0=B8=D1=80=D0=B0=D0=BD=D0=B8, =D0=BF=D1=80= =D0=BE=D0=BC=D0=B5=D0=BD=D0=B5=D0=BD=D0=B8 =D0=B8=D0=BB=D0=B8 =D1=84=D0=B0= =D0=BB=D1=88=D0=B8=D1=84=D0=B8=D1=86=D0=B8=D1=80=D0=B0=D0=BD=D0=B8 =D1=81= =D1=8A=D0=BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D1=8F. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 15:53:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6D1288AB for ; Fri, 25 Oct 2013 15:53:51 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from nm3-vm7.bullet.mail.sg3.yahoo.com (nm3-vm7.bullet.mail.sg3.yahoo.com [106.10.148.118]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A63E22F71 for ; Fri, 25 Oct 2013 15:53:50 +0000 (UTC) Received: from [106.10.166.61] by nm3.bullet.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:48 -0000 Received: from [106.10.151.139] by tm18.bullet.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:48 -0000 Received: from [127.0.0.1] by omp1007.mail.sg3.yahoo.com with NNFMP; 25 Oct 2013 15:53:48 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 206705.62596.bm@omp1007.mail.sg3.yahoo.com Received: (qmail 9130 invoked by uid 60001); 25 Oct 2013 15:53:48 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1382716427; bh=klw1pNG9nVmwNzrlFQTf252xyxIFweAhUlufCCabr3Y=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=d8ikQTBRkOCDXxF5UZSKap4iHvecd2pV8Augzt27GfUpLTA9yi/oZAS9x/Y/RS/VoluwzoiQ/aLK4xF1FYjWSde3F+O9tdsQiiI5Xr4m33Ckpj9N+KtwccimD7jq36XrhLu+4u5XnGxk2ColqkgdsaK+VxrTFpQB9wPFaR4HIi4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=YkgN8a20TJaHCaLvZ17chN+9KyWcyUJA/XRSlAPyYu1533Y4C+/YNOcRcsHhK5uWnED002rQgnvfucKEpAnYe4+gdmqshM1XiVvXbQZEjjJuNn/08pzaeBSvAOoJchq49IKs/b+K3lWe9maBL7ygGXw2Jsz1AeVIlAq5pDcwgfY=; X-YMail-OSG: WEbHG7YVM1laivZl9fMLty_hL8opVycekky8cofQPCnQnDc o1PqSiaqBvuAjKmyLCbvDnqvnofNHpQZ3zE4t0u23Sbr.P.r3QZDU0Qu3kqd qUNPwadIldsvaEL9wBBJdpZgZCdwij3FjLSzPDYAoIfemCMdJl4lIiYPKkCZ 8HFMz9cxdDjLSbFhK_c2wUHOtvI0BM0Z.znLeYTUcCI193FL_Qm2R6e2zqYS Yk0A._YdM6nDhaEvzPpPEkT9ftXazaxtNZUwaNEaoiSQ9zZ0dhw5MTJWL_dc R3btwmWn75u6uEx85wb_Ppj0kzpLlQNQ91TtrJxrrW1hUV9VUFlsEsWzCo.q TKL7jOQ5b2qLMVLxbMMbLU4lZDcf5B5rHTs1DSUps.xTVxtOKNy1pmLwLOfb INmIsjlTCc28u5gme5wzXCdpo5aKBrO57XriN.Ar43PdHWPjhGax3Upb8C_i F.S0n644X5xI_j9ilDNU6nCJmf.7Zyjc2oJ7R0w9dnY6o_Ki_SqHVboZkZw1 A519yre50mc6.8cSu6q5DZW6NsZ0kDAsmaOjkcVYK8dNHjF2i5qiVrsap6ur sldon_y3Lep51hg-- Received: from [61.15.240.133] by web193503.mail.sg3.yahoo.com via HTTP; Fri, 25 Oct 2013 23:53:47 SGT X-Rocket-MIMEInfo: 002.001, PiBJIHdvdWxkIGFsc28gcmVjb21tZW5kIHRvIHVzZSA0SyBzZWN0b3Igc2l6ZSB1c2luZwpnbm9wIGFuZCB6cG9vbCBleHBvcnQvaW1wb3J0LgoKQWdyZWUsIGluIG15IHNtYWxsIHNldHVwLCA0SyBzZWN0b3JzIGFuZCBhbGlnbmVkIHBhcnRpdGlvbiBjYW4gYmUgc2V2ZW4gb3IgZWlnaHQgdGltZXMgZmFzdGVyIHRoYW4gNTEya2Igc2VjdG9ycyB3aGVuIHRoZSBoYXJkZGlzayBpcyBpbiBhZHZhbmNlIGZvcm1hdCAoNGtiIHBlciBzZWN0b3JzKQoKQWxzbyBzdWdnZXN0IHRvIGhhdmUgdHdvIHNlcGFydGUgZmEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> Message-ID: <1382716427.9107.YahooMailNeo@web193503.mail.sg3.yahoo.com> Date: Fri, 25 Oct 2013 23:53:47 +0800 (SGT) From: Patrick Dung Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance To: "Ivailo.TANUSHEFF@raiffeisen.bg" , Michael Schmiedgen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "owner-freebsd-fs@freebsd.org" , "freebsd-fs@freebsd.org" , "pgsql-admin@postgresql.org" , "freebsd-jail@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Patrick Dung List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 15:53:51 -0000 > I would also recommend to use 4K sector size using=0Agnop and zpool expor= t/import.=0A=0AAgree, in my small setup, 4K sectors and aligned partition c= an be seven or eight times faster than 512kb sectors when the harddisk is i= n advance format (4kb per sectors)=0A=0AAlso suggest to have two separte fa= st SSD (in mirror mode) for the ZIL.=0AThis is to avoid ZFS file system fra= gmentation when the ZIL is stored intent with the ZFS.=0A=0APatrick=0A=0A= =0A=0A=0AOn Thursday, October 24, 2013 10:18 PM, "Ivailo.TANUSHEFF@raiffeis= en.bg" wrote:=0A =0Aowner-freebsd-fs@freeb= sd.org wrote on 23.10.2013 23:57:57:=0A=0A> From:=0A> =0A> Michael Schmiedg= en =0A> =0A> To:=0A> =0A> Mark Felder , Albert Shih , =0A> pgsql-admin@postgresql.org, fre= ebsd-fs@freebsd.org, freebsd-jail@freebsd.org, =0A> =0A> Date:=0A> =0A> 23.= 10.2013 23:58=0A> =0A> Subject:=0A> =0A> Re: ZFS-FreeBSD + postgresql perfo= rmance=0A> =0A> Sent by:=0A> =0A> owner-freebsd-fs@freebsd.org=0A> =0A> On = 10/23/13 19:14, Mark Felder wrote:=0A> > On Wed, Oct 23, 2013, at 11:25, Al= bert Shih wrote:=0A> >> My question is about the performance, I known ZFS e= at all=0Amemory he can=0A> >> have (or almost), so what append when we run = database like=0Apostgresql and=0A> >> jail ? (it's also the reason of 96 Go= ram).=0A> >>=0A> >> Sorry for cross-posting but it's about 3 differents th= ings....=0A> >>=0A> >=0A> > To my understanding the solution is to change t= he primarycache=0Ato=0A> > "metadata" for any ZFS filesystem that you do no= t want=0AZFS to heavily=0A> > cache in memory.=0A> >=0A> > Example:=0A> >= =0A> > # zfs set primarycache=3Dmetadata tank/usr/local/pgsql=0A> =0A> In a= ddition to this it is recommended to set the recordsize=0A> to the fixed va= lue of 8k on dataset creation.=0A> =0A> Michael=0A> =0A=0AI would also reco= mmend to use 4K sector size using=0Agnop and zpool export/import.=0AI agree= that using fixed recordsize is a must when=0Ayou have databases and you ha= ve DB on each jail.=0AAlso there are a bunch of sysctl parameters you may= =0Awant to play with, mostly about tuning the memory usage.=0A=0AAlso it mi= ght be an idea to use partitions on disks=0Aand:=0A1. Use first 1-2G on eac= h disk for swap (just in case).=0A2. Use the rest of the disk for 4K aligne= d and 4K=0Asector size partition, labelled in some way (I use disk ID as la= bel, for=0Aeasy identification).=0A=0ABest regards,=0AIvailo Tanusheff=0A= =0A=0A=0ADISCLAIMER: The contents of this e-mail are intended for the named= addressee only. It contains information that may be privileged and/or conf= idential. Unless you are the named addressee or an authorized designee, you= may not copy or use it, or disclose it to anyone else. If you received it = in error please notify us immediately and then destroy it. Messages transpo= rted over the public Internet are susceptible to alteration. Raiffeisenbank= (Bulgaria) EAD, Raiffeisen Asset Management (Bulgaria) EAD, Raiffeisen Ins= urance Broker EOOD, Raiffeisen Services EAD, Raiffeisen Leasing Bulgaria OO= D, Raiffeisen Auto Leasing Bulgaria EOOD and Raiffeisen Real Estate EOOD sh= all not be liable for the message if altered, changed or falsified. =0A=0A= =D0=9E=D0=93=D0=A0=D0=90=D0=9D=D0=98=D0=A7=D0=95=D0=9D=D0=98=D0=95 =D0=9D= =D0=90 =D0=9E=D0=A2=D0=93=D0=9E=D0=92=D0=9E=D0=A0=D0=9D=D0=9E=D0=A1=D0=A2= =D0=A2=D0=90: =D0=A1=D1=8A=D0=B4=D1=8A=D1=80=D0=B6=D0=B0=D0=BD=D0=B8=D0=B5= =D1=82=D0=BE =D0=BD=D0=B0 =D1=82=D0=BE=D0=B2=D0=B0 =D0=B5=D0=BB=D0=B5=D0=BA= =D1=82=D1=80=D0=BE=D0=BD=D0=BD=D0=BE =D1=81=D1=8A=D0=BE=D0=B1=D1=89=D0=B5= =D0=BD=D0=B8=D0=B5 =D0=B5 =D0=BF=D1=80=D0=B5=D0=B4=D0=BD=D0=B0=D0=B7=D0=BD= =D0=B0=D1=87=D0=B5=D0=BD=D0=BE =D0=B5=D0=B4=D0=B8=D0=BD=D1=81=D1=82=D0=B2= =D0=B5=D0=BD=D0=BE =D0=B7=D0=B0 =D0=BF=D0=BE=D1=81=D0=BE=D1=87=D0=B5=D0=BD= =D0=B8=D1=8F =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B0=D1=82=D0=B5=D0=BB. =D0=A1= =D1=8A=D0=BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D0=B5=D1=82=D0=BE =D1=81=D1=8A= =D0=B4=D1=8A=D1=80=D0=B6=D0=B0 =D0=B8=D0=BD=D1=84=D0=BE=D1=80=D0=BC=D0=B0= =D1=86=D0=B8=D1=8F, =D0=BA=D0=BE=D1=8F=D1=82=D0=BE =D0=BC=D0=BE=D0=B6=D0=B5= =D0=B4=D0=B0 =D0=B1=D1=8A=D0=B4=D0=B5 =D0=BF=D1=80=D0=B5=D0=B4=D0=BD=D0=B0= =D0=B7=D0=BD=D0=B0=D1=87=D0=B5=D0=BD=D0=B0 =D0=B7=D0=B0 =D0=BE=D0=B3=D1=80= =D0=B0=D0=BD=D0=B8=D1=87=D0=B5=D0=BD=D0=BE =D0=BF=D0=BE=D0=BB=D0=B7=D0=B2= =D0=B0=D0=BD=D0=B5 =D0=B8/=D0=B8=D0=BB=D0=B8 =D0=BF=D0=BE=D0=B2=D0=B5=D1=80= =D0=B8=D1=82=D0=B5=D0=BB=D0=BD=D0=B0. =D0=92 =D1=81=D0=BB=D1=83=D1=87=D0=B0= =D0=B9 =D1=87=D0=B5 =D0=BD=D0=B5 =D1=81=D1=82=D0=B5 =D1=83=D0=BA=D0=B0=D0= =B7=D0=B0=D0=BD=D0=B8=D1=8F =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B0=D1=82=D0= =B5=D0=BB =D0=BD=D0=B0 =D1=82=D0=BE=D0=B2=D0=B0 =D1=81=D1=8A=D0=BE=D0=B1=D1= =89=D0=B5=D0=BD=D0=B8=D0=B5 =D0=B8=D0=BB=D0=B8 =D0=BD=D0=B5=D0=B3=D0=BE=D0= =B2 =D0=BE=D1=82=D0=BE=D1=80=D0=B8=D0=B7=D0=B8=D1=80=D0=B0=D0=BD =D0=BF=D1= =80=D0=B5=D0=B4=D1=81=D1=82=D0=B0=D0=B2=D0=B8=D1=82=D0=B5=D0=BB, =D0=BD=D1= =8F=D0=BC=D0=B0=D1=82=D0=B5 =D0=BF=D1=80=D0=B0=D0=B2=D0=BE =D0=B4=D0=B0 =D0= =B3=D0=BE =D0=BA=D0=BE=D0=BF=D0=B8=D1=80=D0=B0=D1=82=D0=B5 =D0=B8=D0=BB=D0= =B8 =D0=B8=D0=B7=D0=BF=D0=BE=D0=BB=D0=B7=D0=B2=D0=B0=D1=82=D0=B5, =D0=BA=D0= =B0=D0=BA=D1=82=D0=BE =D0=B8 =D0=B4=D0=B0 =D1=80=D0=B0=D0=B7=D0=BA=D1=80=D0= =B8=D0=B2=D0=B0=D1=82=D0=B5 =D1=81=D1=8A=D0=B4=D1=8A=D1=80=D0=B6=D0=B0=D0= =BD=D0=B8=D0=B5=D1=82=D0=BE =D0=BC=D1=83 =D0=BF=D1=80=D0=B5=D0=B4 =D1=82=D1= =80=D0=B5=D1=82=D0=B8 =D0=BB=D0=B8=D1=86=D0=B0. =D0=90=D0=BA=D0=BE =D1=81= =D1=82=D0=B5 =D0=B3=D0=BE =D0=BF=D0=BE=D0=BB=D1=83=D1=87=D0=B8=D0=BB=D0=B8 = =D0=BF=D0=BE =D0=B3=D1=80=D0=B5=D1=88=D0=BA=D0=B0, =D0=BC=D0=BE=D0=BB=D1=8F= =D0=B4=D0=B0 =D1=83=D0=B2=D0=B5=D0=B4=D0=BE=D0=BC=D0=B8=D1=82=D0=B5 =D0=BF= =D0=BE=D0=B4=D0=B0=D1=82=D0=B5=D0=BB=D1=8F =D0=BD=D0=B5=D0=B7=D0=B0=D0=B1= =D0=B0=D0=B2=D0=BD=D0=BE =D0=B8 =D0=B4=D0=B0 =D0=B8=D0=B7=D1=82=D1=80=D0=B8= =D0=B5=D1=82=D0=B5 =D1=82=D0=BE=D0=B2=D0=B0 =D1=81=D1=8A=D0=BE=D0=B1=D1=89= =D0=B5=D0=BD=D0=B8=D0=B5 =D0=BE=D1=82 =D1=81=D0=B8=D1=81=D1=82=D0=B5=D0=BC=D0=B0=D1=82=D0=B0. =D0=95=D0=BB=D0=B5= =D0=BA=D1=82=D1=80=D0=BE=D0=BD=D0=BD=D0=B8=D1=82=D0=B5 =D1=81=D1=8A=D0=BE= =D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D1=8F, =D1=80=D0=B0=D0=B7=D0=BF=D1=80=D0=BE= =D1=81=D1=82=D1=80=D0=B0=D0=BD=D1=8F=D0=B2=D0=B0=D0=BD=D0=B8 =D0=BF=D0=BE = =D0=98=D0=BD=D1=82=D0=B5=D1=80=D0=BD=D0=B5=D1=82 =D0=BC=D0=BE=D0=B3=D0=B0= =D1=82 =D0=B4=D0=B0 =D0=B1=D1=8A=D0=B4=D0=B0=D1=82 =D0=BF=D1=80=D0=BE=D0=BC= =D0=B5=D0=BD=D1=8F=D0=BD=D0=B8. =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7= =D0=B5=D0=BD=D0=B1=D0=B0=D0=BD=D0=BA (=D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80= =D0=B8=D1=8F) =D0=95=D0=90=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0= =B7=D0=B5=D0=BD =D0=90=D1=81=D0=B5=D1=82 =D0=9C=D0=B5=D0=BD=D0=B8=D0=B4=D0= =B6=D0=BC=D1=8A=D0=BD=D1=82 (=D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80=D0=B8=D1= =8F) =D0=95=D0=90=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5= =D0=BD =D0=97=D0=B0=D1=81=D1=82=D1=80=D0=B0=D1=85=D0=BE=D0=B2=D0=B0=D1=82= =D0=B5=D0=BB=D0=B5=D0=BD =D0=91=D1=80=D0=BE=D0=BA=D0=B5=D1=80 =D0=95=D0=9E= =D0=9E=D0=94, =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=A1= =D1=8A=D1=80=D0=B2=D0=B8=D1=81=D0=B8=D0=B7 =D0=95=D0=90=D0=94, =D0=A0=D0=B0= =D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=9B=D0=B8=D0=B7=D0=B8=D0=BD= =D0=B3 =D0=91=D1=8A=D0=BB=D0=B3=D0=B0=D1=80=D0=B8=D1=8F =D0=9E=D0=9E=D0=94,= =D0=A0=D0=B0=D0=B9=D1=84=D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=90=D1=83=D1=82= =D0=BE =D0=9B=D0=B8=D0=B7=D0=B8=D0=BD=D0=B3 =D0=91=D1=8A=D0=BB=D0=B3=D0=B0= =D1=80=D0=B8=D1=8F =D0=95=D0=9E=D0=9E=D0=94 =D0=B8 =D0=A0=D0=B0=D0=B9=D1=84= =D0=B0=D0=B9=D0=B7=D0=B5=D0=BD =D0=98=D0=BC=D0=BE=D1=82=D0=B8 =D0=95=D0=9E= =D0=9E=D0=94 =D0=BD=D0=B5 =D0=BD=D0=BE=D1=81=D1=8F=D1=82 =D0=BE=D1=82=D0=B3= =D0=BE=D0=B2=D0=BE=D1=80=D0=BD=D0=BE=D1=81=D1=82 =D0=B7=D0=B0 =D0=BC=D0=BE= =D0=B4=D0=B8=D1=84=D0=B8=D1=86=D0=B8=D1=80=D0=B0=D0=BD=D0=B8, =D0=BF=D1=80= =D0=BE=D0=BC=D0=B5=D0=BD=D0=B5=D0=BD=D0=B8 =D0=B8=D0=BB=D0=B8 =D1=84=D0=B0= =D0=BB=D1=88=D0=B8=D1=84=D0=B8=D1=86=D0=B8=D1=80=D0=B0=D0=BD=D0=B8 =D1=81= =D1=8A=D0=BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D1=8F. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 16:19:55 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 171B473A; Fri, 25 Oct 2013 16:19:55 +0000 (UTC) (envelope-from prvs=101050af22=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5CC442162; Fri, 25 Oct 2013 16:19:54 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50006503893.msg; Fri, 25 Oct 2013 17:19:53 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 25 Oct 2013 17:19:53 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=101050af22=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Patrick Dung" , , "Michael Schmiedgen" References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> <1382716422.13782.YahooMailNeo@web193504.mail.sg3.yahoo.com> Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance Date: Fri, 25 Oct 2013 17:19:54 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org, pgsql-admin@postgresql.org, freebsd-jail@freebsd.org, owner-freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 16:19:55 -0000 ----- Original Message ----- From: "Patrick Dung" Sent: Friday, October 25, 2013 4:53 PM Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance >> I would also recommend to use 4K sector size using > > gnop and zpool export/import. This will have no effect, the ashift for a pool is set at creation. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 17:07:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 310E5CEB; Fri, 25 Oct 2013 17:07:58 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 004822400; Fri, 25 Oct 2013 17:07:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=y6TshWEsn56vRtXXF8HJDawcSIby5qM4A1xuTZ/574s=; b=K9XDPc3hNLiA1wa5kMU51jwyI4GTpd1ezso3DdIa3+TTkiQ4/ZuxocJUzZkBQVCIjxdu/HyuMu+VDOruLYrxU7I9qQbZeOMnb1g2/GYmSh7TZbYivrpgJ35GWoBvw2MKtS8zNMS3gyy1qneEq+eqmnnofSjZaK7WZLnIQHQgSdg=; Received: from localhost.lerctr.org ([127.0.0.1]:36833 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1VZkra-000LIK-NZ; Fri, 25 Oct 2013 12:07:56 -0500 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 25 Oct 2013 12:07:54 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Oct 2013 12:07:54 -0500 From: Larry Rosenman To: Freebsd fs , Freebsd current Subject: ZFS: Invalid SEND/RECV stream still Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.4 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 17:07:58 -0000 between 10.0-BETA1 and 11.0-CURRENT. I've been fighting this battle for a LONG time. I had hoped(!) that it would be solved by the systems being closer in rev but that didn't happen. I'd LOVE someone to look at it. I can provide ssh / root access to both machines. Error Message: received 320KB stream in 1 seconds (320KB/sec) receiving incremental stream of vault/var@2013-10-25 into zroot/backups/TBH/var@2013-10-25 cannot receive incremental stream: invalid backup stream $ Script: $ cat backup-TBH-ZFS.sh #!/bin/sh DATE=`date "+%Y-%m-%d"` #DATE2=2013-03-24 DATE2=`date -v "-1d" "+%Y-%m-%d"` # snap the source ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} # zfs copy the source to here. ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} vault@${DATE} | \ ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH\"" # make sure we NEVER allow the backup stuff to automount. /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh $ Uname's: $ uname -a FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #80 r257016: Wed Oct 23 17:36:15 CDT 2013 root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 $ ssh tbh uname -a FreeBSD thebighonker.lerctr.org 10.0-BETA1 FreeBSD 10.0-BETA1 #1 r256791: Sun Oct 20 12:28:55 CDT 2013 root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 $ What else do y'all need? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 17:33:13 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3889449B; Fri, 25 Oct 2013 17:33:13 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0E5A6257F; Fri, 25 Oct 2013 17:33:12 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1VZlFx-0004gj-NX; Fri, 25 Oct 2013 17:33:05 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r9PHX3eS046892; Fri, 25 Oct 2013 11:33:03 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/qoFOJOMjWBN4eSaqADVab Subject: Re: ZFS: Invalid SEND/RECV stream still From: Ian Lepore To: Larry Rosenman In-Reply-To: References: Content-Type: text/plain; charset="us-ascii" Date: Fri, 25 Oct 2013 11:33:03 -0600 Message-ID: <1382722383.1170.107.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Freebsd fs , Freebsd current X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 17:33:13 -0000 On Fri, 2013-10-25 at 12:07 -0500, Larry Rosenman wrote: > between 10.0-BETA1 and 11.0-CURRENT. > I've been fighting this battle for a LONG time. I had hoped(!) that it > would be solved by the systems being > closer in rev but that didn't happen. > > I'd LOVE someone to look at it. > > I can provide ssh / root access to both machines. > > > Error Message: > received 320KB stream in 1 seconds (320KB/sec) > receiving incremental stream of vault/var@2013-10-25 into > zroot/backups/TBH/var@2013-10-25 > cannot receive incremental stream: invalid backup stream > $ > > Script: > $ cat backup-TBH-ZFS.sh > #!/bin/sh > DATE=`date "+%Y-%m-%d"` > #DATE2=2013-03-24 > DATE2=`date -v "-1d" "+%Y-%m-%d"` > # snap the source > ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} > # zfs copy the source to here. > ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} > vault@${DATE} | \ > ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH\"" > # make sure we NEVER allow the backup stuff to automount. > /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ > awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh > $ > > Uname's: > $ uname -a > FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #80 r257016: > Wed Oct 23 17:36:15 CDT 2013 > root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 > $ ssh tbh uname -a > FreeBSD thebighonker.lerctr.org 10.0-BETA1 FreeBSD 10.0-BETA1 #1 > r256791: Sun Oct 20 12:28:55 CDT 2013 > root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 > $ > > What else do y'all need? > > > I notice you're using ssh and a pipe -- does your shell logon script echo anything to stdout when you connect? I've had that interfere with piped ssh stuff. Most shells provide a way for the logon script to determine "is this an interactive session?" and you can use that to avoid echoing anything that would interfere with non-interactive automation. -- Ian From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 17:36:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F154E81A; Fri, 25 Oct 2013 17:36:26 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BCC8025D9; Fri, 25 Oct 2013 17:36:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=KEvZtj/rYUHkioaRtH4m1x3Hn5nWHtoWI2AkwStOdRo=; b=NoxRMWAhICPzdL/7CnFn8K8CQdiJWYL0B7f3VWiMvQXVC/uWCh8NA+yi7YoBIpEwq/vn+myQ3gE/1Sc8fJeczW3Zjol6iZyb9m7wGGwao40kyJlTsMUlyGw5ajdJZBWX+8BxznEwa2PrPDhgS/8tV1Vd4QXW7d7ZIcVe0OUN8bI=; Received: from localhost.lerctr.org ([127.0.0.1]:20320 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1VZlIm-000LdS-59; Fri, 25 Oct 2013 12:36:23 -0500 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 25 Oct 2013 12:35:59 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Oct 2013 12:35:59 -0500 From: Larry Rosenman To: Ian Lepore Subject: Re: ZFS: Invalid SEND/RECV stream still In-Reply-To: <1382722383.1170.107.camel@revolution.hippie.lan> References: <1382722383.1170.107.camel@revolution.hippie.lan> Message-ID: <4b27106e75966fc555478400915d375f@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.4 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 Cc: Freebsd fs , Freebsd current X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 17:36:27 -0000 On 2013-10-25 12:33, Ian Lepore wrote: > On Fri, 2013-10-25 at 12:07 -0500, Larry Rosenman wrote: >> between 10.0-BETA1 and 11.0-CURRENT. >> I've been fighting this battle for a LONG time. I had hoped(!) that >> it >> would be solved by the systems being >> closer in rev but that didn't happen. >> >> I'd LOVE someone to look at it. >> >> I can provide ssh / root access to both machines. >> >> >> Error Message: >> received 320KB stream in 1 seconds (320KB/sec) >> receiving incremental stream of vault/var@2013-10-25 into >> zroot/backups/TBH/var@2013-10-25 >> cannot receive incremental stream: invalid backup stream >> $ >> >> Script: >> $ cat backup-TBH-ZFS.sh >> #!/bin/sh >> DATE=`date "+%Y-%m-%d"` >> #DATE2=2013-03-24 >> DATE2=`date -v "-1d" "+%Y-%m-%d"` >> # snap the source >> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >> # zfs copy the source to here. >> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} >> vault@${DATE} | \ >> ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH\"" >> # make sure we NEVER allow the backup stuff to automount. >> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >> $ >> >> Uname's: >> $ uname -a >> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #80 r257016: >> Wed Oct 23 17:36:15 CDT 2013 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >> $ ssh tbh uname -a >> FreeBSD thebighonker.lerctr.org 10.0-BETA1 FreeBSD 10.0-BETA1 #1 >> r256791: Sun Oct 20 12:28:55 CDT 2013 >> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 >> $ >> >> What else do y'all need? >> >> >> > > I notice you're using ssh and a pipe -- does your shell logon script > echo anything to stdout when you connect? I've had that interfere with > piped ssh stuff. Most shells provide a way for the logon script to > determine "is this an interactive session?" and you can use that to > avoid echoing anything that would interfere with non-interactive > automation. > > -- Ian no, it does NOT. I've verified that, plus this error is a LONG way into the stream. (I cut a bunch of the previous output). I've even gathered tmp files from this before, but no one had the time to look at it. Thanks for at least looking. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 19:36:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A63983F4; Fri, 25 Oct 2013 19:36:33 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 71FB22CAD; Fri, 25 Oct 2013 19:36:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=Ta1WFlndqeSA+7LyOoX/XWU6ekfzHc/nc0CbgJaHHo0=; b=Bx1pUpVWKhIMce5HaLqyFAeKpbOWBhsRX4Hj5vmQC//zQUs3b5/5A3j2d8kae9XxF1PliWu8uc7OXb+SXuh1GAmfEuXB6Rr5ZLeGHDLmHktfMqql4QAhnHqvjDpGbr2t4XjNSP37FPEzWxGhOVpVyQ39o4K6CbpVfRbsAvAzGVI=; Received: from localhost.lerctr.org ([127.0.0.1]:12582 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1VZnBO-0003Xc-VB; Fri, 25 Oct 2013 14:36:32 -0500 Received: from [32.97.110.59] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 25 Oct 2013 14:36:30 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Oct 2013 14:36:30 -0500 From: Larry Rosenman To: Garrett Wollman Subject: Re: ZFS: Invalid SEND/RECV stream still In-Reply-To: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> References: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> Message-ID: <29a687d287cf28d34283566eba509486@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.5 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 Cc: Freebsd fs , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 19:36:33 -0000 On 2013-10-25 14:33, Garrett Wollman wrote: > In article , > Larry Rosenman wrote: > >> Error Message: >> received 320KB stream in 1 seconds (320KB/sec) >> receiving incremental stream of vault/var@2013-10-25 into >> zroot/backups/TBH/var@2013-10-25 >> cannot receive incremental stream: invalid backup stream > > I've seen this happen in one very specific case: if the receiving > machine is doing automatic snapshots, and happens to do a snapshot of > the filesystem being received *while it's being received*. In that > case, it appears that the filesystem on disk gets subtly corrupted, > and the only remedy is to destroy it on the receiving machine and > start all over (making sure to disable automatic snapshots this time). > > "zfs receive -F" will rollback snapshots made on the receiving system > that don't exist on the sending system, but it only does this at the > beginning of the stream -- it can't prevent further snapshots from > being taken during reception of an incremental stream which foul up > the works. > > -GAWollman There are NO automatic snapshots in play here at all :( -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 20:48:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7D568B38 for ; Fri, 25 Oct 2013 20:48:43 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qe0-x235.google.com (mail-qe0-x235.google.com [IPv6:2607:f8b0:400d:c02::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3F3AA202A for ; Fri, 25 Oct 2013 20:48:43 +0000 (UTC) Received: by mail-qe0-f53.google.com with SMTP id cy11so2697298qeb.26 for ; Fri, 25 Oct 2013 13:48:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=GkXZUxYB0gdN173OihEWI6SBwZKXREshB6MNC+vKfuU=; b=S0Q/ncMhlHEmjE4U9OW7L671lY+A8eDiDlMKfFtUclq/pcFoByqcMC3kKm227PCCYP QnM1FMUdgJSVMz9qgNlZTZTX9mYjg5Ya3+LC5Sp5l1Pq3ABZdZeJQlw85rAKJQE4Lkly WaiV5itQQUpuAMov35QJjQwRG8StKUfsftXPSJ785MoE7za/e5pZXt//zbAvuQHi3Oq2 5zAtsf+3eOTuP12b/IhobeW7qLRIN1S04D5gOG81oBrFBBHd+CZDXwONEGx2pUG5wD8U 0sOnxqThdpZIsyH5+clDqzWMocQThaKkpXnMgfNEg5wk7pxW6YAr4K/YSp3flZ//LnD2 ElnQ== MIME-Version: 1.0 X-Received: by 10.224.121.6 with SMTP id f6mr14304940qar.64.1382734122445; Fri, 25 Oct 2013 13:48:42 -0700 (PDT) Received: by 10.96.180.233 with HTTP; Fri, 25 Oct 2013 13:48:42 -0700 (PDT) In-Reply-To: <20131025103746.g0u1s5xlgc0k8c48@imp.redlamb.net> References: <20131025103746.g0u1s5xlgc0k8c48@imp.redlamb.net> Date: Fri, 25 Oct 2013 23:48:42 +0300 Message-ID: Subject: Re: missing files on ext2fs with 9-STABLE From: Kimmo Paasiala To: Pete Erickson Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 20:48:43 -0000 On Fri, Oct 25, 2013 at 6:37 PM, Pete Erickson wrote: > I'm missing files on an ext2 filesystem in 9-STABLE. However, remounting the > drive in Ubuntu (12.04.3) the files are back. The missing files range from a > few KB to several GB so it's not based on the file sizes. While I've found > an ugly way past this, I'm trying to understand why I'm having the problem > in the first place. > > The ext2 fs was created in Ubuntu (12.04.3) in order to backup approx 1.4 TB > from a hardware raid ext4 fs and copy it back once I repurpose the drives as > ZFS in FreeBSD. The ext2 fs is on a gpt-labeled 3.0TB drive and I used rsync > to copy the data. Unfortunately, I'm not getting any error messages so I > have nothing to provide. > > FYI, my temp fix is mounting the raw ext2 fs to a ubuntu VM in VirtualBox > and then rsync'ing it the ZFS fs via the virtual network. While it's ugly, > it appears that it'll work. However, I'm hesitant to destroy the ext4 fs > until I'm sure that I have the data backed up. > > Any help understanding this would be appreciated. Is the filesystem larger than 2TB? I recall that the ext2 driver in FreeBSD has a severe limitation with the filesystem size that limits the usable filesystems to 2TBs, anything larger will give undefined results when used. -Kimmo From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 20:55:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AA075E65 for ; Fri, 25 Oct 2013 20:55:14 +0000 (UTC) (envelope-from redlamb@redlamb.net) Received: from ironhide.redlamb.net (ironhide.redlamb.net [216.17.90.180]) by mx1.freebsd.org (Postfix) with ESMTP id 858FF20DA for ; Fri, 25 Oct 2013 20:55:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ironhide.redlamb.net (Postfix) with ESMTP id D05A7C7912; Fri, 25 Oct 2013 15:55:13 -0500 (CDT) Received: from ironhide.redlamb.net ([127.0.0.1]) by localhost (ironhide.redlamb.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 93879-03; Fri, 25 Oct 2013 15:55:13 -0500 (CDT) Received: from does.not.exist (localhost [127.0.0.1]) by ironhide.redlamb.net (Postfix) with ESMTP id 03C91C67D6; Fri, 25 Oct 2013 15:55:12 -0500 (CDT) Date: Fri, 25 Oct 2013 15:54:57 -0500 From: redlamb@redlamb.net To: Kimmo Paasiala Subject: Re: missing files on ext2fs with 9-STABLE Message-ID: <20131025205457.GA3599@does.not.exist> Mail-Followup-To: Kimmo Paasiala , FreeBSD Filesystems References: <20131025103746.g0u1s5xlgc0k8c48@imp.redlamb.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt X-GPG-Fingerprint: 60EF 3594 0953 353E ABB3 F5D0 EFF5 EAC1 A989 6169 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 20:55:14 -0000 On Fri, Oct 25, 2013 at 11:48:42PM +0300, Kimmo Paasiala wrote: > On Fri, Oct 25, 2013 at 6:37 PM, Pete Erickson wrote: > > I'm missing files on an ext2 filesystem in 9-STABLE. However, remounting the > > drive in Ubuntu (12.04.3) the files are back. The missing files range from a > > few KB to several GB so it's not based on the file sizes. While I've found > > an ugly way past this, I'm trying to understand why I'm having the problem > > in the first place. > > > > The ext2 fs was created in Ubuntu (12.04.3) in order to backup approx 1.4 TB > > from a hardware raid ext4 fs and copy it back once I repurpose the drives as > > ZFS in FreeBSD. The ext2 fs is on a gpt-labeled 3.0TB drive and I used rsync > > to copy the data. Unfortunately, I'm not getting any error messages so I > > have nothing to provide. > > > > FYI, my temp fix is mounting the raw ext2 fs to a ubuntu VM in VirtualBox > > and then rsync'ing it the ZFS fs via the virtual network. While it's ugly, > > it appears that it'll work. However, I'm hesitant to destroy the ext4 fs > > until I'm sure that I have the data backed up. > > > > Is the filesystem larger than 2TB? I recall that the ext2 driver in > FreeBSD has a severe limitation with the filesystem size that limits > the usable filesystems to 2TBs, anything larger will give undefined > results when used. > Yes, the partition is 3TB in size. I'll try and recreate the ext2 filesystem to be less than 2TB and see if that works. It can't be any slower then copying the data using a VM. Thanks for the response. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 22:09:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 52905EC; Fri, 25 Oct 2013 22:09:09 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1B01F257B; Fri, 25 Oct 2013 22:09:09 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id aq17so7669038iec.10 for ; Fri, 25 Oct 2013 15:09:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=03TI34FHCN/OFm94/jNmfqQvbknXGYK+UplLO3zQMEU=; b=bVeaN7seJBHhTzp24dRbu3NN+Qc87zUsSAr+akUVyqAwJX0R0NZXIBXU0kTf86lxdh LMmSXD3RrkHW6Z3FoYoTS/vxsLKuouitKhElap5gSzI4FgdY7nacXoDVQ/1cHs56noY4 ZK0jeKgFeVpHulMk8ujAPBk8s6MR8XhamCr8zCUrMFGj3VSGnF/e066bGT3iT4GTnEUO mSGBCvdhwU0s/OLuHBnT5bR9NH9ecVzJQBjLfyREFnsYkZjLmDxqwr+6lDovhlnoeOwR 1SrwNEiKEZh4uJeIYx0JW6azPYj5EsJ+aBJUj/S4e8co4rywZO2ODsCIFW0p+vTaCQ1O MDow== MIME-Version: 1.0 X-Received: by 10.50.117.40 with SMTP id kb8mr328980igb.60.1382738947820; Fri, 25 Oct 2013 15:09:07 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Fri, 25 Oct 2013 15:09:07 -0700 (PDT) In-Reply-To: <29a687d287cf28d34283566eba509486@webmail.lerctr.org> References: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> <29a687d287cf28d34283566eba509486@webmail.lerctr.org> Date: Fri, 25 Oct 2013 18:09:07 -0400 X-Google-Sender-Auth: RMwkvFJdr8ZZHfK-bvvs409RfKY Message-ID: Subject: Re: ZFS: Invalid SEND/RECV stream still From: J David To: Larry Rosenman Content-Type: text/plain; charset=ISO-8859-1 Cc: Freebsd fs , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 22:09:09 -0000 On Fri, Oct 25, 2013 at 3:36 PM, Larry Rosenman wrote: >>> cannot receive incremental stream: invalid backup stream It may help to decouple the steps. I.e. do a "zfs send blah blah >file" then copy the file and do a "zfs recv blah blah Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 22974202; Fri, 25 Oct 2013 22:11:21 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E230C25E9; Fri, 25 Oct 2013 22:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=+UNADGfiMf/7uB1u/Z5H7LJzXEaPShdNJmIdb+F+GKQ=; b=VIA9rICDUICR2IRjqv/xk94ns+HHJ6Qu8YWPPndF60ClDXGzq+H0MFXa1b4M4k/mWl269iPrgRr0pNm568lgw5mWy+gSr7EV5W7axP/9mnbY0uPc6bKK/GhucD2nUBNTjwxHY5+rUnDnShfiYRa42L7+ih+tUcPLtidAKwiCfsU=; Received: from localhost.lerctr.org ([127.0.0.1]:34699 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1VZpbB-0008DQ-TJ; Fri, 25 Oct 2013 17:11:19 -0500 Received: from cpe-72-182-93-216.austin.res.rr.com ([72.182.93.216]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 25 Oct 2013 17:11:17 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Oct 2013 17:11:17 -0500 From: Larry Rosenman To: J David Subject: Re: ZFS: Invalid SEND/RECV stream still In-Reply-To: References: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> <29a687d287cf28d34283566eba509486@webmail.lerctr.org> Message-ID: <723e41d2533cfb09bf6892a172592744@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.5 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 Cc: Freebsd fs , current@freebsd.org, jdavidlists@gmail.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 22:11:21 -0000 On 2013-10-25 17:09, J David wrote: > On Fri, Oct 25, 2013 at 3:36 PM, Larry Rosenman wrote: >>>> cannot receive incremental stream: invalid backup stream > > It may help to decouple the steps. I.e. do a "zfs send blah blah >> file" then copy the file and do a "zfs recv blah blah second host. > > If nothing else, it should help you demonstrate that it's not the > machine-to-machine copy that's at fault, because you can take the > sha256 sum of the file on both sides and prove it's the same. > > Good luck! we've tried this, and still the same issue. So. I'm not sure how to diagnose it. The message out of zfs send/recv is not very informative. Other ideas? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 22:17:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D64B54F2; Fri, 25 Oct 2013 22:17:43 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22c.google.com (mail-ie0-x22c.google.com [IPv6:2607:f8b0:4001:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D0D5265A; Fri, 25 Oct 2013 22:17:43 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id tp5so7707248ieb.3 for ; Fri, 25 Oct 2013 15:17:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=9y6UAQhUQFJEr8Tt3d/kaRAFniDO2BaG0datjdtr3vc=; b=fndHfQ8gPmpJQg5eqmJwD22jCPuAKNYKii64o9Bfccg9INuLdQn2mVJ0apTd1UpQYi mKtz8m0DfBAMUg0ZPltR4Qh05qIhOjzpDWYVfsZE/MfTjDo7i7zMsl0ED3Dz7TcM5ihV jh+xr64nT5kK2z3CtTk85+piRXVfnvsBmJ9DhsqGgvwEMHQprxy4ZagKjHz5Kkzw1Mxh 1CxNmpevMzJGDCXpQCC8oI3gGwL8y8BnvkuZOOJPi8U9t7ANB3BZAP5Jnm5EDvNj2JNg Pgi3h6rEo7e40afIaHkYa6uWw0wfsf+8K2YAkBWtivNeUOUf0Oon8oBrQpvnP6lnF9/l E4Zg== MIME-Version: 1.0 X-Received: by 10.50.1.78 with SMTP id 14mr292860igk.37.1382739462950; Fri, 25 Oct 2013 15:17:42 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Fri, 25 Oct 2013 15:17:42 -0700 (PDT) In-Reply-To: <723e41d2533cfb09bf6892a172592744@webmail.lerctr.org> References: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> <29a687d287cf28d34283566eba509486@webmail.lerctr.org> <723e41d2533cfb09bf6892a172592744@webmail.lerctr.org> Date: Fri, 25 Oct 2013 18:17:42 -0400 X-Google-Sender-Auth: 0KtW8qaDVnoCW1gqPY4B4qnlYzk Message-ID: Subject: Re: ZFS: Invalid SEND/RECV stream still From: J David To: Larry Rosenman Content-Type: text/plain; charset=ISO-8859-1 Cc: Freebsd fs , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 22:17:43 -0000 On Fri, Oct 25, 2013 at 6:11 PM, Larry Rosenman wrote: > we've tried this, and still the same issue. Post that and the shorter local-only reproduction then, so people don't spend time looking at the network layer. > So. I'm not sure how to diagnose it. > > The message out of zfs send/recv is not very informative. > > Other ideas? Two: 1) Do a non-incremental send/receive and see if it works. 2) Set up this exact test between two 10.0-BETA1 machines and then two 11-CURRENT machines to demonstrate whether the problem is specific to one version or the other, or limited to the interaction between the two versions. If the latter, then you may have to start bisecting commits to find the breaking point. Good luck! From owner-freebsd-fs@FreeBSD.ORG Fri Oct 25 22:20:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0FD8E5AD; Fri, 25 Oct 2013 22:20:15 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D038B267E; Fri, 25 Oct 2013 22:20:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=T2cY7QDLQQzvpoN5EuC81M7mPSGI1eSNYyJF47nl8lg=; b=ilY9L9yAjg4WQofzSH9tQXWwMiYmyf/eoNn50Ulxeh5l6tofhMlE1kvlIGKB5Url4vimt3derlOY0uedwjdbiPKu8Wo26CTqR/7omligFsDp/94LRBxCI4s0s1hqsvN3oxpsmdhC8f+bB1lyKz+fKe1LiDWIUHFw8BKHfylehw4=; Received: from localhost.lerctr.org ([127.0.0.1]:49846 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1VZpjo-0008J6-BP; Fri, 25 Oct 2013 17:20:13 -0500 Received: from cpe-72-182-93-216.austin.res.rr.com ([72.182.93.216]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 25 Oct 2013 17:20:12 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Oct 2013 17:20:12 -0500 From: Larry Rosenman To: J David Subject: Re: ZFS: Invalid SEND/RECV stream still In-Reply-To: References: <201310251933.r9PJX8sx086597@hergotha.csail.mit.edu> <29a687d287cf28d34283566eba509486@webmail.lerctr.org> <723e41d2533cfb09bf6892a172592744@webmail.lerctr.org> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.5 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.375 Cc: Freebsd fs , current@freebsd.org, jdavidlists@gmail.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 22:20:15 -0000 On 2013-10-25 17:17, J David wrote: > On Fri, Oct 25, 2013 at 6:11 PM, Larry Rosenman wrote: >> we've tried this, and still the same issue. > > Post that and the shorter local-only reproduction then, so people > don't spend time looking at the network layer. > >> So. I'm not sure how to diagnose it. >> >> The message out of zfs send/recv is not very informative. >> >> Other ideas? > > Two: > > 1) Do a non-incremental send/receive and see if it works. This worked. > > 2) Set up this exact test between two 10.0-BETA1 machines and then two > 11-CURRENT machines to demonstrate whether the problem is specific to > one version or the other, or limited to the interaction between the > two versions. If the latter, then you may have to start bisecting > commits to find the breaking point. Hard to do, as my choices are the 2 machines I have. This has been going on since 8.x (see posts from me in the list archives from March to now). > > Good luck! -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Sat Oct 26 05:16:37 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1B2A468C; Sat, 26 Oct 2013 05:16:37 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x233.google.com (mail-ve0-x233.google.com [IPv6:2607:f8b0:400c:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD03B2898; Sat, 26 Oct 2013 05:16:36 +0000 (UTC) Received: by mail-ve0-f179.google.com with SMTP id cz12so3484737veb.10 for ; Fri, 25 Oct 2013 22:16:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=/hvMeaanInXv5QqAZtU4dvM/HJ/CJCgogt6t5S1Ky1M=; b=n3Q4ggUXcJ7Km/pYWUpiyb82I3Eyt2wU0pNEayNpueiKJ+PRl0i0av0haSyg1xKT0d 6kiauQ+aSlfEpsoviUAzaXLCFhEg9O3ZVLYlQ7ZSkR8leZyLX9bYv3MvyMHUaGlaQ8Om hLsChT4oP2jfqflgp3a4xkRNgHihUtx7hjJnWsAox/Nq8+h09fpn/oQa+1hUU0hC6NkB uO8uHbrEZVqwElEk8geey0IR37vZNgT+xAtBIY9cB2hKGv2tMEPQM9siCy8iZwEXfzmc RN30m9AjNFeCWa+JSNxFdyAQ+RCHB6kvEJdsuwyWJiJJSoW4DJBhV55+0fQi4Qiqy99G X5Hw== MIME-Version: 1.0 X-Received: by 10.58.146.71 with SMTP id ta7mr6806976veb.23.1382764595491; Fri, 25 Oct 2013 22:16:35 -0700 (PDT) Received: by 10.220.30.130 with HTTP; Fri, 25 Oct 2013 22:16:35 -0700 (PDT) Date: Sat, 26 Oct 2013 01:16:35 -0400 Message-ID: Subject: Or it could be ZFS memory starvation and 9k packets (was Re: istgt causes massive jumbo nmbclusters loss) From: Zaphod Beeblebrox To: FreeBSD Net , freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 05:16:37 -0000 At first I thought this was entirely the interaction of istgt and 9k packets, but after some observation (and a few more hangs) I'm reasonably positive it's a form of resource starvation related to ZFS and 9k packets. To reliably trigger the hang, I need to do something that triggers a demand for 9k packets (like istgt traffic, but also bit torrent traffic --- as you see the MTU is 9014) and it must have been some time since the system booted. ZFS is fairly busy (with both NFS and SMB guests), so it generally takes quite a bit of the 8G of memory for itself. Now... below the netstat -m shows 1399 9k bufs with 376 available. When the network gets busy, I've seen 4k or even 5k bufs in total... never near the 77k max. After some time of lesser activity, the number of 9k buffers returns to this level. When the problem occurs, the number of denied buffers will shoot up at the rate of several hundred or even several thousand per second, but the system will not be "out" of memory. Top will show 800 meg often in the free column when this happens. While it's happening, when I'm logged into the console, none of these stats seem out of place, save the number of denied 9k buffer allocations and the "cache" of 9k buffers will be less than 10 (but I've never seen it at 0). On Tue, Oct 22, 2013 at 3:42 PM, Zaphod Beeblebrox wrote: > I have a server > > FreeBSD virtual.accountingreality.com 9.2-STABLE FreeBSD 9.2-STABLE #13 > r256549M: Tue Oct 15 16:29:48 EDT 2013 > root@virtual.accountingreality.com:/usr/obj/usr/src/sys/VRA amd64 > > That has an em0 with jumbo packets enabled: > > em0: flags=8843 metric 0 mtu 9014 > > It has (among other things): ZFS, NFS, iSCSI (via istgt) and Samba. > > Every day or two, it looses it's ability to talk to the network. ifconfig > down/up on em0 gives the message about not being able to allocate the > receive buffers... > > With everything running, but with specifically iSCSI not used, everything > seems good. When I start hitting istgt, I see the denied stat for 9k mbufs > rise very rapidly (this amount only took a few seconds): > > [1:47:347]root@virtual:/usr/local/etc/iet> netstat -m > 1313/877/2190 mbufs in use (current/cache/total) > 20/584/604/523514 mbuf clusters in use (current/cache/total/max) > 20/364 mbuf+clusters out of packet secondary zone in use (current/cache) > 239/359/598/261756 4k (page size) jumbo clusters in use > (current/cache/total/max) > 1023/376/1399/77557 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/43626 16k jumbo clusters in use (current/cache/total/max) > 10531K/6207K/16738K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 0/50199/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > ... the denied number rises... and somewhere in the millions or more the > machine stops --- but even with the large number of denied 9k clusters, the > "9k jumbo clusters in use" line will always indicate some available. > > ... so is this a tuning or a bug issue? I've tried ietd --- basically it > doesn't want to work with a zfs zvol, it seems (refuses to use it). > > From owner-freebsd-fs@FreeBSD.ORG Sat Oct 26 05:23:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 17EF27CF for ; Sat, 26 Oct 2013 05:23:43 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from nm1-vm1.bullet.mail.sg3.yahoo.com (nm1-vm1.bullet.mail.sg3.yahoo.com [106.10.148.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7C58328EA for ; Sat, 26 Oct 2013 05:23:42 +0000 (UTC) Received: from [106.10.166.61] by nm1.bullet.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:23:34 -0000 Received: from [106.10.150.27] by tm18.bullet.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:23:33 -0000 Received: from [127.0.0.1] by omp1028.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:23:33 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 940745.66242.bm@omp1028.mail.sg3.yahoo.com Received: (qmail 23919 invoked by uid 60001); 26 Oct 2013 05:23:33 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1382765013; bh=Mo3yxvxwwK/n+yKjTq5FeeXPaE+iNZZ5Tj3mx7NCXDw=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=lEjB3jN/mEBN1HejHDUF/pJLEDmA52L1BPQiSpnl1lzRLUibCWuayR9HEI3I1jKDNDevD0gWDldmASLEbU38XMqP1i3WaNzmGt4K2jaj3CVlQ2IVG0LeSSMlumRgfd+6Rl+C9vRKXuC79JY9NLlQbBmdcpPltpJ4gdOvTYojyCw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=xt+Dj2pGACn36EvjozN2qHlqY4lA/U9H6vST07S6OmrWddqMd9J/bJqelG8BWJ4EhePB8Oi4LD3XuQj2V8+K85N5CaOvnQOJipGYk91z6xXk2J0cJx6M6U3SPZUpqNIDINkTsIm+6Hdi0mvdhwkFKWraIq9QSXHUlKxQ5gimL30=; X-YMail-OSG: _hhhqQsVM1lyy4P1tSWoK7QmRjD27b4qRZemOd8BVqKpNmn QjfNlPhAYzQRw4nh_fF230tonyrTkkcGTnIs1xaU4tYtUuM1w9lmRDZsCdAv c9gavmWYbSLFzTSRQrr4WjmxxpIphoMX8k3fkXdEHVK8KRaGsgu3JUspZMyg h7vChCfG7XWB1Zhb4b6Gr06LHUtyPDXpsbYodrqrOSVi9emCx7m0KqVefkJC 1S4ljOhxjHX1spL76Kpj0RPy81zig_ovHzb8o7VXD4nJ0JBpNJjHNlh7Xlgc Zwuwjeqf5zHugjpNSnEwd6ADvXJwfO6ju.6s6m0OnGt8rMhMfM6lQCepGu.G OR9_jSvkpTadU4eluGqbD3H1Gugvbzp3If9ElzhnfFxlszsE3MTyLlV.ggnY EOKmjMzAEFGdxiwBzZ2Qt0rpYay3NSvA5sBiAx0pohAegrSOUt142ggxfxgt 56ss_qOx2Ni9uoSH7MWV0tgQ.EYZdfg55yvpK9onCg.bOSoYJAU4Ote0WjK0 nidffZoGhJwcVaInZCXA.x58wrIrJM0PL0Ej0R1Q_g80136aRA0KczzJOh_X ZrTK06aw3ic_esSN_b5s- Received: from [61.15.240.133] by web193502.mail.sg3.yahoo.com via HTTP; Sat, 26 Oct 2013 13:23:33 SGT X-Rocket-MIMEInfo: 002.001, CgpPbiBTYXR1cmRheSwgT2N0b2JlciAyNiwgMjAxMyAxOjA0IEFNLCBTdGV2ZW4gSGFydGxhbmQgPGtpbGxpbmdAbXVsdGlwbGF5LmNvLnVrPiB3cm90ZToKIAotLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0tLS0tIApGcm9tOiAiUGF0cmljayBEdW5nIiA8cGF0cmlja19ka3RAeWFob28uY29tLmhrPgpTZW50OiBGcmlkYXksIE9jdG9iZXIgMjUsIDIwMTMgNDo1MyBQTQpTdWJqZWN0OiBSZTogW0FETUlOXSBaRlMtRnJlZUJTRCArIHBvc3RncmVzcWwgcGVyZm9ybWFuY2UKCgo.PiBJIHdvdWxkIGFsc28gcmVjb20BMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> <1382716422.13782.YahooMailNeo@web193504.mail.sg3.yahoo.com> Message-ID: <1382765013.22777.YahooMailNeo@web193502.mail.sg3.yahoo.com> Date: Sat, 26 Oct 2013 13:23:33 +0800 (SGT) From: Patrick Dung Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance To: Steven Hartland , "Ivailo.TANUSHEFF@raiffeisen.bg" , Michael Schmiedgen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" , "pgsql-admin@postgresql.org" , "freebsd-jail@freebsd.org" , "owner-freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Patrick Dung List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 05:23:43 -0000 =0A=0AOn Saturday, October 26, 2013 1:04 AM, Steven Hartland wrote:=0A =0A----- Original Message ----- =0AFrom: "Patrick Dun= g" =0ASent: Friday, October 25, 2013 4:53 PM=0ASu= bject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance=0A=0A=0A>> I would = also recommend to use 4K sector size using=0A>=0A> gnop and zpool export/im= port.=0A=0AThis will have no effect, the ashift for a pool is set at creati= on.=0A=0A=A0 =A0 Regards=0A=A0 =A0 Steve=0A=0A=0A---------------------=0A= =0ATo be specific, I think the steps should be gnop create, create zfs pool= , export zfs, remove gnop and reimport zfs.=0AThis link have the deatils an= d steps: From owner-freebsd-fs@FreeBSD.ORG Sat Oct 26 05:24:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 53C4D86E for ; Sat, 26 Oct 2013 05:24:27 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from nm9-vm6.bullet.mail.sg3.yahoo.com (nm9-vm6.bullet.mail.sg3.yahoo.com [106.10.148.213]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9000328FB for ; Sat, 26 Oct 2013 05:24:26 +0000 (UTC) Received: from [106.10.166.124] by nm9.bullet.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:24:24 -0000 Received: from [106.10.151.186] by tm13.bullet.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:24:24 -0000 Received: from [127.0.0.1] by omp1012.mail.sg3.yahoo.com with NNFMP; 26 Oct 2013 05:24:24 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 135079.24890.bm@omp1012.mail.sg3.yahoo.com Received: (qmail 36799 invoked by uid 60001); 26 Oct 2013 05:24:23 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1382765063; bh=XhN/luD6GySM20955Al18hbmlaTTRJ0N6iJ3gTSWdic=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=y5pOGVfMW+b4j/CPpSg2BNho7/DI2HEb7Hw9SYZ1q4iGOs3lHBYFeUf+vnFt6trWr7CPDN5SMTRghOKSLvmOf/Do0Rctbf2z7ZMz8qeYAadkjr4N959IRsylmDGpnX8bEf2dEKEuV4fBuIIdbAWXxFFDojSY9MY62UIznihpfn4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=shWeDuwsO+RtugCTd8ACaeoL5vSXgTrKua/oYFF4otiZrQctr0ZtSevBW4ymBj+3B7QLUiSCxMmZiT2jno96N1V3Jm7122IFvjmdzvWKeUrGdhgcHqubDSygFd2bEJ6ajAtfCsGhor9eDFzmc5F86S0Ns3jyUgP98521s9WE/JM=; X-YMail-OSG: qw7EvoEVM1nXWr6ew5BfdYvLN7x1MUMP_zsmBdL0MCykLVV 8yoo2nzKOWI5i12sdIzzqfFdUlIc_xRwURjGcyiEOSopdogy6pfqkC_MTM20 m3vte8yVD7e2G2J3fVobo9KUtz5G_Df21tCuW_SlJdmWQ65zcS43KEJxzbjS BTy98g_gENdyRZ671lApAVPS2nhFAuwd6LZGZ.NlWisuW1y5_mo2tgiqvMCa QsLqiKGT6RKOsQJ2aMkuV2S5laQLQjCLKzLCREVGb_FlSHGBRebDyUnLzuqB inPra0G3zVuePimLIk9CHMTKM3Aacv_Epi3iMLBO0BUbt2l.YkcvsGDNKzNy 2foklJMpjMwmjS4f6AIIInuZsfmpJknAGlPkdXFueGm7TKivgC54L8fokJFx Fq.lvg312HiJbpzXI9LuSLSHlvHGSbaupvxRXmugd8IO10AL.CQpxHHrqRPE FCXtnmnPDv.FEo9ABZjI4aGB2jjIPlvEcfmw3bcRKEubojxOKPGjpm2MxJqG bQ3qXor3KT3EPQ0gZ3_uMqdJel.myz0KMkmR_XXjfCF_cj2ZCK72UQcdKMDN 90Lv56ugWyBBg2IQXJIVc30deLeUlexPhg2zNO5ZO2i_kDcMYZXq697j46HK 87T109S9yxB6l5IqGBgK7UZdTByNPCcVpqb6xB0IOwNlSl_5m5FLgb6U0__i gQ2QGYGePcgufTr33EdT8OTdIf03IqB5XiciDdA-- Received: from [61.15.240.133] by web193504.mail.sg3.yahoo.com via HTTP; Sat, 26 Oct 2013 13:24:23 SGT X-Rocket-MIMEInfo: 002.001, CgpPbiAsIFBhdHJpY2sgRHVuZyA8cGF0cmlja19ka3RAeWFob28uY29tLmhrPiB3cm90ZToKIAoKCk9uIFNhdHVyZGF5LCBPY3RvYmVyIDI2LCAyMDEzIDE6MDQgQU0sIFN0ZXZlbiBIYXJ0bGFuZCA8a2lsbGluZ0BtdWx0aXBsYXkuY28udWs.IHdyb3RlOgogCi0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0gCkZyb206ICJQYXRyaWNrIER1bmciIDxwYXRyaWNrX2RrdEB5YWhvby5jb20uaGs.ClNlbnQ6IEZyaWRheSwgT2N0b2JlciAyNSwgMjAxMyA0OjUzIFBNClN1YmplY3Q6IFJlOiBbQURNSU5dIFpGUy0BMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <20131023162545.GA19794@pcjas.obspm.fr> <1382548465.32073.37624113.6AA8DB42@webmail.messagingengine.com> <52683855.6060609@gmx.net> <1382716422.13782.YahooMailNeo@web193504.mail.sg3.yahoo.com> <1382765013.22777.YahooMailNeo@web193502.mail.sg3.yahoo.com> Message-ID: <1382765063.35320.YahooMailNeo@web193504.mail.sg3.yahoo.com> Date: Sat, 26 Oct 2013 13:24:23 +0800 (SGT) From: Patrick Dung Subject: Re: [ADMIN] ZFS-FreeBSD + postgresql performance To: Steven Hartland , "Ivailo.TANUSHEFF@raiffeisen.bg" , Michael Schmiedgen In-Reply-To: <1382765013.22777.YahooMailNeo@web193502.mail.sg3.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" , "pgsql-admin@postgresql.org" , "freebsd-jail@freebsd.org" , "owner-freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Patrick Dung List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 05:24:27 -0000 =0A=0AOn , Patrick Dung wrote:=0A =0A=0A=0AOn Sa= turday, October 26, 2013 1:04 AM, Steven Hartland = wrote:=0A =0A----- Original Message ----- =0AFrom: "Patrick Dung" =0ASent: Friday, October 25, 2013 4:53 PM=0ASubject: Re: = [ADMIN] ZFS-FreeBSD + postgresql performance=0A=0A=0A>> I would also recomm= end to use 4K sector size=0A using=0A>=0A> gnop and zpool export/import.=0A= =0AThis will have no effect, the ashift for a pool is set at creation.=0A= =0A=A0 =A0 Regards=0A=A0 =A0 Steve=0A=0A=0A---------------------=0A=0ATo be= specific, I think the steps should be gnop create, create zfs pool, export= zfs, remove gnop and reimport zfs.=0AThis link have the deatils and steps:= =0AThe link is:=0A=0Ahttp://ivoras.sharanet.org/blog/tree/2011-01-01.freebs= d-on-4k-sector-drives.html=0A