From owner-freebsd-geom@FreeBSD.ORG Mon Nov 21 09:12:22 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60B46106564A for ; Mon, 21 Nov 2011 09:12:22 +0000 (UTC) (envelope-from acc@hexadecagram.org) Received: from mail.itproficiency.com (hexadecagram.org [166.70.126.65]) by mx1.freebsd.org (Postfix) with ESMTP id 343558FC14 for ; Mon, 21 Nov 2011 09:12:21 +0000 (UTC) Received: from mail (mail.hexadecagram.org [192.168.133.11]) by mail.itproficiency.com (Postfix) with ESMTP id EDE95B01845 for ; Mon, 21 Nov 2011 01:54:48 -0700 (MST) X-Virus-Scanned: amavisd-new at itproficiency.com Received: from mail.itproficiency.com ([192.168.133.11]) by mail (mail.itproficiency.com [192.168.133.11]) (amavisd-new, port 10024) with LMTP id vzJn6lHexInN for ; Mon, 21 Nov 2011 01:54:34 -0700 (MST) Received: from [192.168.133.166] (ares.aegaeum.hexadecagram.org [192.168.133.166]) by mail.itproficiency.com (Postfix) with ESMTPSA id 40366B01811 for ; Mon, 21 Nov 2011 01:54:34 -0700 (MST) Message-ID: <4ECA11CB.3070506@hexadecagram.org> Date: Mon, 21 Nov 2011 01:54:35 -0700 From: Anthony Chavez Organization: hexadecagram.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111108 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 09:12:22 -0000 The subject describes the configuration that I would like. # uname -a FreeBSD metis.hexadecagram.org 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Nov 10 16:1 3:27 MST 2011 root@metis.hexadecagram.org:/usr/obj/usr/src/sys/METIS i386 All 3 drives are 4096-sector Seagate FreeAgent GoFlex Desk 3TB USB 2.0 drives. I partition them as follows: # for d in /dev/da[345]; do # gpart create -s gpt /dev/da3 # gpart add -t freebsd-vinum $d # done I then glabel them (which would be nice to have as they are USB drives): # glabel extraid0 da3p1 # glabel extraid1 da4p1 # glabel extraid2 da5p1 Create the gvinum object: # gvinum create external.gvinum external.gvinum contains: drive a device /dev/label/extraid0 drive b device /dev/label/extraid1 drive c device /dev/label/extraid2 volume external plex org raid5 256k sd drive a sd drive b sd drive c I attempt to initialize geli and get an error: # dd if=/dev/random of=/boot/keys/external.key bs=64 count=1 # geli init -s 4096 -P -K /boot/keys/external.key /dev/gvinum/external geli: Cannot store metadata on /dev/gvinum/external: Invalid argument. I get a similar error trying to create a journal: # gjournal label -s 14G /dev/gvinum/external gjournal: Cannot clear metadata on /dev/gvinum/external: Invalid argument. It might be worth noting that if I geli init on /dev/da3p1, it succeeds, I am able to geli attach, and then use those devices to create the gvinum object. But it stands to reason that I want to at least have /dev/gvinum/external.journal, which I'm unable to create in any case. Any thoughts on this? -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org From owner-freebsd-geom@FreeBSD.ORG Mon Nov 21 09:55:13 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD1471065670 for ; Mon, 21 Nov 2011 09:55:13 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 892758FC16 for ; Mon, 21 Nov 2011 09:55:13 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RSQak-0001Ha-Vp for freebsd-geom@freebsd.org; Mon, 21 Nov 2011 10:55:10 +0100 Received: from hexadecagram.org ([166.70.126.65]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 21 Nov 2011 10:55:10 +0100 Received: from acc by hexadecagram.org with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 21 Nov 2011 10:55:10 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Anthony Chavez Date: Mon, 21 Nov 2011 02:43:10 -0700 Lines: 40 Message-ID: References: <4ECA11CB.3070506@hexadecagram.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: hexadecagram.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111108 Thunderbird/8.0 In-Reply-To: <4ECA11CB.3070506@hexadecagram.org> Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 09:55:13 -0000 On 11/21/2011 01:54 AM, Anthony Chavez wrote: > The subject describes the configuration that I would like. I neglected to mention the following in my initial post: # gvinum l 3 drives: D c State: up /dev/label/extraid2 A: 0/2861588 MB (0%) D b State: up /dev/label/extraid1 A: 0/2861588 MB (0%) D a State: up /dev/label/extraid0 A: 0/2861588 MB (0%) 1 volume: V external State: up Plexes: 1 Size: 5589 GB 1 plex: P external.p0 R5 State: up Subdisks: 3 Size: 5589 GB 3 subdisks: S external.p0.s2 State: up D: c Size: 2794 GB S external.p0.s1 State: up D: b Size: 2794 GB S external.p0.s0 State: up D: a Size: 2794 GB Also: # newfs -b 65536 -f 8192 -U /dev/gvinum/external newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument # newfs -U /dev/gvinum/external newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument # newfs /dev/gvinum/external newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org From owner-freebsd-geom@FreeBSD.ORG Mon Nov 21 11:07:04 2011 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1780E106566C for ; Mon, 21 Nov 2011 11:07:04 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F0A188FC16 for ; Mon, 21 Nov 2011 11:07:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id pALB73S8053561 for ; Mon, 21 Nov 2011 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id pALB73AE053559 for freebsd-geom@FreeBSD.org; Mon, 21 Nov 2011 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 21 Nov 2011 11:07:03 GMT Message-Id: <201111211107.pALB73AE053559@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-geom@FreeBSD.org X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 11:07:04 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/162147 geom [geom] mirror GPT: GPT rejected -- may not be recovera o kern/162010 geom [geli] panic: Provider's error should be set (error=0) o kern/161979 geom [geom] glabel doesn't update after newfs, and glabel s o kern/161752 geom [geom] glabel(8) doesn't get gpt label change o bin/161677 geom gpart(8) Probably bug in gptboot o kern/160562 geom [geom][patch] Allow to insert new component to geom_ra o kern/160409 geom [geli] failed to attach provider o kern/159595 geom [geom] [panic] panic on gmirror unload in vbox [regres p kern/158398 geom [headers] [patch] includes o kern/158197 geom [geom] geom_cache with size>1000 leads to panics o kern/157879 geom [libgeom] [regression] ABI change without version bump o kern/157863 geom [geli] kbdmux prevents geli passwords from being enter o kern/157739 geom [geom] GPT labels with geom_multipath o kern/157724 geom [geom] gpart(8) 'add' command must preserve gap for sc o kern/157723 geom [geom] GEOM should not process 'c' (raw) partitions fo o kern/157108 geom [gjournal] dumpon(8) fails on gjournal providers o kern/155994 geom [geom] Long "Suspend time" when reading large files fr o kern/154226 geom [geom] GEOM label does not change when you modify them o kern/150858 geom [geom] [geom_label] [patch] glabel(8) is not compatibl o kern/150626 geom [geom] [gjournal] gjournal(8) destroys label o kern/150555 geom [geom] gjournal unusable on GPT partitions o kern/150334 geom [geom] [udf] [patch] geom label does not support UDF o kern/149762 geom volume labels with rogue characters o bin/149215 geom [panic] [geom_part] gpart(8): Delete linux's slice via o kern/147667 geom [gmirror] Booting with one component of a gmirror, the o kern/145818 geom [geom] geom_stat_open showing cached information for n o kern/145042 geom [geom] System stops booting after printing message "GE f kern/144905 geom [geom][geom_part] panic in gpart_ctlreq when unpluggin o kern/143455 geom gstripe(8) in RELENG_8 (31st Jan 2010) broken o kern/142563 geom [geom] [hang] ioctl freeze in zpool o kern/141740 geom [geom] gjournal(8): g_journal_destroy concurrent error o kern/140352 geom [geom] gjournal + glabel not working o kern/135898 geom [geom] Severe filesystem corruption - large files or l o kern/134922 geom [gmirror] [panic] kernel panic when use fdisk on disk o kern/134113 geom [geli] Problem setting secondary GELI key o kern/133931 geom [geli] [request] intentionally wrong password to destr o bin/132845 geom [geom] [patch] ggated(8) does not close files opened a o bin/131415 geom [geli] keystrokes are unregulary sent to Geli when typ o kern/131353 geom [geom] gjournal(8) kernel lock o kern/129674 geom [geom] gjournal root did not mount on boot o kern/129645 geom gjournal(8): GEOM_JOURNAL causes system to fail to boo o kern/129245 geom [geom] gcache is more suitable for suffix based provid f kern/128276 geom [gmirror] machine lock up when gmirror module is used o kern/127420 geom [geom] [gjournal] [panic] Journal overflow on gmirrore o kern/124973 geom [gjournal] [patch] boot order affects geom_journal con o kern/124969 geom gvinum(8): gvinum raid5 plex does not detect missing s o kern/123962 geom [panic] [gjournal] gjournal (455Gb data, 8Gb journal), o kern/123122 geom [geom] GEOM / gjournal kernel lock o kern/122738 geom [geom] gmirror list "losts consumers" after gmirror de o kern/122067 geom [geom] [panic] Geom crashed during boot o kern/121364 geom [gmirror] Removing all providers create a "zombie" mir o kern/120091 geom [geom] [geli] [gjournal] geli does not prompt for pass o kern/115856 geom [geli] ZFS thought it was degraded when it should have o kern/115547 geom [geom] [patch] [request] let GEOM Eli get password fro o kern/114532 geom [geom] GEOM_MIRROR shows up in kldstat even if compile f kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113419 geom [geom] geom fox multipathing not failing back o kern/107707 geom [geom] [patch] [request] add new class geom_xbox360 to o kern/94632 geom [geom] Kernel output resets input while GELI asks for o kern/90582 geom [geom] [panic] Restore cause panic string (ffs_blkfree o bin/90093 geom fdisk(8) incapable of altering in-core geometry o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o bin/86388 geom [geom] [geom_part] periodic(8) daily should backup gpa o kern/84556 geom [geom] [panic] GBDE-encrypted swap causes panic at shu o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/79035 geom [vinum] gvinum unable to create a striped set of mirro o bin/78131 geom gbde(8) "destroy" not working. 68 problems total. From owner-freebsd-geom@FreeBSD.ORG Mon Nov 21 23:38:12 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D82E1065672 for ; Mon, 21 Nov 2011 23:38:12 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 916058FC13 for ; Mon, 21 Nov 2011 23:38:10 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RSdRB-00057T-El for freebsd-geom@freebsd.org; Tue, 22 Nov 2011 00:38:09 +0100 Received: from hexadecagram.org ([166.70.126.65]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 22 Nov 2011 00:38:09 +0100 Received: from acc by hexadecagram.org with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 22 Nov 2011 00:38:09 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Anthony Chavez Date: Mon, 21 Nov 2011 16:37:53 -0700 Lines: 37 Message-ID: References: <4ECA11CB.3070506@hexadecagram.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: hexadecagram.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111108 Thunderbird/8.0 In-Reply-To: Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 23:38:12 -0000 On 11/21/2011 02:43 AM, Anthony Chavez wrote: > # newfs -b 65536 -f 8192 -U /dev/gvinum/external > newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument > # newfs -U /dev/gvinum/external > newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument > # newfs /dev/gvinum/external > newfs: wtfs: 512 bytes at sector 11721065471: Invalid argument After posting, I used diskinfo on the gvinum object, which reported a 512b sectorsize, so I'm thinking this was the issue I was facing. There's also this, which has me thinking gvinum doesn't support 4k sectors yet: % grep -n sectorsize ~metis/sys/geom/vinum/geom_vinum_subr.c 1068: pp->sectorsize = 512; /* XXX */ Anyhow, the last time I had to prepare a large, diverse pool of storage was during the 5.x branch, so I was trying to work with what I was familiar with (vinum). And due to budget constraints, I'm stuck using i386 32-bit for the time being, so ZFS wasn't really an option. After some research, I've got a much better understanding of how the GEOM components fit together, and I must say that I'm quite pleased with what I've found! I have since decided to give sysutils/graid5 a spin, and so far it's doing exactly what I had hoped to accomplish with vinum (including using regular disk partitions as subdisks, which I intended to get to eventually). I have 2 questions regarding graid5: 1. Why hasn't it made its way to the base FreeBSD distribution yet? 2. Has anyone done any rigorous testing with it? -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org From owner-freebsd-geom@FreeBSD.ORG Tue Nov 22 06:25:38 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76B3B106566B for ; Tue, 22 Nov 2011 06:25:38 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 18DB58FC08 for ; Tue, 22 Nov 2011 06:25:38 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:d9a7:15b3:2d22:93e6]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 4CAAB4AC1C; Tue, 22 Nov 2011 10:25:36 +0400 (MSK) Date: Tue, 22 Nov 2011 10:25:33 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <835526911.20111122102533@serebryakov.spb.ru> To: Anthony Chavez In-Reply-To: References: <4ECA11CB.3070506@hexadecagram.org> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2011 06:25:38 -0000 Hello, Anthony. You wrote 22 =ED=EE=FF=E1=F0=FF 2011 =E3., 3:37:53: > I have 2 questions regarding graid5: > 1. Why hasn't it made its way to the base FreeBSD distribution yet? Because its code is ugly, almost non-understandable, raises some questions, and lays from style(9) a miles away. I'm rewriting it now, to support same functionality (and to be compatible with existing arrays), but as I didn't see any interest in it (yes, I'm using it by myself, but this your message is first "external" interest I could see for few years), I spend not much time on it. > 2. Has anyone done any rigorous testing with it? It depends on your meaning of word ``rigorous'' :) I'm using it for couple of years, I had several server failures in this time (panics), Ive chaged two failed drives for same size and after that migrate to larger HDDs (5x500Gb -> 5x2Tb) non-stop (almost, growfs need you to unmount FS), and I run some tests in VM after each change. But there is no formal test-suite (yet, but, again, see above about lack of "external" interest, which discourage me). One known semi-bug semi-feature. It cache writes aggressively (which allows it to have perfect linear write speed and avoid re-reads to compute checksum in most real-life cases), and in case of failures, this could lead to "unexpected softupdates inconsistences," as FFS doesn't send FLUSH commands at all, and module could not now, which writers should go synchronous, with minimal possible delay. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-geom@FreeBSD.ORG Tue Nov 22 06:29:00 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99061106566C for ; Tue, 22 Nov 2011 06:29:00 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6102D8FC0A for ; Tue, 22 Nov 2011 06:29:00 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:d9a7:15b3:2d22:93e6]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 24ED64AC1C; Tue, 22 Nov 2011 10:28:59 +0400 (MSK) Date: Tue, 22 Nov 2011 10:28:56 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <306619320.20111122102856@serebryakov.spb.ru> To: Anthony Chavez In-Reply-To: References: <4ECA11CB.3070506@hexadecagram.org> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2011 06:29:00 -0000 Hello, Anthony. You wrote 22 =ED=EE=FF=E1=F0=FF 2011 =E3., 3:37:53: > I have 2 questions regarding graid5: > 1. Why hasn't it made its way to the base FreeBSD distribution yet? > 2. Has anyone done any rigorous testing with it? BTW, if you could write list of good test-cases which should be in (VM-based) automated test-suite, it will be very helpful. I could easily miss some corner cases. But, please, be aware, that this test-suite should be implemented via standard UNIX API plus "disk disconnection," which is possible under VirtualBox 4.x, so tests should be formulated via newfs, file operations, fsck, etc. toolset, not via some internal kernel APIs. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-geom@FreeBSD.ORG Wed Nov 23 03:51:11 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5628C1065670 for ; Wed, 23 Nov 2011 03:51:11 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id D6C278FC13 for ; Wed, 23 Nov 2011 03:51:10 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RT3rZ-0006uN-66 for freebsd-geom@freebsd.org; Wed, 23 Nov 2011 04:51:09 +0100 Received: from hexadecagram.org ([166.70.126.65]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 23 Nov 2011 04:51:09 +0100 Received: from acc by hexadecagram.org with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 23 Nov 2011 04:51:09 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Anthony Chavez Date: Tue, 22 Nov 2011 20:50:54 -0700 Lines: 81 Message-ID: References: <4ECA11CB.3070506@hexadecagram.org> <835526911.20111122102533@serebryakov.spb.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: hexadecagram.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111108 Thunderbird/8.0 In-Reply-To: <835526911.20111122102533@serebryakov.spb.ru> Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2011 03:51:11 -0000 On 11/21/2011 11:25 PM, Lev Serebryakov wrote: > Hello, Anthony. Hi Lev, thank you for responding! > You wrote 22 ноября 2011 г., 3:37:53: > >> I have 2 questions regarding graid5: >> 1. Why hasn't it made its way to the base FreeBSD distribution yet? > Because its code is ugly, almost non-understandable, raises some > questions, and lays from style(9) a miles away. > I'm rewriting it now, to support same functionality (and to be > compatible with existing arrays), but as I didn't see any interest in > it (yes, I'm using it by myself, but this your message is first > "external" interest I could see for few years), I spend not much time > on it. It seems most admins prefer hardware RAID5, but having a good implementation in software for those under tight budgets seems like it would be very worthwhile. >> 2. Has anyone done any rigorous testing with it? > It depends on your meaning of word ``rigorous'' :) > I'm using it for couple of years, I had several server failures in > this time (panics), Ive chaged two failed drives for same size and > after that migrate to larger HDDs (5x500Gb -> 5x2Tb) non-stop > (almost, growfs need you to unmount FS), and I run some tests in VM > after each change. But there is no formal test-suite (yet, but, > again, see above about lack of "external" interest, which discourage > me). Were the panics that you mentioned directly caused by graid5? If so, under what conditions? Please don't get discouraged. I think graid5 could be very useful. About 5 years ago, I had a client that wanted to set up a fileserver but the lack of software RAID5 at the time turned them off of FreeBSD initially. I managed to convince them that hardware RAID5 was a smart investment, but they weren't terribly happy about having to spend the extra cash. You might want to base your opinion on the number of downloads the code receives rather than responses here. I, for one, have been using FreeBSD since version 1.0, but haven't posted to the mailing lists very often at all. ;-) > One known semi-bug semi-feature. It cache writes aggressively > (which allows it to have perfect linear write speed and avoid > re-reads to compute checksum in most real-life cases), and > in case of failures, this could lead to "unexpected softupdates > inconsistences," as FFS doesn't send FLUSH commands at all, and module > could not now, which writers should go synchronous, with minimal > possible delay. If that's the case, this would not affect me, as I'm using gjournal and mounting asynchronously rather than SoftUpdates. Any other features / bugs I should be aware of? The data that I'm intending to migrate is rather precious. I cannot afford to lose any of it, and will use graid3 if I must. >> I have 2 questions regarding graid5: >> 1. Why hasn't it made its way to the base FreeBSD distribution yet? >> 2. Has anyone done any rigorous testing with it? > BTW, if you could write list of good test-cases which should be in > (VM-based) automated test-suite, it will be very helpful. > I could easily miss some corner cases. > But, please, be aware, that this test-suite should be implemented > via standard UNIX API plus "disk disconnection," which is possible > under VirtualBox 4.x, so tests should be formulated via newfs, file > operations, fsck, etc. toolset, not via some internal kernel APIs. When time permits, I'll have a look at the code, but it may be a few months before I will be able to do so. I'm rather busy at the moment. When you mention "disk disconnection," do you mean *physical* disconnection (e.g., before a umount)? -- Anthony Chavez http://hexadecagram.org/ mailto:acc@hexadecagram.org xmpp:acc@hexadecagram.org From owner-freebsd-geom@FreeBSD.ORG Wed Nov 23 08:52:08 2011 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 889B2106564A for ; Wed, 23 Nov 2011 08:52:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 287948FC13 for ; Wed, 23 Nov 2011 08:52:08 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:5974:a369:b987:bc4d]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 4E9BF4AC1C; Wed, 23 Nov 2011 12:52:06 +0400 (MSK) Date: Wed, 23 Nov 2011 12:52:02 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <715091620.20111123125202@serebryakov.spb.ru> To: Anthony Chavez In-Reply-To: References: <4ECA11CB.3070506@hexadecagram.org> <835526911.20111122102533@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: gvinum + geli + gjournal X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2011 08:52:08 -0000 Hello, Anthony. You wrote 23 =ED=EE=FF=E1=F0=FF 2011 =E3., 7:50:54: > It seems most admins prefer hardware RAID5, but having a good > implementation in software for those under tight budgets seems like it > would be very worthwhile. And not so tight budgets, too, as "true" hardware RAID5 cards are very expensive, and work rather poor without battery module, which adds another ~$300... >>> 2. Has anyone done any rigorous testing with it? >> It depends on your meaning of word ``rigorous'' :) >> I'm using it for couple of years, I had several server failures in >> this time (panics), Ive chaged two failed drives for same size and >> after that migrate to larger HDDs (5x500Gb -> 5x2Tb) non-stop >> (almost, growfs need you to unmount FS), and I run some tests in VM >> after each change. But there is no formal test-suite (yet, but, >> again, see above about lack of "external" interest, which discourage >> me). > Were the panics that you mentioned directly caused by graid5? If so, > under what conditions? No, it was faulty memory, overheated CPU and other hardware problems. My main (and only one physical) server is only cheap desktop hardware, nothing fancy like ECC memory or redundant cooling syste= m. > You might want to base your opinion on the number of downloads the code > receives rather than responses here. I, for one, have been using > FreeBSD since version 1.0, but haven't posted to the mailing lists very > often at all. ;-) :) > Any other features / bugs I should be aware of? The data that I'm It could be misconfigured: long write queue / long write timeout could over-allocate memory. Default settings is very conservative, but if you try to tune it for best write performance, you could be misleaded by "maxmem" sysctl and belive, that it is hard top border for allocated memory. It is, unfortunately, not. I'll change this in future, but now you should be conservative in these settings. > intending to migrate is rather precious. I cannot afford to lose any of > it, and will use graid3 if I must. I should be sure, that port contains latest version with all latest changes, as I've fixed one potential bug (I never encounter it in the wild, though), some time ago. I'll check it tonight and notify you. >>> I have 2 questions regarding graid5: >>> 1. Why hasn't it made its way to the base FreeBSD distribution yet? >>> 2. Has anyone done any rigorous testing with it? >> BTW, if you could write list of good test-cases which should be in >> (VM-based) automated test-suite, it will be very helpful. >> I could easily miss some corner cases. >> But, please, be aware, that this test-suite should be implemented >> via standard UNIX API plus "disk disconnection," which is possible >> under VirtualBox 4.x, so tests should be formulated via newfs, file >> operations, fsck, etc. toolset, not via some internal kernel APIs. > When time permits, I'll have a look at the code, but it may be a few > months before I will be able to do so. I'm rather busy at the moment. > When you mention "disk disconnection," do you mean *physical* > disconnection (e.g., before a umount)? Yep. To simulate disk failure & disk hot-swap. VirtualBox allows to do so (from host OS, of course) and it helps me a lot in graid5 development. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-geom@FreeBSD.ORG Wed Nov 23 22:21:52 2011 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E1B7106564A; Wed, 23 Nov 2011 22:21:52 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id C23DC8FC0A; Wed, 23 Nov 2011 22:21:51 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:5974:a369:b987:bc4d]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id D5DF04AC1C; Thu, 24 Nov 2011 02:21:49 +0400 (MSK) Date: Thu, 24 Nov 2011 02:21:45 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <337241442.20111124022145@serebryakov.spb.ru> To: Kostik Belousov In-Reply-To: <20111123194444.GE50300@deviant.kiev.zoral.com.ua> References: <1957615267.20111123230026@serebryakov.spb.ru> <20111123194444.GE50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-geom@FreeBSD.org Subject: Re: Does UFS2 send BIO_FLUSH to GEOM when update metadata (with softupdates)? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2011 22:21:52 -0000 Hello, Kostik. You wrote 23 =ED=EE=FF=E1=F0=FF 2011 =E3., 23:44:44: > You are making wrong conclusions from the false assumptions. It seems to me, that FFS2/SU and SUJ are built on wrong assumption, that complete bwrite() without ASYNC flag really means landed data on physical device in any case. It is completely wrong, non-reliable, and prevents from building reliable AND high-performance storage on FreeBSD :( Or, may be, I understand code wrong. It is possible. See below. > The only requirement of the SU is that writes reported as done by disk > driver are indeed safely landed in the involatile storage. I've traced code and found next call chain. Please, correct me if I'm wrong. Softupdates writes all data via bwrite() or bawrite(). bawrite() is Ok, it is Async, it should givew any guarantees about immideat cache flush or ordering. bwrite() calls (in most cases, and in case of GEOM backing) ends in strategy, g_vfs_strategy() in case of GEOM (geom uses generic bufwrite(), which tweaks some flags, does some checks and send struct buf to bop_strategy, which is g_vfs_stratedy() in case of GEOM). g_vfs_strategy() sends request WITHOUT looking into ASYNC flag on "struct buf". We have BIO_ORDERED flags, but it is not used on this codepat= h! Maybe, cheap solution will be to set BIO_ORDERED on every struct bio, which is created for struct buf without ASYNC flag? Or it is too strict? Please note: now GEOM could not guarantee even ordering of SU writing requests now! Disk drivers, which sends such requests to hardware at least, could queue them too or leave them in drive's cache. It is COMPLETELY WRONG! With such disconnection between top-level logic (softupdates) and all driver stack (GEOM and disk drives) I surprised, that FFS2 could be repaired after panic at all! IMHO, it should be fixed ASAP and FFS2 should notify lower layers about writes, which is required to be ordered & landed before bwrite() returns! We have BIO_ORDERED flag, it could be used, but if it is too strict, we could add BIO_SYNC flag, too. ATA/SCSI subsystems already have proper support for BIO_ORDERED, and adding BIO_SYNC will not a big deal on low level, also, it could be easily added to g_vfs_strategy(), but I'm not sure that it will not hurt performance too much -- I'm not sure, that every buf write without ASYNC flag should be strict-SYNC. But I AM SURE, that SU/SU+J writes MUST BE DONE STRICT SYNC. P.S. I added geom@ into CC: as it seems to be UFS<->GEOM interaction problem. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-geom@FreeBSD.ORG Fri Nov 25 04:31:59 2011 Return-Path: Delivered-To: freebsd-geom@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF2E5106566B; Fri, 25 Nov 2011 04:31:59 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A754E8FC15; Fri, 25 Nov 2011 04:31:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id pAP4VxSj027837; Fri, 25 Nov 2011 04:31:59 GMT (envelope-from ae@freefall.freebsd.org) Received: (from ae@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id pAP4VxRt027831; Fri, 25 Nov 2011 04:31:59 GMT (envelope-from ae) Date: Fri, 25 Nov 2011 04:31:59 GMT Message-Id: <201111250431.pAP4VxRt027831@freefall.freebsd.org> To: bruce@cran.org.uk, ae@FreeBSD.org, freebsd-geom@FreeBSD.org From: ae@FreeBSD.org Cc: Subject: Re: kern/144905: [geom][geom_part] panic in gpart_ctlreq when unplugging card reader X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2011 04:31:59 -0000 Synopsis: [geom][geom_part] panic in gpart_ctlreq when unplugging card reader State-Changed-From-To: feedback->closed State-Changed-By: ae State-Changed-When: Fri Nov 25 04:31:18 UTC 2011 State-Changed-Why: Feedback timeout. I hope the problem is fixed now. http://www.freebsd.org/cgi/query-pr.cgi?pr=144905