From owner-freebsd-geom@FreeBSD.ORG Sun Dec 14 02:07:29 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CE691065672 for ; Sun, 14 Dec 2008 02:07:29 +0000 (UTC) (envelope-from mikej@paymentallianceintl.com) Received: from mx2.confluenttech.com (mx2.confluentasp.com [216.26.153.14]) by mx1.freebsd.org (Postfix) with ESMTP id 0A5F68FC12 for ; Sun, 14 Dec 2008 02:07:28 +0000 (UTC) (envelope-from mikej@paymentallianceintl.com) Received: from calvin.pai.local (calvin.pai.local [10.0.6.33]) by mx2.confluenttech.com (8.14.1/8.13.8) with ESMTP id mBE27KVa090226; Sat, 13 Dec 2008 21:07:20 -0500 (EST) (envelope-from mikej@paymentallianceintl.com) Authentication-Results: mx2.confluenttech.com from=mikej@paymentallianceintl.com; sender-id=neutral; spf=neutral X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4325 Content-Class: urn:content-classes:message MIME-Version: 1.0 Date: Sat, 13 Dec 2008 21:07:15 -0500 Message-ID: In-Reply-To: <917871cf0812130559r6d423688q57287dd765d6edf4@mail.gmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Encrypting raid5 volume with geli thread-index: AcldKyHy2JmlnXAqRrOoAUbv19RoOwAYie1g References: <20081212155023.GA82667@keira.kiwi-computer.com> <917871cf0812130559r6d423688q57287dd765d6edf4@mail.gmail.com> From: "Michael Jung" Importance: normal To: "Ulf Lilleengen" Priority: normal Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-geom@freebsd.org Subject: RE: Encrypting raid5 volume with geli X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2008 02:07:29 -0000 From: Ulf Lilleengen [mailto:ulf.lilleengen@gmail.com] Sent: Saturday, December 13, 2008 8:59 AM To: Michael Jung Cc: freebsd-geom@freebsd.org Subject: Re: Encrypting raid5 volume with geli On Fri, Dec 12, 2008 at 5:00 PM, Michael Jung wrote: FreeBSD charon.confluentasp.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Thu Sep 4 12:06:08 EDT 2008 In the interest of this thread I tried to duplicate the problem. I created: 10 drives: D d9 State: up /dev/da9 A: 0/17366 MB (0%) D d8 State: up /dev/da8 A: 0/17366 MB (0%) D d7 State: up /dev/da7 A: 0/17366 MB (0%) D d6 State: up /dev/da6 A: 0/17366 MB (0%) D d5 State: up /dev/da5 A: 0/17366 MB (0%) D d4 State: up /dev/da4 A: 0/17366 MB (0%) D d3 State: up /dev/da3 A: 0/17366 MB (0%) D d2 State: up /dev/da2 A: 0/17366 MB (0%) D d1 State: up /dev/da1 A: 0/17366 MB (0%) D d0 State: up /dev/da0 A: 0/17366 MB (0%) 1 volume: V test State: up Plexes: 1 Size: 152 GB 1 plex: P test.p0 R5 State: up Subdisks: 10 Size: 152 GB 10 subdisks: S test.p0.s9 State: up D: d9 Size: 16 GB S test.p0.s8 State: up D: d8 Size: 16 GB S test.p0.s7 State: up D: d7 Size: 16 GB S test.p0.s6 State: up D: d6 Size: 16 GB S test.p0.s5 State: up D: d5 Size: 16 GB S test.p0.s4 State: up D: d4 Size: 16 GB S test.p0.s3 State: up D: d3 Size: 16 GB S test.p0.s2 State: up D: d2 Size: 16 GB S test.p0.s1 State: up D: d1 Size: 16 GB S test.p0.s0 State: up D: d0 Size: 16 GB Which I can newfs and mount (root@charon) /etc# mount /dev/gvinum/test /mnt (root@charon) /etc# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1a 357G 119G 209G 36% / devfs 1.0K 1.0K 0B 100% /dev 172.0.255.28:/data/unix 1.3T 643G 559G 54% /nas1 /dev/gvinum/test 148G 4.0K 136G 0% /mnt But with /dev/gvinum/test unmounted if I try: (root@charon) /etc# geli init -P -K /root/test.key /dev/gvinum/test geli: Cannot store metadata on /dev/gvinum/test: Operation not permitted. (root@charon) /etc# My random file was created like dd if=/dev/random of=/root/test.key bs=64 count=1 I use GELI at home with no trouble, although not with a gvinum volume. Hello, When I tried this myself, I also got the EPERM error in return. I though this was very strange. I went through the gvinum code today, and put debugging prints everywhere, but everything looked fine, and it was only raid5 volumes that failed. Then I saw that the EPERM error came from the underlying providers of geom (more specifially from the read requests to the parity stripes etc), so I was starting to suspect that it was not a gvinum error. But still, I was able to write/read from the disks from outside of gvinum! Then, I discovered in geom userland code that it opens the disk where metadata should be written in write only mode. Then I discovered the reason: gvinum tries to write to the stripe in question, but has to read back the parity data from one of the other stripes. But, they are opened O_WRONLY, so the request fails. I tried opening the device as O_RDWR, and everything is find. Phew :) You can bet I was frustrated I hope to commit the attached change in the near future. -- Ulf Lilleengen I+++++++++++++++++++++++++++++++++ 7.1-PRERELEASE #0: Sat Dec 13 15:09:38 EST 2008 I just cvsup and applied your patch, now: (root@charon) /etc# geli init -P -K /root/test.key /dev/gvinum/test (root@charon) /etc# geli attach -p -k /root/test.key /dev/gvinum/test (root@charon) /etc# newfs /dev/gvinum/test.eli /dev/gvinum/test.eli: 121564.2MB (248963480 sectors) block size 16384, fragment size 2048 using 662 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. super-block backups (for fsck -b #) at: 160, 376512, 752864, 1129216, 1505568, 1881920, 2258272,........... (root@charon) /etc# mount /dev/gvinum/test.eli /mnt (root@charon) /etc# df -h /mnt Filesystem Size Used Avail Capacity Mounted on /dev/gvinum/test.eli 115G 4.0K 106G 0% /mnt (root@charon) /etc# I exercise it some but patch looks good! --mikej CONFIDENTIALITY NOTE: This message is intended only for the use of the individual or entity to whom it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify us by telephone at (502) 212-4001 or notify us at PAI , Dept. 99, 11857 Commonwealth Drive, Louisville, KY 40299. Thank you.