Date: Sat, 13 Dec 2008 21:07:15 -0500 From: "Michael Jung" <mikej@paymentallianceintl.com> To: "Ulf Lilleengen" <ulf.lilleengen@gmail.com> Cc: freebsd-geom@freebsd.org Subject: RE: Encrypting raid5 volume with geli Message-ID: <ADC733B130BF1D4A82795B6B3A2654E20128D2B5@exchange.paymentallianceintl.com> In-Reply-To: <917871cf0812130559r6d423688q57287dd765d6edf4@mail.gmail.com> References: <20081212155023.GA82667@keira.kiwi-computer.com> <ADC733B130BF1D4A82795B6B3A2654E2777381@exchange.paymentallianceintl.com> <917871cf0812130559r6d423688q57287dd765d6edf4@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
From: Ulf Lilleengen [mailto:ulf.lilleengen@gmail.com] Sent: Saturday, December 13, 2008 8:59 AM To: Michael Jung Cc: freebsd-geom@freebsd.org Subject: Re: Encrypting raid5 volume with geli On Fri, Dec 12, 2008 at 5:00 PM, Michael Jung <mikej@paymentallianceintl.com> wrote: FreeBSD charon.confluentasp.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Thu Sep 4 12:06:08 EDT 2008 In the interest of this thread I tried to duplicate the problem. I created: 10 drives: D d9 State: up /dev/da9 A: 0/17366 MB (0%) D d8 State: up /dev/da8 A: 0/17366 MB (0%) D d7 State: up /dev/da7 A: 0/17366 MB (0%) D d6 State: up /dev/da6 A: 0/17366 MB (0%) D d5 State: up /dev/da5 A: 0/17366 MB (0%) D d4 State: up /dev/da4 A: 0/17366 MB (0%) D d3 State: up /dev/da3 A: 0/17366 MB (0%) D d2 State: up /dev/da2 A: 0/17366 MB (0%) D d1 State: up /dev/da1 A: 0/17366 MB (0%) D d0 State: up /dev/da0 A: 0/17366 MB (0%) 1 volume: V test State: up Plexes: 1 Size: 152 GB 1 plex: P test.p0 R5 State: up Subdisks: 10 Size: 152 GB 10 subdisks: S test.p0.s9 State: up D: d9 Size: 16 GB S test.p0.s8 State: up D: d8 Size: 16 GB S test.p0.s7 State: up D: d7 Size: 16 GB S test.p0.s6 State: up D: d6 Size: 16 GB S test.p0.s5 State: up D: d5 Size: 16 GB S test.p0.s4 State: up D: d4 Size: 16 GB S test.p0.s3 State: up D: d3 Size: 16 GB S test.p0.s2 State: up D: d2 Size: 16 GB S test.p0.s1 State: up D: d1 Size: 16 GB S test.p0.s0 State: up D: d0 Size: 16 GB Which I can newfs and mount (root@charon) /etc# mount /dev/gvinum/test /mnt (root@charon) /etc# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1a 357G 119G 209G 36% / devfs 1.0K 1.0K 0B 100% /dev 172.0.255.28:/data/unix 1.3T 643G 559G 54% /nas1 /dev/gvinum/test 148G 4.0K 136G 0% /mnt But with /dev/gvinum/test unmounted if I try: (root@charon) /etc# geli init -P -K /root/test.key /dev/gvinum/test geli: Cannot store metadata on /dev/gvinum/test: Operation not permitted. (root@charon) /etc# My random file was created like dd if=/dev/random of=/root/test.key bs=64 count=1 I use GELI at home with no trouble, although not with a gvinum volume. Hello, When I tried this myself, I also got the EPERM error in return. I though this was very strange. I went through the gvinum code today, and put debugging prints everywhere, but everything looked fine, and it was only raid5 volumes that failed. Then I saw that the EPERM error came from the underlying providers of geom (more specifially from the read requests to the parity stripes etc), so I was starting to suspect that it was not a gvinum error. But still, I was able to write/read from the disks from outside of gvinum! Then, I discovered in geom userland code that it opens the disk where metadata should be written in write only mode. Then I discovered the reason: gvinum tries to write to the stripe in question, but has to read back the parity data from one of the other stripes. But, they are opened O_WRONLY, so the request fails. I tried opening the device as O_RDWR, and everything is find. Phew :) You can bet I was frustrated I hope to commit the attached change in the near future. -- Ulf Lilleengen I+++++++++++++++++++++++++++++++++ 7.1-PRERELEASE #0: Sat Dec 13 15:09:38 EST 2008 I just cvsup and applied your patch, now: (root@charon) /etc# geli init -P -K /root/test.key /dev/gvinum/test (root@charon) /etc# geli attach -p -k /root/test.key /dev/gvinum/test (root@charon) /etc# newfs /dev/gvinum/test.eli /dev/gvinum/test.eli: 121564.2MB (248963480 sectors) block size 16384, fragment size 2048 using 662 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. super-block backups (for fsck -b #) at: 160, 376512, 752864, 1129216, 1505568, 1881920, 2258272,........... (root@charon) /etc# mount /dev/gvinum/test.eli /mnt (root@charon) /etc# df -h /mnt Filesystem Size Used Avail Capacity Mounted on /dev/gvinum/test.eli 115G 4.0K 106G 0% /mnt (root@charon) /etc# I exercise it some but patch looks good! --mikej CONFIDENTIALITY NOTE: This message is intended only for the use of the individual or entity to whom it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify us by telephone at (502) 212-4001 or notify us at PAI , Dept. 99, 11857 Commonwealth Drive, Louisville, KY 40299. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ADC733B130BF1D4A82795B6B3A2654E20128D2B5>