Date: Fri, 18 Jan 2008 10:11:15 +0100 From: =?ISO-8859-1?Q?Johan_Str=F6m?= <johan@stromnet.se> To: Ulrich Spoerlein <uspoerlein@gmail.com> Cc: emj@emj.se, freebsd-stable@freebsd.org Subject: Re: Backup solution suggestions [ggated] Message-ID: <73ED6B7D-49C5-4C69-8BE2-27F0BDAA246E@stromnet.se> In-Reply-To: <7ad7ddd90801170030l790810b5r5bb156e3cda286b1@mail.gmail.com> References: <E6BCC509-6CC8-44F1-98C2-416920A52218@stromnet.se> <39FB5CF3-F2F4-401B-9D6D-7796608152E5@ish.com.au> <4FF9842D-ADC9-4A99-9DC4-E0FE1CC9CDCF@stromnet.se> <20080116222729.GB1529@roadrunner.spoerlein.net> <134BD86C-19CF-42A7-9190-FAD3BB564A06@stromnet.se> <7ad7ddd90801170030l790810b5r5bb156e3cda286b1@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 17, 2008, at 09:30 , Ulrich Spoerlein wrote:
> On Jan 17, 2008 1:31 AM, Johan Str=F6m <johan@stromnet.se> wrote:
>>> Export the disk on the backup server with ggated. Bind it on the
>>> client
>>> with ggatec. Slap a GELI or GBDE encryption on top of it and then
>>> put a
>>> ZFS on top of it.
>>>
>>> You can mount/import this "remote" ZFS at will and do your zfs
>>> send/receive on your local box. Nothing ever leaves your box
>>> unencrypted.
>>
>> Now that is a cool solution! That actually sounds like something =20
>> doable.
>> I tried it out some at home between a 6.2 box (client) and 7.0 box
>> (server), hosting the system in a ZFS "sparse volume" with a
>> predefined size, exported that via ggated and connected ggatec on the
>> client box. I then did some experimentation with just newfs, and it
>> worked great!
>> The only downside with this would be that the size is fixed. So I
>> played around a bit with setting the volsize property in ZFS and it
>> seemd to work just fine. zfs list reported the new, bigger, size.
>> Restarted ggatec and did a growfs, and then remounted.. Yay bigger
>> disk :)
>> Then I went on do do some geli test, geli'ed /dev/ggate0 and
>> newfs'ed, mounted and played around a bit. All fine.. Now came the
>> problem, i unmounetd it, expanded the zfs volume a bit more,
>> restarted ggatec and tried to attach it using geli again (note, I
>> have no idea if this is supposed to work at all, I'm just testing.
>> Havent read such things anywhere). Now I got Invalid argument.
>> Im not realy sure about how GEOM works, but if I recall correct it
>> uses the last sectors of the disk? If I moved X bytes of data from
>> old end of disk to new end of disk, would that make GELI work? If I
>> can get that to work, then this would be a kickass solution (all
>> encryption stuff works great, I don't have to allocate all space
>> immediatly, I can expand it later without destroying data and
>> starting from scratch etc).
>
> I'm pretty certain that GELI cannot handle variable sized disks. But
> you could add GVIRSTOR into the mix. But I'd just allocate the
> necessary space and be done with it. Adding yet another layer is
> asking for trouble, imho.
Okay.
>
>> Some other questions, more related to ggated/c. Is this stable? Good
>> working? how does it handle failure situations? Anyone using it for
>> production systems?
>
> =46rom my personal experience (which is rather limited): No, barely, =20=
> bad, hell no.
>
> There were/are some open PRs about ggate. I had troubles with
> gmirror+ggate in that it would deadlock every other hour on SMP
> systems (try removing option PREEMPTION if that bug hits you).
Your no,barely, bad hell no seems to fit pretty good.. I did some =20
testing during the night with the above (non-production) setup.
What I did was doing some rsyncing over the night:
while true ; do
echo "`date` Clearing vmail" >> logfile
rm -rf vmail
echo "`date` Starting rsync" >> logfile
rsync -vr /usr/var/vmail . |tee -a logfile
echo "`date` Rsync finished " >> logfile
done
I started this at ~02.0. The results? A freshly rebooted 6.2 (6.2-=20
RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Fri Jul 27 15:47:50 UTC 2007) =20
box in the morning..
Looking at the messages:
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844480512, length=3D4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844484608, length=3D4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844488704, length=3D4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844492800, length=3D4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844517376, length=3D4096)]
... more of the same...
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844480512, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844640256, length=3D32768)]
..more of the same...
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844988416, length=3D4096)]
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844484608, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844488704, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844492800, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844517376, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8844992512, length=3D4096)]
...more of the same...
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D65536, length=3D4096)]
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844521472, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844496896, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844500992, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8844505088, length=3D4096)]error =3D 5
..and more of those...
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8845176832, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D5636096, length=3D4096)]error =3D 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D65536, length=3D4096)]error =3D 5
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D5636096, length=3D4096)]
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8127250432, length=3D16384)]
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed =20
(error=3D5). ggate0.eli[WRITE(offset=3D8480587776, length=3D16384)]
...more of those..
Jan 18 05:33:43 phomca kernel: g_vfs_done():ggate0.eli[WRITE=20
(offset=3D8834072576, length=3D16384)]error =3D 5
Jan 18 05:35:42 phomca syslogd: kernel boot file is /boot/kernel/kernel
What I dont have is a coredump, judging from dmesg -a savecore wasnt =20
even run.. running it now, 5 hours later, didnt find any cores.
The other end (7.0 server) wasnt affected at all.
Not realy sure what it had been doing, because looking at my =20
bandwidth graphs from the switch, nothing was done at all.. It didnt =20
even go through one iteration of rsync... ~7.5k files/directorys =20
seems to have been transfered, then the log doesnt say more. But =20
according to the BW graph, after ~03.00 no traffic was sent at all...
Some known bug with 6.2?
>> Yes this is for backup only so minor glitches
>> might be acceptable for me, but I'd rather know about those =20
>> beforehand.
>
> Give it a shot, if your systems stay up and stable, good. If the link
> breaks from time to time, I think ZFS should be able to recover most
> of it. Since it is your backup, I'd try to break it in interessting
> ways first, to get a feel for how robust it is.
>
> Uli
>
So, crashing the backuped box is not a minor glitch. Thus this doesnt =20=
seem to be useable AT ALL :/ Too bad..
--
Johan=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?73ED6B7D-49C5-4C69-8BE2-27F0BDAA246E>