Date: Wed, 11 Apr 2007 04:36:19 +0200 From: Solon Luigi Lutz <solon@pyro.de> To: freebsd-questions@freebsd.org Subject: Terabyte harddisks, GELI, AMD64, Samba and Zen... Message-ID: <1387491461.20070411043619@pyro.de>
next in thread | raw e-mail | index | archive | help
Hello freebsd-questions,
I've setup a Freebsd 6.2 server with a Areca 1280 24-port SATA2-Raid
controller and some 24 500gb harddisks. CPU is a AMD64 Sempron 3000+.
---
FreeBSD rad.net 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Apr 8 12:39:46 UTC 2007
root@rad.net:/usr/obj/usr/src/sys/RADIUM amd64
---
---
Filesystem Size Used Avail Capacity Mounted on
/dev/ad10s1a 496M 70M 386M 15% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad10s1e 496M 18K 456M 0% /tmp
/dev/ad10s1f 67G 3.9G 58G 6% /usr
/dev/ad10s1d 1.9G 1.0G 761M 58% /var
/dev/da0.eli 9.9T 4.0T 5.1T 44% /mnt
---
The raid contains one 10 terabyte volume, encrypted with GELI
---
Geom name: da0.eli
EncryptionAlgorithm: AES-CBC
KeyLength: 128
Crypto: software
UsedKey: 0
Flags: NONE
Providers:
1. Name: da0.eli
Mediasize: 10999996342272 (10T)
Sectorsize: 8192
Mode: r1w1e1
Consumers:
1. Name: da0
Mediasize: 10999996350464 (10T)
Sectorsize: 512
Mode: r1w1e1
---
As I started to fill up the volume, I experienced random crashes -
always while doing high-speed transfers over gigabit ethernet between
this and another server. Mostly 1-2 per day - might be a hardware
problem or not.
Due to the slow CPU and software encryption, the maximum speed is somewhat
limited (raw speed is somewhat around 600 MB/s):
---
File './Bonnie.2117', size: 1048576000
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
1000 37922 31.5 48829 12.0 23827 5.0 30054 36.0 43666 6.0 1120.0 2.5
---
But I have not been able to get more than 17 MB/s when using Samba to
transfer data - FTP maxes out around 27 MB/s. I also tried that on
i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but
nothing to worry.
What made me feel really uncomfortable was the fact, that just some
minutes ago some 3000 1GB files suddenly disappeared while working
in a directory. They where gone, but the filesystem did not report
some additional 3TB to be free and after unmounting and remounting the
filesystem the files were back where the belonged...
This just happened some minutes later again, now with only 2500 files
dis- and -reappearing again.
Questions until now:
1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates)
2. GELI unstable on big disks and/or AMD64?
3. Why is Samba so slow?
4. Does the crypto-framwork gain speed advantages from dual-core CPUs?
5. Will the GPT-stuff change over the next releases in a way I need to
do DUMP/RESTORE?
Thanks for your patience...
Best regards,
Solon Lutz
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1387491461.20070411043619>