From owner-freebsd-questions@FreeBSD.ORG Wed Apr 11 03:09:45 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2174716A403 for ; Wed, 11 Apr 2007 03:09:45 +0000 (UTC) (envelope-from solon@pyro.de) Received: from srv23.fsb.echelon.bnd.org (mail.pyro.de [83.137.99.96]) by mx1.freebsd.org (Postfix) with ESMTP id DB60113C465 for ; Wed, 11 Apr 2007 03:09:44 +0000 (UTC) (envelope-from solon@pyro.de) Received: from i577b7f7c.versanet.de ([87.123.127.124] helo=FLASH) by srv23.fsb.echelon.bnd.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1HbSgE-000C1d-C5 for freebsd-questions@freebsd.org; Wed, 11 Apr 2007 04:35:30 +0200 Date: Wed, 11 Apr 2007 04:36:19 +0200 From: Solon Luigi Lutz X-Mailer: SecureBat! Lite (v2.12.4) Personal Organization: PyroDesign Berlin X-Priority: 3 (Normal) Message-ID: <1387491461.20070411043619@pyro.de> To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Terabyte harddisks, GELI, AMD64, Samba and Zen... X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Solon Luigi Lutz List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Apr 2007 03:09:45 -0000 Hello freebsd-questions, I've setup a Freebsd 6.2 server with a Areca 1280 24-port SATA2-Raid controller and some 24 500gb harddisks. CPU is a AMD64 Sempron 3000+. --- FreeBSD rad.net 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Apr 8 12:39:46 UTC 2007 root@rad.net:/usr/obj/usr/src/sys/RADIUM amd64 --- --- Filesystem Size Used Avail Capacity Mounted on /dev/ad10s1a 496M 70M 386M 15% / devfs 1.0K 1.0K 0B 100% /dev /dev/ad10s1e 496M 18K 456M 0% /tmp /dev/ad10s1f 67G 3.9G 58G 6% /usr /dev/ad10s1d 1.9G 1.0G 761M 58% /var /dev/da0.eli 9.9T 4.0T 5.1T 44% /mnt --- The raid contains one 10 terabyte volume, encrypted with GELI --- Geom name: da0.eli EncryptionAlgorithm: AES-CBC KeyLength: 128 Crypto: software UsedKey: 0 Flags: NONE Providers: 1. Name: da0.eli Mediasize: 10999996342272 (10T) Sectorsize: 8192 Mode: r1w1e1 Consumers: 1. Name: da0 Mediasize: 10999996350464 (10T) Sectorsize: 512 Mode: r1w1e1 --- As I started to fill up the volume, I experienced random crashes - always while doing high-speed transfers over gigabit ethernet between this and another server. Mostly 1-2 per day - might be a hardware problem or not. Due to the slow CPU and software encryption, the maximum speed is somewhat limited (raw speed is somewhat around 600 MB/s): --- File './Bonnie.2117', size: 1048576000 Writing with putc()...done Rewriting...done Writing intelligently...done Reading with getc()...done Reading intelligently...done Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done... -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 1000 37922 31.5 48829 12.0 23827 5.0 30054 36.0 43666 6.0 1120.0 2.5 --- But I have not been able to get more than 17 MB/s when using Samba to transfer data - FTP maxes out around 27 MB/s. I also tried that on i386 32-bit and found it to be 8 MB/s and 17 MB/s - not good, but nothing to worry. What made me feel really uncomfortable was the fact, that just some minutes ago some 3000 1GB files suddenly disappeared while working in a directory. They where gone, but the filesystem did not report some additional 3TB to be free and after unmounting and remounting the filesystem the files were back where the belonged... This just happened some minutes later again, now with only 2500 files dis- and -reappearing again. Questions until now: 1. 10TB as a single volume, too big for good? (fsck time: 30 min with softupdates) 2. GELI unstable on big disks and/or AMD64? 3. Why is Samba so slow? 4. Does the crypto-framwork gain speed advantages from dual-core CPUs? 5. Will the GPT-stuff change over the next releases in a way I need to do DUMP/RESTORE? Thanks for your patience... Best regards, Solon Lutz