From owner-freebsd-geom@FreeBSD.ORG Mon Jul 23 11:08:19 2007 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 80E2916A479 for ; Mon, 23 Jul 2007 11:08:19 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 39CC413C480 for ; Mon, 23 Jul 2007 11:08:19 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l6NB8JTr045308 for ; Mon, 23 Jul 2007 11:08:19 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l6NB8Hwv045304 for freebsd-geom@FreeBSD.org; Mon, 23 Jul 2007 11:08:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 23 Jul 2007 11:08:17 GMT Message-Id: <200707231108.l6NB8Hwv045304@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2007 11:08:19 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/73177 geom kldload geom_* causes panic due to memory exhaustion o kern/76538 geom [gbde] nfs-write on gbde partition stalls and continue o kern/83464 geom [geom] [patch] Unhandled malloc failures within libgeo o kern/84556 geom [geom] GBDE-encrypted swap causes panic at shutdown o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o kern/89102 geom [geom_vfs] [panic] panic when forced unmount FS from u o bin/90093 geom fdisk(8) incapable of altering in-core geometry o kern/90582 geom [geom_mirror] [panic] Restore cause panic string (ffs_ o kern/98034 geom [geom] dereference of NULL pointer in acd_geom_detach o kern/104389 geom [geom] [patch] sys/geom/geom_dump.c doesn't encode XML o kern/113419 geom [geom] geom fox multipathing not failing back o misc/113543 geom [geom] [patch] geom(8) utilities don't work inside the o kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad 13 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/78131 geom gbde "destroy" not working. o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/94632 geom [geom] Kernel output resets input while GELI asks for f kern/105390 geom [geli] filesystem on a md backed by sparse file with s o kern/107707 geom [geom] [patch] add new class geom_xbox360 to slice up p bin/110705 geom gmirror control utility does not exit with correct exi o kern/113790 geom [patch] enable the Camellia block cipher on GEOM ELI ( o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113885 geom [geom] [patch] improved gmirror balance algorithm o kern/114532 geom GEOM_MIRROR shows up in kldstat even if compiled in th 10 problems total. From owner-freebsd-geom@FreeBSD.ORG Wed Jul 25 02:52:38 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A33716A419; Wed, 25 Jul 2007 02:52:38 +0000 (UTC) (envelope-from mi@aldan.algebra.com) Received: from aldan.algebra.com (aldan.algebra.com [216.254.65.224]) by mx1.freebsd.org (Postfix) with ESMTP id 3C9FC13C442; Wed, 25 Jul 2007 02:52:38 +0000 (UTC) (envelope-from mi@aldan.algebra.com) Received: from aldan.algebra.com (localhost [127.0.0.1]) by aldan.algebra.com (8.14.1/8.14.1) with ESMTP id l6P2Q3KS001247 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Jul 2007 22:26:03 -0400 (EDT) (envelope-from mi@aldan.algebra.com) Received: (from mi@localhost) by aldan.algebra.com (8.14.1/8.14.1/Submit) id l6P2Q3UC001246; Tue, 24 Jul 2007 22:26:03 -0400 (EDT) (envelope-from mi) From: "Mikhail T." Message-Id: <200707250226.l6P2Q3UC001246@aldan.algebra.com> To: stable@FreeBSD.org Date: Tue, 24 Jul 2007 22:26:03 -0400 (EDT) X-Face: %UW#n0|w>ydeGt/b@1-.UFP=K^~-:0f#O:D7w hJ5G_<5143Bb3kOIs9XpX+"V+~$adGP:J|SLieM31VIhqXeLBli" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2007 02:52:38 -0000 Hello! I was working remotely, when suddenly the server dropped off the 'net. Upon reboot it was stuck in "unexpected softupdates inconsistency" (I did not have fsck_y_enable set). Here is the panic: Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 1; apic id = 01 instruction pointer = 0x8:0xffffffff803be0ed stack pointer = 0x10:0xffffffffb1ae0b10 frame pointer = 0x10:0xffffff005fa23380 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3 (g_up) trap number = 9 panic: general protection fault cpuid = 1 Uptime: 29m52s Dumping 2047 MB (2 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 2047MB (524016 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (kgdb) #0 doadump () at pcpu.h:172 #1 0x0000000000000004 in ?? () #2 0xffffffff802b2d87 in boot (howto=260) at /var/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff802b3421 in panic (fmt=0xffffff007b9e7000 "°Ö {") at /var/src/sys/kern/kern_shutdown.c:565 #4 0xffffffff804140ff in trap_fatal (frame=0xffffff007b9e7000, eva=18446742976272062128) at /var/src/sys/amd64/amd64/trap.c:668 #5 0xffffffff80414602 in trap (frame= {tf_rdi = -1098100966016, tf_rsi = -1097437646848, tf_rdx = 164931039068161, tf_rcx = 0, tf_r8 = -6241704602446071034, tf_r9 = -2141060032, tf_rax = 0, tf_rbx = -1099511589376, tf_rbp = -1097907162240, tf_r10 = -2141060032, tf_r11 = -2144905952, tf_r12 = -1616167584, tf_r13 = -2144915520, tf_r14 = 0, tf_r15 = -1097907162240, tf_trapno = 9, tf_addr = 0, tf_flags = -1097437646848, tf_err = 0, tf_rip = -2143559443, tf_cs = 8, tf_rflags = 66050, tf_rsp = -1313993952, tf_ss = 16}) at /var/src/sys/amd64/amd64/trap.c:470 #6 0xffffffff803fb35b in calltrap () at /var/src/sys/amd64/amd64/exception.S:168 #7 0xffffffff803be0ed in softdep_disk_write_complete (bp=0xffffffff9fab3d60) at /var/src/sys/ufs/ffs/ffs_softdep.c:4755 #8 0xffffffff8030edba in bufdone (bp=0xffffffff9fab3d60) at buf.h:440 #9 0xffffffff802755a5 in g_vfs_done (bip=0x0) at /var/src/sys/geom/geom_vfs.c:87 #10 0xffffffff80272d3d in g_io_schedule_up (tp=0xffffff005414fd80) at /var/src/sys/geom/geom_io.c:490 #11 0xffffffff8027304c in g_up_procbody () at /var/src/sys/geom/geom_kern.c:95 #12 0xffffffff80297e17 in fork_exit ( callout=0xffffffff80272fc0 , arg=0x0, frame=0xffffffffb1ae0c50) at /var/src/sys/kern/kern_fork.c:821 #13 0xffffffff803fb6be in fork_trampoline () at /var/src/sys/amd64/amd64/exception.S:394 #14 0x0000000000000000 in ?? () #15 0x0000000000000000 in ?? () #16 0x0000000000000001 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0x0000000000000000 in ?? () #46 0x0000000000824000 in ?? () #47 0xffffff007ba0d6b0 in ?? () #48 0x0000000000000104 in ?? () #49 0x0000000000000000 in ?? () #50 0xffffff007ba0d6b0 in ?? () #51 0xffffff007b9e7be0 in ?? () #52 0xffffffffb1ae0818 in ?? () #53 0xffffff007b9e7000 in ?? () #54 0xffffffff802ca546 in sched_switch (td=0x0, newtd=0x0, flags=0) at /var/src/sys/kern/sched_4bsd.c:973 #55 0x0000000000000000 in ?? () #56 0x0000000000000000 in ?? () #57 0x0000000000000000 in ?? () #58 0x0000000000000000 in ?? () #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x0000000000000000 in ?? () #71 0x0000000000000000 in ?? () #72 0x0000000000000000 in ?? () #73 0x0000000000000000 in ?? () #74 0x0000000000000000 in ?? () #75 0x0000000000000000 in ?? () #76 0x0000000000000000 in ?? () #77 0x0000000000000000 in ?? () #78 0x0000000000000000 in ?? () #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () #117 0x0000000000000000 in ?? () #118 0x0000000000000000 in ?? () #119 0x0000000000000000 in ?? () #120 0x0000000000000000 in ?? () #121 0x0000000000000000 in ?? () #122 0x0000000000000000 in ?? () #123 0x0000000000000000 in ?? () #124 0x0000000000000000 in ?? () #125 0x0000000000000000 in ?? () #126 0x0000000000000000 in ?? () (kgdb) #6 0xffffffff803fb35b in calltrap () at /var/src/sys/amd64/amd64/exception.S:168 168 call trap Current language: auto; currently asm (kgdb) From owner-freebsd-geom@FreeBSD.ORG Fri Jul 27 11:06:08 2007 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 220D716A421; Fri, 27 Jul 2007 11:06:08 +0000 (UTC) (envelope-from lulf@FreeBSD.org) Received: from fri.itea.ntnu.no (fri.itea.ntnu.no [129.241.7.60]) by mx1.freebsd.org (Postfix) with ESMTP id B101D13C4CC; Fri, 27 Jul 2007 11:06:07 +0000 (UTC) (envelope-from lulf@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) by fri.itea.ntnu.no (Postfix) with ESMTP id 97F8FB1AA; Fri, 27 Jul 2007 12:39:35 +0200 (CEST) Received: from gaupe.stud.ntnu.no (gaupe.stud.ntnu.no [129.241.56.184]) by fri.itea.ntnu.no (Postfix) with ESMTP; Fri, 27 Jul 2007 12:39:35 +0200 (CEST) Received: by gaupe.stud.ntnu.no (Postfix, from userid 2312) id BEB3DD003A; Fri, 27 Jul 2007 12:39:44 +0200 (CEST) Date: Fri, 27 Jul 2007 12:39:44 +0200 From: Ulf Lilleengen To: freebsd-geom@freebsd.org Message-ID: <20070727103944.GA12318@stud.ntnu.no> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="envbJBWh7q8WU6mo" Content-Disposition: inline User-Agent: Mutt/1.5.9i X-Content-Scanned: with sophos and spamassassin at mailgw.ntnu.no. Cc: pjd@FreeBSD.org, phk@FreeBSD.org Subject: [PATCH] geom_bsd tasting holds topology_lock when doing I/O X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2007 11:06:08 -0000 --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, I started having problems with gvinum hanging itself up because it was waiting for the topology lock. At the same time geom_bsd was trying to taste the new gvinum provider that I created. This resulted in a deadlock where gvinum waited for the lock and g_bsd_taste waited for gvinum to finish a read BIO. The patch also asserts that the lock is not held in g_bsd_taste -- Ulf Lilleengen --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="geom_bsd_toplock.diff" Index: sys/geom/geom_bsd.c =================================================================== RCS file: /home/ncvs/src/sys/geom/geom_bsd.c,v retrieving revision 1.78 diff -u -r1.78 geom_bsd.c --- sys/geom/geom_bsd.c 24 May 2006 11:02:32 -0000 1.78 +++ sys/geom/geom_bsd.c 27 Jul 2007 10:34:49 -0000 @@ -201,6 +201,8 @@ struct disklabel *dl; off_t secoff; + g_topology_assert_not(); + /* * We need to read entire aligned sectors, and we assume that the * disklabel does not span sectors, so one sector is enough. @@ -502,6 +504,7 @@ * using a simple "break". This improves code clarity without * ending up in deep nesting and without using goto or come from. */ + g_topology_unlock(); do { /* * If the provider is an MBR we will only auto attach @@ -571,8 +574,11 @@ * Process the found disklabel, and modify our "slice" * instance to match it, if possible. */ + g_topology_lock(); error = g_bsd_modify(gp, ms->label); + g_topology_unlock(); } while (0); + g_topology_lock(); /* Success or failure, we can close our provider now. */ g_access(cp, -1, 0, 0); --envbJBWh7q8WU6mo-- From owner-freebsd-geom@FreeBSD.ORG Fri Jul 27 19:57:43 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33DB316A418 for ; Fri, 27 Jul 2007 19:57:43 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from carrick.bishnet.net (unknown [IPv6:2001:618:400::54ea:1138]) by mx1.freebsd.org (Postfix) with ESMTP id 231CF13C461 for ; Fri, 27 Jul 2007 19:57:42 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from cpc1-warw1-0-0-cust384.sol2.cable.ntl.com ([86.20.169.129] helo=magellan.dom.bishnet.net ident=mailnull) by carrick.bishnet.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IEVwM-000F5M-CQ for geom@FreeBSD.org; Fri, 27 Jul 2007 20:57:34 +0100 Received: from deimos.dom.bishnet.net ([192.168.3.100] helo=deimos) by magellan.dom.bishnet.net with esmtp (Exim 4.67 (FreeBSD)) (envelope-from ) id 1IEVwM-000NCQ-1T for geom@FreeBSD.org; Fri, 27 Jul 2007 20:57:34 +0100 From: "Dominic Bishop" To: Date: Fri, 27 Jul 2007 20:57:46 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3138 Thread-Index: AcfQiGZ7gwkqpxmaT8eFNyHaKkLssw== X-Bishnet-MailScanner-Information: Contact postmaster@bishnet.net X-Bishnet-MailScanner-VirusCheck: Found to be clean X-Bishnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=0.189, required 5, autolearn=not spam, AWL 1.01, BAYES_05 -1.11, FORGED_RCVD_HELO 0.14, HOT_NASTY 0.16, SPF_PASS -0.00) X-Bishnet-MailScanner-From: dom@bishnet.net Message-Id: <20070727195742.231CF13C461@mx1.freebsd.org> Cc: Subject: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2007 19:57:43 -0000 I've just been testing out GELI performance on an underlying RAID using a 3ware 9550SXU-12 running RELENG_6 as of yesterday and seem to be hitting a performance bottleneck, but I can't see where it is coming from. Testing with an unencrypted 100GB GPT partition (/dev/da0p1) gives me around 200-250MB/s read and write speeds to give an idea of the capability of the disk device itself. Using GELI with a default 128bit AES key seems to limit at ~50MB/s , changing the sector size all the way upto 128KB makes no difference whatsoever to the performance. If I use the threads sysctl in loader.conf and drop the geli threads to 1 thread only (instead of the usual 3 it spawns on this system) the performance still does not change at all. Monitoring during writes with systat confirms that it really is spawning 1 or 3 threads correctly in these cases. Here is a uname -a from the machine FreeBSD 004 6.2-STABLE FreeBSD 6.2-STABLE #2: Fri Jul 27 20:10:05 CEST 2007 dom@004:/u1/obj/u1/src/sys/004 amd64 Kernel is a copy of GENERIC with GELI option added Encrypted partition created using : geli init -s 65536 /dev/da0p1 Simple write test done with: dd if=/dev/zero of=/dev/da0p1.eli bs=1m count=10000 (same as I did on the unencyrpted, a full test with bonnie++ shows similar speeds) Systat output whilst writing, showing 3 threads: /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average |||| /0 /10 /20 /30 /40 /50 /60 /70 /80 /90 /100 root idle: cpu3 XXXXXXXXX root idle: cpu1 XXXXXXXX XXXXXXXX root idle: cpu0 XXXXXXX root idle: cpu2 XXXXXX root g_eli[2] d XXX root g_eli[0] d XXX root g_eli[1] d X root g_up root dd Output from vmstat -w 5 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 da0 in sy cs us sy id 0 1 0 38124 3924428 208 0 1 0 9052 0 0 0 1758 451 6354 1 15 84 0 1 0 38124 3924428 0 0 0 0 13642 0 0 411 2613 128 9483 0 22 78 0 1 0 38124 3924428 0 0 0 0 13649 0 0 411 2614 130 9483 0 22 78 0 1 0 38124 3924428 0 0 0 0 13642 0 0 411 2612 128 9477 0 22 78 0 1 0 38124 3924428 0 0 0 0 13642 0 0 411 2611 128 9474 0 23 77 Output from iostat -x 5 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ad4 2.2 0.7 31.6 8.1 0 3.4 1 da0 0.2 287.8 2.3 36841.5 0 0.4 10 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ad4 0.0 0.0 0.0 0.0 0 0.0 0 da0 0.0 411.1 0.0 52622.1 0 0.4 15 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ad4 0.0 0.0 0.0 0.0 0 0.0 0 da0 0.0 411.1 0.0 52616.2 0 0.4 15 pass0 0.0 0.0 0.0 0.0 0 0.0 0 Looking at these results myself I cannot see where the bottleneck is, I would assume since changing the sector size or the geli threads doesn't affect performance that there is some other single threaded part limiting it but I don't know enough about how it works to say what. CPU in the machine is a pair of these: CPU: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz (1603.92-MHz K8-class CPU) I've also come across some other strange issues with some other machines which have identical arrays but only a pair of 32bit 3.0Ghz xeons in them (Also using releng_6 as of yesterday, just i386 not amd64). On those geli will launch a single thread by default (cores-1 seems to be the default) however I cannot force it to launch 2 by using the sysctl, although on the 4 core machine I can successfully use it to launch 4. It would be nice to be able to use both cores on the 32bit machines for geli but given the results I've shown here I'm not sure it would gain me much at the moment. Another problem I've found is that if I use a sector size for GELI > 8192 bytes then I'm unable to newfs the encrypted partition afterwards, it fails immediately with this error: newfs /dev/da0p1.eli increasing block size from 16384 to fragment size (65536) /dev/da0p1.eli: 62499.9MB (127999872 sectors) block size 65536, fragment size 65536 using 5 cylinder groups of 14514.56MB, 232233 blks, 58112 inodes. newfs: can't read old UFS1 superblock: read error from block device: Invalid argument The underlying device is readable/writeable however as dd can read/write to it without any errors. If anyone has any suggestions/thoughts on any of these points it would be much appreciated, these machines will be performing backups over 1Gbit LAN so more speed than I can currently get would be preferable. Regards, Dominic Bishop From owner-freebsd-geom@FreeBSD.ORG Fri Jul 27 20:42:44 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 331AD16A419 for ; Fri, 27 Jul 2007 20:42:44 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from auriate.fluffles.net (cust.95.160.adsl.cistron.nl [195.64.95.160]) by mx1.freebsd.org (Postfix) with ESMTP id CFCA313C45A for ; Fri, 27 Jul 2007 20:42:43 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from 195-241-125-45.dsl.ip.tiscali.nl ([195.241.125.45] helo=[10.0.0.18]) by auriate.fluffles.net with esmtpa (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IEW6d-000BqF-W8; Fri, 27 Jul 2007 22:08:12 +0200 Message-ID: <46AA50B4.9080901@fluffles.net> Date: Fri, 27 Jul 2007 22:08:20 +0200 From: Fluffles User-Agent: Thunderbird 2.0.0.5 (X11/20070716) MIME-Version: 1.0 To: Dominic Bishop References: <20070727195742.231CF13C461@mx1.freebsd.org> In-Reply-To: <20070727195742.231CF13C461@mx1.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: geom@FreeBSD.org Subject: Re: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2007 20:42:44 -0000 Dominic Bishop wrote: > Another problem I've found is that if I use a sector size for GELI > 8192 > bytes then I'm unable to newfs the encrypted partition afterwards, it fails > immediately with this error: > > newfs /dev/da0p1.eli > increasing block size from 16384 to fragment size (65536) > /dev/da0p1.eli: 62499.9MB (127999872 sectors) block size 65536, fragment > size 65536 > using 5 cylinder groups of 14514.56MB, 232233 blks, 58112 inodes. > newfs: can't read old UFS1 superblock: read error from block device: Invalid > argument > > The underlying device is readable/writeable however as dd can read/write to > it without any errors. > Newfs will use 512-byte sectors by default so if you're using an 8KB sector size you need: newfs -S 8192 /dev/da0p1.eli To view the sector size you can also use diskinfo -v . You can never write or read below the sector size (thats right, not even the filesystem can do that!) I can't help you with your GELI question. I'm assuming you run a quadcore 1.6GHz Xeon CPU? It should give you more performance than 50MB/s especially with 4 threads. On my Athlon 64 X2 3800+ (2x 2.0GHz 512KB) i got around 100 i believe. Do know that AMD processors perform relatively better in AES than Intel's line of CPUs, but with a quadcore i would have expected at least 160MB/s of thoughput. Maybe you should try geli on the raw device? Like: geli onetime da0 dd if=/dev/da0.eli of=/dev/null bs=1m count=1000 dd if=/dev/zero of=/dev/da0.eli bs=1m count=1000 It goes without saying that you should unmount and make sure no geom layer is using the da0 device. Good luck! - Veronica From owner-freebsd-geom@FreeBSD.ORG Fri Jul 27 21:00:32 2007 Return-Path: Delivered-To: geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BB8516A418 for ; Fri, 27 Jul 2007 21:00:32 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from carrick.bishnet.net (unknown [IPv6:2001:618:400::54ea:1138]) by mx1.freebsd.org (Postfix) with ESMTP id 0140413C457 for ; Fri, 27 Jul 2007 21:00:32 +0000 (UTC) (envelope-from dom@bishnet.net) Received: from cpc1-warw1-0-0-cust384.sol2.cable.ntl.com ([86.20.169.129] helo=magellan.dom.bishnet.net ident=mailnull) by carrick.bishnet.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IEWv9-000H9K-Ve for geom@FreeBSD.org; Fri, 27 Jul 2007 22:00:24 +0100 Received: from deimos.dom.bishnet.net ([192.168.3.100] helo=deimos) by magellan.dom.bishnet.net with esmtp (Exim 4.67 (FreeBSD)) (envelope-from ) id 1IEWv9-000NKy-MJ for geom@FreeBSD.org; Fri, 27 Jul 2007 22:00:23 +0100 From: "Dominic Bishop" To: Date: Fri, 27 Jul 2007 22:00:35 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3138 Thread-Index: AcfQieXtn3Z6gHUFR1uBrUNF7dA5kgABmfIQ In-Reply-To: <46AA50B4.9080901@fluffles.net> X-Bishnet-MailScanner-Information: Contact postmaster@bishnet.net X-Bishnet-MailScanner-VirusCheck: Found to be clean X-Bishnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-0.693, required 5, autolearn=not spam, AWL 1.77, BAYES_00 -2.60, FORGED_RCVD_HELO 0.14, SPF_PASS -0.00) X-Bishnet-MailScanner-From: dom@bishnet.net Message-Id: <20070727210032.0140413C457@mx1.freebsd.org> Cc: Subject: RE: Increasing GELI performance X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2007 21:00:32 -0000 > Fluffles wrote: > > Dominic Bishop wrote: > > Another problem I've found is that if I use a sector size > for GELI > > > 8192 bytes then I'm unable to newfs the encrypted partition > > afterwards, it fails immediately with this error: > > > > newfs /dev/da0p1.eli > > increasing block size from 16384 to fragment size (65536) > > /dev/da0p1.eli: 62499.9MB (127999872 sectors) block size 65536, > > fragment size 65536 > > using 5 cylinder groups of 14514.56MB, 232233 blks, > 58112 inodes. > > newfs: can't read old UFS1 superblock: read error from > block device: > > Invalid argument > > > > The underlying device is readable/writeable however as dd can > > read/write to it without any errors. > > > > Newfs will use 512-byte sectors by default so if you're using > an 8KB sector size you need: > > newfs -S 8192 /dev/da0p1.eli > > To view the sector size you can also use diskinfo -v > . You can never write or read below the sector size > (thats right, not even the filesystem can do that!) > > I can't help you with your GELI question. I'm assuming you > run a quadcore 1.6GHz Xeon CPU? It should give you more > performance than 50MB/s especially with 4 threads. On my > Athlon 64 X2 3800+ (2x 2.0GHz > 512KB) i got around 100 i believe. Do know that AMD > processors perform relatively better in AES than Intel's line > of CPUs, but with a quadcore i would have expected at least > 160MB/s of thoughput. Maybe you should try geli on the raw > device? Like: > > geli onetime da0 > dd if=/dev/da0.eli of=/dev/null bs=1m count=1000 dd > if=/dev/zero of=/dev/da0.eli bs=1m count=1000 > > It goes without saying that you should unmount and make sure > no geom layer is using the da0 device. > > Good luck! > > - Veronica > The -S option didn't help, I tried using -S 65536 on a geli partition created with -s 65536 and got exactly the same error. newfs -S 65536 /dev/da0p1.eli increasing block size from 16384 to fragment size (65536) /dev/da0p1.eli: 62499.9MB (127999872 sectors) block size 65536, fragment size 65536 using 5 cylinder groups of 14514.56MB, 232233 blks, 58112 inodes. newfs: can't read old UFS1 superblock: read error from block device: Invalid argument It seems to be completely ignoring it and going with 512 byte sectors anyway, judging from the size versus number of sectors listed. I just tried your suggestion of geli on the raw device and it is no better at all: dd if=/dev/da0.eli of=/dev/null bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 29.739186 secs (35259069 bytes/sec) dd if=/dev/zero of=/dev/da0.eli bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 23.501061 secs (44618241 bytes/sec) Using top -S with 1s refresh to list the geli processes whilst doing this it seems only one of them is doing anything at any given time, the others are sitting in a state of "geli:w", I assume that is a truncation of something, maybe geli:wait at a guess. For comparison, here are the same tests done on the unencrypted raw device: dd if=/dev/da0 of=/dev/null bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 5.802704 secs (180704717 bytes/sec) dd if=/dev/zero of=/dev/da0 bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 4.026869 secs (260394859 bytes/sec) Dominic. From owner-freebsd-geom@FreeBSD.ORG Sat Jul 28 04:54:30 2007 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9387216A468 for ; Sat, 28 Jul 2007 04:54:30 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.175]) by mx1.freebsd.org (Postfix) with ESMTP id 32CF513C459 for ; Sat, 28 Jul 2007 04:54:29 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: by ug-out-1314.google.com with SMTP id o4so888048uge for ; Fri, 27 Jul 2007 21:54:29 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=JCiNZVi4Xw1tMkxdC8ZgPX/875HUmdfA/bQH2gE35A54Fstiz/OWbIGcAjlhbPrgAmtkmd+mFUiNNpy3Em+LU6SkbyDS20QpPoqVwoiEvoMtaKJ4hzErUzQrQPgTWJ4+eFh09V/nmKHv7a3OAXvBvzSeeC1OMQPeJyIaEXRdp/Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=r8BF8cyLxVG8YMf6U9AZolad7xdM9LNiFC5lKgYmLJX1sHTYc0OUt7bb4qgAnoPM5Gm2S+x7ztk3RntwUohcCHQkiKy0XDJmNQwr9lkO8p8opocRMJlYgY20S1PEfNg7zotdl2nX+Efh//0z7OKOb58fuWaaWrmvRuUH4KRHc34= Received: by 10.78.186.9 with SMTP id j9mr926681huf.1185596804868; Fri, 27 Jul 2007 21:26:44 -0700 (PDT) Received: by 10.78.25.16 with HTTP; Fri, 27 Jul 2007 21:26:44 -0700 (PDT) Message-ID: <520894aa0707272126g42b88d41q95477f2d1dd3689b@mail.gmail.com> Date: Sat, 28 Jul 2007 01:26:44 -0300 From: "Fernan Aguero" To: freebsd-geom@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: boot from second disk (gmirror device) ... problems X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2007 04:54:30 -0000 Hi I'm using geom mirror on several FreeBSD boxes. It's been some time since I've set these up, but I remember being bitten by this problem several times: the problem? booting from the second disk after setting it as the first member of a geom mirror pair. I've always followed more or less Ralf's guide http://people.freebsd.org/~rse/mirror/ (I say more or less because I usually mirror the second slice of my disks) But I always end up getting trouble booting from the second disk. (need to have to have physical access to the box). In the previous incantation of Ralf's guide, he suggested booting from the second disk like this: echo "1:ad(1,a)/boot/loader" >/boot.config Now, he replaced this in the new version of the guide by this: echo $d2 | sed -e 's;^\([^0-9]*\)\([0-9][0-9]*\)$;1:\1(\2,a)/boot/loader;' \ >/boot.config where $d2 is the second disk (ad2 in my case), and this is the resulting /boot.config: 1:ad(2,a)/boot/loader After setting up /etc/fstab and /boot/loader.conf in my /dev/mirror/gm0a partition rebooting brings me to the boot loader prompt because: 1:ad(2,a)/boot/loader is not a valid label or /boot/loader does not exist I've tried several combinations, blindly trying to spot the correct one: 1:ad(1,a)/boot/loader 1:ad(0,a)/boot/loader To no avail. However, 0:ad(0,a) or 0:ad(1,a) allowed me to boot from my ad0 (first) disk (i.e. not the gmirror device) What's the right incantation to boot from the second disk? I've already read boot(8) many times, and can't seem to figure out why 1:ad(1,a) doesn't work bios_drive: 1 (second drive) interface: ad unit: 1 (second drive) slice: a Thanks in advance for any tip or suggestion, Fernan PS: my disks are ad0 (at ata0-master) and ad2 (at ata1-master). ad2s1 (single slice) is the first member of the gm0 gmirror gm0a is the partition containing the bootable (active) partition -- fernan From owner-freebsd-geom@FreeBSD.ORG Sat Jul 28 11:44:37 2007 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32DD716A41B for ; Sat, 28 Jul 2007 11:44:37 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from auriate.fluffles.net (cust.95.160.adsl.cistron.nl [195.64.95.160]) by mx1.freebsd.org (Postfix) with ESMTP id F3CD613C480 for ; Sat, 28 Jul 2007 11:44:36 +0000 (UTC) (envelope-from etc@fluffles.net) Received: from 195-241-125-45.dsl.ip.tiscali.nl ([195.241.125.45] helo=[10.0.0.18]) by auriate.fluffles.net with esmtpa (Exim 4.66 (FreeBSD)) (envelope-from ) id 1IEkib-0003iE-DA; Sat, 28 Jul 2007 13:44:21 +0200 Message-ID: <46AB2C1D.9090207@fluffles.net> Date: Sat, 28 Jul 2007 13:44:29 +0200 From: Fluffles User-Agent: Thunderbird 2.0.0.5 (X11/20070716) MIME-Version: 1.0 To: Fernan Aguero References: <520894aa0707272126g42b88d41q95477f2d1dd3689b@mail.gmail.com> In-Reply-To: <520894aa0707272126g42b88d41q95477f2d1dd3689b@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-geom@freebsd.org Subject: Re: boot from second disk (gmirror device) ... problems X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2007 11:44:37 -0000 Fernan Aguero wrote: > After setting up /etc/fstab and /boot/loader.conf in my > /dev/mirror/gm0a partition > rebooting brings me to the boot loader prompt because: > 1:ad(2,a)/boot/loader > is not a valid label or /boot/loader does not exist > On a boot partition you need to have a slice, so you probably should use: 1:ad(2,1,a)/boot/loader this will try to boot from ad2s1a, a likely name for a boot partition on the secondary master PATA disk. You may also press "?" at the prompt and get a list of devices. Also, for your /etc/fstab you may wish to use a label instead. Read up on man glabel on how to set these up. The idea is that, by giving a disk a unique label, you can setup your /etc/fstab without worrying on what controller/cable/connector te disk is on. So doesn't matter if its ad2 or ad8, it will show up as /dev/label/mydisk and you can setup your /etc/fstab accordingly. Good luck! - Veronica