From owner-freebsd-geom@FreeBSD.ORG Mon Mar 26 11:08:11 2007 Return-Path: X-Original-To: freebsd-geom@FreeBSD.org Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 71E2D16A40A for ; Mon, 26 Mar 2007 11:08:11 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 5FF7B13C48A for ; Mon, 26 Mar 2007 11:08:11 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l2QB8Bn9049270 for ; Mon, 26 Mar 2007 11:08:11 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l2QB8ADD049266 for freebsd-geom@FreeBSD.org; Mon, 26 Mar 2007 11:08:10 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Mar 2007 11:08:10 GMT Message-Id: <200703261108.l2QB8ADD049266@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: linimon set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 11:08:11 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/73177 geom kldload geom_* causes panic due to memory exhaustion o kern/76538 geom [gbde] nfs-write on gbde partition stalls and continue o kern/83464 geom [geom] [patch] Unhandled malloc failures within libgeo o kern/84556 geom [geom] GBDE-encrypted swap causes panic at shutdown o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o kern/89102 geom [geom_vfs] [panic] panic when forced unmount FS from u o bin/90093 geom fdisk(8) incapable of altering in-core geometry o kern/90582 geom [geom_mirror] [panic] Restore cause panic string (ffs_ o kern/98034 geom [geom] dereference of NULL pointer in acd_geom_detach o kern/104389 geom [geom] [patch] sys/geom/geom_dump.c doesn't encode XML 10 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/78131 geom gbde "destroy" not working. o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/94632 geom [geom] Kernel output resets input while GELI asks for f kern/105390 geom [geli] filesystem on a md backed by sparse file with s o kern/107707 geom [geom] [patch] add new class geom_xbox360 to slice up p bin/110705 geom gmirror control utility does not exit with correct exi 6 problems total. From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 10:40:09 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AEBD616A400 for ; Wed, 28 Mar 2007 10:40:09 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 6B88A13C484 for ; Wed, 28 Mar 2007 10:40:09 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HWVZT-00076O-He for freebsd-geom@freebsd.org; Wed, 28 Mar 2007 12:40:03 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Mar 2007 12:40:03 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Mar 2007 12:40:03 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Ivan Voras Date: Wed, 28 Mar 2007 12:39:54 +0200 Lines: 40 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig10E2A397132E254BEE6C5522" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Geom_label and multiple devices X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 10:40:09 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig10E2A397132E254BEE6C5522 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable I've just thought of something - the situation where geom_label detects=20 multiple "same" labels is actually very similar to that of=20 geom_multipath. Obviously this doesn't apply 100% to UFS and other=20 filesystems' labels (except if there's a way to identify different file=20 systems by their superblock, for example timestamp of creation?), but in = case of "manual" labels, if the unique ID matches and the label string=20 matches, it's probably the same device. Currently, geom_label discards "additional" devices with the same label, = and I remember there was a discussion about if that should be changed=20 (though I don't remember the conclusions). It would be useful if=20 geom_label would work the same as geom_multipath in this case (i.e.=20 failover to "next" device, add new devices with same label+id to the=20 pool), or even if geom_multipath were merged in geom_label. Comments? --------------enig10E2A397132E254BEE6C5522 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGCkX6ldnAQVacBcgRAvRRAKDF8H65uFimMaGauGmdQDBqicLp/wCgjVAF liP8Uinfy4lvPUG9Nma+dOM= =yZsz -----END PGP SIGNATURE----- --------------enig10E2A397132E254BEE6C5522-- From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 11:49:51 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 66E7616A400 for ; Wed, 28 Mar 2007 11:49:51 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta01.mail.t-online.hu (mta01.mail.t-online.hu [195.228.240.50]) by mx1.freebsd.org (Postfix) with ESMTP id 28C4313C4C2 for ; Wed, 28 Mar 2007 11:49:51 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 249E533C0BE for ; Wed, 28 Mar 2007 13:19:34 +0200 (CEST) Message-ID: <003401c7712a$f71ebb60$6502a8c0@peteruj> From: =?iso-8859-2?B?U3phYvMgUOl0ZXI=?= To: Date: Wed, 28 Mar 2007 13:19:34 +0200 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 11:49:51 -0000 Hi! I have a raid3 setup with 5 consumers. it seems everything is all right, = but riad is very slow imho, but i don't have any reference. Now i copy from raid3 to a single hdd 75GB data. Nothing else run, = except ssh, and ftp. Before the copy my load is 0.1, after i star the = copy the load is 2.92 or higher, and the machine is very slow. I think = this is not normal, or is it? My config is: AMD Barton 2500+ with 1GB memory. I stop the copy, load is 0.05, and start again, after a few seconds load = is 2.02. raid is used to download with torrentflux, but the downloads always = stops because the high load. Matyee From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 11:50:00 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 11D7116A403 for ; Wed, 28 Mar 2007 11:50:00 +0000 (UTC) (envelope-from peter@jobservice.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id C7BE913C448 for ; Wed, 28 Mar 2007 11:49:59 +0000 (UTC) (envelope-from peter@jobservice.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id C1A7133C1FB for ; Wed, 28 Mar 2007 13:17:09 +0200 (CEST) Message-ID: <001e01c7712a$a0766fb0$6502a8c0@peteruj> From: =?iso-8859-2?B?U3phYvMgUOl0ZXI=?= To: Date: Wed, 28 Mar 2007 13:17:09 +0200 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 11:50:00 -0000 Hi! I have a raid3 setup with 5 consumers. it seems everything is all right, = but riad is very slow imho, but i don't have any reference. Now i copy from raid3 to a single hdd 75GB data. Nothing else run, = except ssh, and ftp. Before the copy my load is 0.1, after i star the = copy the load is 2.92 or higher, and the machine is very slow. I think = this is not normal, or is it? My config is: AMD Barton 2500+ with 1GB memory. I stop the copy, load is 0.05, and start again, after a few seconds load = is 2.02. raid is used to download with torrentflux, but the downloads always = stops because the high load. Matyee From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:02:50 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F2E1416A413 for ; Wed, 28 Mar 2007 12:02:49 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 80BEE13C4C5 for ; Wed, 28 Mar 2007 12:02:49 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HWWrM-0005eC-Fk for freebsd-geom@freebsd.org; Wed, 28 Mar 2007 14:02:36 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Mar 2007 14:02:36 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Mar 2007 14:02:36 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Ivan Voras Date: Wed, 28 Mar 2007 14:02:16 +0200 Lines: 47 Message-ID: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig94DA44A3D5E7C984628C02BB" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <003401c7712a$f71ebb60$6502a8c0@peteruj> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:02:50 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig94DA44A3D5E7C984628C02BB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Szab=C3=B3 P=C3=A9ter wrote: > I have a raid3 setup with 5 consumers. it seems everything is all right= , but riad is very slow imho, but i don't have any reference. You need to give more accurate information. What are the "consumers" you = mention? Maybe you mean the disks/components of the RAID? > Now i copy from raid3 to a single hdd 75GB data. Nothing else run, exce= pt ssh, and ftp. Before the copy my load is 0.1, after i star the copy th= e load is 2.92 or higher, and the machine is very slow. I think this is n= ot normal, or is it? Copy how? By "cp" from shell? > My config is: > AMD Barton 2500+ with 1GB memory. >=20 > I stop the copy, load is 0.05, and start again, after a few seconds loa= d is 2.02. Due to the way raid3 works, it's by definition more CPU-intensive than=20 most other RAID levels, but it shouldn't be even remotely slow as that.=20 Also, since your load is > 2, what are the other CPU intensive tasks? --------------enig94DA44A3D5E7C984628C02BB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGCllIldnAQVacBcgRAsoSAKCREdqq6y5w3iXlHbyz/P1JofRpQACfUyQL tWYzgdzFK/imqQNDmY9ZhzI= =oLcw -----END PGP SIGNATURE----- --------------enig94DA44A3D5E7C984628C02BB-- From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:25:25 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 681AC16A407 for ; Wed, 28 Mar 2007 12:25:25 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id E725813C469 for ; Wed, 28 Mar 2007 12:25:24 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 2370E33C1BD for ; Wed, 28 Mar 2007 14:25:23 +0200 (CEST) Message-ID: <005c01c77134$28e0fce0$6502a8c0@peteruj> From: =?UTF-8?B?U3phYsOzIFDDqXRlcg==?= To: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> Date: Wed, 28 Mar 2007 14:25:23 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:25:25 -0000 Yes, consumers are the disks. # graid3 list Geom name: nmivol State: DEGRADED Components: 5 Flags: NONE GenID: 0 SyncID: 73 ID: 2396346217 Zone64kFailed: 0 Zone64kRequested: 13987 Zone16kFailed: 11189 Zone16kRequested: 28827400 Zone4kFailed: 71105 Zone4kRequested: 29207293 Providers: 1. Name: raid3/nmivol Mediasize: 638906845184 (595G) Sectorsize: 2048 Mode: r3w3e4 Consumers: 1. Name: label/nmivol2 Mediasize: 159726711808 (149G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Flags: HARDCODED GenID: 0 SyncID: 73 Number: 1 Type: DATA 2. Name: label/nmivol1 Mediasize: 159726711808 (149G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Flags: HARDCODED GenID: 0 SyncID: 73 Number: 0 Type: DATA 3. Name: label/nmivol3 Mediasize: 159726711808 (149G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Flags: HARDCODED GenID: 0 SyncID: 73 Number: 2 Type: DATA 4. Name: label/nmivol4 Mediasize: 159726711808 (149G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Flags: HARDCODED GenID: 0 SyncID: 73 Number: 3 Type: DATA #systat -vmstat 4 users Load 1.62 1.75 1.82 Mar 28 14:24 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 66488 6968 150740 11396 31172 count All 1017076 10160975701764 16256 pages 1 zfod Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 4066 total 1 60 9820 68312106 4237 752 1 195968 wire 1002 0: clk 79088 act 128 8: rtc 89.2%Sys 3.0%Intr 3.3%User 0.0%Nice 4.5%Idl 714076 inact 10 10: fxp | | | | | | | | | | 29068 cache 1780 11: ata =============================================+>> 2104 free 573 14: ata 12 daefr 573 15: ata Namei Name-cache Dir-cache prcfr Calls hits % hits % react 11 10 91 1 pdwake 4045 pdpgs Disks ad0 ad1 ad2 ad3 ad6 intrn KB/t 8.42 8.42 8.42 8.42 5.67 113904 buf tps 287 287 287 287 1780 133 dirtybuf MB/s 2.36 2.36 2.36 2.36 9.85 69984 desiredvnodes % busy 25 26 24 26 99 21586 numvnodes 17495 freevnodes I use mc to copy. Matyee >Szabó Péter wrote: > >> I have a raid3 setup with 5 consumers. it seems everything is all right, >> but riad is very slow imho, but i don't have any >reference. > >You need to give more accurate information. What are the "consumers" you >mention? Maybe you mean the disks/components of the RAID? > >> Now i copy from raid3 to a single hdd 75GB data. Nothing else run, except >> ssh, and ftp. Before the copy my load is >0.1, after i star the copy the >> load is 2.92 or higher, and the machine is very slow. I think this is not >> normal, or is it? > >>Copy how? By "cp" from shell? > >> My config is: >> AMD Barton 2500+ with 1GB memory. >> >> I stop the copy, load is 0.05, and start again, after a few seconds load >> is 2.02. >Due to the way raid3 works, it's by definition more CPU-intensive than >most other RAID levels, but it shouldn't be even remotely slow as that. >Also, since your load is > 2, what are the other CPU intensive tasks? From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:33:04 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F41EF16A400 for ; Wed, 28 Mar 2007 12:33:03 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id D50B513C46E for ; Wed, 28 Mar 2007 12:33:03 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2SCWrM6064122; Wed, 28 Mar 2007 07:32:53 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460A6075.7000302@freebsd.org> Date: Wed, 28 Mar 2007 07:32:53 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: Ivan Voras References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2946/Wed Mar 28 04:36:58 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-geom@freebsd.org Subject: Re: Geom_label and multiple devices X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:33:04 -0000 On 03/28/07 05:39, Ivan Voras wrote: > I've just thought of something - the situation where geom_label detects > multiple "same" labels is actually very similar to that of > geom_multipath. Obviously this doesn't apply 100% to UFS and other > filesystems' labels (except if there's a way to identify different file > systems by their superblock, for example timestamp of creation?), but in > case of "manual" labels, if the unique ID matches and the label string > matches, it's probably the same device. > > Currently, geom_label discards "additional" devices with the same label, > and I remember there was a discussion about if that should be changed > (though I don't remember the conclusions). It would be useful if > geom_label would work the same as geom_multipath in this case (i.e. > failover to "next" device, add new devices with same label+id to the > pool), or even if geom_multipath were merged in geom_label. > > Comments? > I think that the duplicate label case is not really the same as multipath. I see your point, but the duplicate label issue is more of a cause from mistake rather than intention. To me, these are the cases that could arise for geom_label in the duplicate label case: 1) Same device, multiple paths. Can be detected like you mention above (checking superblocks). That works fine for UFS but not everything else. This case must be certain that it *is* the same device. This is hard to do. 2) Possibly same device with multiple paths. Maybe there's no file system on the device yet, or geom_label can't determine if it's the same device (sizes, etc all match exactly, but it's not 100% certain). 3) Definitely *not* the same device. For (1), once geom_label is certain it is truly the same path, ignore the additional duplicate labels found, but print a warning like: "geom_label: duplicate /dev/... found: multiple paths for same label." "geom_label: removing subsequent duplicates for /dev/..." For (2), we something similar, but rename any duplicates so the administrator can still deal with the duplicate via a label. That helps the admin determine what really went on. Something like this: "geom_label: duplicate /dev/...: possible multiple paths for same label." "geom_label: renaming duplicate label /dev/... to /dev/...-DUP1" And for (3), the result is the same as (2), except the error message is different: "geom_label: duplicate /dev/...: SAME LABEL FOR DIFFERENT DEVICES!" "geom_label: renaming duplicate label /dev/... to /dev/...-DUP1" Maybe it would be a good idea to allow a priority value in the geom_label meta data so that an admin could prioritize his labels. For instance, on my laptop, I have labeled my root partition /dev/label/root. If I have a compact flash plugged in that I'm working on for a small embedded device, I also need it named /dev/label/root, even though I'm already using that label. I want to be able to reboot my laptop with the compact flash inserted still, without it possibly booting off the flash instead. To protect against that, I could prioritize my internal hd's 'root' labeled device to a priority of '1', which means 'nothing is higher priority than this device'. A '0' would mean no priority given, and any number greater than 0 is less priority. That way admins could mark their labels and allow certain devices to be forced more important than others. Comments? Eric From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:35:00 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E4C9A16A401 for ; Wed, 28 Mar 2007 12:35:00 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30301.mail.mud.yahoo.com (web30301.mail.mud.yahoo.com [209.191.69.63]) by mx1.freebsd.org (Postfix) with SMTP id 90D2213C448 for ; Wed, 28 Mar 2007 12:35:00 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: (qmail 51283 invoked by uid 60001); 28 Mar 2007 12:08:19 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=PBX9FzdczA+sXuddjScWHQXlajGSrB37M4hTf0zRakDaQxwH3d9qPJgsFgFFvuRxmmELng41zBxOrK1Gb/JcaCrxXihbTsm1vIWLbj+EIKV//XghxG9eFmH1LHtw8yML0WXD1k3I30CYjN8jALXDVMWC0oAoYTPDOZwUmq/kq6M=; X-YMail-OSG: esHKSWoVM1lI3xHqqQkL07u_es_eMtmgklnpb0.Ghs_ymadcLNhMbQKLDHDw4SwaVJI6xJ9GIfFqo052gp3I0HLjaIy.473xsrga7PiUStUjikhfPsW2yZBYluNOiw-- Received: from [85.212.45.2] by web30301.mail.mud.yahoo.com via HTTP; Wed, 28 Mar 2007 05:08:19 PDT Date: Wed, 28 Mar 2007 05:08:19 -0700 (PDT) From: "R. B. Riddick" To: "Szabó" "Péter" , freebsd-geom@freebsd.org In-Reply-To: <003401c7712a$f71ebb60$6502a8c0@peteruj> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <667838.49636.qm@web30301.mail.mud.yahoo.com> Cc: Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:35:01 -0000 --- Szabó Péter wrote: > I stop the copy, load is 0.05, and start again, after a few seconds load is > 2.02. > Do u use UDMA? What does atacontrol mode say? What if u use gstripe instead of graid3? What if u use "dd if=/dev/zero of=/single-hdd-75GB/test-file-that-did-not-exist-before bs=1m count=128"? What does "systat -vmstat" say (the lower left corner, where the "KB/t[ransfer]" r shown)? -Arne ____________________________________________________________________________________ Never miss an email again! Yahoo! Toolbar alerts you the instant new Mail arrives. http://tools.search.yahoo.com/toolbar/features/mail/ From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:36:10 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 733CB16A402 for ; Wed, 28 Mar 2007 12:36:10 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3FD1213C465 for ; Wed, 28 Mar 2007 12:36:09 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2SCa9bI064736; Wed, 28 Mar 2007 07:36:09 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460A6139.4040401@freebsd.org> Date: Wed, 28 Mar 2007 07:36:09 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Szab=F3_P=E9ter?= References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> In-Reply-To: <005c01c77134$28e0fce0$6502a8c0@peteruj> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/2946/Wed Mar 28 04:36:58 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:36:10 -0000 On 03/28/07 07:25, Szabó Péter wrote: > Yes, consumers are the disks. [..snip..] > #systat -vmstat > > 4 users Load 1.62 1.75 1.82 Mar 28 14:24 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 66488 6968 150740 11396 31172 count > All 1017076 10160975701764 16256 pages > 1 zfod Interrupts > Proc:r p d s w Csw Trp Sys Int Sof Flt cow 4066 total > 1 60 9820 68312106 4237 752 1 195968 wire 1002 0: clk > 79088 act 128 8: rtc > 89.2%Sys 3.0%Intr 3.3%User 0.0%Nice 4.5%Idl 714076 inact 10 10: > fxp > | | | | | | | | | | 29068 cache 1780 11: > ata > =============================================+>> 2104 free 573 14: > ata > 12 daefr 573 15: > ata > Namei Name-cache Dir-cache prcfr > Calls hits % hits % react > 11 10 91 1 pdwake > 4045 pdpgs > Disks ad0 ad1 ad2 ad3 ad6 intrn > KB/t 8.42 8.42 8.42 8.42 5.67 113904 buf > tps 287 287 287 287 1780 133 dirtybuf > MB/s 2.36 2.36 2.36 2.36 9.85 69984 desiredvnodes > % busy 25 26 24 26 99 21586 numvnodes > 17495 freevnodes > > I use mc to copy. > > Matyee Can you send the output of: diskinfo -vt /dev/raid3/nmivol diskinfo -vt /dev/label/nmivol1 Thanks Eric >> Szabó Péter wrote: >> >>> I have a raid3 setup with 5 consumers. it seems everything is all right, >>> but riad is very slow imho, but i don't have any >reference. >> You need to give more accurate information. What are the "consumers" you >> mention? Maybe you mean the disks/components of the RAID? >> >>> Now i copy from raid3 to a single hdd 75GB data. Nothing else run, except >>> ssh, and ftp. Before the copy my load is >0.1, after i star the copy the >>> load is 2.92 or higher, and the machine is very slow. I think this is not >>> normal, or is it? >>> Copy how? By "cp" from shell? >>> My config is: >>> AMD Barton 2500+ with 1GB memory. >>> >>> I stop the copy, load is 0.05, and start again, after a few seconds load >>> is 2.02. > >> Due to the way raid3 works, it's by definition more CPU-intensive than >> most other RAID levels, but it shouldn't be even remotely slow as that. >> Also, since your load is > 2, what are the other CPU intensive tasks? > > _______________________________________________ > freebsd-geom@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 12:48:48 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BAAE516A400; Wed, 28 Mar 2007 12:48:48 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id E788013C46C; Wed, 28 Mar 2007 12:48:47 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id DE8BE487FA; Wed, 28 Mar 2007 14:48:44 +0200 (CEST) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id F080145684; Wed, 28 Mar 2007 14:48:34 +0200 (CEST) Date: Wed, 28 Mar 2007 14:48:32 +0200 From: Pawel Jakub Dawidek To: Eric Anderson Message-ID: <20070328124832.GB35749@garage.freebsd.pl> References: <460A6075.7000302@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KsGdsel6WgEHnImy" Content-Disposition: inline In-Reply-To: <460A6075.7000302@freebsd.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: Ivan Voras , freebsd-geom@freebsd.org Subject: Re: Geom_label and multiple devices X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 12:48:48 -0000 --KsGdsel6WgEHnImy Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 28, 2007 at 07:32:53AM -0500, Eric Anderson wrote: > On 03/28/07 05:39, Ivan Voras wrote: > >I've just thought of something - the situation where geom_label detects = multiple "same" labels is actually very similar to that of geom_multipath. = Obviously this doesn't=20 > >apply 100% to UFS and other filesystems' labels (except if there's a way= to identify different file systems by their superblock, for example timest= amp of creation?), but=20 > >in case of "manual" labels, if the unique ID matches and the label strin= g matches, it's probably the same device. > >Currently, geom_label discards "additional" devices with the same label,= and I remember there was a discussion about if that should be changed (tho= ugh I don't remember the=20 > >conclusions). It would be useful if geom_label would work the same as ge= om_multipath in this case (i.e. failover to "next" device, add new devices = with same label+id to=20 > >the pool), or even if geom_multipath were merged in geom_label. > >Comments? >=20 >=20 > I think that the duplicate label case is not really the same as multipath= =2E I see your point, but the duplicate label issue is more of a cause fr= om mistake rather than=20 > intention. Yeah. When I see duplicated labels, I don't want to show them both with some random names that nobody expect. And think here about automatic processing, not that administrator can take a look and find new name. I can do two things: 1. Present one of those labels. 2. Stop the system (panic?), because it's just administrator fault (misconfiguration). I'd prefer 2, but it's too cruel. Administrators make mistakes and we have some garbage to clean still (eg. 'c' partition). That's why I decided to go with 1. > Maybe it would be a good idea to allow a priority value in the geom_label= meta data so that an admin could prioritize his labels. For instance, on = my laptop, I have=20 > labeled my root partition /dev/label/root. If I have a compact flash plu= gged in that I'm working on for a small embedded device, I also need it nam= ed /dev/label/root, even=20 > though I'm already using that label. I want to be able to reboot my lapt= op with the compact flash inserted still, without it possibly booting off t= he flash instead. To=20 > protect against that, I could prioritize my internal hd's 'root' labeled = device to a priority of '1', which means 'nothing is higher priority than t= his device'. A '0'=20 > would mean no priority given, and any number greater than 0 is less prior= ity. That way admins could mark their labels and allow certain devices to = be forced more important=20 > than others. Comments? I like this idea, but it can only be used for glabel's native labels. You can't use the same method for filesystem volume names, and those are more often used than glabel's native labels. Don't read me wrong, I still like your idea and would be glad to commit something like this. Care to provide a patch?:) --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --KsGdsel6WgEHnImy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFGCmQgForvXbEpPzQRAj9mAJ9ZGU8dDyoL0pqO5sZ0T3KCwoBcsgCcCglG uGhLjdWC154HgMb6pbkMQek= =a2mv -----END PGP SIGNATURE----- --KsGdsel6WgEHnImy-- From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:02:47 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C98C816A408 for ; Wed, 28 Mar 2007 13:02:47 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id 26D4E13C459 for ; Wed, 28 Mar 2007 13:02:46 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 21CE133C35B for ; Wed, 28 Mar 2007 15:02:45 +0200 (CEST) Message-ID: <008701c77139$613694b0$6502a8c0@peteruj> From: =?iso-8859-1?B?U3phYvMgUOl0ZXI=?= To: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <460A6139.4040401@freebsd.org> Date: Wed, 28 Mar 2007 15:02:45 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:02:48 -0000 I stop the copy #diskinfo -vt /dev/raid3/nmivol /dev/raid3/nmivol 2048 # sectorsize 638906845184 # mediasize in bytes (595G) 311966233 # mediasize in sectors Seek times: Full stroke: 250 iter in 11.486148 sec = 45.945 msec Half stroke: 250 iter in 8.818982 sec = 35.276 msec Quarter stroke: 500 iter in 15.365682 sec = 30.731 msec Short forward: 400 iter in 6.013585 sec = 15.034 msec Short backward: 400 iter in 10.356176 sec = 25.890 msec Seq outer: 2048 iter in 0.803544 sec = 0.392 msec Seq inner: 2048 iter in 0.804926 sec = 0.393 msec Transfer rates: outside: 102400 kbytes in 1.179933 sec = 86785 kbytes/sec middle: 102400 kbytes in 1.148816 sec = 89135 kbytes/sec inside: 102400 kbytes in 1.171987 sec = 87373 kbytes/sec #diskinfo -vt /dev/label/nmivol1 /dev/label/nmivol1 512 # sectorsize 159726711808 # mediasize in bytes (149G) 311966234 # mediasize in sectors 309490 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. Seek times: Full stroke: 250 iter in 5.540328 sec = 22.161 msec Half stroke: 250 iter in 3.955182 sec = 15.821 msec Quarter stroke: 500 iter in 6.648877 sec = 13.298 msec Short forward: 400 iter in 2.203517 sec = 5.509 msec Short backward: 400 iter in 1.960238 sec = 4.901 msec Seq outer: 2048 iter in 0.288225 sec = 0.141 msec Seq inner: 2048 iter in 0.292780 sec = 0.143 msec Transfer rates: outside: 102400 kbytes in 1.839020 sec = 55682 kbytes/sec middle: 102400 kbytes in 2.112483 sec = 48474 kbytes/sec inside: 102400 kbytes in 3.416896 sec = 29969 kbytes/sec # atacontrol mode ad0 current mode = UDMA100 # atacontrol mode ad1 current mode = UDMA100 # atacontrol mode ad2 current mode = UDMA100 # atacontrol mode ad3 current mode = UDMA100 # atacontrol mode ad6 current mode = SATA150 gstripe may help, and extends disk space, but less secure. #dd if=/dev/zero of=/mnt/new/x.x bs=1m count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 11.410957 secs (11762180 bytes/sec) Matyee ----- Original Message ----- From: "Eric Anderson" On 03/28/07 07:25, Szabó Péter wrote: > Yes, consumers are the disks. [..snip..] > #systat -vmstat > > 4 users Load 1.62 1.75 1.82 Mar 28 14:24 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 66488 6968 150740 11396 31172 count > All 1017076 10160975701764 16256 pages > 1 zfod > Interrupts > Proc:r p d s w Csw Trp Sys Int Sof Flt cow 4066 > total > 1 60 9820 68312106 4237 752 1 195968 wire 1002 0: > clk > 79088 act 128 8: > rtc > 89.2%Sys 3.0%Intr 3.3%User 0.0%Nice 4.5%Idl 714076 inact 10 10: > fxp > | | | | | | | | | | 29068 cache 1780 11: > ata > =============================================+>> 2104 free 573 14: > ata > 12 daefr 573 15: > ata > Namei Name-cache Dir-cache prcfr > Calls hits % hits % react > 11 10 91 1 pdwake > 4045 pdpgs > Disks ad0 ad1 ad2 ad3 ad6 intrn > KB/t 8.42 8.42 8.42 8.42 5.67 113904 buf > tps 287 287 287 287 1780 133 dirtybuf > MB/s 2.36 2.36 2.36 2.36 9.85 69984 desiredvnodes > % busy 25 26 24 26 99 21586 numvnodes > 17495 freevnodes > > I use mc to copy. > > Matyee Can you send the output of: diskinfo -vt /dev/raid3/nmivol diskinfo -vt /dev/label/nmivol1 Thanks Eric >> Szabó Péter wrote: >> >>> I have a raid3 setup with 5 consumers. it seems everything is all right, >>> but riad is very slow imho, but i don't have any >reference. >> You need to give more accurate information. What are the "consumers" you >> mention? Maybe you mean the disks/components of the RAID? >> >>> Now i copy from raid3 to a single hdd 75GB data. Nothing else run, >>> except ssh, and ftp. Before the copy my load is >0.1, after i star the >>> copy the load is 2.92 or higher, and the machine is very slow. I think >>> this is not normal, or is it? >>> Copy how? By "cp" from shell? >>> My config is: >>> AMD Barton 2500+ with 1GB memory. >>> >>> I stop the copy, load is 0.05, and start again, after a few seconds load >>> is 2.02. > >> Due to the way raid3 works, it's by definition more CPU-intensive than >> most other RAID levels, but it shouldn't be even remotely slow as that. >> Also, since your load is > 2, what are the other CPU intensive tasks? > From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:07:50 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 60A3416A400; Wed, 28 Mar 2007 13:07:50 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3E59213C45D; Wed, 28 Mar 2007 13:07:50 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2SD7lrD070365; Wed, 28 Mar 2007 08:07:47 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460A68A3.1060605@freebsd.org> Date: Wed, 28 Mar 2007 08:07:47 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <460A6075.7000302@freebsd.org> <20070328124832.GB35749@garage.freebsd.pl> In-Reply-To: <20070328124832.GB35749@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2946/Wed Mar 28 04:36:58 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: Ivan Voras , freebsd-geom@freebsd.org Subject: Re: Geom_label and multiple devices X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:07:50 -0000 On 03/28/07 07:48, Pawel Jakub Dawidek wrote: > On Wed, Mar 28, 2007 at 07:32:53AM -0500, Eric Anderson wrote: >> On 03/28/07 05:39, Ivan Voras wrote: >>> I've just thought of something - the situation where geom_label detects multiple "same" labels is actually very similar to that of geom_multipath. Obviously this doesn't >>> apply 100% to UFS and other filesystems' labels (except if there's a way to identify different file systems by their superblock, for example timestamp of creation?), but >>> in case of "manual" labels, if the unique ID matches and the label string matches, it's probably the same device. >>> Currently, geom_label discards "additional" devices with the same label, and I remember there was a discussion about if that should be changed (though I don't remember the >>> conclusions). It would be useful if geom_label would work the same as geom_multipath in this case (i.e. failover to "next" device, add new devices with same label+id to >>> the pool), or even if geom_multipath were merged in geom_label. >>> Comments? >> >> I think that the duplicate label case is not really the same as multipath. I see your point, but the duplicate label issue is more of a cause from mistake rather than >> intention. > > Yeah. When I see duplicated labels, I don't want to show them both with > some random names that nobody expect. And think here about automatic > processing, not that administrator can take a look and find new name. > I can do two things: > 1. Present one of those labels. > 2. Stop the system (panic?), because it's just administrator fault > (misconfiguration). > > I'd prefer 2, but it's too cruel. Administrators make mistakes and we > have some garbage to clean still (eg. 'c' partition). That's why I > decided to go with 1. The problem with 2 is that any random person walking up to your system or handing you a usb stick labeled 'incorrectly' could panic your system. Not a good thing. >> Maybe it would be a good idea to allow a priority value in the geom_label meta data so that an admin could prioritize his labels. For instance, on my laptop, I have >> labeled my root partition /dev/label/root. If I have a compact flash plugged in that I'm working on for a small embedded device, I also need it named /dev/label/root, even >> though I'm already using that label. I want to be able to reboot my laptop with the compact flash inserted still, without it possibly booting off the flash instead. To >> protect against that, I could prioritize my internal hd's 'root' labeled device to a priority of '1', which means 'nothing is higher priority than this device'. A '0' >> would mean no priority given, and any number greater than 0 is less priority. That way admins could mark their labels and allow certain devices to be forced more important >> than others. Comments? > > I like this idea, but it can only be used for glabel's native labels. > You can't use the same method for filesystem volume names, and those are > more often used than glabel's native labels. This is true, and I thought of that. Since the main case is UFS2, we could possibly (dare I suggest) use one of the empty slots in the UFS superblock to put in a priority value, and add an option to tunefs/newfs to set that value. The superblock is already tasted to find the label name, so while we're there, just grab the priority and use the same values as mentioned above. That shouldn't break any backwards compatibility either. That doesn't solve the msdosfs case, but that is probably a lot less common, and possibly less detrimental to a system compared to UFS. > Don't read me wrong, I still like your idea and would be glad to commit > something like this. Care to provide a patch?:) I'll give it a shot - but give me some time, I only have about 5 mins of free time per day. :) Eric From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:21:35 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8681E16A401 for ; Wed, 28 Mar 2007 13:21:35 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30313.mail.mud.yahoo.com (web30313.mail.mud.yahoo.com [209.191.69.75]) by mx1.freebsd.org (Postfix) with SMTP id 51BFC13C44B for ; Wed, 28 Mar 2007 13:21:35 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: (qmail 54721 invoked by uid 60001); 28 Mar 2007 13:21:34 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=vTBBacIiLM5CADIjNMuZn9kX/RQ2u15E4rr3f2N4Ar2auWoiop3Xx23MfBVKh2g4NwOkcclU4c0RUSYkEsWvTNenjvPrEz7h31RydmJ3qcpF3PIb4IrU91e2CAO+DPz1zv4JfFw3IGH9U8QTfoVtoz8Hb6ot+D3X9PrkXXqEjnc=; X-YMail-OSG: lZMElLcVM1l77JBe1zIJNehY9u7h7CvJbKx5k32duj4ZMPZdqbHArDGdGFUstAOXilV_kjhxpwDy6NcTAlSZJOxJP7fUWOJFyQZTOt7NVD1yS4UVRNgfy37eSOHZXw-- Received: from [85.212.45.2] by web30313.mail.mud.yahoo.com via HTTP; Wed, 28 Mar 2007 06:21:34 PDT Date: Wed, 28 Mar 2007 06:21:34 -0700 (PDT) From: "R. B. Riddick" To: "Szabó" "Péter" , freebsd-geom@freebsd.org In-Reply-To: <008701c77139$613694b0$6502a8c0@peteruj> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <657228.54534.qm@web30313.mail.mud.yahoo.com> Cc: Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:21:35 -0000 --- Szabó Péter wrote: > raid3 > Transfer rates: > outside: 102400 kbytes in 1.179933 sec = 86785 kbytes/sec > middle: 102400 kbytes in 1.148816 sec = 89135 kbytes/sec > inside: 102400 kbytes in 1.171987 sec = 87373 kbytes/sec > Looks good... > #diskinfo -vt /dev/label/nmivol1 > outside: 102400 kbytes in 1.839020 sec = 55682 kbytes/sec > middle: 102400 kbytes in 2.112483 sec = 48474 kbytes/sec > inside: 102400 kbytes in 3.416896 sec = 29969 kbytes/sec > Looks slow... > # atacontrol mode ad0 > current mode = UDMA100 > # atacontrol mode ad1 > current mode = UDMA100 > # atacontrol mode ad2 > current mode = UDMA100 > # atacontrol mode ad3 > current mode = UDMA100 > # atacontrol mode ad6 > current mode = SATA150 > Looks good... > gstripe may help, and extends disk space, but less secure. > Yup... > #dd if=/dev/zero of=/mnt/new/x.x bs=1m count=128 > 128+0 records in > 128+0 records out > 134217728 bytes transferred in 11.410957 secs (11762180 bytes/sec) > Hmm And what if u try this: dd if=/dev/raid3/... of=/dev/null bs=1m count=128 ? > > Disks ad0 ad1 ad2 ad3 ad6 intrn > > KB/t 8.42 8.42 8.42 8.42 5.67 113904 buf > > tps 287 287 287 287 1780 133 dirtybuf > > MB/s 2.36 2.36 2.36 2.36 9.85 69984 desiredvnodes > > % busy 25 26 24 26 99 21586 numvnodes > 8KB/transfer is a little low for my taste... Looks like ur copy program uses 32KB transfers, instead of 128KB transfers... > > I use mc to copy. > What if u try dd if=/source-file/on/raid3-device of=/dev/null bs=1m count=128 ? -Arne ____________________________________________________________________________________ Don't pick lemons. See all the new 2007 cars at Yahoo! Autos. http://autos.yahoo.com/new_cars.html From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:42:52 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6FBDB16A404 for ; Wed, 28 Mar 2007 13:42:52 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 43BA913C468 for ; Wed, 28 Mar 2007 13:42:52 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2SDgpmt076664; Wed, 28 Mar 2007 08:42:51 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460A70DB.6030808@freebsd.org> Date: Wed, 28 Mar 2007 08:42:51 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Szab=F3_P=E9ter?= References: <657228.54534.qm@web30313.mail.mud.yahoo.com> In-Reply-To: <657228.54534.qm@web30313.mail.mud.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/2948/Wed Mar 28 07:30:01 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: "R. B. Riddick" , freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:42:52 -0000 On 03/28/07 08:21, R. B. Riddick wrote: > --- Szabó Péter wrote: >> raid3 >> Transfer rates: >> outside: 102400 kbytes in 1.179933 sec = 86785 kbytes/sec >> middle: 102400 kbytes in 1.148816 sec = 89135 kbytes/sec >> inside: 102400 kbytes in 1.171987 sec = 87373 kbytes/sec >> > Looks good... It's about right, I agree. >> #diskinfo -vt /dev/label/nmivol1 >> outside: 102400 kbytes in 1.839020 sec = 55682 kbytes/sec >> middle: 102400 kbytes in 2.112483 sec = 48474 kbytes/sec >> inside: 102400 kbytes in 3.416896 sec = 29969 kbytes/sec >> > Looks slow... You think so? I think that's pretty acceptable for a UDMA100 device. >> # atacontrol mode ad0 >> current mode = UDMA100 >> # atacontrol mode ad1 >> current mode = UDMA100 >> # atacontrol mode ad2 >> current mode = UDMA100 >> # atacontrol mode ad3 >> current mode = UDMA100 >> # atacontrol mode ad6 >> current mode = SATA150 >> > Looks good... > >> gstripe may help, and extends disk space, but less secure. >> > Yup... > >> #dd if=/dev/zero of=/mnt/new/x.x bs=1m count=128 >> 128+0 records in >> 128+0 records out >> 134217728 bytes transferred in 11.410957 secs (11762180 bytes/sec) >> > Hmm That looks pretty poor to me. Can you send the output of a df -ik /mnt? > And what if u try this: > dd if=/dev/raid3/... of=/dev/null bs=1m count=128 > ? > >>> Disks ad0 ad1 ad2 ad3 ad6 intrn >>> KB/t 8.42 8.42 8.42 8.42 5.67 113904 buf >>> tps 287 287 287 287 1780 133 dirtybuf >>> MB/s 2.36 2.36 2.36 2.36 9.85 69984 desiredvnodes >>> % busy 25 26 24 26 99 21586 numvnodes > 8KB/transfer is a little low for my taste... Looks like ur copy program uses > 32KB transfers, instead of 128KB transfers... > >>> I use mc to copy. > What if u try > dd if=/source-file/on/raid3-device of=/dev/null bs=1m count=128 > ? It seems that the write speed is the real issue possibly. A great test would be to write to the device itself, but that would blast the filesystem. If that's ok, this would be a good test: dd if=/dev/zero of=/dev/raid3/... bs=1m count=512 Eric From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:44:28 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0C9C316A406 for ; Wed, 28 Mar 2007 13:44:28 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 92C5213C4B7 for ; Wed, 28 Mar 2007 13:44:27 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E1F9B487F4; Wed, 28 Mar 2007 15:44:25 +0200 (CEST) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id C8568487F3; Wed, 28 Mar 2007 15:44:13 +0200 (CEST) Date: Wed, 28 Mar 2007 15:44:11 +0200 From: Pawel Jakub Dawidek To: "R. B. Riddick" Message-ID: <20070328134411.GC35749@garage.freebsd.pl> References: <008701c77139$613694b0$6502a8c0@peteruj> <657228.54534.qm@web30313.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ftEhullJWpWg/VHq" Content-Disposition: inline In-Reply-To: <657228.54534.qm@web30313.mail.mud.yahoo.com> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: =?iso-8859-2?Q?Szab=F3_P=E9ter?= , freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:44:28 -0000 --ftEhullJWpWg/VHq Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 28, 2007 at 06:21:34AM -0700, R. B. Riddick wrote: > > # atacontrol mode ad0 > > current mode =3D UDMA100 > > # atacontrol mode ad1 > > current mode =3D UDMA100 > > # atacontrol mode ad2 > > current mode =3D UDMA100 > > # atacontrol mode ad3 > > current mode =3D UDMA100 > > # atacontrol mode ad6 > > current mode =3D SATA150 > > > Looks good... Does it? It seems there are two disks per ata channel, which is very bad for performance. I suggesting trying this: # apply "dd if=3D/dev/ad%1 if=3D/dev/null bs=3D1m count=3D250 &" 0 1 2 3 6 and once it completes try this: # dd if=3D/dev/raid3/ of=3D/dev/null bs=3D1m count=3D1000 and send the results here. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --ftEhullJWpWg/VHq Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFGCnErForvXbEpPzQRAnORAJ0aW5VMFuKgplQyDQO+xt5mlcgvowCfUBKi 5MqcuDubXY/RGbh5couCGos= =vssD -----END PGP SIGNATURE----- --ftEhullJWpWg/VHq-- From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 13:46:01 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9DBF116A403 for ; Wed, 28 Mar 2007 13:46:01 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta01.mail.t-online.hu (mta01.mail.t-online.hu [195.228.240.50]) by mx1.freebsd.org (Postfix) with ESMTP id 612C713C448 for ; Wed, 28 Mar 2007 13:46:01 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 87E9F33C1D4 for ; Wed, 28 Mar 2007 15:46:00 +0200 (CEST) Message-ID: <009601c7713f$6b9f2a60$6502a8c0@peteruj> From: =?iso-8859-1?B?U3phYvMgUOl0ZXI=?= To: References: <657228.54534.qm@web30313.mail.mud.yahoo.com> Date: Wed, 28 Mar 2007 15:46:00 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 13:46:01 -0000 > And what if u try this: > dd if=/dev/raid3/... of=/dev/null bs=1m count=128 > ? #dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 1.555843 secs (86266891 bytes/sec) > 8KB/transfer is a little low for my taste... Looks like ur copy program > uses > 32KB transfers, instead of 128KB transfers... > What if u try > dd if=/source-file/on/raid3-device of=/dev/null bs=1m count=128 > ? dd if=/mnt/vol/site/video/Sissi_2-3/SISSI_2.ISO of=/dev/null bs=1m count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 6.096775 secs (22014545 bytes/sec) > -Arne Matyee ____________________________________________________________________________________ Don't pick lemons. See all the new 2007 cars at Yahoo! Autos. http://autos.yahoo.com/new_cars.html _______________________________________________ freebsd-geom@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-geom To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 14:00:31 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 77CD116A402 for ; Wed, 28 Mar 2007 14:00:31 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30305.mail.mud.yahoo.com (web30305.mail.mud.yahoo.com [209.191.69.67]) by mx1.freebsd.org (Postfix) with SMTP id 245AD13C45A for ; Wed, 28 Mar 2007 14:00:31 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: (qmail 73737 invoked by uid 60001); 28 Mar 2007 14:00:29 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=dcolWhGHu6htms96biqIAuKDfxTkmBJmgvGSRQ04Ff3vhdhVsBlAk1TP/IlRRlJP9dD5LHIPHc+BnOAuN2idusJcMkcro9BbJnd6i593frYHxxete01+iLv6s08pzpA60iEtXXGSbdctHWSM1H0MIDJAcM55Db2o++ljnPV8SmA=; X-YMail-OSG: wzi4u5YVM1kZ2BusduWzZDRu2C0zPz0ltTz0WEGq Received: from [85.212.45.2] by web30305.mail.mud.yahoo.com via HTTP; Wed, 28 Mar 2007 07:00:29 PDT Date: Wed, 28 Mar 2007 07:00:29 -0700 (PDT) From: "R. B. Riddick" To: "Szabó" "Péter" , freebsd-geom@freebsd.org In-Reply-To: <009601c7713f$6b9f2a60$6502a8c0@peteruj> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <299130.71872.qm@web30305.mail.mud.yahoo.com> Cc: Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 14:00:31 -0000 Eric wrote: >>> inside: 102400 kbytes in 3.416896 sec = 29969 kbytes/sec >>> >> Looks slow... > > You think so? I think that's pretty acceptable for a UDMA100 device. > Yup... I thought, it was 2MB/sec... Dont know why... 29-55 is good... --- Szabó Péter wrote: > > And what if u try this: > > dd if=/dev/raid3/... of=/dev/null bs=1m count=128 > > ? > > #dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=128 > 128+0 records in > 128+0 records out > 134217728 bytes transferred in 1.555843 secs (86266891 bytes/sec) > OK > > 8KB/transfer is a little low for my taste... Looks like ur copy program > > uses > > 32KB transfers, instead of 128KB transfers... > > > What if u try > > dd if=/source-file/on/raid3-device of=/dev/null bs=1m count=128 > > ? > > dd if=/mnt/vol/site/video/Sissi_2-3/SISSI_2.ISO of=/dev/null bs=1m count=128 > 128+0 records in > 128+0 records out > 134217728 bytes transferred in 6.096775 secs (22014545 bytes/sec) > Hmm... It is still quite slow (the raw device can do about 80MB/sec)... Is it better than with mc? How was CPU-usage / system-load? -Arne ____________________________________________________________________________________ Expecting? Get great news right away with email Auto-Check. Try the Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/newmail_tools.html From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 14:31:06 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7D8FA16A405 for ; Wed, 28 Mar 2007 14:31:06 +0000 (UTC) (envelope-from geoffb@chuggalug.clues.com) Received: from chuggalug.clues.com (chuggalug2.demon.co.uk [83.104.169.191]) by mx1.freebsd.org (Postfix) with ESMTP id E218113C4B9 for ; Wed, 28 Mar 2007 14:31:05 +0000 (UTC) (envelope-from geoffb@chuggalug.clues.com) Received: from chuggalug.clues.com (localhost [127.0.0.1]) by chuggalug.clues.com (8.14.0/8.12.10) with ESMTP id l2SE7JCd065248; Wed, 28 Mar 2007 14:07:19 GMT (envelope-from geoffb@chuggalug.clues.com) Received: (from geoffb@localhost) by chuggalug.clues.com (8.13.8/8.12.10/Submit) id l2SE7JoH065247; Wed, 28 Mar 2007 14:07:19 GMT (envelope-from geoffb) Date: Wed, 28 Mar 2007 14:07:19 +0000 From: Geoff Buckingham To: Szab?? P??ter Message-ID: <20070328140719.GA57134@chuggalug.clues.com> References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <005c01c77134$28e0fce0$6502a8c0@peteruj> User-Agent: Mutt/1.4.1i Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 14:31:06 -0000 On Wed, Mar 28, 2007 at 02:25:23PM +0200, Szab?? P??ter wrote: > Yes, consumers are the disks. > > # graid3 list > > Geom name: nmivol > State: DEGRADED > Components: 5 > Flags: NONE > GenID: 0 > SyncID: 73 > ID: 2396346217 > Zone64kFailed: 0 > Zone64kRequested: 13987 > Zone16kFailed: 11189 > Zone16kRequested: 28827400 > Zone4kFailed: 71105 > Zone4kRequested: 29207293 I could have be missing something here, but does DEGRADED above not have an impact on performance? From owner-freebsd-geom@FreeBSD.ORG Wed Mar 28 14:37:24 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D5E216A401 for ; Wed, 28 Mar 2007 14:37:24 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id E16DA13C4B9 for ; Wed, 28 Mar 2007 14:37:21 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 9A7872084; Wed, 28 Mar 2007 16:37:15 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 8C13B207E; Wed, 28 Mar 2007 16:37:15 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 6440EA1073; Wed, 28 Mar 2007 16:37:15 +0200 (CEST) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: =?iso-8859-1?Q?Szab=F3_P=E9ter?= References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> Date: Wed, 28 Mar 2007 16:37:15 +0200 In-Reply-To: <005c01c77134$28e0fce0$6502a8c0@peteruj> (=?iso-8859-1?Q?Szab?= =?iso-8859-1?Q?=F3_P=E9ter's?= message of "Wed, 28 Mar 2007 14:25:23 +0200") Message-ID: <86zm5xph7o.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 14:37:24 -0000 Szab=F3 P=E9ter writes: > # graid3 list > > Geom name: nmivol > State: DEGRADED First of all, your array is degraded. The graid3 output shows only four components. Unless the missing fifth component happens to be the parity disk, geom_raid3 will have to reconstruct your data on the fly. Second, you say nothing about which spindles the components are on and how those spindles are attached to the system. If some of those components are on the same spindle, or if some of those spindles are on the same ATA channel, performance will suffer considerably. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 08:41:26 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 576AF16A401 for ; Thu, 29 Mar 2007 08:41:26 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id EFD4E13C45A for ; Thu, 29 Mar 2007 08:41:25 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id BC0E133C45B for ; Thu, 29 Mar 2007 10:41:24 +0200 (CEST) Message-ID: <003301c771de$08d03650$6502a8c0@peteruj> From: =?iso-8859-2?B?U3phYvMgUOl0ZXI=?= To: References: <008701c77139$613694b0$6502a8c0@peteruj><657228.54534.qm@web30313.mail.mud.yahoo.com> <20070328134411.GC35749@garage.freebsd.pl> Date: Thu, 29 Mar 2007 10:41:23 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-2"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 08:41:26 -0000 > On Wed, Mar 28, 2007 at 06:21:34AM -0700, R. B. Riddick wrote: > > > # atacontrol mode ad0 > > > current mode = UDMA100 > > > # atacontrol mode ad1 > > > current mode = UDMA100 > > > # atacontrol mode ad2 > > > current mode = UDMA100 > > > # atacontrol mode ad3 > > > current mode = UDMA100 > > > # atacontrol mode ad6 > > > current mode = SATA150 > > > > > Looks good... > > Does it? It seems there are two disks per ata channel, which is very bad > for performance. > > I suggesting trying this: > > # apply "dd if=/dev/ad%1 if=/dev/null bs=1m count=250 &" 0 1 2 3 6 > > and once it completes try this: > # dd if=/dev/raid3/ of=/dev/null bs=1m count=1000 > and send the results here. A SATA cable was wrong, and the 5th disk of the raid was not connected. Now i fix it. Geom name: nmivol State: COMPLETE Components: 5 Flags: NONE GenID: 0 SyncID: 77 ID: 2396346217 Zone64kFailed: 0 Zone64kRequested: 19808642 Zone16kFailed: 0 Zone16kRequested: 4629259 Zone4kFailed: 28270 Zone4kRequested: 2027159 # apply "dd if=/dev/ad%1 of=/dev/null bs=1m count=250 &" 0 1 2 3 4 250+0 records out 262144000 bytes transferred in 4.324466 secs (60618814 bytes/sec) 250+0 records in 250+0 records out 262144000 bytes transferred in 6.289833 secs (41677418 bytes/sec) 250+0 records in 250+0 records out 262144000 bytes transferred in 6.262564 secs (41858894 bytes/sec) 250+0 records in 250+0 records out 262144000 bytes transferred in 7.871292 secs (33303808 bytes/sec) 250+0 records in 250+0 records out 262144000 bytes transferred in 7.857484 secs (33362333 bytes/sec) I forgot to tell, one label of the raid is a gbde encoded partition. If i do a read test from the raid, the load seems normal?!?! I don't know how high is normal. # dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=30000 30000+0 records in 30000+0 records out 31457280000 bytes transferred in 336.545001 secs (93471244 bytes/sec) # top last pid: 16191; load averages: 0.32, 0.36, 0.49 up 0+15:16:03 10:14:12 58 processes: 1 running, 57 sleeping CPU states: 0.0% user, 0.0% nice, 35.9% system, 3.5% interrupt, 60.5% idle Mem: 50M Active, 306M Inact, 165M Wired, 20K Cache, 111M Buf, 475M Free # systat -vmstat 7 users Load 0.45 0.39 0.50 Mar 29 10:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 60900 8108 143488 9684 486056 count All 533396 12740622626264 15932 pages zfod Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 3986 total 1 56 14559 2 418 4171 10 9 169268 wire 1000 0: clk 51560 act 128 8: rtc 38.9%Sys 3.3%Intr 0.0%User 0.0%Nice 57.8%Idl 313632 inact 9 10: fxp | | | | | | | | | | 20 cache 1 11: ata ===================++ 486036 free 1424 14: ata daefr 1424 15: ata Namei Name-cache Dir-cache prcfr Calls hits % hits % react pdwake pdpgs Disks ad0 ad1 ad2 ad3 ad4 intrn KB/t 31.97 31.97 31.97 31.97 3.12 113904 buf tps 712 712 712 712 1 3 dirtybuf MB/s 22.22 22.22 22.22 22.22 0.00 69984 desiredvnodes % busy 33 71 32 59 0 59556 numvnodes 17488 freevnodes The same test from the encoded partition: # dd if=/dev/raid3/nmivole.bde of=/dev/null bs=1m count=30000 30000+0 records in 30000+0 records out 31457280000 bytes transferred in 1282.804938 secs (24522263 bytes/sec) # top last pid: 16322; load averages: 1.07, 0.66, 0.55 up 0+15:22:28 10:20:37 61 processes: 1 running, 60 sleeping CPU states: 0.4% user, 0.0% nice, 90.3% system, 0.4% interrupt, 8.9% idle Mem: 58M Active, 306M Inact, 166M Wired, 20K Cache, 111M Buf, 466M Free # systat -vmstat 7 users Load 1.04 0.70 0.57 Mar 29 10:21 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 60912 8108 143504 9684 486044 count All 533408 12740622626280 15932 pages zfod Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 3014 total 1 56 8575 5 275 3181 20 13 169268 wire 999 0: clk 51572 act 128 8: rtc 81.8%Sys 2.5%Intr 0.2%User 0.0%Nice 15.5%Idl 313632 inact 8 10: fxp | | | | | | | | | | 20 cache 1 11: ata =========================================+ 486024 free 939 14: ata daefr 939 15: ata Namei Name-cache Dir-cache prcfr Calls hits % hits % react pdwake pdpgs Disks ad0 ad1 ad2 ad3 ad4 intrn KB/t 12.88 12.88 12.88 12.88 0.50 113904 buf tps 470 470 470 470 1 22 dirtybuf MB/s 5.92 5.91 5.92 5.92 0.00 69984 desiredvnodes % busy 24 40 20 35 0 59556 numvnodes 17489 freevnodes Maybe is what the hardware can do :( , isn't it? Matyee From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 08:57:51 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7912216A402 for ; Thu, 29 Mar 2007 08:57:51 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30304.mail.mud.yahoo.com (web30304.mail.mud.yahoo.com [209.191.69.66]) by mx1.freebsd.org (Postfix) with SMTP id 448D813C455 for ; Thu, 29 Mar 2007 08:57:51 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: (qmail 40089 invoked by uid 60001); 29 Mar 2007 08:57:50 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=ulxnH9ORvu6/yYIGkEjc8hQTvOsIknRtuEjYAPY5y6S1xJ6DqV4wP7sccUu/7e3gAWcO0lga5RX5Ureke5OL+3KzEUVgmOX08ESrop83TH54pzgf5xzsK/8wveUUMrd5HaK88G7xTuP5EF5JKMq2/GX/xZBHmtA1LiY0Ar1l8H8=; X-YMail-OSG: EIl0QeoVM1nHEwzuKsWnUPvxhPrr6fqzih.uqu1FpMKI848SKAn8JJGmf8GMfItWjAxQSgWfno7PDE6SRQ603SKX71ENuz2EQCmwM9Vi_V2JNeEn.mshTEtechW77g-- Received: from [85.212.19.59] by web30304.mail.mud.yahoo.com via HTTP; Thu, 29 Mar 2007 01:57:50 PDT Date: Thu, 29 Mar 2007 01:57:50 -0700 (PDT) From: "R. B. Riddick" To: "Szabó" "Péter" , freebsd-geom@freebsd.org In-Reply-To: <003301c771de$08d03650$6502a8c0@peteruj> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <707306.40086.qm@web30304.mail.mud.yahoo.com> Cc: Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 08:57:51 -0000 --- Szabó Péter wrote: > A SATA cable was wrong, and the 5th disk of the raid was not connected. Now > i fix it. > > Geom name: nmivol > State: COMPLETE > OK... :-) > # apply "dd if=/dev/ad%1 of=/dev/null bs=1m count=250 &" 0 1 2 3 4 > 250+0 records out > 262144000 bytes transferred in 4.324466 secs (60618814 bytes/sec) > 250+0 records in > 250+0 records out > 262144000 bytes transferred in 6.289833 secs (41677418 bytes/sec) > 250+0 records in > 250+0 records out > 262144000 bytes transferred in 6.262564 secs (41858894 bytes/sec) > 250+0 records in > 250+0 records out > 262144000 bytes transferred in 7.871292 secs (33303808 bytes/sec) > 250+0 records in > 250+0 records out > 262144000 bytes transferred in 7.857484 secs (33362333 bytes/sec) > Looks good... > I forgot to tell, one label of the raid is a gbde encoded partition. If i do > a read test from the raid, the load seems normal?!?! I don't know how high > is normal. > And why is that? I mean: It doesnt encrypt the data really, because graid3 can still read with a missing disk (consumer)... > # dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=30000 > 30000+0 records in > 30000+0 records out > 31457280000 bytes transferred in 336.545001 secs (93471244 bytes/sec) > Doesnt look much faster than before (86MB/sec or so?)... What happens, when u do not use a gbde device as a consumer? What happens when u try to read from a file system on top of the graid3 device? -Arne ____________________________________________________________________________________ Don't pick lemons. See all the new 2007 cars at Yahoo! Autos. http://autos.yahoo.com/new_cars.html From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 09:07:11 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8ACA816A401 for ; Thu, 29 Mar 2007 09:07:11 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id 4FA6713C459 for ; Thu, 29 Mar 2007 09:07:11 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 5F56C33C88F for ; Thu, 29 Mar 2007 11:07:10 +0200 (CEST) Message-ID: <004601c771e1$a2123130$6502a8c0@peteruj> From: =?ISO-8859-1?B?U3phYvMgUOl0ZXI=?= To: References: <707306.40086.qm@web30304.mail.mud.yahoo.com> Date: Thu, 29 Mar 2007 11:07:09 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 09:07:11 -0000 ----- Original Message ----- From: "R. B. Riddick" > And why is that? > I mean: It doesnt encrypt the data really, because graid3 can still read > with a > missing disk (consumer)... > > ...... > > > ..... > > What happens, when u do not use a gbde device as a consumer? > > What happens when u try to read from a file system on top of the graid3 > device? > > -Arne not the consumer encoded. The hole raid. /dev/raid3/nmivol the raid device and the endoced is /dev/raid3/nmivole. Matyee ____________________________________________________________________________________ Don't pick lemons. See all the new 2007 cars at Yahoo! Autos. http://autos.yahoo.com/new_cars.html From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 09:29:22 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C541416A405 for ; Thu, 29 Mar 2007 09:29:22 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta02.mail.t-online.hu (mta02.mail.t-online.hu [195.228.240.51]) by mx1.freebsd.org (Postfix) with ESMTP id 8899C13C4B7 for ; Thu, 29 Mar 2007 09:29:22 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id 48A2C33CA67 for ; Thu, 29 Mar 2007 11:29:19 +0200 (CEST) Message-ID: <005301c771e4$bb0a3900$6502a8c0@peteruj> From: =?ISO-8859-1?B?U3phYvMgUOl0ZXI=?= To: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> Date: Thu, 29 Mar 2007 11:29:17 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 09:29:22 -0000 ----- Original Message ----- > From: "Dag-Erling "Smørgrav"" > > Szabó Péter writes: > > # graid3 list > > > > Geom name: nmivol > > State: DEGRADED > > First of all, your array is degraded. The graid3 output shows only > four components. Unless the missing fifth component happens to be the > parity disk, geom_raid3 will have to reconstruct your data on the fly. > > Second, you say nothing about which spindles the components are on and > how those spindles are attached to the system. If some of those > components are on the same spindle, or if some of those spindles are > on the same ATA channel, performance will suffer considerably. Array problem solved. But my problem is not the low read/write performance, my problem is the high load. I start a single bittorrent download to the encoded raid3 partition with 2.5MB/s speed, and the load is 2.5. Matyee From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 09:41:46 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BAAD916A401 for ; Thu, 29 Mar 2007 09:41:46 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30312.mail.mud.yahoo.com (web30312.mail.mud.yahoo.com [209.191.69.74]) by mx1.freebsd.org (Postfix) with SMTP id 6E1EA13C458 for ; Thu, 29 Mar 2007 09:41:46 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: (qmail 71565 invoked by uid 60001); 29 Mar 2007 09:41:45 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=qgK3uH2asa0K+eIObx8tH7CZ0sTEJAysJfhVyZvg+olv0EasCRLNLqruFqpfVXcRyjKq2k7Qpt7myPn5lFZarV6lE9uHhL84KGydWpdzl9rRW2aGzL0maGwb1KqZfhWTJJuSTVjNLoj6lwRe6WWleo/p0sBZjtiL+L0y1f5Wqf4=; X-YMail-OSG: 6vGQVnAVM1nhXBfC4udlV7CgOI4aLHsXCm0SD6FduiDc7xzrJilxghHkl0U984RFzDno9MYVOca4WNn05cRazuM181M1WgfnoGtOEaAE4E8KeqasiwwFMfCpsKsOWQ-- Received: from [85.212.19.59] by web30312.mail.mud.yahoo.com via HTTP; Thu, 29 Mar 2007 02:41:45 PDT Date: Thu, 29 Mar 2007 02:41:45 -0700 (PDT) From: "R. B. Riddick" To: "Szabó" "Péter" , freebsd-geom@freebsd.org In-Reply-To: <005301c771e4$bb0a3900$6502a8c0@peteruj> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <869857.70440.qm@web30312.mail.mud.yahoo.com> Cc: Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 09:41:46 -0000 --- Szabó Péter wrote: > Array problem solved. But my problem is not the low read/write performance, > my problem is the high load. I start a single bittorrent download to the > encoded raid3 partition with 2.5MB/s speed, and the load is 2.5. > Hmm... As far as I can see it, u have the following tasks: 1. talking to the disks 2. label-related stuff 3. graid3 related stuff (splitting-up requests, XOR-ing all data) 4. gbde related stuff (encoding) 5. apllication stuff 6. network stuff You can see, how those tasks load ur CPU(-s(?)), when u use this command: top -S during this download process... The output of systat -vmstat during the download process might be quite interesting, too... You may want to do the download to another disk or to /dev/null somehow (in order to exclude the influence of graid3)... -Arne ____________________________________________________________________________________ Food fight? Enjoy some healthy debate in the Yahoo! Answers Food & Drink Q&A. http://answers.yahoo.com/dir/?link=list&sid=396545367 From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 10:22:13 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 674C916A401 for ; Thu, 29 Mar 2007 10:22:13 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 28F6A13C469 for ; Thu, 29 Mar 2007 10:22:13 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 9172E2088; Thu, 29 Mar 2007 12:22:06 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 835072085; Thu, 29 Mar 2007 12:22:06 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 7C2B8A1090; Thu, 29 Mar 2007 12:22:06 +0200 (CEST) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: =?iso-8859-1?Q?Szab=F3_P=E9ter?= References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> Date: Thu, 29 Mar 2007 12:22:06 +0200 In-Reply-To: <005301c771e4$bb0a3900$6502a8c0@peteruj> (=?iso-8859-1?Q?Szab?= =?iso-8859-1?Q?=F3_P=E9ter's?= message of "Thu, 29 Mar 2007 11:29:17 +0200") Message-ID: <86lkhg5oz5.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 10:22:13 -0000 Szab=F3 P=E9ter writes: > Array problem solved. But my problem is not the low read/write > performance, my problem is the high load. I start a single bittorrent > download to the encoded raid3 partition with 2.5MB/s speed, and the > load is 2.5. Even after the array has finished rebuilding the reconnected consumer? DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 12:33:52 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3183916A400 for ; Thu, 29 Mar 2007 12:33:52 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from mta01.mail.t-online.hu (mta01.mail.t-online.hu [195.228.240.50]) by mx1.freebsd.org (Postfix) with ESMTP id C8E2613C457 for ; Thu, 29 Mar 2007 12:33:51 +0000 (UTC) (envelope-from matyee@mail.alba.hu) Received: from peteruj (dsl51B6FBF9.fixip.t-online.hu [81.182.251.249]) by mail.axelero.hu (Postfix) with SMTP id B646333C1F4 for ; Thu, 29 Mar 2007 14:33:48 +0200 (CEST) Message-ID: <007c01c771fe$805b2fc0$6502a8c0@peteruj> From: =?ISO-8859-1?B?U3phYvMgUOl0ZXI=?= To: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj><86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> Date: Thu, 29 Mar 2007 14:33:45 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 12:33:52 -0000 I get a lot of new knowledges from the posts, thank you all. With this new knowledges i did a few tests, with very interesting results. Now the array state is COMPLETE. ----------------- Read test from the raid device (no encryption) ----------------- # dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=30000 30000+0 records in 30000+0 records out 31457280000 bytes transferred in 335.859291 secs (93662081 bytes/sec) # top -S last pid: 19340; load averages: 0.46, 1.01, 1.97 up 0+18:31:47 13:29:56 104 processes: 2 running, 89 sleeping, 13 waiting CPU states: 0.0% user, 0.0% nice, 33.2% system, 4.3% interrupt, 62.5% idle Mem: 65M Active, 568M Inact, 316M Wired, 39M Cache, 111M Buf, 10M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 11 root 1 171 52 0K 8K RUN 944:41 59.77% idle 40 root 1 -8 0 0K 8K r3:w1 34:06 19.63% g_raid3 nmivol 3 root 1 -8 0 0K 8K - 19:31 5.91% g_up 4 root 1 -8 0 0K 8K - 33:29 4.79% g_down 19271 root 1 -8 0 3400K 1772K physrd 0:07 1.56% dd ************************************ Everything seems o.k., "g_raid3 nmivol" gets ~20% of WCPU from the begining of the test to the end. The load is ~0.5 under the whole test. ------------- Read test from the encrypted partition of the raid -------------- # dd if=/dev/raid3/nmivole.bde of=/dev/null bs=1m count=30000 30000+0 records in 30000+0 records out 31457280000 bytes transferred in 1281.651229 secs (24544337 bytes/sec) # top -S last pid: 19588; load averages: 1.24, 0.89, 1.08 up 0+18:47:03 13:45:12 110 processes: 4 running, 95 sleeping, 11 waiting CPU states: 0.0% user, 0.0% nice, 88.3% system, 1.6% interrupt, 10.2% idle Mem: 74M Active, 570M Inact, 314M Wired, 37M Cache, 111M Buf, 1656K Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 15400 root 1 -8 0 0K 8K - 65:23 67.43% g_bde raid3/nmivole 11 root 1 171 52 0K 8K RUN 954:01 11.96% idle 40 root 1 -8 0 0K 8K r3:w1 34:36 4.98% g_raid3 nmivol 3 root 1 -8 0 0K 8K - 19:55 4.69% g_up 19588 root 1 96 0 9628K 9096K select 0:00 2.60% perl 4 root 1 -8 0 0K 8K - 33:41 1.51% g_down ************************************ Still o.k. but it seems, the hardware is too weak for raid3 and gbde together. "g_bde raid3/nmivole" gets ~70% WCPU, load is ~ 1.0, and that is the case under the whole test. Now the write test, and the interesting things. I can do it to a file only, i can't blast the filesystem. ------------- Write test to the encrypted partition of the raid -------------- # dd if=/dev/zero of=/mnt/vol/x.x bs=1m count=30000 13252+0 records in 13251+0 records out 13894680576 bytes transferred in 1595.827345 secs (8706882 bytes/sec) >> At the begining # top -S last pid: 19972; load averages: 1.04, 0.50, 0.66 up 0+19:10:26 14:08:35 104 processes: 3 running, 89 sleeping, 12 waiting CPU states: 0.0% user, 0.0% nice, 70.8% system, 0.8% interrupt, 28.4% idle Mem: 65M Active, 679M Inact, 196M Wired, 22M Cache, 111M Buf, 35M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 11 root 1 171 52 0K 8K RUN 963:45 52.15% idle 15400 root 1 -8 0 0K 8K - 75:58 24.27% g_bde raid3/nmivole 3 root 1 -8 0 0K 8K - 20:49 4.88% g_up 19970 root 1 -16 0 3400K 1772K wdrain 0:04 3.29% dd 4 root 1 -8 0 0K 8K - 34:07 2.29% g_down 40 root 1 -8 0 0K 8K r3:w1 35:33 2.05% g_raid3 nmivol It seems fine, but the load 1.0, i think is a litle bit high. gbde gets only ~30% of WCPU and g_down gets ~3%. I don't know what is the task of g_down. >> After a few minutes # top -S last pid: 20351; load averages: 3.23, 2.84, 2.24 up 0+19:34:37 14:32:46 104 processes: 4 running, 88 sleeping, 12 waiting CPU states: 0.0% user, 0.0% nice, 83.6% system, 1.6% interrupt, 14.8% idle Mem: 65M Active, 611M Inact, 266M Wired, 53M Cache, 111M Buf, 1080K Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 4 root 1 -8 0 0K 8K - 42:47 53.27% g_down 15400 root 1 -8 0 0K 8K - 82:18 19.38% g_bde raid3/nmivole 11 root 1 171 52 0K 8K RUN 968:44 12.60% idle 3 root 1 -8 0 0K 8K - 22:12 3.03% g_up 19970 root 1 -16 0 3400K 1772K wdrain 1:12 2.93% dd 40 root 1 -8 0 0K 8K r3:w1 36:18 1.07% g_raid3 nmivol ************************************ At this point i stop the test, before the copmuter goes unreachable. It seems terible!!! g_down gets more and more and more WCPU, and the load goes to the skies. I don't know, what should i think. Now i can see, at the end i will change the raid3 to stripe ;) But i don't want to :( If somebody have an idea, please post it! Matyee ----- Original Message ----- From: "Dag-Erling "Smørgrav"" Szabó Péter writes: > Array problem solved. But my problem is not the low read/write > performance, my problem is the high load. I start a single bittorrent > download to the encoded raid3 partition with 2.5MB/s speed, and the > load is 2.5. Even after the array has finished rebuilding the reconnected consumer? DES -- Dag-Erling Smørgrav - des@des.no From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 12:42:56 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 20A9516A400 for ; Thu, 29 Mar 2007 12:42:56 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A482F13C459 for ; Thu, 29 Mar 2007 12:42:55 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HWtxn-0002xQ-I0 for freebsd-geom@freebsd.org; Thu, 29 Mar 2007 14:42:47 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 Mar 2007 14:42:47 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 Mar 2007 14:42:47 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Ivan Voras Date: Thu, 29 Mar 2007 14:42:32 +0200 Lines: 52 Message-ID: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj><86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3681B5D0626F389387AA5E67" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <007c01c771fe$805b2fc0$6502a8c0@peteruj> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 12:42:56 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3681B5D0626F389387AA5E67 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Szab=C3=B3 P=C3=A9ter wrote: >>> After a few minutes > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 4 root 1 -8 0 0K 8K - 42:47 53.27% g_down > 15400 root 1 -8 0 0K 8K - 82:18 19.38% g_bde=20 > raid3/nmivole > 11 root 1 171 52 0K 8K RUN 968:44 12.60% idle > 3 root 1 -8 0 0K 8K - 22:12 3.03% g_up > 19970 root 1 -16 0 3400K 1772K wdrain 1:12 2.93% dd > 40 root 1 -8 0 0K 8K r3:w1 36:18 1.07% g_raid3 = > nmivol > It seems terible!!! g_down gets more and more and more WCPU, and the=20 > load goes to the skies. I don't know, what should i think. Now i can=20 > see, at the end i will change the raid3 to stripe ;) But i don't want t= o :( Also one of the "shouldn't happen" things - g_down is (roughly) the=20 kernel thread that dispatches IO requests from the file system to the=20 lower level devices, gbde and graid3 in your case. It's strange that it=20 consumes that much CPU - it may mean that one of the GEOM classes is=20 being unusually slow in offloading requests to its own thread, or that=20 there's an unusual number of small requests bouncing between them. Try=20 running 'gstat' during this test and post some results. If you have the time, also try using geli instead of gbde, just to=20 experiment. --------------enig3681B5D0626F389387AA5E67 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGC7Q4ldnAQVacBcgRArf5AJ9IPeJ/9y5c1XyFEH4GSr2nzfPDlQCcDaM+ /aH0Btx3xWPBMFmsjFnAq5Y= =DivZ -----END PGP SIGNATURE----- --------------enig3681B5D0626F389387AA5E67-- From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 13:10:29 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A36816A401 for ; Thu, 29 Mar 2007 13:10:29 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id CD7C013C489 for ; Thu, 29 Mar 2007 13:10:28 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 808572085; Thu, 29 Mar 2007 15:10:15 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id E7A9E2084; Thu, 29 Mar 2007 15:10:14 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id DEA66A1073; Thu, 29 Mar 2007 15:10:14 +0200 (CEST) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: =?iso-8859-1?Q?Szab=F3_P=E9ter?= References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> Date: Thu, 29 Mar 2007 15:10:14 +0200 In-Reply-To: <007c01c771fe$805b2fc0$6502a8c0@peteruj> (=?iso-8859-1?Q?Szab?= =?iso-8859-1?Q?=F3_P=E9ter's?= message of "Thu, 29 Mar 2007 14:33:45 +0200") Message-ID: <86odmc42mh.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 13:10:29 -0000 Szab=F3 P=E9ter writes: > It seems fine, but the load 1.0, i think is a litle bit high. gbde > gets only ~30% of WCPU and g_down gets ~3%. I don't know what is the > task of g_down. You don't seem to understand what the load averages mean. They are the average number of runnable threads in the scheduler queue over the last one, five and fifteen seconds. Certain workloads will drive up the load averages without consuming all available CPU time. This is particularly the case for workloads where small chunks of data (e.g. RAID3 stripes) are passed around between multiple threads. FYI, g_up and g_down are the threads responsible for passing data up and down the GEOM stack, respectively. For further details, see for instance . DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 14:29:33 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 03D2116A404 for ; Thu, 29 Mar 2007 14:29:33 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id CB4E713C48A for ; Thu, 29 Mar 2007 14:29:32 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2TETRrH042400; Thu, 29 Mar 2007 09:29:27 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460BCD47.3070702@freebsd.org> Date: Thu, 29 Mar 2007 09:29:27 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> <86odmc42mh.fsf@dwp.des.no> In-Reply-To: <86odmc42mh.fsf@dwp.des.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/2959/Thu Mar 29 04:32:45 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 14:29:33 -0000 On 03/29/07 08:10, Dag-Erling Smørgrav wrote: > Szabó Péter writes: >> It seems fine, but the load 1.0, i think is a litle bit high. gbde >> gets only ~30% of WCPU and g_down gets ~3%. I don't know what is the >> task of g_down. > > You don't seem to understand what the load averages mean. They are > the average number of runnable threads in the scheduler queue over the > last one, five and fifteen seconds. Are you sure it isn't over the last 1, 5, and 15 *minutes*? :) Eric From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 14:52:28 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 13EE716A401 for ; Thu, 29 Mar 2007 14:52:28 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id C0EB113C4B0 for ; Thu, 29 Mar 2007 14:52:27 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id B50EC2088; Thu, 29 Mar 2007 16:52:18 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 3569B2086; Thu, 29 Mar 2007 16:52:18 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 3BFC0A1073; Thu, 29 Mar 2007 16:52:18 +0200 (CEST) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: Eric Anderson References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> <86odmc42mh.fsf@dwp.des.no> <460BCD47.3070702@freebsd.org> Date: Thu, 29 Mar 2007 16:52:18 +0200 In-Reply-To: <460BCD47.3070702@freebsd.org> (Eric Anderson's message of "Thu, 29 Mar 2007 09:29:27 -0500") Message-ID: <86ps6s2jbx.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 14:52:28 -0000 Eric Anderson writes: > On 03/29/07 08:10, Dag-Erling Sm=F8rgrav wrote: > > You don't seem to understand what the load averages mean. They are > > the average number of runnable threads in the scheduler queue over the > > last one, five and fifteen seconds. > Are you sure it isn't over the last 1, 5, and 15 *minutes*? You're right, of course. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 15:53:22 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7166A16A40E for ; Thu, 29 Mar 2007 15:53:22 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 0082913C489 for ; Thu, 29 Mar 2007 15:53:21 +0000 (UTC) (envelope-from gcubfg-freebsd-geom@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HWww7-0008AQ-Vf for freebsd-geom@freebsd.org; Thu, 29 Mar 2007 17:53:16 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 Mar 2007 17:53:15 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 29 Mar 2007 17:53:15 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-geom@freebsd.org From: Ivan Voras Date: Thu, 29 Mar 2007 17:52:40 +0200 Lines: 36 Message-ID: References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> <86odmc42mh.fsf@dwp.des.no> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA8F5CA0ABA6B1DA375705187" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <86odmc42mh.fsf@dwp.des.no> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 15:53:22 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA8F5CA0ABA6B1DA375705187 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Dag-Erling Sm=C3=B8rgrav wrote: > You don't seem to understand what the load averages mean. They are > the average number of runnable threads in the scheduler queue over the > last one, five and fifteen seconds. Certain workloads will drive up > the load averages without consuming all available CPU time. This is > particularly the case for workloads where small chunks of data (e.g. > RAID3 stripes) are passed around between multiple threads. But, in his case the threads DO seem to consume much more CPU time than=20 they should - especially the g_down thread. In this case, load avg ( as=20 an approximation of real system load) is useful, and he's not concerned=20 without cause. --------------enigA8F5CA0ABA6B1DA375705187 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGC+DPldnAQVacBcgRAvxgAKDZGc66iiLpsjuTVmBGWaWllwf8lgCg0J2Z o/1l9Lm/s5HfOSATilgR+mc= =tXRE -----END PGP SIGNATURE----- --------------enigA8F5CA0ABA6B1DA375705187-- From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 15:58:32 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B4F8F16A401 for ; Thu, 29 Mar 2007 15:58:32 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 86CC613C448 for ; Thu, 29 Mar 2007 15:58:32 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l2TFwMio058650; Thu, 29 Mar 2007 10:58:22 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <460BE21E.7070700@freebsd.org> Date: Thu, 29 Mar 2007 10:58:22 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.10 (X11/20070320) MIME-Version: 1.0 To: Ivan Voras References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> <86odmc42mh.fsf@dwp.des.no> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/2961/Thu Mar 29 09:06:01 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 15:58:32 -0000 On 03/29/07 10:52, Ivan Voras wrote: > Dag-Erling Smørgrav wrote: > >> You don't seem to understand what the load averages mean. They are >> the average number of runnable threads in the scheduler queue over the >> last one, five and fifteen seconds. Certain workloads will drive up >> the load averages without consuming all available CPU time. This is >> particularly the case for workloads where small chunks of data (e.g. >> RAID3 stripes) are passed around between multiple threads. > > But, in his case the threads DO seem to consume much more CPU time than > they should - especially the g_down thread. In this case, load avg ( as > an approximation of real system load) is useful, and he's not concerned > without cause. > > It would be good to see output of a ps -auxl next time he's running the test. Eric From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 16:18:23 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E2AA916A401; Thu, 29 Mar 2007 16:18:23 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 72DCD13C44B; Thu, 29 Mar 2007 16:18:23 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id A982448805; Thu, 29 Mar 2007 18:18:21 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 76BD6487F2; Thu, 29 Mar 2007 18:18:11 +0200 (CEST) Date: Thu, 29 Mar 2007 18:18:07 +0200 From: Pawel Jakub Dawidek To: Eric Anderson Message-ID: <20070329161807.GB3325@garage.freebsd.pl> References: <003401c7712a$f71ebb60$6502a8c0@peteruj> <005c01c77134$28e0fce0$6502a8c0@peteruj> <86zm5xph7o.fsf@dwp.des.no> <005301c771e4$bb0a3900$6502a8c0@peteruj> <86lkhg5oz5.fsf@dwp.des.no> <007c01c771fe$805b2fc0$6502a8c0@peteruj> <86odmc42mh.fsf@dwp.des.no> <460BE21E.7070700@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hHWLQfXTYDoKhP50" Content-Disposition: inline In-Reply-To: <460BE21E.7070700@freebsd.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: Ivan Voras , freebsd-geom@freebsd.org Subject: Re: raid3 is slow X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 16:18:24 -0000 --hHWLQfXTYDoKhP50 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 29, 2007 at 10:58:22AM -0500, Eric Anderson wrote: > On 03/29/07 10:52, Ivan Voras wrote: > >Dag-Erling Sm?=B8rgrav wrote: > >>You don't seem to understand what the load averages mean. They are > >>the average number of runnable threads in the scheduler queue over the > >>last one, five and fifteen seconds. Certain workloads will drive up > >>the load averages without consuming all available CPU time. This is > >>particularly the case for workloads where small chunks of data (e.g. > >>RAID3 stripes) are passed around between multiple threads. > >But, in his case the threads DO seem to consume much more CPU time than = they should - especially the g_down thread. In this case, load avg ( as an = approximation of real=20 > >system load) is useful, and he's not concerned without cause. >=20 >=20 > It would be good to see output of a ps -auxl next time he's running the t= est. Maybe I missed it, but I think we should start from asking about CPU he uses:) Full dmesg would be best. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --hHWLQfXTYDoKhP50 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFGC+a/ForvXbEpPzQRArH9AKDdbz9+hbDpbQgi+WBJJiHmi/mjJwCg05eS BaNOgfF7v/hUTqzRYgHsLl8= =awLv -----END PGP SIGNATURE----- --hHWLQfXTYDoKhP50-- From owner-freebsd-geom@FreeBSD.ORG Thu Mar 29 23:47:45 2007 Return-Path: X-Original-To: freebsd-geom@FreeBSD.org Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F005016A400; Thu, 29 Mar 2007 23:47:45 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from pinus.cc.fer.hr (pinus.cc.fer.hr [161.53.73.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7FB5513C483; Thu, 29 Mar 2007 23:47:45 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from [161.53.72.113] (lara.cc.fer.hr [161.53.72.113]) by pinus.cc.fer.hr (8.12.2/8.12.2) with ESMTP id l2SD5cLT020376; Wed, 28 Mar 2007 15:05:39 +0200 (MEST) Message-ID: <460A6648.1050609@fer.hr> Date: Wed, 28 Mar 2007 14:57:44 +0200 From: Ivan Voras User-Agent: Thunderbird 1.5.0.10 (X11/20060911) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <460A6075.7000302@freebsd.org> <20070328124832.GB35749@garage.freebsd.pl> In-Reply-To: <20070328124832.GB35749@garage.freebsd.pl> X-Enigmail-Version: 0.94.2.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6590A2FCCAB371FB9CE7F096" Cc: freebsd-geom@FreeBSD.org Subject: Re: Geom_label and multiple devices X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 23:47:46 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6590A2FCCAB371FB9CE7F096 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Pawel Jakub Dawidek wrote: > 2. Stop the system (panic?), because it's just administrator fault > (misconfiguration). >=20 > I'd prefer 2, but it's too cruel. Administrators make mistakes and we > have some garbage to clean still (eg. 'c' partition).=20 =2E.. and people using multipath would be angry :) > I like this idea, but it can only be used for glabel's native labels. > You can't use the same method for filesystem volume names, and those ar= e > more often used than glabel's native labels. Well, there's always the "dirty" way, by embedding "hidden" strings like = "#0" in the labels. --------------enig6590A2FCCAB371FB9CE7F096 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGCmZIldnAQVacBcgRAna1AKDa3J7CaxsehzYeDH9r7mLGY1XApgCfS2UV M8bC5iff0ZmGauxAcIMfqPg= =HfiQ -----END PGP SIGNATURE----- --------------enig6590A2FCCAB371FB9CE7F096-- From owner-freebsd-geom@FreeBSD.ORG Sat Mar 31 19:37:59 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 03DA016A408 for ; Sat, 31 Mar 2007 19:37:59 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 4BC8213C4AD for ; Sat, 31 Mar 2007 19:37:58 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 89034 invoked by uid 2001); 31 Mar 2007 19:11:16 -0000 Date: Sat, 31 Mar 2007 13:11:16 -0600 From: "Rick C. Petty" To: freebsd-geom@freebsd.org Message-ID: <20070331191116.GA87710@keira.kiwi-computer.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: gvinum rebuildparity breakage X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 19:37:59 -0000 Due to ata driver problems, one of the disks which holds part of my RAID5 volumes was dropped (see below). gvinum did the right thing and reported the drive lost and all the relevent subdisks in my RAID5 plexes were changed to degraded. Problem 1). The ata driver would not bring up the dropped drive. Numerous attempts at "atacontrol reinit" and detach/attach sequences would not bring the drive back. I've reported this problem in the past with this Promise PDC40718 SATA300 controller card and have not seen or heard of a resolution. My guess is the ata folks blame it on the Promise chip, fine. Problem 2). The geom subsystem continued to spit out error messages yet fail to allow the volumes be unmounted. I've yet to witness geom do anything gracefully such as allow the mountpoints to be forcibly unmounted (without panicking) or perform a shutdown without dirty buffers. Fine. Problems 1 & 2 required a reboot; so much for my 115-day uptime. I'll accept this, because I expect gvinum to do the Right Thing... which brings me to: Problem 3). I have four volumes which have degraded RAID5 plexes, as is expected from the dropped drive. However the drive is up now but gvinum won't let me rebuild the parity: # gvinum rebuildparity music.p0 gvinum: plex music.p0 is not completely accessible What a strange message, unless it's referring to the stale subdisk, but it won't let me do anything about that either: # gvinum start music.p0.s0 gvinum: can't start: cannot start 'music.p0.s0' - not yet supported I'm not keen to try forcibly setting the state with "gvinum setstate -f" because: the man page states this is for diagnostic purposes only, and the man page doesn't give a list of possible states anyway. This documentation has been lacking since the 5.x nerf of vinum/gvinum anyway so the man page and "gvinum help" are quite useless (Problem #4). I'd rather not experiment with this command since the volumes are important, so I'm left without options. So, how do I rebuild by RAID5 volumes? This used to work fine in pre-geom vinum and I'm pretty sure this worked in gvinum at one point. I'm running 6.1-STABLE (RELENG_6 as of 2006-Jul-21) with a GENERIC kernel. Here is my vinum configuration: # gvinum l 8 drives: D k0 State: up /dev/ad18 A: 92253/305245 MB (30%) D k3 State: up /dev/ad16 A: 0/305245 MB (0%) D k1 State: up /dev/ad14 A: 92253/305245 MB (30%) D k2 State: up /dev/ad12 A: 0/305245 MB (0%) D k4 State: up /dev/ad10 A: 77036/286188 MB (26%) D k7 State: up /dev/ad8 A: 77036/286188 MB (26%) D k5 State: up /dev/ad6 A: 77036/286188 MB (26%) D k6 State: up /dev/ad4 A: 77036/286188 MB (26%) 10 volumes: V extra State: up Plexes: 1 Size: 524 GB V 3rd-party State: up Plexes: 2 Size: 16 GB V home State: up Plexes: 2 Size: 32 GB V projects State: up Plexes: 2 Size: 4096 MB V bsd State: up Plexes: 2 Size: 128 GB V media State: up Plexes: 2 Size: 64 GB V picts State: up Plexes: 1 Size: 768 MB V music State: up Plexes: 1 Size: 120 GB V flac State: up Plexes: 1 Size: 192 GB V video State: up Plexes: 1 Size: 300 GB 15 plexes: P extra.p0 S State: up Subdisks: 2 Size: 524 GB P 3rd-party.p0 C State: up Subdisks: 1 Size: 16 GB P 3rd-party.p1 C State: up Subdisks: 1 Size: 16 GB P home.p1 C State: up Subdisks: 1 Size: 32 GB P home.p0 C State: up Subdisks: 1 Size: 32 GB P projects.p1 C State: up Subdisks: 1 Size: 4096 MB P projects.p0 C State: up Subdisks: 1 Size: 4096 MB P bsd.p1 C State: up Subdisks: 1 Size: 128 GB P bsd.p0 C State: up Subdisks: 1 Size: 128 GB P media.p1 C State: up Subdisks: 1 Size: 64 GB P media.p0 C State: up Subdisks: 1 Size: 64 GB P picts.p0 R5 State: degraded Subdisks: 4 Size: 768 MB P music.p0 R5 State: degraded Subdisks: 4 Size: 120 GB P flac.p0 R5 State: degraded Subdisks: 4 Size: 192 GB P video.p0 R5 State: degraded Subdisks: 4 Size: 300 GB 28 subdisks: S extra.p0.s1 State: up D: k3 Size: 262 GB S extra.p0.s0 State: up D: k2 Size: 262 GB S 3rd-party.p0.s0 State: up D: k0 Size: 16 GB S 3rd-party.p1.s0 State: up D: k1 Size: 16 GB S home.p1.s0 State: up D: k3 Size: 32 GB S home.p0.s0 State: up D: k2 Size: 32 GB S projects.p1.s0 State: up D: k3 Size: 4096 MB S projects.p0.s0 State: up D: k2 Size: 4096 MB S bsd.p1.s0 State: up D: k1 Size: 128 GB S bsd.p0.s0 State: up D: k0 Size: 128 GB S media.p1.s0 State: up D: k1 Size: 64 GB S media.p0.s0 State: up D: k0 Size: 64 GB S picts.p0.s0 State: stale D: k4 Size: 256 MB S picts.p0.s1 State: up D: k5 Size: 256 MB S picts.p0.s2 State: up D: k6 Size: 256 MB S picts.p0.s3 State: up D: k7 Size: 256 MB S music.p0.s0 State: stale D: k4 Size: 40 GB S music.p0.s1 State: up D: k5 Size: 40 GB S music.p0.s2 State: up D: k6 Size: 40 GB S music.p0.s3 State: up D: k7 Size: 40 GB S flac.p0.s3 State: up D: k7 Size: 64 GB S flac.p0.s2 State: up D: k6 Size: 64 GB S flac.p0.s1 State: up D: k5 Size: 64 GB S flac.p0.s0 State: stale D: k4 Size: 64 GB S video.p0.s3 State: up D: k7 Size: 100 GB S video.p0.s2 State: up D: k6 Size: 100 GB S video.p0.s1 State: up D: k5 Size: 100 GB S video.p0.s0 State: stale D: k4 Size: 100 GB Although not relevant for Problem #3, here is a history of the controller card failure: First one of the raid5 drives had a fatal read failure, causing that drive to be dropped: Mar 31 04:25:38 kay kernel: ad10: WARNING - READ_DMA UDMA ICRC error (retrying request) LBA=267860265 Mar 31 04:25:47 kay kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly Mar 31 04:25:51 kay kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly Mar 31 04:25:55 kay kernel: ad10: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly Mar 31 04:25:59 kay kernel: ad10: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly Mar 31 04:26:03 kay kernel: ad10: WARNING - SET_MULTI taskqueue timeout - completing request directly Mar 31 04:26:03 kay kernel: ad10: TIMEOUT - READ_DMA retrying (0 retries left) LBA=267860265 Mar 31 04:26:05 kay kernel: ad10: FAILURE - READ_DMA status=ff error=ff LBA=267860265 Mar 31 04:26:05 kay kernel: GEOM_VINUM: subdisk picts.p0.s0 state change: up -> down Mar 31 04:26:05 kay kernel: GEOM_VINUM: plex picts.p0 state change: up -> degraded Mar 31 04:26:05 kay kernel: GEOM_VINUM: subdisk music.p0.s0 state change: up -> down Mar 31 04:26:05 kay kernel: GEOM_VINUM: plex music.p0 state change: up -> degraded Mar 31 04:26:05 kay kernel: GEOM_VINUM: subdisk flac.p0.s0 state change: up -> down Mar 31 04:26:05 kay kernel: GEOM_VINUM: plex flac.p0 state change: up -> degraded Mar 31 04:26:05 kay kernel: GEOM_VINUM: subdisk video.p0.s0 state change: up -> down Mar 31 04:26:05 kay kernel: GEOM_VINUM: plex video.p0 state change: up -> degraded Mar 31 04:26:05 kay kernel: g_vfs_done():gvinum/video[WRITE(offset=75620171776, length=131072)]error = 5 Mar 31 04:26:05 kay kernel: g_vfs_done():gvinum/video[WRITE(offset=75620302848, length=131072)]error = 6 Mar 31 04:26:05 kay kernel: g_vfs_done():gvinum/video[WRITE(offset=75621220352, length=131072)]error = 6 Mar 31 04:26:05 kay kernel: g_vfs_done():gvinum/video[WRITE(offset=75620433920, length=131072)]error = 6 ... [continued similar messages] Mar 31 04:26:10 kay kernel: ad10: TIMEOUT - READ_DMA retrying (1 retry left) LBA=268002601 Mar 31 04:26:10 kay kernel: GEOM_VINUM: lost drive 'k4' A few hundred of the ENXIO (error = 6) kernel messages messages happen. Notice the ata subsystem doesn't report that ad10 was ever detached. It was just unresponsive and a reinit blew away the drive for good. Later, a drive not part of the RAID5 but part of some mirrors and a slice decides to wig out, causing the ata subsystem to drop the drive forever until a reboot (Problem #1): Mar 31 04:53:50 kay kernel: ad12: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly Mar 31 04:53:54 kay kernel: ad12: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly Mar 31 04:53:58 kay kernel: ad12: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly Mar 31 04:54:02 kay kernel: ad12: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly Mar 31 04:54:06 kay kernel: ad12: WARNING - SET_MULTI taskqueue timeout - completing request directly Mar 31 04:54:06 kay kernel: ad12: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=497537109 Mar 31 04:54:21 kay kernel: ad12: FAILURE - device detached Mar 31 04:54:21 kay kernel: subdisk12: detached Mar 31 04:54:21 kay kernel: ad12: detached Which in turn causes the expected: Mar 31 04:54:21 kay kernel: GEOM_VINUM: subdisk projects.p0.s0 state change: up -> down Mar 31 04:54:21 kay kernel: GEOM_VINUM: plex projects.p0 state change: up -> down Mar 31 04:54:21 kay kernel: GEOM_VINUM: subdisk home.p0.s0 state change: up -> down Mar 31 04:54:21 kay kernel: GEOM_VINUM: plex home.p0 state change: up -> down Mar 31 04:54:21 kay kernel: GEOM_VINUM: subdisk extra.p0.s0 state change: up -> down Mar 31 04:54:21 kay kernel: GEOM_VINUM: plex extra.p0 state change: up -> down Mar 31 04:54:21 kay kernel: g_vfs_done():gvinum/extra[READ(offset=432168124416, length=131072)]error = 6 Mar 31 04:54:21 kay kernel: g_vfs_done():gvinum/extra[READ(offset=432168124416, length=65536)]error = 6 Mar 31 04:54:21 kay kernel: g_vfs_done():gvinum/extra[READ(offset=432168189952, length=131072)]error = 6 Mar 31 04:54:21 kay kernel: GEOM_VINUM: g_access failed on drive k2, errno 6 Mar 31 04:54:21 kay kernel: g_vfs_done():gvinum/extra[READ(offset=95073779712, length=2048)]error = 6 Mar 31 04:54:21 kay last message repeated 4 times Mar 31 04:54:21 kay kernel: GEOM_VINUM: lost drive 'k2' Followed by a the failure of the geom subsystem to ever properly deal with the problematic filesystem until forcibly rebooted (Problem 2): Mar 31 04:54:50 kay kernel: g_vfs_done():gvinum/extra[WRITE(offset=94997381120, length=16384)]error = 6 Mar 31 04:55:19 kay kernel: g_vfs_done():gvinum/extra[WRITE(offset=94997381120, length=16384)]error = 6 Mar 31 04:57:15 kay last message repeated 4 times Mar 31 05:06:55 kay last message repeated 20 times Mar 31 05:17:05 kay last message repeated 21 times Mar 31 05:27:15 kay last message repeated 21 times Mar 31 05:36:56 kay last message repeated 20 times Mar 31 05:47:06 kay last message repeated 21 times Mar 31 05:57:15 kay last message repeated 21 times Mar 31 06:06:56 kay last message repeated 20 times Mar 31 06:17:06 kay last message repeated 21 times Mar 31 06:27:16 kay last message repeated 21 times Mar 31 06:36:57 kay last message repeated 20 times Mar 31 06:47:06 kay last message repeated 21 times Mar 31 06:57:16 kay last message repeated 21 times Mar 31 07:06:27 kay last message repeated 19 times Mar 31 07:06:34 kay kernel: g_vfs_done():gvinum/extra[READ(offset=95073779712, length=2048)]error = 6 Mar 31 07:06:57 kay kernel: g_vfs_done():gvinum/extra[WRITE(offset=94997381120, length=16384)]error = 6 Mar 31 07:07:26 kay kernel: g_vfs_done():gvinum/extra[WRITE(offset=94997381120, length=16384)]error = 6 Mar 31 07:09:51 kay last message repeated 5 times Mar 31 07:19:32 kay last message repeated 20 times Mar 31 07:29:42 kay last message repeated 21 times Mar 31 07:39:23 kay last message repeated 20 times Mar 31 07:49:34 kay last message repeated 21 times Mar 31 07:59:43 kay last message repeated 21 times Mar 31 08:09:23 kay last message repeated 20 times Mar 31 08:19:33 kay last message repeated 21 times Mar 31 08:29:43 kay last message repeated 21 times Mar 31 08:39:25 kay last message repeated 20 times Mar 31 08:49:35 kay last message repeated 21 times Mar 31 08:59:45 kay last message repeated 21 times Mar 31 09:09:26 kay last message repeated 20 times Mar 31 09:19:35 kay last message repeated 21 times Mar 31 09:29:46 kay last message repeated 21 times Mar 31 09:39:27 kay last message repeated 20 times Mar 31 09:49:37 kay last message repeated 21 times Mar 31 09:59:47 kay last message repeated 21 times Mar 31 10:09:28 kay last message repeated 20 times Mar 31 10:19:39 kay last message repeated 21 times Mar 31 10:29:49 kay last message repeated 21 times Mar 31 10:39:30 kay last message repeated 20 times Mar 31 10:49:40 kay last message repeated 21 times Mar 31 10:59:51 kay last message repeated 21 times Mar 31 11:09:32 kay last message repeated 20 times Mar 31 11:19:42 kay last message repeated 21 times Mar 31 11:29:23 kay last message repeated 20 times Mar 31 11:39:32 kay last message repeated 21 times Mar 31 11:49:42 kay last message repeated 21 times Mar 31 11:54:23 kay last message repeated 9 times Mar 31 11:54:27 kay syslogd: exiting on signal 15 -- Rick C. Petty