From owner-freebsd-stable@FreeBSD.ORG Sun Jan 10 01:06:33 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5CAB106566B for ; Sun, 10 Jan 2010 01:06:33 +0000 (UTC) (envelope-from garrettmoore@gmail.com) Received: from mail-iw0-f198.google.com (mail-iw0-f198.google.com [209.85.223.198]) by mx1.freebsd.org (Postfix) with ESMTP id 66F718FC0C for ; Sun, 10 Jan 2010 01:06:33 +0000 (UTC) Received: by iwn36 with SMTP id 36so13882154iwn.3 for ; Sat, 09 Jan 2010 17:06:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=lYx1PZbKv6PP/s3jsO9pnyiJlZ+vWl0yBcREHgh4FDs=; b=XEPeKo8kR+at9McPBa3CaWke/97Jrum9Sn0nUh3n3XWc9NZqB3YgZK5uPSr6Ti9ZzY 7Rilpf7mqbG2JCbirIjefeydZMJfSnOECcALRFjDyv0STMleEb+gLRlTS7nby/HelN5l SQa9heb24SYFeivg+7fo7a9wxUBuxQHzJYIKQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=YmnDdZA6YinpeYWghKCWdAf+O2bRGS4K/t0YNO2lK2HQCWurHiKlINW46R3XGp1Zvt MtJrklmrzFPezuRstkmzLpM9+wUpaEOMVJEP68pTHfZTAdYKpoSTXHRHN0l6//vi6Owu lGeWd/wdkFfqc3WTuzUAHCFs+AwHsN56Qz7GU= MIME-Version: 1.0 Received: by 10.231.120.136 with SMTP id d8mr2145175ibr.14.1263085590313; Sat, 09 Jan 2010 17:06:30 -0800 (PST) In-Reply-To: <20100110005621.GA45015@icarus.home.lan> References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <7346c5c61001080831w375d158fu5b1996ee58cb0f8d@mail.gmail.com> <2ae8edf31001091051l156cb57alf549cfe06f1c7197@mail.gmail.com> <7346c5c61001091640w1a0ffb5y3ffe97cb88c76436@mail.gmail.com> <20100110005621.GA45015@icarus.home.lan> Date: Sat, 9 Jan 2010 20:06:30 -0500 Message-ID: <7346c5c61001091706m45a3a2a5k3ca8bb0c4bec5ea8@mail.gmail.com> From: Garrett Moore To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS performance degradation over time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 01:06:33 -0000 Sure: [root@leviathan ~]# sysctl kstat.zfs.misc.arcstats kstat.zfs.misc.arcstats.hits: 32092629 kstat.zfs.misc.arcstats.misses: 1064835 kstat.zfs.misc.arcstats.demand_data_hits: 30542262 kstat.zfs.misc.arcstats.demand_data_misses: 848959 kstat.zfs.misc.arcstats.demand_metadata_hits: 1550367 kstat.zfs.misc.arcstats.demand_metadata_misses: 215876 kstat.zfs.misc.arcstats.prefetch_data_hits: 0 kstat.zfs.misc.arcstats.prefetch_data_misses: 0 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0 kstat.zfs.misc.arcstats.mru_hits: 18329884 kstat.zfs.misc.arcstats.mru_ghost_hits: 114483 kstat.zfs.misc.arcstats.mfu_hits: 13762745 kstat.zfs.misc.arcstats.mfu_ghost_hits: 172573 kstat.zfs.misc.arcstats.deleted: 1735926 kstat.zfs.misc.arcstats.recycle_miss: 2076926 kstat.zfs.misc.arcstats.mutex_miss: 545 kstat.zfs.misc.arcstats.evict_skip: 532474 kstat.zfs.misc.arcstats.hash_elements: 6784 kstat.zfs.misc.arcstats.hash_elements_max: 14351 kstat.zfs.misc.arcstats.hash_collisions: 149862 kstat.zfs.misc.arcstats.hash_chains: 338 kstat.zfs.misc.arcstats.hash_chain_max: 4 kstat.zfs.misc.arcstats.p: 25819136 kstat.zfs.misc.arcstats.c: 107609280 kstat.zfs.misc.arcstats.c_min: 107609280 kstat.zfs.misc.arcstats.c_max: 860874240 kstat.zfs.misc.arcstats.size: 40148272 kstat.zfs.misc.arcstats.hdr_size: 1411072 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 10349 Additionally, from top: Mem: 2869M Active, 440M Inact, 479M Wired, 91M Cache, 418M Buf, 63M Free Getting <20MB/s combined read/write at this point (ie copying files from one directory to another, both source and destination on the zpool). On Sat, Jan 9, 2010 at 7:56 PM, Jeremy Chadwick wrote: > On Sat, Jan 09, 2010 at 07:40:53PM -0500, Garrett Moore wrote: > > If no-one has any questions, I'll try Artem's suggestion of wasting a > bunch > > of memory in Perl/Python and forcing some memory to be swapped out. (I > don't > > want to do it yet in case someone wants a specific number before I do > that). > > It would help if you could provide the entire output from: > > sysctl kstat.zfs.misc.arcstats > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP: 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >