From owner-freebsd-fs@FreeBSD.ORG  Tue Aug 30 23:00:43 2011
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A0597106566B;
	Tue, 30 Aug 2011 23:00:43 +0000 (UTC)
	(envelope-from mckusick@mckusick.com)
Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235])
	by mx1.freebsd.org (Postfix) with ESMTP id 82F168FC14;
	Tue, 30 Aug 2011 23:00:43 +0000 (UTC)
Received: from chez.mckusick.com (localhost [127.0.0.1])
	by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id p7UN0jJ6022811;
	Tue, 30 Aug 2011 16:00:45 -0700 (PDT)
	(envelope-from mckusick@chez.mckusick.com)
Message-Id: <201108302300.p7UN0jJ6022811@chez.mckusick.com>
To: lev@FreeBSD.org
In-reply-to: <1693072185.20110831022933@serebryakov.spb.ru> 
Date: Tue, 30 Aug 2011 16:00:45 -0700
From: Kirk McKusick <mckusick@mckusick.com>
X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY
	autolearn=failed version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com
Cc: freebsd-fs@FreeBSD.org
Subject: Re: Very inconsistent (read) speed on UFS2 
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Aug 2011 23:00:43 -0000

> Date: Wed, 31 Aug 2011 02:29:33 +0400
> From: Lev Serebryakov <lev@FreeBSD.org>
> To: Kirk McKusick <mckusick@mckusick.com>, freebsd-fs@FreeBSD.org
> Subject: Re: Very inconsistent (read) speed on UFS2
> 
> Hello, Kirk.
> 
> >   I'll try this experiment with mmap() and touching every 4096-th byte of
> > mapped memory instead of read(2).
> 
>   Strange enough, it gives only 40-50MiB/s and results are very
>  consistent.
> 
>   It really surprise me. I didn't think, that there will be so much
> difference, I was sure, that it will be almost equivalent speed.
> 
> --
> // Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>

I had not realized that you were using O_DIRECT. That would in fact
avoid most of the caching / memory-recovery effects that I was blaming
earlier. Your test above is definitely hitting them though. My guess
is that the consistency is because you are measuring the rate at which
free memory can be created.

So, my new theory on why your O_DIRECT test is running slowly is due
to the single threading in the GEOM layer. Pawel Jakub Dawidek (pjd@)
gave a very interesting talk on this problem at this year's BSDCan.

	Kirk McKusick