From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 12:31:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7E35FF5 for ; Thu, 10 Jul 2014 12:31:05 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E65D255F for ; Thu, 10 Jul 2014 12:31:05 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hi2so4439411wib.7 for ; Thu, 10 Jul 2014 05:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=KBUCDl8GqdzX7EAaa4Od9BmyV6fmtrqx9OLJK9X7H10=; b=wdn8XKITPyGjjcivp7zIFsMnN29hGIXduosziz2i8O60vDeQX4clj5yRj3VCMy28z9 rcJJR+zHp/4fuj80m/nmd//8qpLanfd1EB4Dd97QAd27AZsjJNjhSsFZicujSIHzHsMU 9r5xccxpNeFMN/3guZJfPIyRCetyB/T4HyHmIadhO5AJuIRcr1k78c2X7omFYlrou5ar zXsaZdPFRbMrTjdmveaEcCOGrmEcXhJdKhKrcZXGk8E1xbqO3XYpiq4k3J5Z5v9KOCUc NtMMyijPC/aPVQEJcSgZdNU9wjVzPZdfKfv8ifaINRE8mrpUDMnUTMYpqoFg9hOIhGLk KkQw== X-Received: by 10.194.89.168 with SMTP id bp8mr56412036wjb.73.1404995463593; Thu, 10 Jul 2014 05:31:03 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id ft17sm110111803wjc.14.2014.07.10.05.31.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 10 Jul 2014 05:31:02 -0700 (PDT) Sender: Alexander Motin Message-ID: <53BE8784.8060503@FreeBSD.org> Date: Thu, 10 Jul 2014 15:31:00 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Kashyap Desai Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> In-Reply-To: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 12:31:06 -0000 Hi, Kashyap. On 10.07.2014 15:00, Kashyap Desai wrote: > I am trying to collect IOPs and throughput using FIO on FreeBSD-10-stable > as below post mentioned that CAM can reach upto 1,000,000 IOPS using > Fine-Grained CAM locking. > > http://www.freebsd.org/news/status/report-2013-07-2013-09.html#GEOM-Direct-Dispatch-and-Fine-Grained-CAM-Locking > > I am using below FIO parameter. > > [global] > ioengine=posixaio > buffered=0 > rw=randread > bs=4K > iodepth=32 > numjobs=2 > direct=1 > runtime=60s > thread > group_reporting=1 > [job1] > filename=/dev/da0 > [job2] > filename=/dev/da1 > [job3] > filename=/dev/da2 > [job4] > filename=/dev/da3 > [job4] > filename=/dev/da4 > .. > > I have 8 SSDs in my setup and all 8 SSDs are behind LSI’s 12Gp/s MegaRaid > Controller as JBOD. I also found FIO can be used in Async mode after > loading “aio” kernel module. > > Using single SSD, I am able to see 110K-130K IOPs. This IOPs counts are > matching with what I see on Linux machine. > > Now, I am not able to scale IOPs on my machine after 200K. I see CPU is > almost occupied and no idle time after IOPs reach to 200K. > > If you have any pointers to try with, I can do some experiment on my setup. Getting such results I would immediately start doing profiling with pmcstat. Quite likely you are hitting some new lock congestion. Start with simple `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say for sure what went wrong there without more data, so just couple thoughts: First of all, I've never tried aio in my benchmarks, only synchronous ones. Try to run 8 instances of `dd if=/dev/daX of=/dev/null bs=512` per each SSD same time, just as I did. You may vary number of dd's, but keep total below 256, or you mad to increase nswbuf limit in kern_vfs_bio_buffer_alloc(). For second, you are using single HBA, that should create significant congestion around its CAM SIM lock. Proper solution would be to add multiple queues support to the driver, and we discussed it with Scott Long for quite some time, but that requires more work (I hope you may be interested in it ;) ). Or you may just insert 3-4 HBAs. My million IOPS I was reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs. Please tell me what you get, so we could continue investigation. -- Alexander Motin