From owner-freebsd-hackers@freebsd.org Fri Feb 22 12:28:45 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28F9314E9F45 for ; Fri, 22 Feb 2019 12:28:45 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [96.73.9.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 339706B3D4 for ; Fri, 22 Feb 2019 12:28:44 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id D353382812; Fri, 22 Feb 2019 05:29:30 -0700 (MST) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id up9mo3BA8mDp; Fri, 22 Feb 2019 05:29:30 -0700 (MST) Received: from photon.int.bluestop.org (gw.bluestop.org [96.73.9.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Fri, 22 Feb 2019 05:29:30 -0700 (MST) Subject: Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD To: Rajesh Kumar , FreeBSD Hackers References: From: Rebecca Cran Message-ID: Date: Fri, 22 Feb 2019 05:28:35 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 339706B3D4 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-2.93)[ip: (-9.73), ipnet: 96.64.0.0/11(-4.30), asn: 7922(-0.53), country: US(-0.07)]; NEURAL_HAM_SHORT(-0.94)[-0.935,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bluestop.org:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; MX_GOOD(-0.01)[mail.bluestop.org]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2019 12:28:45 -0000 On 2/22/19 1:51 AM, Rajesh Kumar wrote: > 1. Should we use "posixaio" as the ioengine (or) something else? > 2. Should we use single thread (or) multiple threads for test? If > multiple threads, how can we decide on the optimal thread count? > 3. Should we use "raw device files" (Eg: nvme namespace file - > /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a > regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better > numbers. > 4. Should we use a shared file (or) one file per thread? > 5. I believe 1Job should be fine for benchmarking. (or) should we try > multiple jobs? I just ran a quick test on a filesystem on my machine which has an M.2 NVMe drive, and it seems posixaio performs pretty poorly compared to the sync ioengine: around 700 MB/s vs. 1100 MB/s! I _was_ going to suggest using posixaio and setting iodepth to something like 32, but since it performs badly I'd suggest playing around with the numjobs parameter and seeing where the best performance is achieved - whether that's latency or throughput. On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150 MB/s and 32 1840 MB/s with a 4 KB block size. Bumping the block size from 4 KB to 16 KB makes the throughput more jumpy, but appears to average 2300 MB/s when used with 32 jobs. -- Rebecca Cran