From owner-freebsd-hackers@freebsd.org Mon Feb 25 12:12:03 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F0D51508547 for ; Mon, 25 Feb 2019 12:12:03 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0548F899B6 for ; Mon, 25 Feb 2019 12:12:02 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x42e.google.com with SMTP id o17so9666675wrw.3 for ; Mon, 25 Feb 2019 04:12:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=79BKp5bWHC7S00+fI1smA1L23BTP0E/3DR+ohhX9UQI=; b=SMXmhQwSPX7fbroYpelp1CTeulLm80VXT2DuWlW0Ku2wHOy5qYoJPF+lL3DU0Yf9R7 iFXe/XzG2y8ahAyhDrv2scYUEdJ8imbLzx/gpjVqNCGCg62N2foH6o33chfEWtYkfcsk HsLSATT+nuj33DZhBtWnOaCPBmSlE7skOA44EkeIc0V1g1jePAqYfZZRjtUicKEFEeQh SMFcnoLA5KE4CFRtiFWS0xDOv2rvQTppfp41lctIs0SLJfA4tk6P9eE1Sko5Df3sjAO3 lugoEgagxU2vubrRaDL2JGf+gfVk15EvAI4oPVn4uN+C5zXB8tyR1wEeOIf8p/l36wSH aeDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=79BKp5bWHC7S00+fI1smA1L23BTP0E/3DR+ohhX9UQI=; b=E3gn5+Vl8+fepoDI8heRsZe+mjOdGHut49ysIO71X23i8/xlJ1JCkkF6prMy2khaRZ QTir+d0ZGWJBp1lhUYK9fRJihZ1EPAUDPNLQjbEEiKIkaPxnEtDihYdgifcX+DWLEdxm PkTiEZVe5thfQK6hsXOFMHWJb0q1HwwZM0OwQg4+eS/hr7PkuI63eQDzdR2vqprGLQEy IWD7Qic0NFHATJJCPk6PA/gpCQ/HmlnJpjC1c2OqXVy1fyLfFNkEixsfLLarv48hAnMq MtgQd1Cj7bMWrN88iHZ2DabQF92NYXCBSiwtfNe4+5RPVGOXXWMGUFGaPAm/HaBPuSd+ yo7w== X-Gm-Message-State: AHQUAuZX5MWijnEzwGzdekaB9ASF8Vp8jO5c5WHc3nKHCO2bfkec8VQt c0C6EhB74NPBudUaCfztJ+SzTp5eWD4VXTrSht6Tnw== X-Google-Smtp-Source: AHgI3IYg3l1GpDn9EkRuHB+Dhh46mQmRB1rl0X++998WFJIjH6tFkHnsMARkx1vJj+i5Ly7+/suxVwLxpSvKuP5EvNo= X-Received: by 2002:a05:6000:1084:: with SMTP id y4mr12541509wrw.14.1551096720814; Mon, 25 Feb 2019 04:12:00 -0800 (PST) MIME-Version: 1.0 References: <0E136DED-C1AD-481C-B243-C943D4F8D9C5@gmail.com> In-Reply-To: From: Rajesh Kumar Date: Mon, 25 Feb 2019 17:41:48 +0530 Message-ID: Subject: Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD To: Enji Cooper Cc: FreeBSD Hackers X-Rspamd-Queue-Id: 0548F899B6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=SMXmhQwS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::42e as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-6.71 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[e.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.70)[ip: (-9.20), ipnet: 2a00:1450::/32(-2.25), asn: 15169(-2.00), country: US(-0.07)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Feb 2019 12:12:03 -0000 Thanks Enji Cooper, Warner Losh, Alan Somers and Rebecca Cran for your Inputs. I did some test and here's my observation. 1) iogine "posixaio" (or) "sync" - no much difference. Is this expected? 2) As said, raw device gives better numbers compared to filesystem. 3) With "thread" option, I see the numbers going considerably down. Is this expected? 4) I see good numbers for IOPS, but the MB/s is less comparatively. Any reasons? Enji Cooper, the mentioned link has all the documents (data sheet, user guide etc.,). Not sure if you have seen them. Anyway, I will consider about the points you have mentioned in my testing and see how they impact the performance. Thanks. Thanks, Rajesh. On Fri, Feb 22, 2019 at 9:24 PM Enji Cooper wrote: > > On Feb 22, 2019, at 01:29, Rajesh Kumar wrote: > > Hi Enji Cooper, > > I am using Samsung 960 PRO > > https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/960pr= o/ > > > Hi Rajesh, > I asked about the datasheet, because there might be some hints in > terms of the number of parallel jobs you might want to apply as well as t= he > I/O queue depth, which will affect the performance of the drive. Otherwis= e > you=E2=80=99ll be throwing values against a wall, seeing what will stick,= which is > sort of ok if you bound your testing and adjust based on performance, but > if your initial hunch is off, it can mislead you. > Similarly, check to see if there are any tunables or sysctls that wil= l > bound the device in terms of the queue depth and I/O requests. > As others have noted, test the device directly if you want to know it= s > raw performance. Only test with a filesystem if your intent is to see how > the device will perform with a filesystem on it (and disable filesystem > sync if you want to test max throughput with the overhead of a filesystem= ). > Testing with a filesystem can reveal some potentially interesting > characteristics, in terms of limitations in VFS / the filesystem > implementation, which might be helpful if you=E2=80=99re trying to determ= ine why > there=E2=80=99s a difference between raw device speed and the speed with = a > filesystem on it. Testing with the same file in different directories is > ok, as long as you blow the drive cache=E2=80=94which will have a noticea= ble > performance impact on conventional (PMR, SMR, etc) drives, as you want to > test the worst case speed of the device, instead of the cache. It should > matter a bit less with faster devices like SSDs/NVMe drives. Testing with > files in the same lateral filesystem hierarchy (same directory), might > reveal issues with filesystem locking (directory inode performance), but > that shouldn=E2=80=99t be the primary goal of your testing. It=E2=80=99s = just something to > keep in mind. > Happy testing! > HTH, > -Enji >