From owner-svn-src-head@freebsd.org Fri Mar 11 18:33:17 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A505ACC4BD; Fri, 11 Mar 2016 18:33:17 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 101D5D74; Fri, 11 Mar 2016 18:33:17 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: by mail-wm0-x229.google.com with SMTP id p65so28419968wmp.0; Fri, 11 Mar 2016 10:33:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=ec2VIZuVof/YTPJZURFQpXnd7tzfuxJfwR3JAHaChxI=; b=R5b23UcANkowYv7lXGP03+xxuGAkJ9I23HnA4UtlQuqV73e5wJRxxfK3wu4DbvDC+I gM8hY49fPlI/bNyWpYTOGELUxKb9wHxfljHUV/WJLHbkYF5RkgTd6NsJXKS1kXuEWRRY cU4WzCSsDj4nUS7jwrNR3McARUt2jcEUAoxvMVrRIRr7pJ08Tby8a3UxloTU+87WL+KE 8TYZiLaDYq5qOKbranJQotnYMj34NwYwluXA4QisauKJrMa1ibEWPxbLstUqE+A1BlR1 E2wga+5/I+qlKkyf5vn3TJHfLTU/aQCZdhQBN95eY5Ib5UOlhTMrAOds9NTeNHO9+O4N njxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=ec2VIZuVof/YTPJZURFQpXnd7tzfuxJfwR3JAHaChxI=; b=EKaXvb4ppw6GRCjIvfbzbPFzUSV5NLAFit87AI2FjIkSTqRe4G5QY/hKrTwSye6G0j o0ZGoDygGckSNNyNOQEs0hKZ5HOVnbBAHmnJ/7ZvMnYeImfpIbwKI7oId/VmjL8+Fotx tTizUivPyS8SPk5HrFPjjFP9Y0tzWobu+819c5zzicb2kjnB/BrP76uqS1xbHvKV6g1b GBJI4mil412x0zUNP3Qu+24Oz8M7LE4VecvEB1tFeFjVV2ELUYtvtnnu612PnU7Ydrt0 ptT1RoCw/vlwHLV1ZSM86AKsWMm0U483QdPTchKpxzNyn8DiJYDVyOvR/xcm5Wb5dXto y/gQ== X-Gm-Message-State: AD7BkJIYKh6uIlyZr4L1j03mG0bvbI5OPp9hwR3wdn8l4dLYimAjbTt3NMohYEZiFzYW/GmVVTaMz68BRZMJLg== MIME-Version: 1.0 X-Received: by 10.28.153.135 with SMTP id b129mr4640899wme.3.1457721194927; Fri, 11 Mar 2016 10:33:14 -0800 (PST) Received: by 10.194.115.99 with HTTP; Fri, 11 Mar 2016 10:33:14 -0800 (PST) In-Reply-To: References: <201512110206.tBB264Ad039486@repo.freebsd.org> <56E28ABD.3060803@FreeBSD.org> Date: Fri, 11 Mar 2016 11:33:14 -0700 Message-ID: Subject: Re: svn commit: r292074 - in head/sys/dev: nvd nvme From: Jim Harris To: Warner Losh Cc: Alan Somers , Alexander Motin , Steven Hartland , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2016 18:33:17 -0000 On Fri, Mar 11, 2016 at 9:31 AM, Warner Losh wrote: > > > > And keep in mind the original description was this: > > Quote: > > Intel NVMe controllers have a slow path for I/Os that span > a 128KB stripe boundary but ZFS limits ashift, which is derived > from d_stripesize, to 13 (8KB) so we limit the stripesize > reported to geom(8) to 4KB. > > This may result in a small number of additional I/Os > to require splitting in nvme(4), however the NVMe I/O > path is very efficient so these additional I/Os will cause > very minimal (if any) difference in performance or > CPU utilisation. > > unquote > > so the issue seems to being blown up a bit. It's better if you > don't generate these I/Os, but the driver copes by splitting them > on the affected drives causing a small inefficiency because you're > increasing the IOs needed to do the I/O, cutting into the IOPS budget. > > Warner > > Warner is correct. This is something specific to some of the Intel NVMe controllers. The core nvme(4) driver detects Intel controllers that benefit from splitting I/O crossing 128KB stripe boundaries, and will do the splitting internal to the driver. Reporting this stripe size further up the stack is only to reduce the number of I/O that require this splitting. In practice, there is no noticeable impact to performance or latency when splitting I/O on 128KB boundaries. Larger I/O are more likely to require splitting, but for larger I/O you will hit overall bandwidth limitations before getting close to IOPs limitations. -Jim