From owner-freebsd-stable@freebsd.org Mon Apr 1 15:22:25 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8681E1566A66 for ; Mon, 1 Apr 2019 15:22:25 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yw1-f67.google.com (mail-yw1-f67.google.com [209.85.161.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8EBE773BE8 for ; Mon, 1 Apr 2019 15:22:24 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yw1-f67.google.com with SMTP id d132so3394507ywa.2 for ; Mon, 01 Apr 2019 08:22:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=gw3C3igBRrNqq1SoZXr5t5Xug7wFSD43831ZUuqh6Eg=; b=MNopiQaxpkB2yFvkzGiiMhK9+tYyb45HMEijdK280R3zWX6S0VYYrOAo68EXq5g+Vo GK/mnxJRjg4I/4D7bQSETTf+Q/il31JfLj+YOWZMZMxDIljOsguj+HYhll90+DOWbCRD 8RgbCBQdcUsIHEmPl8ZOvy9v/ShqLHmVps6WvsB7GR43XglpAF7dCxKfYgRzNay3ZN02 Eb0scLdDk6NqvCNa+VEbDgfsYomgmYEiMrFXHDO/xyCzgcjD6xww2OsCF01n4vSwfkYb vlhK1K6Kzm1538Ya0GNIhY2kmvPyUj0R4XKx+6gb/7moJaannUSDXtJMRqOpJ3wZSCmu spIg== X-Gm-Message-State: APjAAAXtv3mu3NQawmr7gs8m9e9YQokuYdOVgTRt2ssiAB1UOuJ87Icx tGm0/1ep7kwFKFEuoelISjoF3kn49oLAF6woMyQ= X-Google-Smtp-Source: APXvYqwbmFUgCoO7HI8WhhaE38VH/wLc5NvMf0preYn6UW6g7TST96d1W5xfCMnHHlgfhqBvUfv0adgiCvbyEv5msYQ= X-Received: by 2002:a81:3a57:: with SMTP id h84mr52075079ywa.284.1554131799909; Mon, 01 Apr 2019 08:16:39 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Mon, 1 Apr 2019 23:16:28 +0800 Message-ID: Subject: FreeBSD CI Weekly Report 2019-03-31 To: freebsd-testing@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 8EBE773BE8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.161.67 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-4.08 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; RCVD_COUNT_TWO(0.00)[2]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[67.161.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.85)[-0.853,0]; DMARC_NA(0.00)[freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[67.161.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.22)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.15), country: US(-0.06)]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FORGED_RECIPIENTS(0.00)[freebsd-testing@freebsd.org,freebsd-stable@freebsd.org]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Mailman-Approved-At: Mon, 01 Apr 2019 16:03:16 +0000 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2019 15:22:25 -0000 (bcc -current and -stable for more audience) FreeBSD CI Weekly Report 2019-03-31 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-03-25 to 2019-03-31. During this period, we have: * 2095 builds (98.7% passed, 1.3% failed) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 363 test runs (52.6% passed, 46.8% unstable, 0.6% exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 8 doc buils (100% passed) (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/s/BymrvPI_4 and archive is available at http://hackfoldr.org/freebsd-ci-report/, any help is welcome. ## Fixed Tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ * lib.libarchive.functional_test.test_fuzz_zip Fixed in https://svnweb.freebsd.org/changeset/base/345497 * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * lib.libc.regex.exhaust_test.regcomp_too_big * lib.libregex.exhaust_test.regcomp_too_big Fixed in https://svnweb.freebsd.org/changeset/base/345516 ## Failing Tests * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * sys.netmap.ctrl-api-test.main * sys.opencrypto.runtests.main * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 * (New) lib.libc.gen.floatunditf_test.floatunditf * (New) lib.libc.stdio.printfloat_test.hexadecimal_rounding * (New) lib.msun.ctrig_test.test_small_inputs * (New) lib.msun.precision_test.t_precision https://bugs.freebsd.org/236936 * https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/ * sys.netmap.ctrl-api-test.main * sys.opencrypto.runtests.main * lib.libc.regex.exhaust_test.regcomp_too_big * lib.libregex.exhaust_test.regcomp_too_big * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * https://ci.freebsd.org/job/FreeBSD-stable-11-amd64-test/ * usr.bin.procstat.procstat_test.kernel_stacks * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ * sys.netmap.ctrl-api-test.main * sys.opencrypto.runtests.main * usr.bin.procstat.procstat_test.kernel_stacks * local.kyua.* (31 cases) * local.lutok.* (3 cases) * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 ## Failing Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * common.ip.t_dtrace_contrib.tst_ipv4localsctp_ksh * common.ip.t_dtrace_contrib.tst_localsctpstate_ksh * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ There are ~60 failing cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details ## Disabled Tests * lib.libc.sys.mmap_test.mmap_truncate_signal https://bugs.freebsd.org/211924 * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * usr.bin.procstat.procstat_test.command_line_arguments https://bugs.freebsd.org/233587 * usr.bin.procstat.procstat_test.environment https://bugs.freebsd.org/233588 ## Closed Issues * Flakey test case: lib.libarchive.functional_test.test_fuzz_zip https://bugs.freebsd.org/236300 ## Oepn Issues * https://bugs.freebsd.org/236936 4 test cases failing on i386 after r345562 ### Cause build fails * [233735: Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory](https://bugs.freebsd.org/233735) * [233769: Possible build race: ld: error: unable to find library -lgcc_s](https://bugs.freebsd.org/233769) ### Others [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-stable@freebsd.org Tue Apr 2 16:01:08 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58C30156B325 for ; Tue, 2 Apr 2019 16:01:08 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F8116B209 for ; Tue, 2 Apr 2019 16:01:07 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: by mail-ed1-x52b.google.com with SMTP id d1so12116567edd.13 for ; Tue, 02 Apr 2019 09:01:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=LZyNeO1qIcj1mJYMfq8pjORstkn9nQJ2ih1x5MltITQ=; b=Vtl1XfoMqy3+v25yqCLXKgprdVaCyOuvV+bj6Jx2CoznOwuqrFpAYiCgBzhn0QRXoE g3Eb5E7LuWff2U16kJzsVwlyKPXyW2IulI6Uab0FumUIapeh39BtKzxZQVRn8qUeLFVm RWgVnrTWcc86y8VCo5YvHqNGvcX00d/yDWi1Y8CQY8ugWIC8CfWcrwrCXgjAbk3Dj8uv xdmTfnzHetOw2w9sjh2DH/znymY5ZBhFNyvHaCanGTztVlclzN/lIAzDoJztZgYGxeG1 BDMNTySmeC03EklYMVty9MXrBZKFhEL+l8fM5XMqooDzoLIQjEj7Jzb4yjEoyUjIBLW/ 1unA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=LZyNeO1qIcj1mJYMfq8pjORstkn9nQJ2ih1x5MltITQ=; b=aylrnccfTZ0RSR902gqV+ZV5tLKPweKs1QgsiMlk8GcES7SIJ6ETShhHK9U34pzOsW OQckl7MiAzdk6TTA9rnTYiXlyqSeYCwqBHdMnR/xj/ZItonoVmf6OUTSiTsWgI9cgCL0 UptNYBnaNzTSbmLG//Kc61ti/tGTTnLcIAR0V+eRiTBTSiPDqEXFir0icITi4ykdh11x 64lz+vQiJGbJeFHQKk+SK+GFJDiM91wI77XuhWpkx9s3Y71QWt7P8VQ8R1bC1XUELK0r 9cw51qnd29HsYxUpcK7gReR4569cuc+Kpc52ZZW93obnrn8FkZXCRd39nnq3b9TSarfU 6cgA== X-Gm-Message-State: APjAAAXzkXl5UbEpRsmRAHp1GNEP4t3yU8iHIVG2M6LIKCInIZsM4sGP jXsGrgechSpKZTQmliUQIBZ6r0Ba X-Google-Smtp-Source: APXvYqy46xbDK7MMOUvLr2yuEbnMm4i5AGHb+AQMnm3QX6pgBFzf1JZd2ECFUkdWUSziQ+WlVU2YBQ== X-Received: by 2002:a50:ad83:: with SMTP id a3mr10833414edd.21.1554220864421; Tue, 02 Apr 2019 09:01:04 -0700 (PDT) Received: from hollewijn.internal (217-19-30-105.dsl.cambrium.nl. [217.19.30.105]) by smtp.gmail.com with ESMTPSA id h15sm4065176edq.76.2019.04.02.09.01.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 09:01:03 -0700 (PDT) From: Alban Hertroys Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Bridge stopped bridging after upgrade from 11-STABLE to 12-STABLE Message-Id: <988F2E44-5030-4F87-8572-413802356A00@gmail.com> Date: Tue, 2 Apr 2019 18:01:02 +0200 To: freebsd-stable List X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 2F8116B209 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Vtl1XfoM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of haramrae@gmail.com designates 2a00:1450:4864:20::52b as permitted sender) smtp.mailfrom=haramrae@gmail.com X-Spamd-Result: default: False [-6.16 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.94)[-0.944,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.70)[ip: (-8.94), ipnet: 2a00:1450::/32(-2.37), asn: 15169(-2.15), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[b.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Apr 2019 16:01:08 -0000 I=E2=80=99m probably just missing something, but I don=E2=80=99t see = anything obviously wrong and Google is not being helpful. I have em0 and wlan0 bridged on my internal network, but wireless = devices cannot access wired devices and vice versa - except for the = bridging host itself. That used to work before I upgraded to 12-STABLE. How should I recover connectivity between both media? In rc.conf I have: --- # Wireless wlans_ath0=3D"wlan0" create_args_wlan0=3D"wlanmode hostap" # Bridged interfaces (see example at man 4 bridge) cloned_interfaces=3D"bridge0" ifconfig_bridge0=3D"addm em0 addm wlan0 stp em0 stp wlan0 up" ifconfig_bridge0_alias0=3D"inet 10.0.0.1/24" ifconfig_bridge0_ipv6_alias0=3D"inet6 fe80::6efd:b9ff:fe68:db36%bridge0" ifconfig_wlan0=3D"ssid solfertje country NL mode 11g channel 9:ht/40 up" ifconfig_wlan0_ipv6=3D"inet6 accept_rtadv" hostapd_enable=3D"YES" ifconfig_em0=3D"tso up" ifconfig_em0_ipv6=3D"inet6 accept_rtadv=E2=80=9D --- Resulting in these interfaces: --- wlan0: flags=3D8943 = metric 0 mtu 1500 ether 6c:fd:b9:68:db:36 inet6 fe80::6efd:b9ff:fe68:db36%wlan0 prefixlen 64 scopeid 0x4 groups: wlan ssid solfertje channel 9 (2452 MHz 11g) bssid 6c:fd:b9:68:db:36 regdomain 32924 country CN indoor ecm authmode WPA2/802.11i privacy MIXED deftxkey 2 AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 20 scanvalid 60 protmode CTS wme burst dtimperiod 1 -dfs media: IEEE 802.11 Wireless Ethernet autoselect mode 11g = status: running nd6 options=3D23 em0: flags=3D8943 metric = 0 mtu 1500 = options=3D812099 ether 68:05:ca:17:fe:f7 inet6 fe80::6a05:caff:fe17:fef7%em0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D23 bridge0: flags=3D8843 metric 0 = mtu 1500 ether 02:95:bb:bd:98:00 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 id 68:05:ca:17:fe:f7 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 68:05:ca:17:fe:f7 priority 32768 ifcost 0 port 0 member: wlan0 = flags=3D167 ifmaxaddr 0 port 4 priority 128 path cost 33333 proto = rstp role designated state forwarding member: em0 = flags=3D1e7 ifmaxaddr 0 port 1 priority 128 path cost 20000 proto = rstp role designated state forwarding groups: bridge nd6 options=3D9 =E2=80=94 Regards, Alban Hertroys P.S. The package for postgresql10-server links to icu libraries version = 61, while version 64 gets installed. I should be running PG 11, I know, = working on that ;) -- There is always an exception to always. From owner-freebsd-stable@freebsd.org Thu Apr 4 08:38:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 326591568EAA for ; Thu, 4 Apr 2019 08:38:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 053878CCB4 for ; Thu, 4 Apr 2019 08:37:59 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x348bwee010124 for ; Thu, 4 Apr 2019 10:37:58 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x348bw9r096634 for ; Thu, 4 Apr 2019 10:37:58 +0200 (CEST) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: NVME aborting outstanding i/o Message-Id: References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> To: freebsd-stable@freebsd.org Date: Thu, 4 Apr 2019 10:37:58 +0200 X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 053878CCB4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.53)[-0.532,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.36)[ipnet: 217.29.32.0/20(-0.99), asn: 16188(-0.79), country: DE(-0.01)]; DMARC_NA(0.00)[punkt.de]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 08:38:07 -0000 Hi all, I=E2=80=99m currently doing some load tests/burn in for two new servers. These feature all NVME SSDs and run FreeNAS, i.e. FreeBSD 11.2-STABLE. pcib17: at device 3.2 numa-domain 1 on = pci15 pcib17: [GIANT-LOCKED] pci17: numa-domain 1 on pcib17 nvme7: mem 0xeca10000-0xeca13fff at device = 0.0 numa-domain 1 on pci17 When putting some moderate i/o load on the system, the log fills with = these messages: nvme7: aborting outstanding i/o nvme7: DATASET MANAGEMENT sqid:41 cid:91 nsid:1 nvme7: ABORTED - BY REQUEST (00/07) sqid:41 cid:91 cdw0:0 There has been some discussion of this on on the iX Systems forum as = well as various FreeBSD media and one person suggested setting: hw.nvme.per_cpu_io_queues=3D0 This is where I need some help now. This is from the manpage for = nvme(4): ---------- To force a single I/O queue pair shared by all CPUs, set the = following tunable value in loader.conf(5): hw.nvme.per_cpu_io_queues=3D0 To assign more than one CPU per I/O queue pair, thereby reducing the number of MSI-X vectors consumed by the device, set the following = tunable value in loader.conf(5): hw.nvme.min_cpus_per_ioq=3DX To force legacy interrupts for all nvme driver instances, set the following tunable value in loader.conf(5): hw.nvme.force_intx=3D1 Note that use of INTx implies disabling of per-CPU I/O queue pairs. ---------- But: root@freenas01[~]# sysctl hw.nvme.per_cpu_io_queues sysctl: unknown oid 'hw.nvme.per_cpu_io_queues' root@freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq' root@freenas01[~]# sysctl hw.nvme.force_intx sysctl: unknown oid 'hw.nvme.force_intx' Where do I go from here? Thanks! Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Thu Apr 4 11:25:20 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E923C156D565 for ; Thu, 4 Apr 2019 11:25:19 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A65D6C095 for ; Thu, 4 Apr 2019 11:25:18 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x34BPG1j013313 for ; Thu, 4 Apr 2019 13:25:16 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x34BPFwD005293 for ; Thu, 4 Apr 2019 13:25:15 +0200 (CEST) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o Date: Thu, 4 Apr 2019 13:25:15 +0200 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> To: freebsd-stable@freebsd.org In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 8A65D6C095 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-3.10 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.95)[-0.951,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.34)[ipnet: 217.29.32.0/20(-0.95), asn: 16188(-0.76), country: DE(-0.01)]; DMARC_NA(0.00)[punkt.de]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 11:25:20 -0000 Hi, > Am 04.04.2019 um 10:37 schrieb Patrick M. Hausen : > But: >=20 > root@freenas01[~]# sysctl hw.nvme.per_cpu_io_queues > sysctl: unknown oid 'hw.nvme.per_cpu_io_queues' > root@freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq > sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq' > root@freenas01[~]# sysctl hw.nvme.force_intx > sysctl: unknown oid 'hw.nvme.force_intx' Looks like these can be set via loader.conf and reboot although they are not visible in the running system. Even when set they are not visible afterwards ... hw.nvme.per_cpu_io_queues=3D0 seems to cure the NVME errors. I=E2=80=99m still curious about the root problem here, bit at least I can now continue to use the machines for some real loads. Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Thu Apr 4 14:55:11 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 966A6157359B for ; Thu, 4 Apr 2019 14:55:11 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 85F6874EC4; Thu, 4 Apr 2019 14:55:10 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x34Et87d018074; Thu, 4 Apr 2019 16:55:08 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x34Et8Nm016894; Thu, 4 Apr 2019 16:55:08 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o From: "Patrick M. Hausen" In-Reply-To: Date: Thu, 4 Apr 2019 16:55:07 +0200 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <751FDBB4-14C0-420E-B81A-6C66D3564FB6@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> To: Chuck Tuffli X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 85F6874EC4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.995,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.60)[-0.602,0]; IP_SCORE(-0.33)[ipnet: 217.29.32.0/20(-0.92), asn: 16188(-0.74), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 14:55:11 -0000 > Am 04.04.2019 um 16:51 schrieb Chuck Tuffli : > nvmecontrol identify nvme7 Controller Capabilities/Features ================================ Vendor ID: 8086 Subsystem Vendor ID: 8086 Serial Number: BTLJ90230F1R1P0FGN Model Number: INTEL SSDPE2KX010T8 Firmware Version: VDV10131 Recommended Arb Burst: 0 IEEE OUI Identifier: e4 d2 5c Multi-Interface Cap: 00 Max Data Transfer Size: 131072 Controller ID: 0x00 The server is a Supermicro AS-1113S-WN10RT Kind regards, Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Thu Apr 4 15:11:40 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A6621573B4F for ; Thu, 4 Apr 2019 15:11:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D1CA75FDE for ; Thu, 4 Apr 2019 15:11:39 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82f.google.com with SMTP id w30so3619375qta.8 for ; Thu, 04 Apr 2019 08:11:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8yALtkS5XQvx6CIAJD7dEtflwQIWZoLMOp81ZZh1C0A=; b=IWhj57HtFPpcgkXIoTyjTg6vRVGc4554HxamrjtkWYp4J1u30rwG3mzb+U7u8MurOE aFDx8SJM56hrOfJcgPU32uEDdPbPwwtKlIbAL0EKryGVcqSduxxzCL7GOQ7NLaTgR6ng bOdkfqGW5D6kyz444T5iZjw57keQadGIpRJISSivQKELJzV6IMA0XGvSBtjhCKfnRZAC 3CGgtrmySlJTqIg2sVRM4yKEDE0v4XmnRiXiBMgMCxUAwB7HwIPnb++Yj0Rv1xxbWY/u jPu2wcybuCcPfalrkKTqYM7si6ObzT2DfqSsRp3GZFYC0/kOuQtgx1vl/qK12kC9zaEn RtwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8yALtkS5XQvx6CIAJD7dEtflwQIWZoLMOp81ZZh1C0A=; b=popxk6yVXd9fAjgxTcsMxeWdGwGQJxDdYuWrsFJkueYD+pe6zzy8sldyeahx/sa8XR CYb5g8vD+M/jp4rgklzjAMkUzsKAx6cIQ0+GXZXDc68PKOKnKUTECHULFgJpRIW/fxMO QEKHnMMomqF1xQv7ZeoagTKnz8VXYLArOeCVVD0oY3UXdQUawaox0Rkkzo4Pv9VSoIn7 VLCKvkCM/pFpdE220HEuCIrbXKW+O2e9O+19McOQYEbhCJ6PO9h9RpMswQ4UD+88mhP6 Pg4tO1rB081iB0mLO+hZ6Qc5wMmsHCb0X+wpHOdyt38+YOQM1uQO+IXsJek0tLkgLHiu Yk7g== X-Gm-Message-State: APjAAAUqMBZ3p9T7AwAXvBe89J+0fXkgdE/vNHyytNH4PBcJ6nisu8iR wTwVbcPax4rTWE9B/Y/MGwgcVKJhZQhpUKf+4963Ex/Y X-Google-Smtp-Source: APXvYqxMV+F0X/SMNprqK5thuUy73umHugHhXZhU0+CrcnltVUbuI3M7NMpBbV78WMp1npLrVVc8TH2k7fxy07EyNyc= X-Received: by 2002:a0c:ecca:: with SMTP id o10mr5151348qvq.197.1554390698357; Thu, 04 Apr 2019 08:11:38 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> In-Reply-To: From: Warner Losh Date: Thu, 4 Apr 2019 09:11:27 -0600 Message-ID: Subject: Re: NVME aborting outstanding i/o To: "Patrick M. Hausen" Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 1D1CA75FDE X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=IWhj57Ht X-Spamd-Result: default: False [-5.80 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[f.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.80)[ip: (-8.86), ipnet: 2607:f8b0::/32(-2.91), asn: 15169(-2.16), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 15:11:40 -0000 On Thu, Apr 4, 2019 at 2:39 AM Patrick M. Hausen wrote: > Hi all, > > I=E2=80=99m currently doing some load tests/burn in for two new servers. > These feature all NVME SSDs and run FreeNAS, i.e. FreeBSD 11.2-STABLE. > > pcib17: at device 3.2 numa-domain 1 on pci1= 5 > pcib17: [GIANT-LOCKED] > pci17: numa-domain 1 on pcib17 > nvme7: mem 0xeca10000-0xeca13fff at device > 0.0 numa-domain 1 on pci17 > > When putting some moderate i/o load on the system, the log fills with the= se > messages: > > nvme7: aborting outstanding i/o > nvme7: DATASET MANAGEMENT sqid:41 cid:91 nsid:1 > nvme7: ABORTED - BY REQUEST (00/07) sqid:41 cid:91 cdw0:0 > OK. So unless you are suspending and resuming, or the drive is somehow failing, here's what's going on: There's a request that was sent down to the drive. It took longer than 30s to respond. One of them, at least, was a trim request. There's a number of reasons for this. NAND sucks. It's a horrible steaming pile of... silicon. To make it useful, there's a layer of software called the FTL (flash translation layer). NAND is an append-only medium at the lowest level, so the FTL has to take requests and build a map of logical blocks to physical blocks, as well as manage the 'log structured device' in some way. The details of why are too long to get into here (see my BSDCan talk from a few years ago). But what is relevant is that many drives have really crappy FTLs, especially when it comes to TRIMs. They can't handle a lot of them, and when you send a lot down, like FreeBSD will often do with UFS or ZFS, you can trigger the driving doing a bunch of garbage collection. This can cause the drive to delay > 30s before responding to commands. So sometimes you can avoid this by disabling trims. Other times, you have crappy FTL that crashes. This can cause a long timeout because FreeBSD has done something that, while in spec, is unexpected or not well tested. Here you can really only have FreeBSD do less work at once to avoid this issue, or you can upgrade the firmware. There has been some discussion of this on on the iX Systems forum as well > as various > FreeBSD media and one person suggested setting: > > hw.nvme.per_cpu_io_queues=3D0 > > > This is where I need some help now. This is from the manpage for nvme(4): > > ---------- > To force a single I/O queue pair shared by all CPUs, set the followin= g > tunable value in loader.conf(5): > > hw.nvme.per_cpu_io_queues=3D0 > > To assign more than one CPU per I/O queue pair, thereby reducing the > number of MSI-X vectors consumed by the device, set the following > tunable > value in loader.conf(5): > > hw.nvme.min_cpus_per_ioq=3DX > > To force legacy interrupts for all nvme driver instances, set the > following tunable value in loader.conf(5): > > hw.nvme.force_intx=3D1 > > Note that use of INTx implies disabling of per-CPU I/O queue pairs. > ---------- > > But: > > root@freenas01[~]# sysctl hw.nvme.per_cpu_io_queues > sysctl: unknown oid 'hw.nvme.per_cpu_io_queues' > root@freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq > sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq' > root@freenas01[~]# sysctl hw.nvme.force_intx > sysctl: unknown oid 'hw.nvme.force_intx' > > > Where do I go from here? > Did you add it to /boot/loader.conf? There's no sysctl for this. Wanrer > > Thanks! > Patrick > -- > punkt.de GmbH Internet - Dienstleistungen - Beratung > Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 > 76133 Karlsruhe info@punkt.de http://punkt.de > AG Mannheim 108285 Gf: Juergen Egeling > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Thu Apr 4 15:22:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 869571574198 for ; Thu, 4 Apr 2019 15:22:29 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com [209.85.210.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B3B9976A7E for ; Thu, 4 Apr 2019 15:22:28 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-ot1-f44.google.com with SMTP id o74so2698320ota.3 for ; Thu, 04 Apr 2019 08:22:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Nk+FXoQBRhtcIvzO0eYf50J/6j0iCgS3/RqS0tLYjRg=; b=RopUV4sqEszEWQlx/cIbOfcIyHS4HUlO8Wh3135HKyqNxWG1MC87pAIY62fcuxw6CS iNVqGExu3DNgv1RJq/hn6lRCIA0rO3jisPWuIRbnySSecw8V0BU6bg4Z6RrNgqvb2ZLv mAikAcgFkvV16xvlEes8A7cVDBVAIcMyyg/Tr9HdLD/RoeLPozhUaMjoJ4aAL5E5c8zZ f1/Iu+1R38YSlYXQa3BshKnho7ebHO8J+yLdA7BquM95utodSR742E0DOXughYBzr4MZ inU44vEV7SgZuTRkN75gYfzCZrMDhrvk+7YjhV/lVKkS3fFQv+inHKPUB0SwN7PHkhei PiCQ== X-Gm-Message-State: APjAAAXJck7xrwewT9n7o6bNlmEGQnX/BQHmDkqfxyEIB+oKAhud3C9s HrOCCVtBLLjdXszeWiTncl2N0RXR X-Google-Smtp-Source: APXvYqwXMQOrCoPB2GDAhE+wEw1u7bbJRCXR4jlJc6cYcgQChKoQGGSWZzmgemNNASK8HvM1JaIQyA== X-Received: by 2002:a05:6830:164a:: with SMTP id h10mr4473394otr.323.1554389486150; Thu, 04 Apr 2019 07:51:26 -0700 (PDT) Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com. [209.85.167.169]) by smtp.gmail.com with ESMTPSA id 96sm8367259otf.17.2019.04.04.07.51.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 Apr 2019 07:51:25 -0700 (PDT) Received: by mail-oi1-f169.google.com with SMTP id t81so2101824oig.10 for ; Thu, 04 Apr 2019 07:51:25 -0700 (PDT) X-Received: by 2002:aca:eb93:: with SMTP id j141mr3598428oih.178.1554389485196; Thu, 04 Apr 2019 07:51:25 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> In-Reply-To: From: Chuck Tuffli Date: Thu, 4 Apr 2019 07:51:14 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: NVME aborting outstanding i/o To: "Patrick M. Hausen" Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B3B9976A7E X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.210.44 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [-4.18 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[44.210.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.85)[-0.852,0]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; IP_SCORE(-1.32)[ip: (-0.50), ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.16), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Apr 2019 15:22:29 -0000 On Thu, Apr 4, 2019 at 4:27 AM Patrick M. Hausen wrote: > > Hi, > > > Am 04.04.2019 um 10:37 schrieb Patrick M. Hausen : > > But: > > > > root@freenas01[~]# sysctl hw.nvme.per_cpu_io_queues > > sysctl: unknown oid 'hw.nvme.per_cpu_io_queues' > > root@freenas01[~]# sysctl hw.nvme.min_cpus_per_ioq > > sysctl: unknown oid 'hw.nvme.min_cpus_per_ioq' > > root@freenas01[~]# sysctl hw.nvme.force_intx > > sysctl: unknown oid 'hw.nvme.force_intx' > > Looks like these can be set via loader.conf and reboot although > they are not visible in the running system. Even when set they > are not visible afterwards ... > > hw.nvme.per_cpu_io_queues=3D0 > > seems to cure the NVME errors. I=E2=80=99m still curious about the root > problem here, bit at least I can now continue to use the machines > for some real loads. Out of curiosity, what device is this? I.e. nvmecontrol identify nvme7 --chuck From owner-freebsd-stable@freebsd.org Fri Apr 5 07:33:20 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 315B9156D30A for ; Fri, 5 Apr 2019 07:33:20 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 10F6B83439 for ; Fri, 5 Apr 2019 07:33:18 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x357XGV1029671; Fri, 5 Apr 2019 09:33:16 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x357XFkf042698; Fri, 5 Apr 2019 09:33:16 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o From: "Patrick M. Hausen" In-Reply-To: Date: Fri, 5 Apr 2019 09:33:15 +0200 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 10F6B83439 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.69)[-0.691,0]; IP_SCORE(-0.32)[ipnet: 217.29.32.0/20(-0.89), asn: 16188(-0.72), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 07:33:20 -0000 Hi all, > Am 04.04.2019 um 17:11 schrieb Warner Losh : > There's a request that was sent down to the drive. It took longer than = 30s to respond. One of them, at least, was a trim request. > [=E2=80=A6] Thanks for the explanation. This further explains why I was seeing a lot more of those and the = system occasionally froze for a couple of seconds after I increased these: vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 as recommended by Allan Jude reasoning that NVME devices could work on up to 64 requests in parallel. I have since reverted that change and I = am running with the defaults. If I understand correctly, this: > hw.nvme.per_cpu_io_queues=3D0 essentially limits the rate at which the system throws commands at the = devices. Correct? So it=E2=80=99s not a real fix and there=E2=80=99s nothing fundamentally = wrong with the per CPU queue or interrupt implementation. I will look into new firmware for my Intel = devices and try tweaking the vfs.zfs.vdev.trim_max_active and related parameters. Out of curiosity: what happens if I disable TRIM? My knowledge is rather = superficial and I just filed that under =E2=80=9ETRIM is absolutely essential lest = performance will suffer severely and your devices die - plus bad karma, of course =E2=80=A6= =E2=80=9C ;-) Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Fri Apr 5 10:49:47 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9623B157157B for ; Fri, 5 Apr 2019 10:49:47 +0000 (UTC) (envelope-from server@no-reply.com) Received: from auce.dal.corespace.com (auce.dal.corespace.com [64.182.228.108]) by mx1.freebsd.org (Postfix) with ESMTP id D8F2C88CAF for ; Fri, 5 Apr 2019 10:49:46 +0000 (UTC) (envelope-from server@no-reply.com) Received: from WIN-PMEMFBE7L9O.cs1cloud.internal (unknown [149.202.36.57]) by auce.dal.corespace.com (Postfix) with ESMTPA id E7F0F38B081 for ; Fri, 5 Apr 2019 05:40:46 -0500 (CDT) MIME-Version: 1.0 Subject: You have 7 pending mails To: freebsd-stable@freebsd.org From: "E-mail Administrator" Date: Fri, 05 Apr 2019 03:40:54 -0700 X-Rspamd-Queue-Id: D8F2C88CAF X-Spamd-Bar: ++++++++++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [12.55 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MX_INVALID(0.50)[cached]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[1]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:54489, ipnet:64.182.0.0/16, country:US]; IP_SCORE(0.06)[asn: 54489(0.34), country: US(-0.06)]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(3.00)[57.36.202.149.zen.spamhaus.org : 127.0.0.4]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.49)[0.493,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000,0]; RCPT_COUNT_ONE(0.00)[1]; MISSING_MID(2.50)[]; RBL_VIRUSFREE_BOTNET(2.00)[108.228.182.64.bip.virusfree.cz : 127.0.0.2]; NEURAL_SPAM_LONG(1.00)[1.000,0]; DMARC_NA(0.00)[no-reply.com]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; GREYLIST(0.00)[pass,body] X-Spam: Yes Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 10:49:47 -0000 = = = Emails Pending! = = = Dear freebsd-stable@freebsd.org, = You have 7 New pending mails. Your mail version 2.0.1 = is currently being discontinued from receiving incoming mails, and will no longer work as of April 6th, 2019. To retrieve your messages and upgrade to version 3.0.1, kindly follow the upgrade information below: Upgrade to version 3.0.1 now Have a nice day. Administrator Team = = = = =A9 Copyright (2019) Email Administrator. All Rights Reserved . =20 From owner-freebsd-stable@freebsd.org Fri Apr 5 12:40:08 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8898415761A0 for ; Fri, 5 Apr 2019 12:40:08 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5FB48F8F4 for ; Fri, 5 Apr 2019 12:40:01 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x35Cds5o035606 for ; Fri, 5 Apr 2019 14:39:54 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x35CdsuQ057422 for ; Fri, 5 Apr 2019 14:39:54 +0200 (CEST) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o) Date: Fri, 5 Apr 2019 14:39:54 +0200 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> To: freebsd-stable@freebsd.org In-Reply-To: <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> Message-Id: <7070F9C7-556F-4BE5-B274-8318CD62E2DD@punkt.de> X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: E5FB48F8F4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.75 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.63)[-0.629,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.31)[ipnet: 217.29.32.0/20(-0.87), asn: 16188(-0.69), country: DE(-0.01)]; DMARC_NA(0.00)[punkt.de]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 12:40:08 -0000 Hi all, in addition to the aborted commands every dozen of system boots or so (this order of magnitude) the kernel simply hangs during initialisation of one of the NVME devices: https://cloud.hausen.com/s/TxPTDFJwMe6sJr2 The particular device affected is not constant. A power cycle fixes it, the system has not shown hangs/freezes during multiuser operation, yet. Any ideas? Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Fri Apr 5 14:36:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBCC41578943 for ; Fri, 5 Apr 2019 14:36:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6525940A3 for ; Fri, 5 Apr 2019 14:36:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x832.google.com with SMTP id w5so7575356qtb.11 for ; Fri, 05 Apr 2019 07:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZaVy1pm8lTjhngkO5DxX5/yqciB2CTDoqKCD20X2fh0=; b=hhUQOpnwoubWJHD5Ru2wqX1552kzi2SDcLpWx94AGCE95Aaqk2B1Hrk5TVVWcHDpPz Co78Dmnu6kCTyIVwZugRCDV73T9bzmBVHaVH2aOd/ZyxFSKdYxkWdD12S9fgl1aT4VJ/ ePW0IBown8yzg8Hp8SCZRRhjmPAxmzdfOEmkKm7fNLJgYXWyjReJ8GK0Bl4P7VJb3BRL LXeNhBnki6jONWhGOnxS7E5mQiRPnwbIWNa64jZqRWLvqmMnUsKVN2N8WkZEQ2PjajmG w0CgAVB4rtDAu94ftA32Qv+h9suzaFwMW6R3UQmPze0SYJvSF7yzSKG0iEu2/RK7mayx mECw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZaVy1pm8lTjhngkO5DxX5/yqciB2CTDoqKCD20X2fh0=; b=b75rv4bcEqRUsKeDpWFwNSXO4nQUSR0yGM5i/167mpADD/91U/MKTsaahAbRl27+Ow 2kuQNpOmI7IXOnumNmsJv1FAMYfkUnonFAe6Ae9hRgf/SmITmp/Lnhy1trf/3QwA+z6K hXpDKZB/befVDaQUyx3ZCBT0rtQB9DKh63x6Jio9JGHpXQWaiz5P7dja4TSI9/jNWW34 V//FdSmpC1JRfH5JkQyUU4jPgxnm5Youg2HipON/aDWTbzq3+hdl+O8CAEIIuuM20jH7 acRNCPWSAPFqWxBORYLCiZrdLcNdpDcSuH0U8cl/vTNYIQrcIJzPONz0vpIHzoAZn5Lb WlXQ== X-Gm-Message-State: APjAAAVzRlN03/oLzawcbBIDx6hIe1d5Mbyi1mGYVqaKGrMLlSr0XhRL r7RdVgg6rdullhV/+M1ObIi4nsi0+G6ra8HO+tyeP66P X-Google-Smtp-Source: APXvYqx7b6cAiuJ0npa0rrs/UbH9dHuP+fkD+2IUrCSqkusOn5u+ccVAIG0lwFevPxEBWZ+pjCIpxEHhfNdUShFdDAk= X-Received: by 2002:ac8:3328:: with SMTP id t37mr11067678qta.246.1554475000125; Fri, 05 Apr 2019 07:36:40 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <7070F9C7-556F-4BE5-B274-8318CD62E2DD@punkt.de> In-Reply-To: <7070F9C7-556F-4BE5-B274-8318CD62E2DD@punkt.de> From: Warner Losh Date: Fri, 5 Apr 2019 08:36:29 -0600 Message-ID: Subject: Re: Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o) To: "Patrick M. Hausen" Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: A6525940A3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=hhUQOpnw X-Spamd-Result: default: False [-5.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.86)[ip: (-9.15), ipnet: 2607:f8b0::/32(-2.92), asn: 15169(-2.17), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 14:36:42 -0000 On Fri, Apr 5, 2019 at 6:41 AM Patrick M. Hausen wrote: > Hi all, > > in addition to the aborted commands every dozen of system boots or so > (this order of magnitude) the kernel simply hangs during initialisation of > one of the NVME devices: > > https://cloud.hausen.com/s/TxPTDFJwMe6sJr2 > > The particular device affected is not constant. > > A power cycle fixes it, the system has not shown hangs/freezes during > multiuser operation, yet. > > > Any ideas? > What normally comes after the nvme6 line in boot? Often times it's the next thing after the last message that's the issue, not the last thing. Warner From owner-freebsd-stable@freebsd.org Fri Apr 5 14:51:08 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6FE71579604 for ; Fri, 5 Apr 2019 14:51:08 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A97694EE5 for ; Fri, 5 Apr 2019 14:51:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x35Ep4hK038066; Fri, 5 Apr 2019 16:51:04 +0200 (CEST) Received: from [217.29.44.151] ([217.29.44.151]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x35Ep4PP063181; Fri, 5 Apr 2019 16:51:04 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o) From: "Patrick M. Hausen" In-Reply-To: Date: Fri, 5 Apr 2019 16:51:04 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <659E5B01-F6E2-40AC-950D-D8A9833715C9@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <7070F9C7-556F-4BE5-B274-8318CD62E2DD@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 9A97694EE5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.57 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.46)[-0.461,0]; IP_SCORE(-0.30)[ipnet: 217.29.32.0/20(-0.84), asn: 16188(-0.67), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 14:51:08 -0000 Hi! > Am 05.04.2019 um 16:36 schrieb Warner Losh : > What normally comes after the nvme6 line in boot? Often times it's the = next thing after the last message that's the issue, not the last thing. nvme7 ;-) And I had hangs at nvme1, nvme3, =E2=80=A6 as well. Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Fri Apr 5 15:08:19 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23B7C1579BFD for ; Fri, 5 Apr 2019 15:08:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80518959AD for ; Fri, 5 Apr 2019 15:08:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x832.google.com with SMTP id z17so7688788qts.13 for ; Fri, 05 Apr 2019 08:08:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Nh1US6CI4loxkffk3/0LSAd5VaSvnmKkj7E0tL/8sIQ=; b=Zxz7PlSpN/XIYmYywLQaQpo1dINnN/ey4/RDa26V2ajMK4Gtf8gZ8KCLnPy0vC2a7T ylF5QdsCz0C2sQMuhTPcuQUhO43gBvpt0Iu5zT8ny51fHMHt/A2wPB+TgGYss5ioffAk 5oszDcBekkdnlckuyLlmBoUdLoGLJNkW01wvzT0vPNKy7Xemabvyp7h5moBf837UhjVP d4jh6+ZgDPoQtwXOdh7aBNDlibpDyFhQjYQ3c0QCtDRRRP9gXm/R7jAEcGeQgecxYIaI NVykhXpvklt8mGCLVDAM0r5l545Df6sh7woy1dudz+vLFlSKcv3Eh+osykcSq6G67SMp GgHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Nh1US6CI4loxkffk3/0LSAd5VaSvnmKkj7E0tL/8sIQ=; b=JpMgFTvWIaozWCaptsATp23vGZujtNfWNOHwCwcsiQDFBlm2ZmFM9SCjl31hsqCscp vJ9eaHudH+/4r3o2MykWiLxEbQrCo0aTyEYvehIcgA664/fZ8h8ZaA5sFnMioegUcPAl 7LMD2f0Trs6dPj+uWO3TaZXEDt03NXdn2kdftFcOnYwixLl8H8pCfnRl66oLBJcT8t51 pPl4COWURVy94CXbnWbfQ9OmN/dMYqUrDmQ/I2WmUXgkKVGT1ztg4UBrp/UeLCQ0h/d8 2mz7MIBrZKvfhz2LZdLoltsc7Wg4BzMcGWkuX3tNe7nbpMrTpAwRX+NTL6ocEpHLYL2h KxpQ== X-Gm-Message-State: APjAAAXCgr1A+tbAeFtS1KTlb9i6DJQX1r9A12UlD8W7QYIL/nnxyupn 5TZ7tYVfDu2TQmlCf+dUMmx5ivZK0ENpe4cXLhhxSj1m X-Google-Smtp-Source: APXvYqy2jV8IyxR3rVvA/Ctb2VFh4ow9CqChDQ7J0RXYPU6zpSdp6qVv+IOk05XKQiIe9mM2s0yYoWeGcpmAcutuVQc= X-Received: by 2002:a0c:d4a2:: with SMTP id u31mr10941436qvh.139.1554476897726; Fri, 05 Apr 2019 08:08:17 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> In-Reply-To: <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> From: Warner Losh Date: Fri, 5 Apr 2019 09:08:06 -0600 Message-ID: Subject: Re: NVME aborting outstanding i/o To: "Patrick M. Hausen" Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 80518959AD X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=Zxz7PlSp X-Spamd-Result: default: False [-5.72 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.85)[-0.852,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.86)[ip: (-9.14), ipnet: 2607:f8b0::/32(-2.92), asn: 15169(-2.17), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 15:08:19 -0000 On Fri, Apr 5, 2019 at 1:33 AM Patrick M. Hausen wrote: > Hi all, > > > Am 04.04.2019 um 17:11 schrieb Warner Losh : > > There's a request that was sent down to the drive. It took longer than > 30s to respond. One of them, at least, was a trim request. > > [=E2=80=A6] > > Thanks for the explanation. > > This further explains why I was seeing a lot more of those and the system > occasionally froze for a couple of seconds after I increased these: > > vfs.zfs.vdev.async_write_max_active: 10 > vfs.zfs.vdev.async_read_max_active: 3 > vfs.zfs.vdev.sync_write_max_active: 10 > vfs.zfs.vdev.sync_read_max_active: 10 > > as recommended by Allan Jude reasoning that NVME devices could work on > up to 64 requests in parallel. I have since reverted that change and I am > running with the defaults. > > If I understand correctly, this: > > > hw.nvme.per_cpu_io_queues=3D0 > > essentially limits the rate at which the system throws commands at the > devices. Correct? > Yes. It de-facto limits the number of commands the system can throw at a nvme drive. Some drives have trouble with multiple CPUs submitting things. Others just have trouble with the volume of commands sometimes. This limits both. > So it=E2=80=99s not a real fix and there=E2=80=99s nothing fundamentally = wrong with the > per CPU queue or > interrupt implementation. I will look into new firmware for my Intel > devices and > try tweaking the vfs.zfs.vdev.trim_max_active and related parameters. > Correct. It's a hack-a-around. > Out of curiosity: what happens if I disable TRIM? My knowledge is rather > superficial > and I just filed that under =E2=80=9ETRIM is absolutely essential lest pe= rformance > will > suffer severely and your devices die - plus bad karma, of course =E2=80= =A6=E2=80=9C ;-) > TRIMs help the drive optimize their garbage collection by giving it a larger pool of free blocks to work with. This has the effect of reducing write amplification. Write amp is the measure of the amount of extra work the drive has to do for every user write it processes. Ideally, you want this number to be 1.0. You'll never get to 1.0, but numbers less than 1.5 are common and most of the models drive makers use to rate the lifetime of their NAND assume a write amp of about 2. So, if you eliminate the TRIMs you eliminate this optimization and write amp will increase. This has two bad effects. First, wear and tear on the NAND. Second, it takes resources away from the user. In practice, however, the bad effects are quite limited if you don't have a write intensive workload. Your drive is rated for so many drive writes per day (or equivalently total data written over the life of the drive). This will be on the spec sheet somewhere. If you don't have a write intensive workload (which I'd say is any sustained write load greater than about 1/10th the datasheet write limit), then if you think TRIMs are causing issues, you should disable them. The effects of not trimming are likely to be in the noise on such systems, and the benefits of having things TRIMed will be less. At work, for a large video streaming company, we enable the TRIMs, even though we're on the edge of the rule of thumb since we're unsure how long the machines really need to be in the field and don't want to risk it. Except for the version of Samsung nvme drives (PM963, no longer made) we got a while ago... those we turn TRIM off on because UFS' machine-gunning down of TRIMs and nvd's blind pass-through of TRIMs took down the drive. UFS now combines TRIMs and we've moved to using nda since it also combines TRIMs and it won't be so bad if we tried again today. Drive makers optimize different things. Enterprise drives handle TRIMs a lot better than consumer drives. consumer drives are cheaper (in oh so many ways), so some care is needed. Intel makes a wide range of drives, from the super duper awesome (with prices to match) to the somewhat disappointing (but incredibly cheap and good enough for a lot of applicaitons). Not sure where on this scale your drives fall on this spectrum. tl;dr: Unless you are writing the snot out of those Intel drives, disabling TRIM entirely will likely help avoid pushing so many commands they timeout. Warner From owner-freebsd-stable@freebsd.org Fri Apr 5 15:08:27 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51B8D1579C13 for ; Fri, 5 Apr 2019 15:08:27 +0000 (UTC) (envelope-from mach@swishmail.com) Received: from vorlon.swishmail.com (vorlon.swishmail.com [208.72.56.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7ED5F959B5 for ; Fri, 5 Apr 2019 15:08:26 +0000 (UTC) (envelope-from mach@swishmail.com) Received: (qmail 68322 invoked by uid 89); 5 Apr 2019 15:01:43 -0000 Received: from unknown (HELO ?IPv6:2001:b030:14e:100:adf0:7e78:dd58:9ed?) (mach@swishmail.com@2001:b030:14e:100:adf0:7e78:dd58:9ed) by 2602:ffb8::208:72:56:19 with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256 encrypted, authenticated); 5 Apr 2019 15:01:43 -0000 To: freebsd-stable@freebsd.org From: Kris von Mach Subject: em performs worse than igb (latency wise) in 12? Message-ID: Date: Fri, 5 Apr 2019 23:01:42 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Rspamd-Queue-Id: 7ED5F959B5 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mach@swishmail.com designates 208.72.56.19 as permitted sender) smtp.mailfrom=mach@swishmail.com X-Spamd-Result: default: False [0.11 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.58)[-0.580,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:208.72.56.0/22]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.74)[-0.745,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[swishmail.com]; NEURAL_SPAM_SHORT(0.86)[0.862,0]; MX_GOOD(-0.01)[mxfilter1.nyc.swishmail.com,mxfilter2.nyc.swishmail.com]; IP_SCORE(-0.01)[country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[19.56.72.208.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:14469, ipnet:208.72.56.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 15:08:27 -0000 Hello, I've upgraded from Stable 11 to Stable 12, and noticed that igb has been = removed and is now part of em driver. However, the performance seems a=20 lot worse. This is using HP 366FLR which is just HP's version of Intel=20 i350 I believe. I've tried incorporating the sysctl settings I used in 11 into 12: dev.igb.0.fc=3D0 dev.igb.0.iflib.rx_budget=3D65535 hw.igb.rx_process_limit=3D"-1" hw.igb.num_queues=3D"2" hw.igb.rxd=3D"2048"=C2=A0 # (default 1024) hw.igb.txd=3D"2048"=C2=A0 # (default 1024) hw.igb.max_interrupt_rate=3D"32000" # (default 8000) hw.em.rx_process_limit=3D"-1" hw.em.num_queues=3D"2" hw.em.rxd=3D"2048"=C2=A0 # (default 1024) hw.em.txd=3D"2048"=C2=A0 # (default 1024) hw.em.max_interrupt_rate=3D"32000" But that didn't seem to help. Switching back to 11, of course restored the performance. The performance hit I'm talking about seems to be with latency, not=20 throughput. To give you an example, this server is serving a website that is=20 monitored from different locations and all locations the latency has=20 increased by about 30-50% using stable 12. Any ideas as to how to improve latency in 12? Can you compile igb driver = in 12? Has something else other than the network driver changed in 12=20 that could cause an increase in latency? From owner-freebsd-stable@freebsd.org Fri Apr 5 18:57:03 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4778C1556E3B for ; Fri, 5 Apr 2019 18:57:03 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [31.24.6.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A9606F1C3 for ; Fri, 5 Apr 2019 18:57:01 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from [82.47.240.30] (helo=foula.local) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1hCU1E-000DTO-NC for freebsd-stable@freebsd.org; Fri, 05 Apr 2019 18:56:52 +0000 Subject: Re: em performs worse than igb (latency wise) in 12? To: freebsd-stable@freebsd.org References: From: Pete French Message-ID: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> Date: Fri, 5 Apr 2019 19:56:51 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:67.0) Gecko/20100101 Thunderbird/67.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 9A9606F1C3 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=ingresso.co.uk; spf=pass (mx1.freebsd.org: domain of petefrench@ingresso.co.uk designates 31.24.6.74 as permitted sender) smtp.mailfrom=petefrench@ingresso.co.uk X-Spamd-Result: default: False [-1.62 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.88)[-0.884,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:31.24.6.74]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; IP_SCORE(-0.09)[asn: 16082(-0.36), country: GB(-0.09)]; NEURAL_SPAM_SHORT(0.17)[0.169,0]; MX_GOOD(-0.01)[us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mi mecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com]; DMARC_POLICY_ALLOW(-0.50)[ingresso.co.uk,none]; RECEIVED_SPAMHAUS_PBL(0.00)[30.240.47.82.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:16082, ipnet:31.24.0.0/21, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 18:57:03 -0000 On 05/04/2019 16:01, Kris von Mach wrote: > I've upgraded from Stable 11 to Stable 12, and noticed that igb has been > removed and is now part of em driver. However, the performance seems a > lot worse. This is using HP 366FLR which is just HP's version of Intel > i350 I believe. Something odd going on there there - I am using 12-STABLE and I have igb just fine, and it attaches to the same hardware that 11 did: ... [petefrench@dilbert ~]$ ssh turpentine Last login: Fri Apr 5 18:52:50 2019 from 2a02:b90:3002:411::6 FreeBSD 12.0-STABLE r343538 GENERIC Baby baby turpentine... [webadmin@turpentine ~]$ ifconfig igb0 igb0: flags=8843 metric 0 mtu 1500 options=e507bb ether ac:1f:6b:46:5e:32 inet 10.32.10.5 netmask 0xffff0000 broadcast 10.32.255.255 inet6 fe80::ae1f:6bff:fe46:5e32%igb0 prefixlen 64 scopeid 0x1 inet6 2a02:1658:1:2:e550::5 prefixlen 64 media: Ethernet autoselect (1000baseT ) status: active nd6 options=21 Do you have a custom kernel, and if so did you see this note in UPDATING? 20170109: The igb(4), em(4) and lem(4) ethernet drivers are now implemented via IFLIB. If you have a custom kernel configuration that excludes em(4) but you use igb(4), you need to re-add em(4) to your custom configuration. From owner-freebsd-stable@freebsd.org Fri Apr 5 19:12:27 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C1A41558690 for ; Fri, 5 Apr 2019 19:12:27 +0000 (UTC) (envelope-from mach@swishmail.com) Received: from vorlon.swishmail.com (vorlon.swishmail.com [208.72.56.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB6F56FBDA for ; Fri, 5 Apr 2019 19:12:26 +0000 (UTC) (envelope-from mach@swishmail.com) Received: (qmail 89776 invoked by uid 89); 5 Apr 2019 19:12:25 -0000 Received: from unknown (HELO ?IPv6:2001:b030:14e:100:83e:cc03:35:83f6?) (mach@swishmail.com@2001:b030:14e:100:83e:cc03:35:83f6) by 2602:ffb8::208:72:56:19 with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256 encrypted, authenticated); 5 Apr 2019 19:12:25 -0000 Subject: Re: em performs worse than igb (latency wise) in 12? To: freebsd-stable@freebsd.org References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> From: Kris von Mach Message-ID: Date: Sat, 6 Apr 2019 03:12:24 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Rspamd-Queue-Id: AB6F56FBDA X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mach@swishmail.com designates 208.72.56.19 as permitted sender) smtp.mailfrom=mach@swishmail.com X-Spamd-Result: default: False [-0.32 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:208.72.56.0/22]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.00)[0.003,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.75)[-0.749,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[swishmail.com]; MX_GOOD(-0.01)[cached: mxfilter1.nyc.swishmail.com]; NEURAL_HAM_SHORT(-0.15)[-0.147,0]; IP_SCORE(-0.01)[country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[19.56.72.208.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:14469, ipnet:208.72.56.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 19:12:27 -0000 On 4/6/2019 2:56 AM, Pete French wrote: > Something odd going on there there - I am using 12-STABLE and I have=20 > igb just fine, and it attaches to the same hardware that 11 did: It does work in 12, throughput is great, just that the latency is higher = than 11. igb0: flags=3D8843 metric 0 mtu 1= 500 options=3De527bb =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ether 38:ea:a7:8d:c1:6c =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet 208.72.56.19 netmask 0xf= ffffc00 broadcast 208.72.59.255 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet6 fe80::3aea:a7ff:fe8d:c1= 6c%igb0 prefixlen 64 scopeid 0x1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet6 2602:ffb8::208:72:56:9 = prefixlen 64 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 media: Ethernet autoselect (1= 000baseT ) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: active =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 nd6 options=3D21 > Do you have a custom kernel, and if so did you see this note in UPDATIN= G? Yes I do, but it includes all of GENERIC which includes em drivers,=20 otherwise it wouldn't even work with the network card. my custom kernel: include GENERIC ident=C2=A0=C2=A0 CUSTOM makeoptions WITH_EXTRA_TCP_STACKS=3D1 options TCPHPTS options SC_KERNEL_CONS_ATTR=3D(FG_GREEN|BG_BLACK) options IPSTEALTH options=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 AHC_REG_PRETTY_PRINT=C2=A0 # = Print register bitfields in debug options=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 AHD_REG_PRETTY_PRINT=C2=A0 # = Print register bitfields in debug device cryptodev device aesni I did try without RACK just in case that was the culprit. From owner-freebsd-stable@freebsd.org Sat Apr 6 12:58:43 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D292155C2AE for ; Sat, 6 Apr 2019 12:58:43 +0000 (UTC) (envelope-from mach@swishmail.com) Received: from vorlon.swishmail.com (vorlon.swishmail.com [208.72.56.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9BBC3774F3 for ; Sat, 6 Apr 2019 12:58:42 +0000 (UTC) (envelope-from mach@swishmail.com) Received: (qmail 28862 invoked by uid 89); 6 Apr 2019 12:58:41 -0000 Received: from unknown (HELO ?IPv6:2001:b030:14e:100:83e:cc03:35:83f6?) (mach@swishmail.com@2001:b030:14e:100:83e:cc03:35:83f6) by 2602:ffb8::208:72:56:19 with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256 encrypted, authenticated); 6 Apr 2019 12:58:41 -0000 From: Kris von Mach Subject: Re: em performs worse than igb (latency wise) in 12? To: freebsd-stable@freebsd.org References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> Message-ID: Date: Sat, 6 Apr 2019 20:58:39 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: base64 Content-Language: en-US X-Rspamd-Queue-Id: 9BBC3774F3 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mach@swishmail.com designates 208.72.56.19 as permitted sender) smtp.mailfrom=mach@swishmail.com X-Spamd-Result: default: False [-0.61 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.20)[-0.205,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:208.72.56.0/22]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.87)[-0.868,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[swishmail.com]; MX_GOOD(-0.01)[cached: mxfilter1.nyc.swishmail.com]; MIME_BASE64_TEXT(0.10)[]; NEURAL_HAM_SHORT(-0.21)[-0.210,0]; IP_SCORE(-0.01)[country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[19.56.72.208.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:14469, ipnet:208.72.56.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 12:58:43 -0000 T24gNC82LzIwMTkgMjo1NiBBTSwgUGV0ZSBGcmVuY2ggd3JvdGU6DQo+IFNvbWV0aGluZyBv ZGQgZ29pbmcgb24gdGhlcmUgdGhlcmUgLSBJIGFtIHVzaW5nIDEyLVNUQUJMRSBhbmQgSSBo YXZlIA0KPiBpZ2IganVzdCBmaW5lLCBhbmQgaXQgYXR0YWNoZXMgdG8gdGhlIHNhbWUgaGFy ZHdhcmUgdGhhdCAxMSBkaWQ6DQoNCkkgcmFuIGFwYWNoZSBiZW5jaCwgYW5kIEkgZ290IGEg cmVzdWx0IG9mIDEwMCByZXF1ZXN0cy9zZWMgb24gMTItU1RBQkxFIA0KdnMgMTYsMDAwIHJl cXVlc3RzL3NlYyBvbiAxMS1TVEFCTEUuIFNvIHNvbWV0aGluZyBpcyBkZWZpbml0ZWx5IHdy b25nLiANCk5vdGhpbmcgY2hhbmdlZCBvdGhlciB0aGFuIGdvaW5nIGZyb20gMTEgdG8gMTIu DQoNCmFiIG9uIDEyLVNUQUJMRToNCg0KYWIgLW4gMTAwIC1jIDEwIGh0dHA6Ly9pcC8NClRo aXMgaXMgQXBhY2hlQmVuY2gsIFZlcnNpb24gMi4zIDwkUmV2aXNpb246IDE4NDM0MTIgJD4N CkNvcHlyaWdodCAxOTk2IEFkYW0gVHdpc3MsIFpldXMgVGVjaG5vbG9neSBMdGQsIGh0dHA6 Ly93d3cuemV1c3RlY2gubmV0Lw0KTGljZW5zZWQgdG8gVGhlIEFwYWNoZSBTb2Z0d2FyZSBG b3VuZGF0aW9uLCBodHRwOi8vd3d3LmFwYWNoZS5vcmcvDQoNCkJlbmNobWFya2luZyBpcCAo YmUgcGF0aWVudCkuLi4uLmRvbmUNCg0KDQpTZXJ2ZXIgU29mdHdhcmU6wqDCoMKgwqDCoMKg wqAgbmdpbngNClNlcnZlciBIb3N0bmFtZTrCoMKgwqDCoMKgwqDCoCBpcA0KU2VydmVyIFBv cnQ6wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA4MA0KDQpEb2N1bWVudCBQYXRoOsKgwqDCoMKg wqDCoMKgwqDCoCAvDQpEb2N1bWVudCBMZW5ndGg6wqDCoMKgwqDCoMKgwqAgMTYyNDcgYnl0 ZXMNCg0KQ29uY3VycmVuY3kgTGV2ZWw6wqDCoMKgwqDCoCAxMA0KVGltZSB0YWtlbiBmb3Ig dGVzdHM6wqDCoCAxLjAxMyBzZWNvbmRzDQpDb21wbGV0ZSByZXF1ZXN0czrCoMKgwqDCoMKg IDEwMA0KRmFpbGVkIHJlcXVlc3RzOsKgwqDCoMKgwqDCoMKgIDANClRvdGFsIHRyYW5zZmVy cmVkOsKgwqDCoMKgwqAgMTY0OTkwMCBieXRlcw0KSFRNTCB0cmFuc2ZlcnJlZDrCoMKgwqDC oMKgwqAgMTYyNDcwMCBieXRlcw0KUmVxdWVzdHMgcGVyIHNlY29uZDrCoMKgwqAgOTguNjkg WyMvc2VjXSAobWVhbikNClRpbWUgcGVyIHJlcXVlc3Q6wqDCoMKgwqDCoMKgIDEwMS4zMzEg W21zXSAobWVhbikNClRpbWUgcGVyIHJlcXVlc3Q6wqDCoMKgwqDCoMKgIDEwLjEzMyBbbXNd IChtZWFuLCBhY3Jvc3MgYWxsIGNvbmN1cnJlbnQgcmVxdWVzdHMpDQpUcmFuc2ZlciByYXRl OsKgwqDCoMKgwqDCoMKgwqDCoCAxNTkwLjA3IFtLYnl0ZXMvc2VjXSByZWNlaXZlZA0KDQpD b25uZWN0aW9uIFRpbWVzIChtcykNCiDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCBtaW7C oCBtZWFuWysvLXNkXSBtZWRpYW7CoMKgIG1heA0KQ29ubmVjdDrCoMKgwqDCoMKgwqDCoCAw wqDCoMKgIDDCoMKgIDAuMsKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoCAxDQpQcm9jZXNzaW5n OsKgwqAgMTAwwqAgMTAxwqDCoCAwLjXCoMKgwqAgMTAxwqDCoMKgwqAgMTAyDQpXYWl0aW5n OsKgwqDCoMKgwqDCoMKgIDDCoMKgwqAgMMKgwqAgMC4xwqDCoMKgwqDCoCAwwqDCoMKgwqDC oMKgIDENClRvdGFsOsKgwqDCoMKgwqDCoMKgIDEwMMKgIDEwMcKgwqAgMC42wqDCoMKgIDEw McKgwqDCoMKgIDEwMg0KDQpQZXJjZW50YWdlIG9mIHRoZSByZXF1ZXN0cyBzZXJ2ZWQgd2l0 aGluIGEgY2VydGFpbiB0aW1lIChtcykNCiDCoCA1MCXCoMKgwqAgMTAxDQogwqAgNjYlwqDC oMKgIDEwMQ0KIMKgIDc1JcKgwqDCoCAxMDENCiDCoCA4MCXCoMKgwqAgMTAyDQogwqAgOTAl wqDCoMKgIDEwMg0KIMKgIDk1JcKgwqDCoCAxMDINCiDCoCA5OCXCoMKgwqAgMTAyDQogwqAg OTklwqDCoMKgIDEwMg0KIMKgMTAwJcKgwqDCoCAxMDIgKGxvbmdlc3QgcmVxdWVzdCkNCg0K YWIgb24gMTEtU1RBQkxFOg0KDQphYiAtbiAxMDAgLWMgMTAgaHR0cDovL2lwLw0KVGhpcyBp cyBBcGFjaGVCZW5jaCwgVmVyc2lvbiAyLjMgPCRSZXZpc2lvbjogMTg0MzQxMiAkPg0KQ29w eXJpZ2h0IDE5OTYgQWRhbSBUd2lzcywgWmV1cyBUZWNobm9sb2d5IEx0ZCwgaHR0cDovL3d3 dy56ZXVzdGVjaC5uZXQvDQpMaWNlbnNlZCB0byBUaGUgQXBhY2hlIFNvZnR3YXJlIEZvdW5k YXRpb24sIGh0dHA6Ly93d3cuYXBhY2hlLm9yZy8NCg0KQmVuY2htYXJraW5nIGlwIChiZSBw YXRpZW50KS4uLi4uZG9uZQ0KDQoNClNlcnZlciBTb2Z0d2FyZTrCoMKgwqDCoMKgwqDCoCBu Z2lueA0KU2VydmVyIEhvc3RuYW1lOsKgwqDCoMKgwqDCoMKgIGlwDQpTZXJ2ZXIgUG9ydDrC oMKgwqDCoMKgwqDCoMKgwqDCoMKgIDgwDQoNCkRvY3VtZW50IFBhdGg6wqDCoMKgwqDCoMKg wqDCoMKgIC8NCkRvY3VtZW50IExlbmd0aDrCoMKgwqDCoMKgwqDCoCAxNjI0NyBieXRlcw0K DQpDb25jdXJyZW5jeSBMZXZlbDrCoMKgwqDCoMKgIDEwDQpUaW1lIHRha2VuIGZvciB0ZXN0 czrCoMKgIDAuMDA2IHNlY29uZHMNCkNvbXBsZXRlIHJlcXVlc3RzOsKgwqDCoMKgwqAgMTAw DQpGYWlsZWQgcmVxdWVzdHM6wqDCoMKgwqDCoMKgwqAgMA0KVG90YWwgdHJhbnNmZXJyZWQ6 wqDCoMKgwqDCoCAxNjUxNDAwIGJ5dGVzDQpIVE1MIHRyYW5zZmVycmVkOsKgwqDCoMKgwqDC oCAxNjI0NzAwIGJ5dGVzDQpSZXF1ZXN0cyBwZXIgc2Vjb25kOsKgwqDCoCAxNjEyMy44MyBb Iy9zZWNdIChtZWFuKQ0KVGltZSBwZXIgcmVxdWVzdDrCoMKgwqDCoMKgwqAgMC42MjAgW21z XSAobWVhbikNClRpbWUgcGVyIHJlcXVlc3Q6wqDCoMKgwqDCoMKgIDAuMDYyIFttc10gKG1l YW4sIGFjcm9zcyBhbGwgY29uY3VycmVudCByZXF1ZXN0cykNClRyYW5zZmVyIHJhdGU6wqDC oMKgwqDCoMKgwqDCoMKgIDI2MDAyOC4yNyBbS2J5dGVzL3NlY10gcmVjZWl2ZWQNCg0KQ29u bmVjdGlvbiBUaW1lcyAobXMpDQogwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgbWluwqAg bWVhblsrLy1zZF0gbWVkaWFuwqDCoCBtYXgNCkNvbm5lY3Q6wqDCoMKgwqDCoMKgwqAgMMKg wqDCoCAwwqDCoCAwLjDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqAgMA0KUHJvY2Vzc2luZzrC oMKgwqDCoCAwwqDCoMKgIDDCoMKgIDAuMcKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoCAxDQpX YWl0aW5nOsKgwqDCoMKgwqDCoMKgIDDCoMKgwqAgMMKgwqAgMC4wwqDCoMKgwqDCoCAwwqDC oMKgwqDCoMKgIDENClRvdGFsOsKgwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgIDHCoMKgIDAu MMKgwqDCoMKgwqAgMcKgwqDCoMKgwqDCoCAxDQoNClBlcmNlbnRhZ2Ugb2YgdGhlIHJlcXVl c3RzIHNlcnZlZCB3aXRoaW4gYSBjZXJ0YWluIHRpbWUgKG1zKQ0KIMKgIDUwJcKgwqDCoMKg wqAgMQ0KIMKgIDY2JcKgwqDCoMKgwqAgMQ0KIMKgIDc1JcKgwqDCoMKgwqAgMQ0KIMKgIDgw JcKgwqDCoMKgwqAgMQ0KIMKgIDkwJcKgwqDCoMKgwqAgMQ0KIMKgIDk1JcKgwqDCoMKgwqAg MQ0KIMKgIDk4JcKgwqDCoMKgwqAgMQ0KIMKgIDk5JcKgwqDCoMKgwqAgMQ0KIMKgMTAwJcKg wqDCoMKgwqAgMSAobG9uZ2VzdCByZXF1ZXN0KQ0KDQo= From owner-freebsd-stable@freebsd.org Sat Apr 6 20:22:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09ED7156BDC1 for ; Sat, 6 Apr 2019 20:22:42 +0000 (UTC) (envelope-from imb@protected-networks.net) Received: from mail.protected-networks.net (mail.protected-networks.net [202.12.127.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.protected-networks.net", Issuer "Protected Networks CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3D2686AA86 for ; Sat, 6 Apr 2019 20:22:38 +0000 (UTC) (envelope-from imb@protected-networks.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= protected-networks.net; h=content-transfer-encoding :content-language:content-type:content-type:in-reply-to :mime-version:user-agent:date:date:message-id:from:from :references:subject:subject; s=201508; t=1554582111; bh=yrKsfmN8 keo349Wl+AC+l7BYVQFHLzadwgaGuHDUn4U=; b=e7eSVsMIY7oJajJdjA1Gt5Vx Zy+GvA1m5bewE6Gqyf2GkxuCZDvTxFHJzFIkzMlEZHbaUEr6U/ICoFdsBfZl+mr5 V6xtRBrMZjY6C8bzpMWN8JtRsgcZCTueBJdnYsNQRccqBBnFsyWtNIRbr7nLursT W+ujzLvtGhhpbq2oWr0= Received: from toshi.auburn.protected-networks.net (toshi.auburn.protected-networks.net [192.168.1.10]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: imb@mail.protected-networks.net) by mail.protected-networks.net (Postfix) with ESMTPSA id E33071087; Sat, 6 Apr 2019 16:21:51 -0400 (EDT) Subject: Re: em performs worse than igb (latency wise) in 12? To: Kris von Mach , freebsd-stable@freebsd.org References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> From: Michael Butler Openpgp: preference=signencrypt Message-ID: <4f9b9259-f5a1-ecc6-366e-4a26de0ca3dc@protected-networks.net> Date: Sat, 6 Apr 2019 16:21:41 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 3D2686AA86 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=protected-networks.net header.s=201508 header.b=e7eSVsMI; spf=pass (mx1.freebsd.org: domain of imb@protected-networks.net designates 202.12.127.228 as permitted sender) smtp.mailfrom=imb@protected-networks.net X-Spamd-Result: default: False [-0.92 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[protected-networks.net:s=201508]; NEURAL_HAM_MEDIUM(-0.70)[-0.702,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[protected-networks.net]; NEURAL_SPAM_SHORT(0.30)[0.296,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[protected-networks.net:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[sarah.protected-networks.net,mail.protected-networks.net,sarah.protected-networks.net,mail.protected-networks.net]; IP_SCORE(-0.01)[country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5716, ipnet:202.12.127.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 20:22:42 -0000 On 2019-04-06 08:58, Kris von Mach wrote: > On 4/6/2019 2:56 AM, Pete French wrote: >> Something odd going on there there - I am using 12-STABLE and I have >> igb just fine, and it attaches to the same hardware that 11 did: > > I ran apache bench, and I got a result of 100 requests/sec on 12-STABLE > vs 16,000 requests/sec on 11-STABLE. So something is definitely wrong. > Nothing changed other than going from 11 to 12. I'd be interested to see if substituting the port net/intel-em-kmod has any effect on the issue, imb From owner-freebsd-stable@freebsd.org Sat Apr 6 22:49:17 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9942615700E7 for ; Sat, 6 Apr 2019 22:49:17 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17B586FB46 for ; Sat, 6 Apr 2019 22:49:17 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it1-f178.google.com (mail-it1-f178.google.com [209.85.166.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id D39B812E54 for ; Sat, 6 Apr 2019 22:49:16 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it1-f178.google.com with SMTP id v8so13859955itf.0 for ; Sat, 06 Apr 2019 15:49:16 -0700 (PDT) X-Gm-Message-State: APjAAAUggWBf+GtJd+al4mksLkvHUH+/1yJSzB/SgV/+a75W0H44TaeA HjsrKiA7dYN2B6kbQEc2XYrxstPyeFY1szsu01Y= X-Google-Smtp-Source: APXvYqwHZGcODV5jTajn9z3ziPL9LO7eRM4eEbyoOXl47l2vC3iZMgbX6PZYpaIFj/frltqkQFKtJCTwgF1VMYdVMqA= X-Received: by 2002:a02:90c6:: with SMTP id c6mr14873018jag.117.1554590956234; Sat, 06 Apr 2019 15:49:16 -0700 (PDT) MIME-Version: 1.0 References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> <4f9b9259-f5a1-ecc6-366e-4a26de0ca3dc@protected-networks.net> In-Reply-To: <4f9b9259-f5a1-ecc6-366e-4a26de0ca3dc@protected-networks.net> From: Matthew Macy Date: Sat, 6 Apr 2019 15:49:05 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: em performs worse than igb (latency wise) in 12? To: Michael Butler Cc: Kris von Mach , freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 17B586FB46 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 22:49:17 -0000 On Sat, Apr 6, 2019 at 1:23 PM Michael Butler wrote: > > On 2019-04-06 08:58, Kris von Mach wrote: > > On 4/6/2019 2:56 AM, Pete French wrote: > >> Something odd going on there there - I am using 12-STABLE and I have > >> igb just fine, and it attaches to the same hardware that 11 did: > > > > I ran apache bench, and I got a result of 100 requests/sec on 12-STABLE > > vs 16,000 requests/sec on 11-STABLE. So something is definitely wrong. > > Nothing changed other than going from 11 to 12. > > I'd be interested to see if substituting the port net/intel-em-kmod has > any effect on the issue, I would as well. igb, em, and lem are all the same driver in 12. This makes maintenance a lot easier. However, the older NICs have a lot of errata workarounds that aren't explicitly commented as such. My first guess is this card suffers from one such errata workaround that has been dropped in the update. -M