From owner-freebsd-net@freebsd.org Mon Jan 30 14:36:44 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 210BECC89F5 for ; Mon, 30 Jan 2017 14:36:44 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wj0-x242.google.com (mail-wj0-x242.google.com [IPv6:2a00:1450:400c:c01::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8F5DCCF for ; Mon, 30 Jan 2017 14:36:43 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wj0-x242.google.com with SMTP id kq3so8009562wjc.3 for ; Mon, 30 Jan 2017 06:36:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=TUkVCyr/pw8VIJFEXUzK1hjmUuRB+SFqiKqBaJA7rwc=; b=s71+8QMMWDiSVStZ73gKh5f5/mp2wQzXhwVqvSXfoBJQK9axGlyneIM8oqg7VtlIpA oBhGyhqDY/0CdSfASoOxE/pRyKSOFhZO/C7l/ZyKhlqTXyJPIBGeHsdWzshm9XY7pTwT cBGVrqhxb/Yurwn/Dpi+PUp3Sto42NAMAbQyTjLPElIjZDK5U5jE7sTPURoXchoG3Pxn 2Zx0xgClfCDT1XfKPGm/VGZ63T+wZZIMwlN2K4U+WkCpzsmSoJmriD1LdZh2P1lTBult o6V8I9bwYa5x1ukNxT1FR8yBeV38KRnbRnIhTRU29ZRdC7QyWtBVzuRnQmP2heTJc8In v84A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=TUkVCyr/pw8VIJFEXUzK1hjmUuRB+SFqiKqBaJA7rwc=; b=f9uvxwkbJFAF8B/e+RJGe2fDBN5KfzpK9kcGAFDO8GJPLGCw6yzHSXTqAhHzjTMm6c k5HCLzW5pJi6OR1aTP0lP4XQvcoOCqA0mcbuSt1gLbnxwp6AGB2orb6XfHl98v6Q0T86 hwhY1kbe7Q4dLrJP9HRxYMlBMrj5cHk0Xh8gd0kSV4jY8xrCnaHyrirokMuDrILbQw9L J8DXjlvOolh0kF0sZGBTQWy6cqYDdEM/lPu2OKNerM1aopAd6LZUQ+DMQEnw3CD1uk26 tb1eE2BpWJfcxj1ojfPsRjqF5lwg1hYUfNXUrBPb7n7hsWWd+PHD8w3trXasRx5+58A+ 1ftQ== X-Gm-Message-State: AIkVDXKUKPPK9d9PTxJ1Y1juLnRq+GH9yMF5DSW6A2QsTWdwGT2UFX7qH86nbAqphFuq1w== X-Received: by 10.223.138.220 with SMTP id z28mr18571016wrz.26.1485787002187; Mon, 30 Jan 2017 06:36:42 -0800 (PST) Received: from ben.home ([2a01:cb1d:125:8a00:157e:8c8:56b6:3ae7]) by smtp.gmail.com with ESMTPSA id e72sm19062696wma.16.2017.01.30.06.36.41 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 30 Jan 2017 06:36:41 -0800 (PST) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: iSCSI failing, MLX rx_ring errors ? From: Ben RUBSON In-Reply-To: Date: Mon, 30 Jan 2017 15:36:39 +0100 Cc: Hans Petter Selasky , Yuval Bason , Meny Yossefi Content-Transfer-Encoding: quoted-printable Message-Id: <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> To: "freebsd-net@freebsd.org" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2017 14:36:44 -0000 > On 04 Jan 2017, at 14:47, Ben RUBSON wrote: >=20 >> On 03 Jan 2017, at 07:27, Meny Yossefi wrote: >>=20 >>> From: owner-freebsd-net@freebsd.orgOn Behalf OfBen RUBSON >>> Sent: Monday, January 2, 2017 11:09:15 AM (UTC+00:00) Monrovia, = Reykjavik >>> To: freebsd-net@freebsd.org >>> Cc: Meny Yossefi; Yuval Bason; Hans Petter Selasky >>> Subject: Re: iSCSI failing, MLX rx_ring errors ? >>>=20 >>> Hi Meny, >>>=20 >>> Thank you very much for your feedback. >>>=20 >>> I think you are right, this could be a mbufs issue. >>> Here are some more numbers : >>>=20 >>> # vmstat -z | grep -v "0, 0$" >>> ITEM SIZE LIMIT USED FREE REQ = FAIL SLEEP >>> 4 Bucket: 32, 0, 2673, 28327, 88449799, = 17317, 0 >>> 8 Bucket: 64, 0, 449, 15609, 13926386, = 4871, 0 >>> 12 Bucket: 96, 0, 335, 5323, 10293892, = 142872, 0 >>> 16 Bucket: 128, 0, 533, 6070, 7618615, = 472647, 0 >>> 32 Bucket: 256, 0, 8317, 22133, 36020376, = 563479, 0 >>> 64 Bucket: 512, 0, 1238, 3298, 20138111, = 11430742, 0 >>> 128 Bucket: 1024, 0, 1865, 2963, 21162182, = 158752, 0 >>> 256 Bucket: 2048, 0, 1626, 450, 80253784, = 4890164, 0 >>> mbuf_jumbo_9k: 9216, 603712, 16400, 8744, 4128521064, = 2661, 0 >>=20 >> Have you tried increasing the mbufs limit?=20 >> (sysctl) kern.ipc.nmbufs (Maximum number of mbufs allowed) >=20 > Thank you for your suggestion Meny. > No I did not try this yet. > However, from the numbers above (and below), I think I should increase = kern.ipc.nmbjumbo9 instead ? So, to give some news, increasing kern.ipc.nmbjumbo9 helped a lot. Just a very little issue (compared to the others before) over the last 3 = weeks. > # vmstat -z | grep -E "ITEM|mbuf" > ITEM SIZE LIMIT USED FREE REQ FAIL = SLEEP > mbuf_packet: 256, 26080395, 16400, 10418, 572292683, 0, 0 > mbuf: 256, 26080395, 16402, 40525, 20955366061, 0, 0 > mbuf_cluster: 2048, 4075060, 26818, 148, 907005, 0, 0 > mbuf_jumbo_page: 4096, 2037529, 0, 34262, 5194563127, 0, 0 > mbuf_jumbo_9k: 9216, 603712, 16400, 12867, 4362104082, 2676, 0 > mbuf_jumbo_16k: 16384, 339588, 0, 0, 0, 0, 0 >=20 > # sysctl kern.ipc | grep mb > kern.ipc.nmbufs: 26080380 > kern.ipc.nmbclusters: 4075058 > kern.ipc.nmbjumbop: 2037529 > kern.ipc.nmbjumbo9: 1811136 > kern.ipc.nmbjumbo16: 1358352 > kern.ipc.maxmbufmem: 33382879232 >=20 > // note that I don't understand the difference between vmstat and = sysctl nmbjumbo9 / nmbjumbo16 values. > // the first one is /3 (1811136/3=3D603712) the second one is /4 = (1358352/4=3D339588). Strange. Any idea regarding this difference ? In addition : # sysctl kern.ipc.nmbjumbo9 kern.ipc.nmbjumbo9: 1811136 # sysctl kern.ipc.nmbjumbo9=3D2037529 kern.ipc.nmbjumbo9: 1811136 -> 6112587 Really strange. Thank you, Ben