From owner-freebsd-net@freebsd.org Wed Jan 4 13:47:50 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E89AC9E1F2 for ; Wed, 4 Jan 2017 13:47:50 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B33BF18BE for ; Wed, 4 Jan 2017 13:47:49 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x241.google.com with SMTP id m203so91549367wma.3 for ; Wed, 04 Jan 2017 05:47:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=S77rl5l2abvQ/ttlAFj8IkyJu6Ihv3MzryH9M/I25us=; b=RC0JSPF2CNiEK9lzbsawbfJzOKhpzUQwatRV86MfogdGSgv1aii591O5B+DYGjHUDB 3QwssN+WQBy66XwKTePUjHBhxuxNFN40TOjkG/nDd7/6pxzAxA0wDoUBw8/7wRnqaopr leAYUx9eHNuDW9TpudHFhG+pPhd7oA6giyfoYDaFFt54vgvGfMHCwDf/P3quZlV2louo AsmfDYGjdlueEA11v5dGbNSDnM8L4lkpXAoaRX1aXE7dVOvWPrOTYB+PWzJW+7L64A6D uTp0+uIrbW98ez1pKDbXqDPYthixm9NmEiWXaq3XHNv1DEPWOZePZDZxxbg9qOqYoGQk UBAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=S77rl5l2abvQ/ttlAFj8IkyJu6Ihv3MzryH9M/I25us=; b=WbvjjUwaUbbMlCdGDf5QM8+InNeL7ENfVPvrD7rnXRivPDFqRzeYJM+4rLrj/wlR69 PCyaKQe5YnPHxUKIgTG1sOxDqiorIsmdZcCszPunOG1zG0jlEG00wtTFJVfgm1/lXKSj obiQ8dfD9slDoFDOvxjImlt+ZASAnjSCCQtwXbCrWqPZKbMwqCd1jabvwkMhMR0CxsoA GdIEAMZsqP6JDpMvQozX85Fotl6cHcv+Teg9lA6wGoUet7XnwfjjJaPTPO9Y8kie4U2q 7hw6niof2ey9OpAxvTrWIZgnbhkBtcE+JUozMh/RAIoZzgX82sAyVTk6To4Uk+5+j03y h6SA== X-Gm-Message-State: AIkVDXILfpJXV6hWN1e3tS5wLTKJsROtb/1BUYwBU/G/QdkjgzmQQgotla5kNquG52nssg== X-Received: by 10.28.45.212 with SMTP id t203mr58841457wmt.46.1483537667798; Wed, 04 Jan 2017 05:47:47 -0800 (PST) Received: from ben.home (LFbn-1-7159-4.w90-116.abo.wanadoo.fr. [90.116.90.4]) by smtp.gmail.com with ESMTPSA id g184sm94948273wme.23.2017.01.04.05.47.46 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 04 Jan 2017 05:47:47 -0800 (PST) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: iSCSI failing, MLX rx_ring errors ? From: Ben RUBSON In-Reply-To: Date: Wed, 4 Jan 2017 14:47:45 +0100 Cc: Hans Petter Selasky , Yuval Bason , Meny Yossefi Content-Transfer-Encoding: quoted-printable Message-Id: References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> To: "freebsd-net@freebsd.org" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2017 13:47:50 -0000 > On 03 Jan 2017, at 07:27, Meny Yossefi wrote: >=20 >> From: owner-freebsd-net@freebsd.orgOn Behalf OfBen RUBSON >> Sent: Monday, January 2, 2017 11:09:15 AM (UTC+00:00) Monrovia, = Reykjavik >> To: freebsd-net@freebsd.org >> Cc: Meny Yossefi; Yuval Bason; Hans Petter Selasky >> Subject: Re: iSCSI failing, MLX rx_ring errors ? >>=20 >> Hi Meny, >>=20 >> Thank you very much for your feedback. >>=20 >> I think you are right, this could be a mbufs issue. >> Here are some more numbers : >>=20 >> # vmstat -z | grep -v "0, 0$" >> ITEM SIZE LIMIT USED FREE REQ = FAIL SLEEP >> 4 Bucket: 32, 0, 2673, 28327, 88449799, = 17317, 0 >> 8 Bucket: 64, 0, 449, 15609, 13926386, = 4871, 0 >> 12 Bucket: 96, 0, 335, 5323, 10293892, = 142872, 0 >> 16 Bucket: 128, 0, 533, 6070, 7618615, = 472647, 0 >> 32 Bucket: 256, 0, 8317, 22133, 36020376, = 563479, 0 >> 64 Bucket: 512, 0, 1238, 3298, 20138111, = 11430742, 0 >> 128 Bucket: 1024, 0, 1865, 2963, 21162182, = 158752, 0 >> 256 Bucket: 2048, 0, 1626, 450, 80253784, = 4890164, 0 >> mbuf_jumbo_9k: 9216, 603712, 16400, 8744, 4128521064, = 2661, 0 >>=20 >> # netstat -m >> 32801/18814/51615 mbufs in use (current/cache/total) >> 16400/9810/26210/4075058 mbuf clusters in use = (current/cache/total/max) >> 16400/9659 mbuf+clusters out of packet secondary zone in use = (current/cache) >> 0/8647/8647/2037529 4k (page size) jumbo clusters in use = (current/cache/total/max) >> 16400/8744/25144/603712 9k jumbo clusters in use = (current/cache/total/max) >> 0/0/0/339588 16k jumbo clusters in use (current/cache/total/max) = 188600K/137607K/326207K bytes allocated to network (current/cache/total) >> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) >> 0/2661/0 requests for jumbo clusters denied (4k/9k/16k) >> 0 sendfile syscalls >> 0 sendfile syscalls completed without I/O request >> 0 requests for I/O initiated by sendfile >> 0 pages read by sendfile as part of a request >> 0 pages were valid at time of a sendfile request >> 0 pages were requested for read ahead by applications >> 0 pages were read ahead by sendfile >> 0 times sendfile encountered an already busy page >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >>=20 >> I did not perform any mbufs tuning, numbers above are from FreeBSD = itself. >>=20 >> This server has 64GB of memory. >> It has a ZFS pool for which I limit ARC memory impact with : >> vfs.zfs.arc_max=3D64424509440 #60G >>=20 >> The only thing I did is some TCP tuning to improve throughput over = high-latency long-distance private links : >> kern.ipc.maxsockbuf=3D7372800 >> net.inet.tcp.sendbuf_max=3D6553600 >> net.inet.tcp.recvbuf_max=3D6553600 >> net.inet.tcp.sendspace=3D65536 >> net.inet.tcp.recvspace=3D65536 >> net.inet.tcp.sendbuf_inc=3D65536 >> net.inet.tcp.recvbuf_inc=3D65536 >> net.inet.tcp.cc.algorithm=3Dhtcp >>=20 >> Here are some graphs of memory & ARC usage when issue occurs. >> Crosshair (vertical red line) is at the timestamp where I get iSCSI = disconnections. >> https://postimg.org/gallery/1kkekrc4e/ >> What is strange is that each time issue occurs there is around 1GB of = free memory. >> So FreeBSD should still be able to allocate some more mbufs ? >> Unfortunately I do not have graphs about mbufs. >>=20 >> What should I ideally do ? >=20 > Have you tried increasing the mbufs limit?=20 > (sysctl) kern.ipc.nmbufs (Maximum number of mbufs allowed) Thank you for your suggestion Meny. No I did not try this yet. However, from the numbers above (and below), I think I should increase = kern.ipc.nmbjumbo9 instead ? # vmstat -z | grep -E "ITEM|mbuf" ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 26080395, 16400, 10418, 572292683, 0, 0 mbuf: 256, 26080395, 16402, 40525, 20955366061, 0, 0 mbuf_cluster: 2048, 4075060, 26818, 148, 907005, 0, 0 mbuf_jumbo_page: 4096, 2037529, 0, 34262, 5194563127, 0, 0 mbuf_jumbo_9k: 9216, 603712, 16400, 12867, 4362104082, 2676, 0 mbuf_jumbo_16k: 16384, 339588, 0, 0, 0, 0, 0 # sysctl kern.ipc | grep mb kern.ipc.nmbufs: 26080380 kern.ipc.nmbclusters: 4075058 kern.ipc.nmbjumbop: 2037529 kern.ipc.nmbjumbo9: 1811136 kern.ipc.nmbjumbo16: 1358352 kern.ipc.maxmbufmem: 33382879232 // note that I don't understand the difference between vmstat and sysctl = nmbjumbo9 / nmbjumbo16 values. // the first one is /3 (1811136/3=3D603712) the second one is /4 = (1358352/4=3D339588). Strange. // interesting post from Robert Watson which helped me understanding = some numbers : // = https://lists.freebsd.org/pipermail/freebsd-net/2008-December/020350.html