From owner-freebsd-net@freebsd.org Mon Jan 2 11:09:23 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 949F7C9B303 for ; Mon, 2 Jan 2017 11:09:23 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wj0-x244.google.com (mail-wj0-x244.google.com [IPv6:2a00:1450:400c:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 305481C09 for ; Mon, 2 Jan 2017 11:09:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wj0-x244.google.com with SMTP id hb5so34955408wjc.2 for ; Mon, 02 Jan 2017 03:09:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=OOYynTgdBg1PH7WOYrvqbXDg/PsMau+qsILGJHwHK5E=; b=UWSk0hlN2LeIa6kwLlmbFU6ljFwQFZWRjm5rzzMv+SU+TAhOIq/y4bsp/2CODI/PHv 9rFE649rVlT/d3Wl8Ab1Un/MlAtwCoCUiFlE8CKQuHSwzGA17hPyyMUXvHk7J+7sjgBn 5ktQjAaQzqABf8IJnxon67GEDXQKfHxJNclXD8/TuCyF30bDr1+Ce6bum2zGSDeEy3Em dfMfMTyK9Z3dxO+gIjhnS1u04jfNtjN3i68GMe6ZcDb5nznSB1/CysemrmrJrs13cdT6 kz4F6yBry8Nq5NiyuXl/5DBUN6U7mBhvGyPhMGGbsfyAROpMxwCg/8P844KZ+izKABfS QluA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=OOYynTgdBg1PH7WOYrvqbXDg/PsMau+qsILGJHwHK5E=; b=fjpGKI50fO2Jo24ZtTA+BjBnY9w56PsgCWWkoElotoW+Bn4rlMO+e7AnPGY11xK+W0 6o2eW42orpcgyzqjhEvrkE+xrxoe1MkciB1whJW/+C4gzD1TY6rHaiq3v2y7bg5MHyq0 RqwXvlhEhwo1wmdOIdqeGgkY5CBUQpkgxHzSsa+cg/kdhF0tRJ/2isiPrJ5Ju99pkaZC 9x8BGil8rSSeqJRPj5HfOZ2Xgnt/CJE/+5T3ecRy1c59Zs4pAP1rN9WAYqJo9PEx5S6w LIR0YKwVZCVJKUebIv6urcOtUffFcfjRGpQRFE3xjdZgwT5RbDfPZd9RdBJblZQOcjqS edXQ== X-Gm-Message-State: AIkVDXL3A8948Fn0qfAFzRW0qUUVZRolykJg+aKjmTv7uSfmZ6bibUCBriPQ3q+m9RxetA== X-Received: by 10.194.205.225 with SMTP id lj1mr56746322wjc.122.1483355361175; Mon, 02 Jan 2017 03:09:21 -0800 (PST) Received: from ben.home (LFbn-1-7159-4.w90-116.abo.wanadoo.fr. [90.116.90.4]) by smtp.gmail.com with ESMTPSA id i15sm87039004wjs.16.2017.01.02.03.09.20 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Jan 2017 03:09:20 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: iSCSI failing, MLX rx_ring errors ? From: Ben RUBSON In-Reply-To: Date: Mon, 2 Jan 2017 12:09:15 +0100 Cc: Hans Petter Selasky , Yuval Bason , Meny Yossefi Content-Transfer-Encoding: quoted-printable Message-Id: <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> To: "freebsd-net@freebsd.org" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2017 11:09:23 -0000 Hi Meny, Thank you very much for your feedback. I think you are right, this could be a mbufs issue. Here are some more numbers : # vmstat -z | grep -v "0, 0$" ITEM SIZE LIMIT USED FREE REQ = FAIL SLEEP 4 Bucket: 32, 0, 2673, 28327, 88449799, = 17317, 0 8 Bucket: 64, 0, 449, 15609, 13926386, = 4871, 0 12 Bucket: 96, 0, 335, 5323, 10293892, = 142872, 0 16 Bucket: 128, 0, 533, 6070, 7618615, = 472647, 0 32 Bucket: 256, 0, 8317, 22133, 36020376, = 563479, 0 64 Bucket: 512, 0, 1238, 3298, 20138111, = 11430742, 0 128 Bucket: 1024, 0, 1865, 2963, 21162182, = 158752, 0 256 Bucket: 2048, 0, 1626, 450, 80253784, = 4890164, 0 mbuf_jumbo_9k: 9216, 603712, 16400, 8744, 4128521064, = 2661, 0 # netstat -m 32801/18814/51615 mbufs in use (current/cache/total) 16400/9810/26210/4075058 mbuf clusters in use (current/cache/total/max) 16400/9659 mbuf+clusters out of packet secondary zone in use = (current/cache) 0/8647/8647/2037529 4k (page size) jumbo clusters in use = (current/cache/total/max) 16400/8744/25144/603712 9k jumbo clusters in use = (current/cache/total/max) 0/0/0/339588 16k jumbo clusters in use (current/cache/total/max) 188600K/137607K/326207K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/2661/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed I did not perform any mbufs tuning, numbers above are from FreeBSD = itself. This server has 64GB of memory. It has a ZFS pool for which I limit ARC memory impact with : vfs.zfs.arc_max=3D64424509440 #60G The only thing I did is some TCP tuning to improve throughput over = high-latency long-distance private links : kern.ipc.maxsockbuf=3D7372800 net.inet.tcp.sendbuf_max=3D6553600 net.inet.tcp.recvbuf_max=3D6553600 net.inet.tcp.sendspace=3D65536 net.inet.tcp.recvspace=3D65536 net.inet.tcp.sendbuf_inc=3D65536 net.inet.tcp.recvbuf_inc=3D65536 net.inet.tcp.cc.algorithm=3Dhtcp Here are some graphs of memory & ARC usage when issue occurs. Crosshair (vertical red line) is at the timestamp where I get iSCSI = disconnections. https://postimg.org/gallery/1kkekrc4e/ What is strange is that each time issue occurs there is around 1GB of = free memory. So FreeBSD should still be able to allocate some more mbufs ? Unfortunately I do not have graphs about mbufs. What should I ideally do ? Thank you again, Best regards, Ben > On 01 Jan 2017, at 09:16, Meny Yossefi wrote: >=20 > Hi Ben,=20 > =20 > Those are not HW errors, note that: > =20 > hw.mlxen1.stat.rx_dropped: 0 > hw.mlxen1.stat.rx_errors: 0 > =20 > It seems to be triggered when you are failing to allocate a = replacement buffer. > Any chance you ran out of mbufs in the system? > =20 > en_rx.c: > =20 > mlx4_en_process_rx_cq(): > =20 > mb =3D mlx4_en_rx_mb(priv, rx_desc, mb_list, length); > if (!mb) { > ring->errors++; > goto next; > } > =20 > mlx4_en_rx_mb() =C3=A0 mlx4_en_complete_rx_desc(): > =20 > /* Allocate a replacement page */ > if (mlx4_en_alloc_buf(priv, rx_desc, mb_list, nr)) > goto fail; > =20 > -Meny