Date: Mon, 26 Jun 2017 16:36:35 +0300 From: "Andrey V. Elsukov" <bu7cher@yandex.ru> To: Ben RUBSON <ben.rubson@gmail.com> Cc: Ryan Stone <rysto32@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org> Subject: Re: mbuf_jumbo_9k & iSCSI failing Message-ID: <61f98b7d-f55d-aa0f-4aef-1bdfbc7086ff@yandex.ru> In-Reply-To: <D0CF7BD6-C443-4707-9863-9D6ADB0B3025@gmail.com> References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <DB3PR05MB089A5789A0A619FA8B7CA36C36C0@DB3PR05MB089.eurprd05.prod.outlook.com> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> <DB3PR05MB089011A41EF87A40C7AC741C36E0@DB3PR05MB089.eurprd05.prod.outlook.com> <F19B51C7-7DDD-4FAB-9091-0B7C8A7CE649@gmail.com> <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> <9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com> <CAFMmRNzo=xB8XF6SFD%2BwksmBYjRZ_peYjiPBCXNVyqP%2BdxnujQ@mail.gmail.com> <CAFMmRNwbEwn=TmTAd56rViDV5nDXq_hPmTp-cDwmVqu1XYm=fA@mail.gmail.com> <64abec26-e310-d66d-93ae-3536914ddd84@yandex.ru> <86D76532-92F4-479C-A714-126D007AD91F@gmail.com> <a36014ff-0a43-93ef-52f8-2bca1388757e@yandex.ru> <D0CF7BD6-C443-4707-9863-9D6ADB0B3025@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--8VTMwHwj940cm1xRx4AhDLWSOS98RD7OQ
Content-Type: multipart/mixed; boundary="g9A9E4Di6C80mHn9j0G28L5VKTWTg18l6";
protected-headers="v1"
From: "Andrey V. Elsukov" <bu7cher@yandex.ru>
To: Ben RUBSON <ben.rubson@gmail.com>
Cc: Ryan Stone <rysto32@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>,
"freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>
Message-ID: <61f98b7d-f55d-aa0f-4aef-1bdfbc7086ff@yandex.ru>
Subject: Re: mbuf_jumbo_9k & iSCSI failing
References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com>
<6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org>
<DB3PR05MB089A5789A0A619FA8B7CA36C36C0@DB3PR05MB089.eurprd05.prod.outlook.com>
<613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com>
<2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com>
<DB3PR05MB089011A41EF87A40C7AC741C36E0@DB3PR05MB089.eurprd05.prod.outlook.com>
<F19B51C7-7DDD-4FAB-9091-0B7C8A7CE649@gmail.com>
<52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com>
<9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com>
<CAFMmRNzo=xB8XF6SFD+wksmBYjRZ_peYjiPBCXNVyqP+dxnujQ@mail.gmail.com>
<CAFMmRNwbEwn=TmTAd56rViDV5nDXq_hPmTp-cDwmVqu1XYm=fA@mail.gmail.com>
<64abec26-e310-d66d-93ae-3536914ddd84@yandex.ru>
<86D76532-92F4-479C-A714-126D007AD91F@gmail.com>
<a36014ff-0a43-93ef-52f8-2bca1388757e@yandex.ru>
<D0CF7BD6-C443-4707-9863-9D6ADB0B3025@gmail.com>
In-Reply-To: <D0CF7BD6-C443-4707-9863-9D6ADB0B3025@gmail.com>
--g9A9E4Di6C80mHn9j0G28L5VKTWTg18l6
Content-Type: multipart/mixed;
boundary="------------FF1D5726D14DBD643DE462AD"
Content-Language: en-US
This is a multi-part message in MIME format.
--------------FF1D5726D14DBD643DE462AD
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 26.06.2017 16:29, Ben RUBSON wrote:
>=20
>> On 26 Jun 2017, at 15:25, Andrey V. Elsukov <bu7cher@yandex.ru> wrote:=
>>
>> On 26.06.2017 16:27, Ben RUBSON wrote:
>>>
>>>> On 26 Jun 2017, at 15:13, Andrey V. Elsukov <bu7cher@yandex.ru> wrot=
e:
>>>>
>>>> I think it is not mlxen specific problem, we have the same symptoms =
with
>>>> ixgbe(4) driver too. To avoid the problem we have patches that are
>>>> disable using of 9k mbufs, and instead only use 4k mbufs.
>>>
>>> Interesting feedback Andrey, thank you !
>>> The problem may be then "general".
>>> So you still use large MTU (>=3D9000) but only allocating 4k mbufs, a=
s a workaround ?
>>
>> Yes.
>=20
> Is it a kernel patch or a driver/ixgbe patch ?
I attached it.
--=20
WBR, Andrey V. Elsukov
--------------FF1D5726D14DBD643DE462AD
Content-Type: text/x-patch;
name="0004-Add-m_preferredsize-and-use-it-in-all-intel-s-driver.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename*0="0004-Add-m_preferredsize-and-use-it-in-all-intel-s-driver.pa";
filename*1="tch"
=46rom 57b4789b7f6699a581ded2f4e07c7b12737af1e7 Mon Sep 17 00:00:00 2001
From: "Andrey V. Elsukov" <bu7cher@yandex.ru>
Date: Thu, 6 Oct 2016 14:56:37 +0300
Subject: [PATCH 04/65] Add m_preferredsize() and use it in all intel's
drivers.
---
sys/dev/e1000/if_em.c | 7 +------
sys/dev/e1000/if_igb.c | 7 +------
sys/dev/ixgbe/if_ix.c | 5 +----
sys/dev/ixgbe/if_ixv.c | 5 +----
sys/dev/ixl/if_ixlv.c | 5 +----
sys/dev/ixl/ixl_pf_main.c | 5 +----
sys/kern/kern_mbuf.c | 35 +++++++++++++++++++++++++++++++++++
sys/sys/mbuf.h | 1 +
8 files changed, 42 insertions(+), 28 deletions(-)
diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c
index 7e2690eae08..1af66b7c519 100644
--- a/sys/dev/e1000/if_em.c
+++ b/sys/dev/e1000/if_em.c
@@ -1421,12 +1421,7 @@ em_init_locked(struct adapter *adapter)
** Figure out the desired mbuf
** pool for doing jumbos
*/
- if (adapter->hw.mac.max_frame_size <=3D 2048)
- adapter->rx_mbuf_sz =3D MCLBYTES;
- else if (adapter->hw.mac.max_frame_size <=3D 4096)
- adapter->rx_mbuf_sz =3D MJUMPAGESIZE;
- else
- adapter->rx_mbuf_sz =3D MJUM9BYTES;
+ adapter->rx_mbuf_sz =3D m_preferredsize(adapter->hw.mac.max_frame_size)=
;
=20
/* Prepare receive descriptors and buffers */
if (em_setup_receive_structures(adapter)) {
diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c
index 8e018995029..bfaecae1f71 100644
--- a/sys/dev/e1000/if_igb.c
+++ b/sys/dev/e1000/if_igb.c
@@ -1325,12 +1325,7 @@ igb_init_locked(struct adapter *adapter)
** Figure out the desired mbuf pool
** for doing jumbo/packetsplit
*/
- if (adapter->max_frame_size <=3D 2048)
- adapter->rx_mbuf_sz =3D MCLBYTES;
- else if (adapter->max_frame_size <=3D 4096)
- adapter->rx_mbuf_sz =3D MJUMPAGESIZE;
- else
- adapter->rx_mbuf_sz =3D MJUM9BYTES;
+ adapter->rx_mbuf_sz =3D m_preferredsize(adapter->max_frame_size);
=20
/* Prepare receive descriptors and buffers */
if (igb_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixgbe/if_ix.c b/sys/dev/ixgbe/if_ix.c
index cf2231dc8fc..26fce2704ba 100644
--- a/sys/dev/ixgbe/if_ix.c
+++ b/sys/dev/ixgbe/if_ix.c
@@ -1118,10 +1118,7 @@ ixgbe_init_locked(struct adapter *adapter)
ixgbe_set_multi(adapter);
=20
/* Determine the correct mbuf pool, based on frame size */
- if (adapter->max_frame_size <=3D MCLBYTES)
- adapter->rx_mbuf_sz =3D MCLBYTES;
- else
- adapter->rx_mbuf_sz =3D MJUMPAGESIZE;
+ adapter->rx_mbuf_sz =3D m_preferredsize(adapter->max_frame_size);
=20
/* Prepare receive descriptors and buffers */
if (ixgbe_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixgbe/if_ixv.c b/sys/dev/ixgbe/if_ixv.c
index 80fb1b34be3..5062affb779 100644
--- a/sys/dev/ixgbe/if_ixv.c
+++ b/sys/dev/ixgbe/if_ixv.c
@@ -698,10 +698,7 @@ ixv_init_locked(struct adapter *adapter)
** Determine the correct mbuf pool
** for doing jumbo/headersplit
*/
- if (ifp->if_mtu > ETHERMTU)
- adapter->rx_mbuf_sz =3D MJUMPAGESIZE;
- else
- adapter->rx_mbuf_sz =3D MCLBYTES;
+ adapter->rx_mbuf_sz =3D m_preferredsize(ifp->if_mtu);
=20
/* Prepare receive descriptors and buffers */
if (ixgbe_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixl/if_ixlv.c b/sys/dev/ixl/if_ixlv.c
index c447c34689e..608d784bfee 100644
--- a/sys/dev/ixl/if_ixlv.c
+++ b/sys/dev/ixl/if_ixlv.c
@@ -904,10 +904,7 @@ ixlv_init_locked(struct ixlv_sc *sc)
=20
ixl_init_tx_ring(que);
=20
- if (vsi->max_frame_size <=3D MCLBYTES)
- rxr->mbuf_sz =3D MCLBYTES;
- else
- rxr->mbuf_sz =3D MJUMPAGESIZE;
+ rxr->mbuf_sz =3D m_preferredsize(vsi->max_frame_size);
ixl_init_rx_ring(que);
}
=20
diff --git a/sys/dev/ixl/ixl_pf_main.c b/sys/dev/ixl/ixl_pf_main.c
index d8da4cfee10..8600b0f931e 100644
--- a/sys/dev/ixl/ixl_pf_main.c
+++ b/sys/dev/ixl/ixl_pf_main.c
@@ -2067,10 +2067,7 @@ ixl_initialize_vsi(struct ixl_vsi *vsi)
ixl_init_tx_ring(que);
=20
/* Next setup the HMC RX Context */
- if (vsi->max_frame_size <=3D MCLBYTES)
- rxr->mbuf_sz =3D MCLBYTES;
- else
- rxr->mbuf_sz =3D MJUMPAGESIZE;
+ rxr->mbuf_sz =3D m_preferredsize(vsi->max_frame_size);
=20
u16 max_rxmax =3D rxr->mbuf_sz * hw->func_caps.rx_buf_chain_len;
=20
diff --git a/sys/kern/kern_mbuf.c b/sys/kern/kern_mbuf.c
index 0d0c1c86b16..7c10cedb075 100644
--- a/sys/kern/kern_mbuf.c
+++ b/sys/kern/kern_mbuf.c
@@ -103,6 +103,10 @@ int nmbjumbop; /* limits number of page size jumbo=
clusters */
int nmbjumbo9; /* limits number of 9k jumbo clusters */
int nmbjumbo16; /* limits number of 16k jumbo clusters */
=20
+static int nojumbobuf; /* Use MCLBYTES mbufs */
+static int nojumbo9buf; /* Use either MCLBYTES or MJUMPAGESIZE */
+static int nojumbo16buf; /* Use any mbuf size less than MJUM16BYTES */
+
static quad_t maxmbufmem; /* overall real memory limit for all mbufs */
=20
SYSCTL_QUAD(_kern_ipc, OID_AUTO, maxmbufmem, CTLFLAG_RDTUN | CTLFLAG_NOF=
ETCH, &maxmbufmem, 0,
@@ -151,6 +155,16 @@ tunable_mbinit(void *dummy)
if (nmbufs < nmbclusters + nmbjumbop + nmbjumbo9 + nmbjumbo16)
nmbufs =3D lmax(maxmbufmem / MSIZE / 5,
nmbclusters + nmbjumbop + nmbjumbo9 + nmbjumbo16);
+ /*
+ * Defaults to disable 9/16-kbyte pages
+ */
+ nojumbobuf =3D 0;
+ nojumbo9buf =3D 1;
+ nojumbo16buf =3D 1;
+
+ TUNABLE_INT_FETCH("kern.ipc.nojumbobuf", &nojumbobuf);
+ TUNABLE_INT_FETCH("kern.ipc.nojumbo9buf", &nojumbo9buf);
+ TUNABLE_INT_FETCH("kern.ipc.nojumbo16buf", &nojumbo16buf);
}
SYSINIT(tunable_mbinit, SI_SUB_KMEM, SI_ORDER_MIDDLE, tunable_mbinit, NU=
LL);
=20
@@ -261,6 +275,27 @@ SYSCTL_PROC(_kern_ipc, OID_AUTO, nmbufs, CTLTYPE_INT=
|CTLFLAG_RW,
"Maximum number of mbufs allowed");
=20
/*
+ * Determine the correct mbuf pool
+ * for given mtu size
+ */
+int
+m_preferredsize(int mtu)
+{
+ int size;
+
+ if (mtu <=3D 2048 || nojumbobuf !=3D 0)
+ size =3D MCLBYTES;
+ else if (mtu <=3D 4096 || nojumbo9buf !=3D 0)
+ size =3D MJUMPAGESIZE;
+ else if (mtu <=3D 9216 || nojumbo16buf !=3D 0)
+ size =3D MJUM9BYTES;
+ else
+ size =3D MJUM16BYTES;
+
+ return (size);
+}
+
+/*
* Zones from which we allocate.
*/
uma_zone_t zone_mbuf;
diff --git a/sys/sys/mbuf.h b/sys/sys/mbuf.h
index fdd9931515d..b6a81b05e3b 100644
--- a/sys/sys/mbuf.h
+++ b/sys/sys/mbuf.h
@@ -606,6 +606,7 @@ u_int m_length(struct mbuf *, struct mbuf **);
int m_mbuftouio(struct uio *, struct mbuf *, int);
void m_move_pkthdr(struct mbuf *, struct mbuf *);
int m_pkthdr_init(struct mbuf *, int);
+int m_preferredsize(int);
struct mbuf *m_prepend(struct mbuf *, int, int);
void m_print(const struct mbuf *, int);
struct mbuf *m_pulldown(struct mbuf *, int, int, int *);
--=20
2.12.1
--------------FF1D5726D14DBD643DE462AD--
--g9A9E4Di6C80mHn9j0G28L5VKTWTg18l6--
--8VTMwHwj940cm1xRx4AhDLWSOS98RD7OQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAllRDeMACgkQAcXqBBDI
oXqqvAf6AwhLtFohbNT9kfkP6P0pMIGoXCYwQ7ACxYVLmTLzKorGGV4aj+DalTjv
Dv7H2ICYXBcESgH8xjZgMKeAVxIfMlsvGVbRwQs3rnSO9bMjGLXsPxcD6ymvZf4L
tgxm4aBPyPFevdBD6DdU7bdfv+Ml1c15iQ/Vr5khQaplMkcw2q0mUI6efJD6agp0
5fjd7kxrHDxranQr+DcW6lw+pd4GQakBLy5JNODTESMOc4DaUbhGVE79nRvMqTMU
LjJAWxQGqvwRvvy3RkWuCczjuBelM0Cb9U8HxPbDzumyXbY113raLidGJbabnuul
YPGe76RwAwFQiFkVaLiplL8pMkaAZw==
=0JjL
-----END PGP SIGNATURE-----
--8VTMwHwj940cm1xRx4AhDLWSOS98RD7OQ--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?61f98b7d-f55d-aa0f-4aef-1bdfbc7086ff>