From owner-freebsd-net@freebsd.org Thu Apr 26 06:43:02 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7E6EFA291E for ; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 649136C50A for ; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 19924FA291C; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB89BFA2919; Thu, 26 Apr 2018 06:43:00 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4AD066C507; Thu, 26 Apr 2018 06:43:00 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id p5-v6so27749938wre.12; Wed, 25 Apr 2018 23:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kVgABVOp/EFIAlV+m0iQ2340GYLALiZEpjF1ulJ4rqc=; b=cEJ50ODjAxvmk/XoxjV7ZHNfeyedEzbt0j1RbWMDjrlw5O3D9zqaEFP9pEz+59+WcV iMwLxFsVucCO7YrUe68Mxs9rBVO1gEZbm0/j5mMD2VGzm8EbkooxVUmOwfYHW0VlwU5m EgowbjoJ5Z1xgWWUU0aa+OTdGt0C6geKWaiW0nZ+GJGzi2J4omeBB+F+gVQFxfHHO/no 9VCkaNEPYEaU4zh4EdKi9/egvvvdEo8GrilTWnNtjwaiyDp6RD9l8rR977fSwTW9cN/T Xtplf3nPYMvrkS7IwIrPDf6EO+3KbliPzhGzGyUr0hb52E4BEsFbfjSd6aFpE9jI228r 2nSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kVgABVOp/EFIAlV+m0iQ2340GYLALiZEpjF1ulJ4rqc=; b=lPIhglHuSrYANv0Mspsvi4qxWDTyzrkHOWI12p4EWv97Hn3Vbvjg3x/3Q7VVTcBFYB mJrL2k1Y5UFh+H+q0ioPccciaqEz3JalN0KiAPp6WksPBZ+j6tolTnhu5KaQ6QPzY5PK zAj8J1meX76MrJXT8GLKa1NWh9VAs8PxHk9L4MiPpTC4UP5eZ6+7JImnUFMaOQVeXtAR zjVRPpDUZIVJ7N5HGFNlVsQe/KhmfzHyQdD+Yxf5BN8vckrwW0kH4j2ESJHQiyE8ec+3 dHNbgOUy2lMwtczI7fCySUf32TAnkzfgDuXlgbBMyOQJLhhjQbOc1Ksms5jCVQhAkXSx 9mwA== X-Gm-Message-State: ALQs6tBGVkezKfaw5muXatzsQO5FNYr537IPWagT1hGciafnOizqDq/A MXOW1XZ9ScZS5fbWvIFgOE8= X-Google-Smtp-Source: AIpwx4/4jnNhpZQaT4KzNiDX3tUofDS9TZcleDxGW3oeDjsz8WKtjT6Nptdb2ScpxJ7cBzDwhvoDsA== X-Received: by 2002:adf:b741:: with SMTP id n1-v6mr26629144wre.203.1524724979326; Wed, 25 Apr 2018 23:42:59 -0700 (PDT) Received: from bens-mac.home (LFbn-1-7077-100.w90-116.abo.wanadoo.fr. [90.116.246.100]) by smtp.gmail.com with ESMTPSA id q34-v6sm18526218wrb.27.2018.04.25.23.42.58 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 25 Apr 2018 23:42:58 -0700 (PDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mlx5(4) jumbo receive From: Ben RUBSON In-Reply-To: Date: Thu, 26 Apr 2018 08:43:01 +0200 Cc: Ryan Stone , Konstantin Belousov , FreeBSD Current , FreeBSD Net , Meny Yossefi Content-Transfer-Encoding: 7bit Message-Id: References: <20180424085553.GA6887@kib.kiev.ua> To: Rick Macklem X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 06:43:02 -0000 On 26 Apr 2018, Rick Macklem wrote: > Ryan Stone wrote: >> On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov >> >>wrote: >>> +#ifndef MLX5E_MAX_RX_BYTES >>> +#define MLX5E_MAX_RX_BYTES MCLBYTES >>> +#endif >> >> Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? >> MJUMPAGESIZE should offer significantly better performance for jumbo >> frames without increasing the risk of memory fragmentation. > Actually, when I was playing with using jumbo mbuf clusters for NFS, I > was able > to get it to fragment to the point where allocations failed when mixing > 2K and > 4K mbuf clusters. > Admittedly I was using a 256Mbyte i386 and it wasn't easily reproduced, but > it was possible. > --> Using a mix of 2K and 4K mbuf clusters can result in fragmentation, > although > I suspect that it isn't nearly as serious as what can happen when using 9K > mbuf clusters. I used to face the fragmentation issue easily with MTU set to 9000 (x86_64 / 64GB / Connect-X/3). I then decreased the MTU until 9K mbufs are not more used, to 4072 bytes then. The other used interface of this 2-ports card is set with a 1500 MTU. Until now (about a year later), no more issue. # vmstat -z | grep mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 26080155, 16400, 9652, 999757417, 0, 0 mbuf: 256, 26080155, 16419, 11349, 85322301444, 0, 0 mbuf_cluster: 2048, 4075022, 26052, 550, 1059817, 0, 0 mbuf_jumbo_page: 4096, 2037511, 16400, 9522, 45863599682, 0, 0 mbuf_jumbo_9k: 9216, 603707, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 339585, 0, 0, 0, 0, 0 Here's my experience :) Ben