From owner-freebsd-net@FreeBSD.ORG Thu Sep 25 13:20:31 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58E8EBD for ; Thu, 25 Sep 2014 13:20:31 +0000 (UTC) Received: from nbfkord-smmo01.seg.att.com (nbfkord-smmo01.seg.att.com [209.65.160.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E9873C1F for ; Thu, 25 Sep 2014 13:20:30 +0000 (UTC) Received: from unknown [12.187.104.25] (EHLO webmail.solarflare.com) by nbfkord-smmo01.seg.att.com(mxl_mta-7.2.2-0) with ESMTP id e9614245.2b423c81c940.6677.00-2491.13274.nbfkord-smmo01.seg.att.com (envelope-from ); Thu, 25 Sep 2014 13:20:30 +0000 (UTC) X-MXL-Hash: 5424169e623d36b7-de355a2bce70c77e5cf94d4da26b7039320068a4 Received: from unknown [12.187.104.25] (EHLO webmail.solarflare.com) by nbfkord-smmo01.seg.att.com(mxl_mta-7.2.2-0) over TLS secured channel with ESMTP id 67514245.0.5342.00-2363.10640.nbfkord-smmo01.seg.att.com (envelope-from ); Thu, 25 Sep 2014 13:15:35 +0000 (UTC) X-MXL-Hash: 542415772b0b65b6-4f8496eaa1e03e79e3bafb3e895634611c429835 Received: from [192.168.38.17] (84.52.89.52) by webmail.SolarFlare.com (10.20.40.31) with Microsoft SMTP Server (TLS) id 14.3.158.1; Thu, 25 Sep 2014 06:14:57 -0700 Message-ID: <54241572.6000702@solarflare.com> Date: Thu, 25 Sep 2014 17:15:30 +0400 From: Andrew Rybchenko User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Subject: [PATCH 4/4] sfxge: Support tunable to control Tx deferred packet list limits Content-Type: multipart/mixed; boundary="------------080104050708050209000103" X-Originating-IP: [84.52.89.52] X-TM-AS-Product-Ver: SMEX-10.0.0.1412-7.000.1014-20974.005 X-TM-AS-Result: No--7.088200-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-AnalysisOut: [v=2.0 cv=bdHcppzB c=1 sm=1 a=MkjXnYnS3dyNWGSWLXxFFQ==:17 a] X-AnalysisOut: [=Ozv50jBIw7UA:10 a=o_jm4JFojFsA:10 a=1UGWc0DeibYA:10 a=4ox] X-AnalysisOut: [owH2qkH0A:10 a=RB3BGLmKESwA:10 a=BLceEmwcHowA:10 a=zRKbQ67] X-AnalysisOut: [AAAAA:8 a=qg_zykTt5_OaQZwxDF0A:9 a=wPNLvfGTeEIA:10 a=mTkGQ] X-AnalysisOut: [3Rq-wFmDC5QAsQA:9 a=QEXdDO2ut3YA:10 a=s9qa1pk-s7gA:10 a=ax] X-AnalysisOut: [CjAnoL7aCeYB5-:21 a=Std5G_GjueEGEdi_:21] X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2014051901)] X-MAIL-FROM: X-SOURCE-IP: [12.187.104.25] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 13:20:31 -0000 --------------080104050708050209000103 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Support tunable to control Tx deferred packet list limits Also increase default for Tx queue get-list limit. Too small limit results in TCP packets drops especiall when many streams are running simultaneously. Put list may be kept small enought since it is just a temporary location if transmit function can't get Tx queue lock. The information contained in this message is confidential and is intended f= or the addressee(s) only. If you have received this message in error, pleas= e notify the sender immediately and delete the message. Unless you are an a= ddressee (or authorized to receive for an addressee), you may not use, copy= or disclose to anyone this message or any information contained in this me= ssage. The unauthorized use, disclosure, copying or alteration of this mess= age is strictly prohibited. --------------080104050708050209000103 Content-Type: text/plain; charset="UTF-8"; name="dpl_max" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="dpl_max" Support tunable to control Tx deferred packet list limits Also increase default for Tx queue get-list limit. Too small limit results in TCP packets drops especiall when many streams are running simultaneously. Put list may be kept small enought since it is just a temporary location if transmit function can't get Tx queue lock. Submitted by: Andrew Rybchenko Sponsored by: Solarflare Communications, Inc. diff -r 0c812cd721d6 sys/dev/sfxge/sfxge_tx.c --- a/sys/dev/sfxge/sfxge_tx.c Thu Sep 25 16:06:49 2014 +0400 +++ b/sys/dev/sfxge/sfxge_tx.c Thu Sep 25 16:06:56 2014 +0400 @@ -50,6 +50,7 @@ #include #include #include +#include #include #include @@ -77,6 +78,25 @@ #define SFXGE_TSO_MAX_DESC ((65535 / 512) * 2 + SFXGE_TX_MAPPING_MAX_SEG - 1) #define SFXGE_TXQ_BLOCK_LEVEL(_entries) ((_entries) - SFXGE_TSO_MAX_DESC) +#ifdef SFXGE_HAVE_MQ + +#define SFXGE_PARAM_TX_DPL_GET_MAX SFXGE_PARAM(tx_dpl_get_max) +static int sfxge_tx_dpl_get_max = SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT; +TUNABLE_INT(SFXGE_PARAM_TX_DPL_GET_MAX, &sfxge_tx_dpl_get_max); +SYSCTL_INT(_hw_sfxge, OID_AUTO, tx_dpl_get_max, CTLFLAG_RDTUN, + &sfxge_tx_dpl_get_max, 0, + "Maximum number of packets in deferred packet get-list"); + +#define SFXGE_PARAM_TX_DPL_PUT_MAX SFXGE_PARAM(tx_dpl_put_max) +static int sfxge_tx_dpl_put_max = SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT; +TUNABLE_INT(SFXGE_PARAM_TX_DPL_PUT_MAX, &sfxge_tx_dpl_put_max); +SYSCTL_INT(_hw_sfxge, OID_AUTO, tx_dpl_put_max, CTLFLAG_RDTUN, + &sfxge_tx_dpl_put_max, 0, + "Maximum number of packets in deferred packet put-list"); + +#endif + + /* Forward declarations. */ static inline void sfxge_tx_qdpl_service(struct sfxge_txq *txq); static void sfxge_tx_qlist_post(struct sfxge_txq *txq); @@ -476,7 +496,7 @@ sfxge_tx_qdpl_swizzle(txq); - if (stdp->std_get_count >= SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT) + if (stdp->std_get_count >= stdp->std_get_max) return (ENOBUFS); *(stdp->std_getp) = mbuf; @@ -498,7 +518,7 @@ old_len = mp->m_pkthdr.csum_data; } else old_len = 0; - if (old_len >= SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT) + if (old_len >= stdp->std_put_max) return (ENOBUFS); mbuf->m_pkthdr.csum_data = old_len + 1; mbuf->m_nextpkt = (void *)old; @@ -1384,8 +1404,23 @@ goto fail3; #ifdef SFXGE_HAVE_MQ + if (sfxge_tx_dpl_get_max <= 0) { + log(LOG_ERR, "%s=%d must be greater than 0", + SFXGE_PARAM_TX_DPL_GET_MAX, sfxge_tx_dpl_get_max); + rc = EINVAL; + goto fail_tx_dpl_get_max; + } + if (sfxge_tx_dpl_put_max < 0) { + log(LOG_ERR, "%s=%d must be greater or equal to 0", + SFXGE_PARAM_TX_DPL_PUT_MAX, sfxge_tx_dpl_put_max); + rc = EINVAL; + goto fail_tx_dpl_put_max; + } + /* Initialize the deferred packet list. */ stdp = &txq->dpl; + stdp->std_put_max = sfxge_tx_dpl_put_max; + stdp->std_get_max = sfxge_tx_dpl_get_max; stdp->std_getp = &stdp->std_get; mtx_init(&txq->lock, "txq", NULL, MTX_DEF); @@ -1403,6 +1438,8 @@ return (0); +fail_tx_dpl_put_max: +fail_tx_dpl_get_max: fail3: fail_txq_node: free(txq->pend_desc, M_SFXGE); diff -r 0c812cd721d6 sys/dev/sfxge/sfxge_tx.h --- a/sys/dev/sfxge/sfxge_tx.h Thu Sep 25 16:06:49 2014 +0400 +++ b/sys/dev/sfxge/sfxge_tx.h Thu Sep 25 16:06:56 2014 +0400 @@ -75,13 +75,17 @@ enum sfxge_tx_buf_flags flags; }; -#define SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT 64 +#define SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT 1024 #define SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT 64 /* * Deferred packet list. */ struct sfxge_tx_dpl { + unsigned int std_get_max; /* Maximum number of packets + * in get list */ + unsigned int std_put_max; /* Maximum number of packets + * in put list */ uintptr_t std_put; /* Head of put list. */ struct mbuf *std_get; /* Head of get list. */ struct mbuf **std_getp; /* Tail of get list. */ --------------080104050708050209000103--