From owner-freebsd-net@FreeBSD.ORG  Tue Mar 18 09:55:03 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id D184E38F
 for <net@FreeBSD.org>; Tue, 18 Mar 2014 09:55:03 +0000 (UTC)
Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [188.134.15.200])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 8A20B6A5
 for <net@FreeBSD.org>; Tue, 18 Mar 2014 09:55:03 +0000 (UTC)
Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17])
 (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
 (No client certificate requested)
 by shelob.oktetlabs.ru (Postfix) with ESMTPSA id B3BC57F57C;
 Tue, 18 Mar 2014 13:55:00 +0400 (MSK)
X-DKIM: Sendmail DKIM Filter v2.8.2 shelob.oktetlabs.ru B3BC57F57C
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru;
 s=default; t=1395136500;
 bh=8MszJOk6k9O2Zb6wQVR8MBIm4aUDiotW/5dkTw60uHc=; l=2069;
 h=Message-ID:Date:From:MIME-Version:To:CC:Subject:Content-Type:
 Content-Transfer-Encoding;
 b=goV4R/xsBLdAjCu0zy937J8aJz4eGhOgQuD01IOCwN/QDERKbk89GFREAOnoRPpwA
 LqJhf4t7tnoiSU9s6Q0FT8Uzb6EndBFfss8Ko8LNV8NrUIJXTDa468l6zOIKtu0vNI
 CTb5a1aInP6AvAYSoXJBR0+nZbgAPp9+Qdp+3pqM=
Message-ID: <532817F5.8010505@oktetlabs.ru>
Date: Tue, 18 Mar 2014 13:55:01 +0400
From: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
Organization: OKTET Labs
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.3.0
MIME-Version: 1.0
To: net@FreeBSD.org
Subject: [PATCH 2/6] sfxge: limit software Tx queue size
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Mar 2014 09:55:03 -0000

sfxge: limit software Tx queue size

Previous implementation limits put queue size only (when Tx lock can't 
be acquired),
but get queue may grow unboundedly which results in mbuf pools 
exhaustion and
latency growth.

Submitted-by:   Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
Sponsored by:   Solarflare Communications, Inc.

diff -r ff9f5d3dbafe -r 7632a3355224 src/driver/freebsd/sfxge_tx.c
--- a/head/sys/dev/sfxge/sfxge_tx.c    Tue Mar 04 13:15:13 2014 +0400
+++ b/head/sys/dev/sfxge/sfxge_tx.c    Wed Mar 05 09:06:01 2014 +0400
@@ -461,6 +461,9 @@

          sfxge_tx_qdpl_swizzle(txq);

+        if (stdp->std_count >= SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT)
+            return ENOBUFS;
+
          *(stdp->std_getp) = mbuf;
          stdp->std_getp = &mbuf->m_nextpkt;
          stdp->std_count++;
@@ -480,7 +483,7 @@
                  old_len = mp->m_pkthdr.csum_data;
              } else
                  old_len = 0;
-            if (old_len >= SFXGE_TX_MAX_DEFERRED)
+            if (old_len >= SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT)
                  return ENOBUFS;
              mbuf->m_pkthdr.csum_data = old_len + 1;
              mbuf->m_nextpkt = (void *)old;
@@ -507,12 +510,9 @@
       */
      locked = mtx_trylock(&txq->lock);

-    /*
-     * Can only fail if we weren't able to get the lock.
-     */
      if (sfxge_tx_qdpl_put(txq, m, locked) != 0) {
-        KASSERT(!locked,
-            ("sfxge_tx_qdpl_put() failed locked"));
+        if (locked)
+            mtx_unlock(&txq->lock);
          rc = ENOBUFS;
          goto fail;
      }
diff -r ff9f5d3dbafe -r 7632a3355224 src/driver/freebsd/sfxge_tx.h
--- a/head/sys/dev/sfxge/sfxge_tx.h    Tue Mar 04 13:15:13 2014 +0400
+++ b/head/sys/dev/sfxge/sfxge_tx.h    Wed Mar 05 09:06:01 2014 +0400
@@ -75,7 +75,8 @@
      enum sfxge_tx_buf_flags    flags;
  };

-#define SFXGE_TX_MAX_DEFERRED 64
+#define SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT    64
+#define SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT    64

  /*
   * Deferred packet list.