From owner-svn-src-head@freebsd.org Fri Jun 15 23:42:23 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE407102125C; Fri, 15 Jun 2018 23:42:23 +0000 (UTC) (envelope-from np@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7366B68B95; Fri, 15 Jun 2018 23:42:23 +0000 (UTC) (envelope-from np@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3A5A81E50C; Fri, 15 Jun 2018 23:42:23 +0000 (UTC) (envelope-from np@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w5FNgN8F041075; Fri, 15 Jun 2018 23:42:23 GMT (envelope-from np@FreeBSD.org) Received: (from np@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w5FNgMpU041074; Fri, 15 Jun 2018 23:42:22 GMT (envelope-from np@FreeBSD.org) Message-Id: <201806152342.w5FNgMpU041074@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: np set sender to np@FreeBSD.org using -f From: Navdeep Parhar Date: Fri, 15 Jun 2018 23:42:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r335243 - head/sys/dev/cxgbe X-SVN-Group: head X-SVN-Commit-Author: np X-SVN-Commit-Paths: head/sys/dev/cxgbe X-SVN-Commit-Revision: 335243 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 23:42:24 -0000 Author: np Date: Fri Jun 15 23:42:22 2018 New Revision: 335243 URL: https://svnweb.freebsd.org/changeset/base/335243 Log: cxgbe(4): Add a hw.cxgbe.starve_fl sysctl that can be used to starve the freelists of netmap receive queues. This is primarily to test various congestion scenarios in the chip. Sponsored by: Chelsio Communications Modified: head/sys/dev/cxgbe/adapter.h head/sys/dev/cxgbe/t4_netmap.c Modified: head/sys/dev/cxgbe/adapter.h ============================================================================== --- head/sys/dev/cxgbe/adapter.h Fri Jun 15 21:36:16 2018 (r335242) +++ head/sys/dev/cxgbe/adapter.h Fri Jun 15 23:42:22 2018 (r335243) @@ -682,6 +682,7 @@ struct sge_nm_rxq { uint32_t fl_db_val; u_int fl_hwidx:4; + u_int fl_db_saved; u_int nid; /* netmap ring # for this queue */ /* infrequently used items after this */ Modified: head/sys/dev/cxgbe/t4_netmap.c ============================================================================== --- head/sys/dev/cxgbe/t4_netmap.c Fri Jun 15 21:36:16 2018 (r335242) +++ head/sys/dev/cxgbe/t4_netmap.c Fri Jun 15 23:42:22 2018 (r335243) @@ -89,6 +89,10 @@ SYSCTL_INT(_hw_cxgbe, OID_AUTO, nm_holdoff_tmr_idx, CT static int nm_cong_drop = 1; TUNABLE_INT("hw.cxgbe.nm_cong_drop", &nm_cong_drop); +int starve_fl = 0; +SYSCTL_INT(_hw_cxgbe, OID_AUTO, starve_fl, CTLFLAG_RWTUN, + &starve_fl, 0, "Don't ring fl db for netmap rx queues."); + static int alloc_nm_rxq_hwq(struct vi_info *vi, struct sge_nm_rxq *nm_rxq, int cong) { @@ -806,6 +810,13 @@ cxgbe_netmap_rxsync(struct netmap_kring *kring, int fl kring->nr_kflags &= ~NKR_PENDINTR; } + if (nm_rxq->fl_db_saved > 0 && starve_fl == 0) { + wmb(); + t4_write_reg(sc, sc->sge_kdoorbell_reg, + nm_rxq->fl_db_val | V_PIDX(nm_rxq->fl_db_saved)); + nm_rxq->fl_db_saved = 0; + } + /* Userspace done with buffers from kring->nr_hwcur to head */ n = head >= kring->nr_hwcur ? head - kring->nr_hwcur : kring->nkr_num_slots - kring->nr_hwcur + head; @@ -842,8 +853,12 @@ cxgbe_netmap_rxsync(struct netmap_kring *kring, int fl } if (++dbinc == 8 && n >= 32) { wmb(); - t4_write_reg(sc, sc->sge_kdoorbell_reg, - nm_rxq->fl_db_val | V_PIDX(dbinc)); + if (starve_fl) + nm_rxq->fl_db_saved += dbinc; + else { + t4_write_reg(sc, sc->sge_kdoorbell_reg, + nm_rxq->fl_db_val | V_PIDX(dbinc)); + } dbinc = 0; } } @@ -851,8 +866,12 @@ cxgbe_netmap_rxsync(struct netmap_kring *kring, int fl if (dbinc > 0) { wmb(); - t4_write_reg(sc, sc->sge_kdoorbell_reg, - nm_rxq->fl_db_val | V_PIDX(dbinc)); + if (starve_fl) + nm_rxq->fl_db_saved += dbinc; + else { + t4_write_reg(sc, sc->sge_kdoorbell_reg, + nm_rxq->fl_db_val | V_PIDX(dbinc)); + } } }