From owner-svn-src-projects@freebsd.org Wed Dec 2 23:55:00 2015 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADE5FA3F655 for ; Wed, 2 Dec 2015 23:55:00 +0000 (UTC) (envelope-from np@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8BD4D1D58; Wed, 2 Dec 2015 23:55:00 +0000 (UTC) (envelope-from np@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id tB2NsxH1072063; Wed, 2 Dec 2015 23:54:59 GMT (envelope-from np@FreeBSD.org) Received: (from np@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id tB2NsxT8072060; Wed, 2 Dec 2015 23:54:59 GMT (envelope-from np@FreeBSD.org) Message-Id: <201512022354.tB2NsxT8072060@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: np set sender to np@FreeBSD.org using -f From: Navdeep Parhar Date: Wed, 2 Dec 2015 23:54:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r291664 - projects/cxl_iscsi/sys/dev/cxgbe/cxgbei X-SVN-Group: projects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Dec 2015 23:55:00 -0000 Author: np Date: Wed Dec 2 23:54:59 2015 New Revision: 291664 URL: https://svnweb.freebsd.org/changeset/base/291664 Log: Make full use of the pool of worker threads instead of using the first one all the time. Modified: projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.c projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.h projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/icl_cxgbei.c Modified: projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.c ============================================================================== --- projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.c Wed Dec 2 23:44:29 2015 (r291663) +++ projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.c Wed Dec 2 23:54:59 2015 (r291664) @@ -90,6 +90,10 @@ __FBSDID("$FreeBSD$"); #include "cxgbei.h" #include "cxgbei_ulp2_ddp.h" +static int worker_thread_count; +static struct cxgbei_worker_thread_softc *cwt_softc; +static struct proc *cxgbei_proc; + /* XXXNP some header instead. */ struct icl_pdu *icl_cxgbei_new_pdu(int); void icl_cxgbei_new_pdu_set_conn(struct icl_pdu *, struct icl_conn *); @@ -718,7 +722,7 @@ do_rx_iscsi_ddp(struct sge_iq *iq, const STAILQ_INSERT_TAIL(&icc->rcvd_pdus, ip, ip_next); if ((icc->rx_flags & RXF_ACTIVE) == 0) { - struct cxgbei_worker_thread_softc *cwt = icc->cwt; + struct cxgbei_worker_thread_softc *cwt = &cwt_softc[icc->cwt]; mtx_lock(&cwt->cwt_lock); icc->rx_flags |= RXF_ACTIVE; @@ -884,10 +888,6 @@ static struct uld_info cxgbei_uld_info = .deactivate = cxgbei_deactivate, }; -static int worker_thread_count; -static struct cxgbei_worker_thread_softc *cwt_softc; -static struct proc *cxgbei_proc; - static void cwt_main(void *arg) { @@ -1036,6 +1036,25 @@ stop_worker_threads(void) free(cwt_softc, M_CXGBE); } +/* Select a worker thread for a connection. */ +u_int +cxgbei_select_worker_thread(struct icl_cxgbei_conn *icc) +{ + struct adapter *sc = icc->sc; + struct toepcb *toep = icc->toep; + u_int i, n; + + n = worker_thread_count / sc->sge.nofldrxq; + if (n > 0) + i = toep->port->port_id * n + arc4random() % n; + else + i = arc4random() % worker_thread_count; + + CTR3(KTR_CXGBE, "%s: tid %u, cwt %u", __func__, toep->tid, i); + + return (i); +} + static int cxgbei_mod_load(void) { Modified: projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.h ============================================================================== --- projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.h Wed Dec 2 23:44:29 2015 (r291663) +++ projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/cxgbei.h Wed Dec 2 23:54:59 2015 (r291664) @@ -61,9 +61,9 @@ struct icl_cxgbei_conn { /* Receive related. */ u_int rx_flags; /* protected by so_rcv lock */ + u_int cwt; STAILQ_HEAD(, icl_pdu) rcvd_pdus; /* protected by so_rcv lock */ TAILQ_ENTRY(icl_cxgbei_conn) rx_link; /* protected by cwt lock */ - struct cxgbei_worker_thread_softc *cwt; }; static inline struct icl_cxgbei_conn * @@ -153,6 +153,7 @@ struct cxgbei_data { void cxgbei_conn_task_reserve_itt(void *, void **, void *, unsigned int *); void cxgbei_conn_transfer_reserve_ttt(void *, void **, void *, unsigned int *); void cxgbei_cleanup_task(void *, void *); +u_int cxgbei_select_worker_thread(struct icl_cxgbei_conn *); struct cxgbei_ulp2_pagepod_hdr; int t4_ddp_set_map(struct cxgbei_data *, void *, Modified: projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/icl_cxgbei.c ============================================================================== --- projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/icl_cxgbei.c Wed Dec 2 23:44:29 2015 (r291663) +++ projects/cxl_iscsi/sys/dev/cxgbe/cxgbei/icl_cxgbei.c Wed Dec 2 23:54:59 2015 (r291664) @@ -597,9 +597,6 @@ set_ulp_mode_iscsi(struct adapter *sc, s t4_set_tcb_field(sc, toep, 1, 0, 0xfff, val); } -/* XXXNP */ -extern struct cxgbei_worker_thread_softc *cwt_softc; - /* * XXXNP: Who is responsible for cleaning up the socket if this returns with an * error? Review all error paths. @@ -681,7 +678,7 @@ icl_cxgbei_conn_handoff(struct icl_conn toep = tp->t_toe; MPASS(toep->port->adapter == icc->sc); icc->toep = toep; - icc->cwt = &cwt_softc[0]; /* XXXNP */ + icc->cwt = cxgbei_select_worker_thread(icc); icc->ulp_submode = 0; if (ic->ic_header_crc32c) icc->ulp_submode |= ULP_CRC_HEADER;