From owner-dev-commits-src-branches@freebsd.org Mon Jul 26 16:14:05 2021 Return-Path: Delivered-To: dev-commits-src-branches@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C492166F99A; Mon, 26 Jul 2021 16:14:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GYQ3J0spvz3pgj; Mon, 26 Jul 2021 16:14:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 643F31A491; Mon, 26 Jul 2021 16:14:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 16QGE0CR006195; Mon, 26 Jul 2021 16:14:00 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 16QGE0f9006194; Mon, 26 Jul 2021 16:14:00 GMT (envelope-from git) Date: Mon, 26 Jul 2021 16:14:00 GMT Message-Id: <202107261614.16QGE0f9006194@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: fdab56d1f605 - stable/13 - mlx5en: Allow binding channels to CPUs when RSS is not enabled. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: fdab56d1f6054f33ffab0b4498ea114acbb475ac Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-branches@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commits to the stable branches of the FreeBSD src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2021 16:14:06 -0000 The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=fdab56d1f6054f33ffab0b4498ea114acbb475ac commit fdab56d1f6054f33ffab0b4498ea114acbb475ac Author: Hans Petter Selasky AuthorDate: 2021-06-16 13:02:02 +0000 Commit: Hans Petter Selasky CommitDate: 2021-07-26 16:04:33 +0000 mlx5en: Allow binding channels to CPUs when RSS is not enabled. Submitted by: Netflix Reviewed by: kib Sponsored by: Mellanox Technologies // NVIDIA Networking (cherry picked from commit c8d16d1e084dc14191491e95ce226d3ce8b39275) --- sys/dev/mlx5/mlx5_en/en.h | 6 ++++-- sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c | 13 ++++++++++++- sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 24 ++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index c84e2af237b2..8556a4364ddd 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015-2019 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -710,7 +710,9 @@ struct mlx5e_params { m(+1, u64, diag_general_enable, "diag_general_enable", "0: Disabled 1: Enabled") \ m(+1, u64, hw_mtu, "hw_mtu", "Current hardware MTU value") \ m(+1, u64, mc_local_lb, "mc_local_lb", "0: Local multicast loopback enabled 1: Disabled") \ - m(+1, u64, uc_local_lb, "uc_local_lb", "0: Local unicast loopback enabled 1: Disabled") + m(+1, u64, uc_local_lb, "uc_local_lb", "0: Local unicast loopback enabled 1: Disabled") \ + m(+1, s64, irq_cpu_base, "irq_cpu_base", "-1: Don't bind IRQ 0..NCPU-1: select this base CPU when binding IRQs") \ + m(+1, s64, irq_cpu_stride, "irq_cpu_stride", "0..NCPU-1: Distance between IRQ vectors when binding them") #define MLX5E_PARAMS_NUM (0 MLX5E_PARAMS(MLX5E_STATS_COUNT)) diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c b/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c index 4070fe4331a1..f7228989ee04 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015-2019 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -1250,6 +1250,15 @@ mlx5e_ethtool_handler(SYSCTL_HANDLER_ARGS) } break; + case MLX5_PARAM_OFFSET(irq_cpu_base): + case MLX5_PARAM_OFFSET(irq_cpu_stride): + if (was_opened) { + /* network interface must toggled */ + mlx5e_close_locked(priv->ifp); + mlx5e_open_locked(priv->ifp); + } + break; + default: break; } @@ -1413,6 +1422,8 @@ mlx5e_create_ethtool(struct mlx5e_priv *priv) int i; /* set some defaults */ + priv->params_ethtool.irq_cpu_base = -1; /* disabled */ + priv->params_ethtool.irq_cpu_stride = 1; priv->params_ethtool.tx_queue_size_max = 1 << MLX5E_PARAMS_MAXIMUM_LOG_SQ_SIZE; priv->params_ethtool.rx_queue_size_max = 1 << MLX5E_PARAMS_MAXIMUM_LOG_RQ_SIZE; priv->params_ethtool.tx_queue_size = 1 << priv->params.log_sq_size; diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index b67f382522e7..9345798006f2 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -2446,6 +2446,30 @@ mlx5e_open_channels(struct mlx5e_priv *priv) err = mlx5e_open_channel(priv, cparam, &priv->channel[i]); if (err) goto err_close_channels; + + /* Bind interrupt vectors, if any. */ + if (priv->params_ethtool.irq_cpu_base > -1) { + cpuset_t cpuset; + int cpu; + int irq; + int eqn; + int nirq; + + err = mlx5_vector2eqn(priv->mdev, i, + &eqn, &nirq); + + /* error here is non-fatal */ + if (err != 0) + continue; + + irq = priv->mdev->priv.msix_arr[nirq].vector; + cpu = (unsigned)(priv->params_ethtool.irq_cpu_base + + i * priv->params_ethtool.irq_cpu_stride) % (unsigned)mp_ncpus; + + CPU_ZERO(&cpuset); + CPU_SET(cpu, &cpuset); + intr_setaffinity(irq, CPU_WHICH_INTRHANDLER, &cpuset); + } } for (j = 0; j < priv->params.num_channels; j++) {