From nobody Thu Aug 31 19:51:42 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcBd80q1Bz4rXmh for ; Thu, 31 Aug 2023 19:51:56 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcBd71dg4z3C1Z for ; Thu, 31 Aug 2023 19:51:55 +0000 (UTC) (envelope-from v.maffione@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=dYMIWotV; spf=pass (mx1.freebsd.org: domain of v.maffione@gmail.com designates 2607:f8b0:4864:20::102b as permitted sender) smtp.mailfrom=v.maffione@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-27178b6417fso933641a91.0 for ; Thu, 31 Aug 2023 12:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693511513; x=1694116313; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3M9kWrT1XppFW3AOSuBUt40MqEniDCWAccugeFZMV7I=; b=dYMIWotV67mEwWvrnpG1Ta8SlyVBX6nePOuPSrLaW4VnM5NbWAE3pyednSVNn+XKtC B3szE+aUh/Rh2DqLkh0tKGxWHko7BHajR4uUcq0cFhbjHSUwvkVFGAZ6lpiZ00D45Opd uaktgxdWWlKrmBhUatpYi1gpNToiAcq1Ct55AF4uFuynMySxuF8JheTKly7LiEh74C0w w0Mdpj9dHIhOHbYOtj+oT/ZlQvTWem1218GKZa0JrQwou4bsZoU0llEggSp3ffgx3rW4 9G9TplMzphSJdbGvQvkHVxwIEnAtTTVvlTeMuGr1JPWtm81CwiVtbcnVVKTPIkaNdPQS sGZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693511513; x=1694116313; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3M9kWrT1XppFW3AOSuBUt40MqEniDCWAccugeFZMV7I=; b=TYG6pAZgzVlP1rJEANF2eX3KgJttGy3gC/KlyEOhQPraR4ungauyenyoezraUhcGQl 57ANFfKeTE8IaUaACt7/aCVHA6Z/d0FQ+iIQ59VOjlvy0rs/GguhuW8bg/yVHiSKD4xS yULxWNENQksSXL1aUvV9JiTRvLJgK6+DSj+dYCDgsz83pFFIm9UNzrYLOurJr4idCZPk 3JCX7rtqUs6x2aDuQ4Po7X84xiYKnpMQS4v0pWuz8HKAQuhuQAOjDiB+ya2A3PiaB7OE TbSDvYtxd/oMO+UPqmmXu0qWYSn6wWOXNyrPPzHFstwotici+hsL+xEFOd2L+FgqdG0V hQFQ== X-Gm-Message-State: AOJu0Yyb+8z+9u/wda6hopsGyjLXukG6TITLEBnE7zCWIHEGj5CW7206 QgF5Xo6hgGjvY6/JLG3MmoKOcpI03BFxxwWgYN1XaUgm X-Google-Smtp-Source: AGHT+IFETjWbaiC/TbP1EWUssUd3nxmpAsMQE+GORKepsaLKbWs9xtaScPX2GtkEmgXCdGZrt2DZzmPisR/aAt/WMWQ= X-Received: by 2002:a17:90a:c58e:b0:268:14d7:bc34 with SMTP id l14-20020a17090ac58e00b0026814d7bc34mr360287pjt.20.1693511513512; Thu, 31 Aug 2023 12:51:53 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Vincenzo Maffione Date: Thu, 31 Aug 2023 21:51:42 +0200 Message-ID: Subject: Re: About IFLIB compliant network device driver development To: PAVEL POPA Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="0000000000002fa14106043d60aa" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEFALL_USER(0.00)[vmaffione]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102b:from]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::102b:server fail]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RcBd71dg4z3C1Z --0000000000002fa14106043d60aa Content-Type: text/plain; charset="UTF-8" Hi, I think it's pretty common nowadays to have NICs with completion rings/queues. It is definitely appropriate to implement such drivers with iflib, and indeed iflib provides separate callbacks for updating the TX/RX "submission" and "completion" queues: - ift_txd_encap: submit a packet to a TX ring (or submission queue) - ift_txd_flush: flush submitted TX descriptors to the hardware. - ift_txd_credits_update: check the completion queue (or reclaim completed descriptor from the TX ring). - ift_rxd_refill: submit RX descriptors to a RX ring (or submission queue) - ift_rxd_flush: flush submitted RX descriptors to the NIC hardware. - ift_rxd_pkt_get: get a received packet from the completion queue (or reclaim completed descriptor from the RX ring) - ... If you want to look at an example of iflib driver using completion queues, you can check out sys/dev/vmware/vmxnet3/. It's actually a pretty complex example because vmxnet3 is a complex paravirtualized device with multiple versions etc.. However, the complexity does not come from the completion rings... Cheers, Vincenzo Il giorno mer 30 ago 2023 alle ore 15:58 PAVEL POPA ha scritto: > I have a NIC (a SmartNIC actually) for which I have to implement a > driver, which in addition to RX and TX rings exposes also completion > CMPT rings. Due to this additional complication (the CMPT rings), I'm > not sure how appropriate it is to implement such a driver via the > IFLIB framework. Does anyone have any similar experience that can > share? Any suggestions at all, ideas, feedbacks? > > Thanks in advance, > Pavel > > -- Vincenzo --0000000000002fa14106043d60aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
=C2=A0 I think it's pretty common n= owadays to have NICs with completion rings/queues. It is definitely appropr= iate to implement such drivers with iflib, and indeed iflib provides separa= te callbacks for updating the TX/RX "submission" and "comple= tion" queues:
  • ift_txd_encap: submit a packet to a TX= ring (or submission queue)
  • ift_txd_flush: flush submitted TX d= escriptors to the hardware.
  • ift_txd_credits_update: check the compl= etion queue (or reclaim completed descriptor from the TX ring).
  • ift= _rxd_refill: submit RX descriptors to a RX ring (or submission queue)
    • <= li>ift_rxd_flush: flush submitted RX descriptors to the NIC hardware.
  • ift_rxd_pkt_get: get a received packet from the completion queue (or= reclaim completed descriptor from the RX ring)
  • ...
I= f you want to look at an example of iflib driver using completion queues, y= ou can check out sys/dev/vmware/vmxnet3/.

It's= actually a pretty complex example because vmxnet3 is a complex paravirtual= ized device with multiple versions etc..
However, the complexity = does not come from the completion rings...

Cheers,=
=C2=A0 Vincenzo



Il = giorno mer 30 ago 2023 alle ore 15:58 PAVEL POPA <pavel.popa@edu.unife.it> ha scritto:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">I have a NIC (a SmartNIC a= ctually) for which I have to implement a
driver, which in addition to RX and TX rings exposes also completion
CMPT rings. Due to this additional complication (the CMPT rings), I'm not sure how appropriate it is to implement such a driver via the
IFLIB framework. Does anyone have any similar experience that can
share? Any suggestions at all, ideas, feedbacks?

Thanks in advance,
Pavel



--
Vincenzo
--0000000000002fa14106043d60aa--