Date: Sun, 7 May 2023 21:46:38 -0400 From: Cheng Cui <cc@freebsd.org> To: Yuri <yuri@aetern.org> Cc: freebsd-net@freebsd.org Subject: Re: missing hw.em.msix support in FreeBSD 14.0-CURRENT? Message-ID: <CAGaXuiJb3-NWaD955-%2BLLTM5gq0OEd7vsLEq36%2Bs6AuU0BdQDQ@mail.gmail.com> In-Reply-To: <CAGaXuiKsB8YUh_bvewXwkUm2piHLr60PjgQzyek0FQ2OceF_tA@mail.gmail.com> References: <CAGaXuiKNLjWRE=a15k_qU2K-pcHoPfT9rrfLjCqstCaFb7kwbw@mail.gmail.com> <e7deb146-3e3a-2cc8-45e9-2003242a1293@aetern.org> <CAGaXuiKsB8YUh_bvewXwkUm2piHLr60PjgQzyek0FQ2OceF_tA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000fbc44d05fb24cfe7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Found the RSS support, only if the hardware is ">=3D e1000_82571" from the if_em.c. if (hw->mac.type >=3D em_mac_min) // #define em_mac_min e1000_82571 > ... > scctx->isc_txrx =3D &em_txrx; > ... > > Best Regards, Cheng Cui On Sun, May 7, 2023 at 7:28=E2=80=AFPM Cheng Cui <cc@freebsd.org> wrote: > Hi Yuri, > > Thanks for the update. I am using an Emulab testbed node under the same > type of hardware. Below are the new outputs. > It is supposed to have multiple tx/rx queues under msix. Why are there > only rxq0/txq0? Is it because the CPU has only one core? How can I check = if > RSS is supported? > > root@s1:~ # sysctl -d dev.em.0.iflib.disable_msix > dev.em.0.iflib.disable_msix: disable MSI-X (default 0) > root@s1:~ # sysctl dev.em.0.iflib.disable_msix > dev.em.0.iflib.disable_msix: 0 > root@s1:~ # sysctl dev.em.0.iflib > dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 > dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 > dev.em.0.iflib.rxq0.rxq_fl0.cidx: 959 > dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958 > dev.em.0.iflib.rxq0.cpu: 1 > dev.em.0.iflib.txq0.r_abdications: 0 > dev.em.0.iflib.txq0.r_restarts: 0 > dev.em.0.iflib.txq0.r_stalls: 0 > dev.em.0.iflib.txq0.r_starts: 110443 > dev.em.0.iflib.txq0.r_drops: 0 > dev.em.0.iflib.txq0.r_enqueues: 110444 > dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_tail: 1903 cidx: 190= 3 > state: IDLE > dev.em.0.iflib.txq0.txq_cleaned: 115208 > dev.em.0.iflib.txq0.txq_processed: 115251 > dev.em.0.iflib.txq0.txq_in_use: 42 > dev.em.0.iflib.txq0.txq_cidx_processed: 565 > dev.em.0.iflib.txq0.txq_cidx: 525 > dev.em.0.iflib.txq0.txq_pidx: 568 > dev.em.0.iflib.txq0.no_tx_dma_setup: 0 > dev.em.0.iflib.txq0.txd_encap_efbig: 0 > dev.em.0.iflib.txq0.tx_map_failed: 0 > dev.em.0.iflib.txq0.no_desc_avail: 0 > dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 > dev.em.0.iflib.txq0.m_pullups: 0 > dev.em.0.iflib.txq0.mbuf_defrag: 0 > dev.em.0.iflib.txq0.cpu: 0 > dev.em.0.iflib.override_nrxds: 0 > dev.em.0.iflib.override_ntxds: 0 > dev.em.0.iflib.use_logical_cores: 0 > dev.em.0.iflib.separate_txrx: 0 > dev.em.0.iflib.core_offset: 0 > dev.em.0.iflib.tx_abdicate: 0 > dev.em.0.iflib.rx_budget: 0 > dev.em.0.iflib.disable_msix: 0 > dev.em.0.iflib.override_qs_enable: 0 > dev.em.0.iflib.override_nrxqs: 0 > dev.em.0.iflib.override_ntxqs: 0 > dev.em.0.iflib.driver_version: 7.7.8-fbsd > root@s1:~ # > > > Best Regards, > Cheng Cui > > > On Sun, May 7, 2023 at 4:18=E2=80=AFPM Yuri <yuri@aetern.org> wrote: > >> Cheng Cui wrote: >> > Hello, >> > >> > I am using this em(4) driver for some tests. According to the man page >> > below, there should be a "hw.em.msix" sysctl to enable/disable MSI-X >> > style interrupts. But why is it missing in my system below? >> > https://man.freebsd.org/cgi/man.cgi?em(4) >> > <https://man.freebsd.org/cgi/man.cgi?em(4)>; >> > >> > root@s1:~ # dmesg | egrep "8254" >> > Timecounter "i8254" frequency 1193182 Hz quality 0 >> > Event timer "i8254" frequency 1193182 Hz quality 100 >> > em0: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xdcc0-0xdcff mem >> > 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6 >> > em1: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xccc0-0xccff mem >> > 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7 >> > em2: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbcc0-0xbcff >> > mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9 >> > em3: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbc80-0xbcbf >> > mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9 >> > em4: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xacc0-0xacff >> > mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10 >> > em5: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xac80-0xacbf >> > mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10 >> > >> > root@s1:~ # ifconfig em5 >> > em5: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu >> 1500 >> > >> options=3D481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN= _HWFILTER,NOMAP> >> > ether 00:04:23:b7:1e:1d >> > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >> > media: Ethernet 1000baseT <full-duplex> >> > status: active >> > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >> > >> > root@s1:~ # sysctl hw.em >> > hw.em.max_interrupt_rate: 8000 >> > hw.em.eee_setting: 1 >> > hw.em.rx_process_limit: 100 >> > hw.em.sbp: 0 >> > hw.em.smart_pwr_down: 0 >> > hw.em.rx_abs_int_delay: 66 >> > hw.em.tx_abs_int_delay: 66 >> > hw.em.rx_int_delay: 0 >> > hw.em.tx_int_delay: 66 >> > hw.em.disable_crc_stripping: 0 >> > root@s1:~ # >> >> As if_em was converted to iflib, I guess the related sysctl would now be >> dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check). >> > --000000000000fbc44d05fb24cfe7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Found the RSS support, only if the hardware is "= >=3D e1000_82571" from the if_em.c.<br></div><div><br></div><blockq= uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p= x solid rgb(204,204,204);padding-left:1ex"><div><pre><code>if (hw->mac.t= ype >=3D em_mac_min) // #define em_mac_min e1000_82571 ... scctx->isc_txrx =3D &em_txrx; ...</code></pre></div></blockquote><div><div><div dir=3D"ltr" class=3D"= gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><= br></div>Best Regards,<div>Cheng Cui</div></div></div></div><br></div></div= ><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Su= n, May 7, 2023 at 7:28=E2=80=AFPM Cheng Cui <<a href=3D"mailto:cc@freebs= d.org">cc@freebsd.org</a>> wrote:<br></div><blockquote class=3D"gmail_qu= ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex"><div dir=3D"ltr"><div>Hi Yuri,</div><div><br></div><di= v>Thanks for the=C2=A0update. I am using an Emulab testbed node under the s= ame type of hardware. Below are the new outputs.<br></div><div>It is suppos= ed to have multiple tx/rx queues under msix. Why are there only rxq0/txq0? = Is it because the CPU has only one core? How can I check if RSS is supporte= d?<br></div><div><br></div><div>root@s1:~ # sysctl -d dev.em.0.iflib.disabl= e_msix<br>dev.em.0.iflib.disable_msix: disable MSI-X (default 0)<br>root@s1= :~ # sysctl dev.em.0.iflib.disable_msix<br>dev.em.0.iflib.disable_msix: 0<b= r>root@s1:~ # sysctl dev.em.0.iflib<br>dev.em.0.iflib.rxq0.rxq_fl0.buf_size= : 2048<br>dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023<br>dev.em.0.iflib.rxq0.= rxq_fl0.cidx: 959<br>dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958<br>dev.em.0.ifli= b.rxq0.cpu: 1<br>dev.em.0.iflib.txq0.r_abdications: 0<br>dev.em.0.iflib.txq= 0.r_restarts: 0<br>dev.em.0.iflib.txq0.r_stalls: 0<br>dev.em.0.iflib.txq0.r= _starts: 110443<br>dev.em.0.iflib.txq0.r_drops: 0<br>dev.em.0.iflib.txq0.r_= enqueues: 110444<br>dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_ta= il: 1903 cidx: 1903 state: IDLE<br>dev.em.0.iflib.txq0.txq_cleaned: 115208<= br>dev.em.0.iflib.txq0.txq_processed: 115251<br>dev.em.0.iflib.txq0.txq_in_= use: 42<br>dev.em.0.iflib.txq0.txq_cidx_processed: 565<br>dev.em.0.iflib.tx= q0.txq_cidx: 525<br>dev.em.0.iflib.txq0.txq_pidx: 568<br>dev.em.0.iflib.txq= 0.no_tx_dma_setup: 0<br>dev.em.0.iflib.txq0.txd_encap_efbig: 0<br>dev.em.0.= iflib.txq0.tx_map_failed: 0<br>dev.em.0.iflib.txq0.no_desc_avail: 0<br>dev.= em.0.iflib.txq0.mbuf_defrag_failed: 0<br>dev.em.0.iflib.txq0.m_pullups: 0<b= r>dev.em.0.iflib.txq0.mbuf_defrag: 0<br>dev.em.0.iflib.txq0.cpu: 0<br>dev.e= m.0.iflib.override_nrxds: 0<br>dev.em.0.iflib.override_ntxds: 0<br>dev.em.0= .iflib.use_logical_cores: 0<br>dev.em.0.iflib.separate_txrx: 0<br>dev.em.0.= iflib.core_offset: 0<br>dev.em.0.iflib.tx_abdicate: 0<br>dev.em.0.iflib.rx_= budget: 0<br>dev.em.0.iflib.disable_msix: 0<br>dev.em.0.iflib.override_qs_e= nable: 0<br>dev.em.0.iflib.override_nrxqs: 0<br>dev.em.0.iflib.override_ntx= qs: 0<br>dev.em.0.iflib.driver_version: 7.7.8-fbsd<br>root@s1:~ # <br></div= ><div dir=3D"ltr"><br clear=3D"all"><div><div dir=3D"ltr"><div dir=3D"ltr">= <div><br></div>Best Regards,<div>Cheng Cui</div></div></div></div><br></div= ><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Su= n, May 7, 2023 at 4:18=E2=80=AFPM Yuri <<a href=3D"mailto:yuri@aetern.or= g" target=3D"_blank">yuri@aetern.org</a>> wrote:<br></div><blockquote cl= ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid= rgb(204,204,204);padding-left:1ex">Cheng Cui wrote:<br> > Hello,<br> > <br> > I am using this em(4) driver for some tests. According to the man page= <br> > below, there should be a "hw.em.msix" sysctl to enable/disab= le MSI-X<br> > style interrupts. But why is it missing in my system below?<br> > <a href=3D"https://man.freebsd.org/cgi/man.cgi?em(4)" rel=3D"noreferre= r" target=3D"_blank">https://man.freebsd.org/cgi/man.cgi?em(4)</a><br>; > <<a href=3D"https://man.freebsd.org/cgi/man.cgi?em(4)" rel=3D"noref= errer" target=3D"_blank">https://man.freebsd.org/cgi/man.cgi?em(4)</a>><= br> > <br> > root@s1:~ # dmesg | egrep "8254"<br> > Timecounter "i8254" frequency 1193182 Hz quality 0<br> > Event timer "i8254" frequency 1193182 Hz quality 100<br> > em0: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xdcc0-0xdcff me= m<br> > 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6<br> > em1: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xccc0-0xccff me= m<br> > 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7<br> > em2: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbcc0-= 0xbcff<br> > mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9<br> > em3: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbc80-= 0xbcbf<br> > mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9<br> > em4: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xacc0-= 0xacff<br> > mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10<br> > em5: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xac80-= 0xacbf<br> > mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10<br> > <br> > root@s1:~ # ifconfig em5<br> > em5: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric= 0 mtu 1500<br> > options=3D481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM= ,VLAN_HWFILTER,NOMAP><br> > ether 00:04:23:b7:1e:1d<br> > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255<br> > media: Ethernet 1000baseT <full-duplex><br> > status: active<br> > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL><br> > <br> > root@s1:~ # sysctl hw.em<br> > hw.em.max_interrupt_rate: 8000<br> > hw.em.eee_setting: 1<br> > hw.em.rx_process_limit: 100<br> > hw.em.sbp: 0<br> > hw.em.smart_pwr_down: 0<br> > hw.em.rx_abs_int_delay: 66<br> > hw.em.tx_abs_int_delay: 66<br> > hw.em.rx_int_delay: 0<br> > hw.em.tx_int_delay: 66<br> > hw.em.disable_crc_stripping: 0<br> > root@s1:~ #<br> <br> As if_em was converted to iflib, I guess the related sysctl would now be<br= > dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check)= .<br> </blockquote></div></div> </blockquote></div> --000000000000fbc44d05fb24cfe7--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGaXuiJb3-NWaD955-%2BLLTM5gq0OEd7vsLEq36%2Bs6AuU0BdQDQ>