Date: Mon, 11 Aug 2014 11:27:26 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Wei Hu <weh@microsoft.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, "d@delphij.net" <d@delphij.net> Subject: Re: vRSS support on FreeBSD Message-ID: <CAJ-Vmonr6qP=Gwu600bwnQg0o74-V491gr_1u1uJ0qnp=wZ_Zg@mail.gmail.com> In-Reply-To: <e514d7fdb89c4d2988dfb4300c633a0c@BY1PR0301MB0902.namprd03.prod.outlook.com> References: <184b69414bd246eeacc0d4234a730f2f@BY1PR0301MB0902.namprd03.prod.outlook.com> <CAJ-VmokJ8G-mz%2BL=zJkbQnCHHFBcvqhHjCrHjoWS5i9ViF5qrw@mail.gmail.com> <e514d7fdb89c4d2988dfb4300c633a0c@BY1PR0301MB0902.namprd03.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11 August 2014 02:48, Wei Hu <weh@microsoft.com> wrote:
> CC freebsd-net@ for wider discussion.
>
> Hi Adrian,
>
> Many thanks for the explanation. I checked the if_igb.c and found the f=
lowid field was set in the RX side in igb_rxeof():
>
> Igb_rxeof()
> {
> ...
> #ifdef RSS
> /* XXX set flowtype once this works right */
> rxr->fmp->m_pkthdr.flowid =3D
> le32toh(cur->wb.lower.hi_dword.rss);
> rxr->fmp->m_flags |=3D M_FLOWID;
> ...
> }
>
> I have two questions regarding this.
>
> 1. Is the RSS hash value stored in cur->wb.lower.hi_dword.rss set by the =
NIC hardware?
Yup.
> 2. So the hash value and m_flags are stored in the mbuf related to the re=
ceived packet on the rx side(lgb_rxeof()). But we check the hash value and =
m_flags in mbuf related to the send packet on the tx side (in igb_mq_start(=
)). Does the kernel re-use the same mbuf for tx? If so, how does it know fo=
r the same network stream it should use the same mbuf got from the rx for p=
acket sending? If not, how does the kernel preserve the same hash value acr=
oss the rx mbuf and tx mbuf for same network stream? This seems quite magic=
al to me.
The mbuf flowid/flowtype ends up in the inpcb->inp_flowid /
inpcb->inp_flowtype as part of the TCP receive path.
Then whenever the TCP code outputs an mbuf, it copies the inpcb flow
details out to outbound mbufs.
>
> For the Hyper-V case, the host controls which vCPU it wants to interrupt.=
And the rule can change dynamically based on the load. For a non-busy VM, =
host will send most packets to same vCPU for power saving purpose. For a bu=
sy VM, host will distribute the packets evenly across all vCPUs. This means=
host could change the RSS bucket mapping dynamically. Hyper-V does this by=
sending a mapping table to VM whenever the it needs update. This also mean=
s we cannot use FreeBSD's own bucket mapping which I believe is fixed. Also=
Hyper-V use its own hash key. So do you think it is possible we still use =
the exisiting RSS infrastructure built in FreeBSD in this purpose?
Eventually. Doing rebalancing in RSS is on the TODO list, after I get
the rest of the basic packet handling / routing done.
How's vRSS notify the VM that the mapping table has changed? What's
the format of it look like?
-a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmonr6qP=Gwu600bwnQg0o74-V491gr_1u1uJ0qnp=wZ_Zg>