Date: Tue, 18 Feb 2020 16:09:46 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: freebsd-net <freebsd-net@FreeBSD.org> Cc: Patrick Kelsey <pkelsey@FreeBSD.org>, Josh Paetzel <jpaetzel@FreeBSD.org> Subject: terrible if_vmx / vmxnet3 rx performance with lro (post iflib) Message-ID: <40c4a4df-3df6-d95d-53c2-eef905ff45b1@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
At work we are seeing some rather terrible vmxnet3 rx performance when lro is
enabled. We have it enabled because with the pre-iflib vmxnet3 it improved the
performance quite significantly. For example, in a particular test
configuration we could receive ~1 Gbps w/o lro and ~1.8 Gbps with it.
With the latest driver we can see the same ~1 Gbps w/o lro, but enabling lro
kills the performance -- it's ~40 Mbps.
There is one significant difference that I see between the old and the new code.
Previously, vmxnet3 would just communicate to the virtual hardware that it wants
lro enabled, e.g.:
if (ifp->if_capenable & IFCAP_LRO)
ds->upt_features |= UPT1_F_LRO;
At the same time, it never used tcp_lro_* routines.
Nowadays, if lro is enabled, then the iflib code would pass eligible packets to
tcp_lro_rx() and so on.
I hacked iflib code to not do that while leaving vmxnet3 code unchanged.
The performance with lro enabled immediately improved to ~1.4 Gbps.
It's still a bit worse than the pre-iflib code, but much better than iflib+lro.
I cannot explain my observations but I confirmed them multiple times.
Perhaps, the way that the "hardware" lro works hits a degenerate case either in
tcp_lro_* code or in how iflib calls that code.
Perhaps, iflib needs a special lro quirk for a driver such vmxnet3?
P.S.
My general experience with post-iflib vmxnet3 is that vmxnet3 has some
peculiarities that result in a certain "impedance mismatch" with iflib.
Although we now have a bit less code and it is a bit more regular, there are a
few significant (for us, at least) problems:
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243126
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240608
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392
- the problem described above
- a couple of issues that we already fixed or worked around
We are contemplating locally reverting to the pre-iflib vmxnet3 and we are
wondering if the conversion was really worth it in general.
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?40c4a4df-3df6-d95d-53c2-eef905ff45b1>