Date: Wed, 28 Feb 2024 15:26:38 +0100 From: Hannes Hauswedell <h2+lists2024@fsfe.org> To: freebsd-net@freebsd.org Subject: Re: NFS performance with 10GBase-T Message-ID: <2e1bc8fa-2acb-4cbe-beea-5334f0cbacb7@fsfe.org> In-Reply-To: <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com> References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> <CAM5tNy5guh99YZ7DLxgVsaE2LcXAqSSjs2FocMYpGJ9tspkYEQ@mail.gmail.com> <CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY%2BHwpQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------c0lgN0cqLguSgKOE0eOBZm7O
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
> Hannes
> In the dmesg posted I see that you have a epair loaded . Are you
> trying to do NFS inside of a Jail ?
>
No, the NFS server is running on the regular FreeBSD (host). The epair
is used to give an (unrelated) jail a different mac-address for obscure
reasons that I could elaborate but have nothing to do with the matter at
hand :D
> Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames
> a long time ago ? I know in isilon land
> their FreeBSD can do this, but I can't say I have any idea how it's
> being done. They do have some kernel auto-tune magic as well
> to find optimal send and receive buffers. Maybe what we are seeing is
> Linux having better ergonomics on buffers out of the box ?
>
> Hannes
> Can you post your boot.conf and sysctl.conf settings.
Sure! I assume you mean loader.conf? I have posted the contents below.
If any other information would help, let me know!
Best regards,
Hannes
Server's loader.conf:
cryptodev_load="YES"
zfs_load="YES"
kern.ipc.maxsockbuf=4737024
if_ix_updated_load="YES"
hw.pci.do_power_nodriver=3
hw.acpi.cpu.cx_lowest=Cmax
machdep.hwpstate_pkg_ctrl=0
Server's sysctl.conf:
security.bsd.see_other_uids=0
security.bsd.see_jail_proc=0
kern.randompid=1
vfs.zfs.arc.max=10737418240
Client's loader.conf:
zfs_load="YES"
geom_journal_load="YES"
snd_hda_load="YES"
sem_load="YES"
amdtemp_load="YES"
msdosfs_iconv_load="YES"
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
efi_max_resolution=2160p
kern.ipc.maxsockbuf=4737024
vfs.maxbcachebuf=1048576
Client's sysctl.conf:
vfs.usermount=1
hw.snd.vpc_autoreset=0
kern.ipc.shm_allow_removed=1
debug.debugger_on_panic=0
kern.geom.debugflags=16
kern.ipc.maxsockbuf=4737024
kern.ipc.shmmax=67108864
kern.ipc.shmall=32768
kern.sched.preempt_thresh=224
kern.maxfiles=1000000
hw.syscons.bell=0
--------------c0lgN0cqLguSgKOE0eOBZm7O
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body style="padding-bottom: 1px;">
<br>
<blockquote type="cite"
cite="mid:CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY+HwpQ@mail.gmail.com">
<div dir="ltr">
<div> Hannes</div>
<div> In the dmesg posted I see that you have a epair loaded .
Are you trying to do NFS inside of a Jail ? <br>
</div>
<div><br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>No, the NFS server is running on the regular FreeBSD (host). The
epair is used to give an (unrelated) jail a different mac-address
for obscure reasons that I could elaborate but have nothing to do
with the matter at hand :D</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAMXt9NZYKFnaXLnS10PuARvj2B7oYzd98Y5aExCkhz3cY+HwpQ@mail.gmail.com">
<div dir="ltr">
<div>Rick, Didn't someone from Isilon or Dell/EMC post about the
9K frames a long time ago ? I know in isilon land</div>
<div>their FreeBSD can do this, but I can't say I have any idea
how it's being done. They do have some kernel auto-tune magic
as well</div>
<div>to find optimal send and receive buffers. Maybe what we are
seeing is Linux having better ergonomics on buffers out of the
box ?</div>
<div><br>
</div>
<div>Hannes</div>
<div> Can you post your boot.conf and sysctl.conf settings. <br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Sure! I assume you mean loader.conf? I have posted the contents
below.</p>
<p>If any other information would help, let me know!</p>
<p>Best regards,</p>
<p>Hannes<br>
</p>
<p><br>
</p>
<p>Server's loader.conf:<br>
</p>
<p>cryptodev_load="YES" <br>
zfs_load="YES" <br>
kern.ipc.maxsockbuf=4737024 <br>
if_ix_updated_load="YES" <br>
hw.pci.do_power_nodriver=3 <br>
hw.acpi.cpu.cx_lowest=Cmax <br>
machdep.hwpstate_pkg_ctrl=0</p>
<p>Server's sysctl.conf:</p>
<p>security.bsd.see_other_uids=0 <br>
security.bsd.see_jail_proc=0 <br>
kern.randompid=1 <br>
vfs.zfs.arc.max=10737418240</p>
<p>Client's loader.conf:</p>
<p>zfs_load="YES"
<br>
geom_journal_load="YES"
<br>
snd_hda_load="YES"
<br>
sem_load="YES"
<br>
amdtemp_load="YES"
<br>
msdosfs_iconv_load="YES"
<br>
hint.p4tcc.0.disabled=1
<br>
hint.acpi_throttle.0.disabled=1
<br>
efi_max_resolution=2160p
<br>
kern.ipc.maxsockbuf=4737024
<br>
vfs.maxbcachebuf=1048576</p>
<p>Client's sysctl.conf:</p>
<p>vfs.usermount=1
<br>
hw.snd.vpc_autoreset=0
<br>
kern.ipc.shm_allow_removed=1
<br>
debug.debugger_on_panic=0
<br>
kern.geom.debugflags=16
<br>
kern.ipc.maxsockbuf=4737024
<br>
kern.ipc.shmmax=67108864
<br>
kern.ipc.shmall=32768
<br>
kern.sched.preempt_thresh=224
<br>
kern.maxfiles=1000000
<br>
hw.syscons.bell=0<span style="font-family:monospace"><br>
</span></p>
</body>
</html>
--------------c0lgN0cqLguSgKOE0eOBZm7O--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2e1bc8fa-2acb-4cbe-beea-5334f0cbacb7>