From h2+lists2024@fsfe.org Wed Feb 28 14:26:38 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlGrM0tBMz5Bh4k for ; Wed, 28 Feb 2024 14:26:43 +0000 (UTC) (envelope-from h2+lists2024@fsfe.org) Received: from einhorn-mail-out.in-berlin.de (einhorn.in-berlin.de [192.109.42.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.in-berlin.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlGrL01ZTz4kkw for ; Wed, 28 Feb 2024 14:26:41 +0000 (UTC) (envelope-from h2+lists2024@fsfe.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=fsfe.org (policy=none); spf=fail (mx1.freebsd.org: domain of h2+lists2024@fsfe.org does not designate 192.109.42.8 as permitted sender) smtp.mailfrom=h2+lists2024@fsfe.org X-Envelope-From: h2+lists2024@fsfe.org X-Envelope-To: Received: from authenticated.user (localhost [127.0.0.1]) by einhorn.in-berlin.de with ESMTPSA id 41SEQdfb736923 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for ; Wed, 28 Feb 2024 15:26:39 +0100 Content-Type: multipart/alternative; boundary="------------c0lgN0cqLguSgKOE0eOBZm7O" Message-ID: <2e1bc8fa-2acb-4cbe-beea-5334f0cbacb7@fsfe.org> Date: Wed, 28 Feb 2024 15:26:38 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Hannes Hauswedell Subject: Re: NFS performance with 10GBase-T To: freebsd-net@freebsd.org References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> Content-Language: en-US In-Reply-To: X-Spamd-Bar: - X-Spamd-Result: default: False [-1.99 / 15.00]; R_SPF_FAIL(1.00)[-all]; NEURAL_HAM_SHORT(-0.99)[-0.991]; NEURAL_HAM_LONG(-0.99)[-0.990]; NEURAL_HAM_MEDIUM(-0.92)[-0.922]; RCVD_IN_DNSWL_LOW(-0.10)[192.109.42.8:from]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[fsfe.org : No valid SPF, No valid DKIM,none]; RWL_MAILSPIKE_GOOD(-0.10)[192.109.42.8:from]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TAGGED_FROM(0.00)[lists2024]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:29670, ipnet:192.109.42.0/24, country:DE]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~] X-Rspamd-Queue-Id: 4TlGrL01ZTz4kkw This is a multi-part message in MIME format. --------------c0lgN0cqLguSgKOE0eOBZm7O Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit >  Hannes >    In the dmesg posted I see that you have a epair loaded . Are you > trying to do NFS inside of a Jail ? > No, the NFS server is running on the regular FreeBSD (host). The epair is used to give an (unrelated) jail a different mac-address for obscure reasons that I could elaborate but have nothing to do with the matter at hand :D > Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames > a long time ago ?  I know in isilon land > their FreeBSD can do this, but I can't say I have any idea how it's > being done. They do have some kernel auto-tune magic as well > to find optimal send and receive buffers. Maybe what we are seeing is > Linux having better ergonomics on buffers out of the box ? > > Hannes >   Can you post your boot.conf and sysctl.conf settings. Sure! I assume you mean loader.conf? I have posted the contents below. If any other information would help, let me know! Best regards, Hannes Server's loader.conf: cryptodev_load="YES" zfs_load="YES" kern.ipc.maxsockbuf=4737024 if_ix_updated_load="YES" hw.pci.do_power_nodriver=3 hw.acpi.cpu.cx_lowest=Cmax machdep.hwpstate_pkg_ctrl=0 Server's sysctl.conf: security.bsd.see_other_uids=0 security.bsd.see_jail_proc=0 kern.randompid=1 vfs.zfs.arc.max=10737418240 Client's loader.conf: zfs_load="YES" geom_journal_load="YES" snd_hda_load="YES" sem_load="YES" amdtemp_load="YES" msdosfs_iconv_load="YES" hint.p4tcc.0.disabled=1 hint.acpi_throttle.0.disabled=1 efi_max_resolution=2160p kern.ipc.maxsockbuf=4737024 vfs.maxbcachebuf=1048576 Client's  sysctl.conf: vfs.usermount=1 hw.snd.vpc_autoreset=0 kern.ipc.shm_allow_removed=1 debug.debugger_on_panic=0 kern.geom.debugflags=16 kern.ipc.maxsockbuf=4737024 kern.ipc.shmmax=67108864 kern.ipc.shmall=32768 kern.sched.preempt_thresh=224 kern.maxfiles=1000000 hw.syscons.bell=0 --------------c0lgN0cqLguSgKOE0eOBZm7O Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
 Hannes
   In the dmesg posted I see that you have a epair loaded . Are you trying to do NFS inside of a Jail ?


No, the NFS server is running on the regular FreeBSD (host). The epair is used to give an (unrelated) jail a different mac-address for obscure reasons that I could elaborate but have nothing to do with the matter at hand :D


Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a long time ago ?  I know in isilon land
their FreeBSD can do this, but I can't say I have any idea how it's being done. They do have some kernel auto-tune magic as well
to find optimal send and receive buffers. Maybe what we are seeing is Linux having better ergonomics on buffers out of the box ?

Hannes
  Can you post your boot.conf and sysctl.conf settings.


Sure! I assume you mean loader.conf? I have posted the contents below.

If any other information would help, let me know!

Best regards,

Hannes


Server's loader.conf:

cryptodev_load="YES"
zfs_load="YES"
kern.ipc.maxsockbuf=4737024
if_ix_updated_load="YES"
hw.pci.do_power_nodriver=3
hw.acpi.cpu.cx_lowest=Cmax
machdep.hwpstate_pkg_ctrl=0

Server's sysctl.conf:

security.bsd.see_other_uids=0
security.bsd.see_jail_proc=0
kern.randompid=1 
vfs.zfs.arc.max=10737418240

Client's loader.conf:

zfs_load="YES"
geom_journal_load="YES"
snd_hda_load="YES"
sem_load="YES"
amdtemp_load="YES"
msdosfs_iconv_load="YES"
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
efi_max_resolution=2160p
kern.ipc.maxsockbuf=4737024
vfs.maxbcachebuf=1048576

Client's  sysctl.conf:

vfs.usermount=1
hw.snd.vpc_autoreset=0
kern.ipc.shm_allow_removed=1
debug.debugger_on_panic=0
kern.geom.debugflags=16
kern.ipc.maxsockbuf=4737024
kern.ipc.shmmax=67108864
kern.ipc.shmall=32768
kern.sched.preempt_thresh=224
kern.maxfiles=1000000
hw.syscons.bell=0

--------------c0lgN0cqLguSgKOE0eOBZm7O--