Date: Sun, 22 Aug 2021 20:05:52 -1000 From: "parv/freebsd" <parv.0zero9+freebsd@gmail.com> To: freebsd-questions@freebsd.org Subject: NFS v3 - FreeBSD 12 server connection getting timed out on CentOS 8 client Message-ID: <CABObuOr5rLSoUhm8LsWr6hQZVmpBGrYvJE%2BnruV1kmB8W-9XnQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi there, CentOS 8 client (host name is replaced with "CentOS8-nfs3-client") ... Linux CentOS8-nfs3-client 4.18.0-305.10.2.el8_4.x86_64 #1 SMP Tue Jul 20 17:25:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ... is having issues connecting to FreeBSD 12 NFS v3 server (host name has been replaced with "FreeBSD-nfsd") ... FreeBSD FreeBSD-nfsd 12.2-RELEASE-p7 FreeBSD 12.2-RELEASE-p7 GENERIC amd64 ... when writing to one of the 2 ZFS pools. On the server, there is not much CPU load; drive activity does not match the connection being timed. On client, CPU getting stuck messages are being printed to everyone when trying to write to the NFS server (a short log is at the end). On the server side, I had not seen any output -- from smartctl or nfsd -- which could indicate such a problem. The client machine had been recently replaced. Same issue was present then, and is now. On the server, I have set "vfs.nfsd.debuglevel=2" to see if there would be any output when the client would complain after connection getting timed out. How do I go about investigating or solve this? - parv A short log from CentOS8-nfs3-client about connection timed out ... Aug 21 08:31:04 CentOS8-nfs3-client kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u24:0:672075] Aug 21 08:31:04 CentOS8-nfs3-client kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u24:10:672632] Aug 21 08:31:04 CentOS8-nfs3-client kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u24:4:672525] Aug 21 08:31:04 CentOS8-nfs3-client kernel: Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c nfnetlink tun bridge stp llc sunrpc vfat fat intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi snd_sof_pci snd_sof_intel_byt snd_sof_intel_ipc snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_xtensa_dsp snd_sof_intel_hda snd_hda_codec_realtek snd_sof snd_hda_codec_generic ledtrig_audio snd_soc_skl snd_soc_sst_ipc snd_soc_sst_dsp x86_pkg_temp_thermal intel_powerclamp snd_hda_ext_core coretemp snd_soc_acpi_intel_match snd_soc_acpi kvm_intel snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation iTCO_wdt mei_wdt iTCO_vendor_support intel_wmi_thunderbolt wmi_bmof snd_soc_core kvm snd_compress soundwire_cadence soundwire_bus irqbypass snd_hda_codec snd_hda_core crct10dif_pclmul Aug 21 08:31:04 CentOS8-nfs3-client kernel: crc32_pclmul snd_hwdep snd_seq ghash_clmulni_intel rapl snd_seq_device intel_cstate snd_pcm intel_uncore snd_timer snd pcspkr soundcore i2c_i801 mei_me mei intel_pch_thermal ie31200_edac wmi acpi_pad intel_pmc_core ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg i915 cec intel_gtt drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm igb ahci e1000e libahci libata dca i2c_algo_bit crc32c_intel video pinctrl_cannonlake fuse Aug 21 08:31:04 CentOS8-nfs3-client kernel: CPU: 2 PID: 672525 Comm: kworker/u24:4 Kdump: loaded Tainted: G W L --------- - - 4.18.0-305.10.2.el8_4.x86_64 #1 Aug 21 08:31:04 CentOS8-nfs3-client kernel: Hardware name: Puget Systems C246M-WU4/C246M-WU4-CF, BIOS F4e 09/16/2020 Aug 21 08:31:04 CentOS8-nfs3-client kernel: Workqueue: rpciod rpc_async_schedule [sunrpc] Aug 21 08:31:04 CentOS8-nfs3-client kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x18f/0x1d0 Aug 21 08:31:04 CentOS8-nfs3-client kernel: Code: c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 00 ac 02 00 48 03 04 cd a0 b8 b7 a7 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 0f 84 6e ff ff ff 48 89 c1 Aug 21 08:31:04 CentOS8-nfs3-client kernel: RSP: 0018:ffff9d9bc7f57de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 Aug 21 08:31:04 CentOS8-nfs3-client kernel: RAX: 0000000000000000 RBX: ffff8a7c795a3000 RCX: 0000000000000004 Aug 21 08:31:04 CentOS8-nfs3-client kernel: RDX: ffff8a8b7c2aac00 RSI: 00000000000c0000 RDI: ffff8a7c795a3670 Aug 21 08:31:04 CentOS8-nfs3-client kernel: RBP: ffff8a7c795a3670 R08: 0000000000100000 R09: 0000000000000000 Aug 21 08:31:04 CentOS8-nfs3-client kernel: R10: ffff8a8a12ef2800 R11: 0000000000000001 R12: ffff8a816e31d310 Aug 21 08:31:04 CentOS8-nfs3-client kernel: R13: 0000000000000000 R14: ffffffffc0c52b20 R15: ffffffffc0c44cf0 Aug 21 08:31:04 CentOS8-nfs3-client kernel: FS: 0000000000000000(0000) GS:ffff8a8b7c280000(0000) knlGS:0000000000000000 Aug 21 08:31:04 CentOS8-nfs3-client kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 21 08:31:04 CentOS8-nfs3-client kernel: CR2: 000055af47ebff44 CR3: 0000000eb0610001 CR4: 00000000003706e0 Aug 21 08:31:04 CentOS8-nfs3-client kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 21 08:31:04 CentOS8-nfs3-client kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Aug 21 08:31:04 CentOS8-nfs3-client kernel: Call Trace: Aug 21 08:31:04 CentOS8-nfs3-client kernel: _raw_spin_lock+0x1c/0x20 Aug 21 08:31:04 CentOS8-nfs3-client kernel: xprt_prepare_transmit+0x4a/0xb0 [sunrpc] Aug 21 08:31:04 CentOS8-nfs3-client kernel: call_transmit+0x2e/0x80 [sunrpc] Aug 21 08:31:04 CentOS8-nfs3-client kernel: __rpc_execute+0x85/0x3c0 [sunrpc] Aug 21 08:31:04 CentOS8-nfs3-client kernel: ? try_to_wake_up+0x1cd/0x540 Aug 21 08:31:04 CentOS8-nfs3-client kernel: rpc_async_schedule+0x29/0x40 [sunrpc] Aug 21 08:31:04 CentOS8-nfs3-client kernel: process_one_work+0x1a7/0x360 Aug 21 08:31:04 CentOS8-nfs3-client kernel: ? create_worker+0x1a0/0x1a0 Aug 21 08:31:04 CentOS8-nfs3-client kernel: worker_thread+0x30/0x390 Aug 21 08:31:04 CentOS8-nfs3-client kernel: ? create_worker+0x1a0/0x1a0 Aug 21 08:31:04 CentOS8-nfs3-client kernel: kthread+0x116/0x130 Aug 21 08:31:04 CentOS8-nfs3-client kernel: ? kthread_flush_work_fn+0x10/0x10 Aug 21 08:31:04 CentOS8-nfs3-client kernel: ret_from_fork+0x35/0x40 ... similar for many other CPUs ... Aug 21 08:34:06 CentOS8-nfs3-client kernel: rpc_check_timeout: 123 callbacks suppressed Aug 21 08:34:06 CentOS8-nfs3-client kernel: nfs: server FreeBSD-nfsd not responding, timed out
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABObuOr5rLSoUhm8LsWr6hQZVmpBGrYvJE%2BnruV1kmB8W-9XnQ>