From nobody Thu Feb 29 09:24:14 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlm4t66LZz5CXbf for ; Thu, 29 Feb 2024 09:24:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlm4t2Y60z4sLD for ; Thu, 29 Feb 2024 09:24:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709198654; a=rsa-sha256; cv=none; b=cSSEr7o94GhTRY1g9H9YWk/Zk95xdbS0iTwbedWmQMFHnNeItKbwW1rwurCLhbJgX3mqyw Nssf8bhUg9hlya1ovM2GNLJ/VHPIWG8G++BPfN7jQ+aZpFq+ZQWUpOXsE853C4KNfiFNhW dlViHUl7GZhm6gwPO5dCsYkBwWzkCR/Irwgmm0Rr2qI7kLNjingB1yUO1SfK4Dcig7awcU 3E5vDNiK14QGbGAXr+FDnAQlq47wMK4vmI8ceMnVMWuk1PTbrSJXCEHMNo0YH0g13BootD UT1HiKvUjaLT2CJRuq175hznr9OLXsUz7lcddD+mTUTKbdpiSHqOROmQ+hEWZQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709198654; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=G6Z+zvR73+g2Al8nQGblIB7CG4S9L7tn5l8ZJxIdYT0=; b=N+gONYal0O6jwN/lHlWhadVOqhrG7bIM6Q1Cj2K2+F3p2p7qL3vbTrwYOwu0pZJUnkGRxk 3FRT4i8cV66ex1xfRIMHk/Pn5V37PXUf3wQsPR4fyesKDmjQZOda6Fu3Ec/aG4aNCWs+O5 pBqacjV1SbKCDAm5jt8E1Bx2k5IesscEHm7SEgf2RtdHXU5uv78cZpxi0CD/E8KJFcIVCc NDS8EqcP4xU9BgH5eFZPh0w2jGG0fEVzMfyl5ZmdaB7vA7XBNxcVFbcJHu76P1K4qPFjpU ZpFOkg3QkXKmifp8GrrcWfwHxLXruiVBvt6sTJSR4cCAx/pxedIWcO+PCRf4ig== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tlm4t28Tqzc0g for ; Thu, 29 Feb 2024 09:24:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41T9OEde070100 for ; Thu, 29 Feb 2024 09:24:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41T9OEHF070099 for bugs@FreeBSD.org; Thu, 29 Feb 2024 09:24:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 277389] Reproduceable low memory freeze on 14.0-RELEASE-p5 Date: Thu, 29 Feb 2024 09:24:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pascal.guitierrez@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277389 Bug ID: 277389 Summary: Reproduceable low memory freeze on 14.0-RELEASE-p5 Product: Base System Version: 14.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: pascal.guitierrez@gmail.com I can reliably reproduce a freeze using iozone on 14.0-RELEASE-p5 using a simple ZFS pool backed onto SATA HDDs as such: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP=20= =20=20 HEALTH ALTROOT tank 1.81T 512G 1.31T - - 36% 27% 1.00x=20= =20=20 ONLINE - mirror-0 1.81T 512G 1.31T - - 36% 27.6% -=20= =20=20 ONLINE ada0 1.82T - - - - - - -=20= =20=20 ONLINE ada1 1.82T - - - - - - -=20= =20=20 ONLINE To reproduce I create a ZFS filesystem with compression=3Doff, then run ioz= one -i 0,1 -l 512 -r 4k -s 1g The write tests execute fine (writing out 512 1g files), however when it ru= ns the read tests (1=3Dread/re-read) the system freezes due to an apparent low memory condition after only a few minutes. I've tried increasing the following sysctls from their defaults: vm.v_free_target vm.v_free_min vm.v_free_severe vm.v_free_reserved vfs.zfs.arc.sys_free vfs.zfs.arc.free_target and reducing the following: vfs.zfs.arc_max But no effect, free memory reliably drops below 100M and does not recover a= nd soon after the system will freeze and require a hard reset. This machine has 16GB of RAM with no swap (have also tried with swap on zvol but same effect), dmesg is below: ---<>--- Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-RELEASE-p5 #0: Tue Feb 13 23:37:36 UTC 2024 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GEN= ERIC amd64 FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) VT(efifb): resolution 800x600 CPU: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz (2200.21-MHz K8-class CPU) Origin=3D"GenuineIntel" Id=3D0x506f1 Family=3D0x6 Model=3D0x5f Steppi= ng=3D1 =20 Features=3D0xbfebfbff =20 Features2=3D0x4ff8ebbf AMD Features=3D0x2c100800 AMD Features2=3D0x101 Structured Extended Features=3D0x2294e283 Structured Extended Features3=3D0xac000400 XSAVE Features=3D0xf IA32_ARCH_CAPS=3D0x14000c69 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory =3D 17179869184 (16384 MB) avail memory =3D 16552595456 (15785 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1) FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 Launching APs: 2 3 5 1 4 6 7 random: entropy device external interface kbd0 at kbdmux0 efirtc0: efirtc0: registered as a time-of-day clock, resolution 1.000000s smbios0: at iomem 0x7f0c4000-0x7f0c401e smbios0: Version: 3.0, BCD Revision: 3.0 aesni0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 24000000 Hz quality 950 Event timer "HPET" frequency 24000000 Hz quality 550 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 6.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pcib2: mem 0xdf440000-0xdf45ffff at device 9.0 on pci0 pci2: on pcib2 pcib3: mem 0xdf420000-0xdf43ffff at device 16.0 on pc= i0 pci3: on pcib3 pcib4: mem 0xdf400000-0xdf41ffff at device 17.0 on pc= i0 pci4: on pcib4 pcib5: at device 0.0 on pci4 pci5: on pcib5 vgapci0: port 0xd000-0xd07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff at device 0.0 on pci5 ahci0: port 0xe090-0xe097,0xe080-0xe083,0xe040-0xe05f mem 0xdf476000-0xdf477fff,0xdf47d000-0xdf47d0ff,0xdf47c000-0xdf47c7ff at device 19.0 on pci0 ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahcich6: at channel 6 on ahci0 ahcich7: at channel 7 on ahci0 ahciem0: on ahci0 ahci1: port 0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf474000-0xdf475fff,0xdf47b000-0xdf47b0ff,0xdf47a000-0xdf47a7ff at device 20.0 on pci0 ahci1: AHCI v1.31 with 4 6Gbps ports, Port Multiplier not supported ahcich11: at channel 3 on ahci1 ahcich12: at channel 4 on ahci1 ahcich13: at channel 5 on ahci1 ahcich14: at channel 6 on ahci1 ahciem1: on ahci1 xhci0: mem 0xdf460000-0xdf46ffff at de= vice 21.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pcib6: at device 22.0 on pci0 pci6: on pcib6 ix0: mem 0xddc00000-0xdddfffff,0xdde04000-0xdde07fff= at device 0.0 on pci6 ix0: Using 2048 TX descriptors and 2048 RX descriptors ix0: Using 8 RX queues 8 TX queues ix0: Using MSI-X interrupts with 9 vectors ix0: allocated for 8 queues ix0: allocated for 8 rx queues ix0: Ethernet address: 3c:ec:ef:e9:aa:30 ix0: eTrack 0x80000877 ix0: netmap queues/slots: TX 8/2048, RX 8/2048 ix1: mem 0xdda00000-0xddbfffff,0xdde00000-0xdde03fff= at device 0.1 on pci6 ix1: Using 2048 TX descriptors and 2048 RX descriptors ix1: Using 8 RX queues 8 TX queues ix1: Using MSI-X interrupts with 9 vectors ix1: allocated for 8 queues ix1: allocated for 8 rx queues ix1: Ethernet address: 3c:ec:ef:e9:aa:31 ix1: eTrack 0x80000877 ix1: netmap queues/slots: TX 8/2048, RX 8/2048 pcib7: at device 23.0 on pci0 pci7: on pcib7 ix2: mem 0xdd600000-0xdd7fffff,0xdd804000-0xdd807fff= at device 0.0 on pci7 ix2: Using 2048 TX descriptors and 2048 RX descriptors ix2: Using 8 RX queues 8 TX queues ix2: Using MSI-X interrupts with 9 vectors ix2: allocated for 8 queues ix2: allocated for 8 rx queues ix2: Ethernet address: 3c:ec:ef:e9:aa:32 ix2: eTrack 0x8000087c ix2: netmap queues/slots: TX 8/2048, RX 8/2048 ix3: mem 0xdd400000-0xdd5fffff,0xdd800000-0xdd803fff= at device 0.1 on pci7 ix3: Using 2048 TX descriptors and 2048 RX descriptors ix3: Using 8 RX queues 8 TX queues ix3: Using MSI-X interrupts with 9 vectors ix3: allocated for 8 queues ix3: allocated for 8 rx queues ix3: Ethernet address: 3c:ec:ef:e9:aa:33 ix3: eTrack 0x8000087c ix3: netmap queues/slots: TX 8/2048, RX 8/2048 pci0: at device 24.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 pci0: at device 31.2 (no driver attached) pci0: at device 31.5 (no driver attached) apei0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ns8250: UART FCR is broken uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 est0: on cpu0 Timecounter "TSC-low" frequency 1099999978 Hz quality 1000 Timecounters tick every 1.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Trying to mount root from zfs:zroot/ROOT/default []... ugen0.1: at usbus0 uhub0 on usbus0 uhub0: on usbus0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ACS-4 ATA SATA 3.x device ada0: Serial Number WS10R331 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ACS-4 ATA SATA 3.x device ada1: Serial Number WS10R3FB ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors) ses0 at ahciem0 bus 0 scbus8 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus13 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0 ses0: ada1,pass1 in 'Slot 01', SATA Slot: scbus1 target 0 uhub0: 8 ports with 8 removable, self powered Root mount waiting for: usbus0 usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Gen= eric USB Storage (0x05e3:0x0736) Root mount waiting for: usbus0 usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage devi= ce Generic USB Storage (0x05e3:0x0736) usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device Generic USB Storage (0x05e3:0x0736) usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device Generic USB Storage (0x05e3:0x0736) ugen0.2: at usbus0 umass0 on uhub0 umass0: on usbus0 umass0: SCSI over Bulk-Only; quirks =3D 0xc101 umass0:14:0: Attached to scbus14 da0 at umass-sim0 bus 0 scbus14 target 0 lun 0 da0: Removable Direct Access SCSI device da0: Serial Number 000000000272 da0: 40.000MB/s transfers da0: 3968MB (8126464 512 byte sectors) da0: quirks=3D0x3 usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Gen= eric USB Storage (0x05e3:0x0736) Root mount waiting for: usbus0 usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage devi= ce Generic USB Storage (0x05e3:0x0736) usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device Generic USB Storage (0x05e3:0x0736) usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device Generic USB Storage (0x05e3:0x0736) ugen0.3: at usbus0 umass1 on uhub0 umass1: on usbus0 umass1: SCSI over Bulk-Only; quirks =3D 0xc101 umass1:15:1: Attached to scbus15 da1 at umass-sim1 bus 1 scbus15 target 0 lun 0 da1: Removable Direct Access SCSI device da1: Serial Number 000000000272 da1: 40.000MB/s transfers da1: 3968MB (8126464 512 byte sectors) da1: quirks=3D0x3 ugen0.4: at usbus0 uhub1 on uhub0 uhub1: on usb= us0 uhub1: MTT enabled Root mount waiting for: usbus0 uhub1: 4 ports with 4 removable, self powered Root mount waiting for: usbus0 ugen0.5: at usbus0 uhub2 on uhub1 uhub2: on usbus0 uhub2: 4 ports with 3 removable, self powered ugen0.6: at usbus0 ukbd0 on uhub2 ukbd0: on usbus0 kbd1 at ukbd0 Root mount waiting for: usbus0 ugen0.7: at usbus0 uhub3 on uhub1 uhub3: on usb= us0 uhub3: MTT enabled uhub3: 4 ports with 4 removable, self powered Root mount waiting for: usbus0 ugen0.8: at usbus0 ukbd1 on uhub3 ukbd1: = on usbus0 kbd2 at ukbd1 Dual Console: Video Primary, Serial Secondary ichsmb0: port 0xe000-0xe01f mem 0xdf478000-0xdf4780ff at device 31.4 on pci0 smbus0: on ichsmb0 lo0: link state changed to UP ums0 on uhub2 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=3D0 ix0: link state changed to UP --=20 You are receiving this mail because: You are the assignee for the bug.=