Date: Mon, 30 Mar 2015 18:46:15 +0300 From: Artem Kuchin <artem@artem.ru> To: freebsd-fs@freebsd.org Subject: Re: Little research how rm -rf and tar kill server Message-ID: <55196FC7.8090107@artem.ru> In-Reply-To: <1427727936.293597.247070269.5CE0D411@webmail.messagingengine.com> References: <55170D9C.1070107@artem.ru> <1427727936.293597.247070269.5CE0D411@webmail.messagingengine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
30.03.2015 18:05, Mark Felder пишет: > > > On Sat, Mar 28, 2015, at 15:22, Artem Kuchin wrote: >> >> So, questions and thoughts: 1) Why i had no problem such this in >> fbsd 9? I think the reason for the problem is in kernel, not in >> hardware or mariadb+nginx because server load did not increase at >> all, even decreases a little. > > This is only anecdotal until you run the exact same OS version and > load on the old hardware which had a different motherboard and > (probably) disk controller. Can you provide us with any further > hardware details between the two systems? My point was that the change what causes it was in freebsd kernel or ufs driver or sata driver or somewhere there, not the hardware.I even left the Kernel config file the same. I cannot provide exact hardware and dmesg for the old server (it is down and not available), but it was Intel Xeon E3-1245 /Quad Core/ 16 GB DDR3 ECC RAM 2x 3 TB SATA III Enterprise HDD (seagate ST33000650NS) New server kernel: Copyright (c) 1992-2015 The FreeBSD Project. kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 kernel: The Regents of the University of California. All rights reserved. kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. kernel: FreeBSD 10.1-STABLE #0 r279278: Wed Feb 25 15:17:48 MSK 2015 kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 kernel: CPU: Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) kernel: Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3 kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> kernel: Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,<b11>,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AE kernel: AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> kernel: AMD Features2=0x21<LAHF,ABM> kernel: Structured Extended Features=0x2fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM> kernel: XSAVE Features=0x1<XSAVEOPT> kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID kernel: TSC: P-state invariant, performance statistics kernel: real memory = 34359738368 (32768 MB) kernel: avail memory = 33271001088 (31729 MB) kernel: Event timer "LAPIC" quality 600 kernel: ACPI APIC Table: <SUPERM SMCI--MB> kernel: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads kernel: cpu0 (BSP): APIC ID: 0 kernel: cpu1 (AP): APIC ID: 1 kernel: cpu2 (AP): APIC ID: 2 kernel: cpu3 (AP): APIC ID: 3 kernel: cpu4 (AP): APIC ID: 4 kernel: cpu5 (AP): APIC ID: 5 kernel: cpu6 (AP): APIC ID: 6 kernel: cpu7 (AP): APIC ID: 7 kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard kernel: random: <Software, Yarrow> initialized kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff80b6a090, 0) error 19 kernel: kbd1 at kbdmux0 kernel: module_register_init: MOD_LOAD (accf_data, 0xffffffff807e9af0, 0xffffffff8155d0c8) error 17 kernel: acpi0: <SUPERM SMCI--MB> on motherboard kernel: acpi0: Power Button (fixed) kernel: cpu0: <ACPI CPU> on acpi0 kernel: cpu1: <ACPI CPU> on acpi0 kernel: cpu2: <ACPI CPU> on acpi0 kernel: cpu3: <ACPI CPU> on acpi0 kernel: cpu4: <ACPI CPU> on acpi0 kernel: cpu5: <ACPI CPU> on acpi0 kernel: cpu6: <ACPI CPU> on acpi0 kernel: cpu7: <ACPI CPU> on acpi0 kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 kernel: Timecounter "HPET" frequency 14318180 Hz quality 950 kernel: Event timer "HPET" frequency 14318180 Hz quality 550 kernel: atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 kernel: Event timer "RTC" frequency 32768 Hz quality 0 kernel: attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 kernel: Event timer "i8254" frequency 1193182 Hz quality 100 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 kernel: pci0: <ACPI PCI bus> on pcib0 kernel: xhci0: <Intel Lynx Point USB 3.0 controller> mem 0xf7220000-0xf722ffff irq 16 at device 20.0 on pci0 kernel: xhci0: 32 bytes context size, 64-bit DMA kernel: xhci0: Port routing mask set to 0xffffffff kernel: usbus0 on xhci0 kernel: em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xf020-0xf03f mem 0xf7200000-0xf721ffff,0xf7235000-0xf7235fff irq 20 at device 25.0 on pci0 kernel: em0: Using an MSI interrupt kernel: em0: Ethernet address: 00:25:90:47:42:57 kernel: ehci0: <Intel Lynx Point USB 2.0 controller USB-B> mem 0xf7234000-0xf72343ff irq 16 at device 26.0 on pci0 kernel: usbus1: EHCI version 1.0 kernel: usbus1 on ehci0 kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 kernel: pci1: <ACPI PCI bus> on pcib1 kernel: pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 kernel: pci2: <ACPI PCI bus> on pcib2 kernel: vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem 0xf6000000-0xf6ffffff,0xf7000000-0xf701ffff irq 16 at device 0.0 on pci2 kernel: vgapci0: Boot video device kernel: pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0 kernel: pci3: <ACPI PCI bus> on pcib3 kernel: igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0xd000-0xd01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 17 at device 0.0 on pci3 kernel: igb0: Using MSIX interrupts with 5 vectors kernel: igb0: Ethernet address: 00:25:90:47:42:56 kernel: igb0: Bound queue 0 to cpu 0 kernel: igb0: Bound queue 1 to cpu 1 kernel: igb0: Bound queue 2 to cpu 2 kernel: igb0: Bound queue 3 to cpu 3 kernel: ehci1: <Intel Lynx Point USB 2.0 controller USB-A> mem 0xf7233000-0xf72333ff irq 23 at device 29.0 on pci0 kernel: usbus2: EHCI version 1.0 kernel: usbus2 on ehci1 kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0 kernel: isa0: <ISA bus> on isab0 kernel: ahci0: <Intel Lynx Point AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xf7232000-0xf72327ff irq 19 at device 31.2 on pci0 kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported kernel: ahcich0: <AHCI channel> at channel 0 on ahci0 kernel: ahcich1: <AHCI channel> at channel 1 on ahci0 kernel: ahcich2: <AHCI channel> at channel 2 on ahci0 kernel: ahcich3: <AHCI channel> at channel 3 on ahci0 kernel: ahcich4: <AHCI channel> at channel 4 on ahci0 kernel: ahcich5: <AHCI channel> at channel 5 on ahci0 kernel: ahciem0: <AHCI enclosure management bridge> on ahci0 kernel: acpi_button0: <Sleep Button> on acpi0 kernel: acpi_button1: <Power Button> on acpi0 kernel: acpi_tz0: <Thermal Zone> on acpi0 kernel: acpi_tz1: <Thermal Zone> on acpi0 kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 kernel: uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 kernel: sc0: <System console> at flags 0x100 on isa0 kernel: sc0: CGA <16 virtual consoles, flags=0x300> kernel: vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0 kernel: ppc0: cannot reserve I/O port range kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0 kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1 kernel: est2: <Enhanced SpeedStep Frequency Control> on cpu2 kernel: est3: <Enhanced SpeedStep Frequency Control> on cpu3 kernel: est4: <Enhanced SpeedStep Frequency Control> on cpu4 kernel: est5: <Enhanced SpeedStep Frequency Control> on cpu5 kernel: est6: <Enhanced SpeedStep Frequency Control> on cpu6 kernel: est7: <Enhanced SpeedStep Frequency Control> on cpu7 kernel: random: unblocking device. kernel: usbus0: 5.0Gbps Super Speed USB v3.0 kernel: Timecounters tick every 1.000 msec kernel: ipfw2 (+ipv6) initialized, divert enabled, nat loadable, default to accept, logging disabled kernel: DUMMYNET 0 with IPv6 initialized (100409) kernel: load_dn_sched dn_sched FIFO loaded kernel: load_dn_sched dn_sched PRIO loaded kernel: load_dn_sched dn_sched QFQ loaded kernel: load_dn_sched dn_sched RR loaded kernel: load_dn_sched dn_sched WF2Q+ loaded kernel: usbus1: 480Mbps High Speed USB v2.0 kernel: usbus2: 480Mbps High Speed USB v2.0 kernel: ugen1.1: <Intel> at usbus1 kernel: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 kernel: ugen0.1: <0x8086> at usbus0 kernel: ugen2.1: <Intel> at usbus2 kernel: uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 kernel: uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 kernel: ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 kernel: ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device kernel: ses0: SEMB SES Device kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 kernel: ada0: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device kernel: ada0: Serial Number 64O4281GS kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) kernel: ada0: Command Queueing enabled kernel: ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) kernel: ada0: Previously was known as ad4 kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 kernel: ada1: <TOSHIBA DT01ACA300 MX6OABB0> ATA-8 SATA 3.x device kernel: ada1: Serial Number 64O43WZGS kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) kernel: ada1: Command Queueing enabled kernel: ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) kernel: ada1: Previously was known as ad6 kernel: SMP: AP CPU #1 Launched! kernel: SMP: AP CPU #4 Launched! kernel: SMP: AP CPU #2 Launched! kernel: SMP: AP CPU #3 Launched! kernel: SMP: AP CPU #6 Launched! kernel: SMP: AP CPU #7 Launched! kernel: SMP: AP CPU #5 Launched! kernel: Timecounter "TSC-low" frequency 1750036200 Hz quality 1000 kernel: GEOM_MIRROR: Device mirror/boot launched (2/2). kernel: GEOM_MIRROR: Device mirror/swap launched (2/2). kernel: GEOM_MIRROR: Device mirror/root launched (2/2). kernel: Root mount waiting for: usbus2 usbus1 usbus0 kernel: uhub2: 17 ports with 17 removable, self powered kernel: uhub0: 2 ports with 2 removable, self powered kernel: uhub1: 2 ports with 2 removable, self powered kernel: Root mount waiting for: usbus2 usbus1 usbus0 kernel: ugen0.2: <vendor 0x0000> at usbus0 kernel: uhub3: <vendor 0x0000 product 0x0001, class 9/0, rev 2.00/0.00, addr 1> on usbus0 kernel: uhub3: 4 ports with 3 removable, self powered kernel: ugen1.2: <vendor 0x8087> at usbus1 kernel: uhub4: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1 kernel: ugen2.2: <vendor 0x8087> at usbus2 kernel: uhub5: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2 kernel: uhub4: 4 ports with 4 removable, self powered kernel: uhub5: 6 ports with 6 removable, self powered kernel: ugen0.3: <vendor 0x0557> at usbus0 kernel: ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 kernel: kbd0 at ukbd0 kernel: ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0 kernel: ums0: 3 buttons and [Z] coordinates ID=0 kernel: Trying to mount root from ufs:/dev/mirror/root [rw,async,groupquota,noatime]... > > >> 2) I consider it a sever bug, because even normal used (and i have >> plenty of them using ssh) can eventually do rm -rf and kill all >> sites. Which means there are must be some way to limit io usage per >> user >> > > ZFS has far superior disk IO scheduling. You may have more success > with that. Unfortunately I'm not aware of a way to limit user/process > disk IO on FreeBSD. I hesitatated migrating to ZFS because "don't fix it if it ain't broken". And it was not broken.. until now. But now i cannot migrate to ZFS - server is too busy and actually i am afraid something bad will come up with ZFS eventually too. Well, if not limit, then maybe just monitor and kill user processes that create problems. But the problem is how to determine which processes overuse the io system. I have very little clue how to collect such information. Artem
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55196FC7.8090107>