From owner-freebsd-current@FreeBSD.ORG Sat Jul 13 08:14:24 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F3C7B2BD; Sat, 13 Jul 2013 08:14:23 +0000 (UTC) (envelope-from ianf@clue.co.za) Received: from zcs04.jnb1.cloudseed.co.za (zcs04.jnb1.cloudseed.co.za [41.154.0.161]) by mx1.freebsd.org (Postfix) with ESMTP id 6582016DE; Sat, 13 Jul 2013 08:14:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zcs04.jnb1.cloudseed.co.za (Postfix) with ESMTP id A8D572A832E8; Sat, 13 Jul 2013 10:14:17 +0200 (SAST) X-Virus-Scanned: amavisd-new at zcs04.jnb1.cloudseed.co.za Received: from zcs04.jnb1.cloudseed.co.za ([127.0.0.1]) by localhost (zcs04.jnb1.cloudseed.co.za [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9l8HkUnZ+I04; Sat, 13 Jul 2013 10:14:15 +0200 (SAST) Received: from clue.co.za (unknown [41.154.88.19]) by zcs04.jnb1.cloudseed.co.za (Postfix) with ESMTPSA id 349802A832A5; Sat, 13 Jul 2013 10:14:15 +0200 (SAST) Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1Uxuy4-0003KB-OZ; Sat, 13 Jul 2013 10:14:12 +0200 X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.5 To: Konstantin Belousov From: Ian FREISLICH Subject: Re: Filesystem wedges caused by r251446 In-Reply-To: <20130713054220.GJ91021@kib.kiev.ua> References: <20130713054220.GJ91021@kib.kiev.ua> <20130712201051.GI91021@kib.kiev.ua> <201307110923.06548.jhb@freebsd.org> <201307091202.24493.jhb@freebsd.org> X-Attribution: BOFH Mime-Version: 1.0 Content-Type: multipart/mixed ; boundary="==_Exmh_1373703160_23330" Date: Sat, 13 Jul 2013 10:14:06 +0200 Message-Id: Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jul 2013 08:14:24 -0000 This is a multipart MIME message. --==_Exmh_1373703160_23330 Content-Type: text/plain; charset=us-ascii Konstantin Belousov wrote: > On Fri, Jul 12, 2013 at 11:34:18PM +0200, Ian FREISLICH wrote: > > (kgdb) print runningbufreq > > $1 = 1 > > (kgdb) print runningbufspace > > $2 = 0 > > (kgdb) print lorunningspace > > $3 = 4587520 > > (kgdb) print hirunningspace > > $4 = 4194304 > > This is extremely weird. The hirunningspace is less then lorunningspace, > am I right ? This causes the runningbufspace machinery to never wake up Yes. This state of affairs doesn't happen on r251445 and further testing on my side shows it doesn't hapen on all my amd64 servers. It appears that this particular server type (Dell R200) running amd64 with geom_mirror is affected. I will have to test further by destroying the mirror and removing it from the kernel and see if I can still reproduce the issue. Perhaps r251446 exposes insufficient locking on opperations affecting these variables. FWIW, I cannot reproduce the problem if the mirror is rebuilding. > I just verified on the 4G VM on amd64, my numbers for lo is 4587520, > for high 6881280. Verify your tuning and kernel options, which you should > have provided with the original report, I think. Sorry about that (and I'm relieved:) I had originally compiled with CPUTYPE?=opteron which is incorrect for this CPU. However the problem persists with CPUTYPE?=core2, but I'm not sure how much of a difference this makes with clang. Also, I have another affected host that's compiled with gcc and the correct CPUTYPE so I doubt it's the compiler. I've attached make.conf, kernelconfig and dmesg.boot. You'll notice it's r251446M - which is a result of your patch. Ian -- Ian Freislich --==_Exmh_1373703160_23330 Content-Type: text/plain ; name="FIREWALL"; charset=us-ascii Content-Description: FIREWALL Content-Disposition: attachment; filename="FIREWALL" cpu HAMMER ident "FIREWALL" options SCHED_ULE options INET #InterNETworking options FFS #Berkeley Fast Filesystem options UFS_ACL #Support for access control lists options UFS_DIRHASH #Improve performance on big directories options SOFTUPDATES #Enable FFS soft updates support options PSEUDOFS #Pseudo-filesystem framework options PROCFS options GEOM_PART_GPT options GEOM_LABEL options GEOM_MIRROR options GEOM_GATE # Userland services. options COMPAT_43 options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] options COMPAT_FREEBSD32 options COMPAT_FREEBSD4 #Compatible with FreeBSD4 options COMPAT_FREEBSD5 #Compatible with FreeBSD4 options COMPAT_FREEBSD6 #Compatible with FreeBSD4 options COMPAT_FREEBSD7 #Compatible with FreeBSD4 options COMPAT_LINUX32 options LINPROCFS options LINSYSFS options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options CONSPEED=115200 options PRINTF_BUFR_SIZE=128 device pf device pflog device pfsync options ALTQ options ALTQ_CBQ options ALTQ_RED options ALTQ_RIO options ALTQ_HFSC options ALTQ_CDNR options ALTQ_PRIQ # Debugging for use in -current options KDB # Enable kernel debugger support. options DDB # Support DDB. options GDB # Support remote GDB. options KDB_TRACE options KDB_UNATTENDED options ALT_BREAK_TO_DEBUGGER options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC makeoptions DEBUG=-g # To make an SMP kernel, the next two are needed options SMP # Symmetric MultiProcessor Kernel device cpufreq device acpi device pci device smb device smbus device ichsmb # ATA controllers device mfi device scbus # SCSI bus (required for ATA/SCSI) device ahci # AHCI-compatible SATA controllers device ata device ada # Direct Access (disks) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct ATA/SCSI access) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device kbdmux device psm # PS/2 mouse device vga # VGA video card driver device sc device agp # support several AGP chipsets # Serial (COM) ports device uart device smb device smbus device ichsmb device miibus device bce device bge device em device igb device vlan option VLAN_ARRAY device carp # Pseudo devices - the number indicates how many units to allocate. device random # Entropy device device loop # Network loopback device ether # Ethernet support device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! device bpf # Berkeley packet filter device usb device uhci device ehci device ohci device ums device ukbd device ucom device ulpt device uplcom device umass device uhid --==_Exmh_1373703160_23330 Content-Type: text/plain ; name="dmesg.boot"; charset=us-ascii Content-Description: dmesg.boot Content-Disposition: attachment; filename="dmesg.boot" Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #2 r251446M: Sat Jul 13 09:14:34 SAST 2013 ianf@fw1.smmt.gp-online.net:/usr/obj/usr/src/sys/FIREWALL amd64 FreeBSD clang version 3.3 (trunk 178860) 20130405 WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz (2666.82-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Family = 0x6 Model = 0x17 Stepping = 6 Features=0xbfebfbff Features2=0x8e39d AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 3966435328 (3782 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 atrtc0: port 0x70-0x7f irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x5f irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 28.0 on pci0 pci2: on pcib2 pcib3: irq 16 at device 28.4 on pci0 pci3: on pcib3 bge0: mem 0xdfdf0000-0xdfdfffff irq 16 at device 0.0 on pci3 bge0: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:25:64:3c:33:8d pcib4: irq 17 at device 28.5 on pci0 pci4: on pcib4 bge1: mem 0xdfef0000-0xdfefffff irq 17 at device 0.0 on pci4 bge1: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge1: Ethernet address: 00:25:64:3c:33:8e uhci0: port 0xdc60-0xdc7f irq 21 at device 29.0 on pci0 usbus0 on uhci0 uhci1: port 0xdc80-0xdc9f irq 20 at device 29.1 on pci0 usbus1 on uhci1 uhci2: port 0xdca0-0xdcbf irq 21 at device 29.2 on pci0 usbus2 on uhci2 ehci0: mem 0xdfcffc00-0xdfcfffff irq 21 at device 29.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 pcib5: at device 30.0 on pci0 pci5: on pcib5 vgapci0: port 0xec00-0xecff mem 0xd0000000-0xd7ffffff,0xdfff0000-0xdfffffff irq 19 at device 5.0 on pci5 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0xdc30-0xdc37,0xdc28-0xdc2b,0xdc38-0xdc3f,0xdc2c-0xdc2f,0xdc40-0xdc4f,0xdc50-0xdc5f irq 23 at device 31.2 on pci0 ata2: at channel 0 on atapci0 ata3: at channel 1 on atapci0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x90 on acpi0 uart0: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcb7ff,0xec000-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ada0 at ata2 bus 0 scbus0 target 0 lun 0 ada0: ATA-7 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad0 ada1 at ata3 bus 0 scbus1 target 0 lun 0 ada1: ATA-7 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad1 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 1333409705 Hz quality 1000 WARNING: DIAGNOSTIC option enabled, expect reduced performance. cd0 at ata2 bus 0 scbus0 target 1 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered GEOM: ada0: the secondary GPT header is not in the last LBA. GEOM: ada1: the secondary GPT header is not in the last LBA. GEOM_MIRROR: Device mirror/d0 launched (2/2). Root mount waiting for: usbus3 Root mount waiting for: usbus3 uhub3: 6 ports with 6 removable, self powered Root mount waiting for: usbus3 ugen3.2: at usbus3 uhub4: on usbus3 uhub4: MTT enabled Root mount waiting for: usbus3 uhub4: 4 ports with 4 removable, self powered Trying to mount root from ufs:/dev/mirror/d0p2 [rw]... bge0: link state changed to UP carp: VHID 3@vlan3: INIT -> BACKUP carp: demoted by -240 to 2400 (interface up) vlan3: link state changed to UP carp: VHID 4@vlan4: INIT -> BACKUP carp: demoted by -240 to 2160 (interface up) vlan4: link state changed to UP carp: VHID 5@vlan5: INIT -> BACKUP carp: demoted by -240 to 1920 (interface up) vlan5: link state changed to UP carp: VHID 6@vlan6: INIT -> BACKUP carp: demoted by -240 to 1680 (interface up) vlan6: link state changed to UP carp: VHID 7@vlan7: INIT -> BACKUP carp: demoted by -240 to 1440 (interface up) vlan7: link state changed to UP carp: VHID 8@vlan8: INIT -> BACKUP carp: demoted by -240 to 1200 (interface up) vlan8: link state changed to UP carp: VHID 9@vlan9: INIT -> BACKUP carp: demoted by -240 to 960 (interface up) vlan9: link state changed to UP carp: VHID 12@vlan12: INIT -> BACKUP carp: demoted by -240 to 720 (interface up) vlan12: link state changed to UP carp: VHID 13@vlan13: INIT -> BACKUP carp: demoted by -240 to 480 (interface up) vlan13: link state changed to UP carp: VHID 40@vlan40: INIT -> BACKUP carp: demoted by -240 to 240 (interface up) vlan40: link state changed to UP bge1: link state changed to UP --==_Exmh_1373703160_23330 Content-Type: text/plain ; name="make.conf"; charset=us-ascii Content-Description: make.conf Content-Disposition: attachment; filename="make.conf" MAKE_IDEA= YES # IDEA (128 bit symmetric encryption) PRINTERDEVICE= ps USA_RESIDENT=NO CPUTYPE?=core2 HAVE_MOTIF= yes MAKE_KERBEROS5= yes ENABLE_SUID_K5SU= yes KERNCONF=FIREWALL BOOT_COMCONSOLE_SPEED=115200 #WITHOUT_CLANG_IS_CC=yes WITH_PKGNG= yes # added by use.perl 2013-06-13 18:12:04 PERL_VERSION=5.14.4 --==_Exmh_1373703160_23330--