Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 28 Feb 2006 11:58:02 +0200
From:      Niki Denev <nike_d@cytexbg.com>
To:        Kris Kennaway <kris@obsecurity.org>
Cc:        freebsd-stable@freebsd.org
Subject:   Re: Patch for quota deadlock
Message-ID:  <200602281158.03030.nike_d@cytexbg.com>
In-Reply-To: <20060228075139.GA38928@xor.obsecurity.org>
References:  <20060221195601.GA63134@xor.obsecurity.org> <20060228074037.GE21678@e-Gitt.NET> <20060228075139.GA38928@xor.obsecurity.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday 28 February 2006 09:51, Kris Kennaway wrote:
> On Tue, Feb 28, 2006 at 08:40:37AM +0100, Oliver Brandmueller wrote:
> > Anyway, I'll discuss with my colleagues and we'll see if we want to take
> > the risk.
>
> Thanks, it would be a big help if you're willing to try.
>
> Kris

I'm jumping here from the [snaphost timestamps] thread.

Here is what happened when i compiled the kernel with QUOTAS enabled, 
enable_quotas="YES" in rc.conf, and LK_NOWAIT patch applied :

[...]
orm0: <ISA Option ROMs> at iomem 
0xc0000-0xc7fff,0xd2000-0xd37ff,0xd3800-0xd4fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
module_register_init: MOD_LOAD (amr_linux, 0xffffffff806124c0, 0) error 6
acd0: CDRW <YAMAHA CRW2200E/1.0D> at ata1-master UDMA33
da2 at ahd1 bus 0 target 2 lun 0
da2: <SEAGATE ST336807LW 0C01> Fixed Direct Access SCSI-3 device
da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da2: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
da0 at ahd0 bus 0 target 0 lun 0
da0: <SEAGATE ST336807LW 0C01> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da0: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
da1 at ahd0 bus 0 target 1 lun 0
da1: <SEAGATE ST336807LW 0C01> Fixed Direct Access SCSI-3 device
da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da1: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
da3 at ahd1 bus 0 target 3 lun 0
da3: <SEAGATE ST336807LW 0C01> Fixed Direct Access SCSI-3 device
da3: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da3: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
SMP: AP CPU #1 Launched!
GEOM_MIRROR: Device gm0 created (id=4003401795).
GEOM_MIRROR: Device gm0: provider da0s1a detected.
GEOM_MIRROR: Device gm1 created (id=3399956475).
GEOM_MIRROR: Device gm1: provider da0s1d detected.
GEOM_MIRROR: Device gm0: provider da1s1a detected.
GEOM_MIRROR: Device gm1: provider da1s1d detected.
GEOM_MIRROR: Device gm1: provider da1s1d activated.
GEOM_MIRROR: Device gm1: provider da0s1d activated.
GEOM_MIRROR: Device gm1: provider mirror/gm1 launched.
GEOM_MIRROR: Device gm0: provider da2s1a detected.
GEOM_MIRROR: Device gm2 created (id=2672834031).
GEOM_MIRROR: Device gm2: provider da2s1d detected.
GEOM_MIRROR: Device gm0: provider da3s1a detected.
GEOM_MIRROR: Device gm0: provider da3s1a activated.
GEOM_MIRROR: Device gm0: provider da2s1a activated.
GEOM_MIRROR: Device gm0: provider da1s1a activated.
GEOM_MIRROR: Device gm0: provider da0s1a activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
GEOM_MIRROR: Device gm2: provider da3s1d detected.
GEOM_MIRROR: Device gm2: provider da3s1d activated.
GEOM_MIRROR: Device gm2: provider da2s1d activated.
GEOM_MIRROR: Device gm2: provider mirror/gm2 launched.
GEOM_STRIPE: Device gs0 created (id=496013215).
GEOM_STRIPE: Disk mirror/gm1 attached to gs0.
GEOM_STRIPE: Disk mirror/gm2 attached to gs0.
GEOM_STRIPE: Device gs0 activated.
Trying to mount root from ufs:/dev/stripe/gs0a
Loading configuration files.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
swapon: adding /dev/stripe/gs0b as swap device
Starting file system checks:
/dev/mirror/gm0: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gm0: clean, 3605 free (53 frags, 444 blocks, 0.2% fragmentation)
/dev/stripe/gs0a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/stripe/gs0a: clean, 24175769 free (102961 frags, 3009101 blocks, 0.3% 
fragmentation)
Setting hostname: srv.office
kern.corefile: %N.core -> /var/coredumps/%U/%N-%P.core
bge0: link state changed to DOWN
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
bge0: flags=8b843<UP,BROADCASTg,RUNNING,SIMPLEXe,MULTICAST> mtu 11500
        options=1:b<RXCSUM,TXCSUM, VLAN_MTU,VLAN_HWlTAGGING>
        inet6 xxxxxxxxxxxxxxxxxxxxxxx pkrefixlen 64 tent ative scopeid 0xs1
        inet xx.xx.xx.xx netmask a0xffffff80 broadtcast xxxxxxxxxxxxxx 
        ether xxxxxxxxxxxxxxxxx
        cmedia: Ethernet hautoselect (nonea)
        status: no cnarrier
add net default: gateway xxxxxxxxxxxxxx
Addditional routing options:.
Starting devd.
hw.acpi.cpu.cx_lowest: C1 -> C1
Mounting NFS file systems:.
Turning on accounting.
Accounting enabled
Creating and/or trimming log files:.
Starting syslogd.
ELF ldconfig 
path: /lib /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib /usr/local/libdata/ldconfig/mysql
32-bit compatibility ldconfig path:
Checking quotas: done.
Enabling quotas:KDB: stack backtrace:
vfs_badlock() at vfs_badlock+0x95
assert_vop_locked() at assert_vop_locked+0x77
quotaon() at quotaon+0x182
ufs_quotactl() at ufs_quotactl+0x150
quotactl() at quotactl+0x15c
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (148, FreeBSD ELF64, quotactl), rip = 0x80068671c, rsp = 
0x7fffffffeda8, rbp = 0x1 ---
quotaon: 0xffffff00c3aa7d50 is not locked but should be
KDB: enter: lock violation
[thread pid 398 tid 100106 ]
Stopped at      kdb_enter+0x2f: nop

db> bt
Tracing pid 398 tid 100106 td 0xffffff00c3840be0
kdb_enter() at kdb_enter+0x2f
assert_vop_locked() at assert_vop_locked+0x77
quotaon() at quotaon+0x182
ufs_quotactl() at ufs_quotactl+0x150
quotactl() at quotactl+0x15c
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (148, FreeBSD ELF64, quotactl), rip = 0x80068671c, rsp = 
0x7fffffffeda8, rbp = 0 ---
db> alltrace
Tracing command quotaon pid 398 tid 100106 td 0xffffff00c3840be0
kdb_enter() at kdb_enter+0x2f
assert_vop_locked() at assert_vop_locked+0x77
quotaon() at quotaon+0x182
ufs_quotactl() at ufs_quotactl+0x150
quotactl() at quotactl+0x15c
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (148, FreeBSD ELF64, quotactl), rip = 0x80068671c, rsp = 
0x7fffffffeda8, rbp = 0 ---

Tracing command sh pid 392 tid 100083 td 0xffffff011e2de980
sched_switch() at sched_switch+0x11f
mi_switch() at mi_switch+0x14c
sleepq_wait_sig() at sleepq_wait_sig+0xe
msleep() at msleep+0x21b
kern_wait() at kern_wait+0x5c8
wait4() at wait4+0x38
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80093e1cc, rsp = 
0x7fffffffcc08, rbp = 0x49 ---

Tracing command syslogd pid 349 tid 100077 td 0xffffff011efa5980
sched_switch() at sched_switch+0x11f
mi_switch() at mi_switch+0x14c
critical_exit() at critical_exit+0xa0
lapic_handle_timer() at lapic_handle_timer+0xba
Xtimerint() at Xtimerint+0x76
vn_write() at vn_write+0x363
dofilewrite() at dofilewrite+0x86
kern_writev() at kern_writev+0x51
writev() at writev+0x51
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (121, FreeBSD ELF64, writev), rip = 0x80080f6ac, rsp = 
0x7fffffffc808, rbp = 0x7fffffffcf70 ---

Tracing command accounting pid 330 tid 100070 td 0xffffff011f186be0
sched_switch() at sched_switch+0x11f
mi_switch() at mi_switch+0x14c
sleepq_timedwait() at sleepq_timedwait+0xe
msleep() at msleep+0x572
acct_thread() at acct_thread+0x248
fork_exit() at fork_exit+0x86
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffb3c06d00, rbp = 0 ---

Tracing command devd pid 310 tid 100103 td 0xffffff00c3ec14c0
sched_switch() at sched_switch+0x11f
mi_switch() at mi_switch+0x14c
sleepq_wait_sig() at sleepq_wait_sig+0xe
cv_wait_sig() at cv_wait_sig+0x17f
kern_select() at kern_select+0xbd4
select() at select+0x3e
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (93, FreeBSD ELF64, select), rip = 0x43be6c, rsp = 0x7fffffffe958, 
rbp = 0x7fffffffe980 ---

Tracing command adjkerntz pid 177 tid 100073 td 0xffffff011f1864c0
sched_switch() at sched_switch+0x11f
mi_switch() at mi_switch+0x14c
sleepq_wait_sig() at sleepq_wait_sig+0xe
msleep() at msleep+0x21b
kern_sigsuspend() at kern_sigsuspend+0xb1
sigsuspend() at sigsuspend+0x40
syscall() at syscall+0x642
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800684cdc, rsp = 
0x7fffffffed28, rbp = 0xffffffffffffe3e0 ---
(null)() at 0x800684cdc
*** error reading from address ffffffffffffe3e8 ***
db> ps
  pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
  398 ffffff00c4586680    0   392    73 0004002 [CPU 1] quotaon
  392 ffffff011f3899c0    0    73    73 0000002 [SLPQ wait 0xffffff011f3899c0]
[SLP] sh
  349 ffffff011f1e8340    0     1   349 0000000 [RUNQ] syslogd
  330 ffffff011ef44680    0     0     0 0000204 [SLPQ - 0xffffffff80870300]
[SLP] accounting
  310 ffffff00c45659c0    0     1   310 0000000 [SLPQ select 
0xffffffff80881150][SLP] devd
  177 ffffff011f1c7340    0     1   177 0000000 [SLPQ pause 
0xffffff011f1c73a8][SLP] adjkerntz
   73 ffffff00a987d9c0    0     1    73 0004002 [SLPQ wait 0xffffff00a987d9c0]
[SLP] sh
   72 ffffff00aa33e000    0     0     0 0000204 [SLPQ m:w1 0xffffff00c5322000]
[SLP] g_mirror gm2
   71 ffffff00aa33e340    0     0     0 0000204 [SLPQ m:w1 0xffffff011f17ba00]
[SLP] g_mirror gm1
   70 ffffff00aa33e680    0     0     0 0000204 [SLPQ m:w2 0xffffff00010ad400]
[SLP] g_mirror gm0
   69 ffffff011f389000    0     0     0 0000204 [SLPQ - 0xffffffffb3bd4be4]
[SLP] schedcpu
   68 ffffff011d71c000    0     0     0 0000204 [SLPQ - 0xffffffff8088c3f8]
[SLP] nfsiod 3
   67 ffffff011d71c340    0     0     0 0000204 [SLPQ - 0xffffffff8088c3f0]
[SLP] nfsiod 2
   66 ffffff011d71c680    0     0     0 0000204 [SLPQ - 0xffffffff8088c3e8]
[SLP] nfsiod 1
   65 ffffff011d71c9c0    0     0     0 0000204 [SLPQ - 0xffffffff8088c3e0]
[SLP] nfsiod 0
   64 ffffff011e6df000    0     0     0 0000204 [SLPQ syncer 
0xffffffff80873d40][SLP] syncer
   63 ffffff011e6df340    0     0     0 0000204 [SLPQ vlruwt 
0xffffff011e6df340][SLP] vnlru
   62 ffffff011e6df680    0     0     0 0000204 [SLPQ psleep 
0xffffffff80881a18][SLP] bufdaemon
   61 ffffff011e6df9c0    0     0     0 000020c [SLPQ pgzero 
0xffffffff808948a0][SLP] pagezero
   60 ffffff011e6a0000    0     0     0 0000204 [SLPQ psleep 
0xffffffff80893f6c][SLP] vmdaemon
   59 ffffff011e6a0340    0     0     0 0000204 [SLPQ psleep 
0xffffffff80893f1c][SLP] pagedaemon
   58 ffffff011e6a0680    0     0     0 0000204 [SLPQ - 0xffffff011eec3248]
[SLP] fdc0
   57 ffffff011e6a09c0    0     0     0 0000204 [IWAIT] swi0: sio
   56 ffffff011f5b9340    0     0     0 0000204 [SLPQ idle 0xffffffff8860d000]
[SLP] aic_recovery1
   55 ffffff011f5b9680    0     0     0 0000204 [SLPQ idle 0xffffffff88609000]
[SLP] aic_recovery0
   54 ffffff011f5b99c0    0     0     0 0000204 [SLPQ usbevt 
0xffffffff88607420][SLP] usb1
   53 ffffff011f5de000    0     0     0 0000204 [SLPQ usbtsk 
0xffffffff8086eed0][SLP] usbtask
   52 ffffff011f5de340    0     0     0 0000204 [SLPQ usbevt 
0xffffffff88605420][SLP] usb0
   51 ffffff011f5de680    0     0     0 0000204 [IWAIT] swi6: task queue
   50 ffffff011f5de9c0    0     0     0 0000204 [IWAIT] swi6:+
    9 ffffff011f5df000    0     0     0 0000204 [SLPQ - 0xffffff0000caa900]
[SLP] thread taskq
   49 ffffff011f5df340    0     0     0 0000204 [IWAIT] swi5:+
    8 ffffff011f5df680    0     0     0 0000204 [SLPQ - 0xffffff0000caac00]
[SLP] acpi_task2
    7 ffffff011f5df9c0    0     0     0 0000204 [SLPQ - 0xffffff0000caac00]
[SLP] acpi_task1
    6 ffffff011f5879c0    0     0     0 0000204 [SLPQ - 0xffffff0000caac00]
[SLP] acpi_task0
    5 ffffff011f577000    0     0     0 0000204 [SLPQ - 0xffffff0000caad00]
[SLP] kqueue taskq
   48 ffffff011f577340    0     0     0 0000204 [IWAIT] swi2: cambio
   47 ffffff011f577680    0     0     0 0000204 [SLPQ - 0xffffffff8086caa0]
[SLP] yarrow
    4 ffffff011f5779c0    0     0     0 0000204 [SLPQ - 0xffffffff8086f808]
[SLP] g_down
    3 ffffff011f598000    0     0     0 0000204 [SLPQ - 0xffffffff8086f800]
[SLP] g_up
    2 ffffff011f598340    0     0     0 0000204 [SLPQ - 0xffffffff8086f7f0]
[SLP] g_event
   46 ffffff011f598680    0     0     0 0000204 [IWAIT] swi3: vm
   45 ffffff011f5989c0    0     0     0 000020c [CPU 0] swi4: clock sio
   44 ffffff011f5b9000    0     0     0 0000204 [IWAIT] swi1: net
   43 ffffff011f597680    0     0     0 0000204 [IWAIT] irq31:
   42 ffffff011f5979c0    0     0     0 0000204 [IWAIT] irq30:
   41 ffffff011f584000    0     0     0 0000204 [IWAIT] irq29:
   40 ffffff011f584340    0     0     0 0000204 [IWAIT] irq28:
   39 ffffff011f584680    0     0     0 0000204 [IWAIT] irq27: bge1
   38 ffffff011f5849c0    0     0     0 0000204 [IWAIT] irq26: bge0
   37 ffffff011f587000    0     0     0 0000204 [IWAIT] irq25: ahd1
   36 ffffff011f587340    0     0     0 0000204 [IWAIT] irq24: ahd0
   35 ffffff011f587680    0     0     0 0000204 [IWAIT] irq23:
   34 ffffff011f58a680    0     0     0 0000204 [IWAIT] irq22:
   33 ffffff011f58a9c0    0     0     0 0000204 [IWAIT] irq21:
   32 ffffff011f5b5000    0     0     0 0000204 [IWAIT] irq20:
   31 ffffff011f5b5340    0     0     0 0000204 [IWAIT] irq19: ohci0 ohci1
   30 ffffff011f5b5680    0     0     0 0000204 [IWAIT] irq18:
   29 ffffff011f5b59c0    0     0     0 0000204 [IWAIT] irq17:
   28 ffffff011f597000    0     0     0 0000204 [IWAIT] irq16:
   27 ffffff011f597340    0     0     0 0000204 [IWAIT] irq15: ata1
   26 ffffff011f5da9c0    0     0     0 0000204 [IWAIT] irq14: ata0
   25 ffffff011f588000    0     0     0 0000204 [IWAIT] irq13:
   24 ffffff011f588340    0     0     0 0000204 [IWAIT] irq12:
   23 ffffff011f588680    0     0     0 0000204 [IWAIT] irq11:
   22 ffffff011f5889c0    0     0     0 0000204 [IWAIT] irq10:
   21 ffffff011f58a000    0     0     0 0000204 [IWAIT] irq9: acpi0
   20 ffffff011f58a340    0     0     0 0000204 [IWAIT] irq8:
   19 ffffff011f5b6340    0     0     0 0000204 [IWAIT] irq7: ppc0
   18 ffffff011f5b6680    0     0     0 0000204 [IWAIT] irq6: fdc0
   17 ffffff011f5b69c0    0     0     0 0000204 [IWAIT] irq5:
   16 ffffff011f5da000    0     0     0 0000204 [IWAIT] irq4: sio0
   15 ffffff011f5da340    0     0     0 0000204 [IWAIT] irq3:
   14 ffffff011f5da680    0     0     0 0000204 [IWAIT] irq0:
   13 ffffff011f5af000    0     0     0 0000204 [IWAIT] irq1: atkbd0
   12 ffffff011f5af340    0     0     0 000020c [Can run] idle: cpu0
   11 ffffff011f5af680    0     0     0 000020c [Can run] idle: cpu1
    1 ffffff011f5af9c0    0     0     1 0004200 [SLPQ wait 0xffffff011f5af9c0]
[SLP] init
   10 ffffff011f5b6000    0     0     0 0000204 [SLPQ ktrace 
0xffffffff80870900][SLP] ktrace
    0 ffffffff8086f960    0     0     0 0000200 [IWAIT] swapper
db> show lockedvnods
Locked vnodes
db>


The kernel config is :
ident           OFFICE
include         SMP
options         QUOTA
options         ALT_BREAK_TO_DEBUGGER
options         KDB                     # Enable kernel debugger support.
options         DDB                     # Support DDB.
makeoptions     DEBUG=-g
options         DEBUG_VFS_LOCKS
options         DEBUG_LOCKS

I think that i will boot properly without DEBUG_VFS_LOCKS and can test this if 
needed?

Niki



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200602281158.03030.nike_d>