Date: Tue, 17 Aug 2010 19:42:41 +0400 From: pluknet <pluknet@gmail.com> To: Kostik Belousov <kostikbel@gmail.com> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: LOR on nfs: vfs_vnops.c:301 kern_descrip.c:1580 Message-ID: <AANLkTimfZ=UCiX9eo9XCwcROwViFCxJoB-rD1JxdAt2B@mail.gmail.com> In-Reply-To: <20100816185456.GU2396@deviant.kiev.zoral.com.ua> References: <AANLkTimJ=d06D2z24QyRQ98zEa1Pemk4=vkNGLNiX90N@mail.gmail.com> <AANLkTi=22WWy8Z5VWqri1p70iRzyocD9bWvNHwJavZ%2Bj@mail.gmail.com> <20100816185456.GU2396@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--0016367fa1bab09977048e06ccc5
Content-Type: text/plain; charset=ISO-8859-1
2010/8/16 Kostik Belousov <kostikbel@gmail.com>:
> On Mon, Aug 16, 2010 at 09:07:24PM +0400, pluknet wrote:
>> On 16 August 2010 21:05, pluknet <pluknet@gmail.com> wrote:
>> > Hi.
>> >
>> > Seeing on mostly idle, recently updated current, while closing a file.
>> > Presumably never reported on ML.
[...]
>>
> Both LORs are valid. The fork performed deep inside the VFS call stack
> is obviously problematic. As a workaround, you may fix the number of
> nfsiods.
>
> Proper fix might consist of creating a shepherd thread which only task
> is to act on the requests on creating new nfsiods.
>
> Would you try to implement this ? I will provide the assistance, if needed.
Hmm.. I tried to move kproc_create() under shepherd thread and now stuck
with cp process lockup in [bo_wwait] when cp'ing something on nfs: cp a b.
Did I screw up something?
See weird draft patch attached (weird, as I have no idea how to nicely
exchange data between nfs_nfsiodnew() and shep_thread() thread).
load: 1.34 cmd: cp 1348 [bo_wwait] 4.74r 0.00u 0.00s 0% 1204k
tst-web# procstat -k 1348
PID TID COMM TDNAME KSTACK
1348 100095 cp - mi_switch sleepq_switch
sleepq_wait _sleep bufobj_wwait nfs_flush nfs_close vn_close
vn_closefile _fdrop closef kern_close syscallenter syscall
Xfast_syscall
Process 1347 (cp) thread 0xffffff0002ed7000 (100094)
exclusive lockmgr nfs (nfs) r = 0 (0xffffff006a05a638) locked @
/usr/src/sys/kern/vfs_vnops.c:301
(kgdb) bt
#0 sched_switch (td=0xffffff0002ed7000, newtd=0xffffffff80ca17e0,
flags=Variable "flags" is not available.
)
at /usr/src/sys/kern/sched_ule.c:1848
#1 0xffffffff805bf49b in mi_switch (flags=260, newtd=0x0)
at /usr/src/sys/kern/kern_synch.c:449
#2 0xffffffff805f50e3 in sleepq_switch (wchan=0xffffff006a05a720, pri=77)
at /usr/src/sys/kern/subr_sleepqueue.c:530
#3 0xffffffff805f5ccd in sleepq_wait (wchan=0xffffff006a05a720, pri=77)
at /usr/src/sys/kern/subr_sleepqueue.c:609
#4 0xffffffff805bfac9 in _sleep (ident=0xffffff006a05a720,
lock=0xffffff006a05a6c0, priority=Variable "priority" is not available.
) at /usr/src/sys/kern/kern_synch.c:234
#5 0xffffffff80633083 in bufobj_wwait (bo=0xffffff006a05a6c0,
slpflag=Variable "slpflag" is not available.
)
at /usr/src/sys/kern/vfs_bio.c:4016
#6 0xffffffff8077f5af in nfs_flush (vp=0xffffff006a05a5a0, waitfor=1,
commit=Variable "commit" is not available.
)
at /usr/src/sys/nfsclient/nfs_vnops.c:3216
#7 0xffffffff807802e3 in nfs_close (ap=0xffffff8029bd8900)
at /usr/src/sys/nfsclient/nfs_vnops.c:644
#8 0xffffffff8065b3fe in vn_close (vp=0xffffff006a05a5a0, flags=2,
file_cred=0xffffff006a01b600, td=0xffffff0002ed7000) at vnode_if.h:225
#9 0xffffffff8065b4fa in vn_closefile (fp=0xffffff00027c7500,
td=0xffffff0002ed7000) at /usr/src/sys/kern/vfs_vnops.c:942
#10 0xffffffff8057e3e3 in _fdrop (fp=0xffffff00027c7500, td=Variable
"td" is not available.
) at file.h:277
#11 0xffffffff8057ff4b in closef (fp=0xffffff00027c7500, td=0xffffff0002ed7000)
at /usr/src/sys/kern/kern_descrip.c:2117
---Type <return> to continue, or q <return> to quit---
#12 0xffffffff80580530 in kern_close (td=0xffffff0002ed7000, fd=4)
at /usr/src/sys/kern/kern_descrip.c:1162
--
wbr,
pluknet
--0016367fa1bab09977048e06ccc5
Content-Type: application/octet-stream; name="nfs_shep.diff"
Content-Disposition: attachment; filename="nfs_shep.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_gcyx7bby0
SW5kZXg6IHN5cy9uZnNjbGllbnQvbmZzX3Zub3BzLmMKPT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL25mc2Ns
aWVudC9uZnNfdm5vcHMuYwkocmV2aXNpb24gMjExMjc5KQorKysgc3lzL25mc2NsaWVudC9uZnNf
dm5vcHMuYwkod29ya2luZyBjb3B5KQpAQCAtMjEyLDYgKzIxMiw3IEBACiAgKiBHbG9iYWwgdmFy
aWFibGVzCiAgKi8KIHN0cnVjdCBtdHggCW5mc19pb2RfbXR4Oworc3RydWN0IG10eAlzaGVwX210
eDsKIGVudW0gbmZzaW9kX3N0YXRlIG5mc19pb2R3YW50W05GU19NQVhBU1lOQ0RBRU1PTl07CiBz
dHJ1Y3QgbmZzbW91bnQgKm5mc19pb2Rtb3VudFtORlNfTUFYQVNZTkNEQUVNT05dOwogaW50CQkg
bmZzX251bWFzeW5jID0gMDsKSW5kZXg6IHN5cy9uZnNjbGllbnQvbmZzX3N1YnMuYwo9PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09Ci0tLSBzeXMvbmZzY2xpZW50L25mc19zdWJzLmMJKHJldmlzaW9uIDIxMTI3OSkKKysrIHN5
cy9uZnNjbGllbnQvbmZzX3N1YnMuYwkod29ya2luZyBjb3B5KQpAQCAtMzU0LDYgKzM1NCw3IEBA
CiAJICovCiAJbXR4X2luaXQoJm5mc19pb2RfbXR4LCAiTkZTIGlvZCBsb2NrIiwgTlVMTCwgTVRY
X0RFRik7CiAJbXR4X2luaXQoJm5mc194aWRfbXR4LCAiTkZTIHhpZCBsb2NrIiwgTlVMTCwgTVRY
X0RFRik7CisJbXR4X2luaXQoJnNoZXBfbXR4LCAiTkZTIHNoZXBfdGhyZWFkIGxvY2siLCBOVUxM
LCBNVFhfREVGKTsKIAogCW5mc19wYnVmX2ZyZWVjbnQgPSBuc3didWYgLyAyICsgMTsKIApJbmRl
eDogc3lzL25mc2NsaWVudC9uZnNfbmZzaW9kLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL25mc2NsaWVu
dC9uZnNfbmZzaW9kLmMJKHJldmlzaW9uIDIxMTI3OSkKKysrIHN5cy9uZnNjbGllbnQvbmZzX25m
c2lvZC5jCSh3b3JraW5nIGNvcHkpCkBAIC03NSw2ICs3NSwxMyBAQAogCiBzdGF0aWMgdm9pZAlu
ZnNzdmNfaW9kKHZvaWQgKik7CiAKK3N0cnVjdCBzaGVwaGVyZF9zdHIgeworCWludCAqc2hlcF9p
bnN0OworCWludCBzaGVwX25pb2Q7CisJaW50IHNoZXBfZXJyb3I7CisJaW50IHNoZXBfZG9uZTsK
K30gc2hlcF9jaGFuOworCiBzdGF0aWMgaW50IG5mc19hc3luY2RhZW1vbltORlNfTUFYQVNZTkNE
QUVNT05dOwogCiBTWVNDVExfREVDTChfdmZzX25mcyk7CkBAIC0xNTksMTAgKzE2NiwzMiBAQAog
ICAgIHNpemVvZiAobmZzX2lvZG1heCksIHN5c2N0bF9pb2RtYXgsICJJVSIsCiAgICAgIk1heCBu
dW1iZXIgb2YgbmZzaW9kIGt0aHJlYWRzIik7CiAKK3N0YXRpYyB2b2lkCitzaGVwX3RocmVhZCh2
b2lkICpkdW1teSkKK3sKKwlpbnQgZXJyID0gMDsKKworLy8JbXR4X2xvY2soJnNoZXBfbXR4KTsK
Kwlmb3IgKDs7KSB7CisvLwkJZXJyID0gbXNsZWVwKCZzaGVwX2NoYW4sICZzaGVwX210eCwgUFdB
SVQgfCBQQ0FUQ0gsICJzaHB3dCIsIDApOworCQllcnIgPSB0c2xlZXAoJnNoZXBfY2hhbiwgUFdB
SVQgfCBQQ0FUQ0gsICJzaHB3dCIsIDApOworCQlpZiAoZXJyKQorCQkJY29udGludWU7CisvLwkJ
bXR4X3VubG9jaygmc2hlcF9tdHgpOworCQlzaGVwX2NoYW4uc2hlcF9lcnJvciA9IGtwcm9jX2Ny
ZWF0ZShuZnNzdmNfaW9kLAorCQkgICAgJnNoZXBfY2hhbi5zaGVwX2luc3QsIE5VTEwsIFJGSElH
SFBJRCwgMCwgIm5mc2lvZCAlZCIsCisJCSAgICBzaGVwX2NoYW4uc2hlcF9uaW9kKTsKKwkJc2hl
cF9jaGFuLnNoZXBfZG9uZSA9IDE7CisvLwkJbXR4X2xvY2soJnNoZXBfbXR4KTsKKwl9CisvLwlt
dHhfdW5sb2NrKCZzaGVwX210eCk7Cit9CitTWVNJTklUKHNoZXBoZXJkLCBTSV9TVUJfS1RIUkVB
RF9JRExFLCBTSV9PUkRFUl9BTlksIHNoZXBfdGhyZWFkLCBOVUxMKTsKKwogaW50CiBuZnNfbmZz
aW9kbmV3KGludCBzZXRfaW9kd2FudCkKIHsKLQlpbnQgZXJyb3IsIGk7CisJaW50IGk7CiAJaW50
IG5ld2lvZDsKIAogCWlmIChuZnNfbnVtYXN5bmMgPj0gbmZzX2lvZG1heCkKQEAgLTE3OCwxMSAr
MjA3LDE2IEBACiAJCXJldHVybiAoLTEpOwogCWlmIChzZXRfaW9kd2FudCA+IDApCiAJCW5mc19p
b2R3YW50W2ldID0gTkZTSU9EX0NSRUFURURfRk9SX05GU19BU1lOQ0lPOworCWJ6ZXJvKCZzaGVw
X2NoYW4sIHNpemVvZihzaGVwX2NoYW4pKTsKKwlzaGVwX2NoYW4uc2hlcF9pbnN0ID0gbmZzX2Fz
eW5jZGFlbW9uICsgaTsKKwlzaGVwX2NoYW4uc2hlcF9uaW9kID0gbmV3aW9kOwogCW10eF91bmxv
Y2soJm5mc19pb2RfbXR4KTsKLQllcnJvciA9IGtwcm9jX2NyZWF0ZShuZnNzdmNfaW9kLCBuZnNf
YXN5bmNkYWVtb24gKyBpLCBOVUxMLCBSRkhJR0hQSUQsCi0JICAgIDAsICJuZnNpb2QgJWQiLCBu
ZXdpb2QpOworCXdha2V1cCgmc2hlcF9jaGFuKTsKIAltdHhfbG9jaygmbmZzX2lvZF9tdHgpOwot
CWlmIChlcnJvcikgeworCXdoaWxlIChzaGVwX2NoYW4uc2hlcF9kb25lKQorCQljb250aW51ZTsK
KwlzaGVwX2NoYW4uc2hlcF9kb25lID0gMDsKKwlpZiAoc2hlcF9jaGFuLnNoZXBfZXJyb3IpIHsK
IAkJaWYgKHNldF9pb2R3YW50ID4gMCkKIAkJCW5mc19pb2R3YW50W2ldID0gTkZTSU9EX05PVF9B
VkFJTEFCTEU7CiAJCXJldHVybiAoLTEpOwpJbmRleDogc3lzL25mc2NsaWVudC9uZnMuaAo9PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09Ci0tLSBzeXMvbmZzY2xpZW50L25mcy5oCShyZXZpc2lvbiAyMTEyNzkpCisrKyBzeXMv
bmZzY2xpZW50L25mcy5oCSh3b3JraW5nIGNvcHkpCkBAIC0xMjUsNiArMTI1LDcgQEAKIAogZXh0
ZXJuIHN0cnVjdCBuZnNzdGF0cyBuZnNzdGF0czsKIGV4dGVybiBzdHJ1Y3QgbXR4IG5mc19pb2Rf
bXR4OworZXh0ZXJuIHN0cnVjdCBtdHggc2hlcF9tdHg7CiAKIGV4dGVybiBpbnQgbmZzX251bWFz
eW5jOwogZXh0ZXJuIHVuc2lnbmVkIGludCBuZnNfaW9kbWF4Owo=
--0016367fa1bab09977048e06ccc5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTimfZ=UCiX9eo9XCwcROwViFCxJoB-rD1JxdAt2B>