Date: Wed, 7 Apr 2010 09:00:44 +0200 (CEST) From: Petr Salinger <Petr.Salinger@seznam.cz> To: Kostik Belousov <kostikbel@gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: leak of the vnodes Message-ID: <Pine.LNX.4.62.1004070838050.27621@sci.felk.cvut.cz> In-Reply-To: <20100406210321.GE2415@deviant.kiev.zoral.com.ua> References: <20100403210829.GW2415@deviant.kiev.zoral.com.ua> <Pine.LNX.4.62.1004052223210.23732@sci.felk.cvut.cz> <20100405185924.GS2415@deviant.kiev.zoral.com.ua> <Pine.LNX.4.62.1004061011470.24982@sci.felk.cvut.cz> <20100406092429.GV2415@deviant.kiev.zoral.com.ua> <Pine.LNX.4.62.1004061700590.25851@sci.felk.cvut.cz> <20100406140308.GY2415@deviant.kiev.zoral.com.ua> <20100406140852.GZ2415@deviant.kiev.zoral.com.ua> <20100406144402.GB2415@deviant.kiev.zoral.com.ua> <Pine.LNX.4.62.1004062154060.26438@sci.felk.cvut.cz> <20100406210321.GE2415@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --546530387-1330599894-1270623644=:27621 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Wed, 7 Apr 2010, Kostik Belousov wrote: > On Tue, Apr 06, 2010 at 10:01:56PM +0200, Petr Salinger wrote: >>> Can you try to get a backtrace at the points you have shown me ? >> >> All are similar to this, with ptyp5/ptyp6/ptyp7 name changes. >> >> a vnode 0xffffff0058978000: tag devfs, type VCHR >> usecount 1, writecount 1, refcount 2 mountedhere 0xffffff0039cb0c00 >> flags (VI_DOOMED) >> lock type devfs: EXCL by thread 0xffffff0039e16760 (pid 31427) >> dev ptyp5 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> vgonel() at vgonel+0x424 >> vgone() at vgone+0x39 >> devfs_delete() at devfs_delete+0x1b1 >> devfs_populate_loop() at devfs_populate_loop+0x228 >> devfs_populate() at devfs_populate+0x42 >> devfs_lookup() at devfs_lookup+0x258 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x7e >> lookup() at lookup+0x467 >> namei() at namei+0x3ea >> vn_open_cred() at vn_open_cred+0x211 >> kern_openat() at kern_openat+0x188 >> syscall() at syscall+0x168 >> Xfast_syscall() at Xfast_syscall+0xdc >> --- syscall (5, FreeBSD ELF64, open), rip = 0x800622097, rsp = >> 0x7fffffffbb88, rbp = 0x7fffffffbe30 --- >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> vgonel() at vgonel+0x39d >> vgone() at vgone+0x39 >> devfs_delete() at devfs_delete+0x1b1 >> devfs_populate_loop() at devfs_populate_loop+0x228 >> devfs_populate() at devfs_populate+0x42 >> devfs_lookup() at devfs_lookup+0x258 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x7e >> lookup() at lookup+0x467 >> namei() at namei+0x3ea >> vn_open_cred() at vn_open_cred+0x211 >> kern_openat() at kern_openat+0x188 >> syscall() at syscall+0x168 >> Xfast_syscall() at Xfast_syscall+0xdc >> --- syscall (5, FreeBSD ELF64, open), rip = 0x800622097, rsp = >> 0x7fffffffbb88, rbp = 0x7fffffffbe30 --- >> a vnode 0xffffff00589b5b40: tag devfs, type VCHR >> usecount 1, writecount 1, refcount 2 mountedhere 0xffffff0028d75600 >> flags (VI_DOOMED) >> lock type devfs: EXCL by thread 0xffffff0028cfb3b0 (pid 4529) >> dev ptyp6 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> vgonel() at vgonel+0x424 >> vgone() at vgone+0x39 >> devfs_delete() at devfs_delete+0x1b1 >> devfs_populate_loop() at devfs_populate_loop+0x228 >> devfs_populate() at devfs_populate+0x42 >> devfs_lookup() at devfs_lookup+0x258 >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x7e >> lookup() at lookup+0x467 >> namei() at namei+0x3ea >> vn_open_cred() at vn_open_cred+0x211 >> kern_openat() at kern_openat+0x188 >> syscall() at syscall+0x168 >> Xfast_syscall() at Xfast_syscall+0xdc > > Why do you think that this is the problem ? I used the attached diff, with hackish snooping on allocated/freed memory for vnodes. When the vp pointer have been logged as active1/active2, it is (much) later shown with dead_vnodeops in DUMP_VP(). > One refcount unit is coming from devfs_delete() calling vhold() around > vgone() to prevent the vnode from disappearing under it. > > Second refcount unit comes in pair with use count unit. Use count > implies refcount, and use count is allocated when vnode is e.g. opened, > to account for struct file having a reference to struct vnode. > > What *might* happen for the device nodes you have shown, is that some > application has file opened for the node /dev/ttyp*, and then master pty > device closed. The slave /dev/ttyp* node is destroyed, that you see as > devfs_populate->devfs_delete() sequence. The vnode will be eventually > freed when corresponding file is closed. > If you can confirm that some process has file opened with the reclaimed > vnode, then my theory will be confirmed. What have to be logged ? > I think there should be something else going on. May be both processes share file and memory space (RFMEM). Petr --546530387-1330599894-1270623644=:27621 Content-Type: TEXT/x-diff; charset=US-ASCII; name=vfs_subr.diff Content-Transfer-Encoding: BASE64 Content-ID: <Pine.LNX.4.62.1004070900440.27621@sci.felk.cvut.cz> Content-Description: Content-Disposition: attachment; filename=vfs_subr.diff SW5kZXg6IHZmc19zdWJyLmMNCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCi0t LSB2ZnNfc3Vici5jCShyZXZpc2lvbiAyMDYxNDIpDQorKysgdmZzX3N1YnIu Ywkod29ya2luZyBjb3B5KQ0KQEAgLTI4MCw2ICsyODAsNjIgQEANCiAjZGVm aW5lIFZTSE9VTERCVVNZKHZwKSAoKCh2cCktPnZfaWZsYWcgJiBWSV9GUkVF KSAmJiAodnApLT52X2hvbGRjbnQpDQogDQogDQorI2RlZmluZSBNQVhfTFZO T0RFIDIwMDAwDQorc3RydWN0IHZub2RlICpsYnVmW01BWF9MVk5PREVdOw0K K2ludCBsY250Ow0KKw0KKw0KK3N0YXRpYyBpbmxpbmUgaW50IEZJTkRfVlAo c3RydWN0IHZub2RlICp2cCkNCit7DQoraW50IHBvczsNCitmb3IgKHBvcyA9 IDAgOyBwb3MgPCBNQVhfTFZOT0RFOyBwb3MrKykNCit7DQorCWlmIChsYnVm W3Bvc10gPT0gdnApDQorCQlyZXR1cm4gcG9zOw0KK30NCityZXR1cm4gLTE7 DQorDQorfQ0KKw0KK3N0YXRpYyBpbmxpbmUgdm9pZCBEUk9QX1ZQKHN0cnVj dCB2bm9kZSAqdnApDQorew0KKwlpbnQgcG9zID0gIEZJTkRfVlAodnApOw0K KyAgICAgICAgaWYgKHBvcyAhPSAtMSkNCisgICAgICAgICAgICAgICBsYnVm W3Bvc10gPSBOVUxMOw0KKwkgICAgICAgICAgICAgICAgDQorDQorfQ0KKw0K K3N0YXRpYyBpbmxpbmUgdm9pZCBBRERfVlAoc3RydWN0IHZub2RlICp2cCkN Cit7DQorICBpZiAobGNudCA8IE1BWF9MVk5PREUpDQorICB7DQorICAJbGJ1 ZltsY250XSA9IHZwOw0KKyAgCWxjbnQrKzsNCisgIH0NCisgIGVsc2UNCisg IHsNCisgICAgICBpbnQgcG9zID0gRklORF9WUChOVUxMKTsNCisgICAgICBp ZiAocG9zICE9IC0xKQ0KKyAgCSAgbGJ1Zltwb3NdID0gdnA7DQorICB9Owkg IA0KK30NCisNCisNCitzdGF0aWMgaW5saW5lIHZvaWQgRFVNUF9WUCh2b2lk KQ0KK3sNCitpbnQgcG9zOw0KK3N0cnVjdCB2bm9kZSAqdnA7DQorZm9yIChw b3MgPSAwIDsgcG9zIDwgTUFYX0xWTk9ERTsgcG9zKyspDQorew0KKwlpZiAo bGJ1Zltwb3NdICE9IE5VTEwpDQorCXsNCisJICAgICAgICB2cCA9IGxidWZb cG9zXTsNCisJCUNUUjYoS1RSX1NQQVJFMywgInZwICVwIHRhZyAlcyB0eXBl ICVkIGZsYWdzICUwOHggdXNlIGNvdW50ICVkIGhvbGQgY291bnQgJWQiLCB2 cCwgdnAtPnZfdGFnLCB2cC0+dl90eXBlLCB2cC0+dl9pZmxhZywgdnAtPnZf dXNlY291bnQsIHZwLT52X2hvbGRjbnQpOw0KKwl9DQorfQkNCit9Ow0KKw0K IC8qDQogICogSW5pdGlhbGl6ZSB0aGUgdm5vZGUgbWFuYWdlbWVudCBkYXRh IHN0cnVjdHVyZXMuDQogICovDQpAQCAtMzA1LDYgKzM2MSw5IEBADQogCQkJ ICAgIGRlc2lyZWR2bm9kZXMsIE1BWFZOT0RFU19NQVgpOw0KIAkJZGVzaXJl ZHZub2RlcyA9IE1BWFZOT0RFU19NQVg7DQogCX0NCisJbWVtc2V0KGxidWYs IE1BWF9MVk5PREUgKiAoc2l6ZW9mKHN0cnVjdCB2bm9kZSAqKSksIDApOw0K KwlsY250ID0gMDsNCisJDQogCXdhbnRmcmVldm5vZGVzID0gZGVzaXJlZHZu b2RlcyAvIDQ7DQogCW10eF9pbml0KCZtbnRpZF9tdHgsICJtbnRpZCIsIE5V TEwsIE1UWF9ERUYpOw0KIAlUQUlMUV9JTklUKCZ2bm9kZV9mcmVlX2xpc3Qp Ow0KQEAgLTgwMCw2ICs4NTksNyBAQA0KIAkJfQ0KIAkJbXR4X3VubG9jaygm bW91bnRsaXN0X210eCk7DQogCQlpZiAoZG9uZSA9PSAwKSB7DQorCQkJRFVN UF9WUCgpOw0KIAkJCUVWRU5USEFORExFUl9JTlZPS0UodmZzX2xvd3Zub2Rl cywgZGVzaXJlZHZub2RlcyAvIDEwKTsNCiAjaWYgMA0KIAkJCS8qIFRoZXNl IG1lc3NhZ2VzIGFyZSB0ZW1wb3JhcnkgZGVidWdnaW5nIGFpZHMgKi8NCkBA IC04NjQsNiArOTI0LDcgQEANCiAJbG9ja2Rlc3Ryb3kodnAtPnZfdm5sb2Nr KTsNCiAJbXR4X2Rlc3Ryb3koJnZwLT52X2ludGVybG9jayk7DQogCW10eF9k ZXN0cm95KEJPX01UWChibykpOw0KKwlEUk9QX1ZQKHZwKTsNCiAJdW1hX3pm cmVlKHZub2RlX3pvbmUsIHZwKTsNCiB9DQogDQpAQCAtOTcwLDYgKzEwMzEs OCBAQA0KIAludW12bm9kZXMrKzsNCiAJbXR4X3VubG9jaygmdm5vZGVfZnJl ZV9saXN0X210eCk7DQogCXZwID0gKHN0cnVjdCB2bm9kZSAqKSB1bWFfemFs bG9jKHZub2RlX3pvbmUsIE1fV0FJVE9LfE1fWkVSTyk7DQorCUFERF9WUCh2 cCk7DQorCQ0KIAkvKg0KIAkgKiBTZXR1cCBsb2Nrcy4NCiAJICovDQpAQCAt MjI5OSw2ICsyMzYyLDEyIEBADQogCWlmICh2cC0+dl9ob2xkY250IDw9IDAp DQogCQlwYW5pYygidmRyb3A6IGhvbGRjbnQgJWQiLCB2cC0+dl9ob2xkY250 KTsNCiAJdnAtPnZfaG9sZGNudC0tOw0KKwkNCisJaWYgKHZwLT52X2lmbGFn ICYgVklfRE9PTUVEKQ0KKwl7DQorCQlDVFI1KEtUUl9TUEFSRTQsICIlczog dnAgJXAgZmxhZ3MgJTA4eCB1c2UgY291bnQgJWQgaG9sZCBjb3VudCAlZCIs IF9fZnVuY19fLCB2cCwgdnAtPnZfaWZsYWcsIHZwLT52X3VzZWNvdW50LCB2 cC0+dl9ob2xkY250KTsNCisJfQ0KKwkNCiAJaWYgKHZwLT52X2hvbGRjbnQg PT0gMCkgew0KIAkJaWYgKHZwLT52X2lmbGFnICYgVklfRE9PTUVEKSB7DQog CQkJQ1RSMihLVFJfVkZTLCAiJXM6IGRlc3Ryb3lpbmcgdGhlIHZub2RlICVw IiwgX19mdW5jX18sDQpAQCAtMjUyNSwxMiArMjU5NCwxOSBAQA0KIAkgICAg KCJ2Z29uZWw6IHZwICVwIGhhcyBubyByZWZlcmVuY2UuIiwgdnApKTsNCiAJ Q1RSMihLVFJfVkZTLCAiJXM6IHZwICVwIiwgX19mdW5jX18sIHZwKTsNCiAJ dGQgPSBjdXJ0aHJlYWQ7DQorCQ0KKwkNCisJQ1RSNShLVFJfU1BBUkU0LCAi JXM6IHZwICVwIGZsYWdzICUwOHggdXNlIGNvdW50ICVkIGhvbGQgY291bnQg JWQiLCBfX2Z1bmNfXywgdnAsIHZwLT52X2lmbGFnLCB2cC0+dl91c2Vjb3Vu dCwgdnAtPnZfaG9sZGNudCk7DQogDQogCS8qDQogCSAqIERvbid0IHZnb25l bCBpZiB3ZSdyZSBhbHJlYWR5IGRvb21lZC4NCiAJICovDQogCWlmICh2cC0+ dl9pZmxhZyAmIFZJX0RPT01FRCkNCisJew0KKwkJQ1RSNihLVFJfU1BBUkUz LCAiYWxyZWFkeSAlcCB0YWcgJXMgdHlwZSAlZCBmbGFncyAlMDh4IHVzZSBj b3VudCAlZCBob2xkIGNvdW50ICVkIiwgdnAsIHZwLT52X3RhZywgdnAtPnZf dHlwZSwgdnAtPnZfaWZsYWcsIHZwLT52X3VzZWNvdW50LCB2cC0+dl9ob2xk Y250KTsNCisNCiAJCXJldHVybjsNCisJfQkNCiAJdnAtPnZfaWZsYWcgfD0g VklfRE9PTUVEOw0KIAkvKg0KIAkgKiBDaGVjayB0byBzZWUgaWYgdGhlIHZu b2RlIGlzIGluIHVzZS4gIElmIHNvLCB3ZSBoYXZlIHRvIGNhbGwNCkBAIC0y NTU0LDEyICsyNjMwLDIyIEBADQogCSAqIGRlYWN0aXZhdGVkIGJlZm9yZSBi ZWluZyByZWNsYWltZWQuDQogCSAqLw0KIAlpZiAoYWN0aXZlKQ0KKwl7DQor CQlDVFI2KEtUUl9TUEFSRTMsICJhY3RpdmUxICVwIHRhZyAlcyB0eXBlICVk IGZsYWdzICUwOHggdXNlIGNvdW50ICVkIGhvbGQgY291bnQgJWQiLCB2cCwg dnAtPnZfdGFnLCB2cC0+dl90eXBlLCB2cC0+dl9pZmxhZywgdnAtPnZfdXNl Y291bnQsIHZwLT52X2hvbGRjbnQpOw0KKwkJdm5fcHJpbnRmKHZwLCAiYSB2 bm9kZSAiKTsNCisNCisJCWtkYl9iYWNrdHJhY2UoKTsNCiAJCVZPUF9DTE9T RSh2cCwgRk5PTkJMT0NLLCBOT0NSRUQsIHRkKTsNCisJfQ0KIAlpZiAob3dl aW5hY3QgfHwgYWN0aXZlKSB7DQogCQlWSV9MT0NLKHZwKTsNCiAJCWlmICgo dnAtPnZfaWZsYWcgJiBWSV9ET0lOR0lOQUNUKSA9PSAwKQ0KIAkJCXZpbmFj dGl2ZSh2cCwgdGQpOw0KIAkJVklfVU5MT0NLKHZwKTsNCisJCUNUUjYoS1RS X1NQQVJFMywgImFjdGl2ZTIgJXAgdGFnICVzIHR5cGUgJWQgZmxhZ3MgJTA4 eCB1c2UgY291bnQgJWQgaG9sZCBjb3VudCAlZCIsIHZwLCB2cC0+dl90YWcs IHZwLT52X3R5cGUsIHZwLT52X2lmbGFnLCB2cC0+dl91c2Vjb3VudCwgdnAt PnZfaG9sZGNudCk7DQorCQlrZGJfYmFja3RyYWNlKCk7DQorCQkNCisNCiAJ fQ0KIAkvKg0KIAkgKiBSZWNsYWltIHRoZSB2bm9kZS4NCkBAIC0yNTg2LDcg KzI2NzIsNyBAQA0KIAlWSV9MT0NLKHZwKTsNCiAJdnAtPnZfdm5sb2NrID0g JnZwLT52X2xvY2s7DQogCXZwLT52X29wID0gJmRlYWRfdm5vZGVvcHM7DQot CXZwLT52X3RhZyA9ICJub25lIjsNCisvLwl2cC0+dl90YWcgPSAibm9uZSI7 DQogCXZwLT52X3R5cGUgPSBWQkFEOw0KIH0NCiANCkBAIC0yNjM1LDkgKzI3 MjEsOSBAQA0KIAl2YV9zdGFydChhcCwgZm10KTsNCiAJdnByaW50ZihmbXQs IGFwKTsNCiAJdmFfZW5kKGFwKTsNCi0JcHJpbnRmKCIlcDogIiwgKHZvaWQg Kil2cCk7DQotCXByaW50ZigidGFnICVzLCB0eXBlICVzXG4iLCB2cC0+dl90 YWcsIHR5cGVuYW1lW3ZwLT52X3R5cGVdKTsNCi0JcHJpbnRmKCIgICAgdXNl Y291bnQgJWQsIHdyaXRlY291bnQgJWQsIHJlZmNvdW50ICVkIG1vdW50ZWRo ZXJlICVwXG4iLA0KKwlkYl9wcmludGYoIiVwOiAiLCAodm9pZCAqKXZwKTsN CisJZGJfcHJpbnRmKCJ0YWcgJXMsIHR5cGUgJXNcbiIsIHZwLT52X3RhZywg dHlwZW5hbWVbdnAtPnZfdHlwZV0pOw0KKwlkYl9wcmludGYoIiAgICB1c2Vj b3VudCAlZCwgd3JpdGVjb3VudCAlZCwgcmVmY291bnQgJWQgbW91bnRlZGhl cmUgJXBcbiIsDQogCSAgICB2cC0+dl91c2Vjb3VudCwgdnAtPnZfd3JpdGVj b3VudCwgdnAtPnZfaG9sZGNudCwgdnAtPnZfbW91bnRlZGhlcmUpOw0KIAli dWZbMF0gPSAnXDAnOw0KIAlidWZbMV0gPSAnXDAnOw0KQEAgLTI2ODgsMTQg KzI3NzQsMTQgQEANCiAJCXNucHJpbnRmKGJ1ZjIsIHNpemVvZihidWYyKSwg InxWSSgweCVseCkiLCBmbGFncyk7DQogCQlzdHJsY2F0KGJ1ZiwgYnVmMiwg c2l6ZW9mKGJ1ZikpOw0KIAl9DQotCXByaW50ZigiICAgIGZsYWdzICglcylc biIsIGJ1ZiArIDEpOw0KKwlkYl9wcmludGYoIiAgICBmbGFncyAoJXMpXG4i LCBidWYgKyAxKTsNCiAJaWYgKG10eF9vd25lZChWSV9NVFgodnApKSkNCi0J CXByaW50ZigiIFZJX0xPQ0tlZCIpOw0KKwkJZGJfcHJpbnRmKCIgVklfTE9D S2VkIik7DQogCWlmICh2cC0+dl9vYmplY3QgIT0gTlVMTCkNCi0JCXByaW50 ZigiICAgIHZfb2JqZWN0ICVwIHJlZiAlZCBwYWdlcyAlZFxuIiwNCisJCWRi X3ByaW50ZigiICAgIHZfb2JqZWN0ICVwIHJlZiAlZCBwYWdlcyAlZFxuIiwN CiAJCSAgICB2cC0+dl9vYmplY3QsIHZwLT52X29iamVjdC0+cmVmX2NvdW50 LA0KIAkJICAgIHZwLT52X29iamVjdC0+cmVzaWRlbnRfcGFnZV9jb3VudCk7 DQotCXByaW50ZigiICAgICIpOw0KKwlkYl9wcmludGYoIiAgICAiKTsNCiAJ bG9ja21ncl9wcmludGluZm8odnAtPnZfdm5sb2NrKTsNCiAJaWYgKHZwLT52 X2RhdGEgIT0gTlVMTCkNCiAJCVZPUF9QUklOVCh2cCk7DQo= --546530387-1330599894-1270623644=:27621--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.4.62.1004070838050.27621>