Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 16 Dec 2004 14:11:19 +0100
From:      "J. Martin Petersen" <techlists@motrix.dk>
To:        "'Doug White'" <dwhite@gumbysoft.com>
Cc:        freebsd-stable@freebsd.org
Subject:    RE: netstat fails with memory allocation error and error in kvm_read
Message-ID:  <20041216131119.A0DB5C2C2@brugere.aub.dk>
In-Reply-To: <20041213101844.S92964@carver.gumbysoft.com>

next in thread | previous in thread | raw e-mail | index | archive | help
 
> You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'.

I just got another "netstat: kvm_read: Bad address". vm.kvm_free was
675278848 two seconds earlier and a couple of seconds later (and also
currently).



The output from vm.zone a couple of seconds after:
vm.zone:
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS

pfosfp:           28,        0,    155,    226,     1085
pfospfen:        108,        0,    293,     31,     2051
pfiaddrpl:        92,        0,      5,     79,       28
pfstatescrub:      8,        0,  26676,  28540, 37487180
pffrcent:         12,    50141,      0,      0,        0
pffrcache:        48,    10062,      0,      0,        0
pffrag:           48,        0,      0,    234,   378044
pffrent:          16,     5075,      0,    203,   934804
pfrkentry:       152,        0,      4,     48,        8
pfrktable:       248,        0,      4,     28,       12
pfpooladdrpl:     68,        0,     21,     91,      126
pfaltqpl:        128,        0,     19,     71,      133
pfstatepl:       256,    65010,  14396,  23014, 29835551
pfrulepl:        560,        0,    248,    221,     1693
pfsrctrpl:        80,        0,      0,      0,        0
FFS2 dinode:     256,        0,  32828,    427,  1154768
FFS1 dinode:     128,        0,      0,      0,        0
FFS inode:       140,        0,  32828,    352,  1154768
SWAPMETA:        276,    63672,      9,     47,       40
rtentry:         132,        0,    219,    187,     6666
unpcb:           140,    17220,      9,     47,     5217
ripcb:           180,     8140,      0,     44,       15
sackhole:         16,        0,      0,    203,     3107
tcpreass:         20,     1183,      0,    169,      134
hostcache:        88,    15400,      3,     85,       39
syncache:        108,    15372,      0,     72,      374
tcptw:            56,     1675,      2,    132,       48
tcpcb:           448,     8136,      4,     23,      489
inpcb:           180,     8140,      6,     38,      489
udpcb:           180,     8140,     11,     33,  6297398
socket:          324,     8136,     24,     36,  6303121
KNOTE:            68,        0,      0,    112, 12538429
PIPE:            384,        0,      2,     78,   251051
DIRHASH:        1024,        0,   1621,     11,     4318
NAMEI:          1024,        0,      3,     17, 131064562
L VFS Cache:     291,        0,     38,    456,     8482
S VFS Cache:      68,        0,  29954,   5550,  1214821
VNODEPOLL:        64,        0,      0,    118,        2
VNODE:           264,        0,  33205,     20,    33205
ata_request:     200,        0,      0,     38,  2322599
g_bio:           132,        0,      0,   2204,  4301499
MbufClust:      2048,    17216,    906,    124, 12121216
Mbuf:            256,        0,    907,    188, 2520398988
Packet:          256,        0,    740,    355, 2728311990
VMSPACE:         300,        0,     23,     81,   726569
UPCALL:           44,        0,      0,      0,        0
KSEGRP:          104,        0,    144,     31,      144
TID:             140,        0,      1,     53,        1
THREAD:          388,        0,    144,      6,      144
PROC:            452,        0,     73,     71,   726627
Files:            68,        0,     71,    153, 122894112
4096:           4096,        0,    129,     60,   746477
2048:           2048,        0,    130,     30,    24039
1024:           1024,        0,     23,     61, 14341894
512:             512,        0,    244,     92,   656520
256:             256,        0,    256,   1094,  7153814
128:             128,        0,   1842,    318, 15428344
64:               64,        0,   4481,   1183,  8923514
32:               32,        0,   2595,    682, 4033338850
16:               16,        0,   1930,    303, 22115093
DP fakepg:        72,        0,      0,      0,        0
PV ENTRY:         24,   944530,   5540,  17080, 315728548
MAP ENTRY:        68,        0,    439,   1241, 32683625
KMAP ENTRY:       68,    32200,     40,    464,  5444457
MAP:             192,        0,      7,     33,        5
VM OBJECT:       132,        0,  32560,   1428, 14688659
128 Bucket:      524,        0,    966,      0,        0
64 Bucket:       268,        0,     30,     40,        0
32 Bucket:       140,        0,     30,     26,        0
16 Bucket:        76,        0,     15,     35,        0
UMA Hash:        128,        0,      2,     28,        0
UMA RCntSlab:    104,        0,    515,      3,        0
UMA Slabs:        64,        0,    793,     33,        0
UMA Zones:        88,        0,     73,      7,        0
UMA Kegs:        136,        0,     73,     23,        0


vm.zone a couple of seconds before:
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS

pfosfp:           28,        0,    155,    226,     1085
pfospfen:        108,        0,    293,     31,     2051
pfiaddrpl:        92,        0,      5,     79,       28
pfstatescrub:      8,        0,  27146,  28070, 37485813
pffrcent:         12,    50141,      0,      0,        0
pffrcache:        48,    10062,      0,      0,        0
pffrag:           48,        0,      0,    234,   378044
pffrent:          16,     5075,      0,    203,   934804
pfrkentry:       152,        0,      4,     48,        8
pfrktable:       248,        0,      4,     28,       12
pfpooladdrpl:     68,        0,     21,     91,      126
pfaltqpl:        128,        0,     19,     71,      133
pfstatepl:       256,    65010,  14858,  22552, 29834555
pfrulepl:        560,        0,    248,    221,     1693
pfsrctrpl:        80,        0,      0,      0,        0
FFS2 dinode:     256,        0,  32820,    435,  1154765
FFS1 dinode:     128,        0,      0,      0,        0
FFS inode:       140,        0,  32820,    360,  1154765
SWAPMETA:        276,    63672,      9,     47,       40
rtentry:         132,        0,    222,    184,     6666
unpcb:           140,    17220,      9,     47,     5217
ripcb:           180,     8140,      0,     44,       15
sackhole:         16,        0,      0,    203,     3107
tcpreass:         20,     1183,      0,    169,      134
hostcache:        88,    15400,      1,     87,       39
syncache:        108,    15372,      0,     72,      374
tcptw:            56,     1675,      0,    134,       48
tcpcb:           448,     8136,      4,     23,      489
inpcb:           180,     8140,      4,     40,      489
udpcb:           180,     8140,     11,     33,  6296980
socket:          324,     8136,     24,     36,  6302703
KNOTE:            68,        0,      0,    112, 12537593
PIPE:            384,        0,      2,     78,   251017
DIRHASH:        1024,        0,   1621,     11,     4318
NAMEI:          1024,        0,      3,     17, 131046202
L VFS Cache:     291,        0,     38,    456,     8482
S VFS Cache:      68,        0,  29953,   5551,  1214811
VNODEPOLL:        64,        0,      0,    118,        2
VNODE:           264,        0,  33205,     20,    33205
ata_request:     200,        0,      0,     38,  2322417
g_bio:           132,        0,      0,   2204,  4301228
MbufClust:      2048,    17216,    906,    124, 12121216
Mbuf:            256,        0,    907,    188, 2520288299
Packet:          256,        0,    674,    421, 2728222296
VMSPACE:         300,        0,     23,     81,   726488
UPCALL:           44,        0,      0,      0,        0
KSEGRP:          104,        0,    144,     31,      144
TID:             140,        0,      1,     53,        1
THREAD:          388,        0,    144,      6,      144
PROC:            452,        0,     73,     71,   726546
Files:            68,        0,     71,    153, 122876814
4096:           4096,        0,    129,     60,   746392
2048:           2048,        0,    130,     30,    24033
1024:           1024,        0,     23,     61, 14340393
512:             512,        0,    244,     92,   656448
256:             256,        0,    239,   1111,  7152982
128:             128,        0,   1785,    375, 15426920
64:               64,        0,   4243,   1421,  8922809
32:               32,        0,   2519,    758, 4033195752
16:               16,        0,   1928,    305, 22113261
DP fakepg:        72,        0,      0,      0,        0
PV ENTRY:         24,   944530,   5540,  17080, 315699586
MAP ENTRY:        68,        0,    439,   1241, 32680232
KMAP ENTRY:       68,    32200,     40,    464,  5443857
MAP:             192,        0,      7,     33,        5
VM OBJECT:       132,        0,  32552,   1436, 14687110
128 Bucket:      524,        0,    966,      0,        0
64 Bucket:       268,        0,     30,     40,        0
32 Bucket:       140,        0,     30,     26,        0
16 Bucket:        76,        0,     15,     35,        0
UMA Hash:        128,        0,      2,     28,        0
UMA RCntSlab:    104,        0,    515,      3,        0
UMA Slabs:        64,        0,    793,     33,        0
UMA Zones:        88,        0,     73,      7,        0
UMA Kegs:        136,        0,     73,     23,        0

There doesn't seem to be anything really changed between then and now (i.e.
in one hour), but netstat works again. Can anyone recommend a suitable
course of actions? I'm somewhat at a loss as to what I should look for now.

It seems that when the machine has been up for a few days this starts
happening once or twice a day, until the machine finally deadlocks (it
usually does this after running for a week).

I tried to enable WITNESS, but as fas as I can gather from the dmesg, it
doesn't work as there is not enough buffer space or something (the error is
"witness_get: witness exhausted").

-Martin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20041216131119.A0DB5C2C2>