From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 25 17:34:20 2006 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6AA2916A424 for ; Sat, 25 Mar 2006 17:34:20 +0000 (UTC) (envelope-from me@carrollkong.com) Received: from mail.faerunconsulting.com (vzdsl-jcnj-216-182-31-61.static.tellurian.net [216.182.31.61]) by mx1.FreeBSD.org (Postfix) with SMTP id 0817743D49 for ; Sat, 25 Mar 2006 17:34:18 +0000 (GMT) (envelope-from me@carrollkong.com) Received: (qmail 13238 invoked from network); 25 Mar 2006 17:34:17 -0000 Received: from unknown (HELO athena) (192.168.0.2) by dmz.faerunhome.com with SMTP; 25 Mar 2006 17:34:17 -0000 From: "Carroll Kong" To: Date: Sat, 25 Mar 2006 12:34:17 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1506 In-Reply-To: <20060306173558.A19AC43D60@mx1.FreeBSD.org> Thread-Index: AcY8CbIWzZo0izQhTH+KVM6Em5BJowFNaAQAA7yDCFA= Message-Id: <20060325173418.0817743D49@mx1.FreeBSD.org> Cc: Subject: RE: FreeBSD 4.11 P13 Crash X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 17:34:20 -0000 I do not want to jinx myself, but after back revving to FreeBSD 4.9 + patches, the system has been up for nearly 19 days. It just seems odd = that when I moved to 4.11, it worked for maybe 14 days to 30 days without a crash. Then the crashing would randomly occur more often until it got = to a point where it could not make it past the week without a reboot. I = might have upgrading IPFilter in that time frame, but I did not keep track of = that change unfortunately. So, this is a strong indication of a software bug. If I had to guess, I think it is related to IPFilter in conjunction with 4 Intel nics and/or PPPoE. I run the same version of IPfilter on another box with only 2 = nics and it has 100 day+ uptimes. I do not mind staying with 4.9 though and = I suppose interest in resolving some ancient bug that occurs in such a specific setup is pretty low. I'm just glad I can rely on my FreeBSD = box again! I'll keep you guys posted if it makes it past the one month marker and hopefully reach back to the old day days of 110+ day uptimes :) Thanks again guys for your help! - Carroll Kong=20 > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org=20 > [mailto:owner-freebsd-hackers@freebsd.org] On Behalf Of Carroll Kong > Sent: Monday, March 06, 2006 12:36 PM > To: hackers@freebsd.org > Subject: RE: FreeBSD 4.11 P13 Crash >=20 > Well bad news. It happened again even with a new CPU and new=20 > PowerSupply. > However, the good news is that it seems to be saving the core=20 > dumps a bit more consistently now. I swapped the motherboard=20 > back to the old one. > Honestly, I've had similar core dumps in either case, I'm=20 > starting to think it isn't the hardware but more of a=20 > software or software configuration issue (one that works but=20 > is apparently not stable). I can swap the motherboard back=20 > to the new one, but I got the same error in either case. >=20 > It sure looks a lot like the other backtraces, but I suppose=20 > if something corrupted the data in memory, it could still be anything. >=20 > > So far I am thinking > > - IPFilter intermittent bug with some packets, but I run a box with=20 > > 112 days of uptime with the same version of IPFilter,=20 > albeit not with=20 > > 4 NICs. >=20 > It keeps failing around the net or ppp process. It might not=20 > mean anything though since the box will always log 'junk'. =20 > Although, I find it odd that it has never crashed when using=20 > the other, cable link, ONLY on the PPP process. Only an=20 > experienced hacker can tell me if my hypothesis is correct in=20 > pointing the finger closer to PPP. Then again, there is no=20 > 'process' to refer to if the issue was on the cable link! =20 > Maybe a bug in handling 4 FXP Intel cards? I already swapped=20 > one of them (the other 2 are built onboard, and the last one=20 > I have not swapped out yet that connects to the cable link). >=20 > > - 3Ware driver is flakey, but I have a 4.10 box with 3Ware that is=20 > > somewhat stable > > - CPU (I would tend to think this would result in HARD lock ups vs=20 > > Fatal Trap 12s though) >=20 > New CPU didn't fix it, so let's scratch that out. In fact,=20 > it's been pulled from a working system. >=20 > > - PowerSupply (I suppose anything is possible, please note=20 > it is on an=20 > > APC UPS, but the power supply might be delivering bad juice?) >=20 > New Power supply. Antec 380 or so. I don't have a method of=20 > testing it so it 'could' be a bad new power supply but=20 > honestly, I would expect it to have crashed with a different error. >=20 > > - Harddisks and 3Ware driver have incompatible firmware=20 > issue, I doubt=20 > > this is it though since I purchased new Seagates in 9/2004 for the=20 > > RAID1, then I added another Seagate as a JBOD, and that disk is not=20 > > being written to during the crash. >=20 > This is still a possibility, although it seems to fail in a=20 > memory operation. Unless the 3Ware somehow corrupted memory=20 > ahead of time, which seems kind of odd but possible. >=20 > Someone suggested I go back to 4.9. I don't mind doing so=20 > although I wonder if I would be vulnerable to certain=20 > security issues. Furthermore, I'm not sure how to do this=20 > right. I would guess cvsup with 4.9_RELEASE tag? >=20 > Anyway, I am going to try to go back to 4.9, but wanted to=20 > throw some more information to the list to see if anyone had=20 > any other ideas. >=20 > The only hardware I have not changed so far is > - cdrom > - floppy > - case >=20 > :) >=20 > Here is another backtrace. It seems to be doing the standard=20 > log to the ipf.log file thing, then die during an mbuf operation. >=20 > Fatal trap 12: page fault while in kernel mode > fault virtual address =3D 0x41f59 > fault code =3D supervisor read, page not present > instruction pointer =3D 0x8:0xc0192696 > stack pointer =3D 0x10:0xd71cbbd0 > frame pointer =3D 0x10:0xd71cbbd8 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 105 (ppp) > interrupt mask =3D net tty=20 > trap number =3D 12 > panic: page fault >=20 > --------------- >=20 > #0 dumpsys () at ../../kern/kern_shutdown.c:487 > 487 if (dumping++) { > (kgdb) bt > #0 dumpsys () at ../../kern/kern_shutdown.c:487 > #1 0xc0173f3f in boot (howto=3D256) at ../../kern/kern_shutdown.c:316 > #2 0xc0174364 in poweroff_wait (junk=3D0xc02f2aac,=20 > howto=3D-1070651985) at > ../../kern/kern_shutdown.c:595 > #3 0xc02a77ba in trap_fatal (frame=3D0xd71cbb90, eva=3D270169) at > ../../i386/i386/trap.c:974 > #4 0xc02a748d in trap_pfault (frame=3D0xd71cbb90, usermode=3D0,=20 > eva=3D270169) at > ../../i386/i386/trap.c:867 > #5 0xc02a704b in trap (frame=3D{tf_fs =3D -1072562160, tf_es =3D=20 > 16, tf_ds =3D -686030832, tf_edi =3D 6757530, tf_esi =3D -1055310592,=20 > tf_ebp =3D -685982760, tf_isp =3D -685982788, tf_ebx =3D=20 > 270169, tf_edx =3D 6757530, tf_ecx =3D -1055840256, tf_eax =3D -28864, = > tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1072093546, tf_cs=20 > =3D 8, tf_eflags =3D 66054, tf_esp =3D -1055310592, tf_ss =3D = -1055310592}) > at ../../i386/i386/trap.c:466 > #6 0xc0192696 in m_tag_delete_chain (m=3D0xc1193d00, t=3D0x0) at > ../../kern/uipc_mbuf2.c:358 > #7 0xc01904d3 in m_free (m=3D0xc1193d00) at = ../../kern/uipc_mbuf.c:734 > #8 0xc0190606 in m_freem (m=3D0xc117ef00) at = ../../kern/uipc_mbuf.c:763 > #9 0xc0127df8 in fr_check (ip=3D0xc117ef30, hlen=3D20,=20 > ifp=3D0xc21a9508, out=3D0, > mp=3D0xd71cbce8) > at ../../contrib/ipfilter/netinet/fil.c:1387 > #10 0xc01d7d06 in ip_input (m=3D0xc117ef00) at=20 > ../../netinet/ip_input.c:478 > #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 > #12 0xc0299ee9 in swi_net_next () > #13 0xc016e7c8 in lockmgr (lkp=3D0xc21a9600, flags=3D16973826,=20 > interlkp=3D0xd71c162c, p=3D0xd48bd5a0) at ../../kern/kern_lock.c:355 > #14 0xc019fda8 in vop_stdlock (ap=3D0xd71cbdd0) at > ../../kern/vfs_default.c:256 > #15 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbdd0) at > ../../ufs/ufs/ufs_vnops.c:2394 > #16 0xc01a9ffd in vn_lock (vp=3D0xd71c15c0, flags=3D131074,=20 > p=3D0xd48bd5a0) at > vnode_if.h:861 > #17 0xc01ad842 in spec_write (ap=3D0xd71cbe64) at > ../../miscfs/specfs/spec_vnops.c:284 > #18 0xc025a3ac in ufsspec_write (ap=3D0xd71cbe64) at > ../../ufs/ufs/ufs_vnops.c:1827 > #19 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbe64) at > ../../ufs/ufs/ufs_vnops.c:2394 > #20 0xc01a9b9a in vn_write (fp=3D0xc21a8c40, uio=3D0xd71cbed4,=20 > cred=3D0xc219b780, flags=3D0, p=3D0xd48bd5a0) at vnode_if.h:363 > #21 0xc018330d in dofilewrite (p=3D0xd48bd5a0, fp=3D0xc21a8c40,=20 > fd=3D9, buf=3D0xbfbfe89c, nbyte=3D213, offset=3D-1, flags=3D0) > at ../../sys/file.h:163 > #22 0xc01831c6 in write (p=3D0xd48bd5a0, uap=3D0xd71cbf80) at > ../../kern/sys_generic.c:329 > #23 0xc02a7a69 in syscall2 (frame=3D{tf_fs =3D 134938671, tf_es =3D=20 > 47, tf_ds =3D -1078001617, tf_edi =3D 134996736, tf_esi =3D 213,=20 > tf_ebp =3D -1077940064, tf_isp =3D -685981740, tf_ebx =3D=20 > -1077942112, tf_edx =3D 0, tf_ecx =3D 13, tf_eax =3D 4, tf_trapno =3D = 7,=20 > tf_err =3D 2, tf_eip =3D 673683504, tf_cs =3D 31, tf_eflags =3D=20 > 663, tf_esp =3D -1077942172, tf_ss =3D 47}) > at ../../i386/i386/trap.c:1175 > #24 0xc0298a85 in Xint0x80_syscall () > #25 0x80655de in ?? () > #26 0x806c2fb in ?? () > #27 0x806c21d in ?? () > #28 0x807470a in ?? () > #29 0x8083a78 in ?? () > #30 0x805b84b in ?? () > #31 0x804d484 in ?? () > #32 0x806ed77 in ?? () > #33 0x806e967 in ?? () > #34 0x804b62a in ?? () > (kgdb) quit > daemon# nm /kernel | grep c0192696 > daemon# nm /kernel | grep c019269 > daemon# nm /kernel | grep c01926 > c01926e8 T m_tag_copy > c0192658 T m_tag_delete > c0192674 T m_tag_delete_chain > c0192600 T m_tag_free > c01926ac T m_tag_locate > c0192614 T m_tag_prepend > c0192628 T m_tag_unlink >=20 > > -----Original Message----- > > From: Carroll Kong > > Sent: Monday, February 27, 2006 8:53 PM > > To: 'hackers@freebsd.org' > > Subject: FreeBSD 4.11 P13 Crash > >=20 > > Okay this time my kernel was recompiled so there are no modules to=20 > > make it easier to see all of the symbols. > >=20 > > Sometimes the box cycles through the fatal traps 12. Other=20 > times it=20 > > does not. Based on my other Fatal trap errors, it seems to=20 > interrupt=20 > > more often with the m_tag_delete function. > > I don't think this necessarily means the problem is with=20 > IPFilter or=20 > > PPP mostly because this box acts as a firewall and logs=20 > constantly. =20 > > Therefore, it is not surprising it always fails after logging with=20 > > IPFilter, but I am always open to the possibility. > >=20 > > This box was stable before I upgraded from 4.9->4.11. Among one of=20 > > the software changes was probably the change of IPFilter. =20 > I used to=20 > > use the IPFilter 3.4.33pre modules, but after I moved to=20 > 4.11 I just=20 > > used the distribution packaged 3.4.35. This might be the source of=20 > > the problem, but I could not google for relevant entries. > >=20 > > I have since swapped the RAM, motherboard, RAM again (I=20 > bought another=20 > > stick thinking maybe my new RAM was coincidentally bugged),=20 > one of the=20 > > Intel NICs, and my 3Ware controller. The problem still=20 > occurred and=20 > > actually more frequently. The usual frequency was about 14 days or=20 > > so. It just crashed in less than 23 hours and then again within 25=20 > > minutes. > >=20 > > The final pieces of hardware that still can be swapped is the other=20 > > Intel NIC (but this NIC is NOT connected to the PPPoE), CPU, Power=20 > > Supply, CDROM (not used), Harddisks, or Case. :) > >=20 > > I tried disabling physical swap completely, and the system still=20 > > crashed, so I doubt it is the 3Ware, but who knows. > >=20 > > So far I am thinking > > - IPFilter intermittent bug with some packets, but I run a box with=20 > > 112 days of uptime with the same version of IPFilter,=20 > albeit not with=20 > > 4 NICs. > > - 3Ware driver is flakey, but I have a 4.10 box with 3Ware that is=20 > > somewhat stable > > - CPU (I would tend to think this would result in HARD lock ups vs=20 > > Fatal Trap 12s though) > > - PowerSupply (I suppose anything is possible, please note=20 > it is on an=20 > > APC UPS, but the power supply might be delivering bad juice?) > > - Harddisks and 3Ware driver have incompatible firmware=20 > issue, I doubt=20 > > this is it though since I purchased new Seagates in 9/2004 for the=20 > > RAID1, then I added another Seagate as a JBOD, and that disk is not=20 > > being written to during the crash. > >=20 > > I am tempted to consider upgrading to 5.X, but I am a conservative=20 > > person and somehow doubt 4.X is the source of the problem as the=20 > > system worked fine for over a year. > >=20 > > The box does a lot of things however I omitted this information to=20 > > avoid flooding the list with too much information since it=20 > has worked=20 > > fine for a year in the past. > > As a note, the problem is NOT load related. In fact, one time the=20 > > fatal panic said the running process was "idle". :) =20 > Furthermore, I=20 > > haven't really updated the software unnecessarily except=20 > for security=20 > > issues and the system has been stable in the past with the same=20 > > hardware and same software. I am very conservative when it=20 > comes to=20 > > servers, so this seems like a hardware issue but I already=20 > swapped so=20 > > much of it, I am beginning to wonder. > >=20 > > I am going to buy a new CPU and power supply as I have=20 > replaced nearly=20 > > every other part by now. > >=20 > > I have included my dmesg, nm greps for the functions, a backtrace,=20 > > uname output. I have the kernel dump so if there are any commands=20 > > someone needs me to punch through, I will gladly do so. I included=20 > > some of my own feeble debugging. I didn't like the line that said=20 > > "address is out of bounds" in one of the mbuf structures. I am=20 > > guessing that means the mbuf was already corrupted way=20 > before we got=20 > > there. Any suggestions and advice are welcome. Thanks in advance! > >=20 > >=20 > >=20 > > Fatal trap 12: page fault while in kernel mode > > fault virtual address =3D 0xc11e4402 > > fault code =3D supervisor write, page not present > > instruction pointer =3D 0x8:0xc018ffcf > > stack pointer =3D 0x10:0xc02fa6f0 > > frame pointer =3D 0x10:0xc02fa704 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, def32 1, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D Idle > > interrupt mask =3D net tty bio cam=20 > > trap number =3D 12 > > panic: page fault > > Uptime: 6h5m10s > > twe0: Cannot delete unit. error =3D 16 > >=20 > > Fatal trap 12: page fault while in kernel mode > > fault virtual address =3D 0xc11e4402 > > fault code =3D supervisor write, page not present > > instruction pointer =3D 0x8:0xc018ffcf > > stack pointer =3D 0x10:0xc02fa444 > > frame pointer =3D 0x10:0xc02fa458 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, def32 1, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D Idle > > interrupt mask =3D net tty bio cam=20 > > tx0, limit 0xfffff, type 0x1b > >=20 > > nm -n /kernel | grep c018f > > c018f058 T accept_filt_del > > c018f07c T accept_filt_get > > c018f0b4 T accept_filt_generic_mod_event > > c018f134 t net_init_domain > > c018f1bc T net_add_domain > > c018f1ec t domaininit > > c018f244 T pffindtype > > c018f290 T pffindproto > > c018f304 T pfctlinput > > c018f34c T pfctlinput2 > > c018f3a4 t pfslowtimo > > c018f3fc t pffasttimo > > c018f45c t tunable_mbinit > > c018f4ac t mbinit > > c018f53c T m_mballoc > > c018f5f8 T m_mballoc_wait > > c018f7e8 T m_clalloc > > c018f8b4 T m_clalloc_wait > > c018f9a0 T m_retry > > c018fa74 T m_retryhdr > > c018fb60 t m_reclaim > > c018fbb0 T m_get > > c018fc54 T m_gethdr > > c018fd0c T m_getclr > > c018fdd0 T m_getcl > >=20 > > ------------------------------------------------------------------- > >=20 > > Fatal trap 12: page fault while in kernel mode > > fault virtual address =3D 0x28067100 > > fault code =3D supervisor read, page not present > > instruction pointer =3D 0x8:0xc0192696 > > stack pointer =3D 0x10:0xd71cbbd0 > > frame pointer =3D 0x10:0xd71cbbd8 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, def32 1, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D 110 (ppp) > > interrupt mask =3D net tty=20 > > trap number =3D 12 > > panic: page fault > >=20 > > syncing disks... 7 > > done > > Uptime: 25m51s > > twe0: Cannot delete unit. error =3D 16 > >=20 > > dumping to dev #twed/0x20001, offset 3146240 dump 511 510 509=20 > > 508 507 506 505 504 503 502 501 500 499 498 497 496 495 494=20 > > 493 492 491 490 489 488 487 486 485 484 483 482 481 480 479=20 > > 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464=20 > > 463 462 461 460 459 458 457 456 455 454 453 452 451 450 449=20 > > 448 447 446 445 444 443 442 441 440 439 438 437 436 435 434=20 > > 433 432 431 430 429 428 427 426 425 424 423 422 421 420 419=20 > > 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404=20 > > 403 402 401 400 399 398 397 396 395 394 393 392 391 390 389=20 > > 388 387 386 385 384 383 382 381 380 379 378 377 376 375 374=20 > > 373 372 371 370 369 368 367 366 365 364 363 362 361 360 359=20 > > 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344=20 > > 343 342 341 340 339 338 337 336 335 334 333 332 331 330 329=20 > > 328 327 326 325 324 323 322 321 320 319 318 317 316 315 314=20 > > 313 312 311 310 309 308 307 306 305 304 303 302 301 300 299=20 > > 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284=20 > > 283 282 281 280 279 278 277 276 275 274 273 272 271 270 269=20 > > 268 267 266 265 264 263 262 261 260 259 258 257 256 255 254=20 > > 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239=20 > > 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224=20 > > 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209=20 > > 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194=20 > > 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179=20 > > 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164=20 > > 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149=20 > > 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134=20 > > 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119=20 > > 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104=20 > > 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85=20 > > 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65=20 > > 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45=20 > > 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25=20 > > 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2=20 > > 1 0 succeeded Automatic reboot in 15 seconds - press a key on=20 > > the console to abort Rebooting... > >=20 > > nm -n /kernel | grep c01926 > > c0192600 T m_tag_free > > c0192614 T m_tag_prepend > > c0192628 T m_tag_unlink > > c0192658 T m_tag_delete > > c0192674 T m_tag_delete_chain > > c01926ac T m_tag_locate > > c01926e8 T m_tag_copy > >=20 > > --------------- > >=20 > > (kgdb) bt > > #0 dumpsys () at ../../kern/kern_shutdown.c:487 > > #1 0xc0173f3f in boot (howto=3D256) at = ../../kern/kern_shutdown.c:316 > > #2 0xc0174364 in poweroff_wait (junk=3D0xc02f2aac,=20 > > howto=3D-1070651985) at ../../kern/kern_shutdown.c:595 > > #3 0xc02a77ba in trap_fatal (frame=3D0xd71cbb90,=20 > > eva=3D671510784) at ../../i386/i386/trap.c:974 > > #4 0xc02a748d in trap_pfault (frame=3D0xd71cbb90, usermode=3D0,=20 > > eva=3D671510784) at ../../i386/i386/trap.c:867 > > #5 0xc02a704b in trap (frame=3D{tf_fs =3D -1072562160, tf_es =3D=20 > > 16, tf_ds =3D -686030832, tf_edi =3D 6757530, tf_esi =3D = -1055667456,=20 > > tf_ebp =3D -685982760, tf_isp =3D -685982788, tf_ebx =3D=20 > > 671510784, tf_edx =3D 6757530, tf_ecx =3D -1056411648, tf_eax =3D = 28672,=20 > > tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1072093546, tf_cs=20 > > =3D 8, tf_eflags =3D 66054, tf_esp =3D -1055667456, tf_ss =3D = -1055667456}) > > at ../../i386/i386/trap.c:466 > > #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at=20 > > ../../kern/uipc_mbuf2.c:358 > > #7 0xc01904d3 in m_free (m=3D0xc113cb00) at=20 > ../../kern/uipc_mbuf.c:734 > > #8 0xc0190606 in m_freem (m=3D0xc1094000) at=20 > ../../kern/uipc_mbuf.c:763 > > #9 0xc0127df8 in fr_check (ip=3D0xc1094030, hlen=3D20,=20 > > ifp=3D0xc21a7008, out=3D0, mp=3D0xd71cbce8) > > at ../../contrib/ipfilter/netinet/fil.c:1387 > > #10 0xc01d7d06 in ip_input (m=3D0xc1094000) at=20 > > ../../netinet/ip_input.c:478 > > #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 > > #12 0xc0299ee9 in swi_net_next () > > #13 0xc016e7c8 in lockmgr (lkp=3D0xc21a7100, flags=3D16973826,=20 > > interlkp=3D0xd71c0bac, p=3D0xd48bd5a0) at ../../kern/kern_lock.c:355 > > #14 0xc019fda8 in vop_stdlock (ap=3D0xd71cbdd0) at=20 > > ../../kern/vfs_default.c:256 > > #15 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbdd0) at=20 > > ../../ufs/ufs/ufs_vnops.c:2394 > > #16 0xc01a9ffd in vn_lock (vp=3D0xd71c0b40, flags=3D131074,=20 > > p=3D0xd48bd5a0) at vnode_if.h:861 > > #17 0xc01ad842 in spec_write (ap=3D0xd71cbe64) at=20 > > ../../miscfs/specfs/spec_vnops.c:284 > > #18 0xc025a3ac in ufsspec_write (ap=3D0xd71cbe64) at=20 > > ../../ufs/ufs/ufs_vnops.c:1827 > > #19 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbe64) at=20 > > ../../ufs/ufs/ufs_vnops.c:2394 #20 0xc01a9b9a in vn_write=20 > > (fp=3D0xc219b100, uio=3D0xd71cbed4, cred=3D0xc2197080, flags=3D0,=20 > > p=3D0xd48bd5a0) at vnode_if.h:363 > > #21 0xc018330d in dofilewrite (p=3D0xd48bd5a0, fp=3D0xc219b100,=20 > > fd=3D9, buf=3D0xbfbfe89c, nbyte=3D580, offset=3D-1, flags=3D0) > > at ../../sys/file.h:163 > > #22 0xc01831c6 in write (p=3D0xd48bd5a0, uap=3D0xd71cbf80) at=20 > > ../../kern/sys_generic.c:329 > > #23 0xc02a7a69 in syscall2 (frame=3D{tf_fs =3D -1078001617, tf_es=20 > > =3D 134938671, tf_ds =3D -1078001617, tf_edi =3D 135090176, tf_esi = =3D 580,=20 > > tf_ebp =3D -1077940064, tf_isp =3D -685981740, tf_ebx =3D=20 > > -1077942112, tf_edx =3D 0, tf_ecx =3D 13, tf_eax =3D 4, tf_trapno = =3D 7,=20 > > tf_err =3D 2, tf_eip =3D 673683504, tf_cs =3D 31, tf_eflags = =3D=20 > > 663, tf_esp =3D -1077942172, tf_ss =3D 47}) > > at ../../i386/i386/trap.c:1175 > > #24 0xc0298a85 in Xint0x80_syscall () > > #25 0x80655de in ?? () > > #26 0x806c2fb in ?? () > > #27 0x806c21d in ?? () > > #28 0x807470a in ?? () > > #29 0x8083a78 in ?? () > > #30 0x805b84b in ?? () > > #31 0x804d484 in ?? () > > #32 0x806ed77 in ?? () > > #33 0x806e967 in ?? () > > #34 0x804b62a in ?? () > >=20 > > (kgdb) f 11 > > #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 > > 971 ip_input(m); > > (kgdb) print *m > > $13 =3D {m_hdr =3D {mh_next =3D 0xc1128100, mh_nextpkt =3D 0x0,=20 > > mh_data =3D 0xc1094030 "E", mh_len =3D 208, mh_type =3D 0, mh_flags = =3D 2}, > > M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc21a7008, len =3D = 576,=20 > > header =3D 0x0, csum_flags =3D 0, csum_data =3D 0, tags =3D { > > slh_first =3D 0x0}}, MH_dat =3D {MH_ext =3D {ext_buf =3D=20 > > 0x2000000
, ext_free =3D 0x2400045, = > > ext_size =3D 28454, ext_ref =3D 0x6d3d012e},=20 > > MH_databuf =3D=20 > > = "\000\000\000\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\0= 0 > > = 1=BA\006\000\000\000\000E\000\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8= \001e'B > > = \rmy=BB6\216^\222@\002P\020=E1\000=B1=A1\000\000\000\000@\t\a\000\000\ > > = 002r\000\003=C0\000\227=AB\212!\225@\204]\001\214\027=C1=F9\177\232=F9=E3= =F2 > > = \222\016\000=F9au1=3D\216=DD\204=A8\207O\002+=F80=E9H=F0\eD\2056n\001=F7U= \025=E0 > > = \222=FE\f:S=F5PI\037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\n?\217%x= &=B5\177=F4UqX\ > > = 222\020\225\\=D1=D9~\fh=EA=A9\036\t\"Az\206=E1p=FE+}=C7=A3=A3=A2"...}},=20 > > M_databuf =3D "\bp\032=C2@\002", '\000' ,=20 > > = "\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\001=BA\006\00= 0\0 > > = 00\000\000E\000\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8\001e'B\rmy=BB= 6\216^\ > > 222@\002P\020=E1\000=B1=A1\000\000\000\000@\t\a\000\000\002r\000\003 > > = =C0\000\227=AB\212!\225@\204]\001\214\027=C1=F9\177\232=F9=E3=F2\222\016\= 000 > > = =F9au1=3D\216=DD\204=A8\207O\002+=F80=E9H=F0\eD\2056n\001=F7U\025=E0\222=FE= \f:S=F5PI > > = \037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\n?\217%x&=B5\177=F4UqX\2= 22\020\225\ > > \=D1=D9~\fh=EA=A9\036\t"...}} > > (kgdb) f 6 > > #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at=20 > > ../../kern/uipc_mbuf2.c:358 > > 358 m_tag_delete(m, q); > > (kgdb) print *m > > $14 =3D {m_hdr =3D {mh_next =3D 0xc113ca00, mh_nextpkt =3D=20 > > 0x280ef4cd, mh_data =3D 0x14
,=20 > > mh_len =3D 663,=20 > > mh_type =3D 28672, mh_flags =3D 10246}, M_dat =3D {MH =3D=20 > > {MH_pkthdr =3D {rcvif =3D 0x280ef492, len =3D 672120748, header =3D = 0x2,=20 > > csum_flags =3D 16384, csum_data =3D 1, tags =3D {slh_first=20 > > =3D 0x28067100}}, MH_dat =3D {MH_ext =3D { > > ext_buf =3D 0x28067200
> bounds>, ext_free =3D 0x280541fd, ext_size =3D 134516476,=20 > > ext_ref =3D 0x280819da},=20 > > MH_databuf =3D=20 > > = "\000r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\000=A2A\005(=A8: > > \006(@\200\006(\000\000\000\000\000\000\000\000`=FB=BF=BF@\200\006\0 > > = 01\234=FB=BF=BFOA\005(=FC\216\004\b\004=CFe\000\000r\006(\001\000\000\00 > > = 0=A8:\006(\000p\006(=FC\216\004\b=FDA\005(=FC\216\004\b\t\013\005(\200 > > = =E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?\005(5(\005(=A8:\006= ( > > \f=FC=BF=BF=CF@\005(=20 > > \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\0 > > = 06(\000r\006(=DA>\005(=A8:\006(\000p\006(=FC\216\004\b=D0=FC=BF=BFS=D4\00= 4\b\2 > > 00=E6\020(@ \005\b\000r\006("...}},=20 > > M_databuf =3D=20 > > "\222=F4\016(=AC=BF\017(\002\000\000\000\000@\000\000\001\000\000\00 > > 0\000q\006(\000r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\00 > > = 0=A2A\005(=A8:\006(@\200\006(\000\000\000\000\000\000\000\000`=FB=BF=BF@ > > = \200\006\001\234=FB=BF=BFOA\005(=FC\216\004\b\004=CFe\000\000r\006(\001\ > > = 000\000\000=A8:\006(\000p\006(=FC\216\004\b=FDA\005(=FC\216\004\b\t\01 > > = 3\005(\200=E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?\005(5(\0 > > 05(=A8:\006(\f=FC=BF=BF=CF@\005(=20 > > \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\0 > > 06(\000r\006(=DA>\005(=A8:\006(\000p\006("...}} > > (kgdb) print *q > > Cannot access memory at address 0x0. > > (kgdb) > > ---------------------------- > > Copyright (c) 1992-2005 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992,=20 > > 1993, 1994 > > The Regents of the University of California. All=20 > > rights reserved. > > FreeBSD 4.11-RELEASE-p13 #3: Thu Feb 23 13:09:31 EST 2006 > > damascus@daemon.faerunhome.com:/usr/src/sys/compile/DAEMON > > Timecounter "i8254" frequency 1193182 Hz > > CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz (1993.54-MHz 686-class CPU) > > Origin =3D "GenuineIntel" Id =3D 0xf24 Stepping =3D 4 > > =20 > > Features=3D0x3febfbff > P,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SS > > E2,SS,HTT,TM> > > real memory =3D 536608768 (524032K bytes) avail memory =3D=20 > > 518377472 (506228K bytes) Preloaded elf kernel "kernel" at=20 > 0xc03af000. > > Warning: Pentium 4 CPU: PSE disabled > > Pentium Pro MTRR support enabled > > md0: Malloc disk > > Using $PIR table, 11 entries at 0xc00f28c0 > > npx0: on motherboard > > npx0: INT 16 interface > > pcib0: on motherboard > > pci0: on pcib0 > > pcib1: at device 1.0 on pci0 > > pci1: on pcib1 > > pcib2: at device=20 > > 30.0 on pci0 > > pci2: on pcib2 > > twe0: <3ware Storage Controller driver ver. 1.40.01.002> port=20 > > 0xdfa0-0xdfaf mem 0xfe000000-0xfe7fffff,0xfeafec00-0xfeafec0f=20 > > irq 9 at device 9.0 on pci2 > > twe0: 4 ports, Firmware FE7X 1.05.00.068, BIOS BE7X 1.08.00.048 > > fxp0: port 0xdf00-0xdf3f mem=20 > > 0xfeaa0000-0xfeabffff,0xfeafd000-0xfeafdfff irq 11 at device=20 > > 10.0 on pci2 > > fxp0: Ethernet address 00:02:b3:d0:e3:73 > > inphy0: on miibus0 > > inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > > fxp1: port 0xde80-0xdebf mem=20 > > 0xfea80000-0xfea9ffff,0xfeafc000-0xfeafcfff irq 10 at device=20 > > 11.0 on pci2 > > fxp1: Ethernet address 00:02:b3:ee:65:88 > > inphy1: on miibus1 > > inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > > fxp2: port 0xdd80-0xddbf mem=20 > > 0xfea40000-0xfea5ffff,0xfeafb000-0xfeafbfff irq 11 at device=20 > > 12.0 on pci2 > > fxp2: Ethernet address 00:11:11:c1:a2:e5 > > inphy2: on miibus2 > > inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > > fxp3: port 0xdd00-0xdd3f mem=20 > > 0xfea20000-0xfea3ffff,0xfeafa000-0xfeafafff irq 11 at device=20 > > 13.0 on pci2 > > fxp3: Ethernet address 00:11:11:c1:a2:e7 > > inphy3: on miibus3 > > inphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > > atapci0: port=20 > > 0xdcc0-0xdcff,0xdfe0-0xdfe3,0xdf98-0xdf9f,0xdfe4-0xdfe7,0xdff0 > > -0xdff7 mem 0xfe9e0000-0xfe9fffff irq 11 at device 14.0 on pci2 > > ata2: at 0xdff0 on atapci0 > > ata3: at 0xdf98 on atapci0 > > pci2: at 15.0 irq 11 > > isab0: at device=20 > > 31.0 on pci0 > > isa0: on isab0 > > atapci1: port 0xffa0-0xffaf at=20 > > device 31.1 on pci0 > > ata0: at 0x1f0 irq 14 on atapci1 > > ata1: at 0x170 irq 15 on atapci1 > > uhci0: port=20 > > 0xef40-0xef5f irq 11 at device 31.2 on pci0 > > usb0: on uhci0 > > usb0: USB revision 1.0 > > uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > > uhub0: 2 ports with 2 removable, self powered > > pci0: (vendor=3D0x8086, dev=3D0x2443) at 31.3 irq 11 > > uhci1: port=20 > > 0xef80-0xef9f irq 10 at device 31.4 on pci0 > > usb1: on uhci1 > > usb1: USB revision 1.0 > > uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > > uhub1: 2 ports with 2 removable, self powered > > orm0: