From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 6 17:36:26 2006 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 04DA716A422 for ; Mon, 6 Mar 2006 17:36:26 +0000 (GMT) (envelope-from me@carrollkong.com) Received: from mail.faerunconsulting.com (vzdsl-jcnj-216-182-31-61.static.tellurian.net [216.182.31.61]) by mx1.FreeBSD.org (Postfix) with SMTP id A19AC43D60 for ; Mon, 6 Mar 2006 17:35:58 +0000 (GMT) (envelope-from me@carrollkong.com) Received: (qmail 88771 invoked from network); 6 Mar 2006 17:35:57 -0000 Received: from unknown (HELO athena) (192.168.0.2) by dmz.faerunhome.com with SMTP; 6 Mar 2006 17:35:57 -0000 From: "Carroll Kong" To: Date: Mon, 6 Mar 2006 12:35:57 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1506 Thread-Index: AcY8CbIWzZo0izQhTH+KVM6Em5BJowFNaAQA In-Reply-To: Message-Id: <20060306173558.A19AC43D60@mx1.FreeBSD.org> Cc: Subject: RE: FreeBSD 4.11 P13 Crash X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2006 17:36:26 -0000 Well bad news. It happened again even with a new CPU and new = PowerSupply. However, the good news is that it seems to be saving the core dumps a = bit more consistently now. I swapped the motherboard back to the old one. Honestly, I've had similar core dumps in either case, I'm starting to = think it isn't the hardware but more of a software or software configuration = issue (one that works but is apparently not stable). I can swap the = motherboard back to the new one, but I got the same error in either case. It sure looks a lot like the other backtraces, but I suppose if = something corrupted the data in memory, it could still be anything. > So far I am thinking > - IPFilter intermittent bug with some packets, but I run a=20 > box with 112 days of uptime with the same version of=20 > IPFilter, albeit not with 4 NICs. It keeps failing around the net or ppp process. It might not mean = anything though since the box will always log 'junk'. Although, I find it odd = that it has never crashed when using the other, cable link, ONLY on the PPP process. Only an experienced hacker can tell me if my hypothesis is = correct in pointing the finger closer to PPP. Then again, there is no 'process' = to refer to if the issue was on the cable link! Maybe a bug in handling 4 = FXP Intel cards? I already swapped one of them (the other 2 are built = onboard, and the last one I have not swapped out yet that connects to the cable link). > - 3Ware driver is flakey, but I have a 4.10 box with 3Ware=20 > that is somewhat stable > - CPU (I would tend to think this would result in HARD lock=20 > ups vs Fatal Trap 12s though) New CPU didn't fix it, so let's scratch that out. In fact, it's been = pulled from a working system. > - PowerSupply (I suppose anything is possible, please note it=20 > is on an APC UPS, but the power supply might be delivering bad juice?) New Power supply. Antec 380 or so. I don't have a method of testing it = so it 'could' be a bad new power supply but honestly, I would expect it to = have crashed with a different error. > - Harddisks and 3Ware driver have incompatible firmware=20 > issue, I doubt this is it though since I purchased new=20 > Seagates in 9/2004 for the RAID1, then I added another=20 > Seagate as a JBOD, and that disk is not being written to=20 > during the crash. This is still a possibility, although it seems to fail in a memory operation. Unless the 3Ware somehow corrupted memory ahead of time, = which seems kind of odd but possible. Someone suggested I go back to 4.9. I don't mind doing so although I = wonder if I would be vulnerable to certain security issues. Furthermore, I'm = not sure how to do this right. I would guess cvsup with 4.9_RELEASE tag? Anyway, I am going to try to go back to 4.9, but wanted to throw some = more information to the list to see if anyone had any other ideas. The only hardware I have not changed so far is - cdrom - floppy - case :) Here is another backtrace. It seems to be doing the standard log to the ipf.log file thing, then die during an mbuf operation. Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x41f59 fault code =3D supervisor read, page not present instruction pointer =3D 0x8:0xc0192696 stack pointer =3D 0x10:0xd71cbbd0 frame pointer =3D 0x10:0xd71cbbd8 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 105 (ppp) interrupt mask =3D net tty=20 trap number =3D 12 panic: page fault --------------- #0 dumpsys () at ../../kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0173f3f in boot (howto=3D256) at ../../kern/kern_shutdown.c:316 #2 0xc0174364 in poweroff_wait (junk=3D0xc02f2aac, howto=3D-1070651985) = at ../../kern/kern_shutdown.c:595 #3 0xc02a77ba in trap_fatal (frame=3D0xd71cbb90, eva=3D270169) at ../../i386/i386/trap.c:974 #4 0xc02a748d in trap_pfault (frame=3D0xd71cbb90, usermode=3D0, = eva=3D270169) at ../../i386/i386/trap.c:867 #5 0xc02a704b in trap (frame=3D{tf_fs =3D -1072562160, tf_es =3D 16, = tf_ds =3D -686030832, tf_edi =3D 6757530, tf_esi =3D -1055310592,=20 tf_ebp =3D -685982760, tf_isp =3D -685982788, tf_ebx =3D 270169, = tf_edx =3D 6757530, tf_ecx =3D -1055840256, tf_eax =3D -28864,=20 tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1072093546, tf_cs =3D = 8, tf_eflags =3D 66054, tf_esp =3D -1055310592, tf_ss =3D -1055310592}) at ../../i386/i386/trap.c:466 #6 0xc0192696 in m_tag_delete_chain (m=3D0xc1193d00, t=3D0x0) at ../../kern/uipc_mbuf2.c:358 #7 0xc01904d3 in m_free (m=3D0xc1193d00) at ../../kern/uipc_mbuf.c:734 #8 0xc0190606 in m_freem (m=3D0xc117ef00) at ../../kern/uipc_mbuf.c:763 #9 0xc0127df8 in fr_check (ip=3D0xc117ef30, hlen=3D20, = ifp=3D0xc21a9508, out=3D0, mp=3D0xd71cbce8) at ../../contrib/ipfilter/netinet/fil.c:1387 #10 0xc01d7d06 in ip_input (m=3D0xc117ef00) at = ../../netinet/ip_input.c:478 #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 #12 0xc0299ee9 in swi_net_next () #13 0xc016e7c8 in lockmgr (lkp=3D0xc21a9600, flags=3D16973826, interlkp=3D0xd71c162c, p=3D0xd48bd5a0) at ../../kern/kern_lock.c:355 #14 0xc019fda8 in vop_stdlock (ap=3D0xd71cbdd0) at ../../kern/vfs_default.c:256 #15 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbdd0) at ../../ufs/ufs/ufs_vnops.c:2394 #16 0xc01a9ffd in vn_lock (vp=3D0xd71c15c0, flags=3D131074, = p=3D0xd48bd5a0) at vnode_if.h:861 #17 0xc01ad842 in spec_write (ap=3D0xd71cbe64) at ../../miscfs/specfs/spec_vnops.c:284 #18 0xc025a3ac in ufsspec_write (ap=3D0xd71cbe64) at ../../ufs/ufs/ufs_vnops.c:1827 #19 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbe64) at ../../ufs/ufs/ufs_vnops.c:2394 #20 0xc01a9b9a in vn_write (fp=3D0xc21a8c40, uio=3D0xd71cbed4, = cred=3D0xc219b780, flags=3D0, p=3D0xd48bd5a0) at vnode_if.h:363 #21 0xc018330d in dofilewrite (p=3D0xd48bd5a0, fp=3D0xc21a8c40, fd=3D9, buf=3D0xbfbfe89c, nbyte=3D213, offset=3D-1, flags=3D0) at ../../sys/file.h:163 #22 0xc01831c6 in write (p=3D0xd48bd5a0, uap=3D0xd71cbf80) at ../../kern/sys_generic.c:329 #23 0xc02a7a69 in syscall2 (frame=3D{tf_fs =3D 134938671, tf_es =3D 47, = tf_ds =3D -1078001617, tf_edi =3D 134996736, tf_esi =3D 213,=20 tf_ebp =3D -1077940064, tf_isp =3D -685981740, tf_ebx =3D = -1077942112, tf_edx =3D 0, tf_ecx =3D 13, tf_eax =3D 4, tf_trapno =3D 7,=20 tf_err =3D 2, tf_eip =3D 673683504, tf_cs =3D 31, tf_eflags =3D = 663, tf_esp =3D -1077942172, tf_ss =3D 47}) at ../../i386/i386/trap.c:1175 #24 0xc0298a85 in Xint0x80_syscall () #25 0x80655de in ?? () #26 0x806c2fb in ?? () #27 0x806c21d in ?? () #28 0x807470a in ?? () #29 0x8083a78 in ?? () #30 0x805b84b in ?? () #31 0x804d484 in ?? () #32 0x806ed77 in ?? () #33 0x806e967 in ?? () #34 0x804b62a in ?? () (kgdb) quit daemon# nm /kernel | grep c0192696 daemon# nm /kernel | grep c019269=20 daemon# nm /kernel | grep c01926=20 c01926e8 T m_tag_copy c0192658 T m_tag_delete c0192674 T m_tag_delete_chain c0192600 T m_tag_free c01926ac T m_tag_locate c0192614 T m_tag_prepend c0192628 T m_tag_unlink > -----Original Message----- > From: Carroll Kong=20 > Sent: Monday, February 27, 2006 8:53 PM > To: 'hackers@freebsd.org' > Subject: FreeBSD 4.11 P13 Crash >=20 > Okay this time my kernel was recompiled so there are no=20 > modules to make it easier to see all of the symbols. >=20 > Sometimes the box cycles through the fatal traps 12. Other=20 > times it does not. Based on my other Fatal trap errors, it=20 > seems to interrupt more often with the m_tag_delete function.=20 > I don't think this necessarily means the problem is with=20 > IPFilter or PPP mostly because this box acts as a firewall=20 > and logs constantly. Therefore, it is not surprising it=20 > always fails after logging with IPFilter, but I am always=20 > open to the possibility. >=20 > This box was stable before I upgraded from 4.9->4.11. Among=20 > one of the software changes was probably the change of=20 > IPFilter. I used to use the IPFilter 3.4.33pre modules, but=20 > after I moved to 4.11 I just used the distribution packaged=20 > 3.4.35. This might be the source of the problem, but I could=20 > not google for relevant entries. >=20 > I have since swapped the RAM, motherboard, RAM again (I=20 > bought another stick thinking maybe my new RAM was=20 > coincidentally bugged), one of the Intel NICs, and my 3Ware=20 > controller. The problem still occurred and actually more=20 > frequently. The usual frequency was about 14 days or so. It=20 > just crashed in less than 23 hours and then again within 25 minutes. >=20 > The final pieces of hardware that still can be swapped is the=20 > other Intel NIC (but this NIC is NOT connected to the PPPoE),=20 > CPU, Power Supply, CDROM (not used), Harddisks, or Case. :) >=20 > I tried disabling physical swap completely, and the system=20 > still crashed, so I doubt it is the 3Ware, but who knows. >=20 > So far I am thinking > - IPFilter intermittent bug with some packets, but I run a=20 > box with 112 days of uptime with the same version of=20 > IPFilter, albeit not with 4 NICs. > - 3Ware driver is flakey, but I have a 4.10 box with 3Ware=20 > that is somewhat stable > - CPU (I would tend to think this would result in HARD lock=20 > ups vs Fatal Trap 12s though) > - PowerSupply (I suppose anything is possible, please note it=20 > is on an APC UPS, but the power supply might be delivering bad juice?) > - Harddisks and 3Ware driver have incompatible firmware=20 > issue, I doubt this is it though since I purchased new=20 > Seagates in 9/2004 for the RAID1, then I added another=20 > Seagate as a JBOD, and that disk is not being written to=20 > during the crash. >=20 > I am tempted to consider upgrading to 5.X, but I am a=20 > conservative person and somehow doubt 4.X is the source of=20 > the problem as the system worked fine for over a year. >=20 > The box does a lot of things however I omitted this=20 > information to avoid flooding the list with too much=20 > information since it has worked fine for a year in the past. =20 > As a note, the problem is NOT load related. In fact, one=20 > time the fatal panic said the running process was "idle". :)=20 > Furthermore, I haven't really updated the software=20 > unnecessarily except for security issues and the system has=20 > been stable in the past with the same hardware and same=20 > software. I am very conservative when it comes to servers,=20 > so this seems like a hardware issue but I already swapped so=20 > much of it, I am beginning to wonder. >=20 > I am going to buy a new CPU and power supply as I have=20 > replaced nearly every other part by now. >=20 > I have included my dmesg, nm greps for the functions, a=20 > backtrace, uname output. I have the kernel dump so if there=20 > are any commands someone needs me to punch through, I will=20 > gladly do so. I included some of my own feeble debugging. I=20 > didn't like the line that said "address is out of bounds" in=20 > one of the mbuf structures. I am guessing that means the=20 > mbuf was already corrupted way before we got there. Any=20 > suggestions and advice are welcome. Thanks in advance! >=20 >=20 >=20 > Fatal trap 12: page fault while in kernel mode > fault virtual address =3D 0xc11e4402 > fault code =3D supervisor write, page not present > instruction pointer =3D 0x8:0xc018ffcf > stack pointer =3D 0x10:0xc02fa6f0 > frame pointer =3D 0x10:0xc02fa704 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D Idle > interrupt mask =3D net tty bio cam=20 > trap number =3D 12 > panic: page fault > Uptime: 6h5m10s > twe0: Cannot delete unit. error =3D 16 >=20 > Fatal trap 12: page fault while in kernel mode > fault virtual address =3D 0xc11e4402 > fault code =3D supervisor write, page not present > instruction pointer =3D 0x8:0xc018ffcf > stack pointer =3D 0x10:0xc02fa444 > frame pointer =3D 0x10:0xc02fa458 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D Idle > interrupt mask =3D net tty bio cam=20 > tx0, limit 0xfffff, type 0x1b >=20 > nm -n /kernel | grep c018f > c018f058 T accept_filt_del > c018f07c T accept_filt_get > c018f0b4 T accept_filt_generic_mod_event > c018f134 t net_init_domain > c018f1bc T net_add_domain > c018f1ec t domaininit > c018f244 T pffindtype > c018f290 T pffindproto > c018f304 T pfctlinput > c018f34c T pfctlinput2 > c018f3a4 t pfslowtimo > c018f3fc t pffasttimo > c018f45c t tunable_mbinit > c018f4ac t mbinit > c018f53c T m_mballoc > c018f5f8 T m_mballoc_wait > c018f7e8 T m_clalloc > c018f8b4 T m_clalloc_wait > c018f9a0 T m_retry > c018fa74 T m_retryhdr > c018fb60 t m_reclaim > c018fbb0 T m_get > c018fc54 T m_gethdr > c018fd0c T m_getclr > c018fdd0 T m_getcl >=20 > ------------------------------------------------------------------- >=20 > Fatal trap 12: page fault while in kernel mode > fault virtual address =3D 0x28067100 > fault code =3D supervisor read, page not present > instruction pointer =3D 0x8:0xc0192696 > stack pointer =3D 0x10:0xd71cbbd0 > frame pointer =3D 0x10:0xd71cbbd8 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 110 (ppp) > interrupt mask =3D net tty=20 > trap number =3D 12 > panic: page fault >=20 > syncing disks... 7 > done > Uptime: 25m51s > twe0: Cannot delete unit. error =3D 16 >=20 > dumping to dev #twed/0x20001, offset 3146240 dump 511 510 509=20 > 508 507 506 505 504 503 502 501 500 499 498 497 496 495 494=20 > 493 492 491 490 489 488 487 486 485 484 483 482 481 480 479=20 > 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464=20 > 463 462 461 460 459 458 457 456 455 454 453 452 451 450 449=20 > 448 447 446 445 444 443 442 441 440 439 438 437 436 435 434=20 > 433 432 431 430 429 428 427 426 425 424 423 422 421 420 419=20 > 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404=20 > 403 402 401 400 399 398 397 396 395 394 393 392 391 390 389=20 > 388 387 386 385 384 383 382 381 380 379 378 377 376 375 374=20 > 373 372 371 370 369 368 367 366 365 364 363 362 361 360 359=20 > 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344=20 > 343 342 341 340 339 338 337 336 335 334 333 332 331 330 329=20 > 328 327 326 325 324 323 322 321 320 319 318 317 316 315 314=20 > 313 312 311 310 309 308 307 306 305 304 303 302 301 300 299=20 > 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284=20 > 283 282 281 280 279 278 277 276 275 274 273 272 271 270 269=20 > 268 267 266 265 264 263 262 261 260 259 258 257 256 255 254=20 > 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239=20 > 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224=20 > 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209=20 > 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194=20 > 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179=20 > 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164=20 > 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149=20 > 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134=20 > 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119=20 > 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104=20 > 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85=20 > 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65=20 > 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45=20 > 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25=20 > 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2=20 > 1 0 succeeded Automatic reboot in 15 seconds - press a key on=20 > the console to abort Rebooting... >=20 > nm -n /kernel | grep c01926 > c0192600 T m_tag_free > c0192614 T m_tag_prepend > c0192628 T m_tag_unlink > c0192658 T m_tag_delete > c0192674 T m_tag_delete_chain > c01926ac T m_tag_locate > c01926e8 T m_tag_copy >=20 > --------------- >=20 > (kgdb) bt > #0 dumpsys () at ../../kern/kern_shutdown.c:487 > #1 0xc0173f3f in boot (howto=3D256) at ../../kern/kern_shutdown.c:316 > #2 0xc0174364 in poweroff_wait (junk=3D0xc02f2aac,=20 > howto=3D-1070651985) at ../../kern/kern_shutdown.c:595 > #3 0xc02a77ba in trap_fatal (frame=3D0xd71cbb90,=20 > eva=3D671510784) at ../../i386/i386/trap.c:974 > #4 0xc02a748d in trap_pfault (frame=3D0xd71cbb90, usermode=3D0,=20 > eva=3D671510784) at ../../i386/i386/trap.c:867 > #5 0xc02a704b in trap (frame=3D{tf_fs =3D -1072562160, tf_es =3D=20 > 16, tf_ds =3D -686030832, tf_edi =3D 6757530, tf_esi =3D -1055667456,=20 > tf_ebp =3D -685982760, tf_isp =3D -685982788, tf_ebx =3D=20 > 671510784, tf_edx =3D 6757530, tf_ecx =3D -1056411648, tf_eax =3D = 28672,=20 > tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1072093546, tf_cs=20 > =3D 8, tf_eflags =3D 66054, tf_esp =3D -1055667456, tf_ss =3D = -1055667456}) > at ../../i386/i386/trap.c:466 > #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at=20 > ../../kern/uipc_mbuf2.c:358 > #7 0xc01904d3 in m_free (m=3D0xc113cb00) at = ../../kern/uipc_mbuf.c:734 > #8 0xc0190606 in m_freem (m=3D0xc1094000) at = ../../kern/uipc_mbuf.c:763 > #9 0xc0127df8 in fr_check (ip=3D0xc1094030, hlen=3D20,=20 > ifp=3D0xc21a7008, out=3D0, mp=3D0xd71cbce8) > at ../../contrib/ipfilter/netinet/fil.c:1387 > #10 0xc01d7d06 in ip_input (m=3D0xc1094000) at=20 > ../../netinet/ip_input.c:478 > #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 > #12 0xc0299ee9 in swi_net_next () > #13 0xc016e7c8 in lockmgr (lkp=3D0xc21a7100, flags=3D16973826,=20 > interlkp=3D0xd71c0bac, p=3D0xd48bd5a0) at ../../kern/kern_lock.c:355 > #14 0xc019fda8 in vop_stdlock (ap=3D0xd71cbdd0) at=20 > ../../kern/vfs_default.c:256 > #15 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbdd0) at=20 > ../../ufs/ufs/ufs_vnops.c:2394 > #16 0xc01a9ffd in vn_lock (vp=3D0xd71c0b40, flags=3D131074,=20 > p=3D0xd48bd5a0) at vnode_if.h:861 > #17 0xc01ad842 in spec_write (ap=3D0xd71cbe64) at=20 > ../../miscfs/specfs/spec_vnops.c:284 > #18 0xc025a3ac in ufsspec_write (ap=3D0xd71cbe64) at=20 > ../../ufs/ufs/ufs_vnops.c:1827 > #19 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbe64) at=20 > ../../ufs/ufs/ufs_vnops.c:2394 #20 0xc01a9b9a in vn_write=20 > (fp=3D0xc219b100, uio=3D0xd71cbed4, cred=3D0xc2197080, flags=3D0,=20 > p=3D0xd48bd5a0) at vnode_if.h:363 > #21 0xc018330d in dofilewrite (p=3D0xd48bd5a0, fp=3D0xc219b100,=20 > fd=3D9, buf=3D0xbfbfe89c, nbyte=3D580, offset=3D-1, flags=3D0) > at ../../sys/file.h:163 > #22 0xc01831c6 in write (p=3D0xd48bd5a0, uap=3D0xd71cbf80) at=20 > ../../kern/sys_generic.c:329 > #23 0xc02a7a69 in syscall2 (frame=3D{tf_fs =3D -1078001617, tf_es=20 > =3D 134938671, tf_ds =3D -1078001617, tf_edi =3D 135090176, tf_esi =3D = 580,=20 > tf_ebp =3D -1077940064, tf_isp =3D -685981740, tf_ebx =3D=20 > -1077942112, tf_edx =3D 0, tf_ecx =3D 13, tf_eax =3D 4, tf_trapno =3D = 7,=20 > tf_err =3D 2, tf_eip =3D 673683504, tf_cs =3D 31, tf_eflags =3D=20 > 663, tf_esp =3D -1077942172, tf_ss =3D 47}) > at ../../i386/i386/trap.c:1175 > #24 0xc0298a85 in Xint0x80_syscall () > #25 0x80655de in ?? () > #26 0x806c2fb in ?? () > #27 0x806c21d in ?? () > #28 0x807470a in ?? () > #29 0x8083a78 in ?? () > #30 0x805b84b in ?? () > #31 0x804d484 in ?? () > #32 0x806ed77 in ?? () > #33 0x806e967 in ?? () > #34 0x804b62a in ?? () >=20 > (kgdb) f 11 > #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 > 971 ip_input(m); > (kgdb) print *m > $13 =3D {m_hdr =3D {mh_next =3D 0xc1128100, mh_nextpkt =3D 0x0,=20 > mh_data =3D 0xc1094030 "E", mh_len =3D 208, mh_type =3D 0, mh_flags = =3D 2}, > M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc21a7008, len =3D 576, = > header =3D 0x0, csum_flags =3D 0, csum_data =3D 0, tags =3D { > slh_first =3D 0x0}}, MH_dat =3D {MH_ext =3D {ext_buf =3D=20 > 0x2000000
, ext_free =3D 0x2400045,=20 > ext_size =3D 28454, ext_ref =3D 0x6d3d012e},=20 > MH_databuf =3D=20 > = "\000\000\000\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\0= 0 > = 1=BA\006\000\000\000\000E\000\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8= \001e'B > \rmy=BB6\216^\222@\002P\020=E1\000=B1=A1\000\000\000\000@\t\a\000\000\ > = 002r\000\003=C0\000\227=AB\212!\225@\204]\001\214\027=C1=F9\177\232=F9=E3= =F2 > = \222\016\000=F9au1=3D\216=DD\204=A8\207O\002+=F80=E9H=F0\eD\2056n\001=F7U= \025=E0 > = \222=FE\f:S=F5PI\037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\n?\217%x= &=B5\177=F4UqX\ > = 222\020\225\\=D1=D9~\fh=EA=A9\036\t\"Az\206=E1p=FE+}=C7=A3=A3=A2"...}},=20 > M_databuf =3D "\bp\032=C2@\002", '\000' ,=20 > = "\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\001=BA\006\00= 0\0 > = 00\000\000E\000\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8\001e'B\rmy=BB= 6\216^\ > 222@\002P\020=E1\000=B1=A1\000\000\000\000@\t\a\000\000\002r\000\003 > = =C0\000\227=AB\212!\225@\204]\001\214\027=C1=F9\177\232=F9=E3=F2\222\016\= 000 > = =F9au1=3D\216=DD\204=A8\207O\002+=F80=E9H=F0\eD\2056n\001=F7U\025=E0\222=FE= \f:S=F5PI > = \037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\n?\217%x&=B5\177=F4UqX\2= 22\020\225\ > \=D1=D9~\fh=EA=A9\036\t"...}} > (kgdb) f 6 > #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at=20 > ../../kern/uipc_mbuf2.c:358 > 358 m_tag_delete(m, q); > (kgdb) print *m > $14 =3D {m_hdr =3D {mh_next =3D 0xc113ca00, mh_nextpkt =3D=20 > 0x280ef4cd, mh_data =3D 0x14
,=20 > mh_len =3D 663,=20 > mh_type =3D 28672, mh_flags =3D 10246}, M_dat =3D {MH =3D=20 > {MH_pkthdr =3D {rcvif =3D 0x280ef492, len =3D 672120748, header =3D = 0x2,=20 > csum_flags =3D 16384, csum_data =3D 1, tags =3D {slh_first=20 > =3D 0x28067100}}, MH_dat =3D {MH_ext =3D { > ext_buf =3D 0x28067200
bounds>, ext_free =3D 0x280541fd, ext_size =3D 134516476,=20 > ext_ref =3D 0x280819da},=20 > MH_databuf =3D=20 > = "\000r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\000=A2A\005(=A8: > \006(@\200\006(\000\000\000\000\000\000\000\000`=FB=BF=BF@\200\006\0 > = 01\234=FB=BF=BFOA\005(=FC\216\004\b\004=CFe\000\000r\006(\001\000\000\00 > 0=A8:\006(\000p\006(=FC\216\004\b=FDA\005(=FC\216\004\b\t\013\005(\200 > = =E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?\005(5(\005(=A8:\006= ( > \f=FC=BF=BF=CF@\005(=20 > \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\0 > = 06(\000r\006(=DA>\005(=A8:\006(\000p\006(=FC\216\004\b=D0=FC=BF=BFS=D4\00= 4\b\2 > 00=E6\020(@ \005\b\000r\006("...}},=20 > M_databuf =3D=20 > "\222=F4\016(=AC=BF\017(\002\000\000\000\000@\000\000\001\000\000\00 > 0\000q\006(\000r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\00 > = 0=A2A\005(=A8:\006(@\200\006(\000\000\000\000\000\000\000\000`=FB=BF=BF@ > = \200\006\001\234=FB=BF=BFOA\005(=FC\216\004\b\004=CFe\000\000r\006(\001\ > 000\000\000=A8:\006(\000p\006(=FC\216\004\b=FDA\005(=FC\216\004\b\t\01 > = 3\005(\200=E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?\005(5(\0 > 05(=A8:\006(\f=FC=BF=BF=CF@\005(=20 > \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\0 > 06(\000r\006(=DA>\005(=A8:\006(\000p\006("...}} > (kgdb) print *q > Cannot access memory at address 0x0. > (kgdb) > ---------------------------- > Copyright (c) 1992-2005 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992,=20 > 1993, 1994 > The Regents of the University of California. All=20 > rights reserved. > FreeBSD 4.11-RELEASE-p13 #3: Thu Feb 23 13:09:31 EST 2006 > damascus@daemon.faerunhome.com:/usr/src/sys/compile/DAEMON > Timecounter "i8254" frequency 1193182 Hz > CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz (1993.54-MHz 686-class CPU) > Origin =3D "GenuineIntel" Id =3D 0xf24 Stepping =3D 4 > =20 > Features=3D0x3febfbff P,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SS > E2,SS,HTT,TM> > real memory =3D 536608768 (524032K bytes) avail memory =3D=20 > 518377472 (506228K bytes) Preloaded elf kernel "kernel" at 0xc03af000. > Warning: Pentium 4 CPU: PSE disabled > Pentium Pro MTRR support enabled > md0: Malloc disk > Using $PIR table, 11 entries at 0xc00f28c0 > npx0: on motherboard > npx0: INT 16 interface > pcib0: on motherboard > pci0: on pcib0 > pcib1: at device 1.0 on pci0 > pci1: on pcib1 > pcib2: at device=20 > 30.0 on pci0 > pci2: on pcib2 > twe0: <3ware Storage Controller driver ver. 1.40.01.002> port=20 > 0xdfa0-0xdfaf mem 0xfe000000-0xfe7fffff,0xfeafec00-0xfeafec0f=20 > irq 9 at device 9.0 on pci2 > twe0: 4 ports, Firmware FE7X 1.05.00.068, BIOS BE7X 1.08.00.048 > fxp0: port 0xdf00-0xdf3f mem=20 > 0xfeaa0000-0xfeabffff,0xfeafd000-0xfeafdfff irq 11 at device=20 > 10.0 on pci2 > fxp0: Ethernet address 00:02:b3:d0:e3:73 > inphy0: on miibus0 > inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > fxp1: port 0xde80-0xdebf mem=20 > 0xfea80000-0xfea9ffff,0xfeafc000-0xfeafcfff irq 10 at device=20 > 11.0 on pci2 > fxp1: Ethernet address 00:02:b3:ee:65:88 > inphy1: on miibus1 > inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > fxp2: port 0xdd80-0xddbf mem=20 > 0xfea40000-0xfea5ffff,0xfeafb000-0xfeafbfff irq 11 at device=20 > 12.0 on pci2 > fxp2: Ethernet address 00:11:11:c1:a2:e5 > inphy2: on miibus2 > inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > fxp3: port 0xdd00-0xdd3f mem=20 > 0xfea20000-0xfea3ffff,0xfeafa000-0xfeafafff irq 11 at device=20 > 13.0 on pci2 > fxp3: Ethernet address 00:11:11:c1:a2:e7 > inphy3: on miibus3 > inphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > atapci0: port=20 > 0xdcc0-0xdcff,0xdfe0-0xdfe3,0xdf98-0xdf9f,0xdfe4-0xdfe7,0xdff0 > -0xdff7 mem 0xfe9e0000-0xfe9fffff irq 11 at device 14.0 on pci2 > ata2: at 0xdff0 on atapci0 > ata3: at 0xdf98 on atapci0 > pci2: at 15.0 irq 11 > isab0: at device=20 > 31.0 on pci0 > isa0: on isab0 > atapci1: port 0xffa0-0xffaf at=20 > device 31.1 on pci0 > ata0: at 0x1f0 irq 14 on atapci1 > ata1: at 0x170 irq 15 on atapci1 > uhci0: port=20 > 0xef40-0xef5f irq 11 at device 31.2 on pci0 > usb0: on uhci0 > usb0: USB revision 1.0 > uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub0: 2 ports with 2 removable, self powered > pci0: (vendor=3D0x8086, dev=3D0x2443) at 31.3 irq 11 > uhci1: port=20 > 0xef80-0xef9f irq 10 at device 31.4 on pci0 > usb1: on uhci1 > usb1: USB revision 1.0 > uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub1: 2 ports with 2 removable, self powered > orm0: