From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 28 01:53:00 2006 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A65B16A420 for ; Tue, 28 Feb 2006 01:53:00 +0000 (GMT) (envelope-from me@carrollkong.com) Received: from mail.faerunconsulting.com (vzdsl-jcnj-216-182-31-61.static.tellurian.net [216.182.31.61]) by mx1.FreeBSD.org (Postfix) with SMTP id 2220543D48 for ; Tue, 28 Feb 2006 01:52:58 +0000 (GMT) (envelope-from me@carrollkong.com) Received: (qmail 88741 invoked from network); 28 Feb 2006 01:52:57 -0000 Received: from unknown (HELO athena) (192.168.0.2) by dmz.faerunhome.com with SMTP; 28 Feb 2006 01:52:57 -0000 From: "Carroll Kong" To: Date: Mon, 27 Feb 2006 20:52:57 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1506 Thread-Index: AcY8CbIWzZo0izQhTH+KVM6Em5BJow== Message-Id: <20060228015258.2220543D48@mx1.FreeBSD.org> Cc: Subject: FreeBSD 4.11 P13 Crash X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2006 01:53:01 -0000 Okay this time my kernel was recompiled so there are no modules to make = it easier to see all of the symbols. Sometimes the box cycles through the fatal traps 12. Other times it = does not. Based on my other Fatal trap errors, it seems to interrupt more = often with the m_tag_delete function. I don't think this necessarily means = the problem is with IPFilter or PPP mostly because this box acts as a = firewall and logs constantly. Therefore, it is not surprising it always fails = after logging with IPFilter, but I am always open to the possibility. This box was stable before I upgraded from 4.9->4.11. Among one of the software changes was probably the change of IPFilter. I used to use the IPFilter 3.4.33pre modules, but after I moved to 4.11 I just used the distribution packaged 3.4.35. This might be the source of the problem, = but I could not google for relevant entries. I have since swapped the RAM, motherboard, RAM again (I bought another = stick thinking maybe my new RAM was coincidentally bugged), one of the Intel = NICs, and my 3Ware controller. The problem still occurred and actually more frequently. The usual frequency was about 14 days or so. It just = crashed in less than 23 hours and then again within 25 minutes. The final pieces of hardware that still can be swapped is the other = Intel NIC (but this NIC is NOT connected to the PPPoE), CPU, Power Supply, = CDROM (not used), Harddisks, or Case. :) I tried disabling physical swap completely, and the system still = crashed, so I doubt it is the 3Ware, but who knows. So far I am thinking - IPFilter intermittent bug with some packets, but I run a box with 112 = days of uptime with the same version of IPFilter, albeit not with 4 NICs. - 3Ware driver is flakey, but I have a 4.10 box with 3Ware that is = somewhat stable - CPU (I would tend to think this would result in HARD lock ups vs Fatal Trap 12s though) - PowerSupply (I suppose anything is possible, please note it is on an = APC UPS, but the power supply might be delivering bad juice?) - Harddisks and 3Ware driver have incompatible firmware issue, I doubt = this is it though since I purchased new Seagates in 9/2004 for the RAID1, = then I added another Seagate as a JBOD, and that disk is not being written to during the crash. I am tempted to consider upgrading to 5.X, but I am a conservative = person and somehow doubt 4.X is the source of the problem as the system worked = fine for over a year. The box does a lot of things however I omitted this information to avoid flooding the list with too much information since it has worked fine for = a year in the past. As a note, the problem is NOT load related. In fact, = one time the fatal panic said the running process was "idle". :) = Furthermore, I haven't really updated the software unnecessarily except for security issues and the system has been stable in the past with the same hardware = and same software. I am very conservative when it comes to servers, so this seems like a hardware issue but I already swapped so much of it, I am beginning to wonder. I am going to buy a new CPU and power supply as I have replaced nearly = every other part by now. I have included my dmesg, nm greps for the functions, a backtrace, uname output. I have the kernel dump so if there are any commands someone = needs me to punch through, I will gladly do so. I included some of my own = feeble debugging. I didn't like the line that said "address is out of bounds" = in one of the mbuf structures. I am guessing that means the mbuf was = already corrupted way before we got there. Any suggestions and advice are = welcome. Thanks in advance! Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0xc11e4402 fault code =3D supervisor write, page not present instruction pointer =3D 0x8:0xc018ffcf stack pointer =3D 0x10:0xc02fa6f0 frame pointer =3D 0x10:0xc02fa704 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D Idle interrupt mask =3D net tty bio cam=20 trap number =3D 12 panic: page fault Uptime: 6h5m10s twe0: Cannot delete unit. error =3D 16 Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0xc11e4402 fault code =3D supervisor write, page not present instruction pointer =3D 0x8:0xc018ffcf stack pointer =3D 0x10:0xc02fa444 frame pointer =3D 0x10:0xc02fa458 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D Idle interrupt mask =3D net tty bio cam=20 tx0, limit 0xfffff, type 0x1b nm -n /kernel | grep c018f=20 c018f058 T accept_filt_del c018f07c T accept_filt_get c018f0b4 T accept_filt_generic_mod_event c018f134 t net_init_domain c018f1bc T net_add_domain c018f1ec t domaininit c018f244 T pffindtype c018f290 T pffindproto c018f304 T pfctlinput c018f34c T pfctlinput2 c018f3a4 t pfslowtimo c018f3fc t pffasttimo c018f45c t tunable_mbinit c018f4ac t mbinit c018f53c T m_mballoc c018f5f8 T m_mballoc_wait c018f7e8 T m_clalloc c018f8b4 T m_clalloc_wait c018f9a0 T m_retry c018fa74 T m_retryhdr c018fb60 t m_reclaim c018fbb0 T m_get c018fc54 T m_gethdr c018fd0c T m_getclr c018fdd0 T m_getcl ------------------------------------------------------------------- Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x28067100 fault code =3D supervisor read, page not present instruction pointer =3D 0x8:0xc0192696 stack pointer =3D 0x10:0xd71cbbd0 frame pointer =3D 0x10:0xd71cbbd8 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 110 (ppp) interrupt mask =3D net tty=20 trap number =3D 12 panic: page fault syncing disks... 7=20 done Uptime: 25m51s twe0: Cannot delete unit. error =3D 16 dumping to dev #twed/0x20001, offset 3146240 dump 511 510 509 508 507 506 505 504 503 502 501 500 499 498 497 496 495 = 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480 479 478 477 476 = 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460 459 458 457 = 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440 439 438 = 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420 419 = 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400 = 399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 = 380 379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 = 361 360 359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 = 342 341 340 339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 = 323 322 321 320 319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 = 304 303 302 301 300 299 298 297 296 295 294 293 292 291 290 289 288 287 286 = 285 284 283 282 281 280 279 278 277 276 275 274 273 272 271 270 269 268 267 = 266 265 264 263 262 261 260 259 258 257 256 255 254 253 252 251 250 249 248 = 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 = 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 = 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 = 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 = 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 = 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 = 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 = 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 = 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 = 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 = 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 = 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 succeeded Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... nm -n /kernel | grep c01926=20 c0192600 T m_tag_free c0192614 T m_tag_prepend c0192628 T m_tag_unlink c0192658 T m_tag_delete c0192674 T m_tag_delete_chain c01926ac T m_tag_locate c01926e8 T m_tag_copy --------------- (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0173f3f in boot (howto=3D256) at ../../kern/kern_shutdown.c:316 #2 0xc0174364 in poweroff_wait (junk=3D0xc02f2aac, howto=3D-1070651985) = at ../../kern/kern_shutdown.c:595 #3 0xc02a77ba in trap_fatal (frame=3D0xd71cbb90, eva=3D671510784) at ../../i386/i386/trap.c:974 #4 0xc02a748d in trap_pfault (frame=3D0xd71cbb90, usermode=3D0, = eva=3D671510784) at ../../i386/i386/trap.c:867 #5 0xc02a704b in trap (frame=3D{tf_fs =3D -1072562160, tf_es =3D 16, = tf_ds =3D -686030832, tf_edi =3D 6757530, tf_esi =3D -1055667456,=20 tf_ebp =3D -685982760, tf_isp =3D -685982788, tf_ebx =3D = 671510784, tf_edx =3D 6757530, tf_ecx =3D -1056411648, tf_eax =3D 28672,=20 tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1072093546, tf_cs =3D = 8, tf_eflags =3D 66054, tf_esp =3D -1055667456, tf_ss =3D -1055667456}) at ../../i386/i386/trap.c:466 #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at ../../kern/uipc_mbuf2.c:358 #7 0xc01904d3 in m_free (m=3D0xc113cb00) at ../../kern/uipc_mbuf.c:734 #8 0xc0190606 in m_freem (m=3D0xc1094000) at ../../kern/uipc_mbuf.c:763 #9 0xc0127df8 in fr_check (ip=3D0xc1094030, hlen=3D20, = ifp=3D0xc21a7008, out=3D0, mp=3D0xd71cbce8) at ../../contrib/ipfilter/netinet/fil.c:1387 #10 0xc01d7d06 in ip_input (m=3D0xc1094000) at = ../../netinet/ip_input.c:478 #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 #12 0xc0299ee9 in swi_net_next () #13 0xc016e7c8 in lockmgr (lkp=3D0xc21a7100, flags=3D16973826, interlkp=3D0xd71c0bac, p=3D0xd48bd5a0) at ../../kern/kern_lock.c:355 #14 0xc019fda8 in vop_stdlock (ap=3D0xd71cbdd0) at ../../kern/vfs_default.c:256 #15 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbdd0) at ../../ufs/ufs/ufs_vnops.c:2394 #16 0xc01a9ffd in vn_lock (vp=3D0xd71c0b40, flags=3D131074, = p=3D0xd48bd5a0) at vnode_if.h:861 #17 0xc01ad842 in spec_write (ap=3D0xd71cbe64) at ../../miscfs/specfs/spec_vnops.c:284 #18 0xc025a3ac in ufsspec_write (ap=3D0xd71cbe64) at ../../ufs/ufs/ufs_vnops.c:1827 #19 0xc025a9c9 in ufs_vnoperatespec (ap=3D0xd71cbe64) at ../../ufs/ufs/ufs_vnops.c:2394 #20 0xc01a9b9a in vn_write (fp=3D0xc219b100, uio=3D0xd71cbed4, = cred=3D0xc2197080, flags=3D0, p=3D0xd48bd5a0) at vnode_if.h:363 #21 0xc018330d in dofilewrite (p=3D0xd48bd5a0, fp=3D0xc219b100, fd=3D9, buf=3D0xbfbfe89c, nbyte=3D580, offset=3D-1, flags=3D0) at ../../sys/file.h:163 #22 0xc01831c6 in write (p=3D0xd48bd5a0, uap=3D0xd71cbf80) at ../../kern/sys_generic.c:329 #23 0xc02a7a69 in syscall2 (frame=3D{tf_fs =3D -1078001617, tf_es =3D = 134938671, tf_ds =3D -1078001617, tf_edi =3D 135090176, tf_esi =3D 580,=20 tf_ebp =3D -1077940064, tf_isp =3D -685981740, tf_ebx =3D = -1077942112, tf_edx =3D 0, tf_ecx =3D 13, tf_eax =3D 4, tf_trapno =3D 7,=20 tf_err =3D 2, tf_eip =3D 673683504, tf_cs =3D 31, tf_eflags =3D = 663, tf_esp =3D -1077942172, tf_ss =3D 47}) at ../../i386/i386/trap.c:1175 #24 0xc0298a85 in Xint0x80_syscall () #25 0x80655de in ?? () #26 0x806c2fb in ?? () #27 0x806c21d in ?? () #28 0x807470a in ?? () #29 0x8083a78 in ?? () #30 0x805b84b in ?? () #31 0x804d484 in ?? () #32 0x806ed77 in ?? () #33 0x806e967 in ?? () #34 0x804b62a in ?? () (kgdb) f 11 #11 0xc01d838b in ipintr () at ../../netinet/ip_input.c:971 971 ip_input(m); (kgdb) print *m $13 =3D {m_hdr =3D {mh_next =3D 0xc1128100, mh_nextpkt =3D 0x0, mh_data = =3D 0xc1094030 "E", mh_len =3D 208, mh_type =3D 0, mh_flags =3D 2},=20 M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc21a7008, len =3D 576, = header =3D 0x0, csum_flags =3D 0, csum_data =3D 0, tags =3D { slh_first =3D 0x0}}, MH_dat =3D {MH_ext =3D {ext_buf =3D = 0x2000000
, ext_free =3D 0x2400045,=20 ext_size =3D 28454, ext_ref =3D 0x6d3d012e},=20 MH_databuf =3D "\000\000\000\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\0= 01=BA\006\000\000 \000\000E\000\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8\001e'B\rmy=BB= 6\216^\222@\002P\020=E1\0 00=B1=A1\000\000\000\000@\t\a\000\000\002r\000\003=C0\000\227=AB\212!\225= @\204]\001\ 214\027=C1=F9\177\232=F9=E3=F2\222\016\000=F9au1=3D\216=DD\204=A8\207O\00= 2+=F80=E9H=F0\eD\2056n\001=F7 U\025=E0\222=FE\f:S=F5PI\037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\= n?\217%x&=B5\177=F4UqX\222\020\ 225\\=D1=D9~\fh=EA=A9\036\t\"Az\206=E1p=FE+}=C7=A3=A3=A2"...}},=20 M_databuf =3D "\bp\032=C2@\002", '\000' , "\002E\000@\002&o\000\000.\001=3Dm;=BD=F30=D8=B6\037=3D\003\001=BA\006\00= 0\000\000\000E\00 0\005=C8=E9=BA@\000/\006=A2t=D8=B6\037=3D=C0=A8\001e'B\rmy=BB6\216^\222@\= 002P\020=E1\000=B1=A1\000\000 \000\000@\t\a\000\000\002r\000\003=C0\000\227=AB\212!\225@\204]\001\214\0= 27=C1=F9\17 7\232=F9=E3=F2\222\016\000=F9au1=3D\216=DD\204=A8\207O\002+=F80=E9H=F0\eD= \2056n\001=F7U\025=E0\222=FE\ f:S=F5PI\037)T=D8(=FD=A8\r=D3@\210=EA\217(S=F5c=E2=E9=B8\n?\217%x&=B5\177= =F4UqX\222\020\225\\=D1=D9~\fh=EA =A9\036\t"...}} (kgdb) f 6 #6 0xc0192696 in m_tag_delete_chain (m=3D0xc113cb00, t=3D0x0) at ../../kern/uipc_mbuf2.c:358 358 m_tag_delete(m, q); (kgdb) print *m $14 =3D {m_hdr =3D {mh_next =3D 0xc113ca00, mh_nextpkt =3D 0x280ef4cd, = mh_data =3D 0x14
, mh_len =3D 663,=20 mh_type =3D 28672, mh_flags =3D 10246}, M_dat =3D {MH =3D {MH_pkthdr = =3D {rcvif =3D 0x280ef492, len =3D 672120748, header =3D 0x2,=20 csum_flags =3D 16384, csum_data =3D 1, tags =3D {slh_first =3D = 0x28067100}}, MH_dat =3D {MH_ext =3D { ext_buf =3D 0x28067200
, = ext_free =3D 0x280541fd, ext_size =3D 134516476,=20 ext_ref =3D 0x280819da},=20 MH_databuf =3D "\000r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\000=A2A\005(=A8:\= 006(@\200\006 (\000\000\000\000\000\000\000\000`=FB=BF=BF@\200\006\001\234=FB=BF=BFOA\0= 05(=FC\216\004\b\ 004=CFe\000\000r\006(\001\000\000\000=A8:\006(\000p\006(=FC\216\004\b=FDA= \005(=FC\216\ 004\b\t\013\005(\200=E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?= \005(5(\005(=A8 :\006(\f=FC=BF=BF=CF@\005( \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\006(\000= r\006(=DA >\005(=A8:\006(\000p\006(=FC\216\004\b=D0=FC=BF=BFS=D4\004\b\200=E6\020(@= \005\b\000r\006("...}},=20 M_databuf =3D "\222=F4\016(=AC=BF\017(\002\000\000\000\000@\000\000\001\000\000\000\000= q\006(\00 0r\006(=FDA\005(=FC\216\004\b=DA\031\b(\000\000\000\000=A2A\005(=A8:\006(= @\200\006(\00 0\000\000\000\000\000\000\000`=FB=BF=BF@\200\006\001\234=FB=BF=BFOA\005(=FC= \216\004\b\004=CF e\000\000r\006(\001\000\000\000=A8:\006(\000p\006(=FC\216\004\b=FDA\005(=FC= \216\004\ b\t\013\005(\200=E6\020(=A2A\005(=A8:\006(\000=E9\a(\200=3D\006(\227?\005= (5(\005(=A8:\00 6(\f=FC=BF=BF=CF@\005( \006\005(\004=CFe\000\200=3D\006(\001\000\000\000\000p\006(\000q\006(\000= r\006(=DA >\005(=A8:\006(\000p\006("...}} (kgdb) print *q Cannot access memory at address 0x0. (kgdb)=20 ---------------------------- Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD 4.11-RELEASE-p13 #3: Thu Feb 23 13:09:31 EST 2006 damascus@daemon.faerunhome.com:/usr/src/sys/compile/DAEMON Timecounter "i8254" frequency 1193182 Hz CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz (1993.54-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf24 Stepping =3D 4 =20 Features=3D0x3febfbff real memory =3D 536608768 (524032K bytes) avail memory =3D 518377472 (506228K bytes) Preloaded elf kernel "kernel" at 0xc03af000. Warning: Pentium 4 CPU: PSE disabled Pentium Pro MTRR support enabled md0: Malloc disk Using $PIR table, 11 entries at 0xc00f28c0 npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 30.0 on = pci0 pci2: on pcib2 twe0: <3ware Storage Controller driver ver. 1.40.01.002> port = 0xdfa0-0xdfaf mem 0xfe000000-0xfe7fffff,0xfeafec00-0xfeafec0f irq 9 at device 9.0 on = pci2 twe0: 4 ports, Firmware FE7X 1.05.00.068, BIOS BE7X 1.08.00.048 fxp0: port 0xdf00-0xdf3f mem 0xfeaa0000-0xfeabffff,0xfeafd000-0xfeafdfff irq 11 at device 10.0 on = pci2 fxp0: Ethernet address 00:02:b3:d0:e3:73 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: port 0xde80-0xdebf mem 0xfea80000-0xfea9ffff,0xfeafc000-0xfeafcfff irq 10 at device 11.0 on = pci2 fxp1: Ethernet address 00:02:b3:ee:65:88 inphy1: on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp2: port 0xdd80-0xddbf mem 0xfea40000-0xfea5ffff,0xfeafb000-0xfeafbfff irq 11 at device 12.0 on = pci2 fxp2: Ethernet address 00:11:11:c1:a2:e5 inphy2: on miibus2 inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp3: port 0xdd00-0xdd3f mem 0xfea20000-0xfea3ffff,0xfeafa000-0xfeafafff irq 11 at device 13.0 on = pci2 fxp3: Ethernet address 00:11:11:c1:a2:e7 inphy3: on miibus3 inphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto atapci0: port 0xdcc0-0xdcff,0xdfe0-0xdfe3,0xdf98-0xdf9f,0xdfe4-0xdfe7,0xdff0-0xdff7 = mem 0xfe9e0000-0xfe9fffff irq 11 at device 14.0 on pci2 ata2: at 0xdff0 on atapci0 ata3: at 0xdf98 on atapci0 pci2: at 15.0 irq 11 isab0: at device 31.0 on = pci0 isa0: on isab0 atapci1: port 0xffa0-0xffaf at device = 31.1 on pci0 ata0: at 0x1f0 irq 14 on atapci1 ata1: at 0x170 irq 15 on atapci1 uhci0: port = 0xef40-0xef5f irq 11 at device 31.2 on pci0 usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pci0: (vendor=3D0x8086, dev=3D0x2443) at 31.3 irq 11 uhci1: port = 0xef80-0xef9f irq 10 at device 31.4 on pci0 usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered orm0: