Date: Sat, 9 Jun 2007 03:30:08 GMT From: "Jason Naughton" <jnaughto@ee.ryerson.ca> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/113439: 6.2 Kernel Panic Message-ID: <200706090330.l593U8Np050632@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/113439; it has been noted by GNATS. From: "Jason Naughton" <jnaughto@ee.ryerson.ca> To: remko@elvandar.org Cc: freebsd-gnats-submit@freebsd.org Subject: Re: kern/113439: 6.2 Kernel Panic Date: Fri, 8 Jun 2007 23:03:17 -0400 (EDT) Okay I now have a backtrace: # kgdb kernel.debug /var/crash/vmcore.4 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 06 fault virtual address = 0x34 fault code = supervisor read, page not present instruction pointer = 0x20:0xc05c8aec stack pointer = 0x28:0xe6b948d4 frame pointer = 0x28:0xe6b948f4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 807 (perl5.8.8) trap number = 12 panic: page fault cpuid = 2 Uptime: 10m36s Dumping 2047 MB (2 chunks) chunk 0: 1MB (158 pages) ... ok chunk 1: 2047MB (524000 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) list *0xc05c8aec 0xc05c8aec is in vfs_vmio_release (atomic.h:146). 141 142 #else /* defined(CPU_DISABLE_CMPXCHG) */ 143 144 static __inline int 145 atomic_cmpset_int(volatile u_int *dst, u_int exp, u_int src) 146 { 147 int res = exp; 148 149 __asm __volatile ( 150 " " __XSTRING(MPLOCKED) " " (kgdb) backtrace #0 doadump () at pcpu.h:165 #1 0xc05736fe in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0573add in panic (fmt=0xc0799880 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc076d1bb in trap_fatal (frame=0xe6b94894, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc076ce74 in trap_pfault (frame=0xe6b94894, usermode=0, eva=52) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc076c9de in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = -631635928, tf_edi = -631067336, tf_esi = -631067336, tf_ebp = -424064780, tf_isp = -424064832, tf_ebx = -631067336, tf_edx = 4, tf_ecx = -966075776, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1067676948, tf_cs = 32, tf_eflags = 66182, tf_esp = 17104896, tf_ss = -631067240}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc0754cea in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05c8aec in vfs_vmio_release (bp=0xda62ad38) at atomic.h:146 #8 0xc05c94cd in getnewbuf (slpflag=0, slptimeo=0, size=2048, maxsize=8192) at /usr/src/sys/kern/vfs_bio.c:1787 #9 0xc05caec8 in getblk (vp=0xc6cc2000, blkno=648, size=2048, slpflag=0, slptimeo=0, flags=0) at /usr/src/sys/kern/vfs_bio.c:2505 #10 0xc066865c in nfs_getcacheblk (vp=0xc6cc2000, bn=648, size=2048, td=0xc66ada80) at /usr/src/sys/nfsclient/nfs_bio.c:1260 #11 0xc06681df in nfs_write (ap=0x0) at /usr/src/sys/nfsclient/nfs_bio.c:1068 #12 0xc0781add in VOP_WRITE_APV (vop=0xc0811b00, a=0xe6b94be0) at vnode_if.c:698 #13 0xc05ec404 in vn_write (fp=0xc66ff5e8, uio=0xe6b94cb0, active_cred=0xc6c7ca00, flags=0, td=0xc66ada80) at vnode_if.h:372 #14 0xc059d09e in dofilewrite (td=0xc66ada80, fd=0, fp=0xc66ff5e8, auio=0xe6b94cb0, offset=Unhandled dwarf expression opcode 0x93 ) at file.h:253 #15 0xc059cecf in kern_writev (td=0xc66ada80, fd=10, auio=0x0) at /usr/src/sys/kern/sys_generic.c:402 #16 0xc059cda1 in write (td=0x0, uap=0xc66ada80) at /usr/src/sys/kern/sys_generic.c:326 #17 0xc076d5a2 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = -1078001605, tf_edi = 214334464, tf_esi = 10, tf_ebp = -1077942136, tf_isp = -424063644, tf_ebx = 673839744, tf_edx = 512, tf_ecx = 9, tf_eax = 4, tf_trapno = 22, tf_err = 2, tf_eip = 673773747, tf_cs = 51, tf_eflags = 531, tf_esp = -1077942196, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #18 0xc0754d3f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) > Hello, > > Please see the developers-handbook on how to generate proper kerneldumps > which we can use, the current information is not enough to get this > debugged... > > Thanks! > > > On Thu, June 7, 2007 6:43 am, Jason Naughton wrote: >> >>>Number: 113439 >>>Category: kern >>>Synopsis: 6.2 Kernel Panic >>>Confidential: no >>>Severity: critical >>>Priority: high >>>Responsible: freebsd-bugs >>>State: open >>>Quarter: >>>Keywords: >>>Date-Required: >>>Class: sw-bug >>>Submitter-Id: current-users >>>Arrival-Date: Thu Jun 07 04:50:02 GMT 2007 >>>Closed-Date: >>>Last-Modified: >>>Originator: Jason Naughton >>>Release: RELENG_6 >>>Organization: >> Ryerson University >>>Environment: >> FreeBSD server 6.2-STABLE FreeBSD 6.2-STABLE #7: Sat May 26 02:04:17 EDT >> 2007 root@server:/usr/obj/usr/src/sys/eccles2 i386 >> >>>Description: >> Upgraded server from Freebsd 4.11 to Freebsd 6.2 (ie re-installed server). >> Basically did a pkg_info > installed packages on the old server then >> simply >> installed all the packages on the new 6.2 server. Old Freebsd 4.11 server >> ran for 4 years no problems. Upgraded the OS so I could continue to be >> supported >> through the port tree. Yet now the server crashes randomly producing the >> following output: >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 2; apic id = 06 >> fault virtual address = 0x34 >> fault code = supervisor read, page not present >> instruction pointer = 0x20:0xc05c8d1c >> stack pointer = 0x28:0xe8f2d8d4 >> frame pointer = 0x28:0xe8f2d8f4 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, def32 1, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 231 (perl5.8.8) >> trap number = 12 >> panic: page fault >> cpuid = 0 >> Uptime: 11d8h35m49s >> Cannot dump. No dump device defined. >> Automatic reboot in 15 seconds - press a key on the console to abort >> Rebooting... >> >> I have attempted turning of ACPI, used GENERIC kernel configuration >> and a stripped down kernel configuration. Regardless the same panic >> occurs. >> >> I've attempted to install all the kernel debugging so that I could trace >> the >> output but it seems that the server reboots instead of giving prompt with >> KDB. >> >> Present conf is: >> >> machine i386 >> cpu I486_CPU >> cpu I586_CPU >> cpu I686_CPU >> ident eccles >> makeoptions DEBUG=-g # Build kernel with gdb(1) debug >> symbols >> options SCHED_4BSD # 4BSD scheduler >> options PREEMPTION # Enable kernel thread preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates support >> options UFS_ACL # Support for access control lists >> options UFS_DIRHASH # Improve performance on big >> directories >> options MD_ROOT # MD is a potential root device >> options NFSCLIENT # Network Filesystem Client >> options NFSSERVER # Network Filesystem Server >> options NFS_ROOT # NFS usable as /, requires >> NFSCLIENT >> options MSDOSFS # MSDOS Filesystem >> options CD9660 # ISO 9660 Filesystem >> options PROCFS # Process filesystem (requires >> PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_GPT # GUID Partition Tables. >> options COMPAT_43 # Compatible with BSD 4.3 [KEEP >> THIS!] >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options SCSI_DELAY=5000 # Delay (in ms) before probing >> SCSI >> options KTRACE # ktrace(1) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options ADAPTIVE_GIANT # Giant mutex is adaptive. >> device apic # I/O APIC >> options SMP # Symmetric MultiProcessor Kernel >> options PERFMON >> options KDB >> options NPX_DEBUG=1 # enable npx debugging >> options VESA_DEBUG=1 >> options DRM_DEBUG=1 # Include debug printfs (slow) >> options WLDEBUG=1 # enables verbose debugging output >> options TWA_DEBUG=1 # 0-10; 10 prints the most >> messages. >> options SAFE_DEBUG=1 # enable debugging support: >> hw.safe.debug >> options SX_DEBUG=1 >> options DEBUG_SVR4=1 # enable verbose debugging >> options PECOFF_DEBUG=1 >> options KBDIO_DEBUG=2 >> options PSM_DEBUG=1 >> device eisa >> device pci >> device fdc >> device ata >> device atadisk # ATA disk drives >> device ataraid # ATA RAID drives >> device atapicd # ATAPI CDROM drives >> device atapifd # ATAPI floppy drives >> device atapist # ATAPI tape drives >> options ATA_STATIC_ID # Static device numbering >> device mpt # LSI-Logic MPT-Fusion >> device scbus # SCSI bus (required for SCSI) >> device ch # SCSI media changers >> device da # Direct Access (disks) >> device sa # Sequential Access (tape etc) >> device cd # CD >> device pass # Passthrough device (direct SCSI access) >> device ses # SCSI Environmental Services (and SAF-TE) >> device amr # AMI MegaRAID >> device atkbdc # AT keyboard controller >> device atkbd # AT keyboard >> device psm # PS/2 mouse >> device kbdmux # keyboard multiplexer >> device vga # VGA video card driver >> device splash # Splash screen and screen saver support >> device sc >> device agp # support several AGP chipsets >> device pmtimer >> device sio # 8250, 16[45]50 based serial ports >> device de # DEC/Intel DC21x4x (``Tulip'') >> device em # Intel PRO/1000 adapter Gigabit Ethernet >> Card >> device ixgb # Intel PRO/10GbE Ethernet Card >> device txp # 3Com 3cR990 (``Typhoon'') >> device vx # 3Com 3c590, 3c595 (``Vortex'') >> device miibus # MII bus support >> device bce # Broadcom BCM5706/BCM5708 Gigabit >> Ethernet >> device bfe # Broadcom BCM440x 10/100 Ethernet >> device bge # Broadcom BCM570xx Gigabit Ethernet >> device dc # DEC/Intel 21143 and various workalikes >> device fxp # Intel EtherExpress PRO/100B (82557, >> 82558) >> device lge # Level 1 LXT1001 gigabit Ethernet >> device msk # Marvell/SysKonnect Yukon II Gigabit >> Ethernet >> device nge # NatSemi DP83820 gigabit Ethernet >> device nve # nVidia nForce MCP on-board Ethernet >> Networking >> device pcn # AMD Am79C97x PCI 10/100(precedence over >> 'lnc') >> device re # RealTek 8139C+/8169/8169S/8110S >> device rl # RealTek 8129/8139 >> device sf # Adaptec AIC-6915 (``Starfire'') >> device sis # Silicon Integrated Systems SiS 900/SiS >> 7016 >> device sk # SysKonnect SK-984x & SK-982x gigabit >> Ethernet >> device ste # Sundance ST201 (D-Link DFE-550TX) >> device stge # Sundance/Tamarack TC9021 gigabit >> Ethernet >> device ti # Alteon Networks Tigon I/II gigabit >> Ethernet >> device tl # Texas Instruments ThunderLAN >> device tx # SMC EtherPower II (83c170 ``EPIC'') >> device vge # VIA VT612x gigabit Ethernet >> device vr # VIA Rhine, Rhine II >> device wb # Winbond W89C840F >> device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') >> device loop # Network loopback >> device random # Entropy device >> device ether # Ethernet support >> device sl # Kernel SLIP >> device ppp # Kernel PPP >> device tun # Packet tunnel. >> device pty # Pseudo-ttys (telnet etc) >> device md # Memory "disks" >> device gif # IPv6 and IPv4 tunneling >> device faith # IPv6-to-IPv4 relaying (translation) >> device bpf # Berkeley packet filter >> >> Dmesg output: >> >> Copyright (c) 1992-2007 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 6.2-STABLE #7: Sat May 26 02:04:17 EDT 2007 >> root@server:/usr/obj/usr/src/sys/eccles2 >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3192.01-MHz 686-class CPU) >> Origin = "GenuineIntel" Id = 0xf41 Stepping = 1 >> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C >> MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,<b14>> >> AMD Features=0x20100000<NX,LM> >> Logical CPUs per core: 2 >> real memory = 2147352576 (2047 MB) >> avail memory = 2096361472 (1999 MB) >> ACPI APIC Table: <A M I OEMAPIC > >> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 6 >> cpu3 (AP): APIC ID: 7 >> ioapic0 <Version 2.0> irqs 0-23 on motherboard >> ioapic1 <Version 2.0> irqs 24-47 on motherboard >> ioapic2 <Version 2.0> irqs 48-71 on motherboard >> kbd1 at kbdmux0 >> acpi0: <A M I OEMRSDT> on motherboard >> acpi0: Power Button (fixed) >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> cpu0: <ACPI CPU> on acpi0 >> acpi_throttle0: <ACPI CPU Throttling> on cpu0 >> cpu1: <ACPI CPU> on acpi0 >> acpi_throttle1: <ACPI CPU Throttling> on cpu1 >> cpu2: <ACPI CPU> on acpi0 >> cpu3: <ACPI CPU> on acpi0 >> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 >> pci0: <ACPI PCI bus> on pcib0 >> pci0: <unknown> at device 0.1 (no driver attached) >> pci0: <base peripheral> at device 1.0 (no driver attached) >> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0 >> pci1: <ACPI PCI bus> on pcib1 >> pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 >> pci2: <ACPI PCI bus> on pcib2 >> mpt0: <LSILogic 1030 Ultra4 Adapter> port 0xc800-0xc8ff mem >> 0xdedc0000-0xdedcfff >> f,0xdedb0000-0xdedbffff irq 26 at device 5.0 on pci2 >> mpt0: [GIANT-LOCKED] >> mpt0: MPI Version=1.2.14.0 >> mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE ) >> mpt0: 0 Active Volumes (1 Max) >> mpt0: 0 Hidden Drive Members (6 Max) >> mpt1: <LSILogic 1030 Ultra4 Adapter> port 0xcc00-0xccff mem >> 0xdedf0000-0xdedffff >> f,0xdede0000-0xdedeffff irq 25 at device 5.1 on pci2 >> mpt1: [GIANT-LOCKED] >> mpt1: MPI Version=1.2.14.0 >> mpt1: Capabilities: ( RAID-1E RAID-1 SAFTE ) >> mpt1: 0 Active Volumes (1 Max) >> mpt1: 0 Hidden Drive Members (6 Max) >> pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1 >> pci3: <ACPI PCI bus> on pcib3 >> em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port >> 0xdc00-0xdc3f m >> em 0xdeea0000-0xdeebffff irq 54 at device 4.0 on pci3 >> em0: Ethernet address: XX:XX:XX:XX:XX:XX >> em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port >> 0xdc80-0xdcbf m >> em 0xdeee0000-0xdeefffff irq 55 at device 4.1 on pci3 >> em1: Ethernet address: XX:XX:XX:XX:XX:XX >> pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0 >> pci4: <ACPI PCI bus> on pcib4 >> pci4: <display, VGA> at device 12.0 (no driver attached) >> isab0: <PCI-ISA bridge> at device 31.0 on pci0 >> isa0: <ISA bus> on isab0 >> atapci0: <Intel ICH5 UDMA100 controller> port >> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x37 >> 6,0xfc00-0xfc0f at device 31.1 on pci0 >> ata0: <ATA channel 0> on atapci0 >> ata1: <ATA channel 1> on atapci0 >> pci0: <serial bus, SMBus> at device 31.3 (no driver attached) >> acpi_button0: <Power Button> on acpi0 >> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 >> atkbd0: <AT Keyboard> irq 1 on atkbdc0 >> kbd0 at atkbd0 >> atkbd0: [GIANT-LOCKED] >> psm0: current command byte:0065 >> psm0: <PS/2 Mouse> irq 12 on atkbdc0 >> psm0: [GIANT-LOCKED] >> psm0: model Generic PS/2 mouse, device ID 0-00, 2 buttons >> psm0: config:00000000, flags:00000008, packet size:3 >> psm0: syncmask:c0, syncbits:00 >> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 on acpi0 >> sio0: type 16550A >> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 flags 0x30 on >> acpi0 >> sio1: type 16550A, console >> pmtimer0 on isa0 >> orm0: <ISA Option ROMs> at iomem >> 0xc0000-0xca7ff,0xca800-0xce7ff,0xce800-0xcf7ff >> ,0xcf800-0xd07ff,0xdc000-0xdffff on isa0 >> sc0: <System console> at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=0x100> >> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 >> Timecounters tick every 1.000 msec >> acd0: CDROM <SR244W/T01A> at ata1-slave UDMA33 >> Waiting 5 seconds for SCSI devices to settle >> ses0 at mpt0 bus 0 target 6 lun 0 >> ses0: <ESG-SHV SCA HSBP M29 1.06> Fixed Processor SCSI-2 device >> ses0: 3.300MB/s transfers >> ses0: SAF-TE Compliant Device >> da0 at mpt0 bus 0 target 0 lun 0 >> da0: <SEAGATE ST318404LSUN18G 4203> Fixed Direct Access SCSI-3 device >> da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing >> Enable >> d >> da0: 17274MB (35378533 512 byte sectors: 255H 63S/T 2202C) >> da1 at mpt0 bus 0 target 1 lun 0 >> da1: <SEAGATE ST318404LSUN18G 4203> Fixed Direct Access SCSI-3 device >> da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing >> Enable >> d >> da1: 17274MB (35378533 512 byte sectors: 255H 63S/T 2202C) >> da2 at mpt0 bus 0 target 3 lun 0 >> da2: <SEAGATE ST373307LC 0007> Fixed Direct Access SCSI-3 device >> da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing >> Enabl >> ed >> da2: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) >> SMP: AP CPU #2 Launched! >> SMP: AP CPU #1 Launched! >> SMP: AP CPU #3 Launched! >> >> >>>How-To-Repeat: >> I'm monitoring it now but cannot get a consistent method to cause the >> server to crash. I've setup a console log to monitor the console output >> (shown previously). >>>Fix: >> None known. >> >>>Release-Note: >>>Audit-Trail: >>>Unformatted: >> _______________________________________________ >> freebsd-bugs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-bugs >> To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org" >> > > > -- > Kind regards, > > Remko Lodder ** remko@elvandar.org > FreeBSD ** remko@FreeBSD.org > > /* Quis custodiet ipsos custodes */ > Cheers Jason Jason Naughton, Lead Engineer, Department of Electrical Engineering, Ryerson University 245 Church St., Toronto, Ontario, M5B 2K3 Office: (416)-979-5000 x7168 Fax: (416)-979-5280
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200706090330.l593U8Np050632>