Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 12 Jul 2005 13:15:59 -0400 (EDT)
From:      Tuc at T-B-O-H <ml@t-b-o-h.net>
To:        xfb52@dial.pipex.com (Alex Zbyslaw)
Cc:        Tuc at T-B-O-H <ml@t-b-o-h.net>, freebsd-questions@freebsd.org
Subject:   Re: Problems since 5.3-RELEASE-p15
Message-ID:  <200507121715.j6CHFxIw027032@himinbjorg.tucs-beachin-obx-house.com>
In-Reply-To: <42CDC522.5010309@dial.pipex.com> from "Alex Zbyslaw" at Jul 08, 2005 01:13:22 AM

next in thread | previous in thread | raw e-mail | index | archive | help
> However, if you run ktrace on a simple program like ls: ktrace ls
> then do a kdump | less, you will see that after finding ls, 
> /libexec/ld-elf.so.1 is the first thing accessed.  So, when things start 
> working again, /libexec/ld-elf.so.1 is magically fixed, which just makes 
> no sense.
>
	Imagine if it makes no sense to you how *I* must be feeling!
> 
> I assume there are no other messages obvious errors in /var/log/messages?
>
	Nope, I have debug turned all the way up... And just out of the blue
on the 9th at 3am I see :

Jul  9 03:01:30 himinbjorg kernel: pid 49967 (mailwrapper), uid 0: exited on sig
nal 11 (core dumped)
Jul  9 03:01:30 himinbjorg kernel: Jul  9 03:01:30 himinbjorg kernel: pid 49967 
(mailwrapper), uid 0: exited on signal 11 (core dumped)

	and then goes down hill from there :

Jul  9 03:01:31 himinbjorg kernel: pid 49969 (sendmail), uid 25: exited on signa
l 11
Jul  9 03:01:31 himinbjorg kernel: Jul  9 03:01:31 himinbjorg kernel: pid 49969 
(sendmail), uid 25: exited on signal 11
Jul  9 03:10:02 himinbjorg kernel: pid 50135 (rmail), uid 66: exited on signal 1
1 (core dumped)
Jul  9 03:10:02 himinbjorg kernel: Jul  9 03:10:02 himinbjorg kernel: pid 50135 
(rmail), uid 66: exited on signal 11 (core dumped)
Jul  9 03:10:04 himinbjorg kernel: pid 50151 (uuxqt), uid 66: exited on signal 1
1
Jul  9 03:10:04 himinbjorg kernel: Jul  9 03:10:04 himinbjorg kernel: pid 50151 
(uuxqt), uid 66: exited on signal 11
Jul  9 03:10:05 himinbjorg kernel: pid 50152 (sh), uid 0: exited on signal 11 (c
ore dumped)
Jul  9 03:10:05 himinbjorg kernel: Jul  9 03:10:05 himinbjorg kernel: pid 50152 
(sh), uid 0: exited on signal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: pid 50191 (sleep), uid 0: exited on signal 11
 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: pid 50192 (mailwrapper), uid 0: exited on sig
nal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: pid 50193 (stunnel), uid 0: exited on signal 
11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: pid 50194 (uuxqt), uid 66: exited on signal 1
1
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50191 
(sleep), uid 0: exited on signal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50192 
(mailwrapper), uid 0: exited on signal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50193 
(stunnel), uid 0: exited on signal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50194 
(uuxqt), uid 66: exited on signal 11
Jul  9 03:12:01 himinbjorg kernel: pid 50195 (sleep), uid 0: exited on signal 11
 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: pid 50196 (mailwrapper), uid 0: exited on sig
nal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50195 
(sleep), uid 0: exited on signal 11 (core dumped)
Jul  9 03:12:01 himinbjorg kernel: Jul  9 03:12:01 himinbjorg kernel: pid 50196 
(mailwrapper), uid 0: exited on signal 11 (core dumped)
Jul  9 03:15:04 himinbjorg kernel: pid 50229 (sendmail), uid 0: exited on signal
11

	and happens until...........

Jul  9 03:55:00 himinbjorg kernel: Jul  9 03:55:00 himinbjorg kernel: pid 50481 
(sh), uid 0: exited on signal 11 (core dumped)
Jul  9 03:55:00 himinbjorg kernel: Jul  9 03:55:00 himinbjorg kernel: pid 50482 
(sh), uid 0: exited on signal 11 (core dumped)
Jul  9 03:57:00 himinbjorg kernel: pid 50484 (sh), uid 0: exited on signal 11 (c
ore dumped)
Jul  9 03:57:00 himinbjorg kernel: Jul  9 03:57:00 himinbjorg kernel: pid 50484 
(sh), uid 0: exited on signal 11 (core dumped)


	and then everything is fine again.

	I hadn't been home since the morning of the 8th, and if I'm up at 3am
its NOT a good thing usually.... So wasn't like I was here doing anything
at the time.
>
> One final thought is that it could be the disk.  You could try 
> installing smartmontools and see if the disk thinks it is OK -- though 
> of course it could be the controller.  But in such a case I might expect 
> other errors.
>
	Already had that installed....

smartctl version 5.33 [i386-portbld-freebsd5.3] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     FUJITSU MHT2060AH
Serial Number:    NP0DT512J16R
Firmware Version: 006C
User Capacity:    60,011,642,880 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 3a
Local Time is:    Tue Jul 12 13:14:56 2005 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 ( 440) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					No General Purpose Logging support.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  60) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   046    Pre-fail  Always       -       6586
  2 Throughput_Performance  0x0005   100   100   030    Pre-fail  Offline      -       19464192
  3 Spin_Up_Time            0x0003   100   100   025    Pre-fail  Always       -       1
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       89
  5 Reallocated_Sector_Ct   0x0033   100   100   024    Pre-fail  Always       -       8589934592000
  7 Seek_Error_Rate         0x000f   100   100   047    Pre-fail  Always       -       1434
  8 Seek_Time_Performance   0x0005   100   100   019    Pre-fail  Offline      -       0
  9 Power_On_Seconds        0x0032   094   094   000    Old_age   Always       -       3468h+21m+46s
 10 Spin_Retry_Count        0x0013   100   100   020    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       78
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       57
193 Load_Cycle_Count        0x0032   092   092   000    Old_age   Always       -       80308
194 Temperature_Celsius     0x0022   100   065   000    Old_age   Always       -       56 (Lifetime Min/Max 15/67)
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       283049984
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x000f   100   100   060    Pre-fail  Always       -       29362
203 Run_Out_Cancel          0x0002   100   100   000    Old_age   Always       -       1529016811656

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3457         -
# 2  Short offline       Completed without error       00%      3433         -
# 3  Short offline       Completed without error       00%      3409         -
# 4  Extended offline    Completed without error       00%      3386         -
# 5  Short offline       Completed without error       00%      3385         -
# 6  Short offline       Completed without error       00%      3361         -
# 7  Short offline       Completed without error       00%      3312         -
# 8  Short offline       Completed without error       00%      3302         -
# 9  Short offline       Completed without error       00%      3278         -
#10  Extended offline    Completed without error       00%      3256         -
#11  Short offline       Completed without error       00%      3254         -
#12  Short offline       Completed without error       00%      3233         -
#13  Short offline       Completed without error       00%      3216         -
#14  Short offline       Completed without error       00%      3192         -
#15  Short offline       Completed without error       00%      3168         -
#16  Short offline       Completed without error       00%      3144         -
#17  Short offline       Completed without error       00%      3120         -
#18  Extended offline    Completed without error       00%      3098         -
#19  Short offline       Completed without error       00%      3096         -
#20  Short offline       Completed without error       00%      3072         -
#21  Short offline       Completed without error       00%      3048         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> The classic bad memory symptoms are periodic reboots, and random 
> segmentation faults.  The latter is particular to bad memory, I would 
> say.  The fomer can have other causes.
> 
> It's the intermittent nature of the fault that really makes me think 
> hardware.  Am I right in remembering that you upgraded to 5.4 and still 
> had the same problems?
> 
	Correct... 

FreeBSD himinbjorg.tucs-beachin-obx-house.com 5.4-RELEASE-p2 FreeBSD 5.4-RELEASE-p2 #2: Tue Jun 21 00:52:02 EDT 2005     root@himinbjorg.tucs-beachin-obx-house.com:/usr/obj/usr/src/sys/HIMINBJORG53  i386


		Tuc



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200507121715.j6CHFxIw027032>