Date: Thu, 23 Sep 2010 14:38:35 -0400 From: jhell <jhell@DataIX.net> To: Michael BlackHeart <amdmiek@gmail.com> Cc: freebsd-stable@freebsd.org, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: FreeBSD 8.1 Stable Unreasanoble Rebooting Message-ID: <4C9B9EAB.4050704@DataIX.net> In-Reply-To: <AANLkTinZRdrtdehrzqSRyom8hAo=gBLvtnwuSSgCC6B9@mail.gmail.com> References: <AANLkTimGZOwf5USXD4ANLBq0EBR1LvBakKAH88pmTLo%2B@mail.gmail.com> <20100916164930.GA31869@icarus.home.lan> <AANLkTinZRdrtdehrzqSRyom8hAo=gBLvtnwuSSgCC6B9@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/16/2010 15:33, Michael BlackHeart wrote:
> 2010/9/16 Jeremy Chadwick <freebsd@jdc.parodius.com>:
>> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote:
>>> Today I've got a pretty strange event. It looks like a reboot but
>>> unreasonable as far as I see. Before server's uptime was over month,
>>> it's sometimes have to reboot for kernel updates or somethings like
>>> that. I've digen all logs and didn't find a reason, so here they all.
>>>
>>> auth.log
>>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating.
>>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442.
>>>
>>> cron - nothing
>>> debug.log - nothing
>>> dmesg - nothing
>>>
>>> messages
>>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create
>>> socket: Protocol not supported (fdlimit.c:651)
>>> Sep 16 13:45:31 diablo last message repeated 5 times
>>> Sep 16 13:47:23 diablo last message repeated 13 times
>>> Sep 16 13:57:40 diablo last message repeated 51 times
>>> Sep 16 13:59:48 diablo last message repeated 12 times
>>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953
>>> Sep 16 14:00:18 diablo named[1575]: exiting
>>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15
>>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel
>>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project.
>>> {...}
>>
>> This sure looks like a legitimate reboot to me (e.g. shutdown -r now);
>> note how your system daemons (named, syslogd) are being shut down with
>> SIGTERM. You can check with "last" (shutdown/reboot vs. crash).
>>
>> <paranoid>
>> I would highly recommend taking this machine offline and reinstalling
>> the OS, in addition to newfs'ing all existing filesystems (restore from
>> last known good backup). buildworld/installworld and
>> buildkernel/installkernel may not be enough depending on what the
>> individual did. It's likely the machine could be compromised in some
>> way, especially if there's any service on it which is public-facing,
>> regardless of authentication mechanisms you've deployed in front of it.
>> </paranoid>
>>
>> --
>> | Jeremy Chadwick jdc@parodius.com |
>> | Parodius Networking http://www.parodius.com/ |
>> | UNIX Systems Administrator Mountain View, CA, USA |
>> | Making life hard for others since 1977. PGP: 4BD6C0CB |
>>
>>
>
> That looks reasonable
> last says:
> reboot ~ th 16 sen 14:04
> reboot ~ th 16 sen 14:03
> shutdown ~ th 16 sen 13:59
>
> and it's pretty good syncs with logs but there's no anybody access to
> physical console 'cos it's not even plugged in. That's for the first.
> Next, I've got, I believe, pretty strong passwords, and also root
> can't log in directly, but wheel user also is in operators so he also
> can reboot or shutdown, but there's no any attempts or successful
> logins. All potentialy dangerous services run under their own
> unprerileged users, and so on. Crontabs also doesn't contain scripts,
> I prefer periodic system, and there's no anyway anything that cause
> reboot.
> Thing that worries me it that there were multiple reboots and shutdown
> that goes up by itself without anyone pressing a button. And in
> messages log there's fsck segment that indicates to unnormal shutdown
> or reboot. It looks like it started to shutting down but was in some
> case interrupted and after powering up it few times reboots itself.
> But commonly FreeBSD doesn't reboot by it's own will.
> The same hardware worked over a half a year under 8.0 stables without
> such a problem. I just would like to understand from where this
> problem comes up.
> This machine doesn't contain any critical info so I'll wait for a bit.
> Also I'd like to notice that recently I've tuned hdd's spindown exept
> system hdd by atacontrol port, powerd and CPU frequency lowering in
> idle, maybe something of this could cause this problem? And where
> could I check this out?
You might just want to go through your /etc/rc.d/*, crontabs,
/etc/periodic/* and /etc/rc* to check for any commands that call
shutdown(8) or reboot(8).
Not really malicious but a rogue user that was once a staffer can plant
a shutdown/reboot command in any one of the above matching files and
have it run by at(1). This really sounds like the case here.
1). Check the above files. egrep -nr "(shutdown|reboot)" on those.
2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to
unintelligible binaries that have been setuid=0(u+s) owner=root.
3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to
another command or created a script containing shutdown(8) to another
command that matches another system command or has replaced one.
Thats just a few things to go on for now. Others may have more to add to it.
Regards & Good luck,
- --
jhell,v
-----BEGIN PGP SIGNATURE-----
iQEcBAEBAgAGBQJMm56rAAoJEJBXh4mJ2FR+P8wH/RyMxZSsQGKODwtWNNipeUSZ
iMA+n7ub3GStz1k7hfQQfsWqdWtfrO73Nrw6Af5vnGWdLGjatwD36qOmcLqfg1tE
ZBc7QGFYyFkqHxL2Dor58Q6XV2GxS1zEGyT5Rf2mNGqMOyaTVdBpIlfX02mUdpX3
0KDRdNivEXK/A8sGSK0WN46E7+uSO7L7n4zfv3fEFmsZh9VhtRwf558MJxO39UgN
FsLKQRy0DVPMAnb9zr6dWvRtAVpnCgQhlLdspETN7SFamO4CAs/8ZiasPaNzG/jl
G3avQZhhT1Ws/DpDBdYwj5nOpkTU2Y90PUoLXktGjg90tNvOaWfboAhA80tzyWA=
=hDLt
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C9B9EAB.4050704>