From owner-freebsd-stable@FreeBSD.ORG Fri Sep 24 03:58:29 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 236A81065672 for ; Fri, 24 Sep 2010 03:58:29 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 648B68FC12 for ; Fri, 24 Sep 2010 03:58:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id o8O3wAUn062911; Fri, 24 Sep 2010 13:58:11 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 24 Sep 2010 13:58:10 +1000 (EST) From: Ian Smith To: jhell In-Reply-To: <4C9B9EAB.4050704@DataIX.net> Message-ID: <20100924130332.U11124@sola.nimnet.asn.au> References: <20100916164930.GA31869@icarus.home.lan> <4C9B9EAB.4050704@DataIX.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , Michael BlackHeart Subject: Re: FreeBSD 8.1 Stable Unreasanoble Rebooting X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Sep 2010 03:58:29 -0000 On Thu, 23 Sep 2010, jhell wrote: > On 09/16/2010 15:33, Michael BlackHeart wrote: > > 2010/9/16 Jeremy Chadwick : > >> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote: > >>> Today I've got a pretty strange event. It looks like a reboot but > >>> unreasonable as far as I see. Before server's uptime was over month, > >>> it's sometimes have to reboot for kernel updates or somethings like > >>> that. I've digen all logs and didn't find a reason, so here they all. > >>> > >>> auth.log > >>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating. > >>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442. > >>> > >>> cron - nothing > >>> debug.log - nothing > >>> dmesg - nothing > >>> > >>> messages > >>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create > >>> socket: Protocol not supported (fdlimit.c:651) > >>> Sep 16 13:45:31 diablo last message repeated 5 times > >>> Sep 16 13:47:23 diablo last message repeated 13 times > >>> Sep 16 13:57:40 diablo last message repeated 51 times > >>> Sep 16 13:59:48 diablo last message repeated 12 times Michael, were these above messages to be expected? > >>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953 > >>> Sep 16 14:00:18 diablo named[1575]: exiting > >>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15 > >>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel > >>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project. > >>> {...} > >> > >> This sure looks like a legitimate reboot to me (e.g. shutdown -r now); > >> note how your system daemons (named, syslogd) are being shut down with > >> SIGTERM. You can check with "last" (shutdown/reboot vs. crash). > >> > >> > >> I would highly recommend taking this machine offline and reinstalling > >> the OS, in addition to newfs'ing all existing filesystems (restore from > >> last known good backup). buildworld/installworld and > >> buildkernel/installkernel may not be enough depending on what the > >> individual did. It's likely the machine could be compromised in some > >> way, especially if there's any service on it which is public-facing, > >> regardless of authentication mechanisms you've deployed in front of it. > >> [..] > > That looks reasonable > > last says: > > reboot ~ th 16 sen 14:04 > > reboot ~ th 16 sen 14:03 > > shutdown ~ th 16 sen 13:59 > > > > and it's pretty good syncs with logs but there's no anybody access to > > physical console 'cos it's not even plugged in. That's for the first. > > Next, I've got, I believe, pretty strong passwords, and also root > > can't log in directly, but wheel user also is in operators so he also > > can reboot or shutdown, but there's no any attempts or successful > > logins. All potentialy dangerous services run under their own > > unprerileged users, and so on. Crontabs also doesn't contain scripts, > > I prefer periodic system, and there's no anyway anything that cause > > reboot. > > Thing that worries me it that there were multiple reboots and shutdown > > that goes up by itself without anyone pressing a button. And in > > messages log there's fsck segment that indicates to unnormal shutdown > > or reboot. It looks like it started to shutting down but was in some > > case interrupted and after powering up it few times reboots itself. > > But commonly FreeBSD doesn't reboot by it's own will. > > The same hardware worked over a half a year under 8.0 stables without > > such a problem. I just would like to understand from where this > > problem comes up. > > This machine doesn't contain any critical info so I'll wait for a bit. > > Also I'd like to notice that recently I've tuned hdd's spindown exept > > system hdd by atacontrol port, powerd and CPU frequency lowering in > > idle, maybe something of this could cause this problem? And where > > could I check this out? > > You might just want to go through your /etc/rc.d/*, crontabs, > /etc/periodic/* and /etc/rc* to check for any commands that call > shutdown(8) or reboot(8). > > Not really malicious but a rogue user that was once a staffer can plant > a shutdown/reboot command in any one of the above matching files and > have it run by at(1). This really sounds like the case here. > > 1). Check the above files. egrep -nr "(shutdown|reboot)" on those. > 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to > unintelligible binaries that have been setuid=0(u+s) owner=root. > 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to > another command or created a script containing shutdown(8) to another > command that matches another system command or has replaced one. > > Thats just a few things to go on for now. Others may have more to add to it. Might help to turn on process accounting to log who ran what, when? Add accounting_enable="YES" to rc.conf, (touch /var/account/acct; accton) to begin accounting before/without rebooting. Then something like 'lastcomm -eE [-f /var/account/acct.N] | less' to perouse the|[some] day's doings. To adjust how many days to keep etc: grep account /etc/defaults/periodic.conf cheers, Ian