From owner-freebsd-stable@FreeBSD.ORG Thu Sep 23 18:38:40 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E76F9106564A for ; Thu, 23 Sep 2010 18:38:40 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 739268FC1B for ; Thu, 23 Sep 2010 18:38:40 +0000 (UTC) Received: by wyb33 with SMTP id 33so2518693wyb.13 for ; Thu, 23 Sep 2010 11:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=mrBly7zNJR6WY6GpnaZmLkpQ8k/9GlGcyUqWs8H3pqY=; b=uAdaZpEsnguQXRjyObe4/PSVGbDfYrv3YIpSSiA/cgkymfuMrgcGpTCXK/Yes0bgVU 9repCAxNro1d9gb94MEmWCSyOmRUH3hocwNIzfBLDSbIVZ4Vgw/jAyiRv+ep5pBq7Lzi +eSPmJs97FlFXUcZCS+7jNVNlBaW5JTlpGgSI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=Ez5rhFLboYH1vtkh85Q9OM5xvkLnCT+FnnTgT0DV/KyK68zSvwlQXYU4TgFwQGaBa9 fDabqLJmPiZiAZKSK+k30RoVMr8ptqIekwJhW6Z+85dC6BAOJjkgPbsvZc4I158IIKc2 tk+utz0JZkfBsdRrHPXOR/UZd5CKjj9YPn/eY= Received: by 10.227.69.134 with SMTP id z6mr1715160wbi.201.1285267119346; Thu, 23 Sep 2010 11:38:39 -0700 (PDT) Received: from centel.dataix.local ([99.19.43.205]) by mx.google.com with ESMTPS id m5sm964429wbc.15.2010.09.23.11.38.36 (version=SSLv3 cipher=RC4-MD5); Thu, 23 Sep 2010 11:38:37 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C9B9EAB.4050704@DataIX.net> Date: Thu, 23 Sep 2010 14:38:35 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100917 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Michael BlackHeart References: <20100916164930.GA31869@icarus.home.lan> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: FreeBSD 8.1 Stable Unreasanoble Rebooting X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 18:38:41 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/16/2010 15:33, Michael BlackHeart wrote: > 2010/9/16 Jeremy Chadwick : >> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote: >>> Today I've got a pretty strange event. It looks like a reboot but >>> unreasonable as far as I see. Before server's uptime was over month, >>> it's sometimes have to reboot for kernel updates or somethings like >>> that. I've digen all logs and didn't find a reason, so here they all. >>> >>> auth.log >>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating. >>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442. >>> >>> cron - nothing >>> debug.log - nothing >>> dmesg - nothing >>> >>> messages >>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create >>> socket: Protocol not supported (fdlimit.c:651) >>> Sep 16 13:45:31 diablo last message repeated 5 times >>> Sep 16 13:47:23 diablo last message repeated 13 times >>> Sep 16 13:57:40 diablo last message repeated 51 times >>> Sep 16 13:59:48 diablo last message repeated 12 times >>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953 >>> Sep 16 14:00:18 diablo named[1575]: exiting >>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15 >>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel >>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project. >>> {...} >> >> This sure looks like a legitimate reboot to me (e.g. shutdown -r now); >> note how your system daemons (named, syslogd) are being shut down with >> SIGTERM. You can check with "last" (shutdown/reboot vs. crash). >> >> >> I would highly recommend taking this machine offline and reinstalling >> the OS, in addition to newfs'ing all existing filesystems (restore from >> last known good backup). buildworld/installworld and >> buildkernel/installkernel may not be enough depending on what the >> individual did. It's likely the machine could be compromised in some >> way, especially if there's any service on it which is public-facing, >> regardless of authentication mechanisms you've deployed in front of it. >> >> >> -- >> | Jeremy Chadwick jdc@parodius.com | >> | Parodius Networking http://www.parodius.com/ | >> | UNIX Systems Administrator Mountain View, CA, USA | >> | Making life hard for others since 1977. PGP: 4BD6C0CB | >> >> > > That looks reasonable > last says: > reboot ~ th 16 sen 14:04 > reboot ~ th 16 sen 14:03 > shutdown ~ th 16 sen 13:59 > > and it's pretty good syncs with logs but there's no anybody access to > physical console 'cos it's not even plugged in. That's for the first. > Next, I've got, I believe, pretty strong passwords, and also root > can't log in directly, but wheel user also is in operators so he also > can reboot or shutdown, but there's no any attempts or successful > logins. All potentialy dangerous services run under their own > unprerileged users, and so on. Crontabs also doesn't contain scripts, > I prefer periodic system, and there's no anyway anything that cause > reboot. > Thing that worries me it that there were multiple reboots and shutdown > that goes up by itself without anyone pressing a button. And in > messages log there's fsck segment that indicates to unnormal shutdown > or reboot. It looks like it started to shutting down but was in some > case interrupted and after powering up it few times reboots itself. > But commonly FreeBSD doesn't reboot by it's own will. > The same hardware worked over a half a year under 8.0 stables without > such a problem. I just would like to understand from where this > problem comes up. > This machine doesn't contain any critical info so I'll wait for a bit. > Also I'd like to notice that recently I've tuned hdd's spindown exept > system hdd by atacontrol port, powerd and CPU frequency lowering in > idle, maybe something of this could cause this problem? And where > could I check this out? You might just want to go through your /etc/rc.d/*, crontabs, /etc/periodic/* and /etc/rc* to check for any commands that call shutdown(8) or reboot(8). Not really malicious but a rogue user that was once a staffer can plant a shutdown/reboot command in any one of the above matching files and have it run by at(1). This really sounds like the case here. 1). Check the above files. egrep -nr "(shutdown|reboot)" on those. 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to unintelligible binaries that have been setuid=0(u+s) owner=root. 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to another command or created a script containing shutdown(8) to another command that matches another system command or has replaced one. Thats just a few things to go on for now. Others may have more to add to it. Regards & Good luck, - -- jhell,v -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJMm56rAAoJEJBXh4mJ2FR+P8wH/RyMxZSsQGKODwtWNNipeUSZ iMA+n7ub3GStz1k7hfQQfsWqdWtfrO73Nrw6Af5vnGWdLGjatwD36qOmcLqfg1tE ZBc7QGFYyFkqHxL2Dor58Q6XV2GxS1zEGyT5Rf2mNGqMOyaTVdBpIlfX02mUdpX3 0KDRdNivEXK/A8sGSK0WN46E7+uSO7L7n4zfv3fEFmsZh9VhtRwf558MJxO39UgN FsLKQRy0DVPMAnb9zr6dWvRtAVpnCgQhlLdspETN7SFamO4CAs/8ZiasPaNzG/jl G3avQZhhT1Ws/DpDBdYwj5nOpkTU2Y90PUoLXktGjg90tNvOaWfboAhA80tzyWA= =hDLt -----END PGP SIGNATURE-----