From owner-freebsd-stable@FreeBSD.ORG Thu Sep 23 18:42:29 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E68A4106566B for ; Thu, 23 Sep 2010 18:42:29 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id ADD408FC16 for ; Thu, 23 Sep 2010 18:42:29 +0000 (UTC) Received: by pxi17 with SMTP id 17so727545pxi.13 for ; Thu, 23 Sep 2010 11:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=UhUy11T5gict2tvyNduP9knM08YDHSbB1iBdg+UEKSw=; b=P3dR1q+UaE/6qZhbQ5icE5PjhWkHn/SEFG3FjcqXUtkc8XzOM4FpGpQ6J00A3ADfyv MpCFv8gXDRGBzLc4mQtxSiRYbD04cwdrAkl9qY9wFT8Z+iWJHxKUJJ5KcTk9loS5G/WZ JFPoUSouF04ts8sUdfQiQnBC1DJNLhu4nxRlU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=pWtmzoeYf4vi15+9Q4Kd6o5ClbVQL9ERvSTKZllP8izDlk9D0bRvJf08fsOT7jeXb1 AfVTlMevKhALfkez/FG8ZsgagtFm02dK9PCqnOJysaXbU5+OJY4MYhjoCTYLQs+Ry3VM BcYY3vfRwnCnaX4iIq7MPXUybd/C8TrSyW6Es= Received: by 10.142.216.2 with SMTP id o2mr1796503wfg.142.1285267349236; Thu, 23 Sep 2010 11:42:29 -0700 (PDT) Received: from centel.dataix.local (adsl-99-19-43-205.dsl.klmzmi.sbcglobal.net [99.19.43.205]) by mx.google.com with ESMTPS id n9sm349585vch.43.2010.09.23.11.42.27 (version=SSLv3 cipher=RC4-MD5); Thu, 23 Sep 2010 11:42:28 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C9B9F92.5030102@DataIX.net> Date: Thu, 23 Sep 2010 14:42:26 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100917 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Michael BlackHeart References: <20100916164930.GA31869@icarus.home.lan> <4C9B9EAB.4050704@DataIX.net> In-Reply-To: <4C9B9EAB.4050704@DataIX.net> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: FreeBSD 8.1 Stable Unreasanoble Rebooting X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 18:42:30 -0000 On 09/23/2010 14:38, jhell wrote: > On 09/16/2010 15:33, Michael BlackHeart wrote: >> 2010/9/16 Jeremy Chadwick : >>> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote: >>>> Today I've got a pretty strange event. It looks like a reboot but >>>> unreasonable as far as I see. Before server's uptime was over month, >>>> it's sometimes have to reboot for kernel updates or somethings like >>>> that. I've digen all logs and didn't find a reason, so here they all. >>>> >>>> auth.log >>>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating. >>>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442. >>>> >>>> cron - nothing >>>> debug.log - nothing >>>> dmesg - nothing >>>> >>>> messages >>>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create >>>> socket: Protocol not supported (fdlimit.c:651) >>>> Sep 16 13:45:31 diablo last message repeated 5 times >>>> Sep 16 13:47:23 diablo last message repeated 13 times >>>> Sep 16 13:57:40 diablo last message repeated 51 times >>>> Sep 16 13:59:48 diablo last message repeated 12 times >>>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953 >>>> Sep 16 14:00:18 diablo named[1575]: exiting >>>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15 >>>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel >>>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project. >>>> {...} >>> >>> This sure looks like a legitimate reboot to me (e.g. shutdown -r now); >>> note how your system daemons (named, syslogd) are being shut down with >>> SIGTERM. You can check with "last" (shutdown/reboot vs. crash). >>> >>> >>> I would highly recommend taking this machine offline and reinstalling >>> the OS, in addition to newfs'ing all existing filesystems (restore from >>> last known good backup). buildworld/installworld and >>> buildkernel/installkernel may not be enough depending on what the >>> individual did. It's likely the machine could be compromised in some >>> way, especially if there's any service on it which is public-facing, >>> regardless of authentication mechanisms you've deployed in front of it. >>> >>> >>> -- >>> | Jeremy Chadwick jdc@parodius.com | >>> | Parodius Networking http://www.parodius.com/ | >>> | UNIX Systems Administrator Mountain View, CA, USA | >>> | Making life hard for others since 1977. PGP: 4BD6C0CB | >>> >>> > >> That looks reasonable >> last says: >> reboot ~ th 16 sen 14:04 >> reboot ~ th 16 sen 14:03 >> shutdown ~ th 16 sen 13:59 > >> and it's pretty good syncs with logs but there's no anybody access to >> physical console 'cos it's not even plugged in. That's for the first. >> Next, I've got, I believe, pretty strong passwords, and also root >> can't log in directly, but wheel user also is in operators so he also >> can reboot or shutdown, but there's no any attempts or successful >> logins. All potentialy dangerous services run under their own >> unprerileged users, and so on. Crontabs also doesn't contain scripts, >> I prefer periodic system, and there's no anyway anything that cause >> reboot. >> Thing that worries me it that there were multiple reboots and shutdown >> that goes up by itself without anyone pressing a button. And in >> messages log there's fsck segment that indicates to unnormal shutdown >> or reboot. It looks like it started to shutting down but was in some >> case interrupted and after powering up it few times reboots itself. >> But commonly FreeBSD doesn't reboot by it's own will. >> The same hardware worked over a half a year under 8.0 stables without >> such a problem. I just would like to understand from where this >> problem comes up. >> This machine doesn't contain any critical info so I'll wait for a bit. >> Also I'd like to notice that recently I've tuned hdd's spindown exept >> system hdd by atacontrol port, powerd and CPU frequency lowering in >> idle, maybe something of this could cause this problem? And where >> could I check this out? > > You might just want to go through your /etc/rc.d/*, crontabs, > /etc/periodic/* and /etc/rc* to check for any commands that call > shutdown(8) or reboot(8). > > Not really malicious but a rogue user that was once a staffer can plant > a shutdown/reboot command in any one of the above matching files and > have it run by at(1). This really sounds like the case here. > > 1). Check the above files. egrep -nr "(shutdown|reboot)" on those. > 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to > unintelligible binaries that have been setuid=0(u+s) owner=root. > 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to > another command or created a script containing shutdown(8) to another > command that matches another system command or has replaced one. > > Thats just a few things to go on for now. Others may have more to add to it. > > > Regards & Good luck, > One more thing I would like to add to this is audit(8). This can prove to be an invaluable tool to find rogue binaries on the system and track down exactly to where one is being executed. Since your looking at a specific time-frame that this is happening it would be real easy for you to track this back. Regards again, -- jhell,v