From owner-freebsd-questions@FreeBSD.ORG Sun Nov 23 15:54:22 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1698216A4CE for ; Sun, 23 Nov 2003 15:54:22 -0800 (PST) Received: from fw.farid-hajji.net (fw.farid-hajji.net [213.146.115.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F08D43FD7 for ; Sun, 23 Nov 2003 15:54:20 -0800 (PST) (envelope-from cpghost@cordula.ws) Received: from fw.farid-hajji.net (localhost [127.0.0.1]) by fw.farid-hajji.net (8.12.10/8.12.10) with ESMTP id hANNrchQ019370 for ; Mon, 24 Nov 2003 00:53:40 +0100 (CET) (envelope-from cpghost@cordula.ws) Date: Mon, 24 Nov 2003 00:53:38 +0100 (CET) Message-Id: <200311232353.hANNrchQ019370@fw.farid-hajji.net> From: "Cordula's Web" To: freebsd-questions@freebsd.org In-reply-to: <441xryznvf.fsf@be-well.ilk.org> (message from Lowell Gilbert on 23 Nov 2003 18:06:12 -0500) X-Mailer: Emacs-21.3.1/FreeBSD-4.9-STABLE References: <200311222258.hAMMwApd092388@fw.farid-hajji.net> <16320.5175.69241.145102@jerusalem.litteratus.org> <20031123103544.GD9494@happy-idiot-talk.infracaninophile.co.uk> <441xryznvf.fsf@be-well.ilk.org> Subject: Re: Monitoring a file? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: cpghost@cordula.ws List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2003 23:54:22 -0000 > > I've finally found the culprit with a traditional method: > > * md5 (binary from an uncompromised machine) on all files > > * reinstalling from scratch (not buildworld, but really > > installing from FTP) > > * md5 again and diff. > > [snip] > > > Ugh... system clean again at last. :) > > You can't be sure. The attacker probably put an suid binary somewhere > besides the normal system binaries, in which case it's still there and > you may still be vulnerable. When you know you've been hacked, you > need to wipe the disk and *really* reinstall from scratch. And be > very careful about what you restore from backups, too. I've inherited a set of 280 1U rack mount boxes, and I am in the process of reinstalling from scratch every single server. Started with infrastructure (DNS and firewalls), then working down to every server with a fresh FTP install from the first recovered box. Yes, newfs everything, and recompiling _all_ binaries from scratch. I even reconfigured VLANs on the switches to avoid man-in-the-middle attacks like tcp hijacking, while ftp installing, and locked the subnets to these racks until everything's restored. The only backups were databases in SQL and LDIF format and lots of text data. No binaries and no compromised sources to recover from. Of course, the data could've been hacked too, but that would take more time to fix. I've only checked (and cleaned!) authorization and authentication data so far. Sometimes, small incidents trigger major reconfigurations. Good that this happened before monday! ;) Thank you. -- Cordula's Web. http://www.cordula.ws/