From owner-freebsd-questions@FreeBSD.ORG Sun Apr 29 15:16:33 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 747AF1065687 for ; Sun, 29 Apr 2012 15:16:33 +0000 (UTC) (envelope-from aimass@yabarana.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 302D08FC19 for ; Sun, 29 Apr 2012 15:16:33 +0000 (UTC) Received: by iahk25 with SMTP id k25so4332705iah.13 for ; Sun, 29 Apr 2012 08:16:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=5C0Pse/I+IZksAS50AteWNsbR6piuKI8GEWM4ndYnvY=; b=dWwb54inBBY0yJqFB92sudADQ0J7B8iz9jCX95pQTGwvoDPx/qoiiol0D86QsS0h4d Qcm8x6xed3QSiL6yg52F3Ue71dC58+JxpmvlKupEu8gGG0ziwdvdgQemVXBgTFS6sZdf dcG98jP8R8+Dftml9uxxkGxOPFdAKEyWpxH/UxwXqqL5ffuUHDYgZj7r8XvjxLeoowXs sIbVBy4wlgbTmxYfnc16Lz0KMaL8fQRnqu6xpG7+GyLCx1XYCKNFs6l5F19zteFeGFhn 4RWv+/yJ7RWrlWl/cqsDMkG+UE2cEb6JrMUsjQkt6PZj9/uFZEoaUNlIj8ErdLLWfQM0 JbFw== MIME-Version: 1.0 Received: by 10.50.89.200 with SMTP id bq8mr8273150igb.45.1335712592197; Sun, 29 Apr 2012 08:16:32 -0700 (PDT) Sender: aimass@yabarana.com Received: by 10.231.74.138 with HTTP; Sun, 29 Apr 2012 08:16:32 -0700 (PDT) In-Reply-To: <20120429103740.aa7df743.freebsd@edvax.de> References: <201204281731.q3SHVaiM061997@mail.r-bonomi.com> <20120428200116.b2f5820e.freebsd@edvax.de> <4f9ced3a.f7WBDlsMkhxvy+eF%perryh@pluto.rain.com> <20120429103740.aa7df743.freebsd@edvax.de> Date: Sun, 29 Apr 2012 11:16:32 -0400 X-Google-Sender-Auth: s246nhI3dEl4yso7WsW4Pg2G-H0 Message-ID: From: Alejandro Imass To: Polytropon Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQngkeOy3KEH86udja7sehloYlszdmYESGqPsvps/CBB8NsLzZZNNR7NsFD40bvMMrNU5kFb Cc: perryh@pluto.rain.com, freebsd-questions@freebsd.org Subject: Re: UFS Crash and directories now missing X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Apr 2012 15:16:33 -0000 On Sun, Apr 29, 2012 at 4:37 AM, Polytropon wrote: > On Sun, 29 Apr 2012 00:26:50 -0700, perryh@pluto.rain.com wrote: >> Alejandro Imass wrote: >> >> > 3) the directories were moved at reboot by journal recovery, >> > fsck or something else >> >> I think it's *extremely* unlikely that fsck was involved, because >> it just doesn't do things like that. > > The point is: fsck moving directories "looks different". In > case inodes get "de-connected" (their reference entries on > level n-1 are gone, their data on level n is still present), > fsck will access the lost+found/ directory in the corresponding > partition's root directory (or create it, if not present) and > write _new_ directory entries with the inode as their name, > because that's the only naming information possible (as the > original names on n-1 aren't accessible anymore). So those > directories will have names like #177628676/ and they _can_ > contain subtrees full of data, _including_ names from levels > n+1 and onward. Files also are named #4767667892 and their > names can _maybe_ identified from their content (the "file" > command is helpful, and if they are textfiles containing > a CVS or other revision control system data tag, it's possible > to find out what they've been in their previous life). > > However, as it has been explained, fsck will _not_ do so > unless being _allowed explicitely_ to do that kind of > MODIFICATION to the file system. Flags like -yf can do > that, but they are _not_ the default. This is due to the > fact that _any_ critical modification of file systems > requires the _responsible administrator_ to give permission. > OK, so fsck couldn't have done this. Besides fsck reported the fs as clean so I have to conclude as others have commented that it must have been a mv I've been looking at the logs very carefully and trying to make sense of this. There is a possibility that it could have been an attack because we enabled ftp.proxy so that some clients could upload stuff to their jails using ftp. So I was initially wrong in my assessment because on this particular server we are running a service outside of jails and it's this ftp.proxy that was suppose to be a temporary solution but I guess we never got around to fixing this. The ftp.proxy is started via inetd like so: ftp stream tcp nowait nobody /usr/local/sbin/ftp.proxy ftp.proxy -e And there was a log of a couple of ftp connections the same day this happened, the ONLY 3 messages before the reboot at about 6 pm and they were NOT from any of our customers. Here are the log entries: Apr 27 05:54:37 nune ftp.proxy[2726]: connected to client: host-46-50-183-5.bbcustomer.zsttk.net, interface= 207.158.52.74:21 Apr 27 05:54:37 nune ftp.proxy[2726]: info: monitor mode: off, ccp: Apr 27 05:54:38 nune ftp.proxy[2726]: -ERR: missing hostname Apr 27 18:55:42 nune syslogd: kernel boot file is /boot/kernel/kernel OK. So let's suppose ftp.proxy is the culprit is there any way the could have done the mv by cracking ftp and ftp.proxy ?? I have of course disabled the ftp and I am now thinking that another possibility or combination by also using the ftp proxy on the http-proxy jail, that is, the jail that swallowed the other jails. The http-proxy jails was also running apache ftp proxy. So the question now becomes: could a break in ftp.proxy coupled with Apache ftp proxy have caused the http-proxy jails to have swallowed all the other jails into it's configuration directory?? -- Alejandro Imass