From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 5 18:39:15 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63EEC1065696 for ; Sun, 5 Oct 2008 18:39:15 +0000 (UTC) (envelope-from chuckr@telenix.org) Received: from mail2.sea5.speakeasy.net (mail2.sea5.speakeasy.net [69.17.117.4]) by mx1.freebsd.org (Postfix) with ESMTP id 483F58FC14 for ; Sun, 5 Oct 2008 18:39:15 +0000 (UTC) (envelope-from chuckr@telenix.org) Received: (qmail 29178 invoked from network); 5 Oct 2008 18:39:14 -0000 Received: from april.chuckr.org (HELO april.telenix.org) (chuckr@[66.92.151.30]) (envelope-sender ) by mail2.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 5 Oct 2008 18:39:14 -0000 Message-ID: <48E90904.4020007@telenix.org> Date: Sun, 05 Oct 2008 14:35:48 -0400 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: FreeBSD-Hackers X-Enigmail-Version: 0.95.5 OpenPGP: id=F3DCA0E9; url=http://pgp.mit.edu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: touch screen recommendation? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Oct 2008 18:39:15 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I was wondering if anyone here had a recommendation for a touch screen, specifically to run on FreeBSD? Any user report? Thanks -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjpCQQACgkQz62J6PPcoOl5MwCeKJAkbeQ/+GGpWX/b2CAV/F8J c1sAn0d7tXct21/SfdAYY2otfvFVuqK2 =XMw8 -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 5 19:03:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90C4E106569D; Sun, 5 Oct 2008 19:03:14 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id A185F8FC27; Sun, 5 Oct 2008 19:03:13 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=DwFIAmjFoFO6hpRAxLR3PmaRFZJNPkY7IYAUpc4ZmBDTqHMErBxnlQA8TdZkWmMeFgUZeTar2xzXxfCrRuL61aOS3de15CTpMxr/cJQczWiy0RrTk6XnPznhuH+D4lkbHyGMEeP510VG61xeR2MdqbJ3Z/JnR3tdGp9XItr3h/8=; Received: from amnesiac.at.no.dns (ppp83-237-107-240.pppoe.mtu-net.ru [83.237.107.240]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmYsm-000Pxu-KF; Sun, 05 Oct 2008 23:03:11 +0400 Date: Sun, 5 Oct 2008 23:03:17 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aPdhxNJGSeOG9wFI" Content-Disposition: inline In-Reply-To: <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, Roman Kurakin , freebsd-ports@freebsd.org, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Oct 2008 19:03:14 -0000 --aPdhxNJGSeOG9wFI Content-Type: multipart/mixed; boundary="X3gaHHMYHkYqP6yf" Content-Disposition: inline --X3gaHHMYHkYqP6yf Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Miroslav, good day. Sun, Sep 28, 2008 at 04:14:24PM +0400, Eygene Ryabinkin wrote: > > If I read nightly security e-mail with for example 4 vulnerable=20 > > packages, then I need to log in to server and manualy try, if newer=20 > > (fixed) packages are available. It seems not so hard to check output of= =20 > > `pkg_version -vIL =3D` and compare both versions (installed and availab= le)=20 > > with portaudit in some shellscript, I didn't start to write it yet ;). >=20 > I think it won't be very hard: I'll try to see how to extend portaudit > with such functionality -- it would be very handy, in my opinion. OK, I extended portaudit with this -- flag '-n' will do it. Currently this option requires network access, but I think that it is perfectly fits into the security check -- it downloads auditfile anyway. I had greatly reworked the old part of patch and I have series of 4 patches that implement both my pkg_audit stuff and the '-n' stuff. I am also attaching the mega-patch, it can be applied to the current port sources to give the port version that includes both mentioned enchancements. If you have no pkg_audit -- this isn't a problem: portaudit fill fall back to the awk script. I had also changed the output format for pkg_audit, so I am attaching another version of the second patch for the pkg_install bundle. I had briefly tested my modifications -- they work for now, but I will continue testing. Any bug reports or thoughts about these patches are more that welcome. > Hadn't you have a chance to test my patch? Miroslav, still: had you tested the pkg_audit thingy? --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="0002-New-utility-pkg_audit.patch" Content-Transfer-Encoding: quoted-printable =46rom 88a3659c2d941e27de698fe05e4852a9f418f16e Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Tue, 26 Aug 2008 15:08:46 +0400 Subject: [PATCH] New utility: pkg_audit It is mainly a helper for portupgrade to avoid awk scripting and numerous calls for pkg_info. This utility speeds up portaudit by a factor of 10 on a system with 521 installed ports and the auditfile that contains 3213 entries: ----- $ ls -d /var/db/pkg/* | wc -l 521 $ tar xOf /var/db/portaudit/auditfile.tbz auditfile | sed -e'/^#/d' | wc -l 3213 $ time ./portaudit Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- DNS spoofing vulnerability. Reference: Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- DoS vulnerability in WEBrick. Reference: Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- multiple vulnerabilities in safe level. Reference: 3 problem(s) in your installed packages found. You are advised to update or deinstall the affected package(s) immediately. real 0m0.107s user 0m0.116s sys 0m0.012s $ time portaudit Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- multiple vulnerabilities in safe level. Reference: Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- DoS vulnerability in WEBrick. Reference: Affected package: ruby-1.8.6.111_4,1 Type of problem: ruby -- DNS spoofing vulnerability. Reference: 3 problem(s) in your installed packages found. You are advised to update or deinstall the affected package(s) immediately. real 0m1.583s user 0m0.560s sys 0m1.057s ----- Signed-off-by: Eygene Ryabinkin --- Makefile | 2 +- audit/Makefile | 14 +++ audit/audit.h | 43 +++++++++ audit/main.c | 166 ++++++++++++++++++++++++++++++++++ audit/parse.c | 259 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++ audit/pkg_audit.1 | 63 +++++++++++++ 6 files changed, 546 insertions(+), 1 deletions(-) create mode 100644 audit/Makefile create mode 100644 audit/audit.h create mode 100644 audit/main.c create mode 100644 audit/parse.c create mode 100644 audit/pkg_audit.1 diff --git a/Makefile b/Makefile index fefbd08..abc1e65 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ =20 .include =20 -SUBDIR=3D lib add create delete info updating version +SUBDIR=3D lib add create delete info updating version audit =20 .include =20 diff --git a/audit/Makefile b/audit/Makefile new file mode 100644 index 0000000..2ece5f8 --- /dev/null +++ b/audit/Makefile @@ -0,0 +1,14 @@ +# $FreeBSD$ + +PROG=3D pkg_audit +SRCS=3D main.c parse.c + +CFLAGS+=3D -I${.CURDIR}/../lib + +WARNS?=3D 6 +WFORMAT?=3D 1 + +DPADD=3D ${LIBINSTALL} ${LIBFETCH} ${LIBMD} +LDADD=3D ${LIBINSTALL} -lfetch -lmd + +.include diff --git a/audit/audit.h b/audit/audit.h new file mode 100644 index 0000000..1f0a369 --- /dev/null +++ b/audit/audit.h @@ -0,0 +1,43 @@ +/* + * + * FreeBSD install - a package for the installation and maintainance + * of non-core utilities. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * Eygene Ryabinkin + * 26 August 2008 + * + * Parsing module for pkg_audit, header file. + * + */ + +#ifndef __AUDIT_H__ +#define __AUDIT_H__ + +#include + +SLIST_HEAD(audit_contents, audit_entry); + +struct audit_entry { + char *pkgglob; /* Package name glob */ + char *url; /* URL of advisory */ + char *descr; /* Description of vulnerability */ + size_t pfx_size; /* Metacharacter-less glob part size */ + SLIST_ENTRY(audit_entry) entries; +}; + + +/* Function prototypes */ +int +parse_auditfile(FILE *_fp, struct audit_contents *_head); + + +#endif /* defined(__AUDIT_H__) */ diff --git a/audit/main.c b/audit/main.c new file mode 100644 index 0000000..6ad18a9 --- /dev/null +++ b/audit/main.c @@ -0,0 +1,166 @@ +/* + * + * FreeBSD install - a package for the installation and maintainance + * of non-core utilities. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * Eygene Ryabinkin + * 26 August 2008 + * + * This is the audit module -- fast helper for the portaudit script. + * + * It is filter-like utility: it reads the audit file from the + * standard input, intersects the vulnerable port list with the + * packages installed in the system and outputs the entries for + * the vulnerable ports that are present in the system to the + * standard output. + * + * The installed package can be listed multiple times, since it + * can be vulnerable to more than one bug at a time. But the + * whole output entries will be unique -- package name and + * vulnerability details should produce unique entry. + * + * One more field is prepended to the list of the input fields -- + * the name of the matched port. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#ifdef PROFILING +#include +#endif + +#include +#include +#include +#include +#include + +#include "lib.h" +#include "audit.h" + +static inline void +audit_package(const char *_pkgname, struct audit_contents *_head, + struct match_session *_msess, FILE *_fp); + +int +main(int argc, char *argv[]) +{ + char freebsd[sizeof("FreeBSD-XXYYZZXXYYZZ")]; + unsigned long reldate; + size_t reldate_size =3D sizeof(reldate); + int mib[2]; + + FILE *in =3D stdin, *out =3D stdout; + struct match_session *msess; + struct audit_entry *item; +#ifdef PROFILING + struct timeval t1, t2; + double dt; +#endif + + /* Make compiler happy */ + if (argv[argc] =3D=3D NULL) {}; + + struct audit_contents head =3D + SLIST_HEAD_INITIALIZER(head); + + mib[0] =3D CTL_KERN; + mib[1] =3D KERN_OSRELDATE; + if (sysctl(mib, 2, (void *)&reldate, &reldate_size, NULL, 0) !=3D 0) + errx(1, "Unable to get kern.osreldate"); + snprintf(freebsd, sizeof(freebsd), "%s-%lu", "FreeBSD", reldate); +=09 + SLIST_INIT(&head); +#ifdef PROFILING + gettimeofday(&t1, NULL); +#endif + if (parse_auditfile(in, &head) !=3D 0) { + errx(1, "Failed to parse audit entries"); + } +#ifdef PROFILING + gettimeofday(&t2, NULL); + dt =3D t2.tv_sec - t1.tv_sec + 1e-6 * (t2.tv_usec - t1.tv_usec); + fprintf(stderr, "parse_auditfile(): %.6lf sec\n", dt); +#endif + + msess =3D match_begin(MATCH_GLOB); + if (msess =3D=3D NULL) + return 1; +=09 +#ifdef PROFILING + gettimeofday(&t1, NULL); +#endif + + /* Special check: FreeBSD itself */ + SLIST_FOREACH (item, &head, entries) { + if (strncmp(item->pkgglob, + "FreeBSD", sizeof("FreeBSD") - 1) =3D=3D 0 && + pattern_match(MATCH_GLOB, item->pkgglob, freebsd)) { + fprintf(out, "%s|%s|%s\n", + freebsd, item->url, item->descr); + } + } + + /* Installed packages */ + while (match_next_package(msess)) + audit_package(match_get_pkgname(msess), &head, msess, out); + +#ifdef PROFILING + gettimeofday(&t2, NULL); + dt =3D t2.tv_sec - t1.tv_sec + 1e-6 * (t2.tv_usec - t1.tv_usec); + fprintf(stderr, "match loop: %.6lf sec\n", dt); +#endif + + match_end(msess); + + SLIST_FOREACH (item, &head, entries) { + free((void *)item->pkgglob); + free((void *)item); + } + + return 0; +} + +void +cleanup(int sig) +{ + sig =3D 0; + return; +} + +/* + * Loops over audit file contents and checks each entry in turn. + * + * The great speedup is to test the package prefix at the first + * place and only if it matches perform full match -- match_matches + * uses slow matching routines without precompilation and other + * tricks. For hundreds of installed ports and a couple of thousands + * audit entries this slows things down very well. + */ +static inline void +audit_package(const char *pkgname, struct audit_contents *head, + struct match_session *msess, FILE *fp) +{ + struct audit_entry *item; + + SLIST_FOREACH (item, head, entries) { + if (strncmp(pkgname, item->pkgglob, + item->pfx_size) =3D=3D 0 && + match_matches(msess, item->pkgglob)) { + fprintf(fp, "%s|%s|%s|%s\n", + pkgname, + item->pkgglob, item->url, item->descr); + } + } +} diff --git a/audit/parse.c b/audit/parse.c new file mode 100644 index 0000000..fb33f7c --- /dev/null +++ b/audit/parse.c @@ -0,0 +1,259 @@ +/* + * + * FreeBSD install - a package for the installation and maintainance + * of non-core utilities. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * Eygene Ryabinkin + * 26 August 2008 + * + * Parsing module for pkg_audit. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include + +#include "lib.h" +#include "audit.h" + +/* Simple exponentially-growing buffer. */ + +struct dyn_buffer { + char *buf; + size_t size; +}; + +/* Prototypes */ +static int +parse_audit_entry(struct dyn_buffer *_b, struct audit_entry *_e); + +static int +read_line(FILE *_fp, struct dyn_buffer *_b); + +static struct dyn_buffer * +buf_init(size_t _size); +static void +buf_destroy(struct dyn_buffer *_b); +static int +buf_grow(struct dyn_buffer *_b); + +/* + * Parses audit file to the linked list of single entries. + * + * Return values: + * 0 -- file was successfully parsed; + * 1 -- parsing or read error occured; + * -1 -- bad arguments, memory allocation problems, etc. + */ +int +parse_auditfile(FILE *fp, struct audit_contents *head) +{ + struct audit_entry *e; + struct dyn_buffer *b; + int errcode; + + b =3D buf_init(256); + if (b =3D=3D NULL) + return 1; + + while ((errcode =3D read_line(fp, b)) =3D=3D 0) { + if (b->buf[0] =3D=3D '#') + continue; + e =3D (struct audit_entry *)malloc(sizeof(*e)); + if (e =3D=3D NULL) { + buf_destroy(b); + return -1; + } + bzero((void *)e, sizeof(*e)); + if ((errcode =3D parse_audit_entry(b, e)) !=3D 0) { + buf_destroy(b); + return errcode; + } + SLIST_INSERT_HEAD(head, e, entries); + } + + buf_destroy(b); + + if (errcode !=3D 1) + return 1; + else + return 0; +} + +/* + * Helpers for audit file parsing routine. + */ + +static struct dyn_buffer * +buf_init(size_t size) +{ + struct dyn_buffer *b; + + if (size <=3D 0) + return NULL; + + b =3D (struct dyn_buffer *)malloc(sizeof(*b)); + if (b =3D=3D NULL) + return NULL; + + bzero((void *)b, sizeof(*b)); + b->size =3D size; + b->buf =3D (char *)malloc(b->size * sizeof(b->buf[0])); + if (b->buf =3D=3D NULL) { + free((void *)b); + return NULL; + } + + bzero((void *)b->buf, b->size); + return b; +} + +static void +buf_destroy(struct dyn_buffer *b) +{ + if (b =3D=3D NULL) + return; +=09 + if (b->buf !=3D NULL) + free((void *)b->buf); + free((void *)b); + return; +} + +static int +buf_grow(struct dyn_buffer *b) +{ + char *newbuf; + + if (b =3D=3D NULL || b->buf =3D=3D NULL || b->size <=3D 0) + return -1; + + newbuf =3D (char *)malloc(2 * b->size * sizeof(newbuf)); + if (newbuf =3D=3D NULL) + return 1; + + bzero(newbuf, 2 * b->size); + bcopy((void *)b->buf, (void *)newbuf, b->size); + b->buf =3D newbuf; + b->size *=3D 2; + + return 0; +} + +/* + * fgets()-like function that reads the whole input line + * Returns 0 on the successful read, 1 for the end-of-file + * condition, -1 for any error. + * + * Terminating '\n' is removed from the line. + */ +static int +read_line(FILE *fp, struct dyn_buffer *b) +{ + size_t offset =3D 0, len =3D 0; + + if (fp =3D=3D NULL || b =3D=3D NULL) + return -1; + + if (feof(fp)) + return 1; + + /* We need at least two-character buffer */ + if (b->size =3D=3D 1 && buf_grow(b) !=3D 0) + return -1; + + b->buf[b->size - 1] =3D '\0'; + offset =3D 0; + while (fgets(b->buf + offset, b->size - offset, fp) !=3D NULL) { + len =3D strlen(b->buf); + /* + * Read zero characters or buffer even shrinked? + * Strange, let's indicate error. + */ + if (len <=3D offset) + return -1; + if (b->buf[len - 1] =3D=3D '\n') { + b->buf[len - 1] =3D '\0'; + return 0; + } + + offset =3D len; + if (buf_grow(b) !=3D 0) + return -1; + + /* Should not happen, but who knows */ + if (offset >=3D b->size) + return -1; + } + + if (feof(fp)) { + /* + * If we read no characters, if means that we were + * at the EOF, but it was detected only by fgets(), + * not the first feof(). + */ + if (len =3D=3D 0) + return 1; + else + return 0; + } else { + return -1; + } +} + +/* + * Parses single audit line and places it to the structure. + * Calculates length of the package name suffix that is free + * from metacharacters -- it is used for the quick matches + * against port name. + */ +static int +parse_audit_entry(struct dyn_buffer *b, struct audit_entry *e) +{ + size_t len; + char *string =3D NULL, *d1 =3D NULL, *d2 =3D NULL; + static const char globset[] =3D "{*?><=3D!"; + + /* + * At least 5 characters: + * two delimiters and three non-empty fields. + */ + len =3D strlen(b->buf); + if (len < 5) + return 1; + + /* Locate delimiters. */ + d1 =3D strchr(b->buf, '|'); + if (d1 =3D=3D NULL) + return 1; + d2 =3D strchr(d1 + 1, '|'); + if (d2 =3D=3D NULL) + return 1; + + string =3D (char *)malloc((len + 1) * sizeof(string[0])); + if (string =3D=3D NULL) + return -1; + + bcopy((void *)b->buf, (void *)string, (len + 1) * sizeof(string[0])); + string[d1 - b->buf] =3D '\0'; + string[d2 - b->buf] =3D '\0'; + e->pkgglob =3D string; + e->url =3D string + (d1 - b->buf) + 1; + e->descr =3D string + (d2 - b->buf) + 1; + e->pfx_size =3D strcspn(e->pkgglob, globset); + + return 0; +} diff --git a/audit/pkg_audit.1 b/audit/pkg_audit.1 new file mode 100644 index 0000000..cd4abbc --- /dev/null +++ b/audit/pkg_audit.1 @@ -0,0 +1,63 @@ +.\" +.\" FreeBSD install - a package for the installation and maintenance +.\" of non-core utilities. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" Eygene Ryabinkin +.\" +.\" +.\" @(#)pkg_audit.1 +.\" $FreeBSD$ +.\" +.Dd Aug 26, 2008 +.Dt PKG_AUDIT 1 +.Os +.Sh NAME +.Nm pkg_audit +.Nd lists vulnerable ports installed in the system +.Sh SYNOPSIS +.Nm +.Sh DESCRIPTION +The +.Nm +command is used to extract vulnerability information from the audit +file and list vulnerable packages that are present in the system. +It is main purpose to help +.Xr portaudit 1 +utility to avoid time-consuming scripting. +.Nm +reads vulnerability information from the standard input and writes +the list of vulnerable ports to the standard output. +Format of the output lines is the same as for the audit file, but +package matching globs are substituted with the actual package names. +.Sh TECHNICAL DETAILS +First the audit file is parsed to the internal representation +(currently it is linked list). +Then we are traversing installed packages database and trying to +match the package name against each audit entry. +The crucial step for the speeding up the process is to first +match the package prefix that has no CSH-like metacharacters +and perform full comparison only if match is found. +One more name is tested prior to the installed packages: it is +.Qo FreeBSD-`sysctl -n kern.osreldate` Qc , +the version of +.Fx +the current system is running. +.Sh SEE ALSO +.Xr portaudit 1 , +.Xr pkg_add 1 , +.Xr pkg_create 1 , +.Xr pkg_delete 1 , +.Xr pkg_version 1 . +.Sh AUTHORS +.An Eygene Ryabinkin Aq rea-fbsd@codelabs.ru +.Sh BUGS +Sure to be some. --=20 1.6.0.2 --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch" Content-Transfer-Encoding: quoted-printable =46rom b5fc2033e39aecd8b65f3bda45bf71572b72262a Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Sun, 5 Oct 2008 18:04:10 +0400 Subject: [PATCH 1/4] Avoid usage of global variables $N in the print_affect= ed I will need this functionality in the next commits: some invocations of print_affected will be called with shifted variables $N. But this is good anyway: globals are always been bad and now functions are looking more sanely -- one does not need to guess the meaning of $1, $2 and $3. Signed-off-by: Eygene Ryabinkin --- files/portaudit-cmd.sh | 68 ++++++++++++++++++++++++--------------------= --- 1 files changed, 35 insertions(+), 33 deletions(-) diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh index c0eb67b..4583c9c 100755 --- a/files/portaudit-cmd.sh +++ b/files/portaudit-cmd.sh @@ -140,7 +140,8 @@ audit_installed() $1 ~ /^FreeBSD[<=3D>!]/ { if (fixedre && $2 ~ fixedre) next if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\""= )) { - print_affected("FreeBSD-'"$osversion"'", \ + print_affected("FreeBSD-'"$osversion"'",=20 + $1, $2, $3, \ "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PR= EFIX%%/etc/portaudit.conf") } next @@ -152,7 +153,7 @@ audit_installed() cmd=3D"'"$pkg_info"' -E \"" $1 "\"" while((cmd | getline pkg) > 0) { vul++ - print_affected(pkg, "") + print_affected(pkg, $1, $2, $3, "") } close(cmd) } @@ -207,7 +208,7 @@ audit_file() if ($2 !~ /'"$opt_restrict"'/) continue vul++ - print_affected(pkg, "") + print_affected(pkg, $1, $2, $3, "") } close(cmd) } @@ -244,7 +245,8 @@ audit_args() ' | $pkg_version -T "$1" -`; then VULCNT=3D$(($VULCNT+1)) echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' - { print_affected("'"$1"'", "") } + { print_affected("'"$1"'", + $1, $2, $3, "") } ' fi ;; @@ -277,7 +279,7 @@ audit_cwd() { print } ' | $pkg_version -T "$PKGNAME" -`; then echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' - { print_affected("'"$PKGNAME"'", "") } + { print_affected("'"$PKGNAME"'", $1, $2, $3, "") } ' return 1 fi @@ -425,38 +427,38 @@ prerequisites_checked=3Dfalse =20 if $opt_quiet; then PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - print apkg - } - ' +function print_affected(apkg, glob, refs, descr, note) { + print apkg +} +' elif $opt_verbose; then PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - split(apkg, thepkg) - print "Affected package: " thepkg[1] " (matched by " $1 ")" - print "Type of problem: " $3 "." - split($2, ref, / /) - for (r in ref) - print "Reference: <" ref[r] ">" - if (note) - print "Note: " note - print "" - } - ' +function print_affected(apkg, glob, refs, descr, note) { + split(apkg, thepkg) + print "Affected package: " thepkg[1] " (matched by " glob ")" + print "Type of problem: " descr "." + split(refs, ref, / /) + for (r in ref) + print "Reference: <" ref[r] ">" + if (note) + print "Note: " note + print "" +} +' else PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - split(apkg, thepkg) - print "Affected package: " thepkg[1] - print "Type of problem: " $3 "." - split($2, ref, / /) - for (r in ref) - print "Reference: <" ref[r] ">" - if (note) - print "Note: " note - print "" - } - ' +function print_affected(apkg, glob, refs, descr, note) { + split(apkg, thepkg) + print "Affected package: " thepkg[1] + print "Type of problem: " descr "." + split(refs, ref, / /) + for (r in ref) + print "Reference: <" ref[r] ">" + if (note) + print "Note: " note + print "" +} +' fi =20 if $opt_audit; then --=20 1.6.0.2 --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="0002-Separate-vulnerable-ports-search-from-the-formatter.patch" Content-Transfer-Encoding: quoted-printable =46rom b595cf7da5b81489b5e21d85df4dd8e79f3a0b1f Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Sun, 5 Oct 2008 18:56:13 +0400 Subject: [PATCH 2/4] Separate vulnerable ports search from the formatter ro= utine I am planning to insert some other filters between the code that outputs the vulnerable port entries and the code that formats the messages about vulnerabilities that had been found. Such a split provides a big flexibility: one can insert more filters that will transform entries and one can substitute search routine for something else (I am planning to substitute it with the binary utility written in a compiled language). This was done only for the routine that checks the installed ports, because it is my primary target for now. May be later I will split other auditing functions in the same way. Signed-off-by: Eygene Ryabinkin --- files/portaudit-cmd.sh | 101 ++++++++++++++++++++++++++++++--------------= ---- 1 files changed, 63 insertions(+), 38 deletions(-) diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh index 4583c9c..5fc0d1d 100755 --- a/files/portaudit-cmd.sh +++ b/files/portaudit-cmd.sh @@ -125,50 +125,75 @@ portaudit_prerequisites() return 0 } =20 +# +# Helper for audit_installed that actually finds vulnerable packages. +# +# It processes the auditfile entries (that are read from the stdin) +# in the form "glob|refs|desc" and outputs entries in the form +# "pkgname|glob|refs|desc", where "pkgname" is the matched package name. +# +findvuln_installed() +{ + local fixedre=3D`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' |= tr -s ' \t\n' '|'` + local installedre=3D`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d = '|' -` + local osversion=3D`sysctl -n kern.osreldate` + + awk -F\| \ + -v fixedre=3D"$fixedre" -v installedre=3D"$installedre" \ + -v pkg_version=3D"$pkg_version" -v pkg_info=3D"$pkg_info" \ + -v osversion=3D"$osversion" -v opt_restrict=3D"$opt_restrict" \ + ' +/^(#|\$)/ { next } +opt_restrict && $2 !~ opt_restrict { next } +$1 ~ /^FreeBSD[<=3D>!]/ { + if (fixedre && $2 ~ fixedre) next + if (!system(pkg_version " -T \"FreeBSD-" osversion "\" \"" $1 "\"")) { + printf("FreeBSD-%s|%s\n", osversion, $0); + } + next +} +$1 ~ /^[^{}*?]*[<=3D>!]/ { + if ($1 !~ "^(" installedre ")[<=3D>!]") next; +} +{ + cmd=3Dpkg_info " -E \"" $1 "\"" + while((cmd | getline pkg) > 0) { + printf("%s|%s\n", pkg, $0); + } + close(cmd) +} +' +} + audit_installed() { local rc=3D0 local osversion=3D`sysctl -n kern.osreldate` =20 - fixedre=3D`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s= ' \t\n' '|'` - installedre=3D`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -` - - extract_auditfile | awk -F\| "$PRINTAFFECTED_AWK"' - BEGIN { vul=3D0; fixedre=3D"'"$fixedre"'" } - /^(#|\$)/ { next } - $2 !~ /'"$opt_restrict"'/ { next } - $1 ~ /^FreeBSD[<=3D>!]/ { - if (fixedre && $2 ~ fixedre) next - if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\""= )) { - print_affected("FreeBSD-'"$osversion"'",=20 - $1, $2, $3, \ - "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PR= EFIX%%/etc/portaudit.conf") - } - next - } - $1 ~ /^[^{}*?]*[<=3D>!]/ { - if ($1 !~ "^('"$installedre"')[<=3D>!]") next; - } - { - cmd=3D"'"$pkg_info"' -E \"" $1 "\"" - while((cmd | getline pkg) > 0) { - vul++ - print_affected(pkg, $1, $2, $3, "") - } - close(cmd) - } - END { - if ("'$opt_quiet'" =3D=3D "false") { - print vul " problem(s) in your installed packages found." - } - if (vul > 0) { - if ("'$opt_quiet'" =3D=3D "false") { - print "\nYou are advised to update or deinstall" \ - " the affected package(s) immediately." - } - exit(1) - } + extract_auditfile | findvuln_installed | \ + awk -F\| "$PRINTAFFECTED_AWK"' +BEGIN { vul=3D0; } +$1 ~ /^FreeBSD-/ { + print_affected($1, $2, $3, $4, \ + "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFI= X%%/etc/portaudit.conf") + next +} +{ + print_affected($1, $2, $3, $4, ""); + vul++; +} +END { + if ("'$opt_quiet'" =3D=3D "false") { + print vul " problem(s) in your installed packages found." + } + if (vul > 0) { + if ("'$opt_quiet'" =3D=3D "false") { + print "\nYou are advised to update or deinstall" \ + " the affected package(s) immediately." } + exit(1) + } +} ' || rc=3D$? =20 return $rc --=20 1.6.0.2 --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="0003-Use-pkg_audit-utility-if-it-is-available.patch" Content-Transfer-Encoding: quoted-printable =46rom 100bf18057bb16f21f0058e31279b9c3730e097d Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Sun, 5 Oct 2008 19:20:34 +0400 Subject: [PATCH 3/4] Use pkg_audit utility if it is available Pkg_audit provides a good speed-up to the search of vulnerable packages within installed ones. It can be unavailable, so its usage is conditionalized. Signed-off-by: Eygene Ryabinkin --- files/portaudit-cmd.sh | 15 +++++++++++++++ files/portaudit.conf | 4 ++++ 2 files changed, 19 insertions(+), 0 deletions(-) diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh index 5fc0d1d..32c121d 100755 --- a/files/portaudit-cmd.sh +++ b/files/portaudit-cmd.sh @@ -43,6 +43,8 @@ portaudit_confs() =20 : ${portaudit_fixed=3D""} =20 + : ${portaudit_pkg_audit=3D"/usr/sbin/pkg_audit"} + if [ -r %%PREFIX%%/etc/portaudit.conf ]; then . %%PREFIX%%/etc/portaudit.conf fi @@ -135,6 +137,19 @@ portaudit_prerequisites() findvuln_installed() { local fixedre=3D`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' |= tr -s ' \t\n' '|'` + + if [ -x "${portaudit_pkg_audit}" ]; then + "${portaudit_pkg_audit}" | awk -F \| \ + -v fixedre=3D"$fixedre" -v opt_restrict=3D"$opt_restrict" \ + ' +opt_restrict && $3 !~ opt_restrict { next } +$2 ~ /^FreeBSD[<=3D>!]/ && fixedre && $3 ~ fixedre { next } +{ print } +' + + return + fi + local installedre=3D`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d = '|' -` local osversion=3D`sysctl -n kern.osreldate` =20 diff --git a/files/portaudit.conf b/files/portaudit.conf index 4eb08d7..9ea2c4a 100644 --- a/files/portaudit.conf +++ b/files/portaudit.conf @@ -17,3 +17,7 @@ =20 # this vulnerability has been fixed in your FreeBSD version #portaudit_fixed=3D"d2102505-f03d-11d8-81b0-000347a4fa7d" + +# this command will be used to find the vulnerable packages +# instead of awk(1) script. +# portaudit_pkg_audit=3D"/usr/sbin/pkg_audit" --=20 1.6.0.2 --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="0004-Implement-checking-for-a-new-package-versions.patch" Content-Transfer-Encoding: quoted-printable =46rom f4ef2c5c03ea9543ecdfc68a62b26449b8d492ed Mon Sep 17 00:00:00 2001 =46rom: Eygene Ryabinkin Date: Sun, 5 Oct 2008 22:27:53 +0400 Subject: [PATCH 4/4] Implement checking for a new package versions Flag '-n' tells portaudit to check if updated packages are available. It is currently done only for the 'all installed packages' mode, i.e. when '-a' flag is given. To do this, one additional utility, portaudit-checknew, is introduced. It downloads the ports INDEX file for the current FreeBSD version and checks if new versions of packages are available. This utility is spawned at the late stage when vulnerable packages are already determined and thus only they are checked. So the number of input items shouldn't be very large and portaudit-checknew is a plain shell script. Signed-off-by: Eygene Ryabinkin --- Makefile | 4 +- files/portaudit-checknew.sh | 159 +++++++++++++++++++++++++++++++++++++++= ++++ files/portaudit-cmd.sh | 67 +++++++++++++++--- files/portaudit.1 | 4 +- pkg-plist | 1 + 5 files changed, 222 insertions(+), 13 deletions(-) create mode 100755 files/portaudit-checknew.sh diff --git a/Makefile b/Makefile index 885dc27..6083235 100644 --- a/Makefile +++ b/Makefile @@ -37,7 +37,8 @@ SED_SCRIPT=3D -e 's|%%PREFIX%%|${PREFIX}|g' \ -e "s|%%BZIP2_CMD%%|${BZIP2_CMD}|g" \ =20 do-build: -.for f in portaudit-cmd.sh portaudit.sh portaudit.1 portaudit.conf +.for f in portaudit-cmd.sh portaudit-checknew.sh portaudit.sh \ + portaudit.1 portaudit.conf @${SED} ${SED_SCRIPT} ${FILESDIR}/${f} >${WRKDIR}/${f} .endfor =20 @@ -54,6 +55,7 @@ pre-install: =20 do-install: @${INSTALL_SCRIPT} ${WRKDIR}/portaudit-cmd.sh ${PREFIX}/sbin/portaudit + @${INSTALL_SCRIPT} ${WRKDIR}/portaudit-checknew.sh ${PREFIX}/sbin/portaud= it-checknew @${INSTALL_DATA} ${WRKDIR}/portaudit.conf ${PREFIX}/etc/portaudit.conf.sa= mple @${INSTALL_MAN} ${WRKDIR}/portaudit.1 ${MAN1PREFIX}/man/man1 @${MKDIR} ${PERIODICDIR}/security diff --git a/files/portaudit-checknew.sh b/files/portaudit-checknew.sh new file mode 100755 index 0000000..6bc7dfe --- /dev/null +++ b/files/portaudit-checknew.sh @@ -0,0 +1,159 @@ +#!/bin/sh +# $FreeBSD$ +# +# Script to check if newer port versions then the ones installed +# in the system are available. Uses downloaded ports index from +# the master site. +# +# Utility acts as a filter: reads input strings, processes them +# and outputs the result to the standard output. Each input string +# is treated as the collection of '|'-separated fields. First field +# should be the package name with version, other fields can have +# any contents -- they won't be touched. +# +# One more field can be added -- the newest port version, if it can +# be deduced. This field will be added to the end fieldset. +# +# For any error the utility will just output the unmodified input +# contents. +# +# Eygene Ryabinkin, rea-fbsd@codelabs.ru. September 2008. + +BASEURL=3Dhttp://www.freebsd.org/ports/INDEX- +PKGBASE=3D/var/db/pkg + +# Obtains FreeBSD major version number from uname output. +freebsd_major () { + uname -r | cut -f1 -d- | cut -f1 -d. +} + +# Returns the OS version for the INDEX file. It is just a wrapper +# for the freebsd_major routine that applies the fixups for the +# known unsupported versions. +index_version () { + local major + + major=3D`freebsd_major` + case "$major" in + 1|2|3|4) + major=3D5 + ;; + esac + echo "$major" +} + +# Guesses (installed) port origin. +# Arguments: +# - port name with version specification. +# - name of file wit +guess_origin () { + local contents origin + + if [ -z "$1" ]; then + echo "guess_origin(): called without arguments" >&2 + exit 255 + fi + if [ -z "$2" ]; then + echo "guess_origin(): called without second argument" >&2 + exit 255 + fi + + contents=3D"$PKGBASE"/"$1"/"+CONTENTS" + if [ -s "$contents" ]; then + origin=3D`grep '^@comment ORIGIN:' "$contents" | \ + sed -e's/^@comment ORIGIN://'` + if [ -n "$origin" ]; then + echo "$origin" + return + fi + fi + # Not yet implemented: loop over INDEX file and try to get + # the origin from the matched ports description. +} + +# Fetches bzipped port index file and outputs it to the stdout +# Arguments: +# - URL to get file from. +# +# Uses 'opt_cachedir' (if it is not empty) as the name of the directory +# to place the downloaded file to. Download is made in the mirror mode, +# so if file's timestamps and size are checked and no download takes +# place if the remote file has the same characteristics. +fetch_index () { + if [ -z "$1" ]; then + echo "fetch_index(): called without arguments" >&2 + exit 255 + fi + + if [ -z "$opt_cachedir" -o ! -d "$opt_cachedir" ]; then + fetch -qpo - "$1" + return + fi + + local outfile=3D"$opt_cachedir"/`basename "$1"` + fetch -mpo "$outfile" "$1" && cat "$outfile" +} + +TMPINDEX=3D`mktemp -q -t versionaudit-INDEX` +if [ -z "$TMPINDEX" ]; then + echo "Unable to create temporary file for ports index." >&2 + cat + exit 1 +fi + +trap "rm -f \"$TMPINDEX\"" 0 1 2 3 15 + +opt_cachedir=3D + +while getopts c: opt; do + case "$opt" in + c) + opt_cachedir=3D"$OPTARG" + ;; + ?) + echo "Usage: $0 [-c cachedir]" + exit 2 + ;; + esac +done + +url=3D"$BASEURL"`index_version`.bz2 +fetch_index "$url" | bunzip2 -c > "$TMPINDEX" +if ! [ -s "$TMPINDEX" ]; then + echo "Unable to download port index from $url" >&2 + cat + exit 1 +fi + +IFS=3D'|' +while read portspec rest; do + portname=3D`echo "$portspec" | sed -e's/-[^-]*$//'` + origin=3D`guess_origin "$portspec" "$TMPINDEX"` + if [ -z "$origin" ]; then + echo "Unable to get port origin for '$portspec'." >&2 + continue + fi + # The while cycle is hackish: we make "$?" non-zero if no + # matches were found and it is set to zero if match was + # found. All exit paths from the cycle should set "$?" + # properly. + # + # Another way to proceed is to make subroutine that will + # either print something or not, because we can't pass + # variables outside the while loop -- it is done in the + # separate process. + grep "^$portname-" "$TMPINDEX" | while read nportspec ndir nrest + do + nportname=3D`echo "$nportspec" | sed -e's/-[^-]*$//'` + norigin=3D`echo "$ndir" | sed -e's|^/[^/]*/[^/]*/||'` + if [ "$nportname" =3D "$portname" -a \ + "$norigin" =3D "$origin" ]; then + echo "$portspec|$rest|$nportspec" + true + break + fi + false + done || echo "$portspec|$rest" +done + +exit 0 diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh index 32c121d..0ddd3d3 100755 --- a/files/portaudit-cmd.sh +++ b/files/portaudit-cmd.sh @@ -127,6 +127,15 @@ portaudit_prerequisites() return 0 } =20 +checknew() +{ + if [ -n "$opt_checknew" ]; then + portaudit-checknew -c "${portaudit_dir}" + else + cat + fi +} + # # Helper for audit_installed that actually finds vulnerable packages. # @@ -185,16 +194,16 @@ audit_installed() local rc=3D0 local osversion=3D`sysctl -n kern.osreldate` =20 - extract_auditfile | findvuln_installed | \ - awk -F\| "$PRINTAFFECTED_AWK"' + extract_auditfile | findvuln_installed | checknew | \ + awk -F\| -v pkg_version=3D"$pkg_version" "$PRINTAFFECTED_AWK"' BEGIN { vul=3D0; } $1 ~ /^FreeBSD-/ { - print_affected($1, $2, $3, $4, \ + print_affected($1, $2, $3, $4, "", \ "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFI= X%%/etc/portaudit.conf") next } { - print_affected($1, $2, $3, $4, ""); + print_affected($1, $2, $3, $4, $5, ""); vul++; } END { @@ -248,7 +257,7 @@ audit_file() if ($2 !~ /'"$opt_restrict"'/) continue vul++ - print_affected(pkg, $1, $2, $3, "") + print_affected(pkg, $1, $2, $3, "", "") } close(cmd) } @@ -286,7 +295,7 @@ audit_args() VULCNT=3D$(($VULCNT+1)) echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' { print_affected("'"$1"'", - $1, $2, $3, "") } + $1, $2, $3, "", "") } ' fi ;; @@ -319,7 +328,7 @@ audit_cwd() { print } ' | $pkg_version -T "$PKGNAME" -`; then echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' - { print_affected("'"$PKGNAME"'", $1, $2, $3, "") } + { print_affected("'"$PKGNAME"'", $1, $2, $3, "", "") } ' return 1 fi @@ -389,12 +398,13 @@ opt_restrict=3D opt_verbose=3Dfalse opt_version=3Dfalse opt_expiry=3D +opt_checknew=3D =20 if [ $# -eq 0 ] ; then opt_audit=3Dtrue fi =20 -while getopts aCdf:Fqr:vVX: opt; do +while getopts aCdf:Fnqr:vVX: opt; do case "$opt" in a) opt_audit=3Dtrue;; @@ -406,6 +416,8 @@ while getopts aCdf:Fqr:vVX: opt; do opt_file=3D"$OPTARG";; F) opt_fetch=3Dtrue;; + n) + opt_checknew=3Dtrue;; q) opt_quiet=3Dtrue;; r) @@ -465,15 +477,42 @@ fi =20 prerequisites_checked=3Dfalse =20 +# This awk code demands 'pkg_version' variable to be set +# if 'newver' variable is non-empty. +NEWVERS_AWK=3D' + +nparts=3Dsplit(thepkg[1], verarr, /-/) +gtglob=3Dverarr[1] +for (i=3D2; i" verarr[i] + +updated=3D"" +avail=3D0 +if (system(pkg_version " -T \"" newver "\" '\''" gtglob "'\''") =3D=3D 0) { + updated=3D"available, " newver + avail=3D1 +} else { + updated=3D"not available" +} +if (avail !=3D 0) { + if (system(pkg_version " -T \"" newver "\" '\''" glob "'\''") =3D=3D 0) + updated=3Dupdated ", still vulnerable" + else + updated=3Dupdated ", not vulnerable" +} +print "Updated package: " updated +' + if $opt_quiet; then PRINTAFFECTED_AWK=3D' -function print_affected(apkg, glob, refs, descr, note) { +function print_affected(apkg, glob, refs, descr, newver, note) { print apkg } ' elif $opt_verbose; then PRINTAFFECTED_AWK=3D' -function print_affected(apkg, glob, refs, descr, note) { +function print_affected(apkg, glob, refs, descr, newver, note) { split(apkg, thepkg) print "Affected package: " thepkg[1] " (matched by " glob ")" print "Type of problem: " descr "." @@ -482,12 +521,15 @@ function print_affected(apkg, glob, refs, descr, note= ) { print "Reference: <" ref[r] ">" if (note) print "Note: " note + if (newver) { +'"$NEWVERS_AWK"' + } print "" } ' else PRINTAFFECTED_AWK=3D' -function print_affected(apkg, glob, refs, descr, note) { +function print_affected(apkg, glob, refs, descr, newver, note) { split(apkg, thepkg) print "Affected package: " thepkg[1] print "Type of problem: " descr "." @@ -496,6 +538,9 @@ function print_affected(apkg, glob, refs, descr, note) { print "Reference: <" ref[r] ">" if (note) print "Note: " note + if (newver) { +'"$NEWVERS_AWK"' + } print "" } ' diff --git a/files/portaudit.1 b/files/portaudit.1 index c982b41..5cb0ec2 100644 --- a/files/portaudit.1 +++ b/files/portaudit.1 @@ -42,7 +42,7 @@ .Sh SYNOPSIS . .Nm -.Op Fl aCdFqvV +.Op Fl aCdFnqvV .Op Fl X Ar days .Op Fl f Ar file .Op Fl r Ar eregex @@ -85,6 +85,8 @@ Print the creation date of the database. Fetch the current database from the .Fx servers. +.It Fl n +Check if new versions of vulnerable ports are present. .It Fl q Quiet mode. .It Fl V diff --git a/pkg-plist b/pkg-plist index 8edf7bb..3c9f775 100644 --- a/pkg-plist +++ b/pkg-plist @@ -1,4 +1,5 @@ sbin/portaudit +sbin/portaudit-checknew etc/portaudit.conf.sample %%PERIODICDIR%%/security/410.portaudit @dirrmtry %%PERIODICDIR%%/security --=20 1.6.0.2 --X3gaHHMYHkYqP6yf Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="portaudit-megapatch_pkg_audit-and-checknew.diff" Content-Transfer-Encoding: quoted-printable This is the megapatch that adds both pkg_audit support and the ability to check if new versions of vulnerable ports are present. diff --git a/Makefile b/Makefile index 885dc27..6083235 100644 --- a/Makefile +++ b/Makefile @@ -37,7 +37,8 @@ SED_SCRIPT=3D -e 's|%%PREFIX%%|${PREFIX}|g' \ -e "s|%%BZIP2_CMD%%|${BZIP2_CMD}|g" \ =20 do-build: -.for f in portaudit-cmd.sh portaudit.sh portaudit.1 portaudit.conf +.for f in portaudit-cmd.sh portaudit-checknew.sh portaudit.sh \ + portaudit.1 portaudit.conf @${SED} ${SED_SCRIPT} ${FILESDIR}/${f} >${WRKDIR}/${f} .endfor =20 @@ -54,6 +55,7 @@ pre-install: =20 do-install: @${INSTALL_SCRIPT} ${WRKDIR}/portaudit-cmd.sh ${PREFIX}/sbin/portaudit + @${INSTALL_SCRIPT} ${WRKDIR}/portaudit-checknew.sh ${PREFIX}/sbin/portaud= it-checknew @${INSTALL_DATA} ${WRKDIR}/portaudit.conf ${PREFIX}/etc/portaudit.conf.sa= mple @${INSTALL_MAN} ${WRKDIR}/portaudit.1 ${MAN1PREFIX}/man/man1 @${MKDIR} ${PERIODICDIR}/security diff --git a/files/portaudit-checknew.sh b/files/portaudit-checknew.sh new file mode 100755 index 0000000..6bc7dfe --- /dev/null +++ b/files/portaudit-checknew.sh @@ -0,0 +1,159 @@ +#!/bin/sh +# $FreeBSD$ +# +# Script to check if newer port versions then the ones installed +# in the system are available. Uses downloaded ports index from +# the master site. +# +# Utility acts as a filter: reads input strings, processes them +# and outputs the result to the standard output. Each input string +# is treated as the collection of '|'-separated fields. First field +# should be the package name with version, other fields can have +# any contents -- they won't be touched. +# +# One more field can be added -- the newest port version, if it can +# be deduced. This field will be added to the end fieldset. +# +# For any error the utility will just output the unmodified input +# contents. +# +# Eygene Ryabinkin, rea-fbsd@codelabs.ru. September 2008. + +BASEURL=3Dhttp://www.freebsd.org/ports/INDEX- +PKGBASE=3D/var/db/pkg + +# Obtains FreeBSD major version number from uname output. +freebsd_major () { + uname -r | cut -f1 -d- | cut -f1 -d. +} + +# Returns the OS version for the INDEX file. It is just a wrapper +# for the freebsd_major routine that applies the fixups for the +# known unsupported versions. +index_version () { + local major + + major=3D`freebsd_major` + case "$major" in + 1|2|3|4) + major=3D5 + ;; + esac + echo "$major" +} + +# Guesses (installed) port origin. +# Arguments: +# - port name with version specification. +# - name of file wit +guess_origin () { + local contents origin + + if [ -z "$1" ]; then + echo "guess_origin(): called without arguments" >&2 + exit 255 + fi + if [ -z "$2" ]; then + echo "guess_origin(): called without second argument" >&2 + exit 255 + fi + + contents=3D"$PKGBASE"/"$1"/"+CONTENTS" + if [ -s "$contents" ]; then + origin=3D`grep '^@comment ORIGIN:' "$contents" | \ + sed -e's/^@comment ORIGIN://'` + if [ -n "$origin" ]; then + echo "$origin" + return + fi + fi + # Not yet implemented: loop over INDEX file and try to get + # the origin from the matched ports description. +} + +# Fetches bzipped port index file and outputs it to the stdout +# Arguments: +# - URL to get file from. +# +# Uses 'opt_cachedir' (if it is not empty) as the name of the directory +# to place the downloaded file to. Download is made in the mirror mode, +# so if file's timestamps and size are checked and no download takes +# place if the remote file has the same characteristics. +fetch_index () { + if [ -z "$1" ]; then + echo "fetch_index(): called without arguments" >&2 + exit 255 + fi + + if [ -z "$opt_cachedir" -o ! -d "$opt_cachedir" ]; then + fetch -qpo - "$1" + return + fi + + local outfile=3D"$opt_cachedir"/`basename "$1"` + fetch -mpo "$outfile" "$1" && cat "$outfile" +} + +TMPINDEX=3D`mktemp -q -t versionaudit-INDEX` +if [ -z "$TMPINDEX" ]; then + echo "Unable to create temporary file for ports index." >&2 + cat + exit 1 +fi + +trap "rm -f \"$TMPINDEX\"" 0 1 2 3 15 + +opt_cachedir=3D + +while getopts c: opt; do + case "$opt" in + c) + opt_cachedir=3D"$OPTARG" + ;; + ?) + echo "Usage: $0 [-c cachedir]" + exit 2 + ;; + esac +done + +url=3D"$BASEURL"`index_version`.bz2 +fetch_index "$url" | bunzip2 -c > "$TMPINDEX" +if ! [ -s "$TMPINDEX" ]; then + echo "Unable to download port index from $url" >&2 + cat + exit 1 +fi + +IFS=3D'|' +while read portspec rest; do + portname=3D`echo "$portspec" | sed -e's/-[^-]*$//'` + origin=3D`guess_origin "$portspec" "$TMPINDEX"` + if [ -z "$origin" ]; then + echo "Unable to get port origin for '$portspec'." >&2 + continue + fi + # The while cycle is hackish: we make "$?" non-zero if no + # matches were found and it is set to zero if match was + # found. All exit paths from the cycle should set "$?" + # properly. + # + # Another way to proceed is to make subroutine that will + # either print something or not, because we can't pass + # variables outside the while loop -- it is done in the + # separate process. + grep "^$portname-" "$TMPINDEX" | while read nportspec ndir nrest + do + nportname=3D`echo "$nportspec" | sed -e's/-[^-]*$//'` + norigin=3D`echo "$ndir" | sed -e's|^/[^/]*/[^/]*/||'` + if [ "$nportname" =3D "$portname" -a \ + "$norigin" =3D "$origin" ]; then + echo "$portspec|$rest|$nportspec" + true + break + fi + false + done || echo "$portspec|$rest" +done + +exit 0 diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh index c0eb67b..0ddd3d3 100755 --- a/files/portaudit-cmd.sh +++ b/files/portaudit-cmd.sh @@ -43,6 +43,8 @@ portaudit_confs() =20 : ${portaudit_fixed=3D""} =20 + : ${portaudit_pkg_audit=3D"/usr/sbin/pkg_audit"} + if [ -r %%PREFIX%%/etc/portaudit.conf ]; then . %%PREFIX%%/etc/portaudit.conf fi @@ -125,49 +127,97 @@ portaudit_prerequisites() return 0 } =20 +checknew() +{ + if [ -n "$opt_checknew" ]; then + portaudit-checknew -c "${portaudit_dir}" + else + cat + fi +} + +# +# Helper for audit_installed that actually finds vulnerable packages. +# +# It processes the auditfile entries (that are read from the stdin) +# in the form "glob|refs|desc" and outputs entries in the form +# "pkgname|glob|refs|desc", where "pkgname" is the matched package name. +# +findvuln_installed() +{ + local fixedre=3D`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' |= tr -s ' \t\n' '|'` + + if [ -x "${portaudit_pkg_audit}" ]; then + "${portaudit_pkg_audit}" | awk -F \| \ + -v fixedre=3D"$fixedre" -v opt_restrict=3D"$opt_restrict" \ + ' +opt_restrict && $3 !~ opt_restrict { next } +$2 ~ /^FreeBSD[<=3D>!]/ && fixedre && $3 ~ fixedre { next } +{ print } +' + + return + fi + + local installedre=3D`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d = '|' -` + local osversion=3D`sysctl -n kern.osreldate` + + awk -F\| \ + -v fixedre=3D"$fixedre" -v installedre=3D"$installedre" \ + -v pkg_version=3D"$pkg_version" -v pkg_info=3D"$pkg_info" \ + -v osversion=3D"$osversion" -v opt_restrict=3D"$opt_restrict" \ + ' +/^(#|\$)/ { next } +opt_restrict && $2 !~ opt_restrict { next } +$1 ~ /^FreeBSD[<=3D>!]/ { + if (fixedre && $2 ~ fixedre) next + if (!system(pkg_version " -T \"FreeBSD-" osversion "\" \"" $1 "\"")) { + printf("FreeBSD-%s|%s\n", osversion, $0); + } + next +} +$1 ~ /^[^{}*?]*[<=3D>!]/ { + if ($1 !~ "^(" installedre ")[<=3D>!]") next; +} +{ + cmd=3Dpkg_info " -E \"" $1 "\"" + while((cmd | getline pkg) > 0) { + printf("%s|%s\n", pkg, $0); + } + close(cmd) +} +' +} + audit_installed() { local rc=3D0 local osversion=3D`sysctl -n kern.osreldate` =20 - fixedre=3D`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s= ' \t\n' '|'` - installedre=3D`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -` - - extract_auditfile | awk -F\| "$PRINTAFFECTED_AWK"' - BEGIN { vul=3D0; fixedre=3D"'"$fixedre"'" } - /^(#|\$)/ { next } - $2 !~ /'"$opt_restrict"'/ { next } - $1 ~ /^FreeBSD[<=3D>!]/ { - if (fixedre && $2 ~ fixedre) next - if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\""= )) { - print_affected("FreeBSD-'"$osversion"'", \ - "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PR= EFIX%%/etc/portaudit.conf") - } - next - } - $1 ~ /^[^{}*?]*[<=3D>!]/ { - if ($1 !~ "^('"$installedre"')[<=3D>!]") next; - } - { - cmd=3D"'"$pkg_info"' -E \"" $1 "\"" - while((cmd | getline pkg) > 0) { - vul++ - print_affected(pkg, "") - } - close(cmd) - } - END { - if ("'$opt_quiet'" =3D=3D "false") { - print vul " problem(s) in your installed packages found." - } - if (vul > 0) { - if ("'$opt_quiet'" =3D=3D "false") { - print "\nYou are advised to update or deinstall" \ - " the affected package(s) immediately." - } - exit(1) - } + extract_auditfile | findvuln_installed | checknew | \ + awk -F\| -v pkg_version=3D"$pkg_version" "$PRINTAFFECTED_AWK"' +BEGIN { vul=3D0; } +$1 ~ /^FreeBSD-/ { + print_affected($1, $2, $3, $4, "", \ + "To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFI= X%%/etc/portaudit.conf") + next +} +{ + print_affected($1, $2, $3, $4, $5, ""); + vul++; +} +END { + if ("'$opt_quiet'" =3D=3D "false") { + print vul " problem(s) in your installed packages found." + } + if (vul > 0) { + if ("'$opt_quiet'" =3D=3D "false") { + print "\nYou are advised to update or deinstall" \ + " the affected package(s) immediately." } + exit(1) + } +} ' || rc=3D$? =20 return $rc @@ -207,7 +257,7 @@ audit_file() if ($2 !~ /'"$opt_restrict"'/) continue vul++ - print_affected(pkg, "") + print_affected(pkg, $1, $2, $3, "", "") } close(cmd) } @@ -244,7 +294,8 @@ audit_args() ' | $pkg_version -T "$1" -`; then VULCNT=3D$(($VULCNT+1)) echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' - { print_affected("'"$1"'", "") } + { print_affected("'"$1"'", + $1, $2, $3, "", "") } ' fi ;; @@ -277,7 +328,7 @@ audit_cwd() { print } ' | $pkg_version -T "$PKGNAME" -`; then echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"' - { print_affected("'"$PKGNAME"'", "") } + { print_affected("'"$PKGNAME"'", $1, $2, $3, "", "") } ' return 1 fi @@ -347,12 +398,13 @@ opt_restrict=3D opt_verbose=3Dfalse opt_version=3Dfalse opt_expiry=3D +opt_checknew=3D =20 if [ $# -eq 0 ] ; then opt_audit=3Dtrue fi =20 -while getopts aCdf:Fqr:vVX: opt; do +while getopts aCdf:Fnqr:vVX: opt; do case "$opt" in a) opt_audit=3Dtrue;; @@ -364,6 +416,8 @@ while getopts aCdf:Fqr:vVX: opt; do opt_file=3D"$OPTARG";; F) opt_fetch=3Dtrue;; + n) + opt_checknew=3Dtrue;; q) opt_quiet=3Dtrue;; r) @@ -423,40 +477,73 @@ fi =20 prerequisites_checked=3Dfalse =20 +# This awk code demands 'pkg_version' variable to be set +# if 'newver' variable is non-empty. +NEWVERS_AWK=3D' + +nparts=3Dsplit(thepkg[1], verarr, /-/) +gtglob=3Dverarr[1] +for (i=3D2; i" verarr[i] + +updated=3D"" +avail=3D0 +if (system(pkg_version " -T \"" newver "\" '\''" gtglob "'\''") =3D=3D 0) { + updated=3D"available, " newver + avail=3D1 +} else { + updated=3D"not available" +} +if (avail !=3D 0) { + if (system(pkg_version " -T \"" newver "\" '\''" glob "'\''") =3D=3D 0) + updated=3Dupdated ", still vulnerable" + else + updated=3Dupdated ", not vulnerable" +} +print "Updated package: " updated +' + if $opt_quiet; then PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - print apkg - } - ' +function print_affected(apkg, glob, refs, descr, newver, note) { + print apkg +} +' elif $opt_verbose; then PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - split(apkg, thepkg) - print "Affected package: " thepkg[1] " (matched by " $1 ")" - print "Type of problem: " $3 "." - split($2, ref, / /) - for (r in ref) - print "Reference: <" ref[r] ">" - if (note) - print "Note: " note - print "" - } - ' +function print_affected(apkg, glob, refs, descr, newver, note) { + split(apkg, thepkg) + print "Affected package: " thepkg[1] " (matched by " glob ")" + print "Type of problem: " descr "." + split(refs, ref, / /) + for (r in ref) + print "Reference: <" ref[r] ">" + if (note) + print "Note: " note + if (newver) { +'"$NEWVERS_AWK"' + } + print "" +} +' else PRINTAFFECTED_AWK=3D' - function print_affected(apkg, note) { - split(apkg, thepkg) - print "Affected package: " thepkg[1] - print "Type of problem: " $3 "." - split($2, ref, / /) - for (r in ref) - print "Reference: <" ref[r] ">" - if (note) - print "Note: " note - print "" - } - ' +function print_affected(apkg, glob, refs, descr, newver, note) { + split(apkg, thepkg) + print "Affected package: " thepkg[1] + print "Type of problem: " descr "." + split(refs, ref, / /) + for (r in ref) + print "Reference: <" ref[r] ">" + if (note) + print "Note: " note + if (newver) { +'"$NEWVERS_AWK"' + } + print "" +} +' fi =20 if $opt_audit; then diff --git a/files/portaudit.1 b/files/portaudit.1 index c982b41..5cb0ec2 100644 --- a/files/portaudit.1 +++ b/files/portaudit.1 @@ -42,7 +42,7 @@ .Sh SYNOPSIS . .Nm -.Op Fl aCdFqvV +.Op Fl aCdFnqvV .Op Fl X Ar days .Op Fl f Ar file .Op Fl r Ar eregex @@ -85,6 +85,8 @@ Print the creation date of the database. Fetch the current database from the .Fx servers. +.It Fl n +Check if new versions of vulnerable ports are present. .It Fl q Quiet mode. .It Fl V diff --git a/files/portaudit.conf b/files/portaudit.conf index 4eb08d7..9ea2c4a 100644 --- a/files/portaudit.conf +++ b/files/portaudit.conf @@ -17,3 +17,7 @@ =20 # this vulnerability has been fixed in your FreeBSD version #portaudit_fixed=3D"d2102505-f03d-11d8-81b0-000347a4fa7d" + +# this command will be used to find the vulnerable packages +# instead of awk(1) script. +# portaudit_pkg_audit=3D"/usr/sbin/pkg_audit" diff --git a/pkg-plist b/pkg-plist index 8edf7bb..3c9f775 100644 --- a/pkg-plist +++ b/pkg-plist @@ -1,4 +1,5 @@ sbin/portaudit +sbin/portaudit-checknew etc/portaudit.conf.sample %%PERIODICDIR%%/security/410.portaudit @dirrmtry %%PERIODICDIR%%/security --X3gaHHMYHkYqP6yf-- --aPdhxNJGSeOG9wFI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjpD3QACgkQthUKNsbL7YhR0gCcDLqUN4PF6QqCWua2KuIza2OD hfEAn11non2XPdY+r4zIsFYIPC0x4vs1 =kJcQ -----END PGP SIGNATURE----- --aPdhxNJGSeOG9wFI-- From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 5 19:16:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86843106564A; Sun, 5 Oct 2008 19:16:13 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 2F0428FC24; Sun, 5 Oct 2008 19:16:13 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=HKkMFVM/jbovocWWZqxUoJdWTB94xJOD+wM6nRD3TGbdxi1mno9mBVD2wHfy3pY1LUwiGgDeXxyDbEceVj2ZMQfnY2pFgE6sQBz5RMyIXZyHHJhnQ+4RgmOHBvqLZEo+ebU5+JR917f49gEsOz23EFOVVI39sM4SRMfbIzTzBxU=; Received: from amnesiac.at.no.dns (ppp83-237-107-240.pppoe.mtu-net.ru [83.237.107.240]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmZ5P-0000uB-RL; Sun, 05 Oct 2008 23:16:11 +0400 Date: Sun, 5 Oct 2008 23:16:20 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cW+P/jduATWpL925" Content-Disposition: inline In-Reply-To: Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, Roman Kurakin , bug-followup@freebsd.org, freebsd-ports@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Oct 2008 19:16:13 -0000 --cW+P/jduATWpL925 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Sun, Oct 05, 2008 at 11:03:17PM +0400, Eygene Ryabinkin wrote: > I had also changed the output format for pkg_audit, so I am attaching > another version of the second patch for the pkg_install bundle. One neat about new pkg_audit utility: if you already have the build directory for pkg_install in the /usr/obj, you should create subdirectory for the pkg_audit, ----- mkdir /usr/obj/usr/src/usr.sbin/pkg_install/audit ----- or completely remove /usr/obj/usr/src/usr.sbin/pkg_install World build should do it automatically, at least it worked for me. --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --cW+P/jduATWpL925 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjpEoQACgkQthUKNsbL7YjATgCeOBC9gPeUJZJInl1/kluggj6V f6sAn1KmdznY5zPZhUIlx1sQdzQfLCaF =tXav -----END PGP SIGNATURE----- --cW+P/jduATWpL925-- From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 5 22:21:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11A421065687; Sun, 5 Oct 2008 22:21:06 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id D40688FC18; Sun, 5 Oct 2008 22:21:05 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTP id 7E03546B09; Sun, 5 Oct 2008 18:21:05 -0400 (EDT) Date: Sun, 5 Oct 2008 23:21:05 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Danny Braniss In-Reply-To: Message-ID: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org, Robert Watson Subject: Re: bad NFS/UDP performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Oct 2008 22:21:06 -0000 On Sat, 4 Oct 2008, Danny Braniss wrote: > at the moment, the best I can do is run it on a different hardware that has > if_em, the results are in > ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the > benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I > get the same numbers with an older kernel). Dear Danny: Unfortunately, I was left slightly unclear on the comparison you are making above. Could you confirm whether or not, with if_em, you see a performance regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, and if you do, whether or not the RLOCK->WLOCK change has any effect on performance? It would be nice to know on the same hardware but at least with different hardware we get a sense of whether or not this might affect other systems or whether it's limited to a narrower set of configurations. Thanks, Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 03:10:18 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65BA5106568E; Mon, 6 Oct 2008 03:10:18 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: from valentine.liquidneon.com (valentine.liquidneon.com [IPv6:2001:4830:2407:8000:230:48ff:fe71:c2a2]) by mx1.freebsd.org (Postfix) with ESMTP id 4A1B78FC13; Mon, 6 Oct 2008 03:10:18 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: by valentine.liquidneon.com (Postfix, from userid 1018) id 55E1A8FDDF; Sun, 5 Oct 2008 21:10:17 -0600 (MDT) Date: Sun, 5 Oct 2008 21:10:17 -0600 From: Brad Davis To: freebsd-current@FreeBSD.org, freebsd-hackers@FreeBSD.org Message-ID: <20081006031017.GA43283@valentine.liquidneon.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: FreeBSD Status Reports due October 19th, 2008 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 03:10:18 -0000 Hi Everyone, It is that time again. We would like to remind everybody who has exciting news to share to write a report about their project. This is a good way to improve exposure of your work, receive feedback and help. We are looking forward to your reports. As always you can either use the template or the CGI generator and mail the output to monthly@ by Sunday October 19th, 2008. http://www.freebsd.org/news/status/ http://www.freebsd.org/cgi/monthly.cgi http://www.freebsd.org/news/status/report-sample.xml Regards, Brad Davis From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 05:23:40 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7D351065689; Mon, 6 Oct 2008 05:23:40 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 7E6508FC20; Mon, 6 Oct 2008 05:23:40 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=ZIBqWf4Zf+6ZxO6adwu0FQ2qtskZAGP577ffpsGfU1pmbfm8nxflMa53MLPiwA5jkN1hMdKYIjnxrhC6mdgUAM1WOLKFNMREwfPGJOWBrDfKGZXGy7kgXYv0FvHKK8UvMtMuGDxuApEtcSRZ4cpmW4G9BqNpruKcQtpiGMLCBA0=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmiZG-000I6t-Vl; Mon, 06 Oct 2008 09:23:39 +0400 Date: Mon, 6 Oct 2008 09:23:37 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+hZIELbchok2+JHy" Content-Disposition: inline In-Reply-To: <48E94281.8010300@quip.cz> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, bug-followup@freebsd.org, freebsd-ports@freebsd.org, portmgr@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 05:23:40 -0000 --+hZIELbchok2+JHy Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Miroslav, good day. Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote: > I am busy these days, but it is nice to read about your progress. I hope= =20 > I will get some time to test all of these large patches in a few days=20 > and I will report back my experiences! Fine, thank you! I am re-CC'ing bug-followup@ to track this letter, since it contains some useful information that should go into GNATS. > One note before tests... do -n flag always download new INDEX file, or=20 > is it possible to use one already existing in /usr/ports? Currently, it is downloads bzipped INDEX file to /var/db/portaudit every time, but it uses mirror mode, so if remote file hadn't changed at all, all network expences are just the HTTP's HEAD request and reply. I can add another variable to the portaudit to force the usage of the existing INDEX file, if it is needed. By the way, how are you keeping your INDEX file up to date (your proposed usage of 'pkg_version -I' implies that you're always rely on it)? I am just curious -- my INDEX files are almost always stay unupdated, even if I am using portupgrade. And there can be another way if one keeps ports tree updated: utility can use 'make' to determine the version that is currently available on the examined host. But downloading the INDEX file from the central server seemed to be the best way, since it almost always gives one the latest port versions, so I had implemented this in a first place. Don't know, however, how the badly the load to the central HTTP server will be raised. I am using just two first fields from the INDEX file, so I can use such a stripped file. For me, the reduction was about 6x: SIZE(INDEX-7.bz2) =3D 1126189, SIZE(INDEX-7.stripped.bz2) =3D 184345. I am CC'ing the portmgr team. Guys, could you quickly glance over these patches and determine if they are useful to the project in large? If yes, then may be such a stripped INDEX can be created on the FreeBSD servers (via cut -f1-2 -d'|' INDEX-N)? Thanks! --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --+hZIELbchok2+JHy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjpoNkACgkQthUKNsbL7YhirACeIewIwCM0wr/UBHnMp7hieJya eUoAoIAwCdHLh38sZwl+bvw65cB9OvgW =ANyt -----END PGP SIGNATURE----- --+hZIELbchok2+JHy-- From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 06:58:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AFF31065686; Mon, 6 Oct 2008 06:58:08 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id BA6248FC20; Mon, 6 Oct 2008 06:58:07 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m966uv1m036288; Mon, 6 Oct 2008 00:56:57 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Mon, 06 Oct 2008 00:56:57 -0600 (MDT) Message-Id: <20081006.005657.71122961.imp@bsdimp.com> To: bagavathykumar.m@hcl.in From: Warner Losh In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF294028F DEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081002.233328.-432820840.imp@bsdimp.com> <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 06:58:08 -0000 > Thanks for your support. my probe is getting called for all > the bridges not for my pci device. so please provide the fix . > > OR > Is any other way available for making my driver to override the probe of > cbb driver for my corresponding device (With out changing cbb driver). If your probe returns a higher number that's negative, it will. Unless cbb is returning 0, your probe routine will get called. Make sure it isn't. Code inspection suggests that it isn't. Warner > With regards, > Bagavathy kumar .M > > -----Original Message----- > From: M. Warner Losh [mailto:imp@bsdimp.com] > Sent: Friday, October 03, 2008 11:03 AM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called for > mycorrespondingdevice ID and Vendor ID > > In message: > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > "Bagavathy Kumar Mahendran " > writes: > : > : Dear Baldwin, > : Thanks for your support .but my pci probe function is > not > : getting called for my device id and vendor id. Because pccbb driver > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > any > : other option for me to make my_pciprobe function to be called for my > : corresponding device id and vendor id. > > That's not why your probe isn't called. Setting a description is > standard behavior for the probe routine. Are you sure that the device > probe routine is getting called at all for any device? Have you tried > just leaving cbb out of the kernel? I recently fixed the original > problem in cbb (the fact it doesn't check the bridge type too), maybe > you could try to pick up that fix as well? > > Warner > > > : Thanks, > : > : Regards, > : Bagavathy kumar .M > : > : > : > : -----Original Message----- > : From: John Baldwin [mailto:jhb@freebsd.org] > : Sent: Wednesday, October 01, 2008 8:57 PM > : To: freebsd-hackers@freebsd.org > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my > : correspondingdevice ID and Vendor ID > : > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > : wrote: > : > > : > Dear All, > : > Iam writing a new driver for a SAS/SATA Controller > having > : a > : > Class ID -0x01 > : > Sub Class - 0x07 > : > Programming Interface - 0x00 > : > > : > Hence instead of my probe function the Static build Card Bus Driver > : cbb > : > is attaching just by simply checking sub class 0x07 and programming > : > interface 0x00.hence my probe gets failed. Kindly help me in > resolving > : > this .what I thought is to add the card bus driver a checking of > CLASS > : > ID in its pci probe function. > : > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return > : GENERIC > : in the case where it matches only on class codes). Your driver just > : needs to > : return a numerically higher value (but still < 0) to claim the device. > : You > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > : > : -- > : John Baldwin > : > : DISCLAIMER: > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > : It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in > : this email are solely those of the author and may not necessarily > reflect the opinions of HCL or its affiliates. > : Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of > : this message without the prior written consent of the author of this > e-mail is strictly prohibited. If you have > : received this email in error please delete it and notify the sender > immediately. Before opening any mail and > : attachments please check them for viruses and defect. > : > : > ------------------------------------------------------------------------ > ----------------------------------------------- > : > : > > From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 07:28:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C389106569A; Mon, 6 Oct 2008 07:28:53 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 02B368FC0A; Mon, 6 Oct 2008 07:28:52 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m967QKhD036631; Mon, 6 Oct 2008 01:26:20 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Mon, 06 Oct 2008 01:26:19 -0600 (MDT) Message-Id: <20081006.012619.78741615.imp@bsdimp.com> To: bagavathykumar.m@hcl.in From: Warner Losh In-Reply-To: <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> References: <68C9F31EF19DB6448F515EF 294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.005657.71122961.imp@bsdimp.com> <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 07:28:53 -0000 From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, > > My probe is getting called for the parent bridge devices > .but > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. > > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M > > > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com] > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID > > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). > > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. > > Warner > > > > > With regards, > > Bagavathy kumar .M > > > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com] > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > > > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > : > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > > > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > > > > Warner > > > > > > : Thanks, > > : > > : Regards, > > : Bagavathy kumar .M > > : > > : > > : > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org] > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > : > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : > > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : > > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > : > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC > > : in the case where it matches only on class codes). Your driver just > > : needs to > > : return a numerically higher value (but still < 0) to claim the > device. > > : You > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > : > > : -- > > : John Baldwin > > : > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and > > : attachments please check them for viruses and defect. > > : > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > : > > : > > > > > > From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 09:43:27 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD75D1065687; Mon, 6 Oct 2008 09:43:27 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from mail.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id 91A1D8FC1A; Mon, 6 Oct 2008 09:43:27 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from localhost (mail.rachie.is-a-geek.net [192.168.2.101]) by mail.rachie.is-a-geek.net (Postfix) with ESMTP id BC6D5AFBC01; Mon, 6 Oct 2008 01:24:56 -0800 (AKDT) From: Mel To: freebsd-hackers@freebsd.org Date: Mon, 6 Oct 2008 11:24:54 +0200 User-Agent: KMail/1.9.7 References: <48E94281.8010300@quip.cz> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> Cc: freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 09:43:27 -0000 Hello, On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > But downloading the INDEX file from the central server seemed to be the > best way, since it almost always gives one the latest port versions, so > I had implemented this in a first place. I've been following this, but I don't agree that (port|pkg_)audit should do this, from the very perspective you're writing this program from: On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > 4. I feel that it is Unix-way to do the things: create small utilities > that do their (small) job in a proper fashion. Instead, it can provide installed-pkgnamepkgorigin output. Then, any utility can check whether a new version is available, using what ever source it finds relevant. For example, it is completely irrelevant if a new version is available on the FreeBSD servers, when your machine uses a buildserver in a local network. For those machines it's relevant whether their build server has a new version and one can automatically upgrade if one so desires. Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're again reporting false information. It's also quite trivial to provide this availibility information in a daily security script, for the "majority of cases" and it's better to have tunables like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script. -- Mel From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 10:28:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACDCE1065688; Mon, 6 Oct 2008 10:28:52 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 3231B8FC16; Mon, 6 Oct 2008 10:28:52 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=e7n6Smu33wrvqkyTIS5Zkn/mRIxMvatWC5CRsCq21o4pv/4Mh1oQXCVXIswSRDBaSsapW8SLkVOwSehH/mZe629ZEbUvFzjpO5n0khg/ZL3ESatBVYMOo2QbljSJ2PN/W7VKdpenZD/h/V3mJ+O50f5CAOvaVZLVd3/VXV3+gbE=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmnKb-000EU0-Ji; Mon, 06 Oct 2008 14:28:49 +0400 Date: Mon, 6 Oct 2008 14:28:48 +0400 From: Eygene Ryabinkin To: Mel Message-ID: References: <48E94281.8010300@quip.cz> <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0a51aUHJSNYMigeV" Content-Disposition: inline In-Reply-To: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 10:28:52 -0000 --0a51aUHJSNYMigeV Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Mel, good day. Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote: > On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > > But downloading the INDEX file from the central server seemed to be the > > best way, since it almost always gives one the latest port versions, so > > I had implemented this in a first place. >=20 > I've been following this, but I don't agree that (port|pkg_)audit should = do=20 > this, from the very perspective you're writing this program from: The download is done not by the portaudit itself, but by the helper script, portaudit-checknew. > On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > > 4. I feel that it is Unix-way to do the things: create small utilities > > that do their (small) job in a proper fashion. >=20 > Instead, it can provide installed-pkgnamepkgorigin output. The= n,=20 > any utility can check whether a new version is available, using what ever= =20 > source it finds relevant. > > For example, it is completely irrelevant if a new version is available on= the=20 > FreeBSD servers, when your machine uses a buildserver in a local network.= For=20 > those machines it's relevant whether their build server has a new version= and=20 > one can automatically upgrade if one so desires. > Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're= =20 > again reporting false information. I hear you, but it seems to me that I should just equip portaudit-checknew with the other sources of a new ports information and provide tunables for their location (on-disk path, URL, etc). I am planning to do this, but first I want to know if these patches will be viable for the project: feeding these into the /dev/null or just using them locally, but equipping with a lot of functionality, is not what I really want ;)) > It's also quite trivial to provide this availibility information in a dai= ly=20 > security script, for the "majority of cases" Didn't get it, sorry. Could you, please, elaborate a bit? > and it's better to have tunables=20 > like _use_remote_portindex, _use_portsdir=3D/bigdisk/usr/ports in a scrip= t. Yes, it was what I had talked about above in this mail. Thanks for the input! --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --0a51aUHJSNYMigeV Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjp6GAACgkQthUKNsbL7YhIiwCfaxk/raGBCgtCRoPUirA9AM1F QWQAoKHacwvF3w+HOUw1kMJwFJUg3hrq =ee1X -----END PGP SIGNATURE----- --0a51aUHJSNYMigeV-- From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 10:30:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B5C1106568C; Mon, 6 Oct 2008 10:30:32 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id A6E268FC1F; Mon, 6 Oct 2008 10:30:31 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=Lo/oTIPWRq7/qsx+E+tR6hhhfWX6KWmoMzXySKmYaPL9VqTKtNZRIeuUl2Uooy+ZPLx+wuMtWuRB4KGshXWJns09TmyE6a1g/UEqRYXr8o5MAZgsD1NlGz/Xd5BjKuOCXA2FbONqcbTWahyt1mD8PSTB+btNKqg/PLCj6ZSK758=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmnME-000EbU-Oe; Mon, 06 Oct 2008 14:30:30 +0400 Date: Mon, 6 Oct 2008 14:30:29 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+yOEb87YRmRdrPqC" Content-Disposition: inline In-Reply-To: <48E9D382.4000001@quip.cz> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, bug-followup@freebsd.org, freebsd-ports@freebsd.org, portmgr@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 10:30:32 -0000 --+yOEb87YRmRdrPqC Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Miroslav, Mon, Oct 06, 2008 at 10:59:46AM +0200, Miroslav Lachman wrote: > I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in=20 > my crontab, so I get INDEX updated every night before nightly security=20 > e-mail is generated. Ah, I see. Thanks! > > But downloading the INDEX file from the central server seemed to be the > > best way, since it almost always gives one the latest port versions, so > > I had implemented this in a first place. >=20 > My previous question was not against your solution, it seems useful to=20 > have really actual data from the fresh INDEX. It was just a question=20 > "how it is done". Maybe someone will be happier to use the existing=20 > INDEX because of traffic on some GPRS internet connection or because of= =20 > the own INDEX creation. (it is not my case, I have all machines as the=20 > servers with enough connectivity) ;) OK, fine. I will implement the usage of the local INDEX file in some days. --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --+yOEb87YRmRdrPqC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjp6MUACgkQthUKNsbL7YiFPgCfd591IqFL6B2ONvm0PKFNg9Gv xBMAoJmLlL3JUH7heDDVqr8oD6nn1I4G =SxiG -----END PGP SIGNATURE----- --+yOEb87YRmRdrPqC-- From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 11:07:55 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 067791065693; Mon, 6 Oct 2008 11:07:55 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from mail.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id BECEB8FC21; Mon, 6 Oct 2008 11:07:54 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from localhost (mail.rachie.is-a-geek.net [192.168.2.101]) by mail.rachie.is-a-geek.net (Postfix) with ESMTP id A7B6EAFBC01; Mon, 6 Oct 2008 03:07:53 -0800 (AKDT) From: Mel To: freebsd-hackers@freebsd.org Date: Mon, 6 Oct 2008 13:07:51 +0200 User-Agent: KMail/1.9.7 References: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> Cc: freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 11:07:55 -0000 On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > Mel, good day. > > Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote: > > On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote: > > > But downloading the INDEX file from the central server seemed to be the > > > best way, since it almost always gives one the latest port versions, so > > > I had implemented this in a first place. > > > > I've been following this, but I don't agree that (port|pkg_)audit should > > do this, from the very perspective you're writing this program from: > > The download is done not by the portaudit itself, but by the helper > script, portaudit-checknew. > > > On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote: > > > 4. I feel that it is Unix-way to do the things: create small utilities > > > that do their (small) job in a proper fashion. > > > > Instead, it can provide installed-pkgnamepkgorigin output. > > Then, any utility can check whether a new version is available, using > > what ever source it finds relevant. > > > > For example, it is completely irrelevant if a new version is available on > > the FreeBSD servers, when your machine uses a buildserver in a local > > network. For those machines it's relevant whether their build server has > > a new version and one can automatically upgrade if one so desires. > > Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're > > again reporting false information. > > I hear you, but it seems to me that I should just equip > portaudit-checknew with the other sources of a new ports information and > provide tunables for their location (on-disk path, URL, etc). I am > planning to do this, but first I want to know if these patches will be > viable for the project: feeding these into the /dev/null or just using > them locally, but equipping with a lot of functionality, is not what I > really want ;)) > > > It's also quite trivial to provide this availibility information in a > > daily security script, for the "majority of cases" > > Didn't get it, sorry. Could you, please, elaborate a bit? Once you have the origin of the port, you can: - make -C $PORTSDIR/$origin -V PKGNAME - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} - get the matching origin(s) out of a downloaded INDEX.bz2 This covers the majority of cases. What portaudit lacks, is providing the origin along with the installed package name in easily parseable format. So, a central server wanting to query all the machines for vulnerable packages, now has to do an extra step of going into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line, while (port|pkg_)audit has just been there. This would be something I'd expect: ssh clientmachine "/usr/sbin/pkg_audit -l" foo-1.2,3:misc/foo bar-4.5_6:devel/bar ... -- Mel From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 05:18:36 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43FE0106569E; Mon, 6 Oct 2008 05:18:36 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (chn-hclin-gws02.hcl.in [203.105.186.20]) by mx1.freebsd.org (Postfix) with ESMTP id 811158FC22; Mon, 6 Oct 2008 05:18:34 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (gws04 [10.249.64.135]) by localhost.hcl.in (Postfix) with ESMTP id C0F683600BA; Mon, 6 Oct 2008 10:48:31 +0530 (IST) Received: from chn-egw02-out.corp.hcl.in (unknown [10.249.64.38])by gws04.hcl.in (Postfix) with ESMTPid B62DE3600B7; Mon, 6 Oct 2008 10:48:31 +0530 (IST) Received: from chn-hclt-evs05.HCLT.CORP.HCL.IN ([10.101.26.13]) by chn-egw02-out.corp.hcl.in with Microsoft SMTPSVC(6.0.3790.3959); Mon, 6 Oct 2008 10:48:31 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 6 Oct 2008 10:48:29 +0530 Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> In-Reply-To: <20081002.233328.-432820840.imp@bsdimp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID Thread-Index: AcklGZ1ktXMrlLxISoqJo4xaPaYQnQCWMMuw References: <68C9F31EF19DB6448F515EF294028FDEE999AB@chn-hclt-evs05.HCLT.CORP .HCL.IN><200810011127.14593.jhb@freebsd.org><68C9F31EF19DB6448F515EF294028F DEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081002.233328.-432820840.imp@bsdimp.com> From: "Bagavathy Kumar Mahendran " To: "M. Warner Losh" X-OriginalArrivalTime: 06 Oct 2008 05:18:31.0262 (UTC) FILETIME=[F8EA27E0:01C92772] X-imss-version: 2.051 X-imss-result: Passed X-imss-scanInfo: M:T L:E SM:1 X-imss-tmaseResult: TT:1 TS:-25.7814 TC:1F TRN:72 TV:5.5.1026(16200.004) X-imss-scores: Clean:100.00000 C:0 M:0 S:0 R:0 X-imss-settings: Baseline:1 C:1 M:1 S:1 R:1 (0.0000 0.0000) X-Mailman-Approved-At: Mon, 06 Oct 2008 11:17:14 +0000 Cc: freebsd-hackers@freebsd.org Subject: RE: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 05:18:36 -0000 Dear Warner, Thanks for your support. my probe is getting called for all the bridges not for my pci device. so please provide the fix . OR=20 Is any other way available for making my driver to override the probe of cbb driver for my corresponding device (With out changing cbb driver). With regards, Bagavathy kumar .M -----Original Message----- From: M. Warner Losh [mailto:imp@bsdimp.com]=20 Sent: Friday, October 03, 2008 11:03 AM To: Bagavathy Kumar Mahendran=20 Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called for mycorrespondingdevice ID and Vendor ID In message: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> "Bagavathy Kumar Mahendran " writes: :=20 : Dear Baldwin, : Thanks for your support .but my pci probe function is not : getting called for my device id and vendor id. Because pccbb driver : already sets the device_set_desc as PCI-CardBus Bridge. So is there any : other option for me to make my_pciprobe function to be called for my : corresponding device id and vendor id. That's not why your probe isn't called. Setting a description is standard behavior for the probe routine. Are you sure that the device probe routine is getting called at all for any device? Have you tried just leaving cbb out of the kernel? I recently fixed the original problem in cbb (the fact it doesn't check the bridge type too), maybe you could try to pick up that fix as well? Warner : Thanks, :=20 : Regards, : Bagavathy kumar .M :=20 :=20 :=20 : -----Original Message----- : From: John Baldwin [mailto:jhb@freebsd.org]=20 : Sent: Wednesday, October 01, 2008 8:57 PM : To: freebsd-hackers@freebsd.org : Cc: Bagavathy Kumar Mahendran ; Warner Losh : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my : correspondingdevice ID and Vendor ID :=20 : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran : wrote: : >=20 : > Dear All, : > Iam writing a new driver for a SAS/SATA Controller having : a : > Class ID -0x01 : > Sub Class - 0x07 : > Programming Interface - 0x00 : >=20 : > Hence instead of my probe function the Static build Card Bus Driver : cbb : > is attaching just by simply checking sub class 0x07 and programming : > interface 0x00.hence my probe gets failed. Kindly help me in resolving : > this .what I thought is to add the card bus driver a checking of CLASS : > ID in its pci probe function. :=20 : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return : GENERIC=20 : in the case where it matches only on class codes). Your driver just : needs to=20 : return a numerically higher value (but still < 0) to claim the device. : You=20 : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. :=20 : --=20 : John Baldwin :=20 : DISCLAIMER: : ------------------------------------------------------------------------ ----------------------------------------------- :=20 : The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. : It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in=20 : this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. : Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of=20 : this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have : received this email in error please delete it and notify the sender immediately. Before opening any mail and=20 : attachments please check them for viruses and defect. :=20 : ------------------------------------------------------------------------ ----------------------------------------------- :=20 :=20 From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 07:06:43 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7A8B106568F for ; Mon, 6 Oct 2008 07:06:43 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (gws04.mail.hcl.in [203.105.186.20]) by mx1.freebsd.org (Postfix) with ESMTP id 561CE8FC23 for ; Mon, 6 Oct 2008 07:06:42 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (gws04 [10.249.64.135]) by localhost.hcl.in (Postfix) with ESMTP id A281936001A; Mon, 6 Oct 2008 12:36:39 +0530 (IST) Received: from chn-egw01-out.corp.hcl.in (unknown [10.249.64.37])by gws04.hcl.in (Postfix) with ESMTPid 8D98236006B; Mon, 6 Oct 2008 12:36:39 +0530 (IST) Received: from chn-hclt-evs05.HCLT.CORP.HCL.IN ([10.101.26.13]) by chn-egw01-out.corp.hcl.in with Microsoft SMTPSVC(6.0.3790.3959); Mon, 6 Oct 2008 12:36:39 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 6 Oct 2008 12:36:44 +0530 Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> In-Reply-To: <20081006.005657.71122961.imp@bsdimp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Thread-Index: AckngOgREffkjE0QTT+Ia4ph64es9gAAFouA References: <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081002.233328.-432820840.imp@bsdimp.com><68C9F31EF19DB6448F515EF 294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.005657.71122961.imp@bsdimp.com> From: "Bagavathy Kumar Mahendran " To: "Warner Losh" X-OriginalArrivalTime: 06 Oct 2008 07:06:39.0139 (UTC) FILETIME=[13FD9F30:01C92782] X-imss-version: 2.051 X-imss-result: Passed X-imss-scanInfo: M:T L:E SM:1 X-imss-tmaseResult: TT:1 TS:-30.9576 TC:1F TRN:81 TV:5.5.1026(16200.005) X-imss-scores: Clean:100.00000 C:0 M:0 S:0 R:0 X-imss-settings: Baseline:1 C:1 M:1 S:1 R:1 (0.0000 0.0000) X-Mailman-Approved-At: Mon, 06 Oct 2008 11:23:30 +0000 Cc: freebsd-hackers@freebsd.org Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 07:06:43 -0000 Dear Warner, My probe is getting called for the parent bridge devices .but=20 Not for my pci Card. I have tested this by printing the Device ID and Vendor ID of the corresponding device_t in my probe. You are trying to say even cbb probes for my device and return BUS_PROBE_DEFAULT still my probe function will be called. Just clarify it. But my testing seems that my probe is not called for my pci device Bagavathy kumar .M=20 -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com]=20 Sent: Monday, October 06, 2008 12:27 PM To: Bagavathy Kumar Mahendran=20 Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID > Thanks for your support. my probe is getting called for all > the bridges not for my pci device. so please provide the fix . > > OR=20 > Is any other way available for making my driver to override the probe of > cbb driver for my corresponding device (With out changing cbb driver). If your probe returns a higher number that's negative, it will. Unless cbb is returning 0, your probe routine will get called. Make sure it isn't. Code inspection suggests that it isn't. Warner > With regards, > Bagavathy kumar .M >=20 > -----Original Message----- > From: M. Warner Losh [mailto:imp@bsdimp.com]=20 > Sent: Friday, October 03, 2008 11:03 AM > To: Bagavathy Kumar Mahendran=20 > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called for > mycorrespondingdevice ID and Vendor ID >=20 > In message: > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > "Bagavathy Kumar Mahendran " > writes: > :=20 > : Dear Baldwin, > : Thanks for your support .but my pci probe function is > not > : getting called for my device id and vendor id. Because pccbb driver > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > any > : other option for me to make my_pciprobe function to be called for my > : corresponding device id and vendor id. >=20 > That's not why your probe isn't called. Setting a description is > standard behavior for the probe routine. Are you sure that the device > probe routine is getting called at all for any device? Have you tried > just leaving cbb out of the kernel? I recently fixed the original > problem in cbb (the fact it doesn't check the bridge type too), maybe > you could try to pick up that fix as well? >=20 > Warner >=20 >=20 > : Thanks, > :=20 > : Regards, > : Bagavathy kumar .M > :=20 > :=20 > :=20 > : -----Original Message----- > : From: John Baldwin [mailto:jhb@freebsd.org]=20 > : Sent: Wednesday, October 01, 2008 8:57 PM > : To: freebsd-hackers@freebsd.org > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for my > : correspondingdevice ID and Vendor ID > :=20 > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > : wrote: > : >=20 > : > Dear All, > : > Iam writing a new driver for a SAS/SATA Controller > having > : a > : > Class ID -0x01 > : > Sub Class - 0x07 > : > Programming Interface - 0x00 > : >=20 > : > Hence instead of my probe function the Static build Card Bus Driver > : cbb > : > is attaching just by simply checking sub class 0x07 and programming > : > interface 0x00.hence my probe gets failed. Kindly help me in > resolving > : > this .what I thought is to add the card bus driver a checking of > CLASS > : > ID in its pci probe function. > :=20 > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably return > : GENERIC=20 > : in the case where it matches only on class codes). Your driver just > : needs to=20 > : return a numerically higher value (but still < 0) to claim the device. > : You=20 > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > :=20 > : --=20 > : John Baldwin > :=20 > : DISCLAIMER: > : > ------------------------------------------------------------------------ > ----------------------------------------------- > :=20 > : The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > : It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in=20 > : this email are solely those of the author and may not necessarily > reflect the opinions of HCL or its affiliates. > : Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of=20 > : this message without the prior written consent of the author of this > e-mail is strictly prohibited. If you have > : received this email in error please delete it and notify the sender > immediately. Before opening any mail and=20 > : attachments please check them for viruses and defect. > :=20 > : > ------------------------------------------------------------------------ > ----------------------------------------------- > :=20 > :=20 >=20 >=20 From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 07:37:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C652C1065699; Mon, 6 Oct 2008 07:37:10 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws03.hcl.in (chn-hclin-gws01.hcl.in [203.105.186.19]) by mx1.freebsd.org (Postfix) with ESMTP id 1682B8FC2C; Mon, 6 Oct 2008 07:37:10 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws03.hcl.in (gws03 [10.249.64.134]) by localhost.hcl.in (Postfix) with ESMTP id 813BE37C016; Mon, 6 Oct 2008 13:07:08 +0530 (IST) Received: from chn-egw02-out.corp.hcl.in (unknown [10.249.64.38])by gws03.hcl.in (Postfix) with ESMTPid 5052837C099; Mon, 6 Oct 2008 13:07:08 +0530 (IST) Received: from chn-hclt-evs05.HCLT.CORP.HCL.IN ([10.101.26.13]) by chn-egw02-out.corp.hcl.in with Microsoft SMTPSVC(6.0.3790.3959); Mon, 6 Oct 2008 13:07:07 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 6 Oct 2008 13:07:04 +0530 Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A695@chn-hclt-evs05.HCLT.CORP.HCL.IN> In-Reply-To: <20081006.012619.78741615.imp@bsdimp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID Thread-Index: AcknhTPI5lBt49ibRAOXvo7qxSHFYgAAEmGQ References: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081006.005657.71122961.imp@bsdimp.com><68C9F31EF19DB6448F515EF29 4028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.012619.78741615.imp@bsdimp.com> From: "Bagavathy Kumar Mahendran " To: "Warner Losh" X-OriginalArrivalTime: 06 Oct 2008 07:37:07.0999 (UTC) FILETIME=[561382F0:01C92786] X-imss-version: 2.051 X-imss-result: Passed X-imss-scanInfo: M:T L:E SM:1 X-imss-tmaseResult: TT:1 TS:-40.5216 TC:1F TRN:87 TV:5.5.1026(16200.005) X-imss-scores: Clean:100.00000 C:0 M:0 S:0 R:0 X-imss-settings: Baseline:1 C:1 M:1 S:1 R:1 (0.0000 0.0000) X-Mailman-Approved-At: Mon, 06 Oct 2008 11:23:50 +0000 Cc: freebsd-hackers@freebsd.org Subject: RE: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 07:37:11 -0000 static int My_probe(device_t dev) { uint32_t progif; uint32_t subclass; uint32_t class; device_printf(dev, " Probe\n Vendor ID : 0x%x\n Device ID : 0x%x\n",pci_get_vendor(dev), pci_get_device(dev)); class =3D pci_get_class(dev); subclass =3D pci_get_subclass(dev); progif =3D pci_get_progif(dev); if (class =3D=3D 0x1 && subclass =3D=3D 0x07 && progif =3D=3D 0x00) { printf("probe successful!\n"); device_set_desc(dev, "My_Probe"); return (BUS_PROBE_DEFAULT); } return (ENXIO); } Here above I have attached my pci probe. The vendor ID and Device ID of all the above PCI bridges are printing But instead of My_probe function for my PCI Card the probe of the cbb driver is called. This is the exact problem but when I remove the cbb driver from the kernel My_probe function is called for my PCI Card too. Regards, Bagavathy kumar .M -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com]=20 Sent: Monday, October 06, 2008 12:56 PM To: Bagavathy Kumar Mahendran=20 Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, >=20 > My probe is getting called for the parent bridge devices > .but=20 > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. >=20 > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M=20 >=20 >=20 > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com]=20 > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran=20 > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID >=20 > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR=20 > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). >=20 > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. >=20 > Warner >=20 >=20 >=20 > > With regards, > > Bagavathy kumar .M > >=20 > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com]=20 > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran=20 > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > >=20 > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > :=20 > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > >=20 > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > >=20 > > Warner > >=20 > >=20 > > : Thanks, > > :=20 > > : Regards, > > : Bagavathy kumar .M > > :=20 > > :=20 > > :=20 > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org]=20 > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > :=20 > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : >=20 > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : >=20 > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > :=20 > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC=20 > > : in the case where it matches only on class codes). Your driver just > > : needs to=20 > > : return a numerically higher value (but still < 0) to claim the > device. > > : You=20 > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > :=20 > > : --=20 > > : John Baldwin > > :=20 > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > :=20 > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in=20 > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of=20 > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and=20 > > : attachments please check them for viruses and defect. > > :=20 > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > :=20 > > :=20 > >=20 > >=20 >=20 >=20 From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 07:40:57 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75AF1106568F; Mon, 6 Oct 2008 07:40:57 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (chn-hclin-gws02.hcl.in [203.105.186.20]) by mx1.freebsd.org (Postfix) with ESMTP id 8C1C88FC2B; Mon, 6 Oct 2008 07:40:56 +0000 (UTC) (envelope-from bagavathykumar.m@hcl.in) Received: from gws04.hcl.in (gws04 [10.249.64.135]) by localhost.hcl.in (Postfix) with ESMTP id 3F51C360028; Mon, 6 Oct 2008 13:10:55 +0530 (IST) Received: from chn-egw01-out.corp.hcl.in (unknown [10.249.64.37])by gws04.hcl.in (Postfix) with ESMTPid 1078B360051; Mon, 6 Oct 2008 13:10:55 +0530 (IST) Received: from chn-hclt-evs05.HCLT.CORP.HCL.IN ([10.101.26.13]) by chn-egw01-out.corp.hcl.in with Microsoft SMTPSVC(6.0.3790.3959); Mon, 6 Oct 2008 13:10:54 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 6 Oct 2008 13:10:42 +0530 Message-ID: <68C9F31EF19DB6448F515EF294028FDEE9A6A6@chn-hclt-evs05.HCLT.CORP.HCL.IN> In-Reply-To: <20081006.012619.78741615.imp@bsdimp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID Thread-Index: AcknhTPI5lBt49ibRAOXvo7qxSHFYgAAYvzw References: <68C9F31EF19DB6448F515EF294028FDEE9A4A5@chn-hclt-evs05.HCLT.CORP .HCL.IN><20081006.005657.71122961.imp@bsdimp.com><68C9F31EF19DB6448F515EF29 4028FDEE9A624@chn-hclt-evs05.HCLT.CORP.HCL.IN> <20081006.012619.78741615.imp@bsdimp.com> From: "Bagavathy Kumar Mahendran " To: "Warner Losh" X-OriginalArrivalTime: 06 Oct 2008 07:40:54.0599 (UTC) FILETIME=[DD23ED70:01C92786] X-imss-version: 2.051 X-imss-result: Passed X-imss-scanInfo: M:T L:E SM:1 X-imss-tmaseResult: TT:1 TS:-39.0085 TC:1F TRN:82 TV:5.5.1026(16200.005) X-imss-scores: Clean:100.00000 C:0 M:0 S:0 R:0 X-imss-settings: Baseline:1 C:1 M:1 S:1 R:1 (0.0000 0.0000) X-Mailman-Approved-At: Mon, 06 Oct 2008 11:24:07 +0000 Cc: freebsd-hackers@freebsd.org Subject: RE: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 07:40:58 -0000 static devclass_t My_devclass; DRIVER_MODULE(My, pci, My_driver, My_devclass, 0, 0); With Regards, Bagavathy kumar .M -----Original Message----- From: Warner Losh [mailto:imp@bsdimp.com]=20 Sent: Monday, October 06, 2008 12:56 PM To: Bagavathy Kumar Mahendran=20 Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org Subject: Re: i386/127710: My driver PCI probe is not calledformycorrespondingdevice ID and Vendor ID From: "Bagavathy Kumar Mahendran " Subject: RE: i386/127710: My driver PCI probe is not called formycorrespondingdevice ID and Vendor ID Date: Mon, 6 Oct 2008 12:36:44 +0530 > Dear Warner, >=20 > My probe is getting called for the parent bridge devices > .but=20 > Not for my pci Card. I have tested this by printing the Device ID and > Vendor ID of the corresponding device_t in my probe. >=20 > You are trying to say even cbb probes for my device and return > BUS_PROBE_DEFAULT still my probe function will be called. Just clarify > it. > But my testing seems that my probe is not called for my pci device Can you send me the DRIVER_MODULE line in your driver? Warner > Bagavathy kumar .M=20 >=20 >=20 > -----Original Message----- > From: Warner Losh [mailto:imp@bsdimp.com]=20 > Sent: Monday, October 06, 2008 12:27 PM > To: Bagavathy Kumar Mahendran=20 > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > Subject: Re: i386/127710: My driver PCI probe is not called > formycorrespondingdevice ID and Vendor ID >=20 > > Thanks for your support. my probe is getting called for > all > > the bridges not for my pci device. so please provide the fix . > > > > OR=20 > > Is any other way available for making my driver to override the probe > of > > cbb driver for my corresponding device (With out changing cbb driver). >=20 > If your probe returns a higher number that's negative, it will. > Unless cbb is returning 0, your probe routine will get called. Make > sure it isn't. Code inspection suggests that it isn't. >=20 > Warner >=20 >=20 >=20 > > With regards, > > Bagavathy kumar .M > >=20 > > -----Original Message----- > > From: M. Warner Losh [mailto:imp@bsdimp.com]=20 > > Sent: Friday, October 03, 2008 11:03 AM > > To: Bagavathy Kumar Mahendran=20 > > Cc: jhb@freebsd.org; freebsd-hackers@freebsd.org > > Subject: Re: i386/127710: My driver PCI probe is not called for > > mycorrespondingdevice ID and Vendor ID > >=20 > > In message: > > > <68C9F31EF19DB6448F515EF294028FDEE99BCE@chn-hclt-evs05.HCLT.CORP.HCL.IN> > > "Bagavathy Kumar Mahendran " > > writes: > > :=20 > > : Dear Baldwin, > > : Thanks for your support .but my pci probe function is > > not > > : getting called for my device id and vendor id. Because pccbb driver > > : already sets the device_set_desc as PCI-CardBus Bridge. So is there > > any > > : other option for me to make my_pciprobe function to be called for my > > : corresponding device id and vendor id. > >=20 > > That's not why your probe isn't called. Setting a description is > > standard behavior for the probe routine. Are you sure that the device > > probe routine is getting called at all for any device? Have you tried > > just leaving cbb out of the kernel? I recently fixed the original > > problem in cbb (the fact it doesn't check the bridge type too), maybe > > you could try to pick up that fix as well? > >=20 > > Warner > >=20 > >=20 > > : Thanks, > > :=20 > > : Regards, > > : Bagavathy kumar .M > > :=20 > > :=20 > > :=20 > > : -----Original Message----- > > : From: John Baldwin [mailto:jhb@freebsd.org]=20 > > : Sent: Wednesday, October 01, 2008 8:57 PM > > : To: freebsd-hackers@freebsd.org > > : Cc: Bagavathy Kumar Mahendran ; Warner Losh > > : Subject: Re: FW: i386/127710: My driver PCI probe is not called for > my > > : correspondingdevice ID and Vendor ID > > :=20 > > : On Wednesday 01 October 2008 08:50:15 am Bagavathy Kumar Mahendran > > : wrote: > > : >=20 > > : > Dear All, > > : > Iam writing a new driver for a SAS/SATA Controller > > having > > : a > > : > Class ID -0x01 > > : > Sub Class - 0x07 > > : > Programming Interface - 0x00 > > : >=20 > > : > Hence instead of my probe function the Static build Card Bus > Driver > > : cbb > > : > is attaching just by simply checking sub class 0x07 and > programming > > : > interface 0x00.hence my probe gets failed. Kindly help me in > > resolving > > : > this .what I thought is to add the card bus driver a checking of > > CLASS > > : > ID in its pci probe function. > > :=20 > > : The pccbb driver returns BUS_PROBE_DEFAULT (it should probably > return > > : GENERIC=20 > > : in the case where it matches only on class codes). Your driver just > > : needs to=20 > > : return a numerically higher value (but still < 0) to claim the > device. > > : You=20 > > : can probably use BUS_PROBE_VENDOR or BUS_PROBE_DEFAULT + 1. > > :=20 > > : --=20 > > : John Baldwin > > :=20 > > : DISCLAIMER: > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > :=20 > > : The contents of this e-mail and any attachment(s) are confidential > and > > intended for the named recipient(s) only. > > : It shall not attach any liability on the originator or HCL or its > > affiliates. Any views or opinions presented in=20 > > : this email are solely those of the author and may not necessarily > > reflect the opinions of HCL or its affiliates. > > : Any form of reproduction, dissemination, copying, disclosure, > > modification, distribution and / or publication of=20 > > : this message without the prior written consent of the author of this > > e-mail is strictly prohibited. If you have > > : received this email in error please delete it and notify the sender > > immediately. Before opening any mail and=20 > > : attachments please check them for viruses and defect. > > :=20 > > : > > > ------------------------------------------------------------------------ > > ----------------------------------------------- > > :=20 > > :=20 > >=20 > >=20 >=20 >=20 From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 08:59:17 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4F061065690; Mon, 6 Oct 2008 08:59:17 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [91.103.162.4]) by mx1.freebsd.org (Postfix) with ESMTP id 4857B8FC08; Mon, 6 Oct 2008 08:59:17 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id EE88A19E023; Mon, 6 Oct 2008 10:59:15 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B313A19E019; Mon, 6 Oct 2008 10:59:13 +0200 (CEST) Message-ID: <48E9D382.4000001@quip.cz> Date: Mon, 06 Oct 2008 10:59:46 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: Eygene Ryabinkin References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 11:24:16 +0000 Cc: freebsd-hackers@freebsd.org, bug-followup@freebsd.org, freebsd-ports@freebsd.org, portmgr@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 08:59:17 -0000 Eygene Ryabinkin wrote: > Miroslav, good day. > > Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote: > >>I am busy these days, but it is nice to read about your progress. I hope >>I will get some time to test all of these large patches in a few days >>and I will report back my experiences! > > > Fine, thank you! I am re-CC'ing bug-followup@ to track this letter, > since it contains some useful information that should go into GNATS. > > >>One note before tests... do -n flag always download new INDEX file, or >>is it possible to use one already existing in /usr/ports? > > > Currently, it is downloads bzipped INDEX file to /var/db/portaudit every > time, but it uses mirror mode, so if remote file hadn't changed at all, > all network expences are just the HTTP's HEAD request and reply. > > I can add another variable to the portaudit to force the usage of the > existing INDEX file, if it is needed. By the way, how are you keeping > your INDEX file up to date (your proposed usage of 'pkg_version -I' > implies that you're always rely on it)? I am just curious -- my INDEX > files are almost always stay unupdated, even if I am using portupgrade. I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in my crontab, so I get INDEX updated every night before nightly security e-mail is generated. > And there can be another way if one keeps ports tree updated: utility > can use 'make' to determine the version that is currently available on > the examined host. > > But downloading the INDEX file from the central server seemed to be the > best way, since it almost always gives one the latest port versions, so > I had implemented this in a first place. My previous question was not against your solution, it seems useful to have really actual data from the fresh INDEX. It was just a question "how it is done". Maybe someone will be happier to use the existing INDEX because of traffic on some GPRS internet connection or because of the own INDEX creation. (it is not my case, I have all machines as the servers with enough connectivity) ;) > Don't know, however, how the badly the load to the central HTTP server > will be raised. I am using just two first fields from the INDEX file, > so I can use such a stripped file. For me, the reduction was about > 6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345. > > I am CC'ing the portmgr team. Guys, could you quickly glance over these > patches and determine if they are useful to the project in large? If > yes, then may be such a stripped INDEX can be created on the FreeBSD > servers (via cut -f1-2 -d'|' INDEX-N)? > > Thanks! From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 10:25:15 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CB04106568A for ; Mon, 6 Oct 2008 10:25:15 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 5CE738FC12 for ; Mon, 6 Oct 2008 10:25:14 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 1772 invoked by uid 399); 6 Oct 2008 09:58:46 -0000 Received: from a91-152-230-218.elisa-laajakaista.fi (HELO ?192.168.1.34?) (yurtesen@ispro.net@91.152.230.218) by mail.ispro.net with ESMTPAM; 6 Oct 2008 09:58:46 -0000 X-Originating-IP: 91.152.230.218 Message-ID: <48E9E1BB.6020908@ispro.net> Date: Mon, 06 Oct 2008 13:00:27 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 11:24:29 +0000 Cc: Subject: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 10:25:15 -0000 Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414&postcount=9 I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 12:22:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6C051065692; Mon, 6 Oct 2008 12:22:18 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 61C188FC12; Mon, 6 Oct 2008 12:22:18 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=jhQcisJ2fDvgWs8EQuyg49SWqIfjUm2JOmpApnRytxMCCOeVMPRye5hZUQ6WMnYeV/BIUarlxaQHAeV5LLVREvY9WiFvTldAjei/TKX4NjrvWGdtTkILNM6DuyJNN4fNzX9e6T/LxD6bUuoj9OpL4m8PRHY1s/lPUkQVDfhDHFA=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1Kmp6N-000Mb5-3Q; Mon, 06 Oct 2008 16:22:15 +0400 Date: Mon, 6 Oct 2008 16:22:13 +0400 From: Eygene Ryabinkin To: Mel Message-ID: References: <200810061124.55209.fbsd.hackers@rachie.is-a-geek.net> <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3BL0Ng5d1iPP6beU" Content-Disposition: inline In-Reply-To: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 12:22:19 -0000 --3BL0Ng5d1iPP6beU Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Mel, Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote: > On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > Once you have the origin of the port, you can: > - make -C $PORTSDIR/$origin -V PKGNAME > - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} > - get the matching origin(s) out of a downloaded INDEX.bz2 >=20 > This covers the majority of cases. >=20 > What portaudit lacks, is providing the origin along with the installed pa= ckage > name in easily parseable format. So, a central server wanting to query al= l=20 > the machines for vulnerable packages, now has to do an extra step of goin= g=20 > into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line,= =20 > while (port|pkg_)audit has just been there. >=20 > This would be something I'd expect: > ssh clientmachine "/usr/sbin/pkg_audit -l" > foo-1.2,3:misc/foo > bar-4.5_6:devel/bar > ... OK, got it. There is one neat: pkg_audit should be feeded with the contents of the auditfile and the latter is located in the tar archive. So, if you wouldn't mind about the following sequence ----- tar xf /var/db/portaudit/auditfile.tbz pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5 ----- then I can add the flag '-o' to the portaudit-checknew: it will additionally output the port origin along with the new version. Is that what you meant? --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --3BL0Ng5d1iPP6beU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjqAvUACgkQthUKNsbL7YiaFACfVxP/ieDIZZrUGE4O+DKgfvTh YpgAn02ufj2yxuThuKezaIdezmBYuDYt =UeZL -----END PGP SIGNATURE----- --3BL0Ng5d1iPP6beU-- From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 12:29:36 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEF6F1065692 for ; Mon, 6 Oct 2008 12:29:36 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by mx1.freebsd.org (Postfix) with ESMTP id 89F7B8FC15 for ; Mon, 6 Oct 2008 12:29:36 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from gidgate.gid.co.uk (80-46-130-69.static.dsl.as9105.com [80.46.130.69]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id m96CEXuv030231; Mon, 6 Oct 2008 13:14:33 +0100 (BST) (envelope-from rb@gid.co.uk) Received: from rbPBP.gid.co.uk ([194.32.164.6]) by gidgate.gid.co.uk (8.13.8/8.13.8) with ESMTP id m96CE6R2091009; Mon, 6 Oct 2008 13:14:29 +0100 (BST) (envelope-from rb@gid.co.uk) Message-Id: <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> From: Bob Bishop To: Evren Yurtesen In-Reply-To: <48E9E1BB.6020908@ispro.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Mon, 6 Oct 2008 13:14:06 +0100 References: <48E9E1BB.6020908@ispro.net> X-Mailer: Apple Mail (2.929.2) Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 12:29:37 -0000 Hi, On 6 Oct 2008, at 11:00, Evren Yurtesen wrote: > As far as I can see, there is no continuous backup solution for > FreeBSD at the moment. I talked with R1Soft and they seem to not be > able to support FreeBSD and need help. > > Does anybody have free time and skills to give a hand? Please see: > http://forum.r1soft.com/showpost.php?p=3414&postcount=9 Should be possible to do this with a geom(4) class? > I know people who want to switch to using FreeBSD but cant do that > because their backup solution (R1Soft) does not support FreeBSD :( > > Thanks, > Evren > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org > " > -- Bob Bishop rb@gid.co.uk From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 12:40:51 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4CBF106568D; Mon, 6 Oct 2008 12:40:51 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from mail.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id 9C8978FC0A; Mon, 6 Oct 2008 12:40:51 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from localhost (mail.rachie.is-a-geek.net [192.168.2.101]) by mail.rachie.is-a-geek.net (Postfix) with ESMTP id A8DD8AFBC01; Mon, 6 Oct 2008 04:40:50 -0800 (AKDT) From: Mel To: freebsd-hackers@freebsd.org Date: Mon, 6 Oct 2008 14:40:48 +0200 User-Agent: KMail/1.9.7 References: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> Cc: freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 12:40:51 -0000 On Monday 06 October 2008 14:22:13 Eygene Ryabinkin wrote: > Mel, > > Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote: > > On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote: > > Once you have the origin of the port, you can: > > - make -C $PORTSDIR/$origin -V PKGNAME > > - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE} > > - get the matching origin(s) out of a downloaded INDEX.bz2 > > > > This covers the majority of cases. > > > > What portaudit lacks, is providing the origin along with the installed > > package name in easily parseable format. So, a central server wanting to > > query all the machines for vulnerable packages, now has to do an extra > > step of going into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment > > ORIGIN: line, while (port|pkg_)audit has just been there. > > > > This would be something I'd expect: > > ssh clientmachine "/usr/sbin/pkg_audit -l" > > foo-1.2,3:misc/foo > > bar-4.5_6:devel/bar > > ... > > OK, got it. There is one neat: pkg_audit should be feeded with the > contents of the auditfile and the latter is located in the tar archive. > So, if you wouldn't mind about the following sequence > ----- > tar xf /var/db/portaudit/auditfile.tbz > pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5 > ----- > then I can add the flag '-o' to the portaudit-checknew: it will > additionally output the port origin along with the new version. > > Is that what you meant? What I meant is the '-o' flag in pkg_audit, so I can figure out myself whether it's new or not and my buildserver can prioritize it's builds based on vulnerable packages it's clients have installed. The origin is the unique key that identifies any port, so that's vital information in a pipeline. -- Mel From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 13:08:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40D0D1065689; Mon, 6 Oct 2008 13:08:53 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id DE3EE8FC39; Mon, 6 Oct 2008 13:08:52 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=V6A44P8x4I+U6wngkw60b1tJdo7qmPFWS4B0XAwBVLllSIuY0tZXU0j98J+avs03xegTy52cqDE3ZwGeRI8WuLlHfFCxorcKdN3VH5e2ouKj6fw15Wd0io/SOPZKKnGPUNW+O/OK4cSg8uLHtXAIPSHIH/2IWEF+dSOCTrxki6c=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmppR-000PvV-Qu; Mon, 06 Oct 2008 17:08:49 +0400 Date: Mon, 6 Oct 2008 17:08:48 +0400 From: Eygene Ryabinkin To: Mel Message-ID: References: <200810061307.51977.fbsd.hackers@rachie.is-a-geek.net> <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Lcqw/5GzJkKu54yC" Content-Disposition: inline In-Reply-To: <200810061440.49113.fbsd.hackers@rachie.is-a-geek.net> Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, bug-followup@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 13:08:53 -0000 --Lcqw/5GzJkKu54yC Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Mel, Mon, Oct 06, 2008 at 02:40:48PM +0200, Mel wrote: > What I meant is the '-o' flag in pkg_audit, so I can figure out myself wh= ether > it's new or not and my buildserver can prioritize it's builds based on=20 > vulnerable packages it's clients have installed. The origin is the unique= key=20 > that identifies any port, so that's vital information in a pipeline. Ah, OK: no problems, will do it. --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --Lcqw/5GzJkKu54yC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjqDeAACgkQthUKNsbL7Yj93wCgpy+aySRnxYM8m/O33Yey4nbJ HcIAnRQYWuSduoKqgb8xhY4nx6flkavR =Zw+M -----END PGP SIGNATURE----- --Lcqw/5GzJkKu54yC-- From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 14:33:15 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8DA0410656A8 for ; Mon, 6 Oct 2008 14:33:15 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 7467C8FC1E for ; Mon, 6 Oct 2008 14:33:14 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 72815 invoked by uid 399); 6 Oct 2008 14:33:30 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 6 Oct 2008 14:33:30 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48EA21AE.80607@ispro.net> Date: Mon, 06 Oct 2008 17:33:18 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Bob Bishop References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> In-Reply-To: <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 15:05:05 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 14:33:15 -0000 Bob Bishop wrote: >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > Should be possible to do this with a geom(4) class? > I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 15:36:06 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A97D106568B for ; Mon, 6 Oct 2008 15:36:06 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from gv-out-0910.google.com (gv-out-0910.google.com [216.239.58.188]) by mx1.freebsd.org (Postfix) with ESMTP id 8A0E48FC13 for ; Mon, 6 Oct 2008 15:36:05 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: by gv-out-0910.google.com with SMTP id n8so334610gve.39 for ; Mon, 06 Oct 2008 08:36:04 -0700 (PDT) Received: by 10.102.253.13 with SMTP id a13mr2817427mui.111.1223305746604; Mon, 06 Oct 2008 08:09:06 -0700 (PDT) Received: by 10.103.229.14 with HTTP; Mon, 6 Oct 2008 08:09:06 -0700 (PDT) Message-ID: Date: Mon, 6 Oct 2008 18:09:06 +0300 From: "Vlad GALU" To: "Evren Yurtesen" In-Reply-To: <48EA21AE.80607@ispro.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 15:36:06 -0000 On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > Bob Bishop wrote: > >>> Does anybody have free time and skills to give a hand? Please see: >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >> >> Should be possible to do this with a geom(4) class? >> > > I am not saying it is impossible. They just need somebody to put them to > right track I guess. I personally cant do that. It would be nice if somebody > who has knowledge in this area contacts r1soft. At the very least r1soft > seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key feature > for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... > > Thanks, > Evren > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- ~/.signature: no such file or directory From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 15:46:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EAA0106568E; Mon, 6 Oct 2008 15:46:14 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id CAC238FC1D; Mon, 6 Oct 2008 15:46:13 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1KmsHk-0003eV-G4; Mon, 06 Oct 2008 17:46:12 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Robert Watson In-reply-to: References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> Comments: In-reply-to Robert Watson message dated "Sun, 05 Oct 2008 23:21:05 +0100." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 06 Oct 2008 17:46:12 +0200 From: Danny Braniss Message-ID: Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: bad NFS/UDP performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 15:46:14 -0000 > > On Sat, 4 Oct 2008, Danny Braniss wrote: > > > at the moment, the best I can do is run it on a different hardware that has > > if_em, the results are in > > ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the > > benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I > > get the same numbers with an older kernel). > > Dear Danny: > > Unfortunately, I was left slightly unclear on the comparison you are making > above. Could you confirm whether or not, with if_em, you see a performance > regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, > and if you do, whether or not the RLOCK->WLOCK change has any effect on > performance? It would be nice to know on the same hardware but at least with > different hardware we get a sense of whether or not this might affect other > systems or whether it's limited to a narrower set of configurations. > > Thanks, 7.1-1000.em vanilla 7.1 1 x Intel Core Duo 7.1-1000.x2200.em vanilla 7.1 2 x Dual-Core AMD Opteron 7.0-1000.x2200.em 7.0 + RLOCK->WLOCK the plot thickens. I put an em card in, and the throughput is almost the same than with the bge. all the tests were done on the same host, a Sun x2200/amd/2cpux2core except for the one over the weekend that is a intel Core Duo, and not the same if_em card, sorry about that but one has PCI X, the other PCI Express :-(. what is becoming obvious is that NFS/UDP is very temperamental/sensitive :-) danny From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 16:23:02 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57E741065687 for ; Mon, 6 Oct 2008 16:23:02 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-gx0-f21.google.com (mail-gx0-f21.google.com [209.85.217.21]) by mx1.freebsd.org (Postfix) with ESMTP id EC5B88FC12 for ; Mon, 6 Oct 2008 16:23:01 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by gxk14 with SMTP id 14so5004180gxk.19 for ; Mon, 06 Oct 2008 09:23:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=O6msH7629LnwS8pP+gqKYT9lyI/xz5lq7Akg/T3mgM0=; b=XHLy33VpAU/N4iUxXvVwdAzMYFqNzif3g2ZDfs78H10Ry+3CZzKhBJNzxNB8kDiiy+ SHr9hhSWXJ8OoQy374gmu2PL6GWrrUoeEjzKmwTeevArN91lf0PGmvZacfa05WJ1f44z TUcyk7lwqEqznurA4MYU5dM16C5e6DFq8HBrs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=kCV69nqMzcGEJ/vH8TtLPQTHxfr8k5XnUaBkCm+JOJVKxVabTGsblzHqz5qrYldphz 29nQzi0J5Xqq3ZMqltLmw0p/7vewWF6E+J6bj0IMdYri2BIf+zg/RD100JxNWUtC7x/x OVK8hQSge9hr5AhGBNCprvYdP8DNdfiL/iuMQ= Received: by 10.150.228.2 with SMTP id a2mr7885759ybh.111.1223308326870; Mon, 06 Oct 2008 08:52:06 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Mon, 6 Oct 2008 08:52:06 -0700 (PDT) Message-ID: <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> Date: Mon, 6 Oct 2008 11:52:06 -0400 From: "Zaphod Beeblebrox" To: "Evren Yurtesen" In-Reply-To: <48EA21AE.80607@ispro.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 16:23:02 -0000 On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen wrote: [regarding r1soft.com, ...] > I am not saying it is impossible. They just need somebody to put them to > right track I guess. I personally cant do that. It would be nice if somebody > who has knowledge in this area contacts r1soft. At the very least r1soft > seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key feature > for many hosters. FreeBSD is loosing users because of this issue. Actually, having looked at the site, the hammer filesystem and it's replication strategy seem to be the most applicable technology (but then you wouldn't even need these guys --- you'd be doing it yourself). Like anything, though, live applications will require special treatment. Keeping a live filesystem replicated does in no way guarentee that your database (for instance) will be sane at any particular moment. It sounds like these guys have made allowances for MySQL (they specifically mention it), but this won't help the PostgreSQL users, etc. I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The most fundamental example I can think of to show this principle, however, is the fact that if the HTTP standard required web browsers to try all A records (instead of randomly choosing one), web site redundancy would be amazingly simple to achieve. Consider that most other protocols right down to telnet do this, but web browsers don't. As a complete aside, if you have both AAAA and A records for your website, you have a form of poor-man's redundancy available to you --- with the caveat that it only works for people with both IPv6 and IPv4 connectivity. Browsers will try AAAA followed by A if the former doesn't respond. From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 18:05:38 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B7E8106568F for ; Mon, 6 Oct 2008 18:05:38 +0000 (UTC) (envelope-from volker@vwsoft.com) Received: from frontmail.ipactive.de (frontmail.maindns.de [85.214.95.103]) by mx1.freebsd.org (Postfix) with ESMTP id 240788FC19 for ; Mon, 6 Oct 2008 18:05:37 +0000 (UTC) (envelope-from volker@vwsoft.com) Received: from mail.vtec.ipme.de (Q7c2d.q.ppp-pool.de [89.53.124.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by frontmail.ipactive.de (Postfix) with ESMTP id 3475212883F; Mon, 6 Oct 2008 20:05:29 +0200 (CEST) Received: from cesar.sz.vwsoft.com (cesar.sz.vwsoft.com [192.168.16.3]) by mail.vtec.ipme.de (Postfix) with ESMTP id A7EC02E913; Mon, 6 Oct 2008 20:04:17 +0200 (CEST) Message-ID: <48EA5357.8050503@vwsoft.com> Date: Mon, 06 Oct 2008 20:05:11 +0200 From: Volker User-Agent: Thunderbird 2.0.0.17 (X11/20080930) MIME-Version: 1.0 To: Chuck Robey References: <48E90904.4020007@telenix.org> In-Reply-To: <48E90904.4020007@telenix.org> X-Enigmail-Version: 0.95.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit MailScanner-NULL-Check: 1223921072.36924@3pKM85/myTEiDN4IDH12tg X-MailScanner-ID: A7EC02E913.E20B5 X-VWSoft-MailScanner: Found to be clean X-MailScanner-From: volker@vwsoft.com X-ipactive-MailScanner-Information: Please contact the ISP for more information X-ipactive-MailScanner: Found to be clean X-ipactive-MailScanner-From: volker@vwsoft.com Cc: FreeBSD-Hackers Subject: Re: touch screen recommendation? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 18:05:38 -0000 On 12/23/-58 20:59, Chuck Robey wrote: > I was wondering if anyone here had a recommendation for a touch screen, > specifically to run on FreeBSD? Any user report? I am wondering how this belongs to hackers@? Now the real answer: It depends. Touchscreen != Touchscreen, as there are different technologies available (better ask a sales person for your application). If you just want to get a "click" from a screen device, you can use any. First, I'm not aware of any working USB based devices working under FreeBSD (except mine, not yet released to public). There are other devices with RS-232 connectors for which you may find X.Org drivers in the port tree. For the usb devices, I've got working drivers for GeneralTouch ST-601U (buggy firmware, dumb tech support, not my preference, but it's working) and @-Touch SAWUSB. Next week I should have a working driver for Zytronic devices and one or two others. All for usb devices, everything for the BSDs only but not yet released. From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 18:52:21 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D90631065694 for ; Mon, 6 Oct 2008 18:52:21 +0000 (UTC) (envelope-from volker@vwsoft.com) Received: from frontmail.ipactive.de (frontmail.maindns.de [85.214.95.103]) by mx1.freebsd.org (Postfix) with ESMTP id 8F1528FC2E for ; Mon, 6 Oct 2008 18:52:21 +0000 (UTC) (envelope-from volker@vwsoft.com) Received: from mail.vtec.ipme.de (Q7c2d.q.ppp-pool.de [89.53.124.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by frontmail.ipactive.de (Postfix) with ESMTP id 865C212883F; Mon, 6 Oct 2008 20:19:48 +0200 (CEST) Received: from cesar.sz.vwsoft.com (cesar.sz.vwsoft.com [192.168.16.3]) by mail.vtec.ipme.de (Postfix) with ESMTP id 5802B2E913; Mon, 6 Oct 2008 20:18:46 +0200 (CEST) Message-ID: <48EA56BB.6040702@vwsoft.com> Date: Mon, 06 Oct 2008 20:19:39 +0200 From: Volker User-Agent: Thunderbird 2.0.0.17 (X11/20080930) MIME-Version: 1.0 To: Evren Yurtesen References: <48E9E1BB.6020908@ispro.net> In-Reply-To: <48E9E1BB.6020908@ispro.net> X-Enigmail-Version: 0.95.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit MailScanner-NULL-Check: 1223921932.15983@L22lGKvnB1qa2DZLg284Bg X-MailScanner-ID: 5802B2E913.29331 X-VWSoft-MailScanner: Found to be clean X-MailScanner-From: volker@vwsoft.com X-ipactive-MailScanner-Information: Please contact the ISP for more information X-ipactive-MailScanner: Found to be clean X-ipactive-MailScanner-From: volker@vwsoft.com Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 18:52:21 -0000 On 12/23/-58 20:59, Evren Yurtesen wrote: > Hello, > > As far as I can see, there is no continuous backup solution for FreeBSD > at the moment. I talked with R1Soft and they seem to not be able to > support FreeBSD and need help. > > Does anybody have free time and skills to give a hand? Please see: > http://forum.r1soft.com/showpost.php?p=3414&postcount=9 Quoting the thread: > and they are usually very busy... and they like to work for hire. Wrong term: Even open source folks need something for living. That's too cheap and I don't suggest supporting a company like this, as they're expecting somebody else to do their job and make profit of it. They're free to use a system like FreeBSD for their products, they're free to modify and redistribute it but they need to do their job and don't expect anybody else to code for them for free. Next quote of thread: > real world competency writing block device drivers for FreeBSD Let me check that out... %ls -l /dev | grep ^b % Well, I guess, nobody is able to do so and it's noted in the handbook or in GNN's "The Design and Implementation of the FreeBSD operating system". A good read. > I know people who want to switch to using FreeBSD but cant do that > because their backup solution (R1Soft) does not support FreeBSD :( What's wrong with solutions like Bacula? It's working, cross platform, well supported. And now, go ahead and flame me for not doing work for free for other peoples profit but I'm pretty sure there won't be much guys wanting to do that. Volker From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 19:07:13 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F38C6106568F for ; Mon, 6 Oct 2008 19:07:12 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id C0B3A8FC13 for ; Mon, 6 Oct 2008 19:07:12 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 04A645B4C; Mon, 6 Oct 2008 11:49:33 -0700 (PDT) To: "Vlad GALU" In-reply-to: Your message of "Mon, 06 Oct 2008 18:09:06 +0300." References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> Comments: In-reply-to "Vlad GALU" message dated "Mon, 06 Oct 2008 18:09:06 +0300." Date: Mon, 06 Oct 2008 11:49:33 -0700 From: Bakul Shah Message-Id: <20081006184934.04A645B4C@mail.bitblocks.com> Cc: Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 19:07:13 -0000 On Mon, 06 Oct 2008 18:09:06 +0300 "Vlad GALU" wrote: > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > > Bob Bishop wrote: > > > >>> Does anybody have free time and skills to give a hand? Please see: > >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > >> > >> Should be possible to do this with a geom(4) class? > >> > > > > I am not saying it is impossible. They just need somebody to put them to > > right track I guess. I personally cant do that. It would be nice if somebod > y > > who has knowledge in this area contacts r1soft. At the very least r1soft > > seems to be willing to communicate on this issue. > > > > Continuous backups as well as bare-metal-restore seem to be a key feature > > for many hosters. FreeBSD is loosing users because of this issue. > > gmirror+ggate come to mind as a nifty solution ... My guess is these guys do something simpler like keep keep track of changed blocks since the last backup and periodically dump those blocks to a server. This is good enough for backups (but not mirroring) and it has low memory overhead (1 or 2 bits per block), lower network overhead than remote mirroring (you send a block at most once every sync interval), and a tiny loss of performance (over no backups). May be someone ought to do a garchive device! From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 19:28:37 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA160106569C for ; Mon, 6 Oct 2008 19:28:37 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 521AC8FC25 for ; Mon, 6 Oct 2008 19:28:35 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 78107 invoked by uid 399); 6 Oct 2008 19:28:48 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 6 Oct 2008 19:28:48 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48EA66DB.8020903@ispro.net> Date: Mon, 06 Oct 2008 22:28:27 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Vlad GALU References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 19:34:17 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 19:28:37 -0000 Vlad GALU wrote: > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: >> Bob Bishop wrote: >> >>>> Does anybody have free time and skills to give a hand? Please see: >>>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >>> Should be possible to do this with a geom(4) class? >>> >> I am not saying it is impossible. They just need somebody to put them to >> right track I guess. I personally cant do that. It would be nice if somebody >> who has knowledge in this area contacts r1soft. At the very least r1soft >> seems to be willing to communicate on this issue. >> >> Continuous backups as well as bare-metal-restore seem to be a key feature >> for many hosters. FreeBSD is loosing users because of this issue. > > gmirror+ggate come to mind as a nifty solution ... That allows mirroring however not a very practical solution. You would need to mirror every drive in another machine and you cant restore the drives into a earlier time. I am talking about a real backup solution where the backup agent collecting the information about written sectors and sending to backup server. You can then restore a file which existed in the box 1 hour ago if you need to. For example think about a situation where a server is processing important data which shouldnt get lost. A software failure wipes out the hard drive. You would loose all the data in the mirror as well. Also the setup of such mirroring system would be rather complicated. In addition to that, the mirroring does not support for example restore of mysql databases in table level. Think about a customer who wiped out his database accidentally. All the data would be gone in the mirror as well. With near continuous backup you can restore the data to just moments before the deletion process. Traditional backup systems at best daily backups, even if you could restore the data, the data could be up to 1 day old. More on that subject, r1soft supports multiple hosting control panel softwares. For example H-Sphere ( http://www.parallels.com/hsphere/ ) etc. through plugins which allow hosting customers to restore their own data easily. Something impossible with gmirror+ggate combination (since it does not actually backup the data and only mirror it) and not even practical if it was possible, if you have thousands of users. Actually, I am not saying that anybody should be doing about this and neither am I an r1soft advocate. I am just pointing out that there is a company out there which can provide a valuable software tool and they need somebody to put them into right direction only. If you know some FreeBSD developers who know the disk subsystem and can help R1Soft then you can perhaps forward this information to them. This not a feature or help request and I just am mentioning this. However you can imagine that a company which is giving online services on a serious scale probably would be interested in a CDP solution. It is currently impossible with FreeBSD. This makes using FreeBSD servers on critical applications a little bit insecure data protectionwise thus often people prefer Linux which is supported by such continuous data protection type backup solutions. (not that I am advocating FreeBSD or Linux here but this is the situation). Thanks, Evrne From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 19:38:39 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE2D9106568D for ; Mon, 6 Oct 2008 19:38:39 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 9ECC78FC2F for ; Mon, 6 Oct 2008 19:38:38 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 85327 invoked by uid 399); 6 Oct 2008 19:38:52 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 6 Oct 2008 19:38:52 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48EA6939.6090405@ispro.net> Date: Mon, 06 Oct 2008 22:38:33 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> In-Reply-To: <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 19:52:17 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 19:38:40 -0000 Zaphod Beeblebrox wrote: > On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen > wrote: > > [regarding r1soft.com , ...] > > > I am not saying it is impossible. They just need somebody to put > them to right track I guess. I personally cant do that. It would be > nice if somebody who has knowledge in this area contacts r1soft. At > the very least r1soft seems to be willing to communicate on this issue. > > Continuous backups as well as bare-metal-restore seem to be a key > feature for many hosters. FreeBSD is loosing users because of this > issue. > > > Actually, having looked at the site, the hammer filesystem and it's > replication strategy seem to be the most applicable technology (but then > you wouldn't even need these guys --- you'd be doing it yourself). Like > anything, though, live applications will require special treatment. > Keeping a live filesystem replicated does in no way guarentee that your > database (for instance) will be sane at any particular moment. It > sounds like these guys have made allowances for MySQL (they specifically > mention it), but this won't help the PostgreSQL users, etc. I think you didnt get the point here. Replication or mirroring != backup. You cant return back to how things were 1 hour ago. Also they support postgresql as well (while its usage is way smaller than mysql) http://www.r1soft.com/CDP_db_postgreSQL.html In any case, the product guarantees that it can return your databases to any point in the time. Do you see what you are missing? :) > I've spent a lot of time thinking about redundancy and I've come to one > inescapable conclusion: That the further up the stack you design for > redundancy, the cheaper and easier it becomes. Most databases have > replication strategies of one type or another that don't require exotic > hosting solutions to work. The idea/problem is not redundancy here, it is data protection. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 20:31:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7DA11065691; Mon, 6 Oct 2008 20:31:02 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 4D2A28FC1B; Mon, 6 Oct 2008 20:31:02 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=ptqOr/kCBmSm9rmEZPkF3lIHyepaz6K4C0hz7rqH+3DbqVnwHm6hntAxGpiCHjQneHhylzHEARgqkhASeQf+9535wsTmoRwjzd0P9yiO06dh0x5oE4b0lG/v6ELSqUQXptnsPg/WR6wChTKV9AD0m93bMaQkLR+e7a6nIdHhGRM=; Received: from phoenix.codelabs.ru (ppp83-237-105-41.pppoe.mtu-net.ru [83.237.105.41]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmwjM-0005p1-HJ; Tue, 07 Oct 2008 00:31:00 +0400 Date: Tue, 7 Oct 2008 00:30:58 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz>, Mel Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, bug-followup@freebsd.org, portmgr@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 20:31:02 -0000 Mon, Oct 06, 2008 at 02:30:29PM +0400, Eygene Ryabinkin wrote: > OK, fine. I will implement the usage of the local INDEX file in some > days. OK, I had implemented both '-o' option to pkg_audit and the usage of the local INDEX file. I had reworked pkg_audit and portaudit a bit further, mostly fixing some issues (both mine and existing). Here we go. Patches for pkg_install that adds pkg_audit: http://codelabs.ru/fbsd/patches/portaudit/0001-Add-functions-for-traversing-package-database-and-ma.patch http://codelabs.ru/fbsd/patches/portaudit/0002-Add-function-match_get_pkgorigin.patch http://codelabs.ru/fbsd/patches/portaudit/0003-New-utility-pkg_audit.patch http://codelabs.ru/fbsd/patches/portaudit/0004-pkg_audit-add-option-to-print-origins.patch Mega-patch for pkg_install: http://codelabs.ru/fbsd/patches/portaudit/pkg_install-megapatch-pkg_audit.diff Patches for portaudit: http://codelabs.ru/fbsd/patches/portaudit/0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch http://codelabs.ru/fbsd/patches/portaudit/0002-Separate-vulnerable-ports-search-from-the-formatter.patch http://codelabs.ru/fbsd/patches/portaudit/0003-Use-pkg_audit-utility-if-it-is-available.patch http://codelabs.ru/fbsd/patches/portaudit/0004-Implement-checking-for-a-new-package-versions.patch Mega-patch for portaudit: http://codelabs.ru/fbsd/patches/portaudit/portaudit-megapatch_pkg_audit-and-checknew.diff Opinions are welcome! -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 20:36:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE40B106568B; Mon, 6 Oct 2008 20:36:18 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 91C188FC0C; Mon, 6 Oct 2008 20:36:18 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:Content-Transfer-Encoding:In-Reply-To:Sender; b=B2DcjFIBRhC7lVHTG74BQjt29TU+BfboTSpAvQbKYE4Fwu5qnFGoyAy4NOmQSp9sLmfGV6rq/lSxGjIzvMkJb+eKnbfIgx42wbOfbDl7e4fYd8VbtFKVQal0/9iSjsi4suTvrzqOrHdtYK5ef9X4v6wVrX2q7VWEY4d49azIAVU=; Received: from phoenix.codelabs.ru (ppp83-237-105-41.pppoe.mtu-net.ru [83.237.105.41]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KmwoT-0006BZ-5j; Tue, 07 Oct 2008 00:36:17 +0400 Date: Tue, 7 Oct 2008 00:36:14 +0400 From: Eygene Ryabinkin To: Miroslav Lachman <000.fbsd@quip.cz>, Mel Message-ID: References: <48DE5CC0.9000708@localhost.inse.ru> <48DF6735.4030906@quip.cz> <4bESZpNwE3z/DdlE2fwK/BXzQSo@2MQ0uKCiT7mdMUuLeUzs8Nv3ToQ> <48E94281.8010300@quip.cz> <48E9D382.4000001@quip.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: Sender: rea-fbsd@codelabs.ru Cc: freebsd-hackers@freebsd.org, bug-followup@freebsd.org, freebsd-ports@freebsd.org, portmgr@freebsd.org Subject: Re: ports/126853: ports-mgmt/portaudit: speed up audit of installed packages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 20:36:19 -0000 Forgot to say: Tue, Oct 07, 2008 at 12:30:58AM +0400, Eygene Ryabinkin wrote: > OK, I had implemented both '-o' option to pkg_audit and the usage of the > local INDEX file. The latter can be activated by writing something like ----- portaudit_pkg_index=3D"file:///usr/ports/INDEX-%d" ----- to the /usr/local/etc/portaudit.conf. --=20 Eygene From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 21:43:56 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7629D1065688 for ; Mon, 6 Oct 2008 21:43:56 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 5D7ED8FC16 for ; Mon, 6 Oct 2008 21:43:56 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 4EF6833C62; Mon, 6 Oct 2008 14:24:53 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id F0FB033C5B; Mon, 6 Oct 2008 14:24:52 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18666.33296.607120.889620@almost.alerce.com> Date: Mon, 6 Oct 2008 14:24:32 -0700 To: Bakul Shah In-Reply-To: <20081006184934.04A645B4C@mail.bitblocks.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP X-Mailman-Approved-At: Mon, 06 Oct 2008 22:06:32 +0000 Cc: hackers@freebsd.org, Evren Yurtesen , Vlad GALU Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 21:43:56 -0000 Bakul Shah writes: > On Mon, 06 Oct 2008 18:09:06 +0300 "Vlad GALU" wrote: > > On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen wrote: > > > Bob Bishop wrote: > > > > > >>> Does anybody have free time and skills to give a hand? Please see: > > >>> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > >> > > >> Should be possible to do this with a geom(4) class? > > >> > > > > > > I am not saying it is impossible. They just need somebody to put them to > > > right track I guess. I personally cant do that. It would be nice if somebod > > y > > > who has knowledge in this area contacts r1soft. At the very least r1soft > > > seems to be willing to communicate on this issue. > > > > > > Continuous backups as well as bare-metal-restore seem to be a key feature > > > for many hosters. FreeBSD is loosing users because of this issue. > > > > gmirror+ggate come to mind as a nifty solution ... > > My guess is these guys do something simpler like keep keep > track of changed blocks since the last backup and > periodically dump those blocks to a server. This is good > enough for backups (but not mirroring) and it has low memory > overhead (1 or 2 bits per block), lower network overhead than > remote mirroring (you send a block at most once every sync > interval), and a tiny loss of performance (over no backups). > May be someone ought to do a garchive device! There were a couple of threads about using kqueue or other FreeBSD tools to build something like Mac OS X's Time Machine. R1soft's software sounds very similar. The conclusion seemed to be that it'd be doable. Here's a pointer to the start of one of the threads. http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html g. From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 21:56:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5BA31065686 for ; Mon, 6 Oct 2008 21:56:59 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 6C9448FC33 for ; Mon, 6 Oct 2008 21:56:58 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 57443 invoked by uid 399); 6 Oct 2008 21:30:34 -0000 Received: from a91-152-230-218.elisa-laajakaista.fi (HELO ?192.168.1.34?) (yurtesen@ispro.net@91.152.230.218) by mail.ispro.net with ESMTPAM; 6 Oct 2008 21:30:34 -0000 X-Originating-IP: 91.152.230.218 Message-ID: <48EA83CE.4060702@ispro.net> Date: Tue, 07 Oct 2008 00:31:58 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Sean Bruno , freebsd-hackers@freebsd.org References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> In-Reply-To: <48EA3451.7040801@miralink.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 22:06:44 +0000 Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 21:56:59 -0000 Sean Bruno wrote: > Evren Yurtesen wrote: >> Hello, >> >> As far as I can see, there is no continuous backup solution for >> FreeBSD at the moment. I talked with R1Soft and they seem to not be >> able to support FreeBSD and need help. >> >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 >> >> I know people who want to switch to using FreeBSD but cant do that >> because their backup solution (R1Soft) does not support FreeBSD :( >> >> Thanks, >> Evren >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org" > Hmmm....Well, my company (MiraLink) makes a BSD based appliance that > mirrors block level changes between two units. > > It's commercial, not BSD licensed, but it's something. > > www.miralink.com > > sean > > Hello Sean, Thanks for the link. The miralink products seem to be doing data replication similar to gmirror+ggate? This wont protect against accidental deletion of data etc. The near continuous backup solutions do not mirror/replicate the data. Think about it as continuously taking backup. You can return back to 10 minutes or 1 hour before and restore old data. In either case, I wasnt trying to start a debate between backup solutions etc. here. I simply wanted to ask for help if somebody who has in depth knowledge about disk subsystems of FreeBSD and can give some tips to r1soft so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 22:01:37 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C47881065688 for ; Mon, 6 Oct 2008 22:01:37 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 416298FC21 for ; Mon, 6 Oct 2008 22:01:36 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 74206 invoked by uid 399); 6 Oct 2008 22:01:53 -0000 Received: from a91-152-230-218.elisa-laajakaista.fi (HELO ?192.168.1.34?) (yurtesen@ispro.net@91.152.230.218) by mail.ispro.net with ESMTPAM; 6 Oct 2008 22:01:53 -0000 X-Originating-IP: 91.152.230.218 Message-ID: <48EA8B3A.3090609@ispro.net> Date: Tue, 07 Oct 2008 01:03:38 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Volker References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> In-Reply-To: <48EA56BB.6040702@vwsoft.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 22:07:00 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 22:01:38 -0000 Volker wrote: > On 12/23/-58 20:59, Evren Yurtesen wrote: >> Hello, >> >> As far as I can see, there is no continuous backup solution for FreeBSD >> at the moment. I talked with R1Soft and they seem to not be able to >> support FreeBSD and need help. >> >> Does anybody have free time and skills to give a hand? Please see: >> http://forum.r1soft.com/showpost.php?p=3414&postcount=9 > > Quoting the thread: >> and they are usually very busy... and they like to work for hire. > > Wrong term: Even open source folks need something for living. That's too > cheap and I don't suggest supporting a company like this, as they're > expecting somebody else to do their job and make profit of it. > > They're free to use a system like FreeBSD for their products, they're > free to modify and redistribute it but they need to do their job and > don't expect anybody else to code for them for free. I agree, perhaps whoever can help than can ask for money for the job done and I am sure they would pay reasonably since this is a commercial company. But as far as I can see people here do not even know the difference between near continous backup and mirroring. I just wanted to inform that there is such solutions nowadays available but FreeBSD users are not able to take advantage of them and the company who is making the product is interested in supporting FreeBSD but perhaps somebody who has experience can give some hints to them. They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=4224&postcount=3 > Next quote of thread: >> real world competency writing block device drivers for FreeBSD > > Let me check that out... > %ls -l /dev | grep ^b > % > > Well, I guess, nobody is able to do so and it's noted in the handbook or > in GNN's "The Design and Implementation of the FreeBSD operating > system". A good read. You can go ahead and explain it to r1soft or any other software company which makes near continuous backup solutions. So maybe they can find this information and improve their product. So far at least this company thinks that this is impossible to do with FreeBSD as far as I can tell. >> I know people who want to switch to using FreeBSD but cant do that >> because their backup solution (R1Soft) does not support FreeBSD :( > > What's wrong with solutions like Bacula? It's working, cross platform, > well supported. Do you know what is near continuous backup solution or ever heard of such technologies? I dont see how you can compare this with Bacula. They are not the same thing really. I can tell you one thing which is wrong with Bacula, it scans the whole filesystem everytime it takes a backup even incremental backups (while this disk io loading operation is unnecessary with CDP), and Bacula cant restore your data to what there was 10 minutes ago at any time of the day. > And now, go ahead and flame me for not doing work for free for other > peoples profit but I'm pretty sure there won't be much guys wanting to > do that. There is nothing to flame or blame really. I can only blame my own stupidity. I posted this information here in hopes that somebody capable can give some hints to r1soft or to another near continuous backup solution so we could all benefit. But what do I get? All I get is information about solutions which mirror / replicate the data or suggestions of standard backup programs. I am not even surprised that FreeBSD is not supported. FreeBSD users do not even know the difference between a CDP solution or mirroring / replication or traditional backup... Go figure... If you dont know what is CDP then please read and learn (there are links to wikipedia articles in this address also) http://www.r1soft.com/CDP.html I never told that anybody should make the software ready for selling by another company for free, neither the post in the thread is asking for somebody to write a driver for free. The company obviously understands that such job requires an expert who works for money. They probably wouldnt mind if somebody wrote it for free to them (who wouldnt) but it doesnt say that they wouldnt pay. If you can do the job, please contact the company and give your price. If they tell that they want it done for free, then come and complain that they want it done for free. I am sorry if I was a little bit out of line here but a simple question became an unnecessary debate really. It was more or less a yes or no question :) Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 22:08:30 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 273FD106568D for ; Mon, 6 Oct 2008 22:08:30 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.30]) by mx1.freebsd.org (Postfix) with ESMTP id D07C18FC1E for ; Mon, 6 Oct 2008 22:08:29 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so502481ywe.13 for ; Mon, 06 Oct 2008 15:08:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=LbIo4q7m6wyF1zXlgslSrwuYZVC5/NezM29ugWGq63s=; b=adJvjaRG2hMEBMeieQv19j3I/QyumzGGfLWlckoxIqyI8PYdiU78LRk9zdghMjZ8A1 r3P2HgbMum2/zG9T9LDo8Llyq1iRCT+2++vshsa3hzoW7E4DLFdWsvEpU4mwC82oKFDg IeoSp2K+ncGSNVtNHq77Wj/L3DZZuH0yc9QKA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=ptc1C+fr5zkOQGaApPkSLp3Un4y6ADqyBEkISbaBwuOZckLdmCh5OTtph2eQtT9tCQ GWL10jRgJP/sLKR6i2Kho2yTU/tJRSLVPKibhwINp+/7DQAuO38hlEUdCSXpMusM4yVg 6f1DWyLZNddxPbq/qE2E7uGJwV8yLW1BIfoR0= Received: by 10.151.102.16 with SMTP id e16mr8395801ybm.168.1223330908944; Mon, 06 Oct 2008 15:08:28 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Mon, 6 Oct 2008 15:08:28 -0700 (PDT) Message-ID: <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> Date: Mon, 6 Oct 2008 18:08:28 -0400 From: "Zaphod Beeblebrox" To: "Evren Yurtesen" In-Reply-To: <48EA6939.6090405@ispro.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 22:08:30 -0000 On Mon, Oct 6, 2008 at 3:38 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen > yurtesen@ispro.net>> wrote: >> >> [regarding r1soft.com , ...] >> >> I am not saying it is impossible. They just need somebody to put >> them to right track I guess. I personally cant do that. It would be >> nice if somebody who has knowledge in this area contacts r1soft. At >> the very least r1soft seems to be willing to communicate on this issue. >> >> Continuous backups as well as bare-metal-restore seem to be a key >> feature for many hosters. FreeBSD is loosing users because of this >> issue. >> >> >> Actually, having looked at the site, the hammer filesystem and it's >> replication strategy seem to be the most applicable technology (but then you >> wouldn't even need these guys --- you'd be doing it yourself). Like >> anything, though, live applications will require special treatment. Keeping >> a live filesystem replicated does in no way guarentee that your database >> (for instance) will be sane at any particular moment. It sounds like these >> guys have made allowances for MySQL (they specifically mention it), but this >> won't help the PostgreSQL users, etc. >> > > I think you didnt get the point here. Replication or mirroring != backup. > You cant return back to how things were 1 hour ago. Actually, right back at you. You didn't fathom the meaning in my statement. While your post was vague, I read the company's website to determine the featureset they were claiming (although I missed their postgres support --- I only read the "features" page). NetBSD's hammer filesystem achieves replication at the filesystem layer (which will do infinitely better at this problem than a block-only driver) by maintaining a history of what's happened and being able to "select" (as in the database term) changes to the filesystem that have occurred since the last batch of blocks were shiped out to replication. This gives you both fine grained recovery (basically changes to files are kept until your "rules" define they should be freed) and replication and fine grained recovery on the other side of replication. In fact, Hammer delivers what they claim in a far more sophisticated way. (but it's NetBSD, not FreeBSD, unless someone decides to port it) > Also they support postgresql as well (while its usage is way smaller than > mysql) > http://www.r1soft.com/CDP_db_postgreSQL.html > > In any case, the product guarantees that it can return your databases to > any point in the time. Do you see what you are missing? :) > Well... "I" am not missing it. I have that without making my filesystem jump through an enormous hoop. But designing "my" application correctly, I have that feature for far less effort. (that was the other point of my post) > I've spent a lot of time thinking about redundancy and I've come to one >> inescapable conclusion: That the further up the stack you design for >> redundancy, the cheaper and easier it becomes. Most databases have >> replication strategies of one type or another that don't require exotic >> hosting solutions to work. >> > > The idea/problem is not redundancy here, it is data protection. Well... no... you don't need fine grained filesystem history for data integrity (unless you let loose a bunch of summer students armed with the ability to RM or your application is faulty (in which case, your filesystem won't protect you). As I said, This can all be achieved with other far simpler solutions. However, if your dev team isn't smart enough (or don't care for some reason), then you can take advantage of their product and pay their price. From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 22:28:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDC9E1065691 for ; Mon, 6 Oct 2008 22:28:32 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 5523E8FC16 for ; Mon, 6 Oct 2008 22:28:32 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 68670 invoked by uid 1001); 6 Oct 2008 18:26:00 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Mon, 06 Oct 2008 18:25:59 -0400 Date: Mon, 6 Oct 2008 18:25:58 -0400 To: freebsd-hackers@freebsd.org Message-ID: <20081006182558.708e4094@bhuda.mired.org> In-Reply-To: <18666.33296.607120.889620@almost.alerce.com> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> <18666.33296.607120.889620@almost.alerce.com> Organization: Meyer Consulting X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; amd64-portbld-freebsd7.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: hartzell@alerce.com Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 22:28:32 -0000 On Mon, 6 Oct 2008 14:24:32 -0700 George Hartzell wrote: > There were a couple of threads about using kqueue or other FreeBSD > tools to build something like Mac OS X's Time Machine. R1soft's > software sounds very similar. Time machine doesn't do continuous backups, it does them once an hour or so. People have built similar systems on top of rsync; I did it on top of zfs (turned out to be to fragile, though). You then just need a spiffy GUI for wondering through the backups. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 23:09:41 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC599106569E for ; Mon, 6 Oct 2008 23:09:41 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.177]) by mx1.freebsd.org (Postfix) with ESMTP id 97BB48FC08 for ; Mon, 6 Oct 2008 23:09:41 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: by el-out-1112.google.com with SMTP id v27so600429ele.13 for ; Mon, 06 Oct 2008 16:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=ofYZAwKoVDO9qnANlUQoZ9Qz/EXFaA7HSLjt45NOL0w=; b=Y6e4YcCrPekowQmSSmZKDqTWlwch2kwpW1mRCEEaBXdlD8sNxzGGBbjw8MlRvEUNmB OdVndOp23NnX/pGgLYesRUzDV3R4WwSDxqNYy/CGduwHubiWhqIAGcZ04tUt3gcBIC1O X8ZcXzM4qlAb7udPxajvHGKunR7pxWNa9C1WQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=ivpLbIlzbnuX+Qw24OWbNLlorXsMlIV+RwJgKI+wtiwhT//ME5ta/CJGpdIOMIuX9x rpCGFoyo96937XY7NEftn8BYrwasAV73YmvYX4EvRUf7MjGDXYfov+GdfTFu0gpyd2Z7 o4NlsJTQp7NtRwSWUrXvhvgSLiDRrSI2VEe3g= Received: by 10.90.80.19 with SMTP id d19mr6423620agb.65.1223333040142; Mon, 06 Oct 2008 15:44:00 -0700 (PDT) Received: by 10.90.94.14 with HTTP; Mon, 6 Oct 2008 15:44:00 -0700 (PDT) Message-ID: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> Date: Mon, 6 Oct 2008 15:44:00 -0700 From: "alan yang" To: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: kgdb debugging X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 23:09:41 -0000 hi, there, wonder people can shed some lights on remote debugging. i have freebsd7 configured with option DDB / KDB / GDB but after entering the db on the target system the command gdb gives "the remote GDB backend could not be selected". i browsed through the mailing list, and do find 1 similar post but without answer. thanks in advance & apology if i overlooked things ... cheers, alan From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 22:40:33 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81DD51065696 for ; Mon, 6 Oct 2008 22:40:33 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id D11388FC0A for ; Mon, 6 Oct 2008 22:40:31 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 95635 invoked by uid 399); 6 Oct 2008 22:40:49 -0000 Received: from a91-152-230-218.elisa-laajakaista.fi (HELO ?192.168.1.34?) (yurtesen@ispro.net@91.152.230.218) by mail.ispro.net with ESMTPAM; 6 Oct 2008 22:40:49 -0000 X-Originating-IP: 91.152.230.218 Message-ID: <48EA9459.2000807@ispro.net> Date: Tue, 07 Oct 2008 01:42:33 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> In-Reply-To: <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 06 Oct 2008 23:15:29 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 22:40:33 -0000 Zaphod Beeblebrox wrote: > Actually, right back at you. You didn't fathom the meaning in my > statement. While your post was vague, I read the company's website to I am sorry what was vague since I wrote 'continuous backup' in my post? The whole idea is such a basic idea that if you put this to google you can get wikipedia entry about it in first 3 results (I see in 2nd). Maybe you didnt know anything about it makes it vague to you. The message I sent was quite clear and plain. > determine the featureset they were claiming (although I missed their > postgres support --- I only read the "features" page). NetBSD's hammer > filesystem achieves replication at the filesystem layer (which will do > infinitely better at this problem than a block-only driver) by > maintaining a history of what's happened and being able to "select" (as > in the database term) changes to the filesystem that have occurred since > the last batch of blocks were shiped out to replication. This gives you > both fine grained recovery (basically changes to files are kept until > your "rules" define they should be freed) and replication and fine > grained recovery on the other side of replication. In fact, Hammer > delivers what they claim in a far more sophisticated way. You might want to read this page too: http://www.r1soft.com/CDP.html > (but it's NetBSD, not FreeBSD, unless someone decides to port it) While Hammer might be doing a similar job, it is a filesystem not a backup application and it wont replace backups. It doesnt just store the data in a backup server. While eventually it might become a backup solution, it will take years before that can happen. Even then, people will not just switch their current filesystems overnight. The CDP backup softwares are available today, it just needs a sort of driver to function in current system. > Also they support postgresql as well (while its usage is way smaller > than mysql) > http://www.r1soft.com/CDP_db_postgreSQL.html > > In any case, the product guarantees that it can return your > databases to any point in the time. Do you see what you are missing? :) > > > Well... "I" am not missing it. I have that without making my filesystem > jump through an enormous hoop. But designing "my" application > correctly, I have that feature for far less effort. (that was the other > point of my post) Can you just explain how can one currently do that in FreeBSD? Is it as easy as in Linux with CDP backup softwares? such as installing a program and done? > > > I've spent a lot of time thinking about redundancy and I've come > to one inescapable conclusion: That the further up the stack you > design for redundancy, the cheaper and easier it becomes. Most > databases have replication strategies of one type or another > that don't require exotic hosting solutions to work. > > > The idea/problem is not redundancy here, it is data protection. > > > Well... no... you don't need fine grained filesystem history for data > integrity (unless you let loose a bunch of summer students armed with > the ability to RM or your application is faulty (in which case, your > filesystem won't protect you). As I said, This can all be achieved with > other far simpler solutions. However, if your dev team isn't smart > enough (or don't care for some reason), then you can take advantage of > their product and pay their price. There is no perfect system. This is exactly why people take backups. If what you said was applicable then there wouldnt be any need for backup software. People would just make sure that they dont loose their data. In addition to this, I have no control of if my customer will delete his/her data accidentally. I cant make a system which does not allow customers to delete data. I have given an example of a simple solution that Linux users can utilize (which obviously we also can utilize if we put our heads into it and give some directions as r1soft is willing to support FreeBSD). While you are first saying that these can be achieved with far simpler solutions, at the same time you are saying that the dev team must be smart enough. My solution is simple enough to write here. You install Linux on all machines and then CDP backup server on the backup server and CDP agent on the client machines, is yours simpler? Then explain how we can create similar sort of data protection? Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 23:26:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C262106568D; Mon, 6 Oct 2008 23:26:52 +0000 (UTC) (envelope-from gahr@FreeBSD.org) Received: from cpanel03.rubas-s03.net (cpanel03.rubas-s03.net [195.182.222.73]) by mx1.freebsd.org (Postfix) with ESMTP id 193CF8FC08; Mon, 6 Oct 2008 23:26:52 +0000 (UTC) (envelope-from gahr@FreeBSD.org) Received: from [213.142.183.219] (helo=gahrtop.gahr.ch) by cpanel03.rubas-s03.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1KmzTW-0003Y5-Ce; Tue, 07 Oct 2008 01:26:50 +0200 Message-ID: <48EA9E95.80105@FreeBSD.org> Date: Tue, 07 Oct 2008 01:26:13 +0200 From: Pietro Cerutti Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.17 (X11/20081002) MIME-Version: 1.0 To: alan yang References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> In-Reply-To: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> X-Enigmail-Version: 0.95.6 OpenPGP: id=9571F78E; url=http://gahr.ch/pgp/ Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel03.rubas-s03.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - FreeBSD.org X-Source: X-Source-Args: X-Source-Dir: Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: kgdb debugging X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 23:26:52 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 alan yang wrote: | hi, there, | | wonder people can shed some lights on remote debugging. i have | freebsd7 configured with option DDB / KDB / GDB but after entering the | db on the target system the command gdb gives "the remote GDB backend | could not be selected". | | i browsed through the mailing list, and do find 1 similar post but | without answer. | | thanks in advance & apology if i overlooked things ... I suggest the following tutorial: http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf Have fun :) | | cheers, | alan | _______________________________________________ | freebsd-hackers@freebsd.org mailing list | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" - -- Pietro Cerutti gahr@FreeBSD.org PGP Public Key: http://gahr.ch/pgp -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 =6Pdm -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 00:07:26 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7979106569E for ; Tue, 7 Oct 2008 00:07:26 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.235]) by mx1.freebsd.org (Postfix) with ESMTP id 9AE248FC1C for ; Tue, 7 Oct 2008 00:07:25 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by wr-out-0506.google.com with SMTP id c8so597456wra.27 for ; Mon, 06 Oct 2008 17:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=LTAQNKUn73lN8fRMErmxOUKSa0F9edDhDMqn+yuiop0=; b=ANDeLgGCNPX7MGrLex+aZf4pRgrnsSvzMpiGq5btBLAmVG4XMUV9z/rkYB5UK+6frX ENWOK3cK+vkXfMhWl48sA4jWr6ymGS5f9i3V8JovRe1kWjAFNC1/nnxLnvTDpcNcPfkU wPlRV1SFxlReOVfF0UjgaQAspmQt8k1pixGoQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=aBVVrB5ht5zaheQZBQLSQuGPvG31aGO4dD5ROH2xScJalpb0nCAOEdBusX4vf7qs6C bR8+4hW3zmszADPbfzdvhYQVLmFC6sZrvwZzEns/vmvz7gRo2vUPVzrM8Rd50mMrJpq/ Rc8lPOYWnlrGz7za0aCp6uoJTUmc/Osg48Dhg= Received: by 10.151.141.8 with SMTP id t8mr8556476ybn.247.1223338045120; Mon, 06 Oct 2008 17:07:25 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Mon, 6 Oct 2008 17:07:25 -0700 (PDT) Message-ID: <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> Date: Mon, 6 Oct 2008 20:07:25 -0400 From: "Zaphod Beeblebrox" To: "Evren Yurtesen" In-Reply-To: <48EA9459.2000807@ispro.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 00:07:27 -0000 On Mon, Oct 6, 2008 at 6:42 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> Actually, right back at you. You didn't fathom the meaning in my >> statement. While your post was vague, I read the company's website to >> > > While Hammer might be doing a similar job, it is a filesystem not a backup > application and it wont replace backups. It doesnt just store the data in a > backup server. While eventually it might become a backup solution, it will > take years before that can happen. Even then, people will not just switch > their current filesystems overnight. The CDP backup softwares are available > today, it just needs a sort of driver to function in current system. > >From my reading, Hammer is much more than a filesystem, but then you probably havn't read about it yet. By my reading, Hammer hits all their feature points and does it better _because_ it's a filesystem. Well... "I" am not missing it. I have that without making my filesystem > jump through an enormous hoop. But designing "my" application correctly, I > have that feature for far less effort. (that was the other point of my > post) > Can you just explain how can one currently do that in FreeBSD? Is it as easy > as in Linux with CDP backup softwares? such as installing a program and > done? > It's relatively simple. Database replication solves the data backup problem (I don't store application data outside the database). Database replication for both MySQL and PostgreSQL is relatively straight forward. As for the configuration of code and servers --- that is taken care of with configuration management (it's really a bigger issue than just backing up a filesystem) and installing a new server to take a place in the cluster is a straight forward checkout from the CM system. For things I really care about staying up, add VRRP and an application design that is fault tolerant. This actually works rather well if you do your research. Database replication is possible at all kinds of different link speeds and distances. Database replication also allows you to control your data better --- you know more about your data than a block replicator would. It means that your backup is already live and it means that, with the right scripts, invoking a backup on primary failure is simple. Database replication on some databases even allows you to preserve transactions --- which is important in some cases. > There is no perfect system. This is exactly why people take backups. If > what you said was applicable then there wouldnt be any need for backup > software. People would just make sure that they dont loose their data. In > addition to this, I have no control of if my customer will delete his/her > data accidentally. I cant make a system which does not allow customers to > delete data. > > I have given an example of a simple solution that Linux users can utilize > (which obviously we also can utilize if we put our heads into it and give > some directions as r1soft is willing to support FreeBSD). While you are > first saying that these can be achieved with far simpler solutions, at the > same time you are saying that the dev team must be smart enough. > > My solution is simple enough to write here. You install Linux on all > machines and then CDP backup server on the backup server and CDP agent on > the client machines, is yours simpler? Then explain how we can create > similar sort of data protection? Well... no. Backup software and strategies are just one available tool for risk mitigation. To know what tools you require, you must define your risks. Then with your list of risks you look at the cost of each tool and find the toolset that suits you. By the responses in this thread, it seems like the set of FreeBSD developers and the set of people who desire this solution are disjoint. Actually... as some obligatory positive content, the time travel features of Hammer should be straightforward to implement in ZFS... are ZFS modules supported on FreeBSD yet? It would seem to be a logical module. From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 6 23:57:16 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 550901065687 for ; Mon, 6 Oct 2008 23:57:16 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 3CE9D8FC15 for ; Mon, 6 Oct 2008 23:57:16 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 9C70633C62; Mon, 6 Oct 2008 16:27:53 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 0710A33C5B; Mon, 6 Oct 2008 16:27:52 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18666.40675.636312.893786@almost.alerce.com> Date: Mon, 6 Oct 2008 16:27:31 -0700 To: Mike Meyer In-Reply-To: <20081006182558.708e4094@bhuda.mired.org> References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <20081006184934.04A645B4C@mail.bitblocks.com> <18666.33296.607120.889620@almost.alerce.com> <20081006182558.708e4094@bhuda.mired.org> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP X-Mailman-Approved-At: Tue, 07 Oct 2008 02:17:19 +0000 Cc: freebsd-hackers@freebsd.org, hartzell@alerce.com Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 23:57:16 -0000 Mike Meyer writes: > On Mon, 6 Oct 2008 14:24:32 -0700 > George Hartzell wrote: > > There were a couple of threads about using kqueue or other FreeBSD > > tools to build something like Mac OS X's Time Machine. R1soft's > > software sounds very similar. > > Time machine doesn't do continuous backups, it does them once an hour > or so. People have built similar systems on top of rsync; I did it on > top of zfs (turned out to be to fragile, though). You then just need a > spiffy GUI for wondering through the backups. On the other hand Time Machine does take advantage of a kernel based mechanism that watches file activity and does its best to take advantage of that information to avoid scanning the filesystem when it does a backup. That's the context of the message thread that I pointed to (again, for completeness) http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html The thread seemed relevant given the context of backup systems that watch filesystem io. g. From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 10:12:50 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3A711065686 for ; Tue, 7 Oct 2008 10:12:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 936D78FC15 for ; Tue, 7 Oct 2008 10:12:50 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 50F742049; Tue, 7 Oct 2008 11:56:09 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 3CB3684491; Tue, 7 Oct 2008 11:56:09 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Evren Yurtesen References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> Date: Tue, 07 Oct 2008 11:56:09 +0200 In-Reply-To: <48EA8B3A.3090609@ispro.net> (Evren Yurtesen's message of "Tue, 07 Oct 2008 01:03:38 +0300") Message-ID: <861vysiv9i.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 10:12:51 -0000 Evren Yurtesen writes: > They actually do not think that it is an easy job to adapt their > software to support FreeBSD even. See this post: > http://forum.r1soft.com/showpost.php?p=3D4224&postcount=3D3 All this shows is that they don't know anything about FreeBSD at all (plus they need a refresher course in OS design; Linux is also a monolithic kernel) What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 08:11:21 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AED410656A6 for ; Tue, 7 Oct 2008 08:11:21 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id A90878FC25 for ; Tue, 7 Oct 2008 08:11:19 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 90211 invoked by uid 399); 7 Oct 2008 08:11:34 -0000 Received: from a91-152-230-218.elisa-laajakaista.fi (HELO ?192.168.1.34?) (yurtesen@ispro.net@91.152.230.218) by mail.ispro.net with ESMTPAM; 7 Oct 2008 08:11:34 -0000 X-Originating-IP: 91.152.230.218 Message-ID: <48EB1A1C.7020701@ispro.net> Date: Tue, 07 Oct 2008 11:13:16 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> In-Reply-To: <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 07 Oct 2008 11:33:07 +0000 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 08:11:21 -0000 I think here might be a misunderstanding. I was talking about a reliable backup solution whereas you guys are all the time talking about mirroring and replication type solutions. Since you cant be thinking that mirroring and replication can replace backup, there must be a misunderstanding? Zaphod Beeblebrox wrote: > > From my reading, Hammer is much more than a filesystem, but then you > probably havn't read about it yet. By my reading, Hammer hits all their > feature points and does it better _because_ it's a filesystem. I glanced through these actually: http://www.dragonflybsd.org/hammer/ I didnt see anywhere that it will replace backup programs? > It's relatively simple. Database replication solves the data backup > problem (I don't store application data outside the database). Database > replication for both MySQL and PostgreSQL is relatively straight > forward. As for the configuration of code and servers --- that is taken > care of with configuration management (it's really a bigger issue than > just backing up a filesystem) and installing a new server to take a > place in the cluster is a straight forward checkout from the CM system. > For things I really care about staying up, add VRRP and an application > design that is fault tolerant. Have you ever tried to restore data from MySQL replication logs? :) Even if you use binary logging, when you want to go back in time. You will need to first restore the whole database first from normal backups then replay the logs until the point that you wanna be. There is no simple way to go back in time. That is of course you have backups. If you dont have backups because you think replication is a backup solution, you would be screwed. Totally more complicated that clicking from the web to select data and time and table and restore! Also, you are thinking about a very small sized system. While replication might work if you are relatively small sized company (like 1-2 servers). If you have many independent servers with different databases inside you just cant use it. Even if you could replicate multiple boxes into one box, there would eventually be problems such as same named databases etc. and even then, you cant just easily restore the data if the user deletes all his data in his tables. Also that is not practical for users at all. For example I cant give an option to the user to restore his data by himself. While that is possible with most backup software easily. About VRRP etc. I already told that I am not talking about redundancy here. You are talking about totally different things. I need data protection. > This actually works rather well if you do your research. Database > replication is possible at all kinds of different link speeds and > distances. Database replication also allows you to control your data > better --- you know more about your data than a block replicator would. > It means that your backup is already live and it means that, with the > right scripts, invoking a backup on primary failure is simple. Database > replication on some databases even allows you to preserve transactions > --- which is important in some cases. And how do you propose that I restore a table in the database to of 1h before status? like you can do with a data backup solution? You are talking about a spare server solution not backup solution. Replication IS NOT backup. If you look at articles and information about database replication, almost all mention that it DOES NOT replace backups. > Well... no. Backup software and strategies are just one available tool > for risk mitigation. To know what tools you require, you must define > your risks. Then with your list of risks you look at the cost of each > tool and find the toolset that suits you. By the responses in this > thread, it seems like the set of FreeBSD developers and the set of > people who desire this solution are disjoint. Right, I just cant use the tool I require. There is no way to take near continuous backups of FreeBSD filesystems. > Actually... as some obligatory positive content, the time travel > features of Hammer should be straightforward to implement in ZFS... are > ZFS modules supported on FreeBSD yet? It would seem to be a logical module. Those features work within the filesystem, you there is no mention of mirroring to a remote hard drive, if you could mirror to a remote hard drive, you couldnt easily mirror 10 machines into 1 remote hard drive, even if you could do that, it would require total disk space of all those 10 drives to exist in that 1 backup server. Even if you could overcome that, this would mean that all FreeBSD users wanting to take advantage of such system to convert their filesystems. Even if it was easy enough, there is no GUI tools to allow users exist in those systems to restore files by themselves easily. As you can see, none of the solutions you are suggesting is anywhere near simple solutions which can replace near continous backup solutions. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 10:59:17 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C823910656B2 for ; Tue, 7 Oct 2008 10:59:17 +0000 (UTC) (envelope-from Ramachandran.Sathyanarayanan@aricent.com) Received: from jaguar.aricent.com (inoutbound.aricent.com [125.21.164.247]) by mx1.freebsd.org (Postfix) with ESMTP id 998338FC0C for ; Tue, 7 Oct 2008 10:59:16 +0000 (UTC) (envelope-from Ramachandran.Sathyanarayanan@aricent.com) Received: from jaguar.aricent.com (localhost [127.0.0.1]) by jaguar.aricent.com (8.13.8/8.13.8) with ESMTP id m97AYNQN018406 for ; Tue, 7 Oct 2008 16:04:23 +0530 Received: from GUREXHT02.ASIAN.AD.ARICENT.COM (gurexht02.asian.ad.aricent.com [10.203.171.138]) by jaguar.aricent.com (8.13.8/8.13.8) with ESMTP id m97AYNZ4018386 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=FAIL) for ; Tue, 7 Oct 2008 16:04:23 +0530 Received: from GUREXMB02.asian.ad.aricent.com ([10.203.171.132]) by GUREXHT02.ASIAN.AD.ARICENT.COM ([10.203.171.138]) with mapi; Tue, 7 Oct 2008 16:08:23 +0530 From: Ramachandran Sathyanarayanan To: "freebsd-hackers@freebsd.org" Date: Tue, 7 Oct 2008 16:08:22 +0530 Thread-Topic: sleep is not working to avoid starvation among threads of same priority Thread-Index: AckoZFjwx1hc0ptOQfe07GpuldRJxQABEJqA Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 07 Oct 2008 11:33:38 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same priority X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 10:59:17 -0000 Hi I hope this is the right mailer for this question. pls let us know your in= puts on this. Thanks Ram ________________________________ From: Rajeshwar Patil Sent: Tuesday, October 07, 2008 3:36 PM To: freebsd-questions@freebsd.org Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same pri= ority Hi, I have two threads of same priority lets say thread A and thread B, thread = A is starving since thread B has got a for loop of size 100k. So, to avoid = starvation I am processing 1000 iterations of for loop(that of thread B) in= one batch and giving sleep of 1ms, but still thread A is starving. If I in= crease the batch size used in for loop from 1k to 10k then starvation of th= read A is little less whereas it should be more as Im increasing processing= time of thread B. I tried various combinations of sleeps and batches, but = I couldnt solve the starvation of thread A. Is the sleep right solution? I will be grateful if someone could answer on this. Thanks Rajeshwar ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely = for the use of the individual to whom it is addressed. It may contain privi= leged or confidential information and should not be circulated or used for = any purpose other than for what it is intended. If you have received this m= essage in error,please notify the originator immediately. If you are not th= e intended recipient, you are notified that you are strictly prohibited fro= m using, copying, altering, or disclosing the contents of this message. Ari= cent accepts no responsibility for loss or damage arising from the use of t= he information transmitted by this email including damage from virus." From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 11:58:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD067106569E for ; Tue, 7 Oct 2008 11:58:53 +0000 (UTC) (envelope-from Ramachandran.Sathyanarayanan@aricent.com) Received: from jaguar.aricent.com (jaguar.aricent.com [121.241.96.11]) by mx1.freebsd.org (Postfix) with ESMTP id 14B038FC1A for ; Tue, 7 Oct 2008 11:58:52 +0000 (UTC) (envelope-from Ramachandran.Sathyanarayanan@aricent.com) Received: from jaguar.aricent.com (localhost [127.0.0.1]) by jaguar.aricent.com (8.13.8/8.13.8) with ESMTP id m97Bsj12027856 for ; Tue, 7 Oct 2008 17:24:45 +0530 Received: from GUREXHT01.ASIAN.AD.ARICENT.COM (gurexht01.asian.ad.aricent.com [10.203.171.136]) by jaguar.aricent.com (8.13.8/8.13.8) with ESMTP id m97BsjbQ027828 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=FAIL) for ; Tue, 7 Oct 2008 17:24:45 +0530 Received: from GUREXMB02.asian.ad.aricent.com ([10.203.171.132]) by GUREXHT01.ASIAN.AD.ARICENT.COM ([10.203.171.136]) with mapi; Tue, 7 Oct 2008 17:28:48 +0530 From: Ramachandran Sathyanarayanan To: "freebsd-hackers@freebsd.org" Date: Tue, 7 Oct 2008 17:28:47 +0530 Thread-Topic: sleep is not working to avoid starvation among threads of same priority Thread-Index: AckoZFjwx1hc0ptOQfe07GpuldRJxQABEJqAAALY7EA= Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same priority X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 11:58:54 -0000 Hi I hope this is the right mailer for this question. pls let us know your in= puts on this. Thanks Ram ________________________________ From: Rajeshwar Patil Sent: Tuesday, October 07, 2008 3:36 PM To: freebsd-questions@freebsd.org Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil Subject: sleep is not working to avoid starvation among threads of same pri= ority Hi, I have two threads of same priority lets say thread A and thread B, thread = A is starving since thread B has got a for loop of size 100k. So, to avoid = starvation I am processing 1000 iterations of for loop(that of thread B) in= one batch and giving sleep of 1ms, but still thread A is starving. If I in= crease the batch size used in for loop from 1k to 10k then starvation of th= read A is little less whereas it should be more as Im increasing processing= time of thread B. I tried various combinations of sleeps and batches, but = I couldnt solve the starvation of thread A. Is the sleep right solution? I will be grateful if someone could answer on this. Thanks Rajeshwar ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely = for the use of the individual to whom it is addressed. It may contain privi= leged or confidential information and should not be circulated or used for = any purpose other than for what it is intended. If you have received this m= essage in error,please notify the originator immediately. If you are not th= e intended recipient, you are notified that you are strictly prohibited fro= m using, copying, altering, or disclosing the contents of this message. Ari= cent accepts no responsibility for loss or damage arising from the use of t= he information transmitted by this email including damage from virus." From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 12:20:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB4461065689 for ; Tue, 7 Oct 2008 12:20:07 +0000 (UTC) (envelope-from jille@quis.cx) Received: from mulgore.hexon-is.nl (mulgore.hexon-is.nl [82.94.237.14]) by mx1.freebsd.org (Postfix) with ESMTP id 456108FC5C for ; Tue, 7 Oct 2008 12:20:07 +0000 (UTC) (envelope-from jille@quis.cx) X-Hexon-MailScanner-Watermark: 1223985830.56261@cfWsh1CYpRTeWtMdVexUoQ Received: from [10.0.0.72] ([10.15.16.6]) (authenticated bits=0) by mulgore.hexon-is.nl (8.14.1/8.13.8) with ESMTP id m97C3mYd030988; Tue, 7 Oct 2008 14:03:49 +0200 Message-ID: <48EB5036.80209@quis.cx> Date: Tue, 07 Oct 2008 14:04:06 +0200 From: Jille Timmermans User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Ramachandran Sathyanarayanan References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Hexon-MailScanner-Information: Please contact the ISP for more information X-Hexon-MailScanner-ID: m97C3mYd030988 X-Hexon-MailScanner: Found to be clean X-Hexon-MailScanner-From: jille@quis.cx Cc: "freebsd-hackers@freebsd.org" , Rajeshwar Patil Subject: Re: sleep is not working to avoid starvation among threads of same priority X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 12:20:07 -0000 Hello, Take a look at pthread_yield(): DESCRIPTION The pthread_yield() forces the running thread to relinquish the processor until it again becomes the head of its thread list. (Note that it is not portable) And if you want to use sleep, I found out that using sleep with more ms does 'yield' the thread/process. iirc, 20ms was enough to stop it from using 100% CPU. -- Jille Ramachandran Sathyanarayanan wrote: > Hi > > I hope this is the right mailer for this question. pls let us know your inputs on this. > > Thanks > > Ram > > ________________________________ > From: Rajeshwar Patil > Sent: Tuesday, October 07, 2008 3:36 PM > To: freebsd-questions@freebsd.org > Cc: Ramachandran Sathyanarayanan; Rajeshwar Patil > Subject: sleep is not working to avoid starvation among threads of same priority > > Hi, > > I have two threads of same priority lets say thread A and thread B, thread A is starving since thread B has got a for loop of size 100k. So, to avoid starvation I am processing 1000 iterations of for loop(that of thread B) in one batch and giving sleep of 1ms, but still thread A is starving. If I increase the batch size used in for loop from 1k to 10k then starvation of thread A is little less whereas it should be more as Im increasing processing time of thread B. I tried various combinations of sleeps and batches, but I couldnt solve the starvation of thread A. > > Is the sleep right solution? > > I will be grateful if someone could answer on this. > > Thanks > Rajeshwar > > > > > > > ________________________________ > "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 16:25:21 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9A12106568B for ; Tue, 7 Oct 2008 16:25:21 +0000 (UTC) (envelope-from shaun@FreeBSD.org) Received: from dione.picobyte.net (81-86-230-94.dsl.pipex.com [81.86.230.94]) by mx1.freebsd.org (Postfix) with SMTP id 69DC78FC17 for ; Tue, 7 Oct 2008 16:25:21 +0000 (UTC) (envelope-from shaun@FreeBSD.org) Received: from charon.picobyte.net (charon.picobyte.net [IPv6:2001:770:15d::fe03]) by dione.picobyte.net (Postfix) with ESMTP id 51DC4B878; Tue, 7 Oct 2008 16:51:00 +0100 (BST) Date: Tue, 7 Oct 2008 16:50:59 +0100 From: Shaun Amott To: Evren Yurtesen Message-ID: <20081007155059.GA20615@charon.picobyte.net> References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <48EA83CE.4060702@ispro.net> User-Agent: Mutt/1.5.16 (FreeBSD i386) Cc: Sean Bruno , freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 16:25:22 -0000 On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > so FreeBSD could be supported also. As you can imagine, it is not only > important that data can be restored when a box hardware failure etc. it is > also important that data can be restored if deleted by accidents etc. While > traditional backup programs provide this functionality, you cant really go > back to 10 min or 1h ago, often they take daily backups and have to scan > whole filesystem for changed files every time the backup is taken which > stresses out the systems. > This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. -- Shaun Amott // PGP: 0x6B387A9A "A foolish consistency is the hobgoblin of little minds." - Ralph Waldo Emerson From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 16:25:52 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A749A106569A for ; Tue, 7 Oct 2008 16:25:52 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from rn-out-0910.google.com (rn-out-0910.google.com [64.233.170.186]) by mx1.freebsd.org (Postfix) with ESMTP id 53BFF8FC27 for ; Tue, 7 Oct 2008 16:25:52 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by rn-out-0910.google.com with SMTP id j71so1039135rne.12 for ; Tue, 07 Oct 2008 09:25:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=W23/7c1e0Lihwjrk//4SCHhN6z4JuRfpGP+8ZD7hUX4=; b=uI4ScwAR7FAKtQy3dfEEvibhlAGnLpz1SF9E+rUMQYdbOIVrw4mVSwqppwMlTQfOF5 4W7bMWk5PtE8VmyHxsrHYB/ZxbZIm8mmYgodjFDDySNrNmg6xQU9yXzzPO2R6YOCNpN0 8VvzCRgLsnWl4tWuVVVz2ovKxNDev95uEDZrk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=fVRbu+nUT3fqTSCGr3YrNB5JU929ITPjvDh+kj33goLo3j9FBao81Zbef36CvtAr1Q IB9UAF7YNUyTSQEs9CJL7FEytEOPcilKLtTrQEnoXJqLzIRj1NIaWD7wpWlqS0NSWtAt pnrx6tTEMSYMz6erFG0Kyxswu2GJkkTWGXoF0= Received: by 10.151.26.12 with SMTP id d12mr10018371ybj.222.1223396751467; Tue, 07 Oct 2008 09:25:51 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Tue, 7 Oct 2008 09:25:51 -0700 (PDT) Message-ID: <5f67a8c40810070925k109e3510y18fa64861ef06f03@mail.gmail.com> Date: Tue, 7 Oct 2008 12:25:51 -0400 From: "Zaphod Beeblebrox" To: "Evren Yurtesen" In-Reply-To: <48EB1A1C.7020701@ispro.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <001AD718-D25B-421B-8B0F-CE71FA5A7CF0@gid.co.uk> <48EA21AE.80607@ispro.net> <5f67a8c40810060852k4c51c8far511891c4b135a1e2@mail.gmail.com> <48EA6939.6090405@ispro.net> <5f67a8c40810061508t300e77c7n8c1439462622a71c@mail.gmail.com> <48EA9459.2000807@ispro.net> <5f67a8c40810061707m33a52547idae13e2bb9eb2f9a@mail.gmail.com> <48EB1A1C.7020701@ispro.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 16:25:52 -0000 On Tue, Oct 7, 2008 at 4:13 AM, Evren Yurtesen wrote: > > Zaphod Beeblebrox wrote: > > >> From my reading, Hammer is much more than a filesystem, but then you >> probably havn't read about it yet. By my reading, Hammer hits all their >> feature points and does it better _because_ it's a filesystem. >> > > I glanced through these actually: > http://www.dragonflybsd.org/hammer/ > I didnt see anywhere that it will replace backup programs? > You need to read the PDF on that page. On the first page of the PDF the 4th point is "queue-less incremental mirroring" If you read the PDF to determine what this phrase means, you'll find it describes the filesystem as a database indexed by a B+ tree of radix 64. It mentions that you can easily select all changes newer than a certain time. The problem with backup solutions that run on raw blocks is that you need meta-data to queue up the blocks that have changed faster than you can ship them out. Hammer solves this by allowing you to select changes based on when you shipped out the last batch of changes. The granularity of this will be dependent on how fast you can do this. > > It's relatively simple. Database replication solves the data backup >> problem (I don't store application data outside the database). Database >> replication for both MySQL and PostgreSQL is relatively straight forward. >> As for the configuration of code and servers --- that is taken care of with >> configuration management (it's really a bigger issue than just backing up a >> filesystem) and installing a new server to take a place in the cluster is a >> straight forward checkout from the CM system. For things I really care >> about staying up, add VRRP and an application design that is fault tolerant. >> > > Have you ever tried to restore data from MySQL replication logs? :) Even if > you use binary logging, when you want to go back in time. You will need to > first restore the whole database first from normal backups then replay the > logs until the point that you wanna be. There is no simple way to go back in > time. That is of course you have backups. If you dont have backups because > you think replication is a backup solution, you would be screwed. Totally > more complicated that clicking from the web to select data and time and > table and restore! Largely why I use PostgreSQL. > Also, you are thinking about a very small sized system. While replication > might work if you are relatively small sized company (like 1-2 servers). If > you have many independent servers with different databases inside you just > cant use it. Even if you could replicate multiple boxes into one box, there > would eventually be problems such as same named databases etc. and even > then, you cant just easily restore the data if the user deletes all his data > in his tables. Again, this is application design. I know of rather large organizations that use PostgreSQL replication to deliver very serious and onerous SLAs. Several of them have sponsored the Slovney (sp? russian for elephan --- the PostgreSQL multi-master replication thing). > Also that is not practical for users at all. For example I cant give an > option to the user to restore his data by himself. While that is possible > with most backup software easily. Well... if we're talking about a random samba servers for windoze users, ZFS is the tech you want. Users can retrieve their own snapshots, etc. ZFS was designed with generalized fileserver duties in mind. > And how do you propose that I restore a table in the database to of 1h > before status? like you can do with a data backup solution? You are talking > about a spare server solution not backup solution. Replication IS NOT > backup. If you look at articles and information about database replication, > almost all mention that it DOES NOT replace backups. > I use differential backups to give old database restore functionality. I use replication to give me a live alternative database for failures (different risks, different solutions). > Well... no. Backup software and strategies are just one available tool >> for risk mitigation. To know what tools you require, you must define your >> risks. Then with your list of risks you look at the cost of each tool and >> find the toolset that suits you. By the responses in this thread, it seems >> like the set of FreeBSD developers and the set of people who desire this >> solution are disjoint. >> > > Right, I just cant use the tool I require. There is no way to take near > continuous backups of FreeBSD filesystems. > You're free to pay anyone you'd like to implement this solution. I'd be happy to work for hire. I can provide a quote if you're serious about it. But if I was doing it for free, I'd want it to be fun and interesting. I'd be imlementing a server as well as a client. In fact, if I were doing this as my own open source project, I'd be looking at a better way of achieving it. FreeBSD is, after all, about doing things right. But --- as I said --- regardless of what I'd do if I were doing it for free, I will achieve the goals set out for me if I'm paid. From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 16:37:38 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD1D2106569B for ; Tue, 7 Oct 2008 16:37:38 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-gx0-f21.google.com (mail-gx0-f21.google.com [209.85.217.21]) by mx1.freebsd.org (Postfix) with ESMTP id 94AF68FC1C for ; Tue, 7 Oct 2008 16:37:38 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by gxk14 with SMTP id 14so6671425gxk.19 for ; Tue, 07 Oct 2008 09:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=LS+tXDlhE2T1inhyZOrWoAVI8C7hOM3NPz4EPEHQIvA=; b=L6dEj3GtJSQSJNJWO+db0FQh42+DQPJEXcCATXJBHfmrKjxJPnc/BiJ2i14Vvsx5Ah u+vGtuYjCZshRDFctCeOev2dCZzZrXrwDRVNN6Jw5n9uFMG2rvhluzBT1BZgIfHBtJ0X ZRsk8HZ51md/ESdOCDAifLiEGxRY7eop8pX8Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=lbV7tp43FcqjhutdNKUQRl/rFyuDLu/Lvs+ikyRQkevhDu8mwdMnxBW33M1Vvo05Bb oIwb5lTV7rr8fu5oJYmi1XAkn3u3pWKiwwydDiSF065X+tQczQQurdyKaMkgJ6059nGz NkaclpvxfSdtqarhCQ2remMjChlpg4hhRE+F4= Received: by 10.151.145.21 with SMTP id x21mr10080189ybn.97.1223397457469; Tue, 07 Oct 2008 09:37:37 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Tue, 7 Oct 2008 09:37:37 -0700 (PDT) Message-ID: <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> Date: Tue, 7 Oct 2008 12:37:37 -0400 From: "Zaphod Beeblebrox" To: "=?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?=" In-Reply-To: <861vysiv9i.fsf@ds4.des.no> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 16:37:39 -0000 I wanted to respond to DES' email separately --- because he's right. On Tue, Oct 7, 2008 at 5:56 AM, Dag-Erling Sm=F8rgrav wrote: > Evren Yurtesen writes: > > They actually do not think that it is an easy job to adapt their > > software to support FreeBSD even. See this post: > > http://forum.r1soft.com/showpost.php?p=3D4224&postcount=3D3 > > All this shows is that they don't know anything about FreeBSD at all > (plus they need a refresher course in OS design; Linux is also a > monolithic kernel) A very succinct way of making a similar poing :) > What really annoys me with this thread is that nobody has provided any > information at all that would allow someone to understand what needs to > be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). My primary concern about hammer is it's floating history stuff. It seems t= o me that legal compliance might have some things to say about it. It seems (from the document --- I havn't read the source) that the tunability of the "real deletion" of data are "goals" not absolutes. This is a concern. But as filesystems _are_ databases and as they grow database technology (like transactions and B+tree indexes), we should look to database systems for some of the solutions (ie: problems already solved). Replication replacing "mirroring" is just one of them. Doing this at the block level was suggested by someone earlier (BTW). Suggesting that a geom node could store a bit or two per bock (marking it "dirty" prehaps) and shipping off the blocks. That only solves the replication. You'd need something like a transaction ID per block stored o= n the backup server to enable time travel. Probably want a B+tree index ther= e too. If you're not careful, you might find implementing the filesystem solution much easier. You could, however, implement this with hooks for gjournal --- such that the filesystems you backup are always sane. From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 16:40:16 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D82AD106569C for ; Tue, 7 Oct 2008 16:40:16 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id AD0568FC1C for ; Tue, 7 Oct 2008 16:40:16 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 2E3085B29; Tue, 7 Oct 2008 09:40:15 -0700 (PDT) To: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= In-reply-to: Your message of "Tue, 07 Oct 2008 11:56:09 +0200." <861vysiv9i.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> Comments: In-reply-to =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= message dated "Tue, 07 Oct 2008 11:56:09 +0200." Date: Tue, 07 Oct 2008 09:40:15 -0700 From: Bakul Shah Message-Id: <20081007164015.2E3085B29@mail.bitblocks.com> Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 16:40:16 -0000 On Tue, 07 Oct 2008 11:56:09 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Evren Yurtesen writes: > > They actually do not think that it is an easy job to adapt their > > software to support FreeBSD even. See this post: > > http://forum.r1soft.com/showpost.php?p=3D4224&postcount=3D3 > > All this shows is that they don't know anything about FreeBSD at all > (plus they need a refresher course in OS design; Linux is also a > monolithic kernel) > > What really annoys me with this thread is that nobody has provided any > information at all that would allow someone to understand what needs to > be done and estimate how hard it would be. >From their http://forum.r1soft.com/CDP.html page: R1Soft's Continuous Data Protection solution is a ==>>near-Continuous Backups system capable of providing <<== hundreds of recovery points per day scheduled as little as 5 or 10 minutes apart. ... CDP Server works by reading your hard disk volumes at the sector level, bypassing the file system for the ultimate in performance and recovery. This disk sector synchronization is performed while the server is online and provides no interruption to other I/O requests even on a busy server. Clearly "near-Continuous" is *not* the same as "continuous" but never mind -- truthiness in business is so last century! But this could be the cause of some confusion. What they do is backups, not mirroring. A remote mirror would essentially require a continuous "backup" -- every disk write must be sent right away but in pure mirroring there is no access previous snapshots. In a true backup solution you can restore disk state to some number of previous backup points, regardless of whether you have *online* access to them. My guess is they have a driver that keeps track of disk writes. Something like set bit N of a bitmap when sector N is to be written. Then once every 10 minutes (or whatever snapshot interval you have selected) a client app scans the bitmap and sends these sectors to the backup server. If they did *just* this, there'd be consistency issues -- between the time a snapshot is taken and some sector N is actually backed up, there may be new writes to N by the OS. To deal with this, the new write must be delayed until N has been backed up. Another alternative is to "slide" forward the snapshot point. That is, if the snapshot was taken at time T1 and the backup finished by T2, and there were conflicting writes during [T1..T2), backup these writes as well and slide forward this snapshot time from T1 to T2. Repeat until there are no conflicting writes. This latter method won't block any writes from the OS. So my guess is they need an interface where they get notified for every disk write and optionally a way to delay a write. [To respond to an earlier point raised by others, I believe OS X Time Machine does a filesystem level backup and not at the disk level.] From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 16:42:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D12B1065788 for ; Tue, 7 Oct 2008 16:42:35 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.234]) by mx1.freebsd.org (Postfix) with ESMTP id 547CA8FC0A for ; Tue, 7 Oct 2008 16:42:35 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by wx-out-0506.google.com with SMTP id s17so764544wxc.7 for ; Tue, 07 Oct 2008 09:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=0T9nz1rflil8vljRYMwgWRHiMQrOn0weOWNk3CkstX0=; b=ACKWxD13yPgFYSDwowpSaWQFwmVNxF46BGUkpZu+t0GUBgsydU0daMymuv/TD6SnL6 Is5RUZRK+UZEKOcz+bpOJjEPxXJ3fny+OCSxbF6a2kaG+itfZmC6BrxvwC4XWa5ZPKkP hdcSmJX/rjOeuMaOY6y+fkLi6gBVjtk5KMkhU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=AZbV+VwjE5iF/OoPNxOAAdE8zuelY/tk4gJcj14Pufz7uZ71ZKVlSRvF45VoZ4bQY6 RPONTM3eyggT+AvClLqJZQNuojpC+BtFuNrNM/Ym6WgBOz2nrK3UjrKDPb+XiIevmN7G EjLJRON+HNTp0ksjlirwpDphqelc0cTut7Cvk= Received: by 10.151.156.15 with SMTP id i15mr10042365ybo.184.1223397753979; Tue, 07 Oct 2008 09:42:33 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Tue, 7 Oct 2008 09:42:33 -0700 (PDT) Message-ID: <5f67a8c40810070942k317d55f4i5d46b1409ef34034@mail.gmail.com> Date: Tue, 7 Oct 2008 12:42:33 -0400 From: "Zaphod Beeblebrox" To: "Shaun Amott" In-Reply-To: <20081007155059.GA20615@charon.picobyte.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Sean Bruno , Evren Yurtesen , freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 16:42:35 -0000 On Tue, Oct 7, 2008 at 11:50 AM, Shaun Amott wrote: > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > > > so FreeBSD could be supported also. As you can imagine, it is not only > > important that data can be restored when a box hardware failure etc. it > is > > also important that data can be restored if deleted by accidents etc. > While > > traditional backup programs provide this functionality, you cant really > go > > back to 10 min or 1h ago, often they take daily backups and have to scan > > whole filesystem for changed files every time the backup is taken which > > stresses out the systems. > > > > This can (more or less) be achieved with snapshots: you can cheaply > maintain old versions of the file system, and mount an old snapshot at > any time. Hourly is about as fine-grained as you can expect though. > Unfortunately, as mentioned on another recent thread, FreeBSD UFS snapshots are quite fragile. The last time I tried this type-of-thing, it rather quickly resulted in filesystem corruption on moderately busy filesystems. Also, the cost of UFS snapshots (in time, while locking the filesystem) are unacceptable in practice. Now... ZFS seems to do this well enough. I use snapshots and send snapshots to my backup server --- it all seems to work OK. The idea that snapshots have infinite granularity and are automatic (introduced to me by the Hammer filesystem document) has merit, though --- and this is certainly core to the featurelist of the company that started this thread . From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 17:53:23 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C12110656A1 for ; Tue, 7 Oct 2008 17:53:23 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from mail-gx0-f21.google.com (mail-gx0-f21.google.com [209.85.217.21]) by mx1.freebsd.org (Postfix) with ESMTP id 42EC58FC17 for ; Tue, 7 Oct 2008 17:53:23 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: by gxk14 with SMTP id 14so6804473gxk.19 for ; Tue, 07 Oct 2008 10:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=4rmWkFnBya4m3daNDhDWv+9oyr2PdvF1tfR4LwJAjsA=; b=gRzOgW34us9NNTYzk2pqxRp6x+JP58lzEI8onr11MtUcNKAM75b4Hvsf1JTuJ7OTKf 704SinjDXz+PSIYNkfjerexYpJAxB/TX299faHI8PDTC3+aWicAo58nN8Bhe/RkI9Rq2 RIYCOCv8AQrwsCE7hsDYwFrkGvE5igx01zZKI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=l1P8F20BZFTuDnHGUruar1eGn/7ht5GoicXZ6aZDPH3GfgA+K7qYfcWJfAko/Hvylo AeX+3AmkZHIU1VhYNGmx/8GfifRBAyx6Odv2vA/roisUlD29RiRW4Zvn1NlskTcn7y2+ 5PVdNhnTD8+fD8d5abF58DgRraXRaSizQHAuw= Received: by 10.90.104.20 with SMTP id b20mr7795248agc.69.1223402002354; Tue, 07 Oct 2008 10:53:22 -0700 (PDT) Received: by 10.90.94.14 with HTTP; Tue, 7 Oct 2008 10:53:22 -0700 (PDT) Message-ID: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Date: Tue, 7 Oct 2008 10:53:22 -0700 From: "alan yang" To: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org In-Reply-To: <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> Cc: Subject: Re: kgdb debugging X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 17:53:23 -0000 Could people shed some light how to get remote debugging going, must be something that i overlooked, really appreciate. Two FreeBSD7 systems, target and development, connected with null modem cable on each's COM1. step 1) - rebuild kernel with following options: options DDB options KDB options GDB makeoptions DEBUG=-g step 2) - from development system cd to /usr/src/sys/i386/comiple/MYKERNEL kgdb -r /dev/cuad0 kernel.debug it displays the following: Switching to remote protocol Ignoring packet error, continuing ......... Couldn't establish connection to remote target Malformed response to offset query, timeout step 3) - from targetsystem: 1. Ctrl + Alt + Esc to go into db 2. from db> type gdb 3. it displays: "The remote GDB backend could not be selected" On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: > the problem is, when entering gdb from db as described in the following: > -- > Enter ing gdb from ddb > In FreeBSD you can build a kernel with support for both ddb and gdb. > You can then > change backwards and forwards between them. For example, if you're in > ddb, you can > go to gdb like this: > db> gdb > Next trap will enter GDB remote protocol mode > db> si step a single instruction to reenter ddb > -- > after typing gdb command, it says: "The remote GDB backend could not > be selected" > that i am not sure what this indicates; looking at subr_kdb.c, wonder > maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced > anywhere, not sure how to get it right. > > thanks for shed some light ... > > On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> alan yang wrote: >> | hi, there, >> | >> | wonder people can shed some lights on remote debugging. i have >> | freebsd7 configured with option DDB / KDB / GDB but after entering the >> | db on the target system the command gdb gives "the remote GDB backend >> | could not be selected". >> | >> | i browsed through the mailing list, and do find 1 similar post but >> | without answer. >> | >> | thanks in advance & apology if i overlooked things ... >> >> I suggest the following tutorial: >> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >> >> Have fun :) >> >> | >> | cheers, >> | alan >> | _______________________________________________ >> | freebsd-hackers@freebsd.org mailing list >> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> >> >> - -- >> Pietro Cerutti >> gahr@FreeBSD.org >> >> PGP Public Key: >> http://gahr.ch/pgp >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2.0.9 (FreeBSD) >> >> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >> =6Pdm >> -----END PGP SIGNATURE----- >> > From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 18:12:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDB231065692 for ; Tue, 7 Oct 2008 18:12:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.185]) by mx1.freebsd.org (Postfix) with ESMTP id 410278FC1A for ; Tue, 7 Oct 2008 18:12:07 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so2916537fkk.11 for ; Tue, 07 Oct 2008 11:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=DZRObyjMePyn7F5qfwiYcvONkBKmxz4QbogdkIUwBU0=; b=ofjy9XvOInl7sBtnFSMU83OKXJHbSr4vgBMP2XhrZpruluLRFwPA1qYNsSffqTrNKG A9v8FmMPW3ctZ31QJ7Qqj4jNK5KTLyJUwmMjQzEFrpFLnnAnxsvcWoj1aaDAPBIpAT6s JM3+L/1R25t6peBMPkEYYvL2jFQ66mLYzp2I0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=FXitta0IelTQIXIPoBvzoyFOr3zYbX43Gz470l361ZlegIpB4i6ibVW/q/FD5qNZvu TcgrJWVIIkFXmjHo0/FBoQHxFwFgGCXFzNYCVfW34mN7Lcpe2cAz9FWWGcjGOL65JBzI oGpio8f/2zuKlMaohCqJNzhCpyrQq8wDL5fB4= Received: by 10.187.158.11 with SMTP id k11mr1375690fao.16.1223403126254; Tue, 07 Oct 2008 11:12:06 -0700 (PDT) Received: by 10.187.161.6 with HTTP; Tue, 7 Oct 2008 11:12:06 -0700 (PDT) Message-ID: Date: Tue, 7 Oct 2008 11:12:06 -0700 From: "Navdeep Parhar" To: "alan yang" In-Reply-To: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: kgdb debugging X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 18:12:08 -0000 On Tue, Oct 7, 2008 at 10:53 AM, alan yang wrote: > Could people shed some light how to get remote debugging going, must > be something that i overlooked, really appreciate. > Do you have the right flags for sio or uart in /boot/device.hints? I have this (for a recent HEAD):: hint.uart.0.flags="0x90" You are probably using sio as it is a FreeBSD7 system. From the man page of sio (the part that talks about flags):: ... 0x00080 use this port for remote kernel debugging ... Make sure you have this bit set. Regards, Navdeep > Two FreeBSD7 systems, target and development, connected with null > modem cable on each's COM1. > > step 1) > - rebuild kernel with following options: > options DDB > options KDB > options GDB > > makeoptions DEBUG=-g > > step 2) > - from development system > cd to /usr/src/sys/i386/comiple/MYKERNEL > kgdb -r /dev/cuad0 kernel.debug > > it displays the following: > Switching to remote protocol > Ignoring packet error, continuing > ......... > Couldn't establish connection to remote target > Malformed response to offset query, timeout > > step 3) > - from targetsystem: > 1. Ctrl + Alt + Esc to go into db > 2. from db> type gdb > 3. it displays: "The remote GDB backend could not be selected" > > > > On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: >> the problem is, when entering gdb from db as described in the following: >> -- >> Enter ing gdb from ddb >> In FreeBSD you can build a kernel with support for both ddb and gdb. >> You can then >> change backwards and forwards between them. For example, if you're in >> ddb, you can >> go to gdb like this: >> db> gdb >> Next trap will enter GDB remote protocol mode >> db> si step a single instruction to reenter ddb >> -- >> after typing gdb command, it says: "The remote GDB backend could not >> be selected" >> that i am not sure what this indicates; looking at subr_kdb.c, wonder >> maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced >> anywhere, not sure how to get it right. >> >> thanks for shed some light ... >> >> On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA512 >>> >>> alan yang wrote: >>> | hi, there, >>> | >>> | wonder people can shed some lights on remote debugging. i have >>> | freebsd7 configured with option DDB / KDB / GDB but after entering the >>> | db on the target system the command gdb gives "the remote GDB backend >>> | could not be selected". >>> | >>> | i browsed through the mailing list, and do find 1 similar post but >>> | without answer. >>> | >>> | thanks in advance & apology if i overlooked things ... >>> >>> I suggest the following tutorial: >>> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >>> >>> Have fun :) >>> >>> | >>> | cheers, >>> | alan >>> | _______________________________________________ >>> | freebsd-hackers@freebsd.org mailing list >>> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>> >>> >>> - -- >>> Pietro Cerutti >>> gahr@FreeBSD.org >>> >>> PGP Public Key: >>> http://gahr.ch/pgp >>> >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2.0.9 (FreeBSD) >>> >>> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >>> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >>> =6Pdm >>> -----END PGP SIGNATURE----- >>> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 7 18:14:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 810DF106569B for ; Tue, 7 Oct 2008 18:14:13 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outJ.internet-mail-service.net (outj.internet-mail-service.net [216.240.47.233]) by mx1.freebsd.org (Postfix) with ESMTP id 65BDA8FC26 for ; Tue, 7 Oct 2008 18:14:13 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 1A9F12493; Tue, 7 Oct 2008 11:14:14 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 29BCD2D610C; Tue, 7 Oct 2008 11:14:10 -0700 (PDT) Message-ID: <48EBA6F2.2090606@elischer.org> Date: Wed, 08 Oct 2008 02:14:10 +0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.17 (Macintosh/20080914) MIME-Version: 1.0 To: alan yang References: <290865fd0810061544ubbe92fdsf75501bb729da3f0@mail.gmail.com> <48EA9E95.80105@FreeBSD.org> <290865fd0810061712sfdf5a0p4d3954773ee27a3d@mail.gmail.com> <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> In-Reply-To: <290865fd0810071053k6da13d96j391ade1a30599fa1@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: kgdb debugging X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 18:14:13 -0000 alan yang wrote: > Could people shed some light how to get remote debugging going, must > be something that i overlooked, really appreciate. > > Two FreeBSD7 systems, target and development, connected with null > modem cable on each's COM1. > > step 1) > - rebuild kernel with following options: > options DDB > options KDB > options GDB > > makeoptions DEBUG=-g add hints.dev.uart.0.flags=0xc0 (or whatever it is) (see man uart or man sio) to /boot/device.hints > > step 2) > - from development system > cd to /usr/src/sys/i386/comiple/MYKERNEL > kgdb -r /dev/cuad0 kernel.debug > > it displays the following: > Switching to remote protocol > Ignoring packet error, continuing > ......... > Couldn't establish connection to remote target > Malformed response to offset query, timeout > > step 3) > - from targetsystem: > 1. Ctrl + Alt + Esc to go into db > 2. from db> type gdb > 3. it displays: "The remote GDB backend could not be selected" > > > > On Mon, Oct 6, 2008 at 5:12 PM, alan yang wrote: >> the problem is, when entering gdb from db as described in the following: >> -- >> Enter ing gdb from ddb >> In FreeBSD you can build a kernel with support for both ddb and gdb. >> You can then >> change backwards and forwards between them. For example, if you're in >> ddb, you can >> go to gdb like this: >> db> gdb >> Next trap will enter GDB remote protocol mode >> db> si step a single instruction to reenter ddb >> -- >> after typing gdb command, it says: "The remote GDB backend could not >> be selected" >> that i am not sure what this indicates; looking at subr_kdb.c, wonder >> maybe kdb_dbbe_set is not set? seems kdb_dbbe_set is not referenced >> anywhere, not sure how to get it right. >> >> thanks for shed some light ... >> >> On Mon, Oct 6, 2008 at 4:26 PM, Pietro Cerutti wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA512 >>> >>> alan yang wrote: >>> | hi, there, >>> | >>> | wonder people can shed some lights on remote debugging. i have >>> | freebsd7 configured with option DDB / KDB / GDB but after entering the >>> | db on the target system the command gdb gives "the remote GDB backend >>> | could not be selected". >>> | >>> | i browsed through the mailing list, and do find 1 similar post but >>> | without answer. >>> | >>> | thanks in advance & apology if i overlooked things ... >>> >>> I suggest the following tutorial: >>> http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf >>> >>> Have fun :) >>> >>> | >>> | cheers, >>> | alan >>> | _______________________________________________ >>> | freebsd-hackers@freebsd.org mailing list >>> | http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> | To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>> >>> >>> - -- >>> Pietro Cerutti >>> gahr@FreeBSD.org >>> >>> PGP Public Key: >>> http://gahr.ch/pgp >>> >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2.0.9 (FreeBSD) >>> >>> iEYEAREKAAYFAkjqnpQACgkQwMJqmJVx945RSwCgoDb0JTr8LSFDB1vpAbGUjb76 >>> ZH0An19HpFVJJTUB5/XnyZc0pIDzgxc3 >>> =6Pdm >>> -----END PGP SIGNATURE----- >>> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Sat Nov 4 11:13:03 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D8EF16A47C; Sat, 4 Nov 2006 11:13:03 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mfront8.mail.yandex.net (mfront8.mail.yandex.net [213.180.223.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 990AF43D53; Sat, 4 Nov 2006 11:13:02 +0000 (GMT) (envelope-from bu7cher@yandex.ru) Received: from YAMAIL (mfront8.yandex.ru) by mail.yandex.ru id ; Sat, 4 Nov 2006 14:12:57 +0300 Received: from [82.211.152.12] ([82.211.152.12]) by mail.yandex.ru with HTTP; Sat, 4 Nov 2006 14:12:57 +0300 (MSK) From: "Andrey V. Elsukov" Sender: bu7cher@yandex.ru Message-Id: <454C75B9.000002.09555@mfront8.yandex.ru> MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] Errors-To: bu7cher@yandex.ru To: xdivac02@stud.fit.vutbr.cz In-Reply-To: <20061104090204.GA38945@stud.fit.vutbr.cz> References: <454C55BD.000003.22283@webmail11.yandex.ru> <20061104090204.GA38945@stud.fit.vutbr.cz> X-Source-Ip: 82.211.152.12 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, kib@freebsd.org, jroberson@chesapeake.net, joel@freebsd.org Subject: Re: Yet another magic symlinks implementation X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bu7cher@yandex.ru List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Nov 2006 11:13:03 -0000 Date: Sat, 04 Nov 2006 11:13:03 -0000 X-Original-Date: Sat, 4 Nov 2006 14:12:57 +0300 (MSK) X-List-Received-Date: Sat, 04 Nov 2006 11:13:03 -0000 >On Sat, Nov 04, 2006 at 11:56:29AM +0300, Andrey V. Elsukov wrote: >> I've ported NetBSD magic symlinks implementation to FreeBSD. >> The description of magiclinks can been found here: >> http://www.daemon-systems.org/man/symlink.7.html >> >> Patch here: >> http://butcher.heavennet.ru/patches/kernel/magiclinks/ > >thnx for porting this, have you checked locking? netbsd >kernel is still under one giant lock so locking might >differ. Sorry, i'm not locking guru.. Code seems simple.. Mybe somebody from committers can see into code? Konstantin, Jeff, whath you think? -- WBR, Andrey V. Elsukov From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 03:07:40 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 125821065686 for ; Wed, 8 Oct 2008 03:07:40 +0000 (UTC) (envelope-from tummala.ushasri@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id 9692B8FC0A for ; Wed, 8 Oct 2008 03:07:39 +0000 (UTC) (envelope-from tummala.ushasri@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so2434168fgb.35 for ; Tue, 07 Oct 2008 20:07:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=t1+2ENAVcnLjqZVq1kKHnRiEAQ2ZEoT92i7tzzpboUU=; b=U4KiJam+5E2xObnECCExJzmJYL4JO5sAcdCcakFyLnYB6A6+/yH0KKdx2f+oH+IviA qo2EyhHdZ39uiR7MCw3D5+p4RsKTRMnUXDRxYxu+DlKNAn+O6vEZKEvho9M7ZJy3foYa LaGtAnyfEjHpZghSgl5dAf4fLWg9mhpnNdDr0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=EoqcEhtegqxWtW4PcaZK05bwNTqhpWyMverQyt+wXJOUnRbt+pu0mHF111fauwVEi0 CoDAPcFoN0q3XiD+Ow9znI146qAdSjSZH25KF9sLXsf0U4i4tyVLjykL/WvCo4jAGbar JOfUDUy+PcwAV+lnfRQ08SMB/cRNF6XRfQhI4= Received: by 10.86.80.5 with SMTP id d5mr6980988fgb.19.1223434036101; Tue, 07 Oct 2008 19:47:16 -0700 (PDT) Received: by 10.86.27.18 with HTTP; Tue, 7 Oct 2008 19:47:16 -0700 (PDT) Message-ID: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> Date: Tue, 7 Oct 2008 22:47:16 -0400 From: "ushasri tummala" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 03:07:40 -0000 What is the time between 2 mi_switches in freebsd?In which variable is this information stored? Could you plz help me out with this Thank You, Usha. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 08:14:59 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DFC2106568C for ; Wed, 8 Oct 2008 08:14:59 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 262058FC1F for ; Wed, 8 Oct 2008 08:14:59 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 480992083; Wed, 8 Oct 2008 10:14:58 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 23DC4844BA; Wed, 8 Oct 2008 10:14:58 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: "Zaphod Beeblebrox" References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> Date: Wed, 08 Oct 2008 10:14:58 +0200 In-Reply-To: <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> (Zaphod Beeblebrox's message of "Tue, 7 Oct 2008 12:37:37 -0400") Message-ID: <86iqs3sdtp.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 08:14:59 -0000 "Zaphod Beeblebrox" writes: > "Dag-Erling Sm=C3=B8rgrav" writes: > > What really annoys me with this thread is that nobody has provided > > any information at all that would allow someone to understand what > > needs to be done and estimate how hard it would be. > Well... I hinted that a hammer port would be sufficient (although they > need to finish their replication design) and I hinted that the hammer > approach may be graftable to ZFS. Both reasonably large effort-wise > (but probably within the scope of a single developer with sufficient > time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, "CDP is a service that captures changes to data to a separate storage location" - emphasis on "separate". DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 08:18:48 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B240D106571F for ; Wed, 8 Oct 2008 08:18:48 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 6A84A8FC22 for ; Wed, 8 Oct 2008 08:18:48 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 8B5D62049; Wed, 8 Oct 2008 10:18:47 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 6667C844BA; Wed, 8 Oct 2008 10:18:47 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Bakul Shah References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> Date: Wed, 08 Oct 2008 10:18:47 +0200 In-Reply-To: <20081007164015.2E3085B29@mail.bitblocks.com> (Bakul Shah's message of "Tue, 07 Oct 2008 09:40:15 -0700") Message-ID: <86ej2rsdnc.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 08:18:48 -0000 Bakul Shah writes: > "Dag-Erling Sm=C3=B8rgrav" writes: > > What really annoys me with this thread is that nobody has provided any > > information at all that would allow someone to understand what needs to > > be done and estimate how hard it would be. > From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 08:28:31 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 714EC1065687 for ; Wed, 8 Oct 2008 08:28:31 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 1D7DA8FC18 for ; Wed, 8 Oct 2008 08:28:30 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender; b=VKubplA1Sy/mgplVzbl9KCBcRAxuT7VY6pW70MY3uAsq00v3uVtpGwaWNeL0uJLHljMWCGWdUo2rLG7d4Q2wpJL2xMo/vJ4MGn+qnUBzR/+NlHGejk2e7/HH4yv01l+Td1wCHPosFdCsrB6eUDPdhB9R/7lf0HQQAaYU0m06FTg=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1KnUPF-00042U-Bx; Wed, 08 Oct 2008 12:28:29 +0400 Date: Wed, 8 Oct 2008 12:28:28 +0400 From: Eygene Ryabinkin To: "O. Hartmann" Message-ID: References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+M8NfDDHYT2VBqDz" Content-Disposition: inline In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> Sender: rea-fbsd@codelabs.ru Cc: hackers@FreeBSD.ORG Subject: Re: HEADS UP: GCC 4.2.0 is coming X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 08:28:31 -0000 --+M8NfDDHYT2VBqDz Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Good day. Wed, Oct 08, 2008 at 07:57:51AM +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 > > Alexander Kabaev wrote: > >=20 > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > >=20 > > Done. > >=20 >=20 > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not=20 > 4.2.1 as it is used in 7.X? It was the old message that slipped to the list due to some misconfiguration of the mailing list software. Look at its date -- it's more than one year old. --=20 Eygene _ ___ _.--. # \`.|\..----...-'` `-._.-'_.-'` # Remember that it is hard / ' ` , __.--' # to read the on-line manual =20 )/' _/ \ `-_, / # while single-stepping the kernel. `-'" `"\_ ,_.-;_.-\_ ', fsc/as # _.-'_./ {_.' ; / # -- FreeBSD Developers handbook=20 {_.-``-' {_/ # --+M8NfDDHYT2VBqDz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjsbywACgkQthUKNsbL7Yj/6gCfStFXAckzo10H4eBzs1IqEkNp +08AoIPbDm96UsY87DnVgnOeSy/sdmip =Sd6s -----END PGP SIGNATURE----- --+M8NfDDHYT2VBqDz-- From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 08:40:10 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00B89106568A for ; Wed, 8 Oct 2008 08:40:10 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from ch-smtp02.sth.basefarm.net (ch-smtp02.sth.basefarm.net [80.76.149.213]) by mx1.freebsd.org (Postfix) with ESMTP id AC9468FC27 for ; Wed, 8 Oct 2008 08:40:09 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from c83-255-48-78.bredband.comhem.se ([83.255.48.78]:54586 helo=falcon.midgard.homeip.net) by ch-smtp02.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1KnULp-0008O0-8o for hackers@FreeBSD.ORG; Wed, 08 Oct 2008 10:25:01 +0200 Received: (qmail 64325 invoked from network); 8 Oct 2008 10:24:49 +0200 Received: from owl.midgard.homeip.net (10.1.5.7) by falcon.midgard.homeip.net with ESMTP; 8 Oct 2008 10:24:49 +0200 Received: (qmail 3770 invoked by uid 1001); 8 Oct 2008 10:24:49 +0200 Date: Wed, 8 Oct 2008 10:24:49 +0200 From: Erik Trulsson To: "O. Hartmann" Message-ID: <20081008082449.GA3741@owl.midgard.homeip.net> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> User-Agent: Mutt/1.5.18 (2008-05-17) X-Originating-IP: 83.255.48.78 X-Scan-Result: No virus found in message 1KnULp-0008O0-8o. X-Scan-Signature: ch-smtp02.sth.basefarm.net 1KnULp-0008O0-8o 8e70084920f49359eadf340ff3947a12 Cc: hackers@FreeBSD.ORG, current@FreeBSD.ORG Subject: Re: HEADS UP: GCC 4.2.0 is coming X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 08:40:10 -0000 On Wed, Oct 08, 2008 at 07:57:51AM +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 > > Alexander Kabaev wrote: > > > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > > > > Done. > > > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not > 4.2.1 as it is used in 7.X? Take a close look at the date of the message you were replying to. It is somewhat old. (There seems to have been a mail-server hiccup somewhere.) -- Erik Trulsson ertr1013@student.uu.se From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 11:20:05 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B97681065690 for ; Wed, 8 Oct 2008 11:20:05 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 314CE8FC1E for ; Wed, 8 Oct 2008 11:20:04 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id m98BK3Vb043546; Wed, 8 Oct 2008 13:20:03 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id m98BK2fV043545; Wed, 8 Oct 2008 13:20:02 +0200 (CEST) (envelope-from olli) Date: Wed, 8 Oct 2008 13:20:02 +0200 (CEST) Message-Id: <200810081120.m98BK2fV043545@lurza.secnetix.de> From: Oliver Fromme To: freebsd-hackers@FreeBSD.ORG, "Zaphod Beeblebrox" , Volker , Evren Yurtesen In-Reply-To: <86iqs3sdtp.fsf@ds4.des.no> X-Newsgroups: list.freebsd-hackers User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.4-PRERELEASE-20080904 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Wed, 08 Oct 2008 13:20:03 +0200 (CEST) Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@FreeBSD.ORG, Zaphod Beeblebrox , Volker , Evren Yurtesen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 11:20:05 -0000 Dag-Erling Smørgrav wrote: > "Zaphod Beeblebrox" writes: > > "Dag-Erling Smørgrav" writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". FWIW, the HAMMER file system _does_ support replication to remote targets (thus "separate"). Unfortunately they call this feature "mirroring", which is misleading at best. It's really rather a replication mechanism, much like the binlog of MySQL. It can be used for various purposes, including live mirroring, delayed mirroring, archiving, backup and point-in-time recovery. Well, of course, all of that doesn't help us at all because HAMMER doesn't exist on FreeBSD. However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "File names are infinite in length, where infinity is set to 255 characters." -- Peter Collinson, "The Unix File System" From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 08:18:35 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 095501065686; Wed, 8 Oct 2008 08:18:35 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id B74208FC17; Wed, 8 Oct 2008 08:18:34 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1KnTxu-0000i2-Pt>; Wed, 08 Oct 2008 10:00:14 +0200 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1KnTxu-0001SX-Ot>; Wed, 08 Oct 2008 10:00:14 +0200 Message-ID: <48EC67FF.3030900@zedat.fu-berlin.de> Date: Wed, 08 Oct 2008 07:57:51 +0000 From: "O. Hartmann" Organization: Freie =?ISO-8859-15?Q?Universit=E4t_Berlin?= User-Agent: Thunderbird 2.0.0.17 (X11/20080927) MIME-Version: 1.0 To: Alexander Kabaev References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> In-Reply-To: <20070519022016.2b4a6bda@kan.dnsalias.net> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: 130.133.86.198 X-Mailman-Approved-At: Wed, 08 Oct 2008 11:29:08 +0000 Cc: hackers@FreeBSD.ORG, current@FreeBSD.ORG Subject: Re: HEADS UP: GCC 4.2.0 is coming X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 08:18:35 -0000 Alexander Kabaev wrote: > On Fri, 18 May 2007 19:20:07 -0400 > Alexander Kabaev wrote: > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and >> plan to finish in a couple of hours after that. >> >> The src/ tree will be utterly broken meanwhile. I'll send an 'all >> clear' message when done. > > Done. > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not 4.2.1 as it is used in 7.X? Regards, O. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 09:01:57 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD7191065688 for ; Wed, 8 Oct 2008 09:01:57 +0000 (UTC) (envelope-from peter@wemm.org) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.171]) by mx1.freebsd.org (Postfix) with ESMTP id BFEEE8FC17 for ; Wed, 8 Oct 2008 09:01:57 +0000 (UTC) (envelope-from peter@wemm.org) Received: by wf-out-1314.google.com with SMTP id 24so3738262wfg.7 for ; Wed, 08 Oct 2008 02:01:57 -0700 (PDT) Received: by 10.142.84.3 with SMTP id h3mr3273215wfb.309.1223454982785; Wed, 08 Oct 2008 01:36:22 -0700 (PDT) Received: by 10.142.255.21 with HTTP; Wed, 8 Oct 2008 01:36:22 -0700 (PDT) Message-ID: Date: Wed, 8 Oct 2008 01:36:22 -0700 From: "Peter Wemm" To: "O. Hartmann" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Mailman-Approved-At: Wed, 08 Oct 2008 11:29:22 +0000 Cc: hackers@freebsd.org, current@freebsd.org Subject: TIME WARP! Re: HEADS UP: GCC 4.2.0 is coming X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 09:01:57 -0000 On Wed, Oct 8, 2008 at 12:57 AM, O. Hartmann wrote: > Alexander Kabaev wrote: >> >> On Fri, 18 May 2007 19:20:07 -0400 >> Alexander Kabaev wrote: >> >>> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and >>> plan to finish in a couple of hours after that. >>> >>> The src/ tree will be utterly broken meanwhile. I'll send an 'all >>> clear' message when done. >> >> Done. >> > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not 4.2.1 > as it is used in 7.X? > > Regards, > O. Sorry about that. I accidently revived a bunch of stuck email messages from our mailing list processing system. These messages from 2007 came back to life somehow. (Hint: Mailman's 'unshunt' command doesn't give a usage message) -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI6FJV "All of this is for nothing if we don't go to the stars" - JMS/B5 "If Java had true garbage collection, most programs would delete themselves upon execution." -- Robert Sewell From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 09:17:37 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F9061065687 for ; Wed, 8 Oct 2008 09:17:37 +0000 (UTC) (envelope-from vova@sw.ru) Received: from relay.sw.ru (mailhub.sw.ru [195.214.232.25]) by mx1.freebsd.org (Postfix) with ESMTP id DC3138FC2F for ; Wed, 8 Oct 2008 09:17:36 +0000 (UTC) (envelope-from vova@sw.ru) Received: from vbook.fbsd.ru ([10.30.1.111]) (authenticated bits=0) by relay.sw.ru (8.13.4/8.13.4) with ESMTP id m988e01j028671 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 8 Oct 2008 12:40:01 +0400 (MSD) Received: from vova by vbook.fbsd.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KnUaO-0005yu-0w; Wed, 08 Oct 2008 12:40:00 +0400 From: Vladimir Grebenschikov To: "O. Hartmann" In-Reply-To: <48EC67FF.3030900@zedat.fu-berlin.de> References: <20070518192007.6e6a91e1@kan.dnsalias.net> <20070519022016.2b4a6bda@kan.dnsalias.net> <48EC67FF.3030900@zedat.fu-berlin.de> Content-Type: text/plain Content-Transfer-Encoding: 7bit Date: Wed, 08 Oct 2008 12:39:59 +0400 Message-Id: <1223455199.1864.16.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.22.3.1 FreeBSD GNOME Team Port Sender: Vladimir Grebenschikov X-Mailman-Approved-At: Wed, 08 Oct 2008 11:29:32 +0000 Cc: hackers@FreeBSD.ORG, current@FreeBSD.ORG Subject: Re: HEADS UP: GCC 4.2.0 is coming X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 09:17:37 -0000 On Wed, 2008-10-08 at 07:57 +0000, O. Hartmann wrote: > Alexander Kabaev wrote: > > On Fri, 18 May 2007 19:20:07 -0400 -------^^^^^^^^^^^^^^^^^ > > Alexander Kabaev wrote: > > > >> HEADS UP: I will start importing GCC 4.2.0 bits in about one hour and > >> plan to finish in a couple of hours after that. > >> > >> The src/ tree will be utterly broken meanwhile. I'll send an 'all > >> clear' message when done. > > > > Done. > > > > Just for those who aren't on the cutting edge: why gcc 4.2.0 and not > 4.2.1 as it is used in 7.X? Looks like very outdated messages were delivered from spool. > Regards, > O. -- Vladimir B. Grebenschikov vova@fbsd.ru From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 13:50:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBBC51065686 for ; Wed, 8 Oct 2008 13:50:04 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 84DC98FC21 for ; Wed, 8 Oct 2008 13:50:04 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1KnZQQ-0004jB-Ib for freebsd-hackers@freebsd.org; Wed, 08 Oct 2008 13:50:02 +0000 Received: from utwig.xim.bz ([195.184.197.130]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Oct 2008 13:50:02 +0000 Received: from c.kworr by utwig.xim.bz with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Oct 2008 13:50:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Volodymyr Kostyrko Date: Wed, 08 Oct 2008 16:29:46 +0300 Lines: 10 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: utwig.xim.bz User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; ru-RU; rv:1.8.1.17) Gecko/20081001 SeaMonkey/1.1.12 Sender: news Subject: llvm/clang early test X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 13:50:04 -0000 Hi all. Does anyone tried to build world with clang (devel/llvm-devel)? I just have tested clang on some code from our tree, gzip and bzip2 for example. Well... it works. Gzip compiled with clang become faster, bzip2 don't... Right now I'm playing with world making it compile with clang. If anyone doing the same thing we can share some thoughts and patches... -- Sphinx of black quartz judge my vow. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 13:59:39 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22DA410656AF for ; Wed, 8 Oct 2008 13:59:39 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.186]) by mx1.freebsd.org (Postfix) with ESMTP id 9EFCF8FC20 for ; Wed, 8 Oct 2008 13:59:38 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so3366433fkk.11 for ; Wed, 08 Oct 2008 06:59:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=kdu0yn7EhCahdMpbBDZ4/UR6gMETUEbaawwbS5tb1uY=; b=UmwvU+mCMDAdJXg2WOvDOtuBkkaWecsNqLy1hXSu9RPv2Cp/emh//8GDYUpUTQfSrw PQnlNVypSrGvFfCGJY+I/d0crtBv6Qo3T57JSwhC/d2FPhKURzUu2l+0mR0pl7/LuPhi 7xfYHtwcWNR4gbK5mA7wL634y98AKerkLcGzU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=utxY8SS3D5Dma+UsXcQ01KVhI9i56O/F6F4rEiMT5ELVjGRY1FYoTUVtrmgL3F9stx Ive1ONy2vBNpBU4RM1mubLNtD6kiXZI5uZfJCI3DpGEp232SvhZ+HfigbxuR8uRvS1yL q2FXMymlc5ink8PNhvaRS5sEKhS0msvwNrpas= Received: by 10.180.221.13 with SMTP id t13mr6136973bkg.19.1223472699907; Wed, 08 Oct 2008 06:31:39 -0700 (PDT) Received: by 10.180.252.6 with HTTP; Wed, 8 Oct 2008 06:31:39 -0700 (PDT) Message-ID: <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> Date: Wed, 8 Oct 2008 20:31:39 +0700 From: "Outback Dingo" To: freebsd-hackers@freebsd.org, "Zaphod Beeblebrox" , Volker , "Evren Yurtesen" In-Reply-To: <200810081120.m98BK2fV043545@lurza.secnetix.de> MIME-Version: 1.0 References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 13:59:39 -0000 one answer... www.bakbone.com On Wed, Oct 8, 2008 at 6:20 PM, Oliver Fromme wrote= : > Dag-Erling Sm=F8rgrav wrote: > > "Zaphod Beeblebrox" writes: > > > "Dag-Erling Sm=F8rgrav" writes: > > > > What really annoys me with this thread is that nobody has provided > > > > any information at all that would allow someone to understand what > > > > needs to be done and estimate how hard it would be. > > > Well... I hinted that a hammer port would be sufficient (although th= ey > > > need to finish their replication design) and I hinted that the hamme= r > > > approach may be graftable to ZFS. Both reasonably large effort-wise > > > (but probably within the scope of a single developer with sufficient > > > time). > > > > No... you're so far off the mark it's not even funny, especially when > > it's been repeatedly pointed out to you. This is not a file system, > > it's a backup system. It's not designed to survive a disk crash or an > > accidental file deletion, it's designed to survive a direct missile > > strike on your colo center. > > > > To quote Wikipedia, "CDP is a service that captures changes to data to= a > > separate storage location" - emphasis on "separate". > > FWIW, the HAMMER file system _does_ support replication to > remote targets (thus "separate"). Unfortunately they call > this feature "mirroring", which is misleading at best. > It's really rather a replication mechanism, much like the > binlog of MySQL. It can be used for various purposes, > including live mirroring, delayed mirroring, archiving, > backup and point-in-time recovery. > > Well, of course, all of that doesn't help us at all because > HAMMER doesn't exist on FreeBSD. > > However, ZFS does exist on FreeBSD, and I think it wouldn't > be impossible to add similar features to ZFS. > > Another possibility would be to extend gjournal by adding > time stamps to journal transactions and a possibility to > feed the journal to a pipe, socket or whatever. And of > course a client-side implementation that does something > useful with the journal stream. This might even be a good > SoC project. > > Best regards > Oliver > > -- > Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. > Handelsregister: Registergericht Muenchen, HRA 74606, Gesch=E4ftsfuehrun= g: > secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M=FC= n- > chen, HRB 125758, Gesch=E4ftsf=FChrer: Maik Bachmann, Olaf Erb, Ralf Geb= hart > > FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd > > "File names are infinite in length, where infinity is set to 255 > characters." > -- Peter Collinson, "The Unix File System" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 14:09:55 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DCB31065696 for ; Wed, 8 Oct 2008 14:09:55 +0000 (UTC) (envelope-from rdivacky@lev.vlakno.cz) Received: from vlakno.cz (77-93-215-190.static.masterinter.net [77.93.215.190]) by mx1.freebsd.org (Postfix) with ESMTP id 194A98FC17 for ; Wed, 8 Oct 2008 14:09:55 +0000 (UTC) (envelope-from rdivacky@lev.vlakno.cz) Received: from localhost (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id 741FD9CB186; Wed, 8 Oct 2008 15:53:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at vlakno.cz Received: from vlakno.cz ([127.0.0.1]) by localhost (lev.vlakno.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D8X8u28+P5xU; Wed, 8 Oct 2008 15:53:39 +0200 (CEST) Received: from lev.vlakno.cz (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id 24B429CB74D; Wed, 8 Oct 2008 15:53:39 +0200 (CEST) Received: (from rdivacky@localhost) by lev.vlakno.cz (8.14.2/8.14.2/Submit) id m98DrdmK066850; Wed, 8 Oct 2008 15:53:39 +0200 (CEST) (envelope-from rdivacky) Date: Wed, 8 Oct 2008 15:53:39 +0200 From: Roman Divacky To: Volodymyr Kostyrko Message-ID: <20081008135339.GA66730@freebsd.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-hackers@freebsd.org Subject: Re: llvm/clang early test X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 14:09:55 -0000 On Wed, Oct 08, 2008 at 04:29:46PM +0300, Volodymyr Kostyrko wrote: > Hi all. > > Does anyone tried to build world with clang (devel/llvm-devel)? I just > have tested clang on some code from our tree, gzip and bzip2 for > example. Well... it works. Gzip compiled with clang become faster, bzip2 > don't... Right now I'm playing with world making it compile with clang. > If anyone doing the same thing we can share some thoughts and patches... check this patch: www.vlakno.cz/~rdivacky/clang.patch I am playing with it as well :) From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 16:45:48 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F2CA1065692 for ; Wed, 8 Oct 2008 16:45:48 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 37D6E8FC08 for ; Wed, 8 Oct 2008 16:45:47 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 47722207E; Wed, 8 Oct 2008 18:45:46 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 2F6C284483; Wed, 8 Oct 2008 18:45:46 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: "ushasri tummala" References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> Date: Wed, 08 Oct 2008 18:45:46 +0200 In-Reply-To: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> (ushasri tummala's message of "Tue, 7 Oct 2008 22:47:16 -0400") Message-ID: <86ljwzqblx.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 16:45:48 -0000 "ushasri tummala" writes: > What is the time between 2 mi_switches in freebsd? "it depends" I recommend _The Design and Implementation of the FreeBSD Operating Systems_ (http://www.amazon.com/dp/0201702452) or any good textbook on operating system design (e.g. http://www.amazon.com/dp/0136006639) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:14:50 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9ACA1106568F for ; Wed, 8 Oct 2008 17:14:50 +0000 (UTC) (envelope-from tummala.ushasri@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id AC5998FC0A for ; Wed, 8 Oct 2008 17:14:49 +0000 (UTC) (envelope-from tummala.ushasri@gmail.com) Received: by ug-out-1314.google.com with SMTP id m2so567521uge.39 for ; Wed, 08 Oct 2008 10:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=sbh9TECj36248orYAUDBVhLmwtUn1l6p3APE6WXSQE4=; b=FCBzX1gj3ZdOGYIRZDhSuD07x9awqK/L3fXCptghMd85wxrdmwQ2KmF5syDA43S0Hf q30hnT3IyfEtya4LADEdJ092+3RNvQuh32u/kh7Mpa0d7R1QTAPdoafgyR7T2R5tpCIu 09sNnEQI0dqCLWwHmH8elPYIUcwuXq21a9JBU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=GhFib+6KZJSqJZPF0C4DfGvpVEWalW8+yeMY+vtUK3fzBOmXQvC425Z3xrlYtdoNjU oJTcIG2npjQm4d72kGrmBw/2NAkAVFDMaCAc5etMP1XcFYuQl6Hp2gg11/Vj0ImtRQKm zL+TvnzAkKS2Ds+SL0y+6vRtHB3u8a7S/MzAw= Received: by 10.86.1.11 with SMTP id 11mr7805090fga.27.1223486085438; Wed, 08 Oct 2008 10:14:45 -0700 (PDT) Received: by 10.86.27.18 with HTTP; Wed, 8 Oct 2008 10:14:45 -0700 (PDT) Message-ID: <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> Date: Wed, 8 Oct 2008 13:14:45 -0400 From: "ushasri tummala" To: "=?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?=" , freebsd-hackers@freebsd.org In-Reply-To: <86ljwzqblx.fsf@ds4.des.no> MIME-Version: 1.0 References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:14:50 -0000 Thank You Des for your reply. I am using Design and Implementation of the FreeBSD Operating Systems. I just want to know how its(time between 2 mi_switch()) calculated and in which variable is it stored in the code.(FreeBSD 5.2 release) This is not addressed in text book. Thank You, Usha. On Wed, Oct 8, 2008 at 12:45 PM, Dag-Erling Sm=F8rgrav wrote: > "ushasri tummala" writes: > > What is the time between 2 mi_switches in freebsd? > > "it depends" > > I recommend _The Design and Implementation of the FreeBSD Operating > Systems_ (http://www.amazon.com/dp/0201702452) or any good textbook on > operating system design (e.g. http://www.amazon.com/dp/0136006639) > > DES > -- > Dag-Erling Sm=F8rgrav - des@des.no > From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:15:26 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 709F4106568D for ; Wed, 8 Oct 2008 17:15:26 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-gx0-f21.google.com (mail-gx0-f21.google.com [209.85.217.21]) by mx1.freebsd.org (Postfix) with ESMTP id 076328FC14 for ; Wed, 8 Oct 2008 17:15:25 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by gxk14 with SMTP id 14so8244102gxk.19 for ; Wed, 08 Oct 2008 10:15:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=yy4EaRJhIOVySRlTgUeGSvfaLlKa0tHgfM3Sx6l4qsY=; b=sB6gFAnmev9FsN8R/I/Dx5HQTWze0aSpDaXB05+Ge70KEO3tB1TJkU0T7Iki8TDI8s AblGklTfZExjTZieFbiIfrhUBg0klNlQG+MV4ZTKGGk1/lI7CiyYGahEc3fxkfKQD5EF HLPdlcDxZ86yS3eRzgPD4iagASXJLucnhHQSY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=nxiM7kyMy9yiRXV72ALTcslTwwaad4FIrESfzzxwixVWupUeURYdKEKT4bHgz9xxPe QFZxJx5icVC9ZZPpqNeJp0jpzDu03+KSsJcFvESOHdKOqEpE349IdgsvmQiIvHVDWqLf gYdxVCVlDSriw7J2UPFei+4vVxRO770FBRGCE= Received: by 10.150.177.20 with SMTP id z20mr11873433ybe.192.1223486123179; Wed, 08 Oct 2008 10:15:23 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Wed, 8 Oct 2008 10:15:23 -0700 (PDT) Message-ID: <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> Date: Wed, 8 Oct 2008 13:15:23 -0400 From: "Zaphod Beeblebrox" To: "=?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?=" In-Reply-To: <86iqs3sdtp.fsf@ds4.des.no> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:15:26 -0000 On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Sm=F8rgrav wrote: > "Zaphod Beeblebrox" writes: > > "Dag-Erling Sm=F8rgrav" writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:19:27 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 382CC1065686 for ; Wed, 8 Oct 2008 17:19:27 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from hs-out-0708.google.com (hs-out-0708.google.com [64.233.178.241]) by mx1.freebsd.org (Postfix) with ESMTP id E395F8FC0C for ; Wed, 8 Oct 2008 17:19:26 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by hs-out-0708.google.com with SMTP id 54so1404986hsz.11 for ; Wed, 08 Oct 2008 10:19:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=ZXzXCa0xnWRI50arP3UMVvvMjEVRh3IjBZD7by/2rSg=; b=Bdq96wcfocdCxRGk4/rXyZFtCdzkQj3XmucafLtSCu6GhRrumvBfJJgqV0xnbFo2Fn AOFVYJGjzW5gkPZOAuwfL7jfakJY3Wuk91BDR4zlu1QnZHaxjqzEoPuXoDOAGGmdWapJ PAUNpMGIXGZIJN2mQoElm15Vb1ym2K8gqoAIk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=ouIi7iT5Yj9HtEA3gzXAhZi8JjTVZpEmTz+zOH6wRa0b80NODEqGMPIUpDNxhSAegt eC9uolM8pl1O4QdTuEXasjaF53K9ozKxLqf2AcnT6cdWu7cTBbcCHEl8RPQxBUqO1KgF w4kAr2ksDUgotwDGFIG/oxtdORMAgg/4tbabs= Received: by 10.231.36.12 with SMTP id r12mr446505ibd.15.1223486365452; Wed, 08 Oct 2008 10:19:25 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Wed, 8 Oct 2008 10:19:25 -0700 (PDT) Message-ID: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> Date: Wed, 8 Oct 2008 13:19:25 -0400 From: "Zaphod Beeblebrox" To: freebsd-hackers@freebsd.org, "Zaphod Beeblebrox" , Volker , "Evren Yurtesen" In-Reply-To: <200810081120.m98BK2fV043545@lurza.secnetix.de> MIME-Version: 1.0 References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:19:27 -0000 On Wed, Oct 8, 2008 at 7:20 AM, Oliver Fromme wrote= : > Dag-Erling Sm=F8rgrav wrote: > > FWIW, the HAMMER file system _does_ support replication to > remote targets (thus "separate"). Unfortunately they call > this feature "mirroring", which is misleading at best. > It's really rather a replication mechanism, much like the > binlog of MySQL. It can be used for various purposes, > including live mirroring, delayed mirroring, archiving, > backup and point-in-time recovery [thank-you for repeating that, BTW] > However, ZFS does exist on FreeBSD, and I think it wouldn't > be impossible to add similar features to ZFS. Possibly even as a ZFS module? This might be something better addressed a= t the ZFS project level --- but the next question is: does FreeBSD support ZF= S modules? > Another possibility would be to extend gjournal by adding > time stamps to journal transactions and a possibility to > feed the journal to a pipe, socket or whatever. And of > course a client-side implementation that does something > useful with the journal stream. This might even be a good > SoC project. Now this interests me. Firstly, I thought that gjournal might only be responsible for the meta-data (but I'm happy to be wrong on this point). Secondly, is it a) sufficient and b) efficient to attempt to time-travel UF= S with the gjournal log? From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:49:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D1FA106569D for ; Wed, 8 Oct 2008 17:49:59 +0000 (UTC) (envelope-from asmodai@in-nomine.org) Received: from nexus.in-nomine.org (dhammapada.xs4all.nl [82.95.168.248]) by mx1.freebsd.org (Postfix) with ESMTP id C740D8FC24 for ; Wed, 8 Oct 2008 17:49:58 +0000 (UTC) (envelope-from asmodai@in-nomine.org) Received: from localhost (localhost.domini.in-nomine.org [127.0.0.1]) by nexus.in-nomine.org (Postfix) with ESMTP id 500DEF3CE; Wed, 8 Oct 2008 19:49:57 +0200 (CEST) X-Virus-Scanned: by amavisd-new using ClamAV at in-nomine.org Received: from nexus.in-nomine.org ([127.0.0.1]) by localhost (nexus.domini.in-nomine.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EbQMYghRO3zJ; Wed, 8 Oct 2008 19:49:56 +0200 (CEST) Received: by nexus.in-nomine.org (Postfix, from userid 1000) id 2F2B5F3CC; Wed, 8 Oct 2008 19:49:56 +0200 (CEST) Date: Wed, 8 Oct 2008 19:49:56 +0200 From: Jeroen Ruigrok van der Werven To: ushasri tummala Message-ID: <20081008174956.GA98121@nexus.in-nomine.org> References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> Organisation: Ninth Circle Enterprises User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Dag-Erling =?iso-8859-15?Q?Sm=F8rgrav?= , freebsd-hackers@freebsd.org Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:49:59 -0000 -On [20081008 19:15], ushasri tummala (tummala.ushasri@gmail.com) wrote: >I just want to know how its(time between 2 mi_switch()) calculated and in >which variable is it stored in the code.(FreeBSD 5.2 release) >This is not addressed in text book. What Dag-Erling meant to say, and if I recall correctly, a switch() is highly dependent on your hardware. So the time taken for a specific machine can be vastly different from another machine. -- Jeroen Ruigrok van der Werven / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Lead us to the place, guide us with your grace, to a place where we'll be safe... From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:31:28 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56C601065697 for ; Wed, 8 Oct 2008 17:31:28 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id DE61E8FC12 for ; Wed, 8 Oct 2008 17:31:26 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 29395 invoked by uid 399); 8 Oct 2008 16:45:00 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 8 Oct 2008 16:45:00 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48ECE382.4060907@ispro.net> Date: Wed, 08 Oct 2008 19:44:50 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Shaun Amott References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> In-Reply-To: <20081007155059.GA20615@charon.picobyte.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 08 Oct 2008 18:00:08 +0000 Cc: Sean Bruno , freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:31:28 -0000 Shaun Amott wrote: > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: >> so FreeBSD could be supported also. As you can imagine, it is not only >> important that data can be restored when a box hardware failure etc. it is >> also important that data can be restored if deleted by accidents etc. While >> traditional backup programs provide this functionality, you cant really go >> back to 10 min or 1h ago, often they take daily backups and have to scan >> whole filesystem for changed files every time the backup is taken which >> stresses out the systems. >> > > This can (more or less) be achieved with snapshots: you can cheaply > maintain old versions of the file system, and mount an old snapshot at > any time. Hourly is about as fine-grained as you can expect though. > The documentation says one cant do more than 20 snapshots. http://www.freebsd.org/doc/en/books/handbook/snapshots.html Although 20 could be enough combined with a normal backup program. As far as I understand creating snapshots will consume disk space, and freeze the disk writes for a certain amount of time, every time (I read 5 seconds for 8GB system http://www.wave2.org/2007/10/08/mysql-snapshots-on-freebsd/ ) snapshot is created. The snapshots are stored in the local filesystem and it would require manually transferring the data to a remote machine. More importantly, as far as I understand, if the hard drive totally fails, there would be no way to restore a snapshot anymore unless we have a dump of the whole filesystem and first restore it and make sure everything is exactly at the right blocks in the drive. No? Although this probably could be worked out. In my opinion it requires a lot of work, Bbt thanks for the advice. Just that I would rather pay a small amount of fee and use Linux and use a continous backup software which works as easy as install and run. Which also provides utilities for easily recovering files or the whole filesystem or disk. Thanks again for pointing out snapshots. It is more or less suitable :) Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 17:43:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89BD21065696 for ; Wed, 8 Oct 2008 17:43:29 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 7C2628FC08 for ; Wed, 8 Oct 2008 17:43:28 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 26837 invoked by uid 399); 8 Oct 2008 17:43:44 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 8 Oct 2008 17:43:44 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48ECF144.305@ispro.net> Date: Wed, 08 Oct 2008 20:43:32 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Outback Dingo References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> In-Reply-To: <5635aa0d0810080631s41a43444hbfd1eed11f48f6c2@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 08 Oct 2008 18:05:48 +0000 Cc: Volker , freebsd-hackers@freebsd.org, Zaphod Beeblebrox Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 17:43:29 -0000 Outback Dingo wrote: > one answer... www.bakbone.com > Unfortunately if you check their compatibility matrix you can see that I have to use Linux to be able to do CDP :) http://www.bakbone.com/docs/NetVault_Backup_Supported_Platforms_October_2008.pdf or am I reading this wrong? Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 18:01:05 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D68501065689 for ; Wed, 8 Oct 2008 18:01:05 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 7BE518FC19 for ; Wed, 8 Oct 2008 18:01:04 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 45785 invoked by uid 399); 8 Oct 2008 18:01:19 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 8 Oct 2008 18:01:19 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48ECF564.7000204@ispro.net> Date: Wed, 08 Oct 2008 21:01:08 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> In-Reply-To: <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Wed, 08 Oct 2008 18:06:24 +0000 Cc: Volker , =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 18:01:05 -0000 Zaphod Beeblebrox wrote: > On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Smørgrav > wrote: > > "Zaphod Beeblebrox" > > writes: > > "Dag-Erling Smørgrav" > writes: > > > What really annoys me with this thread is that nobody has provided > > > any information at all that would allow someone to understand what > > > needs to be done and estimate how hard it would be. > > Well... I hinted that a hammer port would be sufficient (although > they > > need to finish their replication design) and I hinted that the hammer > > approach may be graftable to ZFS. Both reasonably large effort-wise > > (but probably within the scope of a single developer with sufficient > > time). > > No... you're so far off the mark it's not even funny, especially when > it's been repeatedly pointed out to you. This is not a file system, > it's a backup system. It's not designed to survive a disk crash or an > accidental file deletion, it's designed to survive a direct missile > strike on your colo center. > > To quote Wikipedia, "CDP is a service that captures changes to data to a > separate storage location" - emphasis on "separate". > > > Wow... thanks for the flame, but there's no reason that the device that > is receiving the hammer replication couldn't be on the other side of the > globe and there's no reason it couldn't be considered a backup. Part of > the advantage of the structure that allows you to efficiently select for > new changes allows you to do the same kind of *backup* as they claim. > Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 18:34:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 595F4106569C; Wed, 8 Oct 2008 18:34:14 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from euclid.ucsd.edu (euclid.ucsd.edu [132.239.145.52]) by mx1.freebsd.org (Postfix) with ESMTP id 74EA68FC22; Wed, 8 Oct 2008 18:33:42 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from zeno.ucsd.edu (zeno.ucsd.edu [132.239.145.22]) by euclid.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m98ICF611073; Wed, 8 Oct 2008 11:12:15 -0700 (PDT) Received: from localhost (neldredg@localhost) by zeno.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m98ICFc23754; Wed, 8 Oct 2008 11:12:15 -0700 (PDT) X-Authentication-Warning: zeno.ucsd.edu: neldredg owned process doing -bs Date: Wed, 8 Oct 2008 11:12:15 -0700 (PDT) From: Nate Eldredge X-X-Sender: neldredg@zeno.ucsd.edu To: Evren Yurtesen In-Reply-To: <48ECE382.4060907@ispro.net> Message-ID: References: <48E9E1BB.6020908@ispro.net> <48EA3451.7040801@miralink.com> <48EA83CE.4060702@ispro.net> <20081007155059.GA20615@charon.picobyte.net> <48ECE382.4060907@ispro.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sean Bruno , freebsd-hackers@freebsd.org, Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 18:34:14 -0000 On Wed, 8 Oct 2008, Evren Yurtesen wrote: > Thanks again for pointing out snapshots. It is more or less suitable :) I'll just warn you that if you're planning to use snapshots for your own purposes, to first do an extensive stress test on a non-critical machine with backed up data. I've had a lot of problems with snapshots occasionally causing deadlocks which hang the machine. This was under 6.x but I had the same problem under many previous versions, so I don't necessarily expect that it's fixed. Also, while it's never happened to me, I've heard other people report data corruption. -- Nate Eldredge neldredge@math.ucsd.edu From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 19:03:55 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E43A106568C for ; Wed, 8 Oct 2008 19:03:55 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 3E1108FC08 for ; Wed, 8 Oct 2008 19:03:55 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 264C25B4C; Wed, 8 Oct 2008 12:03:54 -0700 (PDT) To: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= In-reply-to: Your message of "Wed, 08 Oct 2008 10:18:47 +0200." <86ej2rsdnc.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> Comments: In-reply-to =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= message dated "Wed, 08 Oct 2008 10:18:47 +0200." Date: Wed, 08 Oct 2008 12:03:54 -0700 From: Bakul Shah Message-Id: <20081008190354.264C25B4C@mail.bitblocks.com> Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:03:55 -0000 On Wed, 08 Oct 2008 10:18:47 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Bakul Shah writes: > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > What really annoys me with this thread is that nobody has provided any > > > information at all that would allow someone to understand what needs to > > > be done and estimate how hard it would be. > > From their http://forum.r1soft.com/CDP.html page: [...] > > You completely missed the mark. I know what R1Soft's product is. What > I want to know is what needs to be done to port it to FreeBSD. Sorry, my mindreader is broken. If this is what you really wanted to know, why not ask R1Soft? -hackers is not going to shed any light on the specifics of R1Soft's product. I replied to say that implementing a similar solution is not hard (I am sure you knew that too but I wasn't responding just to you). It may even be worth doing. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 19:11:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AF1E106568A for ; Wed, 8 Oct 2008 19:11:11 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id D0EBA8FC12 for ; Wed, 8 Oct 2008 19:11:10 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m98JB4tU057790; Wed, 8 Oct 2008 15:11:04 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Wed, 8 Oct 2008 14:06:14 -0400 User-Agent: KMail/1.9.7 References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> In-Reply-To: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810081406.14916.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Wed, 08 Oct 2008 15:11:05 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/8395/Wed Oct 8 12:44:51 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.4 required=4.2 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: ushasri tummala Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:11:11 -0000 On Tuesday 07 October 2008 10:47:16 pm ushasri tummala wrote: > What is the time between 2 mi_switches in freebsd?In which variable is this > information stored? > > Could you plz help me out with this There isn't a fixed timeslice due to preemption for interrupts, etc. However, the default time slice is available as the kern.sched.quantum sysctl: % sysctl -d kern.sched.quantum kern.sched.quantum: Roundrobin scheduling quantum in microseconds In the kernel it is stored as a static (private) variable in the scheduler implementation. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 19:35:14 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5D0D106568A for ; Wed, 8 Oct 2008 19:35:14 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.230]) by mx1.freebsd.org (Postfix) with ESMTP id 55BFC8FC0A for ; Wed, 8 Oct 2008 19:35:14 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by wr-out-0506.google.com with SMTP id c8so784035wra.27 for ; Wed, 08 Oct 2008 12:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=KoccRp5fL6mHHfy4GvIz2H9QmuAzc1rFwcX9oNNBSN8=; b=pbwnnd5ubag3bl3RkttDkqkNAWwko/ofs4IS+xT2YfpzZ6tDjeuHgQn0eTAa2k46mN wNeIY4+GU0L1wOzSZGdhPxoDKXp/1v6vcAdTS0wyCziIM+lD099ZbC9EMQjkaPPrq/4X luIz5CfgWvk2N2xPx2Oe5V3uo3OO3jQE54wqo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=WjxxgHsUB+/bi5yktmk0AsgfJa+0byW7/vHXgWh4k8eEfv0keu4jukQcSFkuGTsAgD avOOGnemzm6Wo393LBhkme0YxUl79gOYvrPuv35PfZE2l9JFz/g4AZquxnnVS3wKuQqj GREtRw/i5rvWAYknQXiqGPmp4vcjw/kgYvTKY= Received: by 10.231.38.67 with SMTP id a3mr456227ibe.29.1223494512533; Wed, 08 Oct 2008 12:35:12 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Wed, 8 Oct 2008 12:35:12 -0700 (PDT) Message-ID: <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> Date: Wed, 8 Oct 2008 15:35:12 -0400 From: "Zaphod Beeblebrox" To: "Evren Yurtesen" In-Reply-To: <48ECF564.7000204@ispro.net> MIME-Version: 1.0 References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Volker , =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:35:14 -0000 On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen wrote: > Zaphod Beeblebrox wrote: > >> >> Wow... thanks for the flame, but there's no reason that the device that >> is receiving the hammer replication couldn't be on the other side of the >> globe and there's no reason it couldn't be considered a backup. Part of the >> advantage of the structure that allows you to efficiently select for new >> changes allows you to do the same kind of *backup* as they claim. >> >> > Wouldnt that device need to keep the whole filesystem? Like if you have 10 > machines with 10x 1GB drives (lets say each used about 250gb), you will need > 10TB disk space in the backup server? > Urm... I think everything we've been discussing here backs up the whole filesystem (it would be near impossible for a block-oriented system to do elsewise). I suppose you could do something with the archive bit or dump bits with a filesystem based backup. But anyways... in a filesystem based replication system, you'd need enough space to store the data and the history of the data. The sum of the history of the data could even exceed the size of the sum of the input disks. It could also be much smaller. It really depends on how much you change. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 19:46:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48AA4106568B for ; Wed, 8 Oct 2008 19:46:14 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 021D08FC19 for ; Wed, 8 Oct 2008 19:46:13 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 192132049; Wed, 8 Oct 2008 21:46:13 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 02515844C1; Wed, 8 Oct 2008 21:46:13 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Jeroen Ruigrok van der Werven References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <86ljwzqblx.fsf@ds4.des.no> <53fa490b0810081014k7c6c3ddfs8f43526da72d881@mail.gmail.com> <20081008174956.GA98121@nexus.in-nomine.org> Date: Wed, 08 Oct 2008 21:46:12 +0200 In-Reply-To: <20081008174956.GA98121@nexus.in-nomine.org> (Jeroen Ruigrok van der Werven's message of "Wed, 8 Oct 2008 19:49:56 +0200") Message-ID: <86zlleq397.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, ushasri tummala Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:46:14 -0000 Jeroen Ruigrok van der Werven writes: > -On [20081008 19:15], ushasri tummala (tummala.ushasri@gmail.com) wrote: > > I just want to know how its(time between 2 mi_switch()) calculated > > and in which variable is it stored in the code.(FreeBSD 5.2 release) > > This is not addressed in text book. > What Dag-Erling meant to say, and if I recall correctly, a switch() is > highly dependent on your hardware. So the time taken for a specific > machine can be vastly different from another machine. No, no, no. Assuming the question is really "what is the time between two task switches", A task switch can happen for one of many reasons: - first, and simplest, the current task has used up its quantum; - the current task is waiting for an external event (I/O, a mutex, a timeout, etc.) - the current task has terminated; - something happened to make a higher-priority task runnable; - ... The closest you can get to a hard answer is if you consider only the first of the above, in which case the answer is 1/hz second, where "hz" is literally a kernel variable named hz. Its default value is 1,000 on amd64, i386, ia64 and sparc64, and 100 on all other platforms. (actually, it's more complicated than that, because this default value is a preprocessor macro called HZ which you can redefine in your kernel config, and you can also set hz at boot time using the kern.hz loader tunable) A nice little project for someone with a lot of time on their hands would be to make the FreeBSD kernel tickless, thus (to oversimplify) eliminating hz. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 19:47:40 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F345A1065686 for ; Wed, 8 Oct 2008 19:47:40 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id B09458FC22 for ; Wed, 8 Oct 2008 19:47:40 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 5CA8E2049; Wed, 8 Oct 2008 21:47:39 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 482B2844C1; Wed, 8 Oct 2008 21:47:39 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Evren Yurtesen References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <48ECE92D.7050904@ispro.net> Date: Wed, 08 Oct 2008 21:47:39 +0200 In-Reply-To: <48ECE92D.7050904@ispro.net> (Evren Yurtesen's message of "Wed, 08 Oct 2008 20:09:01 +0300") Message-ID: <86vdw2q36s.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:47:41 -0000 Evren Yurtesen writes: > It would perhaps be too much to ask but can you drop a line to them > and ask or is it ok if I give your email address to them and they can > contact you? Feel free. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 20:03:39 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB3101065686 for ; Wed, 8 Oct 2008 20:03:39 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 631FF8FC1F for ; Wed, 8 Oct 2008 20:03:39 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 9121E2049; Wed, 8 Oct 2008 22:03:38 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 6D7B9844ED; Wed, 8 Oct 2008 22:03:38 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Evren Yurtesen References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> Date: Wed, 08 Oct 2008 22:03:38 +0200 In-Reply-To: <48ECF564.7000204@ispro.net> (Evren Yurtesen's message of "Wed, 08 Oct 2008 21:01:08 +0300") Message-ID: <86r66qq2g5.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , Zaphod Beeblebrox , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:03:39 -0000 Evren Yurtesen writes: > Wouldnt that device need to keep the whole filesystem? Like if you > have 10 machines with 10x 1GB drives (lets say each used about 250gb), > you will need 10TB disk space in the backup server? Yes and no and yes... It stores *changes* to the file systems, so how much space it needs depends on how full your file systems are, how often and how much you write to them, etc. Also, at every recovery point, you can discard all but the last change since the previous recovery point for every changed block. FWIW, the exact same answer applies to pretty much any backup solution that supports incremental backups. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 20:19:50 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12877106568B for ; Wed, 8 Oct 2008 20:19:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id C2BFE8FC0A for ; Wed, 8 Oct 2008 20:19:49 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id A1C2F207E; Wed, 8 Oct 2008 22:19:48 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 8B93F844ED; Wed, 8 Oct 2008 22:19:48 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Bakul Shah References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> Date: Wed, 08 Oct 2008 22:19:48 +0200 In-Reply-To: <20081008190354.264C25B4C@mail.bitblocks.com> (Bakul Shah's message of "Wed, 08 Oct 2008 12:03:54 -0700") Message-ID: <86prma3km3.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , Evren Yurtesen , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:19:50 -0000 Bakul Shah writes: > "Dag-Erling Sm=C3=B8rgrav" writes: > > Bakul Shah writes: > > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > > What really annoys me with this thread is that nobody has > > > > provided any information at all that would allow someone to > > > > understand what needs to be done and estimate how hard it would > > > > be. > > > From their http://forum.r1soft.com/CDP.html page: [...] > > You completely missed the mark. I know what R1Soft's product is. > > What I want to know is what needs to be done to port it to FreeBSD. > Sorry, my mindreader is broken. If this is what you really wanted to > know, why not ask R1Soft? -hackers is not going to shed any light on > the specifics of R1Soft's product. I replied to say that implementing > a similar solution is not hard (I am sure you knew that too but I > wasn't responding just to you). It may even be worth doing. I didn't actually ask a question, and I don't mind that you don't have the answer. What I do mind is that you interpreted my statement of frustration as a question, and provided a completely irrelevant answer. You don't need to read minds to understand this, just English. If you take a step back and go through and read the entire thread again from the start, though, I think you will understand my frustration. Evren asked a question which everybody else is doing their best not to answer in as many words as possible; and when I try to answer, I find out that Evren doesn't really know what the question is. This is where - in an ideal world - somebody at R1Soft would jump in and start asking the right questions... DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 20:20:19 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17B461065686 for ; Wed, 8 Oct 2008 20:20:19 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C34178FC13 for ; Wed, 8 Oct 2008 20:20:18 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KnfW0-00058H-LZ for freebsd-hackers@freebsd.org; Wed, 08 Oct 2008 20:20:12 +0000 Received: from 78-0-85-121.adsl.net.t-com.hr ([78.0.85.121]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Oct 2008 20:20:12 +0000 Received: from ivoras by 78-0-85-121.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Oct 2008 20:20:12 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Wed, 08 Oct 2008 22:20:00 +0200 Lines: 45 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigF0FB422781C3B996F373DECF" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-85-121.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) X-Enigmail-Version: 0.95.7 Sender: news Subject: strftime's %c warning? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:20:19 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigF0FB422781C3B996F373DECF Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm trying to use the %c formatter in strftime(3), documented as: " %c is replaced by national representation of time and date. " =2E.. which looks useful, except that in code in which WFORMAT is defined= as "1" I get this error: str.c: In function 'ltime': str.c:141: warning: '%c' yields only last 2 digits of year in some locales on non-BSD systems *** Error code 1 Since the code I'm developing is definitely BSD-only (patch to pkg_* infrastructure), should I: a) stop using locale-based %c and choose my own date/time format b) remove WFORMAT from the Makefile? The same warning/error is generated by %x and %X, and %+ described in the strftime man page isn't recognized. --------------enigF0FB422781C3B996F373DECF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjtFfAACgkQldnAQVacBcjElACfU3N++OdqFXu1aoK237uoj1Xf YpQAnRppy9+aTnom7Zv0m1lBjUCkX8Yp =RpYV -----END PGP SIGNATURE----- --------------enigF0FB422781C3B996F373DECF-- From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 20:20:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 972C1106568D for ; Wed, 8 Oct 2008 20:20:54 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 548908FC0C for ; Wed, 8 Oct 2008 20:20:54 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 9150F2049; Wed, 8 Oct 2008 22:20:53 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 80263844ED; Wed, 8 Oct 2008 22:20:53 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: "Zaphod Beeblebrox" References: <86iqs3sdtp.fsf@ds4.des.no> <200810081120.m98BK2fV043545@lurza.secnetix.de> <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> Date: Wed, 08 Oct 2008 22:20:53 +0200 In-Reply-To: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> (Zaphod Beeblebrox's message of "Wed, 8 Oct 2008 13:19:25 -0400") Message-ID: <86ljwy3kka.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Volker , freebsd-hackers@freebsd.org, Evren Yurtesen Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:20:54 -0000 "Zaphod Beeblebrox" writes: >> Dag-Erling Sm=C3=B8rgrav wrote: >> >> FWIW, the HAMMER file system _does_ support replication [...] No, actually, I didn't write that. You need to learn to quote properly. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 20:23:45 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08E771065687 for ; Wed, 8 Oct 2008 20:23:45 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id B43D28FC0C for ; Wed, 8 Oct 2008 20:23:44 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id E7361208A; Wed, 8 Oct 2008 22:23:43 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id D26C2844ED; Wed, 8 Oct 2008 22:23:43 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: John Baldwin References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <200810081406.14916.jhb@freebsd.org> Date: Wed, 08 Oct 2008 22:23:43 +0200 In-Reply-To: <200810081406.14916.jhb@freebsd.org> (John Baldwin's message of "Wed, 8 Oct 2008 14:06:14 -0400") Message-ID: <86hc7m3kfk.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, ushasri tummala Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:23:45 -0000 John Baldwin writes: > There isn't a fixed timeslice due to preemption for interrupts, etc. How= ever,=20 > the default time slice is available as the kern.sched.quantum sysctl: > > % sysctl -d kern.sched.quantum > kern.sched.quantum: Roundrobin scheduling quantum in microseconds des@ds4 ~% uname -a FreeBSD ds4.des.no 8.0-CURRENT FreeBSD 8.0-CURRENT #48 r183493M: Tue Sep 30= 23:35:48 CEST 2008 des@ds4.des.no:/usr/obj/usr/src/sys/ds4 amd64 des@ds4 ~% sysctl kern.sched.quantum sysctl: unknown oid 'kern.sched.quantum' DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 21:02:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9E4F10656B1 for ; Wed, 8 Oct 2008 21:02:22 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 789B78FC0C for ; Wed, 8 Oct 2008 21:02:22 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m98L20sv058715; Wed, 8 Oct 2008 17:02:07 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: Dag-Erling =?utf-8?q?Sm=C3=B8rgrav?= Date: Wed, 8 Oct 2008 17:01:44 -0400 User-Agent: KMail/1.9.7 References: <53fa490b0810071947j23fc0f72n5360b6f174ddc96d@mail.gmail.com> <200810081406.14916.jhb@freebsd.org> <86hc7m3kfk.fsf@ds4.des.no> In-Reply-To: <86hc7m3kfk.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200810081701.45199.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Wed, 08 Oct 2008 17:02:07 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/8396/Wed Oct 8 14:48:47 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-hackers@freebsd.org, ushasri tummala Subject: Re: What is the time between 2 mi_switches in freebsd. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 21:02:23 -0000 On Wednesday 08 October 2008 04:23:43 pm Dag-Erling Sm=C3=B8rgrav wrote: > John Baldwin writes: > > There isn't a fixed timeslice due to preemption for interrupts, etc. =20 However,=20 > > the default time slice is available as the kern.sched.quantum sysctl: > > > > % sysctl -d kern.sched.quantum > > kern.sched.quantum: Roundrobin scheduling quantum in microseconds >=20 > des@ds4 ~% uname -a > FreeBSD ds4.des.no 8.0-CURRENT FreeBSD 8.0-CURRENT #48 r183493M: Tue Sep = 30=20 23:35:48 CEST 2008 des@ds4.des.no:/usr/obj/usr/src/sys/ds4 amd64 > des@ds4 ~% sysctl kern.sched.quantum > sysctl: unknown oid 'kern.sched.quantum' Apparently the quantum isn't exposed anywhere for ULE. =2D-=20 John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 21:30:55 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D9861065687 for ; Wed, 8 Oct 2008 21:30:55 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 428A08FC20 for ; Wed, 8 Oct 2008 21:30:55 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 5AF645B4C; Wed, 8 Oct 2008 14:30:54 -0700 (PDT) To: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= In-reply-to: Your message of "Wed, 08 Oct 2008 22:19:48 +0200." <86prma3km3.fsf@ds4.des.no> References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> <86prma3km3.fsf@ds4.des.no> Comments: In-reply-to =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= message dated "Wed, 08 Oct 2008 22:19:48 +0200." Date: Wed, 08 Oct 2008 14:30:54 -0700 From: Bakul Shah Message-Id: <20081008213054.5AF645B4C@mail.bitblocks.com> Cc: Evren Yurtesen , Volker , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 21:30:55 -0000 On Wed, 08 Oct 2008 22:19:48 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wrote: > Bakul Shah writes: > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > Bakul Shah writes: > > > > "Dag-Erling Sm=C3=B8rgrav" writes: > > > > > What really annoys me with this thread is that nobody has > > > > > provided any information at all that would allow someone to > > > > > understand what needs to be done and estimate how hard it would > > > > > be. > > > > From their http://forum.r1soft.com/CDP.html page: [...] > > > You completely missed the mark. I know what R1Soft's product is. > > > What I want to know is what needs to be done to port it to FreeBSD. > > Sorry, my mindreader is broken. If this is what you really wanted to > > know, why not ask R1Soft? -hackers is not going to shed any light on > > the specifics of R1Soft's product. I replied to say that implementing > > a similar solution is not hard (I am sure you knew that too but I > > wasn't responding just to you). It may even be worth doing. > > I didn't actually ask a question, and I don't mind that you don't have > the answer. What I do mind is that you interpreted my statement of > frustration as a question, and provided a completely irrelevant answer. > You don't need to read minds to understand this, just English. Interpreting an expression of frustration as a request for a solution is a common engineering trait:-) I can see you may not prefer my interpretation but I can't understand why you "mind" it. But so be it. I do not wish to annoy you. > If you take a step back and go through and read the entire thread again > from the start, though, I think you will understand my frustration. I understand your frustration but I chose to instead focus on the technical part. I too can get frustrated in similar situations but every time that happens I can trace it back to my own stress. I can't really control what others say so the way I deal with it is to ignore it or joke about it. I do try to clear misunderstandings but people don't always understand my point of view! As for feeling frustrated, I now view that as a warning signal. > Evren asked a question which everybody else is doing their best not to > answer in as many words as possible; and when I try to answer, I find > out that Evren doesn't really know what the question is. Actually that was clear in his very first email. > This is where - in an ideal world - somebody at R1Soft would jump in and > start asking the right questions... Don't bet on it. My musings even makes me wonder if they do it right. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 21:33:22 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E881106568D for ; Wed, 8 Oct 2008 21:33:22 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 08A9E8FC12 for ; Wed, 8 Oct 2008 21:33:21 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id AEC065B4C for ; Wed, 8 Oct 2008 14:33:21 -0700 (PDT) To: hackers@freebsd.org In-reply-to: Your message of "Wed, 08 Oct 2008 14:30:54 PDT." Comments: In-reply-to Bakul Shah message dated "Wed, 08 Oct 2008 14:30:54 -0700." Date: Wed, 08 Oct 2008 14:33:21 -0700 From: Bakul Shah Message-Id: <20081008213321.AEC065B4C@mail.bitblocks.com> Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 21:33:22 -0000 Sorry about that. Didn't mean to continue this discussion in -hackers but forgot to remove the cc list. From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 8 23:28:19 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E47F7106568D for ; Wed, 8 Oct 2008 23:28:19 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 688198FC15 for ; Wed, 8 Oct 2008 23:28:18 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 90987 invoked by uid 399); 8 Oct 2008 23:28:35 -0000 Received: from localhost (127.0.0.1) by localhost with ESMTPM; 8 Oct 2008 23:28:35 -0000 X-Originating-IP: 127.0.0.1 Received: from a91-152-230-218.elisa-laajakaista.fi (a91-152-230-218.elisa-laajakaista.fi [91.152.230.218]) by mail.ispro.net (Horde Framework) with HTTP; Thu, 09 Oct 2008 02:28:33 +0300 Message-ID: <20081009022833.73943nda9muz0qo0@mail.ispro.net> Date: Thu, 09 Oct 2008 02:28:33 +0300 From: yurtesen@ispro.net To: "Zaphod Beeblebrox" References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <5f67a8c40810070937r5ba89773ncee407ace25fa0dd@mail.gmail.com> <86iqs3sdtp.fsf@ds4.des.no> <5f67a8c40810081015p2c14e38evbeed0a97242a7c4a@mail.gmail.com> <48ECF564.7000204@ispro.net> <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> In-Reply-To: <5f67a8c40810081235k227dc870tce5fcbcbca61d3c1@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.6) X-Mailman-Approved-At: Thu, 09 Oct 2008 00:00:01 +0000 Cc: Volker , Dag-Erling =?iso-8859-1?b?U234cmdyYXY=?= , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 23:28:20 -0000 Quoting "Zaphod Beeblebrox" : > On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen wrote: > >> Zaphod Beeblebrox wrote: >> >>> >>> Wow... thanks for the flame, but there's no reason that the device that >>> is receiving the hammer replication couldn't be on the other side of the >>> globe and there's no reason it couldn't be considered a backup. =20 >>> Part of the >>> advantage of the structure that allows you to efficiently select for new >>> changes allows you to do the same kind of *backup* as they claim. >>> >>> >> Wouldnt that device need to keep the whole filesystem? Like if you have 1= 0 >> machines with 10x 1GB drives (lets say each used about 250gb), you will n= eed >> 10TB disk space in the backup server? >> > > > Urm... I think everything we've been discussing here backs up the whole > filesystem (it would be near impossible for a block-oriented system to do > elsewise). I suppose you could do something with the archive bit or dump > bits with a filesystem based backup. > > But anyways... in a filesystem based replication system, you'd need enough > space to store the data and the history of the data. The sum of the histo= ry > of the data could even exceed the size of the sum of the input disks. It > could also be much smaller. It really depends on how much you change. > The CDP backup solutions can manage that, even better you dont have to =20 change to a totally new filesystem to use them. I really do not want to have a large argument about this but the only =20 reason we cant do this on FreeBSD is because the companies who write =20 CDP type backup solutions either do not know much about FreeBSD or =20 they just cant find anybody who they can hire to do the job on FreeBSD. So that is why I posted this information to this list. If there is =20 anybody who is capable of porting such software to FreeBSD then they =20 can, 1- probably make money by doing the job, 2- if they have very =20 little free time and not interested in making money then they can =20 perhaps contact some of these companies (for example r1soft seems to =20 be interested in supporting FreeBSD) and give them some hints and =20 ideas on how this can be done on FreeBSD and where to find example =20 codes and more information etc. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 00:16:37 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B14351065686 for ; Thu, 9 Oct 2008 00:16:37 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 8310D8FC14 for ; Thu, 9 Oct 2008 00:16:36 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 34100 invoked by uid 399); 9 Oct 2008 00:16:55 -0000 Received: from localhost (127.0.0.1) by localhost with ESMTPM; 9 Oct 2008 00:16:55 -0000 X-Originating-IP: 127.0.0.1 Received: from a91-152-230-218.elisa-laajakaista.fi (a91-152-230-218.elisa-laajakaista.fi [91.152.230.218]) by mail.ispro.net (Horde Framework) with HTTP; Thu, 09 Oct 2008 03:16:53 +0300 Message-ID: <20081009031653.19603w0x984w8l6o@mail.ispro.net> Date: Thu, 09 Oct 2008 03:16:53 +0300 From: yurtesen@ispro.net To: "Dag-Erling =?utf-8?b?U23DuHJncmF2?=" References: <48E9E1BB.6020908@ispro.net> <48EA56BB.6040702@vwsoft.com> <48EA8B3A.3090609@ispro.net> <861vysiv9i.fsf@ds4.des.no> <20081007164015.2E3085B29@mail.bitblocks.com> <86ej2rsdnc.fsf@ds4.des.no> <20081008190354.264C25B4C@mail.bitblocks.com> <86prma3km3.fsf@ds4.des.no> In-Reply-To: <86prma3km3.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.6) X-Mailman-Approved-At: Thu, 09 Oct 2008 00:23:55 +0000 Cc: Volker , hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 00:16:37 -0000 Quoting "Dag-Erling Sm=C3=B8rgrav" : > If you take a step back and go through and read the entire thread again > from the start, though, I think you will understand my frustration. > Evren asked a question which everybody else is doing their best not to > answer in as many words as possible; and when I try to answer, I find > out that Evren doesn't really know what the question is. > > This is where - in an ideal world - somebody at R1Soft would jump in and > start asking the right questions... Actually, I was expecting that somebody would contact to R1Soft and =20 offer them help for $$$. But I have given your contact information to =20 them now. So they can forward their questions to you and probably you =20 can let them know who knows what and who they can contact with more =20 information or ask $$$ from them to do the job :) Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 03:22:22 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2287B1065686; Thu, 9 Oct 2008 03:22:22 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from mail.farley.org (farley.org [67.64.95.201]) by mx1.freebsd.org (Postfix) with ESMTP id E2FA78FC12; Thu, 9 Oct 2008 03:22:21 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from thor.farley.org (HPooka@thor.farley.org [192.168.1.5]) by mail.farley.org (8.14.3/8.14.3) with ESMTP id m9937WF7010679; Wed, 8 Oct 2008 22:07:32 -0500 (CDT) (envelope-from scf@FreeBSD.org) Date: Wed, 8 Oct 2008 22:07:32 -0500 (CDT) From: "Sean C. Farley" To: Ivan Voras In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Status: No, score=-4.3 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.farley.org Cc: freebsd-hackers@FreeBSD.org Subject: Re: strftime's %c warning? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 03:22:22 -0000 On Wed, 8 Oct 2008, Ivan Voras wrote: > I'm trying to use the %c formatter in strftime(3), documented as: > > " > %c is replaced by national representation of time and date. > " > > ... which looks useful, except that in code in which WFORMAT is defined > as "1" I get this error: > > str.c: In function 'ltime': > str.c:141: warning: '%c' yields only last 2 digits of year in some > locales on non-BSD systems > *** Error code 1 > > Since the code I'm developing is definitely BSD-only (patch to pkg_* > infrastructure), should I: > > a) stop using locale-based %c and choose my own date/time format > b) remove WFORMAT from the Makefile? > > The same warning/error is generated by %x and %X, and %+ described in > the strftime man page isn't recognized. You are hitting a gcc builtin. Have you tried adding -fno-builtin-strftime? Sean -- scf@FreeBSD.org From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 03:32:25 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26C261065695 for ; Thu, 9 Oct 2008 03:32:25 +0000 (UTC) (envelope-from brucem@mail.cruzio.com) Received: from cruzio.com (dsl-63-249-85-132.cruzio.com [63.249.85.132]) by mx1.freebsd.org (Postfix) with ESMTP id C25FA8FC0A for ; Thu, 9 Oct 2008 03:32:24 +0000 (UTC) (envelope-from brucem@mail.cruzio.com) Received: from mail.cruzio.com (localhost [127.0.0.1]) by cruzio.com (8.12.10/8.12.10) with ESMTP id m984irm6000401 for ; Tue, 7 Oct 2008 21:44:53 -0700 (PDT) (envelope-from brucem@mail.cruzio.com) Received: (from brucem@localhost) by mail.cruzio.com (8.12.10/8.12.10/Submit) id m984ireq000400 for hackers@freebsd.org; Tue, 7 Oct 2008 21:44:53 -0700 (PDT) (envelope-from brucem) Date: Tue, 7 Oct 2008 21:44:53 -0700 (PDT) From: "Bruce R. Montague" Message-Id: <200810080444.m984ireq000400@mail.cruzio.com> To: hackers@freebsd.org Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 03:32:25 -0000 Hi, re Continuous Data Protection (an enterprise market niche, today), a recent paper that provides some background and has basic references might be: http://www.usenix.org/event/fast08/tech/verma.html "SWEEPER: An Efficient Disaster Recovery Point Identification Mechanism", FAST '08, Akshat Verma, IBM India Research; Kaladhar Voruganti, Network Appliance; Ramani Routray, IBM Almaden Research; Rohit Jain, Yahoo India. -bruce From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 07:17:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D32781065688 for ; Thu, 9 Oct 2008 07:17:07 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.227]) by mx1.freebsd.org (Postfix) with ESMTP id A3F9E8FC14 for ; Thu, 9 Oct 2008 07:17:07 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so4327455rvf.43 for ; Thu, 09 Oct 2008 00:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=0DX8GaANugWkcM7GnyLmAmqsXPSwkSmGf39wyB0GLMo=; b=iYXwX0ry3FuPRYBwO8lFoMo7c4Itf2sLzjEJJzyjkWcfSnrmwXDCrHF38WeizY5VCF PY2QfbacASqR2GeS+XtIgeuA1NfUELPL31Eq0bCf3pQXJNEOBUICyd8rMWeAqJEEZeVL e/tAIXgZ8bxthvESOKldgieu0mLvVcOE2Z1+s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=fAw+giqXsuD6xst7WM04Jp1Wnk2hLuX3q2W4oOz1od+93OIfzhfN0chihMY2maAQ7v oYCWc6N7thmUhRS2Oti9JX5+Xs3WQGv02Nkje/JbX2G7kXdJbgvglCsnDIedivG67Ve2 DLIH9ZtVvJYVPbhScQF7PWHVJJ591syB2Rosg= Received: by 10.141.43.19 with SMTP id v19mr5836548rvj.154.1223534979268; Wed, 08 Oct 2008 23:49:39 -0700 (PDT) Received: by 10.141.153.13 with HTTP; Wed, 8 Oct 2008 23:49:39 -0700 (PDT) Message-ID: <9bbcef730810082349t733a0295hf9227d35534fdfcb@mail.gmail.com> Date: Thu, 9 Oct 2008 08:49:39 +0200 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Sean C. Farley" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: X-Google-Sender-Auth: 48b5bce0e845267b Cc: freebsd-hackers@freebsd.org Subject: Re: strftime's %c warning? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 07:17:07 -0000 2008/10/9 Sean C. Farley : > On Wed, 8 Oct 2008, Ivan Voras wrote: > >> str.c: In function 'ltime': >> str.c:141: warning: '%c' yields only last 2 digits of year in some >> locales on non-BSD systems >> *** Error code 1 >> >> Since the code I'm developing is definitely BSD-only (patch to pkg_* >> infrastructure), should I: >> >> a) stop using locale-based %c and choose my own date/time format >> b) remove WFORMAT from the Makefile? >> >> The same warning/error is generated by %x and %X, and %+ described in >> the strftime man page isn't recognized. > > You are hitting a gcc builtin. Have you tried adding > -fno-builtin-strftime? The code im working on is part of FreeBSD and uses FreeBSD's infrastructure for building. I can avoid the warning by defining WFORMAT to be 0 but I don't know how to add -fno-builtin-strftime except directly into CFLAGS. I don't think the original author (John Hubbard) is around anymore so the question is - should I just remove WFPORMAT from this Makefile? From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 09:16:51 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9953E106568B; Thu, 9 Oct 2008 09:16:51 +0000 (UTC) (envelope-from z@zlo.nu) Received: from mzh.zlo.nu (ns0.zlo.nu [85.17.141.90]) by mx1.freebsd.org (Postfix) with ESMTP id 569F28FC15; Thu, 9 Oct 2008 09:16:51 +0000 (UTC) (envelope-from z@zlo.nu) Received: by mzh.zlo.nu (Postfix, from userid 1000) id EB312140FA; Thu, 9 Oct 2008 10:54:09 +0200 (CEST) Date: Thu, 9 Oct 2008 10:54:09 +0200 From: Marc Olzheim To: Ivan Voras Message-ID: <20081009085409.GA11669@zlo.nu> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="82I3+IH0IqGh5yIs" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org Subject: Re: strftime's %c warning? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 09:16:51 -0000 --82I3+IH0IqGh5yIs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 08, 2008 at 10:20:00PM +0200, Ivan Voras wrote: > I'm trying to use the %c formatter in strftime(3), documented as: >=20 > " > %c is replaced by national representation of time and date. > " >=20 > ... which looks useful, except that in code in which WFORMAT is defined > as "1" I get this error: >=20 > str.c: In function 'ltime': > str.c:141: warning: '%c' yields only last 2 digits of year in some > locales on non-BSD systems > *** Error code 1 >=20 > Since the code I'm developing is definitely BSD-only (patch to pkg_* > infrastructure), should I: >=20 > a) stop using locale-based %c and choose my own date/time format > b) remove WFORMAT from the Makefile? >=20 > The same warning/error is generated by %x and %X, and %+ described in > the strftime man page isn't recognized. Yes. In FreeBSD 4.x, gcc was patched not to warn about these, so compiling with -Werror was possible even when using this. Since FreeBSD 5.x, this patch wasn't applied anymore. If you only want it to compile on your own system with -Werror, you can use this patch (and probably adapt it to %c, %x and %X as well): http://www.stack.nl/~marcolz/FreeBSD/gcc-strftime-format-freebsd7.patch.txt Marc --82I3+IH0IqGh5yIs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkjtxrEACgkQezjnobFOgrGY7wCfTsv6/EU4tbDV+u0R+n7FiXAI ML8An3PCMFn4cDGQcoPQ8hg4/BA8BwLU =b0Sp -----END PGP SIGNATURE----- --82I3+IH0IqGh5yIs-- From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 12:29:43 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16B3C106568B for ; Thu, 9 Oct 2008 12:29:43 +0000 (UTC) (envelope-from sigtrm@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.230]) by mx1.freebsd.org (Postfix) with ESMTP id E00538FC17 for ; Thu, 9 Oct 2008 12:29:42 +0000 (UTC) (envelope-from sigtrm@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so4441408rvf.43 for ; Thu, 09 Oct 2008 05:29:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=Xx4F7l6e5nlmXwD5cyrDYmq45dbhkvc52u+PWoUrjzI=; b=cIUQXe9mxBXIRoEd7Pd6MO64ZCdnE4+af8B7ihiqZLZPsLExqqeNM6jUAQFE2uPATR JZzzuOFZzQUCU2gLd0ec2KhPU9yHhayVIdAgKxi/XFHdTK60k5v/kXLPpqEiQdzpWR4E z5XNLqYz28a0i72bOO7IQXdCUz5s7VTcKftFY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=kMcihb4IU6odG2y2Uu0sELSQqAFd9EHvWl0o2EwzpfKAgM+YwVn1N6imxMrj1Mg3QC R9opqu3uOsakgZdiH1YSnbjUO7hY7uver+hmb4rWBRcrL0cGWZxkIaKO6V3eJUJe3Zen kbB7hlOnwqUz2Ubf8AiKPXaPN3evG8U6pCeqw= Received: by 10.141.75.17 with SMTP id c17mr6038796rvl.212.1223554093706; Thu, 09 Oct 2008 05:08:13 -0700 (PDT) Received: by 10.141.185.21 with HTTP; Thu, 9 Oct 2008 05:08:13 -0700 (PDT) Message-ID: Date: Thu, 9 Oct 2008 14:08:13 +0200 From: "Lukasz Jaroszewski" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Sockstress X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 12:29:43 -0000 Hi, I am wondering about sockstres informations recently published. I cant really figure what new they could found. Do we have anything to worry about? ;-) http://searchsecurity.techtarget.com/news/article/0,289142,sid14_gci1332898,00.html ``(...)Sockstress computes and stores so-called client-side SYN cookies and enables Lee and Louis to specify a destination port and IP address. The method allows them to complete the TCP handshake without having to store any values, which takes time and resources. "We can then say that we want to establish X number of TCP connections on that address and that we want to use this attack type, and it does it," Lee said.(...)'' ``(...)Lee said that when and _if_ specific vendors develop workarounds for the issues, they will release details of those issues.(...)'' Was FreeBSD team contacted? ;) -- Regards/Pozdrawiam LVJ -------------------------------------------------------------------------------------------------- They must find it difficult, those that take authority as truth, instead of truth as the authority From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 13:38:35 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 394F21065686; Thu, 9 Oct 2008 13:38:35 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id A7ECF8FC20; Thu, 9 Oct 2008 13:38:34 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id m99DcWMQ006321; Thu, 9 Oct 2008 15:38:32 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id m99DcW3a006320; Thu, 9 Oct 2008 15:38:32 +0200 (CEST) (envelope-from olli) Date: Thu, 9 Oct 2008 15:38:32 +0200 (CEST) Message-Id: <200810091338.m99DcW3a006320@lurza.secnetix.de> From: Oliver Fromme To: freebsd-hackers@FreeBSD.ORG, freebsd-security@FreeBSD.ORG, sigtrm@gmail.com In-Reply-To: X-Newsgroups: list.freebsd-hackers User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.4-PRERELEASE-20080904 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Thu, 09 Oct 2008 15:38:33 +0200 (CEST) Cc: Subject: Re: Sockstress X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-security@FreeBSD.ORG, sigtrm@gmail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 13:38:35 -0000 This is the wrong mailing list, you should send this to the -security list. By the way, this kind of attack isn't really new (as far as I can tell from the few information that have been made public so far). One way to mitigate it is to limit the number of open connections per remote IP address; you can easily do that with PF or IPFW ("limit" option). Best regards Oliver Lukasz Jaroszewski wrote: > Hi, > I am wondering about sockstres informations recently published. I cant > really figure what new they could found. Do we have anything to worry about? > ;-) > > http://searchsecurity.techtarget.com/news/article/0,289142,sid14_gci1332898,00.html > > ``(...)Sockstress computes and stores so-called client-side SYN cookies and > enables Lee and Louis to specify a destination port and IP address. The > method allows them to complete the TCP handshake without having to store any > values, which takes time and resources. "We can then say that we want to > establish X number of TCP connections on that address and that we want to > use this attack type, and it does it," Lee said.(...)'' > > ``(...)Lee said that when and _if_ specific vendors develop workarounds for > the issues, they will release details of those issues.(...)'' > > Was FreeBSD team contacted? ;) > -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Unix gives you just enough rope to hang yourself -- and then a couple of more feet, just to be sure." -- Eric Allman From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 14:11:05 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B89D3106568A; Thu, 9 Oct 2008 14:11:05 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2EFAC8FC1F; Thu, 9 Oct 2008 14:11:04 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id m99EB36N007552; Thu, 9 Oct 2008 16:11:03 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id m99EB0Vo007538; Thu, 9 Oct 2008 16:11:00 +0200 (CEST) (envelope-from olli) Date: Thu, 9 Oct 2008 16:11:00 +0200 (CEST) Message-Id: <200810091411.m99EB0Vo007538@lurza.secnetix.de> From: Oliver Fromme To: freebsd-hackers@FreeBSD.ORG, yurtesen@ispro.net, Shaun Amott , Sean Bruno In-Reply-To: <48ECE382.4060907@ispro.net> X-Newsgroups: list.freebsd-hackers User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.4-PRERELEASE-20080904 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Thu, 09 Oct 2008 16:11:03 +0200 (CEST) Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@FreeBSD.ORG, yurtesen@ispro.net, Shaun Amott , Sean Bruno List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 14:11:05 -0000 Evren Yurtesen wrote: > Shaun Amott wrote: > > On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: > > > so FreeBSD could be supported also. As you can imagine, it is not only > > > important that data can be restored when a box hardware failure etc. it is > > > also important that data can be restored if deleted by accidents etc. While > > > traditional backup programs provide this functionality, you cant really go > > > back to 10 min or 1h ago, often they take daily backups and have to scan > > > whole filesystem for changed files every time the backup is taken which > > > stresses out the systems. > > > > This can (more or less) be achieved with snapshots: you can cheaply > > maintain old versions of the file system, and mount an old snapshot at > > any time. Hourly is about as fine-grained as you can expect though. > > The documentation says one cant do more than 20 snapshots. > http://www.freebsd.org/doc/en/books/handbook/snapshots.html I wouldn't use UFS snapshots for this purpose. They have a few well-known disadvantages. However, ZFS snapshots should work very well for this. They're not limited to 20, and you can create them very quickly and with low overhead. You could create a new snapshot every 5 minutes if you want. Then you can use the "zfs send" command to produce a stream that contains the incremental differences between the snapshot and the previous snapshot, i.e. the stream represents the changes to the filesystem within the past five minutes (or whatever snapshot interval you choose). You can store that stream in a file, on backup medium, or send it with ssh to a different continent. Every once in a while you should make a full backup from a snapshot, of course. Basically it works like any other incremental backup mechanism, except that you can make the time interval between incremental backups very small (like five minutes in the above example), so you get a nearly continous backup solution. By the way, if you accidentally deleted something, you can of course simply copy it back from a snapshot. You can even revert the whole file system to a previous snapshot using the "zfs rollback" command. This is like going back in time. There is no need to touch your backups for that. These features are readily available right now on FreeBSD. You don't have to code anything. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 14:21:16 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8203106568E for ; Thu, 9 Oct 2008 14:21:16 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0C7768FC24 for ; Thu, 9 Oct 2008 14:21:15 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.3/8.14.3) with ESMTP id m99ELEwd007902; Thu, 9 Oct 2008 16:21:14 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.3/8.14.3/Submit) id m99ELEcm007901; Thu, 9 Oct 2008 16:21:14 +0200 (CEST) (envelope-from olli) Date: Thu, 9 Oct 2008 16:21:14 +0200 (CEST) Message-Id: <200810091421.m99ELEcm007901@lurza.secnetix.de> From: Oliver Fromme To: freebsd-hackers@FreeBSD.ORG, zbeeble@gmail.com In-Reply-To: <5f67a8c40810081019w79e0bb42i49c4da623b6e08ab@mail.gmail.com> X-Newsgroups: list.freebsd-hackers User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.4-PRERELEASE-20080904 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Thu, 09 Oct 2008 16:21:14 +0200 (CEST) Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@FreeBSD.ORG, zbeeble@gmail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 14:21:16 -0000 Zaphod Beeblebrox wrote: > Oliver Fromme wrote: > > However, ZFS does exist on FreeBSD, and I think it wouldn't > > be impossible to add similar features to ZFS. > > Possibly even as a ZFS module? This might be something better addressed at > the ZFS project level --- but the next question is: does FreeBSD support ZFS > modules? I'm sorry I don't know. But also see my other reply regarding ZFS snapshots ans "zfs send". > > Another possibility would be to extend gjournal by adding > > time stamps to journal transactions and a possibility to > > feed the journal to a pipe, socket or whatever. And of > > course a client-side implementation that does something > > useful with the journal stream. This might even be a good > > SoC project. > > Now this interests me. Firstly, I thought that gjournal might only be > responsible for the meta-data (but I'm happy to be wrong on this point). Nope, gjournal handles _all_ data, i.e. meta data and file contents. > Secondly, is it a) sufficient and b) efficient to attempt to time-travel UFS > with the gjournal log? I almost don't dare to mention DragonFly BSD again, but they do have a UFS journaling implementation that does exactly that: http://leaf.dragonflybsd.org/cgi/web-man?command=mountctl http://leaf.dragonflybsd.org/cgi/web-man?command=jscan However, I think the implementation was abandoned because it was obsoleted by the development of the HAMMER file system. But the basic functionality is there and works. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "IRIX is about as stable as a one-legged drunk with hypothermia in a four-hundred mile per hour wind, balancing on a banana peel on a greased cookie sheet -- when someone throws him an elephant with bad breath and a worse temper." -- Ralf Hildebrandt From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 15:46:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1D671065690 for ; Thu, 9 Oct 2008 15:46:33 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from mail-gx0-f21.google.com (mail-gx0-f21.google.com [209.85.217.21]) by mx1.freebsd.org (Postfix) with ESMTP id 8B4448FC21 for ; Thu, 9 Oct 2008 15:46:33 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: by gxk14 with SMTP id 14so102152gxk.19 for ; Thu, 09 Oct 2008 08:46:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=QPF8Uts58HrVzzS4ZevqCzHWxo0b3Nxbu7I9AG9w7Vk=; b=f2dp+XT8HP/6admQMRVFv0R+nO4Iz8jBP/hVSDypwROeJ2KM4yG+uusTlUAe040hdR bjDKCL+krlJA3sHv9grMvNLwZEMr50yyYktgUQ8PmLtTmJpdRefEiCeb2WOUFjXER1jV C6HE6Y+o4wLTTdpwkyoFQaWuty7+2hkHMIlZ4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=PG96hPPn1dwUbhLdpxIQsz9I/8HiphXVN4oZ98ApF+DTOvDRpy4omCOjo4/gma0oks t/N8uAdGx0EF8AD9lfM88b9DbYI388VmuSZnHCgU7w+PW8QYp9AzyRKtfvs+c1bfsLKm RT7FmBG6MooQDtMiZblXS+zr1bBDZwQzK7P40= Received: by 10.90.94.3 with SMTP id r3mr104302agb.62.1223567192143; Thu, 09 Oct 2008 08:46:32 -0700 (PDT) Received: by 10.90.94.14 with HTTP; Thu, 9 Oct 2008 08:46:32 -0700 (PDT) Message-ID: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> Date: Thu, 9 Oct 2008 08:46:32 -0700 From: "alan yang" To: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: setkey panic freebsd7 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 15:46:33 -0000 i wonder people ran into similar issue on setkey with freebsd7 that panic at ~/crypto/sha1.c:263 within sha1_result() digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; on the following sadb add with setkey: add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc "12345678" -A keyed-sha1 "12345678123456781234" thanks in advance on any hints. From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 17:00:00 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F31CB1065686; Thu, 9 Oct 2008 16:59:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (www.tegenbosch28.nl [217.21.251.97]) by mx1.freebsd.org (Postfix) with ESMTP id A62088FC13; Thu, 9 Oct 2008 16:59:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from localhost (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 1B461173C9; Thu, 9 Oct 2008 18:40:56 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by localhost (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6SGKaNbgWG9h; Thu, 9 Oct 2008 18:40:50 +0200 (CEST) Received: from [212.61.27.67] (opteron [212.61.27.67]) by mail.digiware.nl (Postfix) with ESMTP id 580FE17170; Thu, 9 Oct 2008 18:40:50 +0200 (CEST) Message-ID: <48EE340B.9010507@IMAP> Date: Thu, 09 Oct 2008 18:40:43 +0200 From: Willem Jan Withagen User-Agent: Thunderbird 2.0.0.17 (X11/20080925) MIME-Version: 1.0 To: Kip Macy References: <3c1674c90808221503v5ee48f05td71f70f152e71ef8@mail.gmail.com> In-Reply-To: <3c1674c90808221503v5ee48f05td71f70f152e71ef8@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 09 Oct 2008 17:02:56 +0000 Cc: freebsd-hackers@freebsd.org, FreeBSD Current Subject: Re: request for testers - xen support for domU in head X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 17:00:00 -0000 Kip Macy wrote: > Basic Xen support for 32-bit in PAE mode is in CVS. Please see the > wiki for general information: > > http://wiki.freebsd.org/FreeBSD/Xen > > Please be forewarned that I am not claiming that this is > production-ready. There are many known limitations. If you would like > to take it for a test drive and report bugs please give it a spin. Well it took me some figuring out, before I found this message. Getting it to run with a current-kernel is a lot simpeler. So what I've done is: take your mdroot-disk and your config build the current XEN kernel So I'm running a current-kernel with 7.0 world, I guess. That boots just fine after I changed the disk to xbd1a... So as a nice stresstest I thought: lets mount /usr/src8/src and try to build world. :) Well that is killed in the first serious work with sig 11. Which I at first expected to be due the small memsize. But even upping this on to 256Mb, the compiler still crashes. Also the system suffers from rather seriously long freezes. It can sit there for like > 30 secs. But then on the other hand: It is getting real close to being usefull. --WjW Underlying stuff: dual opteron 245 with 2gb RAM running Ubuntu Hardy en Xen 3.2 PAE --WjW From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 19:40:40 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D01D1065691 for ; Thu, 9 Oct 2008 19:40:40 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id EE4548FC1F for ; Thu, 9 Oct 2008 19:40:39 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m99JedBx041574 for ; Thu, 9 Oct 2008 12:40:39 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m99Jed1t041572; Thu, 9 Oct 2008 12:40:39 -0700 (PDT) Date: Thu, 9 Oct 2008 12:40:39 -0700 (PDT) From: Matthew Dillon Message-Id: <200810091940.m99Jed1t041572@apollo.backplane.com> To: freebsd-hackers@freebsd.org References: <200810091421.m99ELEcm007901@lurza.secnetix.de> Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 19:40:40 -0000 Go here: http://forums.mysql.com/read.php?28,197644,197644 There are a ton of ways to maintain mysql backups without having to replay the entire log. Google some keywords. With regards to solutions based on filesystem snapshots, such as partial log replaying (snapshot + new-log, then replay from snapshot after crash), HAMMER and ZFS are your only real choices in the BSD world. UFS snapshots have all sorts of issues that make them unsuitable. Block level snapshots are unreliable. When interacting with a high level program for crash recovery purposes you really want to use a filesystem's native snapshot capability (e.g. HAMMER or ZFS. UFS's isn't good enough). You do not need to explicitly backup the filesystem to other media. That is, you of course still make such backups, but they would only be used in case of a catastrophic loss of the production filesystem, not for simple crash recovery which can be done from the same-media snapshots. -- In anycase, HAMMER has two native backup solutions. Both are easy to use. (1) Just use cpdup. This works nicely as the source filesystem, if HAMMER, can be snapshotted, and then used as a stable cpdup source. cpdup to the target and let HAMMER manage the history for you. No hardlink trick needed or anything like that. Use HAMMER's snapshots on the target to determine what history you want to retain. Advantages: Extremely convenient, target storage is reasonably efficient. Great if sources are mixed filesystem types. Disadvantages: Runs cpdup so you are scanning the directory hierarchy on both. Inode numbers not retained, so not suitable for fallback. (this sort of methodology can be used w/ ZFS too). (2) For HAMMER-to-HAMMER you can use HAMMER's native mirroring. Simply create a PFS (pseudo hammer filesystem) slave on the target, which takes 2 seconds, then use the 'hammer mirror-copy' or 'hammer mirror-stream' directives (which can take remote host specifications and will run ssh) to mirror from source to target. HAMMER's mirroring is incremental from the protocol right down to the media accesses. No pre-scan occurs is needed. Advantages: Provides bandwidth-controlled incremental streaming, near-realtime operation (30-60 second lag though finer-grained operation is possible). Extremely robust (can be ^C'd and restarted, or left offline for long periods of time, etc). Mirrors all fine-grained history on source and can be re-pruned on the target to the desired interval. Has little effect on production machines (no queues means backups can't feed-back and effect the performance of the production box). Mirrors inode numbers etc. Mirroring target can become a new master in a pinch (but can't be made a slave again, sorry). Should serve the same NFS fsid, etc. Disadvantages: Must mirror the entire PFS, only works between HAMMER filesystems. Near-real-time (30-60 second lag) is not real time, but it's probably close enough. (ZFS has a way to do something similar but I do not know what the various advantages or disadvantages of using the feature are). DragonFly also has real-time journaling at the VFS layer, which is not HAMMER-specific, but it requires queuing and isn't really suitable on a production environment for off-machine copying because the queue can fill up and block the filesystem. HAMMER's mirroring is far more robust. -Matt From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 20:44:44 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D4331065692; Thu, 9 Oct 2008 20:44:44 +0000 (UTC) (envelope-from jgrosch@mooseriver.com) Received: from gdead.mooseriver.com (gdead.mooseriver.com [205.166.121.45]) by mx1.freebsd.org (Postfix) with ESMTP id 6145F8FC08; Thu, 9 Oct 2008 20:44:44 +0000 (UTC) (envelope-from jgrosch@mooseriver.com) Received: by gdead.mooseriver.com (Postfix, from userid 2010) id 71D9563333; Thu, 9 Oct 2008 13:25:23 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gdead.mooseriver.com X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 Received: from mooseriver.com (berkeley.mooseriver.com [75.61.201.134]) by gdead.mooseriver.com (Postfix) with ESMTP id 026CE63330; Thu, 9 Oct 2008 13:25:22 -0700 (PDT) Received: by mooseriver.com (Postfix, from userid 0) id CB2C22F7108; Thu, 9 Oct 2008 13:25:21 -0700 (PDT) Date: Thu, 9 Oct 2008 13:25:21 -0700 From: Josef Grosch To: questions@freebsd.org Message-ID: <20081009202521.GA57222@mooseriver.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Organization: Moose River, LLC Cc: hackers@freebsd.org, hardware@freebsd.org Subject: FreeBSD support for HP DL180/G5 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jgrosch@MooseRiver.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 20:44:44 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Does anyone have experience running FreeBSD 6.x and 7.x on an HP DL180/G5? The company I work for is looking to get a number of these to be put in production. Your general impressions would be a good start. Josef --=20 Josef Grosch | Another day closer to a | FreeBSD 6.3 jgrosch@MooseRiver.com | Micro$oft free world | Berkeley, Ca. --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- iD8DBQFI7miry8prLS1GYSERAgbPAKChLT0QDIVGiWzZWpkU8+pZDGK3UACeOhBd kNsXOyeO+6oD5pQ9HX0bBWs= =lon3 -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF-- From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 21:16:20 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78FE91065695; Thu, 9 Oct 2008 21:16:20 +0000 (UTC) (envelope-from vanhu@zeninc.net) Received: from smtp.zeninc.net (smtp.zeninc.net [80.67.176.25]) by mx1.freebsd.org (Postfix) with ESMTP id 34DA38FC29; Thu, 9 Oct 2008 21:16:20 +0000 (UTC) (envelope-from vanhu@zeninc.net) Received: from albator.zen.inc (albator.zen.inc [192.168.1.5]) by smtp.zeninc.net (smtpd) with ESMTP id 232B1279899; Thu, 9 Oct 2008 22:56:37 +0200 (CEST) Received: by albator.zen.inc (Postfix, from userid 1000) id 82770731D6; Thu, 9 Oct 2008 22:56:36 +0200 (CEST) Date: Thu, 9 Oct 2008 22:56:36 +0200 From: VANHULLEBUS Yvan To: alan yang Message-ID: <20081009205636.GA3002@zeninc.net> References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> User-Agent: All mail clients suck. This one just sucks less. Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: setkey panic freebsd7 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 21:16:20 -0000 Hi. On Thu, Oct 09, 2008 at 08:46:32AM -0700, alan yang wrote: > i wonder people ran into similar issue on setkey with freebsd7 that > panic at ~/crypto/sha1.c:263 within sha1_result() > digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; > > on the following sadb add with setkey: > add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc > "12345678" -A keyed-sha1 "12345678123456781234" > > thanks in advance on any hints. I guess most people just don't use static SAs anymore :-) Can you reproduce the bug ? Are you using /sbin/setkey (provided by FreeBSD), /usr/local/sbin/setkey (provided by ipsec-tools), or does it crash with both ? If you can reproduce it, please fill in a PR, Bjoern or I will take it. Anyways, I'll have a look asap at that part of the code, to see if I can find "something". Any extra information on how to reproduce the bug is welcome ! :-) Yvan. From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 21:48:54 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96BCC106568A for ; Thu, 9 Oct 2008 21:48:54 +0000 (UTC) (envelope-from subhro.kar@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.232]) by mx1.freebsd.org (Postfix) with ESMTP id 689898FC15 for ; Thu, 9 Oct 2008 21:48:54 +0000 (UTC) (envelope-from subhro.kar@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so194998rvf.43 for ; Thu, 09 Oct 2008 14:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=vdhiMfaVat00+0HeVl6MJX3fOyxMORM1095aS3tg5Zg=; b=GqY8T0PXZqAfC3m0hgkASennqoI5UMAIoBpiadpSojcbs8Cks6k6W9PaaeargysBaC oV0XIVQCVKFZ5nUJtwUtGECEIcj91Ly6EbqdpLHPbYYOLZGSGMcqMYDQQPPY17z0E5Ts MmhY77MgDAITB5I43aiLoo1hzM1UMqIb232OQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=JdsKQyzbTlCLnrWwzHICk7FDpod36QqCgF/dVGGjF8xB/yJ0pZyjsqxGW30CKetIYX 1KJ9IQ1Y/MYliNWKPMPPDA0wX0EbJ7/TUhS27pU54PfZSb4xlEGTqJDU/Ik7Ns0BDuaM lWJGUxckkU1Ux0Ga0k9aVrvDVJ6n0nEJb7BYU= Received: by 10.114.26.18 with SMTP id 18mr905374waz.162.1223587373989; Thu, 09 Oct 2008 14:22:53 -0700 (PDT) Received: by 10.114.15.4 with HTTP; Thu, 9 Oct 2008 14:22:53 -0700 (PDT) Message-ID: Date: Fri, 10 Oct 2008 02:52:53 +0530 From: Subhro To: jgrosch@mooseriver.com In-Reply-To: <20081009202521.GA57222@mooseriver.com> MIME-Version: 1.0 References: <20081009202521.GA57222@mooseriver.com> X-Mailman-Approved-At: Thu, 09 Oct 2008 22:05:27 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: hackers@freebsd.org, questions@freebsd.org, hardware@freebsd.org Subject: Re: FreeBSD support for HP DL180/G5 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 21:48:54 -0000 HP produces pretty good boxes and historically I have been able to get them working without any troubles. However I would say DL180 is a pretty non customizable box. The hardware works perfectly with FreeBSD 7.0. I didnt try it with 6.3, so cant comment on that. However I would say DL380 is a better off. The main advantage of DL3xx boxes are there is a lot of room to play with add-on cards. Also not all the latest and greatest processors are available with DL1xx family of servers. Also make sure that you go for an external RAID controller like 3ware or Areca. I prefer Areca more :-D. The HP RAID controller cant take the beating I give to it. Thanks Subhro On Fri, Oct 10, 2008 at 1:55 AM, Josef Grosch wrote: > > Does anyone have experience running FreeBSD 6.x and 7.x on an HP DL180/G5? > The company I work for is looking to get a number of these to be put in > production. Your general impressions would be a good start. > > > > Josef > > -- > Josef Grosch | Another day closer to a | FreeBSD 6.3 > jgrosch@MooseRiver.com | Micro$oft free world | Berkeley, Ca. > From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 22:35:16 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90443106568A for ; Thu, 9 Oct 2008 22:35:16 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.233]) by mx1.freebsd.org (Postfix) with ESMTP id 286CA8FC08 for ; Thu, 9 Oct 2008 22:35:15 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by wx-out-0506.google.com with SMTP id s17so106401wxc.7 for ; Thu, 09 Oct 2008 15:35:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=dRfH6EVdAuShGa+AWz183PSfQqmN+INHbDvVUIPieq4=; b=oCJKKA8eEZIYaE5SL67Bef/ubEs7y8qoUA0qk9yHghQ3HPau0FXdicvHzzTLvrG9jE WqeOBBMTtcn3/5VTkHYb3tGcNZ/QRM/s0OMHG0eWIJp+k+MZ9WIuMAMcdLu9xgH998A+ JEERmSw3xKgtCikf8jUfRHU9Wuu8kdOWGEt/o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=RYfcJHh+NjPDkZ9cNFtDlqr0sypgcyMIaPoVgPphWepAhO3Iv+qwNucBFW+vzVtNcJ OzfzNvbvnU2TgrKcMn+OsCAvjH1r2AxcZqOdPWNoWbfVrEu9J+9bZEnQ/8jupt1QSa4a bqPT1J17t5SZTqPB714+9mizNmIorz8X8vfaU= Received: by 10.150.203.8 with SMTP id a8mr700604ybg.146.1223582411840; Thu, 09 Oct 2008 13:00:11 -0700 (PDT) Received: by 10.150.137.11 with HTTP; Thu, 9 Oct 2008 13:00:11 -0700 (PDT) Message-ID: <5f67a8c40810091300v37f089a8pa8828f7fba5e9c@mail.gmail.com> Date: Thu, 9 Oct 2008 16:00:11 -0400 From: "Zaphod Beeblebrox" To: "Matthew Dillon" In-Reply-To: <200810091940.m99Jed1t041572@apollo.backplane.com> MIME-Version: 1.0 References: <200810091421.m99ELEcm007901@lurza.secnetix.de> <200810091940.m99Jed1t041572@apollo.backplane.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 22:35:16 -0000 On Thu, Oct 9, 2008 at 3:40 PM, Matthew Dillon wrote: > (ZFS has a way to do something similar but I do not know what the > various advantages or disadvantages of using the feature are). The only current way to do this on ZFS is to snapshot (very cheap) and stream the differences between the current snapshot and the previous snapshot to the remote host. The remote host can store the flat files or store the filesystem (that is: the streams can feed into new snapshots of the filesystem on the remote machine). Like Hammer, this gives history on both the local machine and the remote machine and is amazingly efficient. Unlike hammer, the process is not automated by the filesystem. You need a script that does "zfs snaphot..." followed by "zfs send ... | ssh zfs receive ..." --- such that each individual backup is a job rather than the connection approach you discribe for Hammer. Unlike Hammer, ZFS doesn't, by default, keep all history. I was speculating earlier that this might be possible to make as a ZFS module, though. From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 9 23:34:12 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6ACA71065689 for ; Thu, 9 Oct 2008 23:34:12 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 0FFB28FC14 for ; Thu, 9 Oct 2008 23:34:10 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 15155 invoked by uid 399); 9 Oct 2008 23:34:29 -0000 Received: from localhost (127.0.0.1) by localhost with ESMTPM; 9 Oct 2008 23:34:29 -0000 X-Originating-IP: 127.0.0.1 Received: from a91-152-230-218.elisa-laajakaista.fi (a91-152-230-218.elisa-laajakaista.fi [91.152.230.218]) by mail.ispro.net (Horde Framework) with HTTP; Fri, 10 Oct 2008 02:34:28 +0300 Message-ID: <20081010023428.87556dt18ejyzf48@mail.ispro.net> Date: Fri, 10 Oct 2008 02:34:28 +0300 From: yurtesen@ispro.net To: freebsd-hackers@FreeBSD.ORG, yurtesen@ispro.net, "Shaun Amott" , "Sean Bruno" , "Oliver Fromme" References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> In-Reply-To: <200810091411.m99EB0Vo007538@lurza.secnetix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.6) X-Mailman-Approved-At: Thu, 09 Oct 2008 23:45:31 +0000 Cc: Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 23:34:12 -0000 Quoting "Oliver Fromme" : > These features are readily available right now on FreeBSD. > You don't have to code anything. Well with 2 downsides, The fact that I still would need to take full backups once in a while =20 if I do this and Linux users do not have to because the CDP software =20 on Linux does not need to do this. The software expires the old data =20 automatically and you only need a full backup at first run only. The bigger problem is that I have to convert all my filesystems to =20 ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a =20 large part of the year doing such job while Linux users can just do =20 this on 'any' filesystem they use. How am I suppose to compete with =20 companies which use Linux otherwise if I am doing this sort of tasks =20 all the time? Thanks for all the advices but my original question was if somebody =20 can give inside information to a company(for example r1soft) which is =20 writing CDP backup solutions so they could implement such solution on =20 FreeBSD also. Do you know such person? I am not really looking for alternatives because there is none. You =20 cant just expect commercial companies to convert to a new filesystem =20 to add a feature which other OSes manage without going to such =20 measures. Can you imagine the monetary cost if all FreeBSD users had =20 to convert to ZFS (or another filesystem) to take near cdp level =20 backups? This simply would make people think 'I wish I used Linux from =20 the beginning'. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 00:09:16 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EE1C106564A for ; Fri, 10 Oct 2008 00:09:16 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 9AD658FC0A for ; Fri, 10 Oct 2008 00:09:15 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 47107 invoked by uid 1001); 9 Oct 2008 20:06:43 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Thu, 09 Oct 2008 20:06:42 -0400 Date: Thu, 9 Oct 2008 20:06:41 -0400 To: yurtesen@ispro.net Message-ID: <20081009200641.60d0b236@bhuda.mired.org> In-Reply-To: <20081010023428.87556dt18ejyzf48@mail.ispro.net> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> Organization: Meyer Consulting X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; amd64-portbld-freebsd7.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: Sean Bruno , freebsd-hackers@FreeBSD.ORG, Shaun Amott , Oliver Fromme , yurtesen@ispro.net Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 00:09:16 -0000 On Fri, 10 Oct 2008 02:34:28 +0300 yurtesen@ispro.net wrote: > Quoting "Oliver Fromme" : > > > These features are readily available right now on FreeBSD. > > You don't have to code anything. > > Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your "downsides" are largely figments of your imagination. > The fact that I still would need to take full backups once in a while Every time you started backing up a new file system. This is a requirement of all backup solutions. After that, never again. > The bigger problem is that I have to convert all my filesystems to > ZFS. Can one convert UFS2 to ZFS easily even? I didn't have any trouble. And once you do it, you have advantages that the poor schmucks using Linux can only dream about: like self-healing file systems that are so cheap and easy to create that it makes sense to put each application or jail on it's own file system, one that's tuned for the application. The ability to set up raid and mirror devices without ever having to deal with LLVM (worth the cost of entry all by itself), not having to worry about allocating partitions, and - well, the list just goes on and on. Having converted to ZFS on my FreeBSD boxes, the only thing I feel for my clients still using Linux file systems is pity. > this on 'any' filesystem they use I seriously doubt that it supports things like GMailFS. > How am I suppose to compete with > companies which use Linux otherwise if I am doing this sort of tasks > all the time? Well, once you've done the conversion, by outproducing them while they waste time dealing with LLVM, partitions, and other such crap that ZFS frees you from. > Thanks for all the advices but my original question was if somebody > can give inside information to a company(for example r1soft) which is > writing CDP backup solutions so they could implement such solution on > FreeBSD also. Do you know such person? The only "inside information" here is held by the company (for example rlsoft) providing the CDP software. On the FreeBSD side, the source and documentation are all freely available to anyone who wants to look at it. But it doesn't matter how well you know FreeBSD, you aren't going to get anywhere unless you also you know what the software from that company needs in order to operate. If said company wanted to hire someone to either write this or to help get their people started working with FreeBSD, then the thing to do is send mail to jobs@freebsd.org announcing the position. If they aren't interested in hiring someone, but hope to get it done for free, then they should set up a web page providing the technical details that someone who knows FreeBSD (or is willing to learn it) needs to do the job. If all you want is a CDP solution and you don't care where it comes from - well, you pretty much get the same two choices. It's an interesting enough problem that you might get someone to build it for free, but don't expect it to use proprietary software from some company that already provides such a thing for other systems. Nor - if you don't provide any incentive for meeting your requirements - should you expect it to actually meet them. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 01:14:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C653E1065686 for ; Fri, 10 Oct 2008 01:14:22 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 86BB28FC20 for ; Fri, 10 Oct 2008 01:14:22 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m9A1ELxd043984; Thu, 9 Oct 2008 18:14:21 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m9A1ELOG043983; Thu, 9 Oct 2008 18:14:21 -0700 (PDT) Date: Thu, 9 Oct 2008 18:14:21 -0700 (PDT) From: Matthew Dillon Message-Id: <200810100114.m9A1ELOG043983@apollo.backplane.com> To: yurtesen@ispro.net References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> Cc: freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 01:14:23 -0000 :The fact that I still would need to take full backups once in a while =20 :if I do this and Linux users do not have to because the CDP software =20 :on Linux does not need to do this. The software expires the old data =20 :automatically and you only need a full backup at first run only. You have to do this anyway. Nobody in their right mind trusts a single storage solution for all of their backups. CDP is a brute-force block-level continuous backup mechanism and while it works, to a point, it also has all the drawbacks that ANY block level backup system has... it is discontinuous from the high level filesystem structure and while you can pretty much guarantee that it is possible to recover from a disaster eventually, the filesystem you choose to run on top of that block device still matters a lot. On top of that being a client-server solution CDP is going to have some significant bottlenecks too. Even more telling is the fact that block level storage solutions tend to migrate corruption instead of detect it early. Big oopses can wind up stealthily winding their way through all your backups when you use a block-level solution. I stopped using block-level solutions almost 10 years ago... that's how little I trust them. Only a 'modern' linux filesystem (ext3 or the upcoming ext4, reiser, xfs, and few others), or something like ZFS or HAMMER, really have the ability to reliably recover from a point-in-time block level snapshot. Filesystems such as ZFS and HAMMER also give you insanely good snapshotting solutions that are far, FAR more flexible then what CDP gives you. You can upgrade between EXT filesystems without having to copy, but if you decided the best filesystem for you was one of the other many Linux filesystems, such as Reiser4 or XFS, and you were running EXT3, then you would have to copy. There are massive lists of pluses and minuses to each of the linux filesystem choices. Data expiration is a non-issue. You have to think about it either way. You have to test that the backup system actually works. You have to carefully control the backup policy and in particular not allow heavy disk I/O (such as a hacker DD'ing to your filesystem for 24 hours) to blow out your ability to recover the system. It requires time and effort no matter how well automated it is. :The bigger problem is that I have to convert all my filesystems to =20 :ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a =20 :large part of the year doing such job while Linux users can just do =20 :this on 'any' filesystem they use. How am I suppose to compete with =20 :companies which use Linux otherwise if I am doing this sort of tasks =20 :all the time? I converted our main developer machine from UFS to HAMMER in about 12 hours. 99% of that time simply waiting for the cpdup to finish copying a few hundred gigabytes from the old filesystem to the new. I think we're talking a few days here... time enough to learn how the filesystem works and play with it, make a few test copies (as you would with ANY new system that you did not have previous experienced with), and then do it for real. Linux users cannot just do this with a flip of a switch either. The new filesystem has to be constructed and the old data copied over. The old data set has to be retained, you cannot convert in-place, it's just too dangerous. It takes a certain amount of time to copy the data no matter what OS you are running, based on the amount of data you have. Some filesystem transitions such as going from ext2 to ext3 or ext3 to ext4 (if I remember right) are forwards compatible and do not require copying, but that sort of transition is NOT to a new filesystem, it's to a newer version of the same filesystem. With storage capacities increasing so quickly and mandating the replacement of whole disk subsystems (for running and electricity cost more then anything else, and more convenience, and less maintainance) it is a small convenience at best if you are going to copy the data anyway. Frankly, even if I were upgrading from ext2 to ext3 the last thing I would do is run it in-place. There's just too high a chance of software bugs creeping in. I would want a fresh ext3 and I would want my old ext2 data sitting on a shelf for a few days in case something were to go terribly wrong. I'd copy there too, just for safety. What is a few extra hours compared to blowing up the life-blood of your company? :I am not really looking for alternatives because there is none. You =20 :cant just expect commercial companies to convert to a new filesystem =20 :to add a feature which other OSes manage without going to such =20 :measures. Can you imagine the monetary cost if all FreeBSD users had =20 :to convert to ZFS (or another filesystem) to take near cdp level =20 :backups? This simply would make people think 'I wish I used Linux from =20 :the beginning'. : :Thanks, :Evren Commercial companies like to farm things out and they certainly like turnkey solutions. There are many more linux companies then BSD companies which offer turnkey solutions for various narrow bands of problems. I'm not knocking it, it's how the world works. Linux has a great deal of momentum despite the fracturing of the distributions. Linux has many other things going for it including a far better package updating scheme then any of the BSDs, but there are also downsides such as that huge ssh public key mess that was basically one programmer committing a change to a piece of code he was clueless about. The BSDs don't have the vendor support that linux has. There are many reasons for this, and it's really too bad that it turned out that way, but if you are a sysop in a company that doesn't have infinite $$ to spend, and you can't afford the high costs of turnkey solutions, then rolling your own on the core platforms (Linux or BSD) require only having a good head on your shoulders to turn into a success. It will take about the same amount of time either way. You might be pleasantly surprised! I know a lot of people who tear their hair out on Linux (but unfortunately the same people mostly think BSD's upsides don't make up for its downsides, sigh). But be careful not to confuse turnkey solutions with flexibility. Turnkey solutions... things like CDP, are narrowly focused and tend to be very inflexible when it comes to integration beyond their basic design. If something breaks in the middle of that black box you will be in 'houston we have a problem! mode' and it won't be fixed quickly. These sorts of companies expect to be paid, most of them with ongoing support contracts. Turnkey solutions are not going to be inexpensive. Even for open-source software, if you intend to manage both sides of the turnkey equation yourself you are essentially going to be devoting the same amount of time to it as someone rolling their own solution based on lower level (but still substantial) building blocks, such as the native features of ZFS. I can give you nightmare stories about turnkey backup solutions which either fail unexpectedly and f*ck up a company, or worse: Become obsolete and the programmers have so little visibility into the turnkey 'solution' they are unable to upgrade it. Lots of people have the latter problem. Turnkey can be good if carefully managed, and a nightmare if not. -Matt Matthew Dillon From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 03:19:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C612106568B for ; Fri, 10 Oct 2008 03:19:01 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.183]) by mx1.freebsd.org (Postfix) with ESMTP id 056D78FC15 for ; Fri, 10 Oct 2008 03:19:00 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: by el-out-1112.google.com with SMTP id v27so145463ele.13 for ; Thu, 09 Oct 2008 20:18:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=VAnPbAu/MO7XtlaPa8I22Z81stCr+b7iN7pvsSXgAuw=; b=GLL+gG5koiPLGSwe0vOOkUar1VtuDfwmLXdv5bR8SoqrdkA4oSSoZIpdJT44xc0EnJ rfOesRsOxs+ehSa+PU9R2MLWHY3FoTMrgfUDu1ci4WU3m7Xtcjm+YMk7aGAq6H/7ia+D Y4sJ2oxPlsv4IZQVgemsGi/+zr1xaKyj2D2Y8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=UB9HavH3lsHKp+Z8KpsfozEnDeCzS+f7Zr8yGlcCe0PVs/RoTlipt7sQiOj8n6/Yx+ R7YTxYkI+3BlMUnIS73HXF0FLm+Pj0LTq0nzwf7yr7o+lFgyT1Nxoqe8eenPTu9WRNOH 7YOMqNxG4BfAR2zgPJKBM97mN2VqLrD957weA= Received: by 10.90.98.13 with SMTP id v13mr965800agb.105.1223608739826; Thu, 09 Oct 2008 20:18:59 -0700 (PDT) Received: by 10.90.94.14 with HTTP; Thu, 9 Oct 2008 20:18:59 -0700 (PDT) Message-ID: <290865fd0810092018v59aeecc1rfc38790c0fde2f5e@mail.gmail.com> Date: Thu, 9 Oct 2008 20:18:59 -0700 From: "alan yang" To: "VANHULLEBUS Yvan" In-Reply-To: <20081009205636.GA3002@zeninc.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> <20081009205636.GA3002@zeninc.net> Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: setkey panic freebsd7 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 03:19:01 -0000 that single line of adding SA in a setkey.conf file with /sbin/setkey -f setkey.conf would fail 100% from all my try. /usr/local/sbin/setkey just tried, failed also. 'fill in PR' haven't done that before, could you please advise. thanks for looking into! On Thu, Oct 9, 2008 at 1:56 PM, VANHULLEBUS Yvan wrote: > Hi. > > > On Thu, Oct 09, 2008 at 08:46:32AM -0700, alan yang wrote: >> i wonder people ran into similar issue on setkey with freebsd7 that >> panic at ~/crypto/sha1.c:263 within sha1_result() >> digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; >> >> on the following sadb add with setkey: >> add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc >> "12345678" -A keyed-sha1 "12345678123456781234" >> >> thanks in advance on any hints. > > I guess most people just don't use static SAs anymore :-) > > Can you reproduce the bug ? > Are you using /sbin/setkey (provided by FreeBSD), > /usr/local/sbin/setkey (provided by ipsec-tools), or does it crash > with both ? > > > If you can reproduce it, please fill in a PR, Bjoern or I will take > it. > > Anyways, I'll have a look asap at that part of the code, to see if I > can find "something". > > Any extra information on how to reproduce the bug is welcome ! :-) > > > Yvan. > From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 09:20:27 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12C061065693 for ; Fri, 10 Oct 2008 09:20:27 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id AFFEA8FC1B for ; Fri, 10 Oct 2008 09:20:26 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id B2A3728454 for ; Fri, 10 Oct 2008 17:20:25 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id 3F360101D13B; Fri, 10 Oct 2008 17:20:25 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id rcR9XPaJKikW; Fri, 10 Oct 2008 17:20:20 +0800 (CST) Received: from delta.delphij.net (c-76-103-40-85.hsd1.ca.comcast.net [76.103.40.85]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 49634101D124; Fri, 10 Oct 2008 17:20:19 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:subject:x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=gkwbq6Fs5nKrKeAJ5QzSfVd+L5cD3no9yfS5O2hGoEl8+3bgItqtqTn7olIjy2oPu D4g2SvyPWsKSWzqk6RoGQ== Message-ID: <48EF1E51.4050205@delphij.net> Date: Fri, 10 Oct 2008 02:20:17 -0700 From: Xin LI Organization: The Geek China Organization User-Agent: Thunderbird 2.0.0.17 (X11/20080926) MIME-Version: 1.0 To: freebsd-hackers@FreeBSD.org X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Debugging modify-after-free issue X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 09:20:27 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Recently I had some crashes and other issues on my laptop and I found that, among some bugs I already caught and fixed on my local tree, there is still something like this: Oct 10 01:35:53 delta kernel: Memory modified after free 0xffffff00a2ef9600(256) val=6438300 @ 0xffffff00a2ef9608 Is there any way to track this down (looks like to be wpi(4) related but still keep trying)? I can not seem to reliably trigger it so it would be helpful if such thing would trigger a panic. Cheers, -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjvHlAACgkQi+vbBBjt66B2YQCeMXQwUWJOOm9PdCiFnsQD7Qba exkAnAz/4VRm+ZrU83XudWeo6lLTlkHO =N2pO -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 12:53:45 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B65AA106568A for ; Fri, 10 Oct 2008 12:53:45 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 3C9798FC0A for ; Fri, 10 Oct 2008 12:53:41 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 87647 invoked by uid 399); 10 Oct 2008 12:53:55 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 10 Oct 2008 12:53:55 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48EF5052.2000707@ispro.net> Date: Fri, 10 Oct 2008 15:53:38 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Mike Meyer References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> In-Reply-To: <20081009200641.60d0b236@bhuda.mired.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 10 Oct 2008 12:56:21 +0000 Cc: Sean Bruno , freebsd-hackers@FreeBSD.ORG, Oliver Fromme , Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 12:53:45 -0000 Mike Meyer wrote: > On Fri, 10 Oct 2008 02:34:28 +0300 > yurtesen@ispro.net wrote: > >> Quoting "Oliver Fromme" : >> >>> These features are readily available right now on FreeBSD. >>> You don't have to code anything. >> Well with 2 downsides, > > Once you actually try and implement these solutions, you'll see that > your "downsides" are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. When you look at this from a single point then you might be right. But next thing in my agenda is to provide restore services to hosting customers. Now when I use a commercial solution like r1soft backup, I can just install the plugin for the control panel software (like cPanel or H-Sphere). If it is imagination and things are so easy, I can give you $500(r1soft pricing) for every 5servers I have, and you can give the same features that r1soft gives, deal? I have no doubt that similar solution can be achieved on FreeBSD. If you sit and think for a second, the problem is the amount of time, resources and knowledge it requires to do the same solution in FreeBSD is way higher than if one was using Linux. > I seriously doubt that it supports things like GMailFS. You are right, my mistake. But it supports the most commonly used Linux filesystems, still better than changing to a brand new filesystem. But sorry for the mistake. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 14:42:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05B0E1065688 for ; Fri, 10 Oct 2008 14:42:33 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA08.westchester.pa.mail.comcast.net (qmta08.westchester.pa.mail.comcast.net [76.96.62.80]) by mx1.freebsd.org (Postfix) with ESMTP id 921A78FC1C for ; Fri, 10 Oct 2008 14:42:32 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA06.westchester.pa.mail.comcast.net ([76.96.62.51]) by QMTA08.westchester.pa.mail.comcast.net with comcast id R2dQ1a01A16LCl0582hETK; Fri, 10 Oct 2008 14:41:14 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA06.westchester.pa.mail.comcast.net with comcast id R2hC1a00F2P6wsM3S2hCCP; Fri, 10 Oct 2008 14:41:14 +0000 X-Authority-Analysis: v=1.0 c=1 a=nOJ1pGDTh_cA:10 a=QycZ5dHgAAAA:8 a=yIrVwuVQQqKs7Gv1ov4A:9 a=RSr9XloNvUsri024B51Oe-uKWjMA:4 a=EoioJ0NPDVgA:10 a=ttedcufwrkMA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id D267AC9419; Fri, 10 Oct 2008 07:41:11 -0700 (PDT) Date: Fri, 10 Oct 2008 07:41:11 -0700 From: Jeremy Chadwick To: Evren Yurtesen Message-ID: <20081010144111.GA34609@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48EF5052.2000707@ispro.net> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Sean Bruno , freebsd-hackers@FreeBSD.ORG, Mike Meyer , Oliver Fromme , Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 14:42:33 -0000 On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > Mike Meyer wrote: >> On Fri, 10 Oct 2008 02:34:28 +0300 >> yurtesen@ispro.net wrote: >> >>> Quoting "Oliver Fromme" : >>> >>>> These features are readily available right now on FreeBSD. >>>> You don't have to code anything. >>> Well with 2 downsides, >> >> Once you actually try and implement these solutions, you'll see that >> your "downsides" are largely figments of your imagination. > > So if it is my imagination, how can I actually convert UFS to ZFS > easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons "See! It's like I said! EASY!" You can do this with /var as well. Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. And finally, the root fs. Whoever says "this is easy" is kidding themselves; it's a pain. You get to make a new filesystem called /boot, and have all sorts of fun. It's really not a snap-fingers-voila thing, and I will gladly argue with anyone who thinks otherwise. Is it do-able though? Yes. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 15:32:27 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CB39106568E for ; Fri, 10 Oct 2008 15:32:27 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id A0B758FC18 for ; Fri, 10 Oct 2008 15:32:26 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 61605 invoked by uid 1001); 10 Oct 2008 11:29:54 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Fri, 10 Oct 2008 11:29:53 -0400 Date: Fri, 10 Oct 2008 11:29:52 -0400 To: Jeremy Chadwick Message-ID: <20081010112952.52b8209b@bhuda.mired.org> In-Reply-To: <20081010144111.GA34609@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> Organization: Meyer Consulting X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; amd64-portbld-freebsd7.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: Sean Bruno , Evren Yurtesen , Shaun Amott , Oliver Fromme , freebsd-hackers@FreeBSD.ORG Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 15:32:27 -0000 On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick wrote: > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > Mike Meyer wrote: > >> On Fri, 10 Oct 2008 02:34:28 +0300 > >> yurtesen@ispro.net wrote: > >> > >>> Quoting "Oliver Fromme" : > >>> > >>>> These features are readily available right now on FreeBSD. > >>>> You don't have to code anything. > >>> Well with 2 downsides, > >> > >> Once you actually try and implement these solutions, you'll see that > >> your "downsides" are largely figments of your imagination. > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > easily? Everybody seems to say that this is easy and that is easy. > > It's not that easy. I really don't know why people are telling you it > is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. > Converting some filesystems are easier than others; /home (if you > create one) for example is generally easy: > > 1) ZFS fs is called foo/home, mounted as /mnt > 2) fstat, ensure nothing is using /home -- if something is, shut it > down or kill it > 3) rsync or cpdup /home files to /mnt > 4) umount /home > 5) zfs set mountpoint=/home foo/home > 6) Restart said processes or daemons > > "See! It's like I said! EASY!" You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: > Now try /usr. Hope you've got /rescue available, because once /usr/lib > and /usr/libexec disappear, you're in trouble. Good luck doing this in > multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. > And finally, the root fs. Whoever says "this is easy" is kidding > themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap & data 3) Build zfs & /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your "easy" 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a good senior admin, it's a lot easier than doing an OS upgrade from source, which should be the next step up from trivial. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 15:42:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E8481065687 for ; Fri, 10 Oct 2008 15:42:53 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA06.westchester.pa.mail.comcast.net (qmta06.westchester.pa.mail.comcast.net [76.96.62.56]) by mx1.freebsd.org (Postfix) with ESMTP id B1B278FC0C for ; Fri, 10 Oct 2008 15:42:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA11.westchester.pa.mail.comcast.net ([76.96.62.36]) by QMTA06.westchester.pa.mail.comcast.net with comcast id R2r31a0050mv7h0563injF; Fri, 10 Oct 2008 15:42:47 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA11.westchester.pa.mail.comcast.net with comcast id R3iL1a00E2P6wsM3X3iLzs; Fri, 10 Oct 2008 15:42:22 +0000 X-Authority-Analysis: v=1.0 c=1 a=nOJ1pGDTh_cA:10 a=QycZ5dHgAAAA:8 a=yiZZxKLCH73OoRo35TcA:9 a=ex2T7DQ-rB3BF-4tJy8A:7 a=QPaZ599o6QYSpy-uI6WeQydEMpkA:4 a=EoioJ0NPDVgA:10 a=SV7veod9ZcQA:10 a=ttedcufwrkMA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 84806C9419; Fri, 10 Oct 2008 08:42:49 -0700 (PDT) Date: Fri, 10 Oct 2008 08:42:49 -0700 From: Jeremy Chadwick To: Mike Meyer Message-ID: <20081010154249.GA35859@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081010112952.52b8209b@bhuda.mired.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Sean Bruno , Evren Yurtesen , Oliver Fromme , freebsd-hackers@FreeBSD.ORG, Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 15:42:53 -0000 On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > On Fri, 10 Oct 2008 07:41:11 -0700 > Jeremy Chadwick wrote: > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > Mike Meyer wrote: > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > >> yurtesen@ispro.net wrote: > > >> > > >>> Quoting "Oliver Fromme" : > > >>> > > >>>> These features are readily available right now on FreeBSD. > > >>>> You don't have to code anything. > > >>> Well with 2 downsides, > > >> > > >> Once you actually try and implement these solutions, you'll see that > > >> your "downsides" are largely figments of your imagination. > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > easily? Everybody seems to say that this is easy and that is easy. > > > > It's not that easy. I really don't know why people are telling you it > > is. > > Maybe because it is? Of course, it *does* require a little prior > planning, but anyone with more than a few months experience as a > sysadmin should be able to deal with it without to much hassle. > > > Converting some filesystems are easier than others; /home (if you > > create one) for example is generally easy: > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > down or kill it > > 3) rsync or cpdup /home files to /mnt > > 4) umount /home > > 5) zfs set mountpoint=/home foo/home > > 6) Restart said processes or daemons > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > Yup. Of course, if you've done it that way, you're not thinking ahead, > because: > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > multi-user, too. > > Oops. You F'ed up. If you'd done a little planning, you would have > realized that / and /usr would be a bit of extra trouble, and planned > accordingly. > > > And finally, the root fs. Whoever says "this is easy" is kidding > > themselves; it's a pain. > > Um, no, it wasn't. Of course, I've been doing this long enough to have > a system set up to make this kind of thing easy. My system disk is on > a mirror, and I do system upgrades by breaking the mirror and > upgrading one disk, making everything work, then putting the mirror > back together. And moving to zfs on root is a lot like a system > upgrade: > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > 2) Repartition the unused drive into /boot, swap & data > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > wiki, just copying /boot and / at this point. > 4) Boot the zfs disk in single user mode. > 5) If 4 fails, boot back to the ufs disk so you're operational while > you contemplate what went wrong, then repeat step 3. Otherwise, go > on to step 6. > 6) Create zfs file systems as appropriate (given that zfs file > systems are cheap, and have lots of cool features that ufs > file systems don't have, you probably want to create more than > you had before, doing thing like putting SQL serves on their > own file system with appropriate blocking, etc, but you'll want to > have figured all this out before starting step 1). > 7) Copy data from the ufs file systems to their new homes, > not forgetting to take them out of /etc/fstab. > 8) Reboot on the zfs disk. > 9) Test until you're happy that everything is working properly, > and be prepared to reboot on the ufs disk if something is broken. > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > add the data partition to the zfs pool so it's mirrored, and > you should have already been using swap. > > This is 10 steps to your "easy" 6, but two of the extra steps are > testing you didn't include, and 1 of the steps is a failure recovery > step that shouldn't be necessary. So - one more step than your easy > process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's "how do I do this? You say it's easy..." comment, and I think his viewpoint is very reasonable. > Yeah, this isn't something you do on a whim. On the other hand, it's > not something that any competent sysadmin would consider a pain. For a > good senior admin, it's a lot easier than doing an OS upgrade from > source, which should be the next step up from trivial. I guess you have a very different definition of "easy". :-) The above procedure, in no way shape or form, will be classified as "easy" by the user (or even junior sysadmin) community, I can assure you of that. I'll also throw this in the mix: the fact that we are *expecting* users to know how to do this is unreasonable. It's even *more* rude to expect that mid-level or senior SAs have to do it "the hard way". Why? I'll explain: I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk partitioning, sectors vs. blocks, slices, filesystems, and whatever else. You want me to do it by hand, say, with bsdlabel -e? Fine, I will -- but I will not be happy about it. I have the knowledge, I know how to do it, so why must the process continue to be a PITA and waste my time? This is exactly why I hate things like the OpenBSD installer: do not make me do it the hard way. We're living in 2008, not 1989. Evolve. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 15:59:36 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 597131065687 for ; Fri, 10 Oct 2008 15:59:36 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id E1D1D8FC0A for ; Fri, 10 Oct 2008 15:59:35 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 62237 invoked by uid 1001); 10 Oct 2008 11:57:04 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Fri, 10 Oct 2008 11:57:03 -0400 Date: Fri, 10 Oct 2008 11:57:02 -0400 To: Evren Yurtesen Message-ID: <20081010115702.05440c6f@bhuda.mired.org> In-Reply-To: <48EF79FE.9010302@ispro.net> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <48EF79FE.9010302@ispro.net> Organization: Meyer Consulting X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; amd64-portbld-freebsd7.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: Sean Bruno , freebsd-hackers@FreeBSD.ORG, Jeremy Chadwick , Oliver Fromme , Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 15:59:36 -0000 On Fri, 10 Oct 2008 18:51:26 +0300 Evren Yurtesen wrote: > The original question (which was lost) was if somebody who has technical > knowledge and coding skills who can put r1soft into the right track so > their software can support FreeBSD. Because r1soft is interested in > supporting FreeBSD. I answered that, and you ignored it: If rlsoft is interested in hiring someone for this, they should sedn a message to freebsd-jobs. If they aren't interested in hiring someone, they need to release the technical information that would be required to do it, and then provide pointers to that information to the FreeBSD community (i.e. - by posting it here, rather than just posting trolls). http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 15:51:22 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54F331065689 for ; Fri, 10 Oct 2008 15:51:22 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: from mail.ispro.net (mail.ispro.net [87.251.0.19]) by mx1.freebsd.org (Postfix) with ESMTP id 503DC8FC25 for ; Fri, 10 Oct 2008 15:51:20 +0000 (UTC) (envelope-from yurtesen@ispro.net) Received: (qmail 66755 invoked by uid 399); 10 Oct 2008 15:51:39 -0000 Received: from perpetual.yok.utu.fi (HELO ?130.232.138.155?) (yurtesen@ispro.net@130.232.138.155) by mail.ispro.net with ESMTPAM; 10 Oct 2008 15:51:39 -0000 X-Originating-IP: 130.232.138.155 Message-ID: <48EF79FE.9010302@ispro.net> Date: Fri, 10 Oct 2008 18:51:26 +0300 From: Evren Yurtesen User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Jeremy Chadwick References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> In-Reply-To: <20081010144111.GA34609@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 10 Oct 2008 16:03:49 +0000 Cc: Sean Bruno , freebsd-hackers@FreeBSD.ORG, Mike Meyer , Oliver Fromme , Shaun Amott Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 15:51:22 -0000 I just wanted to say thanks to all the replies to this thread. It has been insightful even though the suggestions I have received were not really answers to what I asked. I dont see any reason why we should continue to argue about if this can be done using ZFS or Hammer or any other filesystem. The fact is that it can be done eventually with something else if one has the will, time and money to put into it. One can even write his own filesystem code for it eh? The original question (which was lost) was if somebody who has technical knowledge and coding skills who can put r1soft into the right track so their software can support FreeBSD. Because r1soft is interested in supporting FreeBSD. Most of you probably know that FreeBSD support in products of commercial companies are scarce already. This is causing reductions in user base and popularity, although I am not saying that the more users are the better or I am not saying that popularity is everything. But you have to be able to combine one issue with another and see what the results might be in the long run. For example, wouldnt it be good for FreeBSD itself if Oracle supported FreeBSD? While I cant make Oracle to support FreeBSD, there is a company out there with a popular and quality product which is interested in supporting FreeBSD in their products. So why not take advantage of the situation? While I didnt think about some details when I sent the first post, some people thought that I would want somebody to code for r1soft for free so they can sell the software. No I do not and did not expect anybody to do anything for free. So, to summarize, if there is anybody who wants to help the 'actual' issue here, either by getting payed or just giving free hints and tips to r1soft then please contact me (I can forward your contact info) or with r1soft. Dag-Erling Smørgrav already told me that I can send his info and I have sent his contact info to r1soft for this issue. I hope we can now stop arguing about all the other things :) It wasnt my intention to start such arguments. Thanks, Evren From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 16:25:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF0931065693 for ; Fri, 10 Oct 2008 16:25:02 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 4ADE08FC16 for ; Fri, 10 Oct 2008 16:25:01 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 62662 invoked by uid 1001); 10 Oct 2008 12:22:29 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Fri, 10 Oct 2008 12:22:28 -0400 Date: Fri, 10 Oct 2008 12:22:28 -0400 To: freebsd-hackers@freebsd.org Message-ID: <20081010122228.355c2c3e@bhuda.mired.org> In-Reply-To: <20081010154249.GA35859@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> Organization: Meyer Consulting X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; amd64-portbld-freebsd7.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 16:25:02 -0000 On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick wrote: > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > On Fri, 10 Oct 2008 07:41:11 -0700 > > Jeremy Chadwick wrote: > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > Mike Meyer wrote: > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > >> yurtesen@ispro.net wrote: > > > >> > > > >>> Quoting "Oliver Fromme" : > > > >>> > > > >>>> These features are readily available right now on FreeBSD. > > > >>>> You don't have to code anything. > > > >>> Well with 2 downsides, > > > >> > > > >> Once you actually try and implement these solutions, you'll see that > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > It's not that easy. I really don't know why people are telling you it > > > is. > > > > Maybe because it is? Of course, it *does* require a little prior > > planning, but anyone with more than a few months experience as a > > sysadmin should be able to deal with it without to much hassle. > > > > > Converting some filesystems are easier than others; /home (if you > > > create one) for example is generally easy: > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > down or kill it > > > 3) rsync or cpdup /home files to /mnt > > > 4) umount /home > > > 5) zfs set mountpoint=/home foo/home > > > 6) Restart said processes or daemons > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > because: > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > multi-user, too. > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > realized that / and /usr would be a bit of extra trouble, and planned > > accordingly. > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > themselves; it's a pain. > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > a system set up to make this kind of thing easy. My system disk is on > > a mirror, and I do system upgrades by breaking the mirror and > > upgrading one disk, making everything work, then putting the mirror > > back together. And moving to zfs on root is a lot like a system > > upgrade: > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > 2) Repartition the unused drive into /boot, swap & data > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > wiki, just copying /boot and / at this point. > > 4) Boot the zfs disk in single user mode. > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > on to step 6. > > 6) Create zfs file systems as appropriate (given that zfs file > > systems are cheap, and have lots of cool features that ufs > > file systems don't have, you probably want to create more than > > you had before, doing thing like putting SQL serves on their > > own file system with appropriate blocking, etc, but you'll want to > > have figured all this out before starting step 1). > > 7) Copy data from the ufs file systems to their new homes, > > not forgetting to take them out of /etc/fstab. > > 8) Reboot on the zfs disk. > > 9) Test until you're happy that everything is working properly, > > and be prepared to reboot on the ufs disk if something is broken. > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > add the data partition to the zfs pool so it's mirrored, and > > you should have already been using swap. > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > testing you didn't include, and 1 of the steps is a failure recovery > > step that shouldn't be necessary. So - one more step than your easy > > process. > > Of course, the part you seem to be (intentionally?) forgetting: most > people are not using gmirror. There is no 2nd disk. They have one disk > with a series of UFS2 filesystems, and they want to upgrade. That's how > I read Evren's "how do I do this? You say it's easy..." comment, and I > think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the server that all this data is going to flow to, for a backup solution. Compared to that, the cost of a few spare disks and the work to install them are trivial. > > Yeah, this isn't something you do on a whim. On the other hand, it's > > not something that any competent sysadmin would consider a pain. For a > > good senior admin, it's a lot easier than doing an OS upgrade from > > source, which should be the next step up from trivial. > I guess you have a very different definition of "easy". :-) Given that mine is based on years of working with the kinds of backup solutions that Evren is asking for: ones that an enterprise deploys for backing up a data center, the answer may well be "yes". > The above procedure, in no way shape or form, will be classified as > "easy" by the user (or even junior sysadmin) community, I can assure you > of that. I never said it would be easy for a user. Then again, your average user doesn't do backups, and wouldn't know a continuous backup solution from a credit default swap. We're not talking about ghosting a disk partition for a backup, we're talking about enterprise-level backup solutions for data centers. People deploying those kinds of solutions tend to have multiple senior sysadmins around. I wouldn't expect a junior admin to call it easy. At least, not the first two or three times. If they still have problems with it after that, they should find a new career path, as they aren't ever going to advance beyond junior. > I'll also throw this in the mix: the fact that we are *expecting* users > to know how to do this is unreasonable. It's even *more* rude to expect Um, is anyone expecting users to do this? I'm not. ZFS is still marked as "experimental" in FreeBSD. That means that, among other things, it's not really well-supported by the installer, etc. Nuts, as of January of this year, there wasn't an operating system on the planet that would install and boot from ZFS. I'm willing to jump through some hoops to get ZFS's advantages. Those happen to include some things that go a long way to solving Zefren's problems, so it was suggested as the basis for such (not by me, mind you). Having done the conversion, and found it easy, I responded when he asked how hard it was. But I'd never recommend this for your average user - which pretty much excludes anyone contemplating continuous backup solutions. > that mid-level or senior SAs have to do it "the hard way". Why? I'll > explain: > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > partitioning, sectors vs. blocks, slices, filesystems, and whatever > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > will -- but I will not be happy about it. I have the knowledge, I > know how to do it, so why must the process continue to be a PITA and > waste my time? Did I ever mention bsdlabel? But in any case, ZFS makes pretty much *all* that crap obsolete. You still have to deal with getting a boot loader installed, but after that, you never have to worry about partitioning, blocks, sectors, or slices again - until you go to an operating system that doesn't have ZFS. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 18:28:55 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEBC6106569E for ; Fri, 10 Oct 2008 18:28:55 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from smtp.timeweb.ru (smtp.timeweb.ru [217.170.79.85]) by mx1.freebsd.org (Postfix) with ESMTP id 7A8468FC2D for ; Fri, 10 Oct 2008 18:28:55 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from [213.148.20.85] (helo=hive.panopticon) by smtp.timeweb.ru with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1KoLzu-0002nZ-JY; Fri, 10 Oct 2008 21:41:54 +0400 Received: from hades.panopticon (hades.panopticon [192.168.0.32]) by hive.panopticon (Postfix) with ESMTP id B7F5E1255A; Fri, 10 Oct 2008 21:41:14 +0400 (MSD) Received: by hades.panopticon (Postfix, from userid 1000) id D544017031; Fri, 10 Oct 2008 21:42:04 +0400 (MSD) Date: Fri, 10 Oct 2008 21:42:04 +0400 From: Dmitry Marakasov To: freebsd-hackers@FreeBSD.org, freebsd-ports@freebsd.org Message-ID: <20081010174204.GB90757@hades.panopticon> References: <48C52718.5080807@sun.com> <48C8F051.7060107@sun.com> <20080919082719.GH676@e.0x20.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20080919082719.GH676@e.0x20.net> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Subject: Re: VirtualBox looks for FreeBSD developer X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 18:28:55 -0000 Hi! Little time ago I was misleaded by the certain people and got an idea that VirtualBox actually works on FreeBSD, so I've made a draft port for it. It doesn't actually work, but since I've spent several hours hacking it and made bunch of (likely) useful patches, here it is, feel free to use it for any purpose. I hope someone of kernel hackers will make it work actually ;) Summary: - installation doesn't work - only 7.0 (7.x maybe) / i386 is supported - VM doesn't run - You'll be able to run QT4 GUI and some tests if you're lucky :) See README.txt inside for additional details. http://amdmi3.ru/files/virtualbox-port.tar.gz -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://www.amdmi3.ru From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 20:25:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0405A1065689 for ; Fri, 10 Oct 2008 20:25:04 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from hs-out-0708.google.com (hs-out-0708.google.com [64.233.178.243]) by mx1.freebsd.org (Postfix) with ESMTP id B75638FC1A for ; Fri, 10 Oct 2008 20:25:03 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: by hs-out-0708.google.com with SMTP id 54so267644hsz.11 for ; Fri, 10 Oct 2008 13:25:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=dw+quje5Pi9zkGKZLomYkCBtj6qUr+NE48q+8PKTHbQ=; b=gByjc99epUIyyCrOIFB14AhRnTySqgQLPpf7LHR1WvWz7XS/uzHSMyjJmXAKH46+P2 2+npcsTPT0e7FLUBuAfKIkSigWVbR490TOz8M/ulxaase6AFAFwXmrQZtm4ra1BMR8O3 F3UJ2LjjtMpQx3ZV0rrL/KcnBBVCR6Fpkc61o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=Xbc+KD6xbCXJdaA9RM0jrDfQYc0AyaRH3yn0fnO252ti3Aa6o9dQYexUavMeTAhOCJ +QQx2ETsyin36BHtc34zik4yfLsw+lIqipvdhn2CZeBq+mrAGo0b37q66DTgWby83pMs M6BWX/FU9A9dAD9ywGgdwOLXskDTvXa9O9Ymw= Received: by 10.90.53.1 with SMTP id b1mr640996aga.51.1223670302624; Fri, 10 Oct 2008 13:25:02 -0700 (PDT) Received: by 10.90.94.14 with HTTP; Fri, 10 Oct 2008 13:25:02 -0700 (PDT) Message-ID: <290865fd0810101325kbd4caber6b132253784523a5@mail.gmail.com> Date: Fri, 10 Oct 2008 13:25:02 -0700 From: "alan yang" To: freebsd-questions@freebsd.org, freebsd-hackers@freebsd.org In-Reply-To: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <290865fd0810090846y57bbdc1fs3db5c5334fe80c09@mail.gmail.com> Cc: Subject: Re: setkey panic freebsd7 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 20:25:04 -0000 sorry, /usr/local/sbin/setkey failed on parsing that specific add, not panic. no specific info, just say parse failed. maybe something is not supported ...? On Thu, Oct 9, 2008 at 8:46 AM, alan yang wrote: > i wonder people ran into similar issue on setkey with freebsd7 that > panic at ~/crypto/sha1.c:263 within sha1_result() > digest[0] = ctxt->h.b8[3]; digest[1] = ctxt->h.b8[2]; > > on the following sadb add with setkey: > add 192.168.0.101 192.168.0.110 esp-old 0x10001 -m any -E des-cbc > "12345678" -A keyed-sha1 "12345678123456781234" > > thanks in advance on any hints. > From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 21:30:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2BD51065692 for ; Fri, 10 Oct 2008 21:30:53 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.208.78.105]) by mx1.freebsd.org (Postfix) with ESMTP id BC5ED8FC0C for ; Fri, 10 Oct 2008 21:30:53 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.3/8.14.3) with ESMTP id m9ALUgDN096853; Fri, 10 Oct 2008 14:30:42 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.3/8.14.3/Submit) id m9ALUg46096852; Fri, 10 Oct 2008 14:30:42 -0700 (PDT) (envelope-from sgk) Date: Fri, 10 Oct 2008 14:30:42 -0700 From: Steve Kargl To: freebsd-hackers@freebsd.org Message-ID: <20081010213042.GA96822@troutmask.apl.washington.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Fri, 10 Oct 2008 22:02:54 +0000 Cc: sgk@troutmask.apl.washington.edu Subject: HPC with ULE vs 4BSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 21:30:54 -0000 Yes, this is a long email. In working with a colleague to diagnosis poor performance of his MPI code, we've discovered that ULE is drastically inferior to 4BSD in utilizing a system with 2 physical cpus (opteron) and a total of 8 cores. We have observed this problem with the Open MPI implementation of MPI and with the MPICH2 implementation. Note, I am using the exact same hardware and FreeBSD-current code dated Sep 22, 2008. The only difference in the kernel config file is whether ULE or 4BSD is used. Using the following command, % time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log we have ULE --> 546.99 real 0.02 user 0.03 sys 4BSD -> 218.96 real 0.03 user 0.02 sys where the machinefile simply tells Open MPI to launch 8 jobs on the local node. Test_mpi uses MPI's scatter, gather, and all_to_all functions to transmit various arrays between the 8 jobs. To get meaningful numbers, a number of iterations are done in a tight loop. With ULE, a snapshot of top(1) shows last pid: 33765; load averages: 7.98, 7.51, 5.63 up 10+03:20:30 13:13:56 43 processes: 9 running, 34 sleeping CPU: 68.6% user, 0.0% nice, 18.9% system, 0.0% interrupt, 12.5% idle Mem: 296M Active, 20M Inact, 192M Wired, 1112K Cache, 132M Buf, 31G Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 33743 kargl 1 118 0 300M 22788K CPU7 7 4:48 100.00% Test_mpi 33747 kargl 1 118 0 300M 22820K CPU3 3 4:43 100.00% Test_mpi 33742 kargl 1 118 0 300M 22692K CPU5 5 4:42 100.00% Test_mpi 33744 kargl 1 117 0 300M 22752K CPU6 6 4:29 100.00% Test_mpi 33748 kargl 1 117 0 300M 22768K CPU2 2 4:31 96.39% Test_mpi 33741 kargl 1 112 0 299M 43628K CPU1 1 4:40 80.08% Test_mpi 33745 kargl 1 113 0 300M 44272K RUN 0 4:27 76.17% Test_mpi 33746 kargl 1 109 0 300M 22740K RUN 0 4:25 57.86% Test_mpi 33749 kargl 1 44 0 8196K 2280K CPU4 4 0:00 0.20% top while with 4BSD, a snapshot of top(1) shows last pid: 1019; load averages: 7.24, 3.05, 1.25 up 0+00:04:40 13:27:09 43 processes: 9 running, 34 sleeping CPU: 45.4% user, 0.0% nice, 54.5% system, 0.1% interrupt, 0.0% idle Mem: 329M Active, 33M Inact, 107M Wired, 104K Cache, 14M Buf, 31G Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 1012 kargl 1 126 0 300M 44744K CPU6 6 2:16 99.07% Test_mpi 1016 kargl 1 126 0 314M 59256K RUN 4 2:16 99.02% Test_mpi 1011 kargl 1 126 0 300M 44652K CPU5 5 2:16 99.02% Test_mpi 1013 kargl 1 126 0 300M 44680K CPU2 2 2:16 99.02% Test_mpi 1010 kargl 1 126 0 300M 44740K CPU7 7 2:16 99.02% Test_mpi 1009 kargl 1 126 0 299M 43884K CPU0 0 2:16 98.97% Test_mpi 1014 kargl 1 126 0 300M 44664K CPU1 1 2:16 98.97% Test_mpi 1015 kargl 1 126 0 300M 44620K CPU3 3 2:16 98.93% Test_mpi 989 kargl 1 96 0 8196K 2460K CPU4 4 0:00 0.10% top Notice the interesting, or even perhaps odd, scheduling with ULE that results in a 20 second gap between the "fastest" job (4:48) and the "slowest" (4:25). With ULE, 2 Test_mpi jobs are always scheduled on the same core while one core remains idle. Also, note the difference in the reported load averages. Various stats are generated by and collected from executing the MPI program With ULE, the numbers are Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) 8 800000 3125 100 scatter 12.58386 0.24251367 8 800000 3125 100 all_to_all 17.24503 0.17696444 8 800000 3125 100 gather 14.82058 0.20591355 8 1600000 6250 100 scatter 28.25922 0.21598316 8 1600000 6250 100 all_to_all 1985.74915 0.00307366 8 1600000 6250 100 gather 30.42038 0.20063902 8 2400000 9375 100 scatter 44.65615 0.20501709 8 2400000 9375 100 all_to_all 16.09386 0.56886748 8 2400000 9375 100 gather 44.38801 0.20625555 8 3200000 12500 100 scatter 60.04160 0.20330956 8 3200000 12500 100 all_to_all 2157.10010 0.00565900 8 3200000 12500 100 gather 59.72242 0.20439614 8 4000000 15625 100 scatter 86.65769 0.17608117 8 4000000 15625 100 all_to_all 2081.25195 0.00733154 8 4000000 15625 100 gather 27.47257 0.55541896 8 4800000 18750 100 scatter 33.02306 0.55447768 8 4800000 18750 100 all_to_all 200.09908 0.09150740 8 4800000 18750 100 gather 91.08742 0.20102168 8 5600000 21875 100 scatter 109.82005 0.19452098 8 5600000 21875 100 all_to_all 76.87574 0.27788095 8 5600000 21875 100 gather 41.67106 0.51264128 8 6400000 25000 100 scatter 26.92482 0.90674917 8 6400000 25000 100 all_to_all 64.74528 0.37707868 8 6400000 25000 100 gather 41.29724 0.59117904 and with 4BSD, the numbers are Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) 8 800000 3125 100 scatter 21.33697 0.14302677 8 800000 3125 100 all_to_all 3941.39624 0.00077428 8 800000 3125 100 gather 24.75520 0.12327747 8 1600000 6250 100 scatter 45.20134 0.13502954 8 1600000 6250 100 all_to_all 1987.94348 0.00307027 8 1600000 6250 100 gather 42.02498 0.14523541 8 2400000 9375 100 scatter 63.03553 0.14523989 8 2400000 9375 100 all_to_all 2015.19580 0.00454312 8 2400000 9375 100 gather 66.72807 0.13720272 8 3200000 12500 100 scatter 91.90541 0.13282169 8 3200000 12500 100 all_to_all 2029.62622 0.00601442 8 3200000 12500 100 gather 87.99693 0.13872112 8 4000000 15625 100 scatter 107.48991 0.14195556 8 4000000 15625 100 all_to_all 1970.66907 0.00774295 8 4000000 15625 100 gather 110.70226 0.13783630 8 4800000 18750 100 scatter 140.39014 0.13042616 8 4800000 18750 100 all_to_all 2401.80054 0.00762367 8 4800000 18750 100 gather 134.60948 0.13602717 8 5600000 21875 100 scatter 152.31958 0.14024661 8 5600000 21875 100 all_to_all 2379.12207 0.00897907 8 5600000 21875 100 gather 154.60051 0.13817745 8 6400000 25000 100 scatter 190.03561 0.12847099 8 6400000 25000 100 all_to_all 2661.36963 0.00917350 8 6400000 25000 100 gather 183.08250 0.13335006 Noting that all communication is over the memory bus, a comparison of the Bandwidth columns suggests that ULE is causing the MPI jobs to stall waiting for data. This has potentially serious negative impact on clusters used for HPC. -- Steve From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 10 22:29:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 865161065690 for ; Fri, 10 Oct 2008 22:29:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 334C28FC1B for ; Fri, 10 Oct 2008 22:29:06 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA13.westchester.pa.mail.comcast.net ([76.96.62.52]) by QMTA07.westchester.pa.mail.comcast.net with comcast id R8Wt1a00517dt5G57AV6UC; Fri, 10 Oct 2008 22:29:06 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA13.westchester.pa.mail.comcast.net with comcast id RAV41a00P2P6wsM3ZAV59K; Fri, 10 Oct 2008 22:29:06 +0000 X-Authority-Analysis: v=1.0 c=1 a=QycZ5dHgAAAA:8 a=C1AWhVUT42d_71i-uLQA:9 a=AI56OMMXf-ldpafupfgA:7 a=ycrOskimge5EXfuqaha76G6AFBcA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 6D004C9419; Fri, 10 Oct 2008 15:29:04 -0700 (PDT) Date: Fri, 10 Oct 2008 15:29:04 -0700 From: Jeremy Chadwick To: Steve Kargl Message-ID: <20081010222904.GA44873@icarus.home.lan> References: <20081010213042.GA96822@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081010213042.GA96822@troutmask.apl.washington.edu> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org, jeff@freebsd.org Subject: Re: HPC with ULE vs 4BSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2008 22:29:07 -0000 On Fri, Oct 10, 2008 at 02:30:42PM -0700, Steve Kargl wrote: > Yes, this is a long email. > > In working with a colleague to diagnosis poor performance of > his MPI code, we've discovered that ULE is drastically inferior > to 4BSD in utilizing a system with 2 physical cpus (opteron) and > a total of 8 cores. We have observed this problem with the Open MPI > implementation of MPI and with the MPICH2 implementation. > > Note, I am using the exact same hardware and FreeBSD-current > code dated Sep 22, 2008. The only difference in the kernel > config file is whether ULE or 4BSD is used. > > Using the following command, > > % time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log > > we have > > ULE --> 546.99 real 0.02 user 0.03 sys > 4BSD -> 218.96 real 0.03 user 0.02 sys > > where the machinefile simply tells Open MPI to launch 8 jobs on the > local node. Test_mpi uses MPI's scatter, gather, and all_to_all > functions to transmit various arrays between the 8 jobs. To get > meaningful numbers, a number of iterations are done in a tight loop. > > With ULE, a snapshot of top(1) shows > > last pid: 33765; load averages: 7.98, 7.51, 5.63 up 10+03:20:30 13:13:56 > 43 processes: 9 running, 34 sleeping > CPU: 68.6% user, 0.0% nice, 18.9% system, 0.0% interrupt, 12.5% idle > Mem: 296M Active, 20M Inact, 192M Wired, 1112K Cache, 132M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 33743 kargl 1 118 0 300M 22788K CPU7 7 4:48 100.00% Test_mpi > 33747 kargl 1 118 0 300M 22820K CPU3 3 4:43 100.00% Test_mpi > 33742 kargl 1 118 0 300M 22692K CPU5 5 4:42 100.00% Test_mpi > 33744 kargl 1 117 0 300M 22752K CPU6 6 4:29 100.00% Test_mpi > 33748 kargl 1 117 0 300M 22768K CPU2 2 4:31 96.39% Test_mpi > 33741 kargl 1 112 0 299M 43628K CPU1 1 4:40 80.08% Test_mpi > 33745 kargl 1 113 0 300M 44272K RUN 0 4:27 76.17% Test_mpi > 33746 kargl 1 109 0 300M 22740K RUN 0 4:25 57.86% Test_mpi > 33749 kargl 1 44 0 8196K 2280K CPU4 4 0:00 0.20% top > > while with 4BSD, a snapshot of top(1) shows > > last pid: 1019; load averages: 7.24, 3.05, 1.25 up 0+00:04:40 13:27:09 > 43 processes: 9 running, 34 sleeping > CPU: 45.4% user, 0.0% nice, 54.5% system, 0.1% interrupt, 0.0% idle > Mem: 329M Active, 33M Inact, 107M Wired, 104K Cache, 14M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1012 kargl 1 126 0 300M 44744K CPU6 6 2:16 99.07% Test_mpi > 1016 kargl 1 126 0 314M 59256K RUN 4 2:16 99.02% Test_mpi > 1011 kargl 1 126 0 300M 44652K CPU5 5 2:16 99.02% Test_mpi > 1013 kargl 1 126 0 300M 44680K CPU2 2 2:16 99.02% Test_mpi > 1010 kargl 1 126 0 300M 44740K CPU7 7 2:16 99.02% Test_mpi > 1009 kargl 1 126 0 299M 43884K CPU0 0 2:16 98.97% Test_mpi > 1014 kargl 1 126 0 300M 44664K CPU1 1 2:16 98.97% Test_mpi > 1015 kargl 1 126 0 300M 44620K CPU3 3 2:16 98.93% Test_mpi > 989 kargl 1 96 0 8196K 2460K CPU4 4 0:00 0.10% top > > Notice the interesting, or even perhaps odd, scheduling with ULE that results > in a 20 second gap between the "fastest" job (4:48) and the "slowest" (4:25). > With ULE, 2 Test_mpi jobs are always scheduled on the same core while one > core remains idle. Also, note the difference in the reported load averages. > > Various stats are generated by and collected from executing the MPI program > With ULE, the numbers are > > Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) > 8 800000 3125 100 scatter 12.58386 0.24251367 > 8 800000 3125 100 all_to_all 17.24503 0.17696444 > 8 800000 3125 100 gather 14.82058 0.20591355 > > 8 1600000 6250 100 scatter 28.25922 0.21598316 > 8 1600000 6250 100 all_to_all 1985.74915 0.00307366 > 8 1600000 6250 100 gather 30.42038 0.20063902 > > 8 2400000 9375 100 scatter 44.65615 0.20501709 > 8 2400000 9375 100 all_to_all 16.09386 0.56886748 > 8 2400000 9375 100 gather 44.38801 0.20625555 > > 8 3200000 12500 100 scatter 60.04160 0.20330956 > 8 3200000 12500 100 all_to_all 2157.10010 0.00565900 > 8 3200000 12500 100 gather 59.72242 0.20439614 > > 8 4000000 15625 100 scatter 86.65769 0.17608117 > 8 4000000 15625 100 all_to_all 2081.25195 0.00733154 > 8 4000000 15625 100 gather 27.47257 0.55541896 > > 8 4800000 18750 100 scatter 33.02306 0.55447768 > 8 4800000 18750 100 all_to_all 200.09908 0.09150740 > 8 4800000 18750 100 gather 91.08742 0.20102168 > > 8 5600000 21875 100 scatter 109.82005 0.19452098 > 8 5600000 21875 100 all_to_all 76.87574 0.27788095 > 8 5600000 21875 100 gather 41.67106 0.51264128 > > 8 6400000 25000 100 scatter 26.92482 0.90674917 > 8 6400000 25000 100 all_to_all 64.74528 0.37707868 > 8 6400000 25000 100 gather 41.29724 0.59117904 > > and with 4BSD, the numbers are > > Procs Array size Kb Iters Function Bandwidth(Mbs) Time(s) > 8 800000 3125 100 scatter 21.33697 0.14302677 > 8 800000 3125 100 all_to_all 3941.39624 0.00077428 > 8 800000 3125 100 gather 24.75520 0.12327747 > > 8 1600000 6250 100 scatter 45.20134 0.13502954 > 8 1600000 6250 100 all_to_all 1987.94348 0.00307027 > 8 1600000 6250 100 gather 42.02498 0.14523541 > > 8 2400000 9375 100 scatter 63.03553 0.14523989 > 8 2400000 9375 100 all_to_all 2015.19580 0.00454312 > 8 2400000 9375 100 gather 66.72807 0.13720272 > > 8 3200000 12500 100 scatter 91.90541 0.13282169 > 8 3200000 12500 100 all_to_all 2029.62622 0.00601442 > 8 3200000 12500 100 gather 87.99693 0.13872112 > > 8 4000000 15625 100 scatter 107.48991 0.14195556 > 8 4000000 15625 100 all_to_all 1970.66907 0.00774295 > 8 4000000 15625 100 gather 110.70226 0.13783630 > > 8 4800000 18750 100 scatter 140.39014 0.13042616 > 8 4800000 18750 100 all_to_all 2401.80054 0.00762367 > 8 4800000 18750 100 gather 134.60948 0.13602717 > > 8 5600000 21875 100 scatter 152.31958 0.14024661 > 8 5600000 21875 100 all_to_all 2379.12207 0.00897907 > 8 5600000 21875 100 gather 154.60051 0.13817745 > > 8 6400000 25000 100 scatter 190.03561 0.12847099 > 8 6400000 25000 100 all_to_all 2661.36963 0.00917350 > 8 6400000 25000 100 gather 183.08250 0.13335006 > > Noting that all communication is over the memory bus, a comparison of > the Bandwidth columns suggests that ULE is causing the MPI jobs to stall > waiting for data. This has potentially serious negative impact on > clusters used for HPC. What surprises me is that you didn't CC the individual who wrote ULE: Jeff Roberson. :-) I've CC'd him here. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 10:35:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B28E51065694 for ; Sat, 11 Oct 2008 10:35:18 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id 419FB8FC17 for ; Sat, 11 Oct 2008 10:35:18 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1Koboa-000CyO-8l; Sat, 11 Oct 2008 12:35:16 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Mike Meyer In-reply-to: <20081010122228.355c2c3e@bhuda.mired.org> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> Comments: In-reply-to Mike Meyer message dated "Fri, 10 Oct 2008 12:22:28 -0400." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 11 Oct 2008 12:35:16 +0200 From: Danny Braniss Message-ID: Cc: freebsd-hackers@freebsd.org Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 10:35:18 -0000 > On Fri, 10 Oct 2008 08:42:49 -0700 > Jeremy Chadwick wrote: > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > Jeremy Chadwick wrote: > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > Mike Meyer wrote: > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > >> yurtesen@ispro.net wrote: > > > > >> > > > > >>> Quoting "Oliver Fromme" : > > > > >>> > > > > >>>> These features are readily available right now on FreeBSD. > > > > >>>> You don't have to code anything. > > > > >>> Well with 2 downsides, > > > > >> > > > > >> Once you actually try and implement these solutions, you'll see that > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > is. > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > planning, but anyone with more than a few months experience as a > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > create one) for example is generally easy: > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > down or kill it > > > > 3) rsync or cpdup /home files to /mnt > > > > 4) umount /home > > > > 5) zfs set mountpoint=/home foo/home > > > > 6) Restart said processes or daemons > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > because: > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > multi-user, too. > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > realized that / and /usr would be a bit of extra trouble, and planned > > > accordingly. > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > themselves; it's a pain. > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > a system set up to make this kind of thing easy. My system disk is on > > > a mirror, and I do system upgrades by breaking the mirror and > > > upgrading one disk, making everything work, then putting the mirror > > > back together. And moving to zfs on root is a lot like a system > > > upgrade: > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > 2) Repartition the unused drive into /boot, swap & data > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > wiki, just copying /boot and / at this point. > > > 4) Boot the zfs disk in single user mode. > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > on to step 6. > > > 6) Create zfs file systems as appropriate (given that zfs file > > > systems are cheap, and have lots of cool features that ufs > > > file systems don't have, you probably want to create more than > > > you had before, doing thing like putting SQL serves on their > > > own file system with appropriate blocking, etc, but you'll want to > > > have figured all this out before starting step 1). > > > 7) Copy data from the ufs file systems to their new homes, > > > not forgetting to take them out of /etc/fstab. > > > 8) Reboot on the zfs disk. > > > 9) Test until you're happy that everything is working properly, > > > and be prepared to reboot on the ufs disk if something is broken. > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > add the data partition to the zfs pool so it's mirrored, and > > > you should have already been using swap. > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > testing you didn't include, and 1 of the steps is a failure recovery > > > step that shouldn't be necessary. So - one more step than your easy > > > process. > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > people are not using gmirror. There is no 2nd disk. They have one disk > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > think his viewpoint is very reasonable. > > Granted, most people don't think about system upgrades when they build > a system, so they wind up having to do extra work. In particular, > Evren is talking about spending thousands of dollars on proprietary > software, not to mention the cost of the server that all this data is > going to flow to, for a backup solution. Compared to that, the cost of > a few spare disks and the work to install them are trivial. > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > not something that any competent sysadmin would consider a pain. For a > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > source, which should be the next step up from trivial. > > I guess you have a very different definition of "easy". :-) > > Given that mine is based on years of working with the kinds of backup > solutions that Evren is asking for: ones that an enterprise deploys > for backing up a data center, the answer may well be "yes". > > > The above procedure, in no way shape or form, will be classified as > > "easy" by the user (or even junior sysadmin) community, I can assure you > > of that. > > I never said it would be easy for a user. Then again, your average > user doesn't do backups, and wouldn't know a continuous backup > solution from a credit default swap. We're not talking about ghosting > a disk partition for a backup, we're talking about enterprise-level > backup solutions for data centers. People deploying those kinds of > solutions tend to have multiple senior sysadmins around. > > I wouldn't expect a junior admin to call it easy. At least, not the > first two or three times. If they still have problems with it after > that, they should find a new career path, as they aren't ever going to > advance beyond junior. > > > I'll also throw this in the mix: the fact that we are *expecting* users > > to know how to do this is unreasonable. It's even *more* rude to expect > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > as "experimental" in FreeBSD. That means that, among other things, > it's not really well-supported by the installer, etc. Nuts, as of > January of this year, there wasn't an operating system on the planet > that would install and boot from ZFS. > > I'm willing to jump through some hoops to get ZFS's advantages. Those > happen to include some things that go a long way to solving Zefren's > problems, so it was suggested as the basis for such (not by me, mind > you). Having done the conversion, and found it easy, I responded when > he asked how hard it was. > > But I'd never recommend this for your average user - which pretty much > excludes anyone contemplating continuous backup solutions. > > > that mid-level or senior SAs have to do > it "the hard way". Why? I'll > > explain: > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > will -- but I will not be happy about it. I have the knowledge, I > > know how to do it, so why must the process continue to be a PITA and > > waste my time? > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > *all* that crap obsolete. You still have to deal with getting a boot > loader installed, but after that, you never have to worry about > partitioning, blocks, sectors, or slices again - until you go to an > operating system that doesn't have ZFS. so can Freebsd boot off a ZFS root? in stable? current? ... danny From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 10:44:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D82A810656A7 for ; Sat, 11 Oct 2008 10:44:11 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA01.westchester.pa.mail.comcast.net (qmta01.westchester.pa.mail.comcast.net [76.96.62.16]) by mx1.freebsd.org (Postfix) with ESMTP id 7C1108FC1C for ; Sat, 11 Oct 2008 10:44:11 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA02.westchester.pa.mail.comcast.net ([76.96.62.19]) by QMTA01.westchester.pa.mail.comcast.net with comcast id RNfp1a0030QuhwU51NkAtH; Sat, 11 Oct 2008 10:44:10 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA02.westchester.pa.mail.comcast.net with comcast id RNk91a0022P6wsM3NNk98e; Sat, 11 Oct 2008 10:44:10 +0000 X-Authority-Analysis: v=1.0 c=1 a=nOJ1pGDTh_cA:10 a=QycZ5dHgAAAA:8 a=l6LQfVVH01YERIII-cEA:9 a=GDxiEFyuD8kOKKtZWhMA:7 a=VU_J0uaJtxqMAlOzYRp2fLz1YOQA:4 a=EoioJ0NPDVgA:10 a=SV7veod9ZcQA:10 a=ttedcufwrkMA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 26F2AC9419; Sat, 11 Oct 2008 03:44:09 -0700 (PDT) Date: Sat, 11 Oct 2008 03:44:09 -0700 From: Jeremy Chadwick To: Danny Braniss Message-ID: <20081011104409.GA58698@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 10:44:11 -0000 On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > On Fri, 10 Oct 2008 08:42:49 -0700 > > Jeremy Chadwick wrote: > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > Jeremy Chadwick wrote: > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > Mike Meyer wrote: > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > >> yurtesen@ispro.net wrote: > > > > > >> > > > > > >>> Quoting "Oliver Fromme" : > > > > > >>> > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > >>>> You don't have to code anything. > > > > > >>> Well with 2 downsides, > > > > > >> > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > is. > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > planning, but anyone with more than a few months experience as a > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > create one) for example is generally easy: > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > down or kill it > > > > > 3) rsync or cpdup /home files to /mnt > > > > > 4) umount /home > > > > > 5) zfs set mountpoint=/home foo/home > > > > > 6) Restart said processes or daemons > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > because: > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > multi-user, too. > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > accordingly. > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > themselves; it's a pain. > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > a system set up to make this kind of thing easy. My system disk is on > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > upgrading one disk, making everything work, then putting the mirror > > > > back together. And moving to zfs on root is a lot like a system > > > > upgrade: > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > 2) Repartition the unused drive into /boot, swap & data > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > wiki, just copying /boot and / at this point. > > > > 4) Boot the zfs disk in single user mode. > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > on to step 6. > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > systems are cheap, and have lots of cool features that ufs > > > > file systems don't have, you probably want to create more than > > > > you had before, doing thing like putting SQL serves on their > > > > own file system with appropriate blocking, etc, but you'll want to > > > > have figured all this out before starting step 1). > > > > 7) Copy data from the ufs file systems to their new homes, > > > > not forgetting to take them out of /etc/fstab. > > > > 8) Reboot on the zfs disk. > > > > 9) Test until you're happy that everything is working properly, > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > add the data partition to the zfs pool so it's mirrored, and > > > > you should have already been using swap. > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > step that shouldn't be necessary. So - one more step than your easy > > > > process. > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > think his viewpoint is very reasonable. > > > > Granted, most people don't think about system upgrades when they build > > a system, so they wind up having to do extra work. In particular, > > Evren is talking about spending thousands of dollars on proprietary > > software, not to mention the cost of the server that all this data is > > going to flow to, for a backup solution. Compared to that, the cost of > > a few spare disks and the work to install them are trivial. > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > not something that any competent sysadmin would consider a pain. For a > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > source, which should be the next step up from trivial. > > > I guess you have a very different definition of "easy". :-) > > > > Given that mine is based on years of working with the kinds of backup > > solutions that Evren is asking for: ones that an enterprise deploys > > for backing up a data center, the answer may well be "yes". > > > > > The above procedure, in no way shape or form, will be classified as > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > of that. > > > > I never said it would be easy for a user. Then again, your average > > user doesn't do backups, and wouldn't know a continuous backup > > solution from a credit default swap. We're not talking about ghosting > > a disk partition for a backup, we're talking about enterprise-level > > backup solutions for data centers. People deploying those kinds of > > solutions tend to have multiple senior sysadmins around. > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > first two or three times. If they still have problems with it after > > that, they should find a new career path, as they aren't ever going to > > advance beyond junior. > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > as "experimental" in FreeBSD. That means that, among other things, > > it's not really well-supported by the installer, etc. Nuts, as of > > January of this year, there wasn't an operating system on the planet > > that would install and boot from ZFS. > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > happen to include some things that go a long way to solving Zefren's > > problems, so it was suggested as the basis for such (not by me, mind > > you). Having done the conversion, and found it easy, I responded when > > he asked how hard it was. > > > > But I'd never recommend this for your average user - which pretty much > > excludes anyone contemplating continuous backup solutions. > > > > > that mid-level or senior SAs have to do > > it "the hard way". Why? I'll > > > explain: > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > will -- but I will not be happy about it. I have the knowledge, I > > > know how to do it, so why must the process continue to be a PITA and > > > waste my time? > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > *all* that crap obsolete. You still have to deal with getting a boot > > loader installed, but after that, you never have to worry about > > partitioning, blocks, sectors, or slices again - until you go to an > > operating system that doesn't have ZFS. > > so can Freebsd boot off a ZFS root? in stable? current? ... boot0 doesn't apply here; it cares about what's at sector 0 on the disk, not filesystems. boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 partition. This is an annoyance. For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in loader.conf will cause FreeBSD to mount the root filesystem from ZFS. This works fine. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 11:07:46 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 847CE106568C; Sat, 11 Oct 2008 11:07:46 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id 109008FC12; Sat, 11 Oct 2008 11:07:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1KocK0-000DEX-PV; Sat, 11 Oct 2008 13:07:44 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Jeremy Chadwick In-reply-to: <20081011104409.GA58698@icarus.home.lan> References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Comments: In-reply-to Jeremy Chadwick message dated "Sat, 11 Oct 2008 03:44:09 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 11 Oct 2008 13:07:44 +0200 From: Danny Braniss Message-ID: Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 11:07:46 -0000 > On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > > On Fri, 10 Oct 2008 08:42:49 -0700 > > > Jeremy Chadwick wrote: > > > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > > Jeremy Chadwick wrote: > > > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > > Mike Meyer wrote: > > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > > >> yurtesen@ispro.net wrote: > > > > > > >> > > > > > > >>> Quoting "Oliver Fromme" : > > > > > > >>> > > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > > >>>> You don't have to code anything. > > > > > > >>> Well with 2 downsides, > > > > > > >> > > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > > is. > > > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > > planning, but anyone with more than a few months experience as a > > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > > create one) for example is generally easy: > > > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > > down or kill it > > > > > > 3) rsync or cpdup /home files to /mnt > > > > > > 4) umount /home > > > > > > 5) zfs set mountpoint=/home foo/home > > > > > > 6) Restart said processes or daemons > > > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > > because: > > > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > > multi-user, too. > > > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > > accordingly. > > > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > > themselves; it's a pain. > > > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > > a system set up to make this kind of thing easy. My system disk is on > > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > > upgrading one disk, making everything work, then putting the mirror > > > > > back together. And moving to zfs on root is a lot like a system > > > > > upgrade: > > > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > > 2) Repartition the unused drive into /boot, swap & data > > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > > wiki, just copying /boot and / at this point. > > > > > 4) Boot the zfs disk in single user mode. > > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > > on to step 6. > > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > > systems are cheap, and have lots of cool features that ufs > > > > > file systems don't have, you probably want to create more than > > > > > you had before, doing thing like putting SQL serves on their > > > > > own file system with appropriate blocking, etc, but you'll want to > > > > > have figured all this out before starting step 1). > > > > > 7) Copy data from the ufs file systems to their new homes, > > > > > not forgetting to take them out of /etc/fstab. > > > > > 8) Reboot on the zfs disk. > > > > > 9) Test until you're happy that everything is working properly, > > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > > add the data partition to the zfs pool so it's mirrored, and > > > > > you should have already been using swap. > > > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > > step that shouldn't be necessary. So - one more step than your easy > > > > > process. > > > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > > think his viewpoint is very reasonable. > > > > > > Granted, most people don't think about system upgrades when they build > > > a system, so they wind up having to do extra work. In particular, > > > Evren is talking about spending thousands of dollars on proprietary > > > software, not to mention the cost of the server that all this data is > > > going to flow to, for a backup solution. Compared to that, the cost of > > > a few spare disks and the work to install them are trivial. > > > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > > not something that any competent sysadmin would consider a pain. For a > > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > > source, which should be the next step up from trivial. > > > > I guess you have a very different definition of "easy". :-) > > > > > > Given that mine is based on years of working with the kinds of backup > > > solutions that Evren is asking for: ones that an enterprise deploys > > > for backing up a data center, the answer may well be "yes". > > > > > > > The above procedure, in no way shape or form, will be classified as > > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > > of that. > > > > > > I never said it would be easy for a user. Then again, your average > > > user doesn't do backups, and wouldn't know a continuous backup > > > solution from a credit default swap. We're not talking about ghosting > > > a disk partition for a backup, we're talking about enterprise-level > > > backup solutions for data centers. People deploying those kinds of > > > solutions tend to have multiple senior sysadmins around. > > > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > > first two or three times. If they still have problems with it after > > > that, they should find a new career path, as they aren't ever going to > > > advance beyond junior. > > > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > > as "experimental" in FreeBSD. That means that, among other things, > > > it's not really well-supported by the installer, etc. Nuts, as of > > > January of this year, there wasn't an operating system on the planet > > > that would install and boot from ZFS. > > > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > > happen to include some things that go a long way to solving Zefren's > > > problems, so it was suggested as the basis for such (not by me, mind > > > you). Having done the conversion, and found it easy, I responded when > > > he asked how hard it was. > > > > > > But I'd never recommend this for your average user - which pretty much > > > excludes anyone contemplating continuous backup solutions. > > > > > > > that mid-level or senior SAs have to do > > > it "the hard way". Why? I'll > > > > explain: > > > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > > will -- but I will not be happy about it. I have the knowledge, I > > > > know how to do it, so why must the process continue to be a PITA and > > > > waste my time? > > > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > > *all* that crap obsolete. You still have to deal with getting a boot > > > loader installed, but after that, you never have to worry about > > > partitioning, blocks, sectors, or slices again - until you go to an > > > operating system that doesn't have ZFS. > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > boot0 doesn't apply here; it cares about what's at sector 0 on the > disk, not filesystems. > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > partition. This is an annoyance. > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > This works fine. so the answer is: yes, if you have only one disk. no, if you have ZFS over many disks because I see no advantage in the springboard solution where ZFS is used to cover several disks. I'm asking, because I want to deploy some zfs fileservers soon, and so far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), and is readonly, so having 1 disk UFS seems to be a pitty. danny From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 11:24:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EC8C1065686 for ; Sat, 11 Oct 2008 11:24:34 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA04.westchester.pa.mail.comcast.net (qmta04.westchester.pa.mail.comcast.net [76.96.62.40]) by mx1.freebsd.org (Postfix) with ESMTP id C699A8FC18 for ; Sat, 11 Oct 2008 11:24:32 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA03.westchester.pa.mail.comcast.net ([76.96.62.27]) by QMTA04.westchester.pa.mail.comcast.net with comcast id RPDN1a00K0bG4ec54PQYEX; Sat, 11 Oct 2008 11:24:32 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA03.westchester.pa.mail.comcast.net with comcast id RPQX1a0022P6wsM3PPQXoe; Sat, 11 Oct 2008 11:24:32 +0000 X-Authority-Analysis: v=1.0 c=1 a=nOJ1pGDTh_cA:10 a=QycZ5dHgAAAA:8 a=yCzQ9mfbob5pQYgKny0A:9 a=XcD4ReMTFUOGBQ-zZIUA:7 a=kmpxTe3segXdmO5VXMdvlTw8BE4A:4 a=EoioJ0NPDVgA:10 a=SV7veod9ZcQA:10 a=ttedcufwrkMA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 38C6AC9419; Sat, 11 Oct 2008 04:24:31 -0700 (PDT) Date: Sat, 11 Oct 2008 04:24:31 -0700 From: Jeremy Chadwick To: Danny Braniss Message-ID: <20081011112431.GA60924@icarus.home.lan> References: <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 11:24:34 -0000 On Sat, Oct 11, 2008 at 01:07:44PM +0200, Danny Braniss wrote: > > On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: > > > > On Fri, 10 Oct 2008 08:42:49 -0700 > > > > Jeremy Chadwick wrote: > > > > > > > > > On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: > > > > > > On Fri, 10 Oct 2008 07:41:11 -0700 > > > > > > Jeremy Chadwick wrote: > > > > > > > > > > > > > On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: > > > > > > > > Mike Meyer wrote: > > > > > > > >> On Fri, 10 Oct 2008 02:34:28 +0300 > > > > > > > >> yurtesen@ispro.net wrote: > > > > > > > >> > > > > > > > >>> Quoting "Oliver Fromme" : > > > > > > > >>> > > > > > > > >>>> These features are readily available right now on FreeBSD. > > > > > > > >>>> You don't have to code anything. > > > > > > > >>> Well with 2 downsides, > > > > > > > >> > > > > > > > >> Once you actually try and implement these solutions, you'll see that > > > > > > > >> your "downsides" are largely figments of your imagination. > > > > > > > > > > > > > > > > So if it is my imagination, how can I actually convert UFS to ZFS > > > > > > > > easily? Everybody seems to say that this is easy and that is easy. > > > > > > > > > > > > > > It's not that easy. I really don't know why people are telling you it > > > > > > > is. > > > > > > > > > > > > Maybe because it is? Of course, it *does* require a little prior > > > > > > planning, but anyone with more than a few months experience as a > > > > > > sysadmin should be able to deal with it without to much hassle. > > > > > > > > > > > > > Converting some filesystems are easier than others; /home (if you > > > > > > > create one) for example is generally easy: > > > > > > > > > > > > > > 1) ZFS fs is called foo/home, mounted as /mnt > > > > > > > 2) fstat, ensure nothing is using /home -- if something is, shut it > > > > > > > down or kill it > > > > > > > 3) rsync or cpdup /home files to /mnt > > > > > > > 4) umount /home > > > > > > > 5) zfs set mountpoint=/home foo/home > > > > > > > 6) Restart said processes or daemons > > > > > > > > > > > > > > "See! It's like I said! EASY!" You can do this with /var as well. > > > > > > > > > > > > Yup. Of course, if you've done it that way, you're not thinking ahead, > > > > > > because: > > > > > > > > > > > > > Now try /usr. Hope you've got /rescue available, because once /usr/lib > > > > > > > and /usr/libexec disappear, you're in trouble. Good luck doing this in > > > > > > > multi-user, too. > > > > > > > > > > > > Oops. You F'ed up. If you'd done a little planning, you would have > > > > > > realized that / and /usr would be a bit of extra trouble, and planned > > > > > > accordingly. > > > > > > > > > > > > > And finally, the root fs. Whoever says "this is easy" is kidding > > > > > > > themselves; it's a pain. > > > > > > > > > > > > Um, no, it wasn't. Of course, I've been doing this long enough to have > > > > > > a system set up to make this kind of thing easy. My system disk is on > > > > > > a mirror, and I do system upgrades by breaking the mirror and > > > > > > upgrading one disk, making everything work, then putting the mirror > > > > > > back together. And moving to zfs on root is a lot like a system > > > > > > upgrade: > > > > > > > > > > > > 1) Break the mirror (mirrors actually, as I mirrored file systems). > > > > > > 2) Repartition the unused drive into /boot, swap & data > > > > > > 3) Build zfs & /boot according to the instructions on ZFSOnRoot > > > > > > wiki, just copying /boot and / at this point. > > > > > > 4) Boot the zfs disk in single user mode. > > > > > > 5) If 4 fails, boot back to the ufs disk so you're operational while > > > > > > you contemplate what went wrong, then repeat step 3. Otherwise, go > > > > > > on to step 6. > > > > > > 6) Create zfs file systems as appropriate (given that zfs file > > > > > > systems are cheap, and have lots of cool features that ufs > > > > > > file systems don't have, you probably want to create more than > > > > > > you had before, doing thing like putting SQL serves on their > > > > > > own file system with appropriate blocking, etc, but you'll want to > > > > > > have figured all this out before starting step 1). > > > > > > 7) Copy data from the ufs file systems to their new homes, > > > > > > not forgetting to take them out of /etc/fstab. > > > > > > 8) Reboot on the zfs disk. > > > > > > 9) Test until you're happy that everything is working properly, > > > > > > and be prepared to reboot on the ufs disk if something is broken. > > > > > > 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, > > > > > > add the data partition to the zfs pool so it's mirrored, and > > > > > > you should have already been using swap. > > > > > > > > > > > > This is 10 steps to your "easy" 6, but two of the extra steps are > > > > > > testing you didn't include, and 1 of the steps is a failure recovery > > > > > > step that shouldn't be necessary. So - one more step than your easy > > > > > > process. > > > > > > > > > > Of course, the part you seem to be (intentionally?) forgetting: most > > > > > people are not using gmirror. There is no 2nd disk. They have one disk > > > > > with a series of UFS2 filesystems, and they want to upgrade. That's how > > > > > I read Evren's "how do I do this? You say it's easy..." comment, and I > > > > > think his viewpoint is very reasonable. > > > > > > > > Granted, most people don't think about system upgrades when they build > > > > a system, so they wind up having to do extra work. In particular, > > > > Evren is talking about spending thousands of dollars on proprietary > > > > software, not to mention the cost of the server that all this data is > > > > going to flow to, for a backup solution. Compared to that, the cost of > > > > a few spare disks and the work to install them are trivial. > > > > > > > > > > Yeah, this isn't something you do on a whim. On the other hand, it's > > > > > > not something that any competent sysadmin would consider a pain. For a > > > > > > good senior admin, it's a lot easier than doing an OS upgrade from > > > > > > source, which should be the next step up from trivial. > > > > > I guess you have a very different definition of "easy". :-) > > > > > > > > Given that mine is based on years of working with the kinds of backup > > > > solutions that Evren is asking for: ones that an enterprise deploys > > > > for backing up a data center, the answer may well be "yes". > > > > > > > > > The above procedure, in no way shape or form, will be classified as > > > > > "easy" by the user (or even junior sysadmin) community, I can assure you > > > > > of that. > > > > > > > > I never said it would be easy for a user. Then again, your average > > > > user doesn't do backups, and wouldn't know a continuous backup > > > > solution from a credit default swap. We're not talking about ghosting > > > > a disk partition for a backup, we're talking about enterprise-level > > > > backup solutions for data centers. People deploying those kinds of > > > > solutions tend to have multiple senior sysadmins around. > > > > > > > > I wouldn't expect a junior admin to call it easy. At least, not the > > > > first two or three times. If they still have problems with it after > > > > that, they should find a new career path, as they aren't ever going to > > > > advance beyond junior. > > > > > > > > > I'll also throw this in the mix: the fact that we are *expecting* users > > > > > to know how to do this is unreasonable. It's even *more* rude to expect > > > > > > > > Um, is anyone expecting users to do this? I'm not. ZFS is still marked > > > > as "experimental" in FreeBSD. That means that, among other things, > > > > it's not really well-supported by the installer, etc. Nuts, as of > > > > January of this year, there wasn't an operating system on the planet > > > > that would install and boot from ZFS. > > > > > > > > I'm willing to jump through some hoops to get ZFS's advantages. Those > > > > happen to include some things that go a long way to solving Zefren's > > > > problems, so it was suggested as the basis for such (not by me, mind > > > > you). Having done the conversion, and found it easy, I responded when > > > > he asked how hard it was. > > > > > > > > But I'd never recommend this for your average user - which pretty much > > > > excludes anyone contemplating continuous backup solutions. > > > > > > > > > that mid-level or senior SAs have to do > > > > it "the hard way". Why? I'll > > > > > explain: > > > > > > > > > > I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk > > > > > partitioning, sectors vs. blocks, slices, filesystems, and whatever > > > > > else. You want me to do it by hand, say, with bsdlabel -e? Fine, I > > > > > will -- but I will not be happy about it. I have the knowledge, I > > > > > know how to do it, so why must the process continue to be a PITA and > > > > > waste my time? > > > > > > > > Did I ever mention bsdlabel? But in any case, ZFS makes pretty much > > > > *all* that crap obsolete. You still have to deal with getting a boot > > > > loader installed, but after that, you never have to worry about > > > > partitioning, blocks, sectors, or slices again - until you go to an > > > > operating system that doesn't have ZFS. > > > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > > > boot0 doesn't apply here; it cares about what's at sector 0 on the > > disk, not filesystems. > > > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > > partition. This is an annoyance. > > > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > > This works fine. > > so the answer is: > yes, if you have only one disk. > no, if you have ZFS over many disks > > because I see no advantage in the springboard solution where ZFS is used to > cover several disks. > > I'm asking, because I want to deploy some zfs fileservers soon, and so > far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > and is readonly, so having 1 disk UFS seems to be a pitty. Hold on a minute. "One disk" has nothing to do with the filesystem. You asked if FreeBSD could boot off of a specific filesystem, and I answered that -- I didn't state anything about disk counts. Now you're changing the focus. :-) I'm pretty sure FreeBSD can boot off of gmirror setups (see above, boot2/loader should work off of gmirror), which means >1 disk. You do not have to gmirror the entire disk, you can gmirror just a slice (AFAIK). I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) part of the 2nd disk for ZFS as well, otherwise the space would go to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page seems to imply you can. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 12:02:51 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1FA41065688; Sat, 11 Oct 2008 12:02:51 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: from k7.mavetju.org (ppp121-44-105-133.lns10.syd6.internode.on.net [121.44.105.133]) by mx1.freebsd.org (Postfix) with ESMTP id 8A66A8FC08; Sat, 11 Oct 2008 12:02:51 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: by k7.mavetju.org (Postfix, from userid 1001) id A1EF145039; Sat, 11 Oct 2008 22:45:57 +1100 (EST) Date: Sat, 11 Oct 2008 22:45:57 +1100 From: Edwin Groothuis To: Dmitry Marakasov Message-ID: <20081011114557.GA71472@mavetju.org> References: <48C52718.5080807@sun.com> <48C8F051.7060107@sun.com> <20080919082719.GH676@e.0x20.net> <20081010174204.GB90757@hades.panopticon> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081010174204.GB90757@hades.panopticon> User-Agent: Mutt/1.4.2.3i Cc: freebsd-hackers@FreeBSD.org, freebsd-ports@freebsd.org Subject: Re: VirtualBox looks for FreeBSD developer X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 12:02:51 -0000 On Fri, Oct 10, 2008 at 09:42:04PM +0400, Dmitry Marakasov wrote: > Little time ago I was misleaded by the certain people and got an > idea that VirtualBox actually works on FreeBSD, so I've made a draft > port for it. It doesn't actually work, but since I've spent several > hours hacking it and made bunch of (likely) useful patches, here > it is, feel free to use it for any purpose. I hope someone of kernel > hackers will make it work actually ;) Have a talk with bms@ about it, he had some interesting working code too. Edwin -- Edwin Groothuis Website: http://www.mavetju.org/ edwin@mavetju.org Weblog: http://www.mavetju.org/weblog/ From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 12:08:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F3CD1065688; Sat, 11 Oct 2008 12:08:02 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from itchy.rabson.org (unknown [IPv6:2002:50b1:e8f2:1::143]) by mx1.freebsd.org (Postfix) with ESMTP id 2E8F78FC54; Sat, 11 Oct 2008 12:08:01 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc] (unknown [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc]) by itchy.rabson.org (Postfix) with ESMTP id ED25F3F9C; Sat, 11 Oct 2008 13:07:58 +0100 (BST) Message-Id: <12DB2D13-1F6F-4CA5-B583-CCDD678419FF@rabson.org> From: Doug Rabson To: Danny Braniss In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Sat, 11 Oct 2008 13:07:58 +0100 References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> X-Mailer: Apple Mail (2.929.2) Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 12:08:02 -0000 On 11 Oct 2008, at 12:07, Danny Braniss wrote: >> On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: >>>> On Fri, 10 Oct 2008 08:42:49 -0700 >>>> Jeremy Chadwick wrote: >>>> >>>>> On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: >>>>>> On Fri, 10 Oct 2008 07:41:11 -0700 >>>>>> Jeremy Chadwick wrote: >>>>>> >>>>>>> On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: >>>>>>>> Mike Meyer wrote: >>>>>>>>> On Fri, 10 Oct 2008 02:34:28 +0300 >>>>>>>>> yurtesen@ispro.net wrote: >>>>>>>>> >>>>>>>>>> Quoting "Oliver Fromme" : >>>>>>>>>> >>>>>>>>>>> These features are readily available right now on FreeBSD. >>>>>>>>>>> You don't have to code anything. >>>>>>>>>> Well with 2 downsides, >>>>>>>>> >>>>>>>>> Once you actually try and implement these solutions, you'll >>>>>>>>> see that >>>>>>>>> your "downsides" are largely figments of your imagination. >>>>>>>> >>>>>>>> So if it is my imagination, how can I actually convert UFS to >>>>>>>> ZFS >>>>>>>> easily? Everybody seems to say that this is easy and that is >>>>>>>> easy. >>>>>>> >>>>>>> It's not that easy. I really don't know why people are >>>>>>> telling you it >>>>>>> is. >>>>>> >>>>>> Maybe because it is? Of course, it *does* require a little prior >>>>>> planning, but anyone with more than a few months experience as a >>>>>> sysadmin should be able to deal with it without to much hassle. >>>>>> >>>>>>> Converting some filesystems are easier than others; /home (if >>>>>>> you >>>>>>> create one) for example is generally easy: >>>>>>> >>>>>>> 1) ZFS fs is called foo/home, mounted as /mnt >>>>>>> 2) fstat, ensure nothing is using /home -- if something is, >>>>>>> shut it >>>>>>> down or kill it >>>>>>> 3) rsync or cpdup /home files to /mnt >>>>>>> 4) umount /home >>>>>>> 5) zfs set mountpoint=/home foo/home >>>>>>> 6) Restart said processes or daemons >>>>>>> >>>>>>> "See! It's like I said! EASY!" You can do this with /var as >>>>>>> well. >>>>>> >>>>>> Yup. Of course, if you've done it that way, you're not thinking >>>>>> ahead, >>>>>> because: >>>>>> >>>>>>> Now try /usr. Hope you've got /rescue available, because >>>>>>> once /usr/lib >>>>>>> and /usr/libexec disappear, you're in trouble. Good luck >>>>>>> doing this in >>>>>>> multi-user, too. >>>>>> >>>>>> Oops. You F'ed up. If you'd done a little planning, you would >>>>>> have >>>>>> realized that / and /usr would be a bit of extra trouble, and >>>>>> planned >>>>>> accordingly. >>>>>> >>>>>>> And finally, the root fs. Whoever says "this is easy" is >>>>>>> kidding >>>>>>> themselves; it's a pain. >>>>>> >>>>>> Um, no, it wasn't. Of course, I've been doing this long enough >>>>>> to have >>>>>> a system set up to make this kind of thing easy. My system disk >>>>>> is on >>>>>> a mirror, and I do system upgrades by breaking the mirror and >>>>>> upgrading one disk, making everything work, then putting the >>>>>> mirror >>>>>> back together. And moving to zfs on root is a lot like a system >>>>>> upgrade: >>>>>> >>>>>> 1) Break the mirror (mirrors actually, as I mirrored file >>>>>> systems). >>>>>> 2) Repartition the unused drive into /boot, swap & data >>>>>> 3) Build zfs & /boot according to the instructions on ZFSOnRoot >>>>>> wiki, just copying /boot and / at this point. >>>>>> 4) Boot the zfs disk in single user mode. >>>>>> 5) If 4 fails, boot back to the ufs disk so you're operational >>>>>> while >>>>>> you contemplate what went wrong, then repeat step 3. >>>>>> Otherwise, go >>>>>> on to step 6. >>>>>> 6) Create zfs file systems as appropriate (given that zfs file >>>>>> systems are cheap, and have lots of cool features that ufs >>>>>> file systems don't have, you probably want to create more than >>>>>> you had before, doing thing like putting SQL serves on their >>>>>> own file system with appropriate blocking, etc, but you'll >>>>>> want to >>>>>> have figured all this out before starting step 1). >>>>>> 7) Copy data from the ufs file systems to their new homes, >>>>>> not forgetting to take them out of /etc/fstab. >>>>>> 8) Reboot on the zfs disk. >>>>>> 9) Test until you're happy that everything is working properly, >>>>>> and be prepared to reboot on the ufs disk if something is >>>>>> broken. >>>>>> 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, >>>>>> add the data partition to the zfs pool so it's mirrored, and >>>>>> you should have already been using swap. >>>>>> >>>>>> This is 10 steps to your "easy" 6, but two of the extra steps are >>>>>> testing you didn't include, and 1 of the steps is a failure >>>>>> recovery >>>>>> step that shouldn't be necessary. So - one more step than your >>>>>> easy >>>>>> process. >>>>> >>>>> Of course, the part you seem to be (intentionally?) forgetting: >>>>> most >>>>> people are not using gmirror. There is no 2nd disk. They have >>>>> one disk >>>>> with a series of UFS2 filesystems, and they want to upgrade. >>>>> That's how >>>>> I read Evren's "how do I do this? You say it's easy..." comment, >>>>> and I >>>>> think his viewpoint is very reasonable. >>>> >>>> Granted, most people don't think about system upgrades when they >>>> build >>>> a system, so they wind up having to do extra work. In particular, >>>> Evren is talking about spending thousands of dollars on proprietary >>>> software, not to mention the cost of the server that all this >>>> data is >>>> going to flow to, for a backup solution. Compared to that, the >>>> cost of >>>> a few spare disks and the work to install them are trivial. >>>> >>>>>> Yeah, this isn't something you do on a whim. On the other hand, >>>>>> it's >>>>>> not something that any competent sysadmin would consider a >>>>>> pain. For a >>>>>> good senior admin, it's a lot easier than doing an OS upgrade >>>>>> from >>>>>> source, which should be the next step up from trivial. >>>>> I guess you have a very different definition of "easy". :-) >>>> >>>> Given that mine is based on years of working with the kinds of >>>> backup >>>> solutions that Evren is asking for: ones that an enterprise deploys >>>> for backing up a data center, the answer may well be "yes". >>>> >>>>> The above procedure, in no way shape or form, will be classified >>>>> as >>>>> "easy" by the user (or even junior sysadmin) community, I can >>>>> assure you >>>>> of that. >>>> >>>> I never said it would be easy for a user. Then again, your average >>>> user doesn't do backups, and wouldn't know a continuous backup >>>> solution from a credit default swap. We're not talking about >>>> ghosting >>>> a disk partition for a backup, we're talking about enterprise-level >>>> backup solutions for data centers. People deploying those kinds of >>>> solutions tend to have multiple senior sysadmins around. >>>> >>>> I wouldn't expect a junior admin to call it easy. At least, not the >>>> first two or three times. If they still have problems with it after >>>> that, they should find a new career path, as they aren't ever >>>> going to >>>> advance beyond junior. >>>> >>>>> I'll also throw this in the mix: the fact that we are >>>>> *expecting* users >>>>> to know how to do this is unreasonable. It's even *more* rude >>>>> to expect >>>> >>>> Um, is anyone expecting users to do this? I'm not. ZFS is still >>>> marked >>>> as "experimental" in FreeBSD. That means that, among other things, >>>> it's not really well-supported by the installer, etc. Nuts, as of >>>> January of this year, there wasn't an operating system on the >>>> planet >>>> that would install and boot from ZFS. >>>> >>>> I'm willing to jump through some hoops to get ZFS's advantages. >>>> Those >>>> happen to include some things that go a long way to solving >>>> Zefren's >>>> problems, so it was suggested as the basis for such (not by me, >>>> mind >>>> you). Having done the conversion, and found it easy, I responded >>>> when >>>> he asked how hard it was. >>>> >>>> But I'd never recommend this for your average user - which pretty >>>> much >>>> excludes anyone contemplating continuous backup solutions. >>>> >>>>> that mid-level or senior SAs have to do >>>> it "the hard way". Why? I'll >>>>> explain: >>>>> >>>>> I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, >>>>> general disk >>>>> partitioning, sectors vs. blocks, slices, filesystems, and >>>>> whatever >>>>> else. You want me to do it by hand, say, with bsdlabel -e? >>>>> Fine, I >>>>> will -- but I will not be happy about it. I have the knowledge, I >>>>> know how to do it, so why must the process continue to be a PITA >>>>> and >>>>> waste my time? >>>> >>>> Did I ever mention bsdlabel? But in any case, ZFS makes pretty much >>>> *all* that crap obsolete. You still have to deal with getting a >>>> boot >>>> loader installed, but after that, you never have to worry about >>>> partitioning, blocks, sectors, or slices again - until you go to an >>>> operating system that doesn't have ZFS. >>> >>> so can Freebsd boot off a ZFS root? in stable? current? ... >> >> boot0 doesn't apply here; it cares about what's at sector 0 on the >> disk, not filesystems. >> >> boot2/loader does not speak ZFS -- this is why you need the /boot >> UFS2 >> partition. This is an annoyance. >> >> For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in >> loader.conf will cause FreeBSD to mount the root filesystem from ZFS. >> This works fine. > > so the answer is: > yes, if you have only one disk. > no, if you have ZFS over many disks > > because I see no advantage in the springboard solution where ZFS is > used to > cover several disks. > > I'm asking, because I want to deploy some zfs fileservers soon, and so > far the solution is either PXE boot, or keep one disk UFS (or boot > off a USB) > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > and is readonly, so having 1 disk UFS seems to be a pitty. ZFS boot is coming. Currently its part of pjd's perforce branch and supports disks, mirrors and collections of disks or mirrors with the only restriction being that enough drives in the pool must be accessible from the bios (i.e. at least one element of a mirror must be seen by the bios). Currently we don't support booting from raidz or raidz2 pools but there is no fundamental reason why that can't be added - someone just needs to implement the code to understand the raidz layout. From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 12:30:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 278FE1065695; Sat, 11 Oct 2008 12:30:35 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id DE51E8FC1C; Sat, 11 Oct 2008 12:30:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m9BCUMdm057392; Sat, 11 Oct 2008 05:30:22 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m9BCUMxp057391; Sat, 11 Oct 2008 05:30:22 -0700 (PDT) Date: Sat, 11 Oct 2008 05:30:22 -0700 (PDT) From: Matthew Dillon Message-Id: <200810111230.m9BCUMxp057391@apollo.backplane.com> To: Danny Braniss References: <200810091411.m99EB0Vo007538@lurza.secnetix.de> <20081010023428.87556dt18ejyzf48@mail.ispro.net> <20081009200641.60d0b236@bhuda.mired.org> <48EF5052.2000707@ispro.net> <20081010144111.GA34609@icarus.home.lan> <20081010112952.52b8209b@bhuda.mired.org> <20081010154249.GA35859@icarus.home.lan> <20081010122228.355c2c3e@bhuda.mired.org> <20081011104409.GA58698@icarus.home.lan> Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , Mike Meyer Subject: Re: continuous backup solution for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 12:30:35 -0000 :> boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 :> partition. This is an annoyance. :> :> For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in :> loader.conf will cause FreeBSD to mount the root filesystem from ZFS. :> This works fine. : :so the answer is: : yes, if you have only one disk. : no, if you have ZFS over many disks : :because I see no advantage in the springboard solution where ZFS is used to :cover several disks. : :I'm asking, because I want to deploy some zfs fileservers soon, and so :far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) :Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), :and is readonly, so having 1 disk UFS seems to be a pitty. : :danny I think it is is perfectly acceptable to have a /boot + ZFS style solution, where /boot is a small ~256M UFS filesystem that the system actually boots from (containing only the kernel, modules, loader.conf, etc), and ZFS is the root filesystem. In a running system /boot would be mounted under the ZFS root. All I needed was a line in /boot/loader.conf to tell the kernel where the root FS was. In my case, I pointed it at HAMMER. vfs.root.mountfrom="hammer:ad0s1d" This gives you the flexibility of being able to have as complex a root FS as you want. Filesystem 1K-blocks Used Avail Capacity Mounted on HAMMER_ROOT 36388864 9789440 26599424 27% / /dev/ad0s1a 257998 100074 137286 42% /boot /pfs/@@-1:00001 36388864 9789440 26599424 27% /usr /pfs/@@-1:00003 36388864 9789440 26599424 27% /var /pfs/@@-1:00006 36388864 9789440 26599424 27% /tmp /pfs/@@-1:00007 36388864 9789440 26599424 27% /home /pfs/@@-1:00005 36388864 9789440 26599424 27% /var/tmp /pfs/@@-1:00002 36388864 9789440 26599424 27% /usr/obj /pfs/@@-1:00004 36388864 9789440 26599424 27% /var/crash procfs 4 4 0 100% /proc The /boot is small enough that it can be dealt with numerous ways, including simple duplication if you have multiple disks (have a adXs1a on two drives). And if you were really that worried you could put /boot on a SSD. Frankly, anything that has approximately the same MTBF as the motherboard itself is suitable, there's really no point trying to make /boot disk-redundant when the motherboard and memory aren't redundant. If you have more then one HD connected to the system, and you want boot redundancy, then you also likely have the $$ to purchase a tiny SSD for your /boot. The big problem trying to boot from a completely generic FS setup is that it tends to severely limit your options. You might want to have more flexibility in your root filesystem that you could otherwise accomodate if /boot were integrated into it. -Matt Matthew Dillon From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 13:28:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 404E1106568E; Sat, 11 Oct 2008 13:28:04 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id EFDBD8FC13; Sat, 11 Oct 2008 13:28:03 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1KoeVm-000ELP-4b; Sat, 11 Oct 2008 15:28:02 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Jeremy Chadwick Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 11 Oct 2008 15:28:01 +0200 From: Danny Braniss Message-ID: Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 13:28:04 -0000 > > > > so can Freebsd boot off a ZFS root? in stable? current? ... > > > > > > boot0 doesn't apply here; it cares about what's at sector 0 on the > > > disk, not filesystems. > > > > > > boot2/loader does not speak ZFS -- this is why you need the /boot UFS2 > > > partition. This is an annoyance. > > > > > > For the final "stage/step", vfs.root.mountfrom="zfs:mypool/root" in > > > loader.conf will cause FreeBSD to mount the root filesystem from ZFS. > > > This works fine. > > > > so the answer is: > > yes, if you have only one disk. > > no, if you have ZFS over many disks > > > > because I see no advantage in the springboard solution where ZFS is used to > > cover several disks. > > > > I'm asking, because I want to deploy some zfs fileservers soon, and so > > far the solution is either PXE boot, or keep one disk UFS (or boot off a USB) > > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), > > and is readonly, so having 1 disk UFS seems to be a pitty. > > Hold on a minute. "One disk" has nothing to do with the filesystem. > You asked if FreeBSD could boot off of a specific filesystem, and I > answered that -- I didn't state anything about disk counts. Now you're > changing the focus. :-) > not intentionaly, but once you mention boot0/2, bsdlabel, slice/partition ... /OT Initially, I was not thrilled with ZFS, but once you cross the few hundred gigabyte filesystems UFS is impractical, and though old sysadmins will have to be re-educated (zfs catch-all-commands, instead of mount/fsck/export/blah...) it is the (only?) way for the new-terrabyte-world. So having bitten the bullet, and doing some experiments i'm stuck as to what to do with / OT/ > I'm pretty sure FreeBSD can boot off of gmirror setups (see above, > boot2/loader should work off of gmirror), which means >1 disk. You > do not have to gmirror the entire disk, you can gmirror just a slice > (AFAIK). but gmirror is not ZFS, and yes it can, why not. > > I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) > part of the 2nd disk for ZFS as well, otherwise the space would go > to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page > seems to imply you can. The idea is to used the 'free/remaining' as part of the BIG ZFS 'array' [ED note : I've highjacked the 'Re: continuous backup solution for FreeBSD'] To Matt: since 'small' nowadays is big enough to hold /, what advantages are there in having root split up? also, having this split personality, what if the disk goes? the hammer/zfs is probably raided ... [btw, having a small-boot-partition brings back bad memories: the first thing I did on a new Compact was boot it diskless, repartition the disk, newfs, and I could no longer boot it :-) - part of the BIOS was there] To Doug: > ZFS boot is coming. great! any time estimate?, just curious, no preasure :-) some food for thought: In the past raid 5 would reduce the throughput conciderably, though nowadays it's hardly notisable, so I guess my reluctance to having a swap partition raided is gone. danny From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 18:10:34 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4732106568E; Sat, 11 Oct 2008 18:10:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7DC9A8FC18; Sat, 11 Oct 2008 18:10:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m9BIAHff059978; Sat, 11 Oct 2008 11:10:17 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m9BIAGPw059975; Sat, 11 Oct 2008 11:10:16 -0700 (PDT) Date: Sat, 11 Oct 2008 11:10:16 -0700 (PDT) From: Matthew Dillon Message-Id: <200810111810.m9BIAGPw059975@apollo.backplane.com> To: Danny Braniss References: Cc: freebsd-hackers@FreeBSD.org, Jeremy Chadwick , Mike Meyer Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 18:10:34 -0000 :To Matt: : since 'small' nowadays is big enough to hold /, what advantages are there :in having root split up? :also, having this split personality, what if the disk goes? the hammer/zfs :is probably raided ... You mean /boot + root , or do you mean /root vs /usr vs /home? I'll answer both. With regards to /boot + root. A small separate /boot partition (256m) allows your root filesystem to use an arbitrarily complex topology. e.g. multiple geom layers, weird zfs setups, etc. So you get flexibility that you would otherwise not have if you went with a directly-bootable ZFS root. /boot can be as complex as boot2 allows. There's nothing preventing it from being RAIDed if boot2 supported that, and there's nothing preventing it (once you had ZFS boot capabilities) from being ZFS using a topology supported by boot2. Having a sparate /boot allows your filesystem root to use topologues boot2 would otherwise not support. With regards to the traditional BSD partitioning scheme, having a separate /usr, /home, /tmp, etc... there's no reason to do that stuff any more with ZFS (or HAMMER). You just need one, and can break it down into separate management domains within the filesystem (e.g. HAMMER PFS's). That's a generic statement of course, there will always be situations where you might want to partition things out separately. Most linux dists don't bother with multiple partitions any more. They just have '/' and maybe a small boot partition, and that's it. -Matt Matthew Dillon From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 20:44:40 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 014BD106568A for ; Sat, 11 Oct 2008 20:44:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.226]) by mx1.freebsd.org (Postfix) with ESMTP id CA83A8FC13 for ; Sat, 11 Oct 2008 20:44:39 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so959066rvf.43 for ; Sat, 11 Oct 2008 13:44:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=kr/GVxHX55WDTMjkU4na7SRV8NhlEt5OPDftz/RMFzw=; b=NJHphhKUCvricrUBlMSazvlrcApNwYtBERWRLW+74t5rdpReuK8EeEIOG74e54HKzp aLyzHEfSOH82m99QJkxAbkd86hZ3AKts54mO+A6eozMCUdbv6aLSnLJl+R3OGgJxES7i X78izG8hd0+3YF7BWPSgEAG3zripn6r00w90E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=FUvGHakzj/ppHJExCCqO5CgGGl3Hp5fTisrtfMcG9uZnnej8lcXJWKytXQMJvY0uVA vJ2vxSHRrmAe/LNuiMBZWcIg603Gby0fCiQzNJD0SApSYwSOeOwdHv2Nh0S8PtV139BL sg6arjlJco/Cy8nOF0VzvYxmNKg9KHjiUSt1Q= Received: by 10.141.153.16 with SMTP id f16mr1740812rvo.17.1223757430695; Sat, 11 Oct 2008 13:37:10 -0700 (PDT) Received: by 10.141.168.12 with HTTP; Sat, 11 Oct 2008 13:37:10 -0700 (PDT) Message-ID: Date: Sat, 11 Oct 2008 20:37:10 +0000 From: "Freddie Cash" To: freebsd-hackers@freebsd.org In-Reply-To: <200810111810.m9BIAGPw059975@apollo.backplane.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200810111810.m9BIAGPw059975@apollo.backplane.com> Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 20:44:40 -0000 On 10/11/08, Matthew Dillon wrote: > With regards to the traditional BSD partitioning scheme, having a > separate /usr, /home, /tmp, etc... there's no reason to do that stuff > any more with ZFS (or HAMMER). As separate partitions, no. As separate filesystems, definitely. While HAMMER PFSes may not support these things yet, ZFS allows you to tailor each filesystem to its purpose. For example, you can enable compression on /usr/ports, but have a separate /usr/ports/distfilles and /usr/ports/work that aren't compressed. Or /usr/src compressed and /usr/obj not. Have a small record (block) size for /usr/src, but a larger one for /home. Give each user a separate filesystem for their /home/, with separate snapshot policies, quotas, and reservations (initial filesystem size). Creating new filesystems with ZFS is as simple as "zfs create -o mountpoint=/wherever pool/fsname". If you put a little time into planning the hierarchy/structure, you can take advantage off the properties inheritance features of ZFS as well. > You just need one, and can break it > down into separate management domains within the filesystem > (e.g. HAMMER PFS's). Similar kind of idea. > Most linux dists don't bother with multiple partitions any more. > They just have '/' and maybe a small boot partition, and that's it. Heh, that's more proof of the difficulties inherent with old-school disk partitioning, compared to pooled storage setups, than an endorsement of using a single partition/filesystem. :) -- Freddie Cash fjwcash@gmail.com From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 20:53:36 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D8B81065689 for ; Sat, 11 Oct 2008 20:53:36 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from euclid.ucsd.edu (euclid.ucsd.edu [132.239.145.52]) by mx1.freebsd.org (Postfix) with ESMTP id 7E2BE8FC0A for ; Sat, 11 Oct 2008 20:53:36 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from zeno.ucsd.edu (zeno.ucsd.edu [132.239.145.22]) by euclid.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9BKra610590; Sat, 11 Oct 2008 13:53:36 -0700 (PDT) Received: from localhost (neldredg@localhost) by zeno.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9BKraK10673; Sat, 11 Oct 2008 13:53:36 -0700 (PDT) X-Authentication-Warning: zeno.ucsd.edu: neldredg owned process doing -bs Date: Sat, 11 Oct 2008 13:53:35 -0700 (PDT) From: Nate Eldredge X-X-Sender: neldredg@zeno.ucsd.edu To: Freddie Cash In-Reply-To: Message-ID: References: <200810111810.m9BIAGPw059975@apollo.backplane.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 20:53:36 -0000 On Sat, 11 Oct 2008, Freddie Cash wrote: > On 10/11/08, Matthew Dillon wrote: >> With regards to the traditional BSD partitioning scheme, having a >> separate /usr, /home, /tmp, etc... there's no reason to do that stuff >> any more with ZFS (or HAMMER). > > As separate partitions, no. As separate filesystems, definitely. > > While HAMMER PFSes may not support these things yet, ZFS allows you to > tailor each filesystem to its purpose. For example, you can enable > compression on /usr/ports, but have a separate /usr/ports/distfilles > and /usr/ports/work that aren't compressed. Or /usr/src compressed > and /usr/obj not. Have a small record (block) size for /usr/src, but > a larger one for /home. Give each user a separate filesystem for > their /home/, with separate snapshot policies, quotas, and > reservations (initial filesystem size). All this about ZFS sounds great, and I'd like to try it out, but some of the bugs, etc, listed at http://wiki.freebsd.org/ZFSKnownProblems are rather alarming. Even on a personal machine, I don't want these features at the cost of an unstable system. Is that list still current? FWIW, my system is amd64 with 1 G of memory, which the page implies is insufficient. Is it really? -- Nate Eldredge neldredge@math.ucsd.edu From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 20:54:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D53D310656A4 for ; Sat, 11 Oct 2008 20:54:26 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.225]) by mx1.freebsd.org (Postfix) with ESMTP id A9F878FC16 for ; Sat, 11 Oct 2008 20:54:26 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so960966rvf.43 for ; Sat, 11 Oct 2008 13:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=GFWAbjKG8AhMCovDIqXlP/LECdgNNTBh0W45msQTr9E=; b=ZXSdHzIzLCwWUrOBn2vvb7zVVpZ/ItMLvyz5hQeMQqGEA1JKbbQFhOivY3AkIhftWD KLmQPvlwlliYoqwegmnFTJc99uvSXuC/tzjBdkLcleIKSZMDKBpP80PVtziz7sqHSG4H y7Zc1Gk03HwUNG+UTJGUkq4NYSbnCYgRu6AEU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=C24N8KEnq3Kb1rM/ftUu/HP/0sirVIa9ahDVuf45CDtSjWfUo2Nril0Aqqn3J1MMNC bdzFySEmdf1BL4ses3HWpKV2uv81j/Q3jM6JTlYVfCCntNFdVKpWXbkW9EdGrvMeNC9M pGY3iuODbZEKWmFmEIffJoUuRyT6+SxFKRMMo= Received: by 10.141.205.10 with SMTP id h10mr2355933rvq.54.1223756979977; Sat, 11 Oct 2008 13:29:39 -0700 (PDT) Received: by 10.141.168.12 with HTTP; Sat, 11 Oct 2008 13:29:39 -0700 (PDT) Message-ID: Date: Sat, 11 Oct 2008 20:29:39 +0000 From: "Freddie Cash" To: freebsd-hackers@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 20:54:27 -0000 On 10/11/08, Danny Braniss wrote: >> > I'm asking, because I want to deploy some zfs fileservers soon, and so >> > far the solution is either PXE boot, or keep one disk UFS (or boot off a >> > USB) For the servers we're deploying FreeBSD+ZFS on, mainly large backup systems with 24 drives, we're putting / onto either CompactFlash (using IDE adapters) or USB sticks (using internal connectors), using gmirror to provide fail-over for /. That way, we can boot off UFS, have full access to single-user mode and /rescue, and use every bit of each disk for ZFS. Works quite nicely. >> > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), >> > and is readonly, so having 1 disk UFS seems to be a pitty. / by itself (no /usr, /home, /tmp, or /var) is under 300 MB on our systems (FreeBSD 7-STABLE from August, amd64). Definitely not worth dedicating an entire 500 GB drive to, or even a single slice or partition to. By putting / onto separate media (like CF, USB, whatever), you can dedicate all your harddrive space to ZFS. > /OT > Initially, I was not thrilled with ZFS, but once you cross the Once you start using ZFS features, especially snapshots, it's really hard to move to non-pooled-storage setups. Even LVM on Linux becomes hard to work with. There's just no easier way to work with multi-TB storage setups using 10+ drives. Even for smaller systems with only 3 drives, it's so much nicer working with pooled storage systems like ZFS. My home server uses a 2 GB USB stick for / with 3x 120 GB drives for ZFS, with separate filesystems for /usr, /usr/ports, /usr/src, /usr/obj, /usr/local, /home, /var, and /tmp. No fussing around with partition sizes ahead of time is probably the single greatest feature, with instant/unlimited snapshots a very close second. >> I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) >> part of the 2nd disk for ZFS as well, otherwise the space would go >> to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page >> seems to imply you can. I did this for awhile. 3x 120 GB drives configured as: 10 GB slice for / 2 GB slice for swap 108 GB slice to ZFS The first slice was configured as a 3-way gmirror, and the last slice was configured as a raidz pool. But performance wasn't that great. Moved / to a USB stick, and dedicated the entire drives to the zpool, and things have been a lot smoother. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 21:40:56 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F07821065688; Sat, 11 Oct 2008 21:40:56 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 929928FC14; Sat, 11 Oct 2008 21:40:56 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [211.166.10.233]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id B952D28449; Sun, 12 Oct 2008 05:40:55 +0800 (CST) Received: from localhost (tarsier.geekcn.org [211.166.10.233]) by tarsier.geekcn.org (Postfix) with ESMTP id 51414101D555; Sun, 12 Oct 2008 05:40:55 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([211.166.10.233]) by localhost (mail.geekcn.org [211.166.10.233]) (amavisd-new, port 10024) with ESMTP id z0O200vJ4hwx; Sun, 12 Oct 2008 05:40:50 +0800 (CST) Received: from delta.delphij.net (c-76-103-40-85.hsd1.ca.comcast.net [76.103.40.85]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 0828E101C86B; Sun, 12 Oct 2008 05:40:48 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=ja7xySETppg3Gg8FmKeIdQQIRYDPvfgYskvhBRVtDYpOcjYHkFs6I9BtuwDyI7Idy DLh50/PiHApXgQK5xPFsg== Message-ID: <48F11D5E.7090108@delphij.net> Date: Sat, 11 Oct 2008 14:40:46 -0700 From: Xin LI Organization: The Geek China Organization User-Agent: Thunderbird 2.0.0.17 (X11/20080926) MIME-Version: 1.0 To: Matthew Dillon References: <200810111810.m9BIAGPw059975@apollo.backplane.com> In-Reply-To: <200810111810.m9BIAGPw059975@apollo.backplane.com> X-Enigmail-Version: 0.95.7 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, Jeremy Chadwick , Mike Meyer Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 21:40:57 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Matt, Matthew Dillon wrote: [...] > /boot can be as complex as boot2 allows. There's nothing preventing > it from being RAIDed if boot2 supported that, and there's nothing > preventing it (once you had ZFS boot capabilities) from being ZFS > using a topology supported by boot2. Having a sparate /boot allows > your filesystem root to use topologues boot2 would otherwise not > support. I believe that it's a good idea to separate / from the zpool for other file systems, or even use a UFS /. My experience with ZFS on my laptop shows that disk failures can be more easily fixed if there are some utilities available in the UFS /, even when ZFS is used as /. Another issue with a ZFS / is that the snapshot rollback feature generally does not work on / since it needs the mountpoint to be unmounted. One thing that I found very useful is the new GPT boot feature on 8.0, which also works on older BIOS because the protected MBR would deal with the bootstrap to the actual GPT boot. Now we have a 15-block sized gptboot that can boot FreeBSD from UFS, however this 'boot' can be in virtually any size that the BIOS supports, so we can embed more logic there. Cheers, -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkjxHV0ACgkQi+vbBBjt66CpXgCfWstsxNc3B4xOzNTxz9/kdl3Y /WYAnjqiV5H8xQYxGgZTnwWieuG6ZZij =LH+x -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 22:30:57 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D7E21065697 for ; Sat, 11 Oct 2008 22:30:57 +0000 (UTC) (envelope-from ken@mthelicon.com) Received: from hercules.mthelicon.com (hercules.mthelicon.com [IPv6:2001:49f0:2023::2]) by mx1.freebsd.org (Postfix) with ESMTP id 028458FC16 for ; Sat, 11 Oct 2008 22:30:56 +0000 (UTC) (envelope-from ken@mthelicon.com) Received: from feathers.peganest.com ([IPv6:2001:4d48:ad51:32:21b:21ff:fe1c:3ce]) (authenticated bits=0) by hercules.mthelicon.com (8.14.3/8.14.2) with ESMTP id m9BMUsF9021884 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Sat, 11 Oct 2008 22:30:55 GMT (envelope-from ken@mthelicon.com) From: Pegasus Mc Cleaft Organization: Feathers To: freebsd-hackers@freebsd.org Date: Sat, 11 Oct 2008 23:30:53 +0100 User-Agent: KMail/1.10.1 (FreeBSD/7.1-PRERELEASE; KDE/4.1.1; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810112330.53214.ken@mthelicon.com> Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 22:30:57 -0000 On Saturday 11 October 2008 21:53:35 Nate Eldredge wrote: > On Sat, 11 Oct 2008, Freddie Cash wrote: > > On 10/11/08, Matthew Dillon wrote: > >> With regards to the traditional BSD partitioning scheme, having a > >> separate /usr, /home, /tmp, etc... there's no reason to do that > >> stuff any more with ZFS (or HAMMER). > > > > As separate partitions, no. As separate filesystems, definitely. > > > > While HAMMER PFSes may not support these things yet, ZFS allows you to > > tailor each filesystem to its purpose. For example, you can enable > > compression on /usr/ports, but have a separate /usr/ports/distfilles > > and /usr/ports/work that aren't compressed. Or /usr/src compressed > > and /usr/obj not. Have a small record (block) size for /usr/src, but > > a larger one for /home. Give each user a separate filesystem for > > their /home/, with separate snapshot policies, quotas, and > > reservations (initial filesystem size). > > All this about ZFS sounds great, and I'd like to try it out, but some of > the bugs, etc, listed at http://wiki.freebsd.org/ZFSKnownProblems are > rather alarming. Even on a personal machine, I don't want these features > at the cost of an unstable system. Is that list still current? I dont know if that list is completely accurate any more, but I can tell you from my own personal experience with ZFS that it has been quite good. I have two servers (one is my test-bed at home) and the other is a production server running mostly mysql at work and I have never experienced the dead-locking problem. > > FWIW, my system is amd64 with 1 G of memory, which the page implies is > insufficient. Is it really? This may be purely subjective, as I have never bench marked the speeds, but when I was first testing zfs on a i386 machine with 1gig ram, I thought the performance was mediocre. However, when I loaded the system on a quad core - core2 with 8 gigs ram, I was quite impressed. I put localized changes in my /boot/loader.conf to give the kernel more breathing room and disabled the prefetch for zfs. #more loader.conf vm.kmem_size_max="1073741824" vm.kmem_size="1073741824" vfs.zfs.prefetch_disable=1 The best advice I can give is for you to find an old machine and test-bed zfs for yourself. I personally have been pleased with it and It has saved my machines data 4 times already (dieing hardware, unexpected power bounces, etc) As a side note, my production machine boots off a dedicated UFS drive (where I also have a slice for the swap). /usr, /var, /var/db and /usr/home are zfs. My test server at home only has /usr/home as zfs. I found it easier for me, when I kill the home machine to just do a reload/rebuild of the OS, rebuild the applications, and rechown/grp the home directories. Peg From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 23:21:57 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1985B1065699 for ; Sat, 11 Oct 2008 23:21:57 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from euclid.ucsd.edu (euclid.ucsd.edu [132.239.145.52]) by mx1.freebsd.org (Postfix) with ESMTP id EAB378FC0A for ; Sat, 11 Oct 2008 23:21:56 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from zeno.ucsd.edu (zeno.ucsd.edu [132.239.145.22]) by euclid.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9BNLt622709; Sat, 11 Oct 2008 16:21:55 -0700 (PDT) Received: from localhost (neldredg@localhost) by zeno.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9BNLtA11857; Sat, 11 Oct 2008 16:21:55 -0700 (PDT) X-Authentication-Warning: zeno.ucsd.edu: neldredg owned process doing -bs Date: Sat, 11 Oct 2008 16:21:55 -0700 (PDT) From: Nate Eldredge X-X-Sender: neldredg@zeno.ucsd.edu To: Pegasus Mc Cleaft In-Reply-To: <200810112330.53214.ken@mthelicon.com> Message-ID: References: <200810112330.53214.ken@mthelicon.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 23:21:57 -0000 On Sat, 11 Oct 2008, Pegasus Mc Cleaft wrote: >> FWIW, my system is amd64 with 1 G of memory, which the page implies is >> insufficient. Is it really? > > This may be purely subjective, as I have never bench marked the speeds, but > when I was first testing zfs on a i386 machine with 1gig ram, I thought the > performance was mediocre. However, when I loaded the system on a quad core - > core2 with 8 gigs ram, I was quite impressed. I put localized changes in my > /boot/loader.conf to give the kernel more breathing room and disabled the > prefetch for zfs. > > #more loader.conf > vm.kmem_size_max="1073741824" > vm.kmem_size="1073741824" > vfs.zfs.prefetch_disable=1 I was somewhat confused by the suggestions on the wiki. Do the kmem_size sysctls affect the allocation of *memory* or of *address space*? It seems a bit much to reserve 1 G of memory solely for the use of the kernel, expecially in my case when that's all I have :) But on amd64, it's welcome to have terabytes of address space if it will help. > The best advice I can give is for you to find an old machine and test-bed zfs > for yourself. I personally have been pleased with it and It has saved my > machines data 4 times already (dieing hardware, unexpected power bounces, etc) Sure, but if my "new" machine isn't studly enough to run it, there's no hope for an old machine. So I'm trying to figure out what I actually need. -- Nate Eldredge neldredge@math.ucsd.edu