From owner-svn-src-all@FreeBSD.ORG Mon Mar 28 17:41:10 2011 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BC501065677; Mon, 28 Mar 2011 17:41:10 +0000 (UTC) (envelope-from trociny@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 683098FC1A; Mon, 28 Mar 2011 17:41:10 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id p2SHfAR3001563; Mon, 28 Mar 2011 17:41:10 GMT (envelope-from trociny@svn.freebsd.org) Received: (from trociny@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id p2SHfAFu001547; Mon, 28 Mar 2011 17:41:10 GMT (envelope-from trociny@svn.freebsd.org) Message-Id: <201103281741.p2SHfAFu001547@svn.freebsd.org> From: Mikolaj Golub Date: Mon, 28 Mar 2011 17:41:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-8@freebsd.org X-SVN-Group: stable-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r220104 - in stable/8: etc etc/mail etc/rc.d sbin/hastctl sbin/hastd X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2011 17:41:10 -0000 Author: trociny Date: Mon Mar 28 17:41:10 2011 New Revision: 220104 URL: http://svn.freebsd.org/changeset/base/220104 Log: MFC r217729, r217730, r217731, r217732, r217737, r217784, r217958, r217961, r217962, r217964, r217965, r217966, r217967, r217969, r218040, r218041, r218042, r218043, r218044, r218045, r218048, r218049, r218119, r218132, r218138, r218139, r218147, r218148, r218158, r218185, r218191, r218192, r218193, r218194, r218201, r218214, r218215, r218217, r218218, r218370, r218373, r218374, r218375, r218376, r218464, r218465, r218474, r219082: r217729 (pjd): - On primary worker reload, update hr_exec field. - Update comment. r217730 (pjd): Use int16 for error. r217731 (pjd): Use more consistent function name with the others (pjdlogv_prefix_set() instead of pjdlog_prefix_setv()). r217732 (pjd): Add nv_assert() which allows to assert that the given name exists. r217737 (pjd): Add missing logs. r217784 (pjd): Don't open configuration file from worker process. Handle SIGHUP in the master process only and pass changes to the worker processes over control socket. This removes access to global namespace in preparation for capsicum sandboxing. r217958 (pjd): Remove __dead2 from pjdlog_verify() prototype, it does return sometimes. r217961 (pjd): - Remove obvious NOTREACHED comment after abort() call. - Remove redundant newline at the end of the file. r217962 (pjd): Add LOG_NDELAY flag to openlog(3) - we want descriptor to be immediately open so there are no surprises once we start chrooting or using capsicum. r217964 (pjd): Use pjd copyright for 2011 work. r217965 (pjd): Add functions to initialize/finalize pjdlog. This allows to open/close log file at will. r217966 (pjd): Extend pjdlog_verify() to support the following additional macros: PJDLOG_RVERIFY() - always check expression and on false log the given message and exit. PJDLOG_RASSERT() - check expression when NDEBUG is not defined and on false log given message and exit. PJDLOG_ABORT() - log the given message and exit. r217967 (pjd): Close the control socket before exiting, so it will be unlinked. r217969 (pjd): Remember created control connection so on fork(2) we can close it in child. r218040 (pjd): Initialize all global variables on pjdlog_init(). r218041 (pjd): Add function to close all unneeded descriptors after fork(2). r218042 (pjd): Add comments to places where we treat errors as ciritical, but it is possible to handle them more gracefully. r218043 (pjd): Close all unneeded descriptors after fork(2). r218044 (pjd): Add function to assert that the only descriptors we have open are the ones we expect to be open. Also assert that they point at expected type. Because openlog(3) API is unable to tell us descriptor number it is using, we have to close syslog socket, remember assert message in local buffer and if we fail on assertion, reopen syslog socket and log the message. r218045 (pjd): Use newly added descriptors_assert() function to ensure only expected descriptors are open. r218046 (pjd), r218047 (pjd), r218119 (maxim): Add 'hast' user and 'hast' group that will be used by hastd (and maybe hastctl) to drop privileges. r218048 (pjd): Implement function that drops privileges by: - chrooting to /var/empty (user hast home directory), - setting groups to 'hast' (user hast primary group), - setting real group id, effective group id and saved group id to 'hast', - setting real user id, effective user id and saved user id to 'hast'. At the end verify that those operations where successfull. r218049 (pjd): Drop privileges in worker processes. Accepting connections and handshaking in secondary is still done before dropping privileges. It should be implemented by only accepting connections in privileged main process and passing connection descriptors to the worker, but is not implemented yet. r218132 (pjd): Rename pjdlog_verify() to pjdlog_abort() as it better describes what the the function does and mark it with __dead2. r218138 (pjd): - Use pjdlog for assertions and aborts as this will log assert/abort message to syslog if we run in background. - Asserts in proto.c that method we want to call is implemented and remove dummy methods from protocols implementation that are only there to abort the program with nice message. r218139 (pjd): Implement two new functions for sending descriptor and receving descriptor over UNIX domain sockets and socket pairs. This is in preparation for capsicum. r218147 (pjd), r218148 (pjd): Fix build on ia64. r218158 (pjd): Do not set socket send and receive buffer. It will be auto-tuned. Confirmed by: rwatson r218185 (pjd): Be prepared that hp_client or hp_server might be NULL now. r218191 (pjd): Move protocol allocation and deallocation to separate functions. r218192 (pjd), r218201 (bz): Allow to specify connection timeout by the caller. r218193 (pjd): Add proto_connect_wait() to wait for connection to finish. If timeout argument to proto_connect() is -1, then the caller needs to use this new function to wait for connection. This change is in preparation for capsicum, where sandboxed worker wants to ask main process to connect in worker's behalf and pass descriptor to the worker. Because we don't want the main process to wait for the connection, it will start async connection and pass descriptor to the worker who will be responsible for waiting for the connection to finish. r218194 (pjd): - Rename proto_descriptor_{send,recv}() functions to proto_connection_{send,recv} and change them to return proto_conn structure. We don't operate directly on descriptors, but on proto_conns. - Add wrap method to wrap descriptor with proto_conn. - Remove methods to send and receive descriptors and implement this functionality as additional argument to send and receive methods. r218214 (pjd): Let the caller log info about successful privilege drop. We don't want to log this in hastctl. r218215 (pjd): Drop privileges after connecting to hastd, but before sending or receiving anything. r218217 (pjd): Add missing locking after moving keepalive_send() to remote send thread in r214692. r218218 (pjd): Setup another socketpair between parent and child, so that primary sandboxed worker can ask the main privileged process to connect in worker's behalf and then we can migrate descriptor using this socketpair to worker. This is not really needed now, but will be needed once we start to use capsicum for sandboxing. r218370 (pjd): Close more descriptors that can be open if the worker process for the given resource is already running. Submitted by: Mikolaj Golub r218373 (pjd): Open syslog when logging sysconf(3) failure. Reported by: Mikolaj Golub r218374 (pjd): Treat fstat(2) failure (different than EBADF) as fatal error. Reported by: Mikolaj Golub r218375 (pjd): Add (void) cast before snprintf(3)s for which we are not interested in return values. r218376 (pjd): Now that we break the loop on fstat(2) failure we no longer need to satisfy gcc's imperfections. r218464 (pjd): Unlink UNIX domain socket file only if: 1. The descriptor is the one we are listening on (not the one when we connect as a client and not the one which is created on accept(2)). 2. Descriptor was created by us (PID matches with the PID stored on bind(2)). Reported by: Mikolaj Golub r218465 (pjd): Explicitly include as suggested by getpid(2) and don't rely on including what's needed. r218474 (pjd): When we decide to unlink socket file, sun_path must be set. If it is set, but there is problem unlinking the file, log a warning. r219082 (pjd): Recognize 'reload' command, as hastd can be reloaded with the SIGHUP signal. Approved by: pjd (mentor) Modified: stable/8/etc/ftpusers stable/8/etc/group stable/8/etc/mail/aliases stable/8/etc/master.passwd stable/8/etc/rc.d/hastd stable/8/sbin/hastctl/hastctl.c stable/8/sbin/hastd/control.c stable/8/sbin/hastd/control.h stable/8/sbin/hastd/hast.h stable/8/sbin/hastd/hastd.c stable/8/sbin/hastd/hastd.h stable/8/sbin/hastd/nv.c stable/8/sbin/hastd/nv.h stable/8/sbin/hastd/pjdlog.c stable/8/sbin/hastd/pjdlog.h stable/8/sbin/hastd/primary.c stable/8/sbin/hastd/proto.c stable/8/sbin/hastd/proto.h stable/8/sbin/hastd/proto_common.c stable/8/sbin/hastd/proto_impl.h stable/8/sbin/hastd/proto_socketpair.c stable/8/sbin/hastd/proto_tcp4.c stable/8/sbin/hastd/proto_uds.c stable/8/sbin/hastd/secondary.c stable/8/sbin/hastd/subr.c stable/8/sbin/hastd/subr.h Directory Properties: stable/8/etc/ (props changed) stable/8/sbin/hastctl/ (props changed) stable/8/sbin/hastd/ (props changed) Modified: stable/8/etc/ftpusers ============================================================================== --- stable/8/etc/ftpusers Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/etc/ftpusers Mon Mar 28 17:41:10 2011 (r220104) @@ -20,6 +20,7 @@ _dhcp uucp pop www +hast nobody mailnull smmsp Modified: stable/8/etc/group ============================================================================== --- stable/8/etc/group Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/etc/group Mon Mar 28 17:41:10 2011 (r220104) @@ -27,5 +27,6 @@ dialer:*:68: network:*:69: audit:*:77: www:*:80: +hast:*:845: nogroup:*:65533: nobody:*:65534: Modified: stable/8/etc/mail/aliases ============================================================================== --- stable/8/etc/mail/aliases Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/etc/mail/aliases Mon Mar 28 17:41:10 2011 (r220104) @@ -30,6 +30,7 @@ bin: root bind: root daemon: root games: root +hast: root kmem: root mailnull: postmaster man: root Modified: stable/8/etc/master.passwd ============================================================================== --- stable/8/etc/master.passwd Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/etc/master.passwd Mon Mar 28 17:41:10 2011 (r220104) @@ -20,4 +20,5 @@ _dhcp:*:65:65::0:0:dhcp programs:/var/em uucp:*:66:66::0:0:UUCP pseudo-user:/var/spool/uucppublic:/usr/local/libexec/uucp/uucico pop:*:68:6::0:0:Post Office Owner:/nonexistent:/usr/sbin/nologin www:*:80:80::0:0:World Wide Web Owner:/nonexistent:/usr/sbin/nologin +hast:*:845:845::0:0:HAST unprivileged user:/var/empty:/usr/sbin/nologin nobody:*:65534:65534::0:0:Unprivileged user:/nonexistent:/usr/sbin/nologin Modified: stable/8/etc/rc.d/hastd ============================================================================== --- stable/8/etc/rc.d/hastd Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/etc/rc.d/hastd Mon Mar 28 17:41:10 2011 (r220104) @@ -18,6 +18,7 @@ hastctl="/sbin/hastctl" required_files="/etc/hast.conf" stop_precmd="hastd_stop_precmd" required_modules="geom_gate:g_gate" +extra_commands="reload" hastd_stop_precmd() { Modified: stable/8/sbin/hastctl/hastctl.c ============================================================================== --- stable/8/sbin/hastctl/hastctl.c Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastctl/hastctl.c Mon Mar 28 17:41:10 2011 (r220104) @@ -430,6 +430,7 @@ main(int argc, char *argv[]) break; } + pjdlog_init(PJDLOG_MODE_STD); pjdlog_debug_set(debug); cfg = yy_config_parse(cfgpath, true); @@ -486,10 +487,15 @@ main(int argc, char *argv[]) cfg->hc_controladdr); } /* ...and connect to hastd. */ - if (proto_connect(controlconn) < 0) { + if (proto_connect(controlconn, HAST_TIMEOUT) < 0) { pjdlog_exit(EX_OSERR, "Unable to connect to hastd via %s", cfg->hc_controladdr); } + + if (drop_privs() != 0) + exit(EX_CONFIG); + pjdlog_debug(1, "Privileges successfully dropped."); + /* Send the command to the server... */ if (hast_proto_send(NULL, controlconn, nv, NULL, 0) < 0) { pjdlog_exit(EX_UNAVAILABLE, Modified: stable/8/sbin/hastd/control.c ============================================================================== --- stable/8/sbin/hastd/control.c Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/control.c Mon Mar 28 17:41:10 2011 (r220104) @@ -62,6 +62,10 @@ child_cleanup(struct hast_resource *res) proto_close(res->hr_event); res->hr_event = NULL; } + if (res->hr_conn != NULL) { + proto_close(res->hr_conn); + res->hr_conn = NULL; + } res->hr_workerpid = 0; } @@ -159,12 +163,13 @@ control_status_worker(struct hast_resour nv_add_uint8(cnvout, HASTCTL_STATUS, "cmd"); error = nv_error(cnvout); if (error != 0) { - /* LOG */ + pjdlog_common(LOG_ERR, 0, error, + "Unable to prepare control header"); goto end; } if (hast_proto_send(res, res->hr_ctrl, cnvout, NULL, 0) < 0) { error = errno; - /* LOG */ + pjdlog_errno(LOG_ERR, "Unable to send control header"); goto end; } @@ -173,17 +178,17 @@ control_status_worker(struct hast_resour */ if (hast_proto_recv_hdr(res->hr_ctrl, &cnvin) < 0) { error = errno; - /* LOG */ + pjdlog_errno(LOG_ERR, "Unable to receive control header"); goto end; } - error = nv_get_int64(cnvin, "error"); + error = nv_get_int16(cnvin, "error"); if (error != 0) goto end; if ((str = nv_get_string(cnvin, "status")) == NULL) { error = ENOENT; - /* LOG */ + pjdlog_errno(LOG_ERR, "Field 'status' is missing."); goto end; } nv_add_string(nvout, str, "status%u", no); @@ -277,6 +282,7 @@ control_handle(struct hastd_config *cfg) return; } + cfg->hc_controlin = conn; nvin = nvout = NULL; role = HAST_ROLE_UNDEF; @@ -383,6 +389,7 @@ close: if (nvout != NULL) nv_free(nvout); proto_close(conn); + cfg->hc_controlin = NULL; } /* @@ -410,7 +417,6 @@ ctrl_thread(void *arg) nv_free(nvin); continue; } - nv_free(nvin); nvout = nv_alloc(); switch (cmd) { case HASTCTL_STATUS: @@ -432,11 +438,23 @@ ctrl_thread(void *arg) nv_add_uint32(nvout, (uint32_t)0, "keepdirty"); nv_add_uint64(nvout, (uint64_t)0, "dirty"); } + nv_add_int16(nvout, 0, "error"); + break; + case HASTCTL_RELOAD: + /* + * When parent receives SIGHUP and discovers that + * something related to us has changes, it sends reload + * message to us. + */ + assert(res->hr_role == HAST_ROLE_PRIMARY); + primary_config_reload(res, nvin); + nv_add_int16(nvout, 0, "error"); break; default: nv_add_int16(nvout, EINVAL, "error"); break; } + nv_free(nvin); if (nv_error(nvout) != 0) { pjdlog_error("Unable to create answer on control message."); nv_free(nvout); Modified: stable/8/sbin/hastd/control.h ============================================================================== --- stable/8/sbin/hastd/control.h Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/control.h Mon Mar 28 17:41:10 2011 (r220104) @@ -34,6 +34,7 @@ #define HASTCTL_SET_ROLE 1 #define HASTCTL_STATUS 2 +#define HASTCTL_RELOAD 3 struct hastd_config; struct hast_resource; Modified: stable/8/sbin/hastd/hast.h ============================================================================== --- stable/8/sbin/hastd/hast.h Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/hast.h Mon Mar 28 17:41:10 2011 (r220104) @@ -81,6 +81,7 @@ #define HIO_FLUSH 4 #define HIO_KEEPALIVE 5 +#define HAST_USER "hast" #define HAST_TIMEOUT 5 #define HAST_CONFIG "/etc/hast.conf" #define HAST_CONTROL "/var/run/hastctl" @@ -101,6 +102,8 @@ struct hastd_config { char hc_controladdr[HAST_ADDRSIZE]; /* Protocol-specific data. */ struct proto_conn *hc_controlconn; + /* Incoming control connection. */ + struct proto_conn *hc_controlin; /* Address to listen on. */ char hc_listenaddr[HAST_ADDRSIZE]; /* Protocol-specific data. */ @@ -179,10 +182,12 @@ struct hast_resource { int hr_previous_role; /* PID of child worker process. 0 - no child. */ pid_t hr_workerpid; - /* Control connection between parent and child. */ + /* Control commands from parent to child. */ struct proto_conn *hr_ctrl; /* Events from child to parent. */ struct proto_conn *hr_event; + /* Connection requests from child to parent. */ + struct proto_conn *hr_conn; /* Activemap structure. */ struct activemap *hr_amp; Modified: stable/8/sbin/hastd/hastd.c ============================================================================== --- stable/8/sbin/hastd/hastd.c Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/hastd.c Mon Mar 28 17:41:10 2011 (r220104) @@ -1,6 +1,6 @@ /*- * Copyright (c) 2009-2010 The FreeBSD Foundation - * Copyright (c) 2010 Pawel Jakub Dawidek + * Copyright (c) 2010-2011 Pawel Jakub Dawidek * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from @@ -34,9 +34,9 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include -#include #include #include #include @@ -93,6 +93,194 @@ g_gate_load(void) } } +void +descriptors_cleanup(struct hast_resource *res) +{ + struct hast_resource *tres; + + TAILQ_FOREACH(tres, &cfg->hc_resources, hr_next) { + if (tres == res) { + PJDLOG_VERIFY(res->hr_role == HAST_ROLE_SECONDARY || + (res->hr_remotein == NULL && + res->hr_remoteout == NULL)); + continue; + } + if (tres->hr_remotein != NULL) + proto_close(tres->hr_remotein); + if (tres->hr_remoteout != NULL) + proto_close(tres->hr_remoteout); + if (tres->hr_ctrl != NULL) + proto_close(tres->hr_ctrl); + if (tres->hr_event != NULL) + proto_close(tres->hr_event); + if (tres->hr_conn != NULL) + proto_close(tres->hr_conn); + } + if (cfg->hc_controlin != NULL) + proto_close(cfg->hc_controlin); + proto_close(cfg->hc_controlconn); + proto_close(cfg->hc_listenconn); + (void)pidfile_close(pfh); + hook_fini(); + pjdlog_fini(); +} + +static const char * +dtype2str(mode_t mode) +{ + + if (S_ISBLK(mode)) + return ("block device"); + else if (S_ISCHR(mode)) + return ("character device"); + else if (S_ISDIR(mode)) + return ("directory"); + else if (S_ISFIFO(mode)) + return ("pipe or FIFO"); + else if (S_ISLNK(mode)) + return ("symbolic link"); + else if (S_ISREG(mode)) + return ("regular file"); + else if (S_ISSOCK(mode)) + return ("socket"); + else if (S_ISWHT(mode)) + return ("whiteout"); + else + return ("unknown"); +} + +void +descriptors_assert(const struct hast_resource *res, int pjdlogmode) +{ + char msg[256]; + struct stat sb; + long maxfd; + bool isopen; + mode_t mode; + int fd; + + /* + * At this point descriptor to syslog socket is closed, so if we want + * to log assertion message, we have to first store it in 'msg' local + * buffer and then open syslog socket and log it. + */ + msg[0] = '\0'; + + maxfd = sysconf(_SC_OPEN_MAX); + if (maxfd < 0) { + pjdlog_init(pjdlogmode); + pjdlog_prefix_set("[%s] (%s) ", res->hr_name, + role2str(res->hr_role)); + pjdlog_errno(LOG_WARNING, "sysconf(_SC_OPEN_MAX) failed"); + pjdlog_fini(); + maxfd = 16384; + } + for (fd = 0; fd <= maxfd; fd++) { + if (fstat(fd, &sb) == 0) { + isopen = true; + mode = sb.st_mode; + } else if (errno == EBADF) { + isopen = false; + mode = 0; + } else { + (void)snprintf(msg, sizeof(msg), + "Unable to fstat descriptor %d: %s", fd, + strerror(errno)); + break; + } + if (fd == STDIN_FILENO || fd == STDOUT_FILENO || + fd == STDERR_FILENO) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (%s) is closed, but should be open.", + fd, (fd == STDIN_FILENO ? "stdin" : + (fd == STDOUT_FILENO ? "stdout" : "stderr"))); + break; + } + } else if (fd == proto_descriptor(res->hr_event)) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (event) is closed, but should be open.", + fd); + break; + } + if (!S_ISSOCK(mode)) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (event) is %s, but should be %s.", + fd, dtype2str(mode), dtype2str(S_IFSOCK)); + break; + } + } else if (fd == proto_descriptor(res->hr_ctrl)) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (ctrl) is closed, but should be open.", + fd); + break; + } + if (!S_ISSOCK(mode)) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (ctrl) is %s, but should be %s.", + fd, dtype2str(mode), dtype2str(S_IFSOCK)); + break; + } + } else if (fd == proto_descriptor(res->hr_conn)) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (conn) is closed, but should be open.", + fd); + break; + } + if (!S_ISSOCK(mode)) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (conn) is %s, but should be %s.", + fd, dtype2str(mode), dtype2str(S_IFSOCK)); + break; + } + } else if (res->hr_role == HAST_ROLE_SECONDARY && + fd == proto_descriptor(res->hr_remotein)) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (remote in) is closed, but should be open.", + fd); + break; + } + if (!S_ISSOCK(mode)) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (remote in) is %s, but should be %s.", + fd, dtype2str(mode), dtype2str(S_IFSOCK)); + break; + } + } else if (res->hr_role == HAST_ROLE_SECONDARY && + fd == proto_descriptor(res->hr_remoteout)) { + if (!isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (remote out) is closed, but should be open.", + fd); + break; + } + if (!S_ISSOCK(mode)) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d (remote out) is %s, but should be %s.", + fd, dtype2str(mode), dtype2str(S_IFSOCK)); + break; + } + } else { + if (isopen) { + (void)snprintf(msg, sizeof(msg), + "Descriptor %d is open (%s), but should be closed.", + fd, dtype2str(mode)); + break; + } + } + } + if (msg[0] != '\0') { + pjdlog_init(pjdlogmode); + pjdlog_prefix_set("[%s] (%s) ", res->hr_name, + role2str(res->hr_role)); + PJDLOG_ABORT("%s", msg); + } +} + static void child_exit_log(unsigned int pid, int status) { @@ -161,7 +349,7 @@ resource_needs_restart(const struct hast const struct hast_resource *res1) { - assert(strcmp(res0->hr_name, res1->hr_name) == 0); + PJDLOG_ASSERT(strcmp(res0->hr_name, res1->hr_name) == 0); if (strcmp(res0->hr_provname, res1->hr_provname) != 0) return (true); @@ -186,9 +374,9 @@ resource_needs_reload(const struct hast_ const struct hast_resource *res1) { - assert(strcmp(res0->hr_name, res1->hr_name) == 0); - assert(strcmp(res0->hr_provname, res1->hr_provname) == 0); - assert(strcmp(res0->hr_localpath, res1->hr_localpath) == 0); + PJDLOG_ASSERT(strcmp(res0->hr_name, res1->hr_name) == 0); + PJDLOG_ASSERT(strcmp(res0->hr_provname, res1->hr_provname) == 0); + PJDLOG_ASSERT(strcmp(res0->hr_localpath, res1->hr_localpath) == 0); if (res0->hr_role != HAST_ROLE_PRIMARY) return (false); @@ -205,6 +393,45 @@ resource_needs_reload(const struct hast_ } static void +resource_reload(const struct hast_resource *res) +{ + struct nv *nvin, *nvout; + int error; + + PJDLOG_ASSERT(res->hr_role == HAST_ROLE_PRIMARY); + + nvout = nv_alloc(); + nv_add_uint8(nvout, HASTCTL_RELOAD, "cmd"); + nv_add_string(nvout, res->hr_remoteaddr, "remoteaddr"); + nv_add_int32(nvout, (int32_t)res->hr_replication, "replication"); + nv_add_int32(nvout, (int32_t)res->hr_timeout, "timeout"); + nv_add_string(nvout, res->hr_exec, "exec"); + if (nv_error(nvout) != 0) { + nv_free(nvout); + pjdlog_error("Unable to allocate header for reload message."); + return; + } + if (hast_proto_send(res, res->hr_ctrl, nvout, NULL, 0) < 0) { + pjdlog_errno(LOG_ERR, "Unable to send reload message"); + nv_free(nvout); + return; + } + nv_free(nvout); + + /* Receive response. */ + if (hast_proto_recv_hdr(res->hr_ctrl, &nvin) < 0) { + pjdlog_errno(LOG_ERR, "Unable to receive reload reply"); + return; + } + error = nv_get_int16(nvin, "error"); + nv_free(nvin); + if (error != 0) { + pjdlog_common(LOG_ERR, 0, error, "Reload failed"); + return; + } +} + +static void hastd_reload(void) { struct hastd_config *newcfg; @@ -306,8 +533,9 @@ hastd_reload(void) * recreating it. * * We do just reload (send SIGHUP to worker process) if we act as - * PRIMARY, but only remote address, replication mode and timeout - * has changed. For those, there is no need to restart worker process. + * PRIMARY, but only if remote address, replication mode, timeout or + * execution path has changed. For those, there is no need to restart + * worker process. * If PRIMARY receives SIGHUP, it will reconnect if remote address or * replication mode has changed or simply set new timeout if only * timeout has changed. @@ -317,7 +545,7 @@ hastd_reload(void) if (strcmp(cres->hr_name, nres->hr_name) == 0) break; } - assert(cres != NULL); + PJDLOG_ASSERT(cres != NULL); if (resource_needs_restart(cres, nres)) { pjdlog_info("Resource %s configuration was modified, restarting it.", cres->hr_name); @@ -335,13 +563,10 @@ hastd_reload(void) sizeof(cres->hr_remoteaddr)); cres->hr_replication = nres->hr_replication; cres->hr_timeout = nres->hr_timeout; - if (cres->hr_workerpid != 0) { - if (kill(cres->hr_workerpid, SIGHUP) < 0) { - pjdlog_errno(LOG_WARNING, - "Unable to send SIGHUP to worker process %u", - (unsigned int)cres->hr_workerpid); - } - } + strlcpy(cres->hr_exec, nres->hr_exec, + sizeof(cres->hr_exec)); + if (cres->hr_workerpid != 0) + resource_reload(cres); } } @@ -496,10 +721,10 @@ listen_accept(void) * we have to cancel those and accept the new connection. */ if (token == NULL) { - assert(res->hr_remoteout == NULL); + PJDLOG_ASSERT(res->hr_remoteout == NULL); pjdlog_debug(1, "Initial connection from %s.", raddr); if (res->hr_workerpid != 0) { - assert(res->hr_remotein == NULL); + PJDLOG_ASSERT(res->hr_remotein == NULL); pjdlog_debug(1, "Worker process exists (pid=%u), stopping it.", (unsigned int)res->hr_workerpid); @@ -599,6 +824,41 @@ close: } static void +connection_migrate(struct hast_resource *res) +{ + struct proto_conn *conn; + int16_t val = 0; + + if (proto_recv(res->hr_conn, &val, sizeof(val)) < 0) { + pjdlog_errno(LOG_WARNING, + "Unable to receive connection command"); + return; + } + if (proto_client(res->hr_remoteaddr, &conn) < 0) { + val = errno; + pjdlog_errno(LOG_WARNING, + "Unable to create outgoing connection to %s", + res->hr_remoteaddr); + goto out; + } + if (proto_connect(conn, -1) < 0) { + val = errno; + pjdlog_errno(LOG_WARNING, "Unable to connect to %s", + res->hr_remoteaddr); + proto_close(conn); + goto out; + } + val = 0; +out: + if (proto_send(res->hr_conn, &val, sizeof(val)) < 0) { + pjdlog_errno(LOG_WARNING, + "Unable to send reply to connection request"); + } + if (val == 0 && proto_connection_send(res->hr_conn, conn) < 0) + pjdlog_errno(LOG_WARNING, "Unable to send connection"); +} + +static void main_loop(void) { struct hast_resource *res; @@ -629,6 +889,7 @@ main_loop(void) case SIGTERM: sigexit_received = true; terminate_workers(); + proto_close(cfg->hc_controlconn); exit(EX_OK); break; case SIGCHLD: @@ -638,29 +899,37 @@ main_loop(void) hastd_reload(); break; default: - assert(!"invalid condition"); + PJDLOG_ABORT("Unexpected signal (%d).", signo); } } /* Setup descriptors for select(2). */ FD_ZERO(&rfds); maxfd = fd = proto_descriptor(cfg->hc_controlconn); - assert(fd >= 0); + PJDLOG_ASSERT(fd >= 0); FD_SET(fd, &rfds); fd = proto_descriptor(cfg->hc_listenconn); - assert(fd >= 0); + PJDLOG_ASSERT(fd >= 0); FD_SET(fd, &rfds); maxfd = fd > maxfd ? fd : maxfd; TAILQ_FOREACH(res, &cfg->hc_resources, hr_next) { if (res->hr_event == NULL) continue; + PJDLOG_ASSERT(res->hr_conn != NULL); fd = proto_descriptor(res->hr_event); - assert(fd >= 0); + PJDLOG_ASSERT(fd >= 0); FD_SET(fd, &rfds); maxfd = fd > maxfd ? fd : maxfd; + if (res->hr_role == HAST_ROLE_PRIMARY) { + /* Only primary workers asks for connections. */ + fd = proto_descriptor(res->hr_conn); + PJDLOG_ASSERT(fd >= 0); + FD_SET(fd, &rfds); + maxfd = fd > maxfd ? fd : maxfd; + } } - assert(maxfd + 1 <= (int)FD_SETSIZE); + PJDLOG_ASSERT(maxfd + 1 <= (int)FD_SETSIZE); ret = select(maxfd + 1, &rfds, NULL, NULL, &seltimeout); if (ret == 0) hook_check(); @@ -678,12 +947,20 @@ main_loop(void) TAILQ_FOREACH(res, &cfg->hc_resources, hr_next) { if (res->hr_event == NULL) continue; + PJDLOG_ASSERT(res->hr_conn != NULL); if (FD_ISSET(proto_descriptor(res->hr_event), &rfds)) { if (event_recv(res) == 0) continue; /* The worker process exited? */ proto_close(res->hr_event); res->hr_event = NULL; + proto_close(res->hr_conn); + res->hr_conn = NULL; + continue; + } + if (res->hr_role == HAST_ROLE_PRIMARY && + FD_ISSET(proto_descriptor(res->hr_conn), &rfds)) { + connection_migrate(res); } } } @@ -735,6 +1012,7 @@ main(int argc, char *argv[]) argc -= optind; argv += optind; + pjdlog_init(PJDLOG_MODE_STD); pjdlog_debug_set(debuglevel); g_gate_load(); @@ -751,7 +1029,7 @@ main(int argc, char *argv[]) } cfg = yy_config_parse(cfgpath, true); - assert(cfg != NULL); + PJDLOG_ASSERT(cfg != NULL); /* * Restore default actions for interesting signals in case parent Modified: stable/8/sbin/hastd/hastd.h ============================================================================== --- stable/8/sbin/hastd/hastd.h Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/hastd.h Mon Mar 28 17:41:10 2011 (r220104) @@ -43,7 +43,12 @@ extern const char *cfgpath; extern bool sigexit_received; extern struct pidfh *pfh; +void descriptors_cleanup(struct hast_resource *res); +void descriptors_assert(const struct hast_resource *res, int pjdlogmode); + void hastd_primary(struct hast_resource *res); void hastd_secondary(struct hast_resource *res, struct nv *nvin); +void primary_config_reload(struct hast_resource *res, struct nv *nv); + #endif /* !_HASTD_H_ */ Modified: stable/8/sbin/hastd/nv.c ============================================================================== --- stable/8/sbin/hastd/nv.c Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/nv.c Mon Mar 28 17:41:10 2011 (r220104) @@ -563,11 +563,10 @@ nv_get_string(struct nv *nv, const char return (str); } -bool -nv_exists(struct nv *nv, const char *namefmt, ...) +static bool +nv_vexists(struct nv *nv, const char *namefmt, va_list nameap) { struct nvhdr *nvh; - va_list nameap; int snverror, serrno; if (nv == NULL) @@ -576,9 +575,7 @@ nv_exists(struct nv *nv, const char *nam serrno = errno; snverror = nv->nv_error; - va_start(nameap, namefmt); nvh = nv_find(nv, NV_TYPE_NONE, namefmt, nameap); - va_end(nameap); errno = serrno; nv->nv_error = snverror; @@ -586,6 +583,29 @@ nv_exists(struct nv *nv, const char *nam return (nvh != NULL); } +bool +nv_exists(struct nv *nv, const char *namefmt, ...) +{ + va_list nameap; + bool ret; + + va_start(nameap, namefmt); + ret = nv_vexists(nv, namefmt, nameap); + va_end(nameap); + + return (ret); +} + +void +nv_assert(struct nv *nv, const char *namefmt, ...) +{ + va_list nameap; + + va_start(nameap, namefmt); + assert(nv_vexists(nv, namefmt, nameap)); + va_end(nameap); +} + /* * Dump content of the nv structure. */ Modified: stable/8/sbin/hastd/nv.h ============================================================================== --- stable/8/sbin/hastd/nv.h Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/nv.h Mon Mar 28 17:41:10 2011 (r220104) @@ -127,6 +127,7 @@ const char *nv_get_string(struct nv *nv, __printflike(2, 3); bool nv_exists(struct nv *nv, const char *namefmt, ...) __printflike(2, 3); +void nv_assert(struct nv *nv, const char *namefmt, ...) __printflike(2, 3); void nv_dump(struct nv *nv); #endif /* !_NV_H_ */ Modified: stable/8/sbin/hastd/pjdlog.c ============================================================================== --- stable/8/sbin/hastd/pjdlog.c Mon Mar 28 16:58:48 2011 (r220103) +++ stable/8/sbin/hastd/pjdlog.c Mon Mar 28 17:41:10 2011 (r220104) @@ -1,5 +1,6 @@ /*- * Copyright (c) 2009-2010 The FreeBSD Foundation + * Copyright (c) 2011 Pawel Jakub Dawidek * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from @@ -33,6 +34,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -40,10 +42,38 @@ __FBSDID("$FreeBSD$"); #include "pjdlog.h" -static int pjdlog_mode = PJDLOG_MODE_STD; -static int pjdlog_debug_level = 0; +static bool pjdlog_initialized = false; +static int pjdlog_mode, pjdlog_debug_level; static char pjdlog_prefix[128]; +void +pjdlog_init(int mode) +{ + + assert(!pjdlog_initialized); + assert(mode == PJDLOG_MODE_STD || mode == PJDLOG_MODE_SYSLOG); + + if (mode == PJDLOG_MODE_SYSLOG) + openlog(NULL, LOG_PID | LOG_NDELAY, LOG_DAEMON); + pjdlog_mode = mode; + pjdlog_debug_level = 0; + bzero(pjdlog_prefix, sizeof(pjdlog_prefix)); + + pjdlog_initialized = true; +} + +void +pjdlog_fini(void) +{ + + assert(pjdlog_initialized); + + if (pjdlog_mode == PJDLOG_MODE_SYSLOG) + closelog(); + + pjdlog_initialized = false; +} + /* * Configure where the logs should go. * By default they are send to stdout/stderr, but after going into background @@ -54,12 +84,18 @@ void pjdlog_mode_set(int mode) { + assert(pjdlog_initialized); assert(mode == PJDLOG_MODE_STD || mode == PJDLOG_MODE_SYSLOG); - pjdlog_mode = mode; + if (pjdlog_mode == mode) + return; if (mode == PJDLOG_MODE_SYSLOG) - openlog(NULL, LOG_PID, LOG_DAEMON); + openlog(NULL, LOG_PID | LOG_NDELAY, LOG_DAEMON); + else /* if (mode == PJDLOG_MODE_STD) */ + closelog(); + + pjdlog_mode = mode; } /* @@ -69,6 +105,8 @@ int pjdlog_mode_get(void) { + assert(pjdlog_initialized); + return (pjdlog_mode); } @@ -80,6 +118,7 @@ void pjdlog_debug_set(int level) { + assert(pjdlog_initialized); assert(level >= 0); pjdlog_debug_level = level; @@ -92,6 +131,8 @@ int pjdlog_debug_get(void) { + assert(pjdlog_initialized); + return (pjdlog_debug_level); } @@ -104,8 +145,10 @@ pjdlog_prefix_set(const char *fmt, ...) { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); - pjdlog_prefix_setv(fmt, ap); + pjdlogv_prefix_set(fmt, ap); va_end(ap); } @@ -114,9 +157,10 @@ pjdlog_prefix_set(const char *fmt, ...) * Setting prefix to NULL will remove it. */ void -pjdlog_prefix_setv(const char *fmt, va_list ap) +pjdlogv_prefix_set(const char *fmt, va_list ap) { + assert(pjdlog_initialized); assert(fmt != NULL); vsnprintf(pjdlog_prefix, sizeof(pjdlog_prefix), fmt, ap); @@ -159,6 +203,8 @@ pjdlog_common(int loglevel, int debuglev { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv_common(loglevel, debuglevel, error, fmt, ap); va_end(ap); @@ -173,6 +219,7 @@ pjdlogv_common(int loglevel, int debugle va_list ap) { + assert(pjdlog_initialized); assert(loglevel == LOG_EMERG || loglevel == LOG_ALERT || loglevel == LOG_CRIT || loglevel == LOG_ERR || loglevel == LOG_WARNING || loglevel == LOG_NOTICE || @@ -249,6 +296,8 @@ void pjdlogv(int loglevel, const char *fmt, va_list ap) { + assert(pjdlog_initialized); + /* LOG_DEBUG is invalid here, pjdlogv?_debug() should be used. */ assert(loglevel == LOG_EMERG || loglevel == LOG_ALERT || loglevel == LOG_CRIT || loglevel == LOG_ERR || @@ -266,6 +315,8 @@ pjdlog(int loglevel, const char *fmt, .. { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv(loglevel, fmt, ap); va_end(ap); @@ -278,6 +329,8 @@ void pjdlogv_debug(int debuglevel, const char *fmt, va_list ap) { + assert(pjdlog_initialized); + pjdlogv_common(LOG_DEBUG, debuglevel, -1, fmt, ap); } @@ -289,6 +342,8 @@ pjdlog_debug(int debuglevel, const char { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv_debug(debuglevel, fmt, ap); va_end(ap); @@ -301,6 +356,8 @@ void pjdlogv_errno(int loglevel, const char *fmt, va_list ap) { + assert(pjdlog_initialized); + pjdlogv_common(loglevel, 0, errno, fmt, ap); } @@ -312,6 +369,8 @@ pjdlog_errno(int loglevel, const char *f { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv_errno(loglevel, fmt, ap); va_end(ap); @@ -324,6 +383,8 @@ void pjdlogv_exit(int exitcode, const char *fmt, va_list ap) { + assert(pjdlog_initialized); + pjdlogv_errno(LOG_ERR, fmt, ap); exit(exitcode); /* NOTREACHED */ @@ -337,6 +398,8 @@ pjdlog_exit(int exitcode, const char *fm { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv_exit(exitcode, fmt, ap); /* NOTREACHED */ @@ -350,6 +413,8 @@ void pjdlogv_exitx(int exitcode, const char *fmt, va_list ap) { + assert(pjdlog_initialized); + pjdlogv(LOG_ERR, fmt, ap); exit(exitcode); /* NOTREACHED */ @@ -363,6 +428,8 @@ pjdlog_exitx(int exitcode, const char *f { va_list ap; + assert(pjdlog_initialized); + va_start(ap, fmt); pjdlogv_exitx(exitcode, fmt, ap); /* NOTREACHED */ @@ -370,21 +437,42 @@ pjdlog_exitx(int exitcode, const char *f *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***