From owner-dev-commits-src-main@freebsd.org Sat Jun 12 01:00:42 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D101F64011F; Sat, 12 Jun 2021 01:00:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4G1zsk5dJFz3G4L; Sat, 12 Jun 2021 01:00:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A9C3D2D00D; Sat, 12 Jun 2021 01:00:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 15C10gRK084686; Sat, 12 Jun 2021 01:00:42 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 15C10gg8084685; Sat, 12 Jun 2021 01:00:42 GMT (envelope-from git) Date: Sat, 12 Jun 2021 01:00:42 GMT Message-Id: <202106120100.15C10gg8084685@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: e8424e2947dd - main - bhyve: Register new kevents synchronously. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: e8424e2947dd8a0e1d3f08295432941b5c5aafb9 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jun 2021 01:00:42 -0000 The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=e8424e2947dd8a0e1d3f08295432941b5c5aafb9 commit e8424e2947dd8a0e1d3f08295432941b5c5aafb9 Author: John Baldwin AuthorDate: 2021-06-12 00:58:54 +0000 Commit: John Baldwin CommitDate: 2021-06-12 01:00:24 +0000 bhyve: Register new kevents synchronously. Change mevent_add*() to synchronously add the new kevent. This permits reporting event registration failures to the caller and avoids failing the registration of other, unrelated events queued up in the same batch. Reviewed by: grehan, markj MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D30502 --- usr.sbin/bhyve/mevent.c | 96 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 63 insertions(+), 33 deletions(-) diff --git a/usr.sbin/bhyve/mevent.c b/usr.sbin/bhyve/mevent.c index 2e9059837011..a394b7c5d69f 100644 --- a/usr.sbin/bhyve/mevent.c +++ b/usr.sbin/bhyve/mevent.c @@ -64,8 +64,10 @@ __FBSDID("$FreeBSD$"); #define MEVENT_MAX 64 static pthread_t mevent_tid; +static pthread_once_t mevent_once = PTHREAD_ONCE_INIT; static int mevent_timid = 43; static int mevent_pipefd[2]; +static int mfd; static pthread_mutex_t mevent_lmutex = PTHREAD_MUTEX_INITIALIZER; struct mevent { @@ -124,6 +126,26 @@ mevent_notify(void) } } +static void +mevent_init(void) +{ +#ifndef WITHOUT_CAPSICUM + cap_rights_t rights; +#endif + + mfd = kqueue(); + assert(mfd > 0); + +#ifndef WITHOUT_CAPSICUM + cap_rights_init(&rights, CAP_KQUEUE); + if (caph_rights_limit(mfd, &rights) == -1) + errx(EX_OSERR, "Unable to apply rights for sandbox"); +#endif + + LIST_INIT(&change_head); + LIST_INIT(&global_head); +} + static int mevent_kq_filter(struct mevent *mevp) { @@ -159,8 +181,24 @@ mevent_kq_fflags(struct mevent *mevp) return (0); } +static void +mevent_populate(struct mevent *mevp, struct kevent *kev) +{ + if (mevp->me_type == EVF_TIMER) { + kev->ident = mevp->me_timid; + kev->data = mevp->me_msecs; + } else { + kev->ident = mevp->me_fd; + kev->data = 0; + } + kev->filter = mevent_kq_filter(mevp); + kev->flags = mevent_kq_flags(mevp); + kev->fflags = mevent_kq_fflags(mevp); + kev->udata = mevp; +} + static int -mevent_build(int mfd, struct kevent *kev) +mevent_build(struct kevent *kev) { struct mevent *mevp, *tmpp; int i; @@ -177,17 +215,8 @@ mevent_build(int mfd, struct kevent *kev) */ close(mevp->me_fd); } else { - if (mevp->me_type == EVF_TIMER) { - kev[i].ident = mevp->me_timid; - kev[i].data = mevp->me_msecs; - } else { - kev[i].ident = mevp->me_fd; - kev[i].data = 0; - } - kev[i].filter = mevent_kq_filter(mevp); - kev[i].flags = mevent_kq_flags(mevp); - kev[i].fflags = mevent_kq_fflags(mevp); - kev[i].udata = mevp; + assert((mevp->me_state & EV_ADD) == 0); + mevent_populate(mevp, &kev[i]); i++; } @@ -197,12 +226,6 @@ mevent_build(int mfd, struct kevent *kev) if (mevp->me_state & EV_DELETE) { free(mevp); } else { - /* - * We need to add the event only once, so we can - * reset the EV_ADD bit after it has been propagated - * to the kevent() arguments the first time. - */ - mevp->me_state &= ~EV_ADD; LIST_INSERT_HEAD(&global_head, mevp, me_list); } @@ -234,7 +257,9 @@ mevent_add_state(int tfd, enum ev_type type, void (*func)(int, enum ev_type, void *), void *param, int state) { + struct kevent kev; struct mevent *lp, *mevp; + int ret; if (tfd < 0 || func == NULL) { return (NULL); @@ -242,6 +267,8 @@ mevent_add_state(int tfd, enum ev_type type, mevp = NULL; + pthread_once(&mevent_once, mevent_init); + mevent_qlock(); /* @@ -262,7 +289,7 @@ mevent_add_state(int tfd, enum ev_type type, } /* - * Allocate an entry, populate it, and add it to the change list. + * Allocate an entry and populate it. */ mevp = calloc(1, sizeof(struct mevent)); if (mevp == NULL) { @@ -277,11 +304,22 @@ mevent_add_state(int tfd, enum ev_type type, mevp->me_type = type; mevp->me_func = func; mevp->me_param = param; - - LIST_INSERT_HEAD(&change_head, mevp, me_list); - mevp->me_cq = 1; mevp->me_state = state; - mevent_notify(); + + /* + * Try to add the event. If this fails, report the failure to + * the caller. + */ + mevent_populate(mevp, &kev); + ret = kevent(mfd, &kev, 1, NULL, 0, NULL); + if (ret == -1) { + free(mevp); + mevp = NULL; + goto exit; + } + + mevp->me_state &= ~EV_ADD; + LIST_INSERT_HEAD(&global_head, mevp, me_list); exit: mevent_qunlock(); @@ -415,7 +453,6 @@ mevent_dispatch(void) struct kevent changelist[MEVENT_MAX]; struct kevent eventlist[MEVENT_MAX]; struct mevent *pipev; - int mfd; int numev; int ret; #ifndef WITHOUT_CAPSICUM @@ -425,14 +462,7 @@ mevent_dispatch(void) mevent_tid = pthread_self(); mevent_set_name(); - mfd = kqueue(); - assert(mfd > 0); - -#ifndef WITHOUT_CAPSICUM - cap_rights_init(&rights, CAP_KQUEUE); - if (caph_rights_limit(mfd, &rights) == -1) - errx(EX_OSERR, "Unable to apply rights for sandbox"); -#endif + pthread_once(&mevent_once, mevent_init); /* * Open the pipe that will be used for other threads to force @@ -466,7 +496,7 @@ mevent_dispatch(void) * to eliminate the extra syscall. Currently better for * debug. */ - numev = mevent_build(mfd, changelist); + numev = mevent_build(changelist); if (numev) { ret = kevent(mfd, changelist, numev, NULL, 0, NULL); if (ret == -1) {