From owner-svn-src-projects@freebsd.org Sun May 17 20:37:18 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ACE522F41EF for ; Sun, 17 May 2020 20:37:18 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49QDTn4x7Xz3RMk; Sun, 17 May 2020 20:37:17 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 99C071650B; Sun, 17 May 2020 20:37:17 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04HKbHRZ087262; Sun, 17 May 2020 20:37:17 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04HKbF2p087253; Sun, 17 May 2020 20:37:15 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005172037.04HKbF2p087253@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sun, 17 May 2020 20:37:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361141 - in projects/nfs-over-tls: sys/rpc sys/rpc/rpcsec_tls usr.sbin/rpctlscd usr.sbin/rpctlssd X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls: sys/rpc sys/rpc/rpcsec_tls usr.sbin/rpctlscd usr.sbin/rpctlssd X-SVN-Commit-Revision: 361141 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2020 20:37:18 -0000 Author: rmacklem Date: Sun May 17 20:37:15 2020 New Revision: 361141 URL: https://svnweb.freebsd.org/changeset/base/361141 Log: Separate errors doing the rpctlscd/rpctlssd upcalls from RPC errors. Without this patch, the RPC layer errors were being overloaded and used to indicate failures in the upcalls in the daemons, usually RPC_FAILED. This patch separates the error returns from the upcalls into a separate returned value called "reterr". This cleans up the code, but does not really change any semantics. Modified: projects/nfs-over-tls/sys/rpc/clnt_rc.c projects/nfs-over-tls/sys/rpc/clnt_vc.c projects/nfs-over-tls/sys/rpc/rpcsec_tls.h projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlscd.x projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlssd.x projects/nfs-over-tls/sys/rpc/svc_vc.c projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Modified: projects/nfs-over-tls/sys/rpc/clnt_rc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/clnt_rc.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/clnt_rc.c Sun May 17 20:37:15 2020 (r361141) @@ -132,6 +132,7 @@ clnt_reconnect_connect(CLIENT *cl) struct ucred *oldcred; CLIENT *newclient = NULL; uint64_t ssl[3]; + uint32_t reterr; mtx_lock(&rc->rc_lock); while (rc->rc_connecting) { @@ -198,9 +199,11 @@ clnt_reconnect_connect(CLIENT *cl) rc->rc_sendsz, rc->rc_recvsz, rc->rc_intr); if (rc->rc_tls && newclient != NULL) { printf("at rpctls_connect\n"); - stat = rpctls_connect(newclient, so, ssl); + stat = rpctls_connect(newclient, so, ssl, &reterr); printf("aft rpctls_connect=%d ssl=%jd\n", stat, (uintmax_t)ssl[2]); - if (stat != RPC_SUCCESS) { + if (stat != RPC_SUCCESS || reterr != RPCTLSERR_OK) { + if (stat == RPC_SUCCESS) + stat = RPC_FAILED; stat = rpc_createerr.cf_stat = stat; rpc_createerr.cf_error.re_errno = 0; CLNT_CLOSE(newclient); @@ -282,6 +285,7 @@ clnt_reconnect_call( if (!rc->rc_client) { mtx_unlock(&rc->rc_lock); stat = clnt_reconnect_connect(cl); +printf("reconnect_connect=%d\n", stat); if (stat == RPC_SYSTEMERROR) { error = tsleep(&fake_wchan, rc->rc_intr ? PCATCH : 0, "rpccon", hz); @@ -307,6 +311,7 @@ clnt_reconnect_call( mtx_unlock(&rc->rc_lock); stat = CLNT_CALL_MBUF(client, ext, proc, args, resultsp, utimeout); +if (stat != RPC_SUCCESS) printf("clnt_reconnect_call=%d\n", stat); if (stat != RPC_SUCCESS) { if (!ext) Modified: projects/nfs-over-tls/sys/rpc/clnt_vc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/clnt_vc.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/clnt_vc.c Sun May 17 20:37:15 2020 (r361141) @@ -875,6 +875,7 @@ clnt_vc_destroy(CLIENT *cl) struct socket *so = NULL; SVCXPRT *xprt; enum clnt_stat stat; + uint32_t reterr; clnt_vc_close(cl); @@ -900,10 +901,12 @@ clnt_vc_destroy(CLIENT *cl) mtx_destroy(&ct->ct_lock); if (so) { stat = RPC_FAILED; + reterr = RPCTLSERR_OK; if (ct->ct_sslrefno != 0) stat = rpctls_cl_disconnect(ct->ct_sslsec, - ct->ct_sslusec, ct->ct_sslrefno); - if (stat != RPC_SUCCESS) { + ct->ct_sslusec, ct->ct_sslrefno, + &reterr); + if (stat != RPC_SUCCESS || reterr == RPCTLSERR_NOCLOSE) { soshutdown(so, SHUT_WR); soclose(so); } @@ -1283,15 +1286,16 @@ static void clnt_vc_dotlsupcall(struct ct_data *ct) { enum clnt_stat ret; + uint32_t reterr; mtx_assert(&ct->ct_lock, MA_OWNED); if (ct->ct_rcvstate == UPCALLNEEDED) { ct->ct_rcvstate = UPCALLINPROG; mtx_unlock(&ct->ct_lock); ret = rpctls_cl_handlerecord(ct->ct_sslsec, ct->ct_sslusec, - ct->ct_sslrefno); + ct->ct_sslrefno, &reterr); mtx_lock(&ct->ct_lock); - if (ret == RPC_SUCCESS) + if (ret == RPC_SUCCESS && reterr == RPCTLSERR_OK) ct->ct_rcvstate = RCVNORMAL; else ct->ct_rcvstate = RCVNONAPPDATA; Modified: projects/nfs-over-tls/sys/rpc/rpcsec_tls.h ============================================================================== --- projects/nfs-over-tls/sys/rpc/rpcsec_tls.h Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/rpcsec_tls.h Sun May 17 20:37:15 2020 (r361141) @@ -43,17 +43,24 @@ #define RPCTLS_FLAGS_DISABLED 0x10 #define RPCTLS_FLAGS_CERTUSER 0x20 +/* Error return values for upcall rpcs. */ +#define RPCTLSERR_OK 0 +#define RPCTLSERR_NOCLOSE 1 +#define RPCTLSERR_NOSSL 2 +#define RPCTLSERR_NOSOCKET 3 + #ifdef _KERNEL /* Functions that perform upcalls to the rpctlsd daemon. */ enum clnt_stat rpctls_connect(CLIENT *newclient, struct socket *so, - uint64_t *sslp); + uint64_t *sslp, uint32_t *reterr); enum clnt_stat rpctls_cl_handlerecord(uint64_t sec, uint64_t usec, - uint64_t ssl); + uint64_t ssl, uint32_t *reterr); enum clnt_stat rpctls_srv_handlerecord(uint64_t sec, uint64_t usec, - uint64_t ssl); -enum clnt_stat rpctls_cl_disconnect(uint64_t sec, uint64_t usec, uint64_t ssl); + uint64_t ssl, uint32_t *reterr); +enum clnt_stat rpctls_cl_disconnect(uint64_t sec, uint64_t usec, + uint64_t ssl, uint32_t *reterr); enum clnt_stat rpctls_srv_disconnect(uint64_t sec, uint64_t usec, - uint64_t ssl); + uint64_t ssl, uint32_t *reterr); /* Initialization function for rpcsec_tls. */ int rpctls_init(void); Modified: projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c Sun May 17 20:37:15 2020 (r361141) @@ -317,7 +317,8 @@ rpctls_server_client(void) /* Do an upcall for a new socket connect using TLS. */ enum clnt_stat -rpctls_connect(CLIENT *newclient, struct socket *so, uint64_t *sslp) +rpctls_connect(CLIENT *newclient, struct socket *so, uint64_t *sslp, + uint32_t *reterr) { struct rpctlscd_connect_res res; struct rpc_callextra ext; @@ -366,9 +367,12 @@ printf("rpctls_conect so=%p\n", so); stat = rpctlscd_connect_1(NULL, &res, cl); printf("aft connect upcall=%d\n", stat); if (stat == RPC_SUCCESS) { - *sslp++ = res.sec; - *sslp++ = res.usec; - *sslp = res.ssl; + *reterr = res.reterr; + if (res.reterr == 0) { + *sslp++ = res.sec; + *sslp++ = res.usec; + *sslp = res.ssl; + } } CLNT_RELEASE(cl); @@ -389,9 +393,11 @@ printf("aft wakeup\n"); /* Do an upcall to handle an non-application data record using TLS. */ enum clnt_stat -rpctls_cl_handlerecord(uint64_t sec, uint64_t usec, uint64_t ssl) +rpctls_cl_handlerecord(uint64_t sec, uint64_t usec, uint64_t ssl, + uint32_t *reterr) { struct rpctlscd_handlerecord_arg arg; + struct rpctlscd_handlerecord_res res; enum clnt_stat stat; CLIENT *cl; @@ -399,22 +405,26 @@ printf("In rpctls_cl_handlerecord\n"); cl = rpctls_connect_client(); printf("handlerecord_client=%p\n", cl); if (cl == NULL) - return (RPC_FAILED); + return (RPC_AUTHERROR); /* Do the handlerecord upcall. */ arg.sec = sec; arg.usec = usec; arg.ssl = ssl; - stat = rpctlscd_handlerecord_1(&arg, NULL, cl); + stat = rpctlscd_handlerecord_1(&arg, &res, cl); printf("aft handlerecord upcall=%d\n", stat); CLNT_RELEASE(cl); + if (stat == RPC_SUCCESS) + *reterr = res.reterr; return (stat); } enum clnt_stat -rpctls_srv_handlerecord(uint64_t sec, uint64_t usec, uint64_t ssl) +rpctls_srv_handlerecord(uint64_t sec, uint64_t usec, uint64_t ssl, + uint32_t *reterr) { struct rpctlssd_handlerecord_arg arg; + struct rpctlssd_handlerecord_res res; enum clnt_stat stat; CLIENT *cl; @@ -422,23 +432,27 @@ printf("In rpctls_srv_handlerecord\n"); cl = rpctls_server_client(); printf("srv handlerecord_client=%p\n", cl); if (cl == NULL) - return (RPC_FAILED); + return (RPC_AUTHERROR); /* Do the handlerecord upcall. */ arg.sec = sec; arg.usec = usec; arg.ssl = ssl; - stat = rpctlssd_handlerecord_1(&arg, NULL, cl); + stat = rpctlssd_handlerecord_1(&arg, &res, cl); printf("aft srv handlerecord upcall=%d\n", stat); CLNT_RELEASE(cl); + if (stat == RPC_SUCCESS) + *reterr = res.reterr; return (stat); } /* Do an upcall to shut down a socket using TLS. */ enum clnt_stat -rpctls_cl_disconnect(uint64_t sec, uint64_t usec, uint64_t ssl) +rpctls_cl_disconnect(uint64_t sec, uint64_t usec, uint64_t ssl, + uint32_t *reterr) { struct rpctlscd_disconnect_arg arg; + struct rpctlscd_disconnect_res res; enum clnt_stat stat; CLIENT *cl; @@ -446,22 +460,26 @@ printf("In rpctls_cl_disconnect\n"); cl = rpctls_connect_client(); printf("disconnect_client=%p\n", cl); if (cl == NULL) - return (RPC_FAILED); + return (RPC_AUTHERROR); /* Do the disconnect upcall. */ arg.sec = sec; arg.usec = usec; arg.ssl = ssl; - stat = rpctlscd_disconnect_1(&arg, NULL, cl); + stat = rpctlscd_disconnect_1(&arg, &res, cl); printf("aft disconnect upcall=%d\n", stat); CLNT_RELEASE(cl); + if (stat == RPC_SUCCESS) + *reterr = res.reterr; return (stat); } enum clnt_stat -rpctls_srv_disconnect(uint64_t sec, uint64_t usec, uint64_t ssl) +rpctls_srv_disconnect(uint64_t sec, uint64_t usec, uint64_t ssl, + uint32_t *reterr) { struct rpctlssd_disconnect_arg arg; + struct rpctlssd_disconnect_res res; enum clnt_stat stat; CLIENT *cl; @@ -469,15 +487,17 @@ printf("In rpctls_srv_disconnect\n"); cl = rpctls_server_client(); printf("srv disconnect_client=%p\n", cl); if (cl == NULL) - return (RPC_FAILED); + return (RPC_AUTHERROR); /* Do the disconnect upcall. */ arg.sec = sec; arg.usec = usec; arg.ssl = ssl; - stat = rpctlssd_disconnect_1(&arg, NULL, cl); + stat = rpctlssd_disconnect_1(&arg, &res, cl); printf("aft srv disconnect upcall=%d\n", stat); CLNT_RELEASE(cl); + if (stat == RPC_SUCCESS) + *reterr = res.reterr; return (stat); } Modified: projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlscd.x ============================================================================== --- projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlscd.x Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlscd.x Sun May 17 20:37:15 2020 (r361141) @@ -30,6 +30,7 @@ /* $FreeBSD:$ */ struct rpctlscd_connect_res { + uint32_t reterr; uint64_t sec; uint64_t usec; uint64_t ssl; @@ -41,12 +42,20 @@ struct rpctlscd_handlerecord_arg { uint64_t ssl; }; +struct rpctlscd_handlerecord_res { + uint32_t reterr; +}; + struct rpctlscd_disconnect_arg { uint64_t sec; uint64_t usec; uint64_t ssl; }; +struct rpctlscd_disconnect_res { + uint32_t reterr; +}; + program RPCTLSCD { version RPCTLSCDVERS { void RPCTLSCD_NULL(void) = 0; @@ -54,8 +63,10 @@ program RPCTLSCD { rpctlscd_connect_res RPCTLSCD_CONNECT(void) = 1; - void RPCTLSCD_HANDLERECORD(rpctlscd_handlerecord_arg) = 2; + rpctlscd_handlerecord_res + RPCTLSCD_HANDLERECORD(rpctlscd_handlerecord_arg) = 2; - void RPCTLSCD_DISCONNECT(rpctlscd_disconnect_arg) = 3; + rpctlscd_disconnect_res + RPCTLSCD_DISCONNECT(rpctlscd_disconnect_arg) = 3; } = 1; } = 0x40677374; Modified: projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlssd.x ============================================================================== --- projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlssd.x Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctlssd.x Sun May 17 20:37:15 2020 (r361141) @@ -44,12 +44,20 @@ struct rpctlssd_handlerecord_arg { uint64_t ssl; }; +struct rpctlssd_handlerecord_res { + uint32_t reterr; +}; + struct rpctlssd_disconnect_arg { uint64_t sec; uint64_t usec; uint64_t ssl; }; +struct rpctlssd_disconnect_res { + uint32_t reterr; +}; + program RPCTLSSD { version RPCTLSSDVERS { void RPCTLSSD_NULL(void) = 0; @@ -57,8 +65,10 @@ program RPCTLSSD { rpctlssd_connect_res RPCTLSSD_CONNECT(void) = 1; - void RPCTLSSD_HANDLERECORD(rpctlssd_handlerecord_arg) = 2; + rpctlssd_handlerecord_res + RPCTLSSD_HANDLERECORD(rpctlssd_handlerecord_arg) = 2; - void RPCTLSSD_DISCONNECT(rpctlssd_disconnect_arg) = 3; + rpctlssd_disconnect_res + RPCTLSSD_DISCONNECT(rpctlssd_disconnect_arg) = 3; } = 1; } = 0x40677375; Modified: projects/nfs-over-tls/sys/rpc/svc_vc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/svc_vc.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/sys/rpc/svc_vc.c Sun May 17 20:37:15 2020 (r361141) @@ -452,13 +452,15 @@ static void svc_vc_destroy_common(SVCXPRT *xprt) { enum clnt_stat stat; + uint32_t reterr; if (xprt->xp_socket) { stat = RPC_FAILED; if ((xprt->xp_tls & RPCTLS_FLAGS_HANDSHAKE) != 0) stat = rpctls_srv_disconnect(xprt->xp_sslsec, - xprt->xp_sslusec, xprt->xp_sslrefno); - if (stat != RPC_SUCCESS) + xprt->xp_sslusec, xprt->xp_sslrefno, + &reterr); + if (stat != RPC_SUCCESS || reterr == RPCTLSERR_NOCLOSE) (void)soclose(xprt->xp_socket); } @@ -671,7 +673,7 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_msg *msg, struct socket* so = xprt->xp_socket; XDR xdrs; int error, rcvflag; - uint32_t xid_plus_direction[2]; + uint32_t reterr, xid_plus_direction[2]; struct cmsghdr *cmsg; struct tls_get_record tgr; enum clnt_stat ret; @@ -801,10 +803,11 @@ tryagain: sx_xunlock(&xprt->xp_lock); printf("Call rpctls_srv_handlerecord\n"); ret = rpctls_srv_handlerecord(xprt->xp_sslsec, - xprt->xp_sslusec, xprt->xp_sslrefno); + xprt->xp_sslusec, xprt->xp_sslrefno, + &reterr); sx_xlock(&xprt->xp_lock); xprt->xp_dontrcv = FALSE; - if (ret != RPC_SUCCESS) { + if (ret != RPC_SUCCESS || reterr != RPCTLSERR_OK) { /* * All we can do is soreceive() it and * then toss it. Modified: projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Sun May 17 20:37:15 2020 (r361141) @@ -79,7 +79,6 @@ __FBSDID("$FreeBSD$"); static struct pidfh *rpctls_pfh = NULL; static int rpctls_debug_level; static bool rpctls_verbose; -static int testnossl; static SSL_CTX *rpctls_ctx = NULL; static const char *rpctls_verify_cafile = NULL; static const char *rpctls_verify_capath = NULL; @@ -153,9 +152,8 @@ main(int argc, char **argv) rpctls_ssl_usec = tm.tv_usec; rpctls_verbose = false; - testnossl = 0; cert = false; - while ((ch = getopt(argc, argv, "D:dl:mp:r:tv")) != -1) { + while ((ch = getopt(argc, argv, "D:dl:mp:r:v")) != -1) { switch (ch) { case 'D': rpctls_certdir = optarg; @@ -175,9 +173,6 @@ main(int argc, char **argv) case 'r': rpctls_crlfile = optarg; break; - case 't': - testnossl = 1; - break; case 'v': rpctls_verbose = true; break; @@ -325,15 +320,19 @@ rpctlscd_connect_1_svc(void *argp, /* Get the socket fd from the kernel. */ s = gssd_syscall("C"); rpctlscd_verbose_out("rpctlsd_connect s=%d\n", s); - if (s < 0) - return (FALSE); + if (s < 0) { + result->reterr = RPCTLSERR_NOSOCKET; + return (TRUE); + } /* Do a TLS connect handshake. */ ssl = rpctls_connect(rpctls_ctx, s); - if (ssl == NULL) + if (ssl == NULL) { rpctlscd_verbose_out("rpctlsd_connect: can't do TLS " "handshake\n"); - else { + result->reterr = RPCTLSERR_NOSSL; + } else { + result->reterr = RPCTLSERR_OK; result->sec = rpctls_ssl_sec; result->usec = rpctls_ssl_usec; result->ssl = ++rpctls_ssl_refno; @@ -341,15 +340,6 @@ rpctlscd_verbose_out("rpctlsd_connect s=%d\n", s); if (rpctls_ssl_refno == 0) result->ssl = ++rpctls_ssl_refno; } - if (testnossl != 0 && ssl != NULL) { - /* Read the 478 bytes of junk off the socket. */ - siz = 478; - ret = 1; - while (siz > 0 && ret > 0) { - ret = recv(s, &buf[478 - siz], siz, 0); - siz -= ret; - } - } if (ssl == NULL) { /* @@ -358,7 +348,7 @@ rpctlscd_verbose_out("rpctlsd_connect s=%d\n", s); */ shutdown(s, SHUT_WR); close(s); - return (FALSE); + return (TRUE); } /* Maintain list of all current SSL *'s */ @@ -372,7 +362,7 @@ rpctlscd_verbose_out("rpctlsd_connect s=%d\n", s); bool_t rpctlscd_handlerecord_1_svc(struct rpctlscd_handlerecord_arg *argp, - void *result, struct svc_req *rqstp) + struct rpctlscd_handlerecord_res *result, struct svc_req *rqstp) { struct ssl_entry *slp; int ret; @@ -401,14 +391,15 @@ rpctlscd_handlerecord_1_svc(struct rpctlscd_handlereco else fprintf(stderr, "SSL_read returned %d\n", ret); } + result->reterr = RPCTLSERR_OK; } else - return (FALSE); + result->reterr = RPCTLSERR_NOSSL; return (TRUE); } bool_t rpctlscd_disconnect_1_svc(struct rpctlscd_disconnect_arg *argp, - void *result, struct svc_req *rqstp) + struct rpctlscd_disconnect_res *result, struct svc_req *rqstp) { struct ssl_entry *slp; int ret; @@ -441,8 +432,9 @@ rpctlscd_verbose_out("get_shutdown=%d\n", ret); shutdown(slp->s, SHUT_WR); close(slp->s); free(slp); + result->reterr = RPCTLSERR_OK; } else - return (FALSE); + result->reterr = RPCTLSERR_NOCLOSE; return (TRUE); } Modified: projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Sun May 17 20:14:49 2020 (r361140) +++ projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Sun May 17 20:37:15 2020 (r361141) @@ -420,7 +420,7 @@ rpctlssd_verbose_out("rpctlsd_connect_svc s=%d\n", s); bool_t rpctlssd_handlerecord_1_svc(struct rpctlssd_handlerecord_arg *argp, - void *result, struct svc_req *rqstp) + struct rpctlssd_handlerecord_res *result, struct svc_req *rqstp) { struct ssl_entry *slp; int ret; @@ -456,14 +456,15 @@ rpctlssd_verbose_out("get_shutdown=%d\n", ret); else fprintf(stderr, "SSL_read returned %d\n", ret); } + result->reterr = RPCTLSERR_OK; } else - return (FALSE); + result->reterr = RPCTLSERR_NOSSL; return (TRUE); } bool_t rpctlssd_disconnect_1_svc(struct rpctlssd_disconnect_arg *argp, - void *result, struct svc_req *rqstp) + struct rpctlssd_disconnect_res *result, struct svc_req *rqstp) { struct ssl_entry *slp; @@ -488,8 +489,9 @@ rpctlssd_disconnect_1_svc(struct rpctlssd_disconnect_a shutdown(slp->s, SHUT_WR); close(slp->s); free(slp); + result->reterr = RPCTLSERR_OK; } else - return (FALSE); + result->reterr = RPCTLSERR_NOCLOSE; return (TRUE); } From owner-svn-src-projects@freebsd.org Sun May 17 22:05:27 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 018852F7958 for ; Sun, 17 May 2020 22:05:26 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49QGRV67TVz3Y8p; Sun, 17 May 2020 22:05:26 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CD3AC17700; Sun, 17 May 2020 22:05:26 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04HM5QA5042437; Sun, 17 May 2020 22:05:26 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04HM5Q4X042433; Sun, 17 May 2020 22:05:26 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005172205.04HM5Q4X042433@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sun, 17 May 2020 22:05:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361144 - in projects/nfs-over-tls/sys/rpc: . rpcsec_tls X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls/sys/rpc: . rpcsec_tls X-SVN-Commit-Revision: 361144 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2020 22:05:27 -0000 Author: rmacklem Date: Sun May 17 22:05:25 2020 New Revision: 361144 URL: https://svnweb.freebsd.org/changeset/base/361144 Log: Assorted error handling fixups plus a fix for client side handling of non-application data. The error handling changes more clearly separate upcall errors from RPC errors. The cases where the disconnect upcall fails now assumes that the socket has been closed by a crashed/restarted daemon and does not attempt to close the socket again. The ct_rcvstate variable becomes a set of flag bits, so a flag bit can be set to tell clnt_vc_dotlsupcall() to call clnt_vc_soupcall() when a socket upcall needed to just return due to a upcall to the rpctlscd daemon was in progress. Modified: projects/nfs-over-tls/sys/rpc/clnt_vc.c projects/nfs-over-tls/sys/rpc/krpc.h projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c projects/nfs-over-tls/sys/rpc/svc_vc.c Modified: projects/nfs-over-tls/sys/rpc/clnt_vc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/clnt_vc.c Sun May 17 21:54:59 2020 (r361143) +++ projects/nfs-over-tls/sys/rpc/clnt_vc.c Sun May 17 22:05:25 2020 (r361144) @@ -159,7 +159,7 @@ clnt_vc_create( ct->ct_closing = FALSE; ct->ct_closed = FALSE; ct->ct_upcallrefs = 0; - ct->ct_rcvstate = RCVNORMAL; + ct->ct_rcvstate = RPCRCVSTATE_NORMAL; if ((so->so_state & (SS_ISCONNECTED|SS_ISCONFIRMING)) == 0) { error = soconnect(so, raddr, curthread); @@ -278,6 +278,7 @@ clnt_vc_create( ct->ct_raw = NULL; ct->ct_record = NULL; ct->ct_record_resid = 0; + ct->ct_sslrefno = 0; TAILQ_INIT(&ct->ct_pending); return (cl); @@ -424,8 +425,8 @@ call_again: * Wait until any upcall is completed. */ clnt_vc_dotlsupcall(ct); - while (ct->ct_rcvstate != RCVNORMAL && - ct->ct_rcvstate != RCVNONAPPDATA) + while ((ct->ct_rcvstate & (RPCRCVSTATE_NORMAL | + RPCRCVSTATE_NONAPPDATA)) == 0) msleep(&ct->ct_rcvstate, &ct->ct_lock, 0, "rpcrcvst", hz); TAILQ_INSERT_TAIL(&ct->ct_pending, cr, cr_link); @@ -802,10 +803,13 @@ printf("backch tls=0x%x xprt=%p\n", xprt->xp_tls, xprt break; case CLSET_BLOCKRCV: - if (*(int *) info) - ct->ct_rcvstate = TLSHANDSHAKE; - else - ct->ct_rcvstate = RCVNORMAL; + if (*(int *) info) { + ct->ct_rcvstate &= ~RPCRCVSTATE_NORMAL; + ct->ct_rcvstate |= RPCRCVSTATE_TLSHANDSHAKE; + } else { + ct->ct_rcvstate &= ~RPCRCVSTATE_TLSHANDSHAKE; + ct->ct_rcvstate |= RPCRCVSTATE_NORMAL; + } break; default: @@ -900,13 +904,17 @@ clnt_vc_destroy(CLIENT *cl) mtx_destroy(&ct->ct_lock); if (so) { - stat = RPC_FAILED; - reterr = RPCTLSERR_OK; - if (ct->ct_sslrefno != 0) + if (ct->ct_sslrefno != 0) { + /* + * If the upcall fails, the socket has + * probably been closed via the rpctlscd + * daemon having crashed or been + * restarted. + */ stat = rpctls_cl_disconnect(ct->ct_sslsec, ct->ct_sslusec, ct->ct_sslrefno, &reterr); - if (stat != RPC_SUCCESS || reterr == RPCTLSERR_NOCLOSE) { + } else { soshutdown(so, SHUT_WR); soclose(so); } @@ -955,8 +963,12 @@ clnt_vc_soupcall(struct socket *so, void *arg, int wai * the socket via openssl library calls. */ mtx_lock(&ct->ct_lock); - if (ct->ct_rcvstate != RCVNORMAL && ct->ct_rcvstate != - RCVNONAPPDATA) { + if ((ct->ct_rcvstate & (RPCRCVSTATE_NORMAL | + RPCRCVSTATE_NONAPPDATA)) == 0) { + /* Mark that a socket upcall needs to be done. */ + if ((ct->ct_rcvstate & (RPCRCVSTATE_UPCALLNEEDED | + RPCRCVSTATE_UPCALLINPROG)) != 0) + ct->ct_rcvstate |= RPCRCVSTATE_SOUPCALLNEEDED; mtx_unlock(&ct->ct_lock); return (SU_OK); } @@ -982,7 +994,8 @@ clnt_vc_soupcall(struct socket *so, void *arg, int wai uio.uio_td = curthread; m2 = m = NULL; rcvflag = MSG_DONTWAIT | MSG_SOCALLBCK; - if (ct->ct_sslrefno != 0 && ct->ct_rcvstate == RCVNORMAL) { + if (ct->ct_sslrefno != 0 && (ct->ct_rcvstate & + RPCRCVSTATE_NORMAL) != 0) { rcvflag |= MSG_TLSAPPDATA; ctrlp = NULL; } else @@ -1021,7 +1034,7 @@ clnt_vc_soupcall(struct socket *so, void *arg, int wai if (ct->ct_sslrefno != 0 && error == ENXIO) { /* Disable reception, marking an upcall needed. */ mtx_lock(&ct->ct_lock); - ct->ct_rcvstate = UPCALLNEEDED; + ct->ct_rcvstate |= RPCRCVSTATE_UPCALLNEEDED; /* * If an upcall in needed, wake up all thread(s) * in clnt_vc_call() so that one of them can do it. @@ -1057,16 +1070,18 @@ if (m2->m_next != NULL) printf("EEK! list of controls\ memcpy(&tgr, CMSG_DATA(cmsg), sizeof(tgr)); /* * This should have been handled by - * setting ct_rcvstate == UPCALLNEEDED, - * but if not, all we can do is toss - * it away. + * setting RPCRCVSTATE_UPCALLNEEDED in + * ct_rcvstate but if not, all we can do + * is toss it away. */ if (tgr.tls_type != TLS_RLTYPE_APP) { printf("Got weird type=%d\n", tgr.tls_type); m_freem(m); m_free(m2); mtx_lock(&ct->ct_lock); - ct->ct_rcvstate = RCVNORMAL; + ct->ct_rcvstate &= + ~RPCRCVSTATE_NONAPPDATA; + ct->ct_rcvstate |= RPCRCVSTATE_NORMAL; mtx_unlock(&ct->ct_lock); continue; } @@ -1289,16 +1304,29 @@ clnt_vc_dotlsupcall(struct ct_data *ct) uint32_t reterr; mtx_assert(&ct->ct_lock, MA_OWNED); - if (ct->ct_rcvstate == UPCALLNEEDED) { - ct->ct_rcvstate = UPCALLINPROG; - mtx_unlock(&ct->ct_lock); - ret = rpctls_cl_handlerecord(ct->ct_sslsec, ct->ct_sslusec, - ct->ct_sslrefno, &reterr); - mtx_lock(&ct->ct_lock); - if (ret == RPC_SUCCESS && reterr == RPCTLSERR_OK) - ct->ct_rcvstate = RCVNORMAL; - else - ct->ct_rcvstate = RCVNONAPPDATA; - wakeup(&ct->ct_rcvstate); + while ((ct->ct_rcvstate & (RPCRCVSTATE_UPCALLNEEDED | + RPCRCVSTATE_SOUPCALLNEEDED)) != 0) { + if ((ct->ct_rcvstate & RPCRCVSTATE_UPCALLNEEDED) != 0) { + ct->ct_rcvstate &= ~RPCRCVSTATE_UPCALLNEEDED; + ct->ct_rcvstate |= RPCRCVSTATE_UPCALLINPROG; + mtx_unlock(&ct->ct_lock); + ret = rpctls_cl_handlerecord(ct->ct_sslsec, ct->ct_sslusec, + ct->ct_sslrefno, &reterr); + mtx_lock(&ct->ct_lock); + ct->ct_rcvstate &= ~RPCRCVSTATE_UPCALLINPROG; + if (ret == RPC_SUCCESS && reterr == RPCTLSERR_OK) + ct->ct_rcvstate |= RPCRCVSTATE_NORMAL; + else + ct->ct_rcvstate |= RPCRCVSTATE_NONAPPDATA; + wakeup(&ct->ct_rcvstate); + } + if ((ct->ct_rcvstate & RPCRCVSTATE_SOUPCALLNEEDED) != 0) { + ct->ct_rcvstate &= ~RPCRCVSTATE_SOUPCALLNEEDED; + mtx_unlock(&ct->ct_lock); + SOCKBUF_LOCK(&ct->ct_socket->so_rcv); + clnt_vc_soupcall(ct->ct_socket, ct, M_NOWAIT); + SOCKBUF_UNLOCK(&ct->ct_socket->so_rcv); + mtx_lock(&ct->ct_lock); + } } } Modified: projects/nfs-over-tls/sys/rpc/krpc.h ============================================================================== --- projects/nfs-over-tls/sys/rpc/krpc.h Sun May 17 21:54:59 2020 (r361143) +++ projects/nfs-over-tls/sys/rpc/krpc.h Sun May 17 22:05:25 2020 (r361144) @@ -82,13 +82,13 @@ struct rc_data { bool rc_tls; /* Enable TLS on connection */ }; -enum clnt_rcvstate { - RCVNORMAL = 0, /* Normal reception. */ - RCVNONAPPDATA = 1, /* Reception of a non-application record. */ - TLSHANDSHAKE = 2, /* Reception blocked for TLS handshake. */ - UPCALLNEEDED = 3, /* Upcall to rpctlscd needed. */ - UPCALLINPROG = 4 /* Upcall to rpctlscd in progress. */ -}; +/* Bits for ct_rcvstate. */ +#define RPCRCVSTATE_NORMAL 0x01 /* Normal reception. */ +#define RPCRCVSTATE_NONAPPDATA 0x02 /* Reception of a non-application record. */ +#define RPCRCVSTATE_TLSHANDSHAKE 0x04 /* Reception blocked for TLS handshake. */ +#define RPCRCVSTATE_UPCALLNEEDED 0x08 /* Upcall to rpctlscd needed. */ +#define RPCRCVSTATE_UPCALLINPROG 0x10 /* Upcall to rpctlscd in progress. */ +#define RPCRCVSTATE_SOUPCALLNEEDED 0x20 /* Socket upcall needed. */ struct ct_data { struct mtx ct_lock; @@ -114,7 +114,7 @@ struct ct_data { uint64_t ct_sslsec; /* RPC-over-TLS connection. */ uint64_t ct_sslusec; uint64_t ct_sslrefno; - enum clnt_rcvstate ct_rcvstate; /* Block receiving for TLS upcalls */ + uint32_t ct_rcvstate; /* Handle receiving for TLS upcalls */ struct mbuf *ct_raw; /* Raw mbufs recv'd */ }; Modified: projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c Sun May 17 21:54:59 2020 (r361143) +++ projects/nfs-over-tls/sys/rpc/rpcsec_tls/rpctls_impl.c Sun May 17 22:05:25 2020 (r361144) @@ -404,8 +404,10 @@ rpctls_cl_handlerecord(uint64_t sec, uint64_t usec, ui printf("In rpctls_cl_handlerecord\n"); cl = rpctls_connect_client(); printf("handlerecord_client=%p\n", cl); - if (cl == NULL) - return (RPC_AUTHERROR); + if (cl == NULL) { + *reterr = RPCTLSERR_NOSSL; + return (RPC_SUCCESS); + } /* Do the handlerecord upcall. */ arg.sec = sec; @@ -431,8 +433,10 @@ rpctls_srv_handlerecord(uint64_t sec, uint64_t usec, u printf("In rpctls_srv_handlerecord\n"); cl = rpctls_server_client(); printf("srv handlerecord_client=%p\n", cl); - if (cl == NULL) - return (RPC_AUTHERROR); + if (cl == NULL) { + *reterr = RPCTLSERR_NOSSL; + return (RPC_SUCCESS); + } /* Do the handlerecord upcall. */ arg.sec = sec; @@ -459,8 +463,10 @@ rpctls_cl_disconnect(uint64_t sec, uint64_t usec, uint printf("In rpctls_cl_disconnect\n"); cl = rpctls_connect_client(); printf("disconnect_client=%p\n", cl); - if (cl == NULL) - return (RPC_AUTHERROR); + if (cl == NULL) { + *reterr = RPCTLSERR_NOSSL; + return (RPC_SUCCESS); + } /* Do the disconnect upcall. */ arg.sec = sec; @@ -486,8 +492,10 @@ rpctls_srv_disconnect(uint64_t sec, uint64_t usec, uin printf("In rpctls_srv_disconnect\n"); cl = rpctls_server_client(); printf("srv disconnect_client=%p\n", cl); - if (cl == NULL) - return (RPC_AUTHERROR); + if (cl == NULL) { + *reterr = RPCTLSERR_NOSSL; + return (RPC_SUCCESS); + } /* Do the disconnect upcall. */ arg.sec = sec; Modified: projects/nfs-over-tls/sys/rpc/svc_vc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/svc_vc.c Sun May 17 21:54:59 2020 (r361143) +++ projects/nfs-over-tls/sys/rpc/svc_vc.c Sun May 17 22:05:25 2020 (r361144) @@ -455,12 +455,17 @@ svc_vc_destroy_common(SVCXPRT *xprt) uint32_t reterr; if (xprt->xp_socket) { - stat = RPC_FAILED; - if ((xprt->xp_tls & RPCTLS_FLAGS_HANDSHAKE) != 0) + if ((xprt->xp_tls & RPCTLS_FLAGS_HANDSHAKE) != 0) { + /* + * If the upcall fails, the socket has + * probably been closed via the rpctlssd + * daemon having crashed or been + * restarted. + */ stat = rpctls_srv_disconnect(xprt->xp_sslsec, xprt->xp_sslusec, xprt->xp_sslrefno, &reterr); - if (stat != RPC_SUCCESS || reterr == RPCTLSERR_NOCLOSE) + } else (void)soclose(xprt->xp_socket); } From owner-svn-src-projects@freebsd.org Tue May 19 01:32:16 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7BC8B2F3C9C for ; Tue, 19 May 2020 01:32:16 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Qyzh2khjz4gHB; Tue, 19 May 2020 01:32:16 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3F0B02ACAE; Tue, 19 May 2020 01:32:16 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04J1WGZ0061897; Tue, 19 May 2020 01:32:16 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04J1WFug061895; Tue, 19 May 2020 01:32:15 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005190132.04J1WFug061895@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Tue, 19 May 2020 01:32:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361233 - in projects/nfs-over-tls/usr.sbin: rpctlscd rpctlssd X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls/usr.sbin: rpctlscd rpctlssd X-SVN-Commit-Revision: 361233 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2020 01:32:16 -0000 Author: rmacklem Date: Tue May 19 01:32:15 2020 New Revision: 361233 URL: https://svnweb.freebsd.org/changeset/base/361233 Log: Fix the daemons so that they do not terminate upon SIGPIPE. There are cases where the other end has closed the TCP connection and then, if the daemon does an SSL_shutdown(), it gets a SIGPIPE. Fix the daemons so they just ignore SIGPIPE. (The code inheritted termination upon SIGPIPE from the gssd, where the only socket is the one used for doing upcalls.) While here, I tweaked the handling of SSL_shutdown() so that the calls are done only if the value returned via SSL_get_shutdown() indicates that a call is needed. It turns out that NFSv3 over TCP is useful for testing the non-application data records, since the server side krpc shuts down a connection that is idle for 6minutes. This never happens normally for NFSv4, but does happen for NFSv3 if you just leave the mount idle for 6minutes. This generates close alert messages and then a new connection. Modified: projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Modified: projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Tue May 19 01:06:31 2020 (r361232) +++ projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Tue May 19 01:32:15 2020 (r361233) @@ -220,7 +220,7 @@ main(int argc, char **argv) signal(SIGHUP, SIG_IGN); } signal(SIGTERM, rpctlscd_terminate); - signal(SIGPIPE, rpctlscd_terminate); + signal(SIGPIPE, SIG_IGN); signal(SIGHUP, rpctls_huphandler); pidfile_write(rpctls_pfh); @@ -385,7 +385,14 @@ rpctlscd_handlerecord_1_svc(struct rpctlscd_handlereco * handle the non-application data record before doing so. */ ret = SSL_read(slp->ssl, &junk, 0); - if (ret > 0) { + if (ret <= 0) { + /* Check to see if this was a close alert. */ + ret = SSL_get_shutdown(slp->ssl); +rpctlscd_verbose_out("get_shutdown2=%d\n", ret); + if ((ret & (SSL_SENT_SHUTDOWN | + SSL_RECEIVED_SHUTDOWN)) == SSL_RECEIVED_SHUTDOWN) + SSL_shutdown(slp->ssl); + } else { if (rpctls_debug_level == 0) syslog(LOG_ERR, "SSL_read returned %d", ret); else @@ -417,12 +424,13 @@ rpctlscd_disconnect_1_svc(struct rpctlscd_disconnect_a rpctlscd_verbose_out("rpctlscd_disconnect: fd=%d closed\n", slp->s); LIST_REMOVE(slp, next); - SSL_shutdown(slp->ssl); - /* Check to see if the peer has sent a close alert. */ ret = SSL_get_shutdown(slp->ssl); -rpctlscd_verbose_out("get_shutdown=%d\n", ret); - if ((ret & (SSL_SENT_SHUTDOWN | SSL_RECEIVED_SHUTDOWN)) == - SSL_SENT_SHUTDOWN) +rpctlscd_verbose_out("get_shutdown0=%d\n", ret); + /* + * Do an SSL_shutdown() unless a close alert has + * already been sent. + */ + if ((ret & SSL_SENT_SHUTDOWN) == 0) SSL_shutdown(slp->ssl); SSL_free(slp->ssl); /* Modified: projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Tue May 19 01:06:31 2020 (r361232) +++ projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Tue May 19 01:32:15 2020 (r361233) @@ -260,7 +260,7 @@ fprintf(stderr, "dnsname=%s\n", rpctls_dnsname); signal(SIGHUP, SIG_IGN); } signal(SIGTERM, rpctlssd_terminate); - signal(SIGPIPE, rpctlssd_terminate); + signal(SIGPIPE, SIG_IGN); signal(SIGHUP, rpctls_huphandler); pidfile_write(rpctls_pfh); @@ -467,6 +467,7 @@ rpctlssd_disconnect_1_svc(struct rpctlssd_disconnect_a struct rpctlssd_disconnect_res *result, struct svc_req *rqstp) { struct ssl_entry *slp; + int ret; slp = NULL; if (argp->sec == rpctls_ssl_sec && argp->usec == @@ -481,6 +482,14 @@ rpctlssd_disconnect_1_svc(struct rpctlssd_disconnect_a rpctlssd_verbose_out("rpctlssd_disconnect fd=%d closed\n", slp->s); LIST_REMOVE(slp, next); + ret = SSL_get_shutdown(slp->ssl); +rpctlssd_verbose_out("get_shutdown1=%d\n", ret); + /* + * Do an SSL_shutdown() unless a close alert has + * already been sent. + */ + if ((ret & SSL_SENT_SHUTDOWN) == 0) + SSL_shutdown(slp->ssl); SSL_free(slp->ssl); /* * For RPC-over-TLS, this upcall is expected From owner-svn-src-projects@freebsd.org Wed May 20 01:25:46 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C0A4D2FB3A0 for ; Wed, 20 May 2020 01:25:46 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49RZnk4gjHz4Tt8; Wed, 20 May 2020 01:25:46 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9B95C14054; Wed, 20 May 2020 01:25:46 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04K1PkPt055898; Wed, 20 May 2020 01:25:46 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04K1Pk9N055897; Wed, 20 May 2020 01:25:46 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005200125.04K1Pk9N055897@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Wed, 20 May 2020 01:25:46 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361270 - projects/nfs-over-tls/sys/rpc X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/nfs-over-tls/sys/rpc X-SVN-Commit-Revision: 361270 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2020 01:25:46 -0000 Author: rmacklem Date: Wed May 20 01:25:46 2020 New Revision: 361270 URL: https://svnweb.freebsd.org/changeset/base/361270 Log: Modify the client side non-application data record upcall so that it is done via a kthread. There was a glitch in the upcall handling without this patch, in that the upcall was done by a thread doing an RPC. The problem with this was that there might not be an RPC done for minutes, hours,... This patch changes the code so that a kthread runs for each client side TLS connection and does the upcalls. Modified: projects/nfs-over-tls/sys/rpc/clnt_vc.c projects/nfs-over-tls/sys/rpc/krpc.h Modified: projects/nfs-over-tls/sys/rpc/clnt_vc.c ============================================================================== --- projects/nfs-over-tls/sys/rpc/clnt_vc.c Tue May 19 22:09:59 2020 (r361269) +++ projects/nfs-over-tls/sys/rpc/clnt_vc.c Wed May 20 01:25:46 2020 (r361270) @@ -62,6 +62,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -101,7 +102,7 @@ static void clnt_vc_close(CLIENT *); static void clnt_vc_destroy(CLIENT *); static bool_t time_not_ok(struct timeval *); static int clnt_vc_soupcall(struct socket *so, void *arg, int waitflag); -static void clnt_vc_dotlsupcall(struct ct_data *ct); +static void clnt_vc_dotlsupcall(void *data); static struct clnt_ops clnt_vc_ops = { .cl_call = clnt_vc_call, @@ -418,13 +419,7 @@ call_again: goto out; } - /* - * For TLS, do an upcall, as required. - * clnt_vc_dotlsupcall() will just return unless an - * an upcall is needed. - * Wait until any upcall is completed. - */ - clnt_vc_dotlsupcall(ct); + /* For TLS, wait for an upcall to be done, as required. */ while ((ct->ct_rcvstate & (RPCRCVSTATE_NORMAL | RPCRCVSTATE_NONAPPDATA)) == 0) msleep(&ct->ct_rcvstate, &ct->ct_lock, 0, "rpcrcvst", hz); @@ -502,30 +497,8 @@ printf("TRY AGAIN!!\n"); goto out; } - /* - * For TLS, msleep() can be awakened to handle - * an upcall via a call to clnt_vc_dotlsupcall(). - * If there was no error, it needs to loop - * around and wait for the reply. - */ - do { - /* - * Call clnt_vc_dotlsupcall() both before - * and after the msleep(). If there is - * no upcall to do, clnt_vc_dotlsupcall() - * simply returns. - * Call before msleep() in case ct_rcvstate - * is already set to UPCALLNEEDED and the - * wakeup(ct) has already been done. - * Call after msleep() in case it has its - * reply and will not be looping. - */ - clnt_vc_dotlsupcall(ct); - error = msleep(cr, &ct->ct_lock, ct->ct_waitflag, - ct->ct_waitchan, tvtohz(&timeout)); - clnt_vc_dotlsupcall(ct); - } while (cr->cr_mrep == NULL && error == 0 && - cr->cr_error == 0); + error = msleep(cr, &ct->ct_lock, ct->ct_waitflag, ct->ct_waitchan, + tvtohz(&timeout)); TAILQ_REMOVE(&ct->ct_pending, cr, cr_link); @@ -673,6 +646,8 @@ clnt_vc_control(CLIENT *cl, u_int request, void *info) void *infop = info; SVCXPRT *xprt; uint64_t *p; + int error; + static u_int thrdnum = 0; mtx_lock(&ct->ct_lock); @@ -800,7 +775,13 @@ printf("backch tls=0x%x xprt=%p\n", xprt->xp_tls, xprt ct->ct_sslsec = *p++; ct->ct_sslusec = *p++; ct->ct_sslrefno = *p; - break; + mtx_unlock(&ct->ct_lock); + /* Start the kthread that handles upcalls. */ + error = kthread_add(clnt_vc_dotlsupcall, ct, + NULL, NULL, 0, 0, "krpctls%u", thrdnum++); + if (error != 0) + panic("Can't add KRPC thread error %d", error); + return (TRUE); case CLSET_BLOCKRCV: if (*(int *) info) { @@ -868,6 +849,7 @@ clnt_vc_close(CLIENT *cl) ct->ct_closing = FALSE; ct->ct_closed = TRUE; + wakeup(&ct->ct_sslrefno); mtx_unlock(&ct->ct_lock); wakeup(ct); } @@ -900,6 +882,10 @@ clnt_vc_destroy(CLIENT *cl) } } + /* Wait for the upcall kthread to terminate. */ + while ((ct->ct_rcvstate & RPCRCVSTATE_UPCALLTHREAD) != 0) + msleep(&ct->ct_sslrefno, &ct->ct_lock, 0, + "clntvccl", hz); mtx_unlock(&ct->ct_lock); mtx_destroy(&ct->ct_lock); @@ -1036,13 +1022,10 @@ clnt_vc_soupcall(struct socket *so, void *arg, int wai mtx_lock(&ct->ct_lock); ct->ct_rcvstate |= RPCRCVSTATE_UPCALLNEEDED; /* - * If an upcall in needed, wake up all thread(s) - * in clnt_vc_call() so that one of them can do it. - * Not efficient, but this should not happen - * frequently. + * If an upcall in needed, wake up the kthread + * that runs clnt_vc_dotlsupcall(). */ - TAILQ_FOREACH(cr, &ct->ct_pending, cr_link) - wakeup(cr); + wakeup(&ct->ct_sslrefno); mtx_unlock(&ct->ct_lock); printf("Mark upcallneeded\n"); break; @@ -1296,16 +1279,19 @@ clnt_vc_upcallsdone(struct ct_data *ct) /* * Do a TLS upcall to the rpctlscd daemon, as required. + * This function runs as a kthread. */ static void -clnt_vc_dotlsupcall(struct ct_data *ct) +clnt_vc_dotlsupcall(void *data) { + struct ct_data *ct = (struct ct_data *)data; enum clnt_stat ret; uint32_t reterr; - mtx_assert(&ct->ct_lock, MA_OWNED); - while ((ct->ct_rcvstate & (RPCRCVSTATE_UPCALLNEEDED | - RPCRCVSTATE_SOUPCALLNEEDED)) != 0) { +printf("TLSupcall started\n"); + mtx_lock(&ct->ct_lock); + ct->ct_rcvstate |= RPCRCVSTATE_UPCALLTHREAD; + while (!ct->ct_closed) { if ((ct->ct_rcvstate & RPCRCVSTATE_UPCALLNEEDED) != 0) { ct->ct_rcvstate &= ~RPCRCVSTATE_UPCALLNEEDED; ct->ct_rcvstate |= RPCRCVSTATE_UPCALLINPROG; @@ -1328,5 +1314,11 @@ clnt_vc_dotlsupcall(struct ct_data *ct) SOCKBUF_UNLOCK(&ct->ct_socket->so_rcv); mtx_lock(&ct->ct_lock); } + msleep(&ct->ct_sslrefno, &ct->ct_lock, 0, "clntvcdu", hz); } + ct->ct_rcvstate &= ~RPCRCVSTATE_UPCALLTHREAD; + wakeup(&ct->ct_sslrefno); + mtx_unlock(&ct->ct_lock); +printf("TLSupcall exit\n"); + kthread_exit(); } Modified: projects/nfs-over-tls/sys/rpc/krpc.h ============================================================================== --- projects/nfs-over-tls/sys/rpc/krpc.h Tue May 19 22:09:59 2020 (r361269) +++ projects/nfs-over-tls/sys/rpc/krpc.h Wed May 20 01:25:46 2020 (r361270) @@ -89,6 +89,7 @@ struct rc_data { #define RPCRCVSTATE_UPCALLNEEDED 0x08 /* Upcall to rpctlscd needed. */ #define RPCRCVSTATE_UPCALLINPROG 0x10 /* Upcall to rpctlscd in progress. */ #define RPCRCVSTATE_SOUPCALLNEEDED 0x20 /* Socket upcall needed. */ +#define RPCRCVSTATE_UPCALLTHREAD 0x40 /* Upcall kthread running. */ struct ct_data { struct mtx ct_lock; From owner-svn-src-projects@freebsd.org Wed May 20 04:59:54 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05C9F2FEAB5 for ; Wed, 20 May 2020 04:59:54 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49RgXm6P3Nz4g27; Wed, 20 May 2020 04:59:52 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from kduck.mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 04K4xmlo031313 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 20 May 2020 00:59:50 -0400 Date: Tue, 19 May 2020 21:59:47 -0700 From: Benjamin Kaduk To: Rick Macklem Cc: Benjamin Kaduk , John Baldwin , Rick Macklem , "src-committers@freebsd.org" , "svn-src-projects@freebsd.org" Subject: Re: svn commit: r360859 - projects/nfs-over-tls/sys/rpc Message-ID: <20200520045947.GX58497@kduck.mit.edu> References: <202005100017.04A0Hd7I058863@repo.freebsd.org> <6739df0b-e621-2ca5-8f92-821822733772@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Rspamd-Queue-Id: 49RgXm6P3Nz4g27 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of kaduk@mit.edu designates 18.9.28.11 as permitted sender) smtp.mailfrom=kaduk@mit.edu X-Spamd-Result: default: False [-1.87 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[24.16.140.251:received]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:18.9.28.0/24]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[mit.edu]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[18.9.28.11:from]; NEURAL_HAM_SHORT(-0.59)[-0.587]; NEURAL_HAM_MEDIUM(-0.78)[-0.780]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2020 04:59:54 -0000 On Tue, May 12, 2020 at 02:32:29PM +0000, Rick Macklem wrote: > Benjamin Kaduk wrote: > [stuff snipped] > >You can avoid having to play games with putting stuff back on the socket > >receive buffer by using a custom BIO implementation in userspace that knows > >how to inject the received message. > >Rick Macklem wrote: > >>Actually, what might work for the krpc code is a new MSG_TLSAPPDATA > >>flag for soreceive_generic(), which says "if the record is not application > >>data, return an error". (Sort of the opposite of what you said above, but > >>would perform the same thing.) > >>This could be used for the krpc soreceive() calls, so that the non-application > >>data record remains on the socket's receive buffer. > Well, I'd find it a lot easier to implement MSG_TLSAPPDATA, since I've been > looking at soreceive_generic() recently. > I'm guessing that a custom BIO would need to be written and the upstreamed > to openssl? It doesn't have to be upstreamed; the idea of the API is that it's modular and anyone can slot in their own implementation for their particular communication needs. -Ben From owner-svn-src-projects@freebsd.org Wed May 20 14:58:48 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B85962DBFD2 for ; Wed, 20 May 2020 14:58:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670048.outbound.protection.outlook.com [40.107.67.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Rwqq4hQ2z4Jtl; Wed, 20 May 2020 14:58:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f13U7VEvISL2NGDSX/iblA10ZBufrIB0NxuQUzsl70akZRX6hQj5ZAL+B0NcCbUth/TDx9WGiB84WWCdUi6O7yZwtCX3LCTlI7aRc8djRc2PioS3qHa4Fek7IIaSj4Pvv+GD79NbX6WZVoagz1wEhcKvi6OCaqC7t68/Y+OluWpLI7k7BmagPmT0ZAroaAhywEoN2ofg1aWM0hlyCDcECp7ATfYDchoA5nQdAT5Kh4l9aTGUD5VYjGZPPpDirZPq2SMDp8UhYLugn4YSEljJ6w55t9Ia8OZUeSAp0p72NcGYiyydpZKMYt+eTOfQmjiTWhu4gCZ1+6O81jGqZWNp3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1/XnNneCp9OV8MV3FPCkAvuR6U036wGW7e4ivG/WUYA=; b=lfLP9a0ajiPF4ZD5fLWtTlbLKDM0jidFgRiy9nELR5OzLKGk50dsoyDAQDE6rCzaO0PQoexLmtxV38LVZ1FMJVJcB1w48tKNG4WIWb0o7kT5wjbpsuAwi8BWmC7YFxyPm8lYwZj9ooTyuhCPVlEJvT/pTcdf8HWIagBJXtQXgZFh6Tk2k73WJAqjmr8oMgXdr1Kf/xZNZeHLnd6LMKFjUy5+rVujR5GSml1Bp20HGjpPs+ibcrT8QynqVFCwc92j0GSvyljROf9nSyggyTtYQC5yethq/3h3mQ8XeFQJdghfkbb6zbBzDVkqnwgwnx/HSUafHK5RbnFUgG7mwqpoHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1/XnNneCp9OV8MV3FPCkAvuR6U036wGW7e4ivG/WUYA=; b=WdNtczUQfZF9Fi1sjuQ628q2X1058QPtGrMS+BEoG6a8yt7Hvk4mKuccIAaZTLwqhzqNKWavyoZnVHq00Eb3+mFvYAp0BveLZ7Gs5Ysbmq+5N4MytMpahnV16ka8jbdd76H7LFqPgAyFDkws744vz6M+E75WZJy6/HiVLfNAEi68ipvtpn7ChZms9pjp/OB+UzKq+Ul/vf/pE2FuOqoJLEHin+lMXymYWZdcCwfOYgIad5yy/eP13tiSXm7lA16IZidS3gLfQcuYF3cDxOJqcTcWb7eO9ezT4C8yxUe87OCZuJuCYkkUd4xFVv54bniUXnr27X2tt/y/UGMaFJmXvQ== Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::26) by QB1PR01MB2564.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:34::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.26; Wed, 20 May 2020 14:58:46 +0000 Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2]) by QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2%6]) with mapi id 15.20.3000.034; Wed, 20 May 2020 14:58:37 +0000 From: Rick Macklem To: Benjamin Kaduk CC: Benjamin Kaduk , John Baldwin , Rick Macklem , "src-committers@freebsd.org" , "svn-src-projects@freebsd.org" Subject: Re: svn commit: r360859 - projects/nfs-over-tls/sys/rpc Thread-Topic: svn commit: r360859 - projects/nfs-over-tls/sys/rpc Thread-Index: AQHWJ9HZ1BQ89BXo/ki0EXCO1Y+2cKijtTOsgAAIx/+AABDwDYAAF9MAgACaLemAC/ZqgIAApYC/ Date: Wed, 20 May 2020 14:58:37 +0000 Message-ID: References: <202005100017.04A0Hd7I058863@repo.freebsd.org> <6739df0b-e621-2ca5-8f92-821822733772@FreeBSD.org> , <20200520045947.GX58497@kduck.mit.edu> In-Reply-To: <20200520045947.GX58497@kduck.mit.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a43e9b6e-0cea-4240-3fba-08d7fcce46e8 x-ms-traffictypediagnostic: QB1PR01MB2564: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 04097B7F7F x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ywwta+vwK/INOs0NZhuuvnQCYMI7Fr4H2SmuVk3yw/D8WiNXnDLmKSfITRKsUuTa6Sc/MiPdlZ+XugqGl+kszNrD1bYYPHzGBvl4cPNbzeWfazCpqG37MS7FzC6b+eCJ+GPrJYpSHMrm2qvGcbLPDoBLEh39QYEBObE4TEQiq4UtuE4aHlui/895pMPtbYmI4CgZ6q7jb2dnrylCQcbmb8Y5jqjEvW5TYJw6WZe8xOUkalxMNw70jy0lpqx5X2rtwPsKM+UDgJEniP/dT/KBN/XaMO1DcaMyccvwg2FnItMGr+97QGu5hw/ufSRp9YONhoTU5Jqb8bKsj9DB2qMVX/cKGWt1naCJYt49yfdNpwofEPS9DGJVLIQKudxtD8ZFe91gzS+gh0pUvnNfhn+AvTnF0QvwqYV+TuQQfOcDrvkmERh7LF0ghoI/2Ix8H/9j x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(346002)(366004)(376002)(136003)(396003)(39860400002)(4326008)(2906002)(55016002)(9686003)(71200400001)(86362001)(33656002)(52536014)(7696005)(8936002)(66446008)(8676002)(76116006)(66946007)(6916009)(66476007)(54906003)(64756008)(66556008)(786003)(186003)(6506007)(316002)(478600001)(5660300002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: BI7EQoSo/lAtSnq4bEHgyyNVORCXqvuhAlMl2O1o5jUJqq/DDxSKkHXAdbyRpCF//1DAJ5ZdolHX0LmeKUmOlgZO9eaaEcZZDKrtHJUOT4e1KY57ApVIT21XYm/N8Bxk2+Zng46yJE1TuGDMSfQGB0PA/332kGxVHV5i0LAsJ2OJ2IYC5V6tvgteuv/naciqQFwtK6c3H1GKFBrNcLU5EoxM1yOWyo536lAwfzmapuiH3Vx8CfICVRwnaf1lcOpjP5FY/rkYVbYtFPd4KxHpzlrBBVubSnL3qhwItPAhZzdg/lSy+5TXVcfn/rq835fnLB56e5wNYIAdO2ZvLg3VsWSwQ7bGHLxisU74qCnG+Hlz6K80YAupaGbEttW26y4JyTCETWr3+SoN48pgaJ4WXYtbVhZQdr8uW9Y6QuUtR7Ib+l+b7sobYUpAVxznb+P7tm8DP6OOndJ1Papnnw+8evZhLOKbvpBH8TTZ/tN91iJXULt0OnIdnKBjqzyYrKCOCayiHXRsPIZMamPR9rb7xyZNJAPIxbCYCtze/oPqynw= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: a43e9b6e-0cea-4240-3fba-08d7fcce46e8 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 May 2020 14:58:37.8473 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mU3lNH/lb9nAOgNW7A7w6UfNmkm3BjFODfVPiSF73fFYMdSbToTzWPNSfTvEeQDTtV/+4B+HwttOmEefLg7u2A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2564 X-Rspamd-Queue-Id: 49Rwqq4hQ2z4Jtl X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=WdNtczUQ; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.48 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.96 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.004]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.48:from]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.04)[-1.045]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RCPT_COUNT_FIVE(0.00)[6]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-1.31)[-1.315]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; FREEMAIL_CC(0.00)[gmail.com,FreeBSD.org,freebsd.org]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.48:from] X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2020 14:58:48 -0000 Benjamin Kaduk wrote:=0A= >On Tue, May 12, 2020 at 02:32:29PM +0000, Rick Macklem wrote:=0A= >> Benjamin Kaduk wrote:=0A= >> [stuff snipped]=0A= >> >You can avoid having to play games with putting stuff back on the socke= t=0A= >> >receive buffer by using a custom BIO implementation in userspace that k= nows=0A= >> >how to inject the received message.=0A= >> >Rick Macklem wrote:=0A= >> >>Actually, what might work for the krpc code is a new MSG_TLSAPPDATA=0A= >> >>flag for soreceive_generic(), which says "if the record is not applica= tion=0A= >> >>data, return an error". (Sort of the opposite of what you said above, = but=0A= >> >>would perform the same thing.)=0A= >> >>This could be used for the krpc soreceive() calls, so that the non-app= lication=0A= >> >>data record remains on the socket's receive buffer.=0A= >> Well, I'd find it a lot easier to implement MSG_TLSAPPDATA, since I've b= een=0A= >> looking at soreceive_generic() recently.=0A= >> I'm guessing that a custom BIO would need to be written and the upstream= ed=0A= >> to openssl?=0A= >=0A= >It doesn't have to be upstreamed; the idea of the API is that it's modular= =0A= >and anyone can slot in their own implementation for their particular=0A= >communication needs.=0A= Well, I've already done MSG_TLSAPPDATA (a kernel only flag for soreceive())= and=0A= it seems to work fine. It makes soreceive() return an error instead of the= =0A= non-application data record, when that is what is at the head of the receiv= e queue.=0A= It is pretty straightforward, so unless someone has a problem with putting = that=0A= in head someday, I'm happy with it.=0A= =0A= I now seem to have upcalls for non-application data records working ok=0A= for both client and server.=0A= When the upcalls happen, the daemon just does a SSL_read(..,0), which seems= =0A= to work fine sofar. (Thanks go to Ben for that hint.)=0A= =0A= Thanks everyone, for your helpful comments, rick=0A= =0A= -Ben=0A= =0A= From owner-svn-src-projects@freebsd.org Wed May 20 23:18:48 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BBCC72F7E12 for ; Wed, 20 May 2020 23:18:48 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49S7wm4L9Gz3yCW; Wed, 20 May 2020 23:18:48 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9047423D2C; Wed, 20 May 2020 23:18:48 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04KNIm7T063561; Wed, 20 May 2020 23:18:48 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04KNIm3i063560; Wed, 20 May 2020 23:18:48 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005202318.04KNIm3i063560@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Wed, 20 May 2020 23:18:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361308 - in projects/nfs-over-tls/usr.sbin: rpctlscd rpctlssd X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls/usr.sbin: rpctlscd rpctlssd X-SVN-Commit-Revision: 361308 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2020 23:18:48 -0000 Author: rmacklem Date: Wed May 20 23:18:47 2020 New Revision: 361308 URL: https://svnweb.freebsd.org/changeset/base/361308 Log: Fix the daemons so that they use the preferred calls for openssl3 instead of SSL_CTX_load_verify_locations(). This should not have any semantics change. Modified: projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Modified: projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Wed May 20 22:25:46 2020 (r361307) +++ projects/nfs-over-tls/usr.sbin/rpctlscd/rpctlscd.c Wed May 20 23:18:47 2020 (r361308) @@ -538,9 +538,19 @@ rpctls_setupcl_ssl(bool cert) return (NULL); } } +#if OPENSSL_VERSION_NUMBER >= 0x30000000 + ret = 1; + if (rpctls_verify_cafile != NULL) + ret = SSL_CTX_load_verify_file(ctx, + rpctls_verify_cafile); + if (ret != 0 && rpctls_verify_capath != NULL) + ret = SSL_CTX_load_verify_dir(ctx, + rpctls_verify_capath); +#else ret = SSL_CTX_load_verify_locations(ctx, rpctls_verify_cafile, rpctls_verify_capath); - if (ret != 1) { +#endif + if (ret == 0) { rpctlscd_verbose_out("rpctls_setupcl_ssl: " "Can't load verify locations\n"); SSL_CTX_free(ctx); Modified: projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c ============================================================================== --- projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Wed May 20 22:25:46 2020 (r361307) +++ projects/nfs-over-tls/usr.sbin/rpctlssd/rpctlssd.c Wed May 20 23:18:47 2020 (r361308) @@ -604,8 +604,18 @@ rpctls_setup_ssl(const char *certdir) return (NULL); } } +#if OPENSSL_VERSION_NUMBER >= 0x30000000 + ret = 1; + if (rpctls_verify_cafile != NULL) + ret = SSL_CTX_load_verify_file(ctx, + rpctls_verify_cafile); + if (ret != 0 && rpctls_verify_capath != NULL) + ret = SSL_CTX_load_verify_dir(ctx, + rpctls_verify_capath); +#else ret = SSL_CTX_load_verify_locations(ctx, rpctls_verify_cafile, rpctls_verify_capath); +#endif if (ret == 0) { rpctlssd_verbose_out("rpctls_setup_ssl: " "Can't load verify locations\n"); From owner-svn-src-projects@freebsd.org Sat May 23 15:49:11 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3C3E62CE7B0 for ; Sat, 23 May 2020 15:49:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Tnpb184Rz4CXR; Sat, 23 May 2020 15:49:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 21CAA12420; Sat, 23 May 2020 15:49:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04NFnB80066069; Sat, 23 May 2020 15:49:11 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04NFn8g1066054; Sat, 23 May 2020 15:49:08 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005231549.04NFn8g1066054@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 23 May 2020 15:49:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361416 - in projects/nfs-over-tls/sys: amd64/amd64 amd64/include amd64/vmm amd64/vmm/amd amd64/vmm/intel amd64/vmm/io arm/arm arm/mv/discovery arm/mv/kirkwood arm/mv/orion arm/ralink a... X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls/sys: amd64/amd64 amd64/include amd64/vmm amd64/vmm/amd amd64/vmm/intel amd64/vmm/io arm/arm arm/mv/discovery arm/mv/kirkwood arm/mv/orion arm/ralink arm/xilinx arm64/arm64 arm... X-SVN-Commit-Revision: 361416 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2020 15:49:11 -0000 Author: rmacklem Date: Sat May 23 15:49:07 2020 New Revision: 361416 URL: https://svnweb.freebsd.org/changeset/base/361416 Log: Merge in an up-to-date kernel from head and make the changes required by recent modifications to the mbuf structure for ext_pgs mbufs. Now I just need to see if this all builds and works. Added: projects/nfs-over-tls/sys/amd64/include/vmm_snapshot.h - copied unchanged from r361415, head/sys/amd64/include/vmm_snapshot.h projects/nfs-over-tls/sys/amd64/vmm/vmm_snapshot.c - copied unchanged from r361415, head/sys/amd64/vmm/vmm_snapshot.c projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/irq_work.h - copied unchanged from r361415, head/sys/compat/linuxkpi/common/include/linux/irq_work.h projects/nfs-over-tls/sys/dev/hyperv/hvsock/ - copied from r361415, head/sys/dev/hyperv/hvsock/ projects/nfs-over-tls/sys/modules/hyperv/hvsock/ - copied from r361415, head/sys/modules/hyperv/hvsock/ projects/nfs-over-tls/sys/powerpc/aim/mmu_radix.c - copied unchanged from r361415, head/sys/powerpc/aim/mmu_radix.c Deleted: projects/nfs-over-tls/sys/arm/mv/discovery/ projects/nfs-over-tls/sys/arm/mv/kirkwood/ projects/nfs-over-tls/sys/arm/mv/orion/ projects/nfs-over-tls/sys/arm/ralink/ projects/nfs-over-tls/sys/cddl/contrib/opensolaris/common/atomic/aarch64/ projects/nfs-over-tls/sys/cddl/contrib/opensolaris/common/atomic/amd64/ projects/nfs-over-tls/sys/cddl/contrib/opensolaris/common/atomic/powerpc64/ projects/nfs-over-tls/sys/cddl/contrib/opensolaris/common/atomic/sparc64/ projects/nfs-over-tls/sys/crypto/blowfish/ projects/nfs-over-tls/sys/dev/auxio/ projects/nfs-over-tls/sys/dev/bktr/ projects/nfs-over-tls/sys/dev/sound/sbus/ projects/nfs-over-tls/sys/dev/ubsec/ projects/nfs-over-tls/sys/modules/auxio/ projects/nfs-over-tls/sys/modules/bktr/ projects/nfs-over-tls/sys/modules/epic/ projects/nfs-over-tls/sys/modules/libalias/modules/cuseeme/ projects/nfs-over-tls/sys/modules/sound/driver/audiocs/ projects/nfs-over-tls/sys/modules/ubsec/ projects/nfs-over-tls/sys/modules/vpo/ projects/nfs-over-tls/sys/netinet/libalias/alias_cuseeme.c projects/nfs-over-tls/sys/opencrypto/cast.c projects/nfs-over-tls/sys/opencrypto/cast.h projects/nfs-over-tls/sys/opencrypto/castsb.h projects/nfs-over-tls/sys/opencrypto/skipjack.c projects/nfs-over-tls/sys/opencrypto/skipjack.h projects/nfs-over-tls/sys/opencrypto/xform_blf.c projects/nfs-over-tls/sys/opencrypto/xform_cast5.c projects/nfs-over-tls/sys/opencrypto/xform_des1.c projects/nfs-over-tls/sys/opencrypto/xform_des3.c projects/nfs-over-tls/sys/opencrypto/xform_md5.c projects/nfs-over-tls/sys/opencrypto/xform_skipjack.c projects/nfs-over-tls/sys/opencrypto/xform_userland.h Modified: projects/nfs-over-tls/sys/amd64/amd64/cpu_switch.S projects/nfs-over-tls/sys/amd64/amd64/initcpu.c projects/nfs-over-tls/sys/amd64/amd64/machdep.c projects/nfs-over-tls/sys/amd64/amd64/mp_machdep.c projects/nfs-over-tls/sys/amd64/amd64/pmap.c projects/nfs-over-tls/sys/amd64/amd64/support.S projects/nfs-over-tls/sys/amd64/include/vmm.h projects/nfs-over-tls/sys/amd64/include/vmm_dev.h projects/nfs-over-tls/sys/amd64/vmm/amd/svm.c projects/nfs-over-tls/sys/amd64/vmm/amd/svm.h projects/nfs-over-tls/sys/amd64/vmm/amd/svm_msr.c projects/nfs-over-tls/sys/amd64/vmm/amd/vmcb.c projects/nfs-over-tls/sys/amd64/vmm/amd/vmcb.h projects/nfs-over-tls/sys/amd64/vmm/intel/vmcs.c projects/nfs-over-tls/sys/amd64/vmm/intel/vmcs.h projects/nfs-over-tls/sys/amd64/vmm/intel/vmx.c projects/nfs-over-tls/sys/amd64/vmm/io/vatpic.c projects/nfs-over-tls/sys/amd64/vmm/io/vatpic.h projects/nfs-over-tls/sys/amd64/vmm/io/vatpit.c projects/nfs-over-tls/sys/amd64/vmm/io/vatpit.h projects/nfs-over-tls/sys/amd64/vmm/io/vhpet.c projects/nfs-over-tls/sys/amd64/vmm/io/vhpet.h projects/nfs-over-tls/sys/amd64/vmm/io/vioapic.c projects/nfs-over-tls/sys/amd64/vmm/io/vioapic.h projects/nfs-over-tls/sys/amd64/vmm/io/vlapic.c projects/nfs-over-tls/sys/amd64/vmm/io/vlapic.h projects/nfs-over-tls/sys/amd64/vmm/io/vpmtmr.c projects/nfs-over-tls/sys/amd64/vmm/io/vpmtmr.h projects/nfs-over-tls/sys/amd64/vmm/io/vrtc.c projects/nfs-over-tls/sys/amd64/vmm/io/vrtc.h projects/nfs-over-tls/sys/amd64/vmm/vmm.c projects/nfs-over-tls/sys/amd64/vmm/vmm_dev.c projects/nfs-over-tls/sys/amd64/vmm/x86.c projects/nfs-over-tls/sys/amd64/vmm/x86.h projects/nfs-over-tls/sys/arm/arm/mp_machdep.c projects/nfs-over-tls/sys/arm/xilinx/zy7_mp.c projects/nfs-over-tls/sys/arm/xilinx/zy7_slcr.h projects/nfs-over-tls/sys/arm64/arm64/cpufunc_asm.S projects/nfs-over-tls/sys/arm64/arm64/gicv3_its.c projects/nfs-over-tls/sys/arm64/arm64/identcpu.c projects/nfs-over-tls/sys/arm64/arm64/mp_machdep.c projects/nfs-over-tls/sys/arm64/arm64/pmap.c projects/nfs-over-tls/sys/arm64/include/cpufunc.h projects/nfs-over-tls/sys/cam/ata/ata_xpt.c projects/nfs-over-tls/sys/cam/cam_ccb.h projects/nfs-over-tls/sys/cam/cam_xpt.c projects/nfs-over-tls/sys/cam/ctl/ctl.c projects/nfs-over-tls/sys/cam/ctl/ctl_backend.c projects/nfs-over-tls/sys/cam/ctl/ctl_backend.h projects/nfs-over-tls/sys/cam/ctl/ctl_backend_block.c projects/nfs-over-tls/sys/cam/ctl/ctl_backend_ramdisk.c projects/nfs-over-tls/sys/cam/ctl/ctl_frontend_iscsi.c projects/nfs-over-tls/sys/cam/ctl/ctl_private.h projects/nfs-over-tls/sys/cam/mmc/mmc_xpt.c projects/nfs-over-tls/sys/cam/nvme/nvme_da.c projects/nfs-over-tls/sys/cam/nvme/nvme_xpt.c projects/nfs-over-tls/sys/cam/scsi/scsi_enc_ses.c projects/nfs-over-tls/sys/cam/scsi/scsi_targ_bh.c projects/nfs-over-tls/sys/cam/scsi/scsi_xpt.c projects/nfs-over-tls/sys/cddl/contrib/opensolaris/common/lz4/lz4.c projects/nfs-over-tls/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zfs_rlock.h projects/nfs-over-tls/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c projects/nfs-over-tls/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c projects/nfs-over-tls/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/bitmap.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/errno.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/kernel.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/lockdep.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/math64.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/mutex.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/pci.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/scatterlist.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/srcu.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/wait.h projects/nfs-over-tls/sys/compat/linuxkpi/common/include/linux/workqueue.h projects/nfs-over-tls/sys/compat/linuxkpi/common/src/linux_compat.c projects/nfs-over-tls/sys/compat/linuxkpi/common/src/linux_pci.c projects/nfs-over-tls/sys/compat/linuxkpi/common/src/linux_work.c projects/nfs-over-tls/sys/conf/Makefile.arm projects/nfs-over-tls/sys/conf/Makefile.powerpc projects/nfs-over-tls/sys/conf/Makefile.riscv projects/nfs-over-tls/sys/conf/NOTES projects/nfs-over-tls/sys/conf/config.mk projects/nfs-over-tls/sys/conf/files projects/nfs-over-tls/sys/conf/files.amd64 projects/nfs-over-tls/sys/conf/files.arm projects/nfs-over-tls/sys/conf/files.arm64 projects/nfs-over-tls/sys/conf/files.i386 projects/nfs-over-tls/sys/conf/files.mips projects/nfs-over-tls/sys/conf/files.powerpc projects/nfs-over-tls/sys/conf/files.riscv projects/nfs-over-tls/sys/conf/files.x86 projects/nfs-over-tls/sys/conf/kern.mk projects/nfs-over-tls/sys/conf/kern.opts.mk projects/nfs-over-tls/sys/conf/kern.post.mk projects/nfs-over-tls/sys/conf/kern.pre.mk projects/nfs-over-tls/sys/conf/kmod.mk projects/nfs-over-tls/sys/conf/ldscript.powerpc projects/nfs-over-tls/sys/conf/ldscript.powerpc64 projects/nfs-over-tls/sys/conf/ldscript.powerpcspe projects/nfs-over-tls/sys/conf/options projects/nfs-over-tls/sys/conf/options.amd64 projects/nfs-over-tls/sys/contrib/dev/acpica/changes.txt projects/nfs-over-tls/sys/contrib/dev/acpica/common/acgetline.c projects/nfs-over-tls/sys/contrib/dev/acpica/common/dmtbdump2.c projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslcompiler.l projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslload.c projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslmessages.c projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslmessages.h projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslpredef.c projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/aslxref.c projects/nfs-over-tls/sys/contrib/dev/acpica/compiler/dtutils.c projects/nfs-over-tls/sys/contrib/dev/acpica/components/debugger/dbhistry.c projects/nfs-over-tls/sys/contrib/dev/acpica/components/disassembler/dmbuffer.c projects/nfs-over-tls/sys/contrib/dev/acpica/components/dispatcher/dsfield.c projects/nfs-over-tls/sys/contrib/dev/acpica/components/executer/exfield.c projects/nfs-over-tls/sys/contrib/dev/acpica/include/acglobal.h projects/nfs-over-tls/sys/contrib/dev/acpica/include/acpixf.h projects/nfs-over-tls/sys/contrib/dev/acpica/include/acpredef.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300_eeprom.c projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300_phy.c projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300eep.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_ap121.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_aphrodite.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_cus157.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_generic.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_hb112.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_hb116.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_osprey_k31.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_wasp_2.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_xb112.h projects/nfs-over-tls/sys/contrib/dev/ath/ath_hal/ar9300/ar9300template_xb113.h projects/nfs-over-tls/sys/crypto/ccp/ccp.c projects/nfs-over-tls/sys/crypto/chacha20/chacha-sw.c projects/nfs-over-tls/sys/crypto/via/padlock_hash.c projects/nfs-over-tls/sys/ddb/db_expr.c projects/nfs-over-tls/sys/dev/acpica/acpi_lid.c projects/nfs-over-tls/sys/dev/acpica/acpi_pxm.c projects/nfs-over-tls/sys/dev/acpica/acpi_video.c projects/nfs-over-tls/sys/dev/acpica/acpivar.h projects/nfs-over-tls/sys/dev/amdtemp/amdtemp.c projects/nfs-over-tls/sys/dev/ath/ath_hal/ah.h projects/nfs-over-tls/sys/dev/ath/ath_hal/ar5416/ar2133.c projects/nfs-over-tls/sys/dev/ath/ath_rate/amrr/amrr.c projects/nfs-over-tls/sys/dev/ath/ath_rate/onoe/onoe.c projects/nfs-over-tls/sys/dev/ath/ath_rate/sample/sample.c projects/nfs-over-tls/sys/dev/ath/ath_rate/sample/sample.h projects/nfs-over-tls/sys/dev/ath/if_ath.c projects/nfs-over-tls/sys/dev/ath/if_ath_misc.h projects/nfs-over-tls/sys/dev/ath/if_ath_rx_edma.c projects/nfs-over-tls/sys/dev/ath/if_ath_tx.c projects/nfs-over-tls/sys/dev/ath/if_ath_tx_ht.c projects/nfs-over-tls/sys/dev/ath/if_athrate.h projects/nfs-over-tls/sys/dev/ath/if_athvar.h projects/nfs-over-tls/sys/dev/bnxt/bnxt.h projects/nfs-over-tls/sys/dev/bnxt/if_bnxt.c projects/nfs-over-tls/sys/dev/cesa/cesa.c projects/nfs-over-tls/sys/dev/cxgbe/adapter.h projects/nfs-over-tls/sys/dev/cxgbe/crypto/t4_crypto.c projects/nfs-over-tls/sys/dev/cxgbe/crypto/t4_kern_tls.c projects/nfs-over-tls/sys/dev/cxgbe/iw_cxgbe/cm.c projects/nfs-over-tls/sys/dev/cxgbe/iw_cxgbe/device.c projects/nfs-over-tls/sys/dev/cxgbe/offload.h projects/nfs-over-tls/sys/dev/cxgbe/t4_main.c projects/nfs-over-tls/sys/dev/cxgbe/t4_sge.c projects/nfs-over-tls/sys/dev/cxgbe/tom/t4_cpl_io.c projects/nfs-over-tls/sys/dev/cxgbe/tom/t4_tls.c projects/nfs-over-tls/sys/dev/e1000/if_em.c projects/nfs-over-tls/sys/dev/ena/ena.c projects/nfs-over-tls/sys/dev/ena/ena.h projects/nfs-over-tls/sys/dev/ena/ena_sysctl.c projects/nfs-over-tls/sys/dev/ena/ena_sysctl.h projects/nfs-over-tls/sys/dev/etherswitch/etherswitch.h projects/nfs-over-tls/sys/dev/evdev/evdev_utils.c projects/nfs-over-tls/sys/dev/evdev/input-event-codes.h projects/nfs-over-tls/sys/dev/evdev/input.h projects/nfs-over-tls/sys/dev/glxsb/glxsb.c projects/nfs-over-tls/sys/dev/gpio/gpioiic.c projects/nfs-over-tls/sys/dev/hifn/hifn7751.c projects/nfs-over-tls/sys/dev/hyperv/include/vmbus.h projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus.c projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus_br.c projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus_brvar.h projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus_chan.c projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus_chanvar.h projects/nfs-over-tls/sys/dev/hyperv/vmbus/vmbus_reg.h projects/nfs-over-tls/sys/dev/ichiic/ig4_acpi.c projects/nfs-over-tls/sys/dev/iscsi/icl_soft.c projects/nfs-over-tls/sys/dev/iwm/if_iwm.c projects/nfs-over-tls/sys/dev/iwm/if_iwmvar.h projects/nfs-over-tls/sys/dev/ixgbe/if_ix.c projects/nfs-over-tls/sys/dev/ixgbe/if_ixv.c projects/nfs-over-tls/sys/dev/ixl/if_iavf.c projects/nfs-over-tls/sys/dev/ixl/if_ixl.c projects/nfs-over-tls/sys/dev/mlx5/device.h projects/nfs-over-tls/sys/dev/mlx5/mlx5_core/mlx5_eq.c projects/nfs-over-tls/sys/dev/mlx5/mlx5_core/mlx5_main.c projects/nfs-over-tls/sys/dev/mlx5/mlx5_en/mlx5_en_hw_tls.c projects/nfs-over-tls/sys/dev/nvdimm/nvdimm.c projects/nfs-over-tls/sys/dev/nvme/nvme.c projects/nfs-over-tls/sys/dev/nvme/nvme_ctrlr.c projects/nfs-over-tls/sys/dev/nvme/nvme_ns.c projects/nfs-over-tls/sys/dev/nvme/nvme_pci.c projects/nfs-over-tls/sys/dev/nvme/nvme_qpair.c projects/nfs-over-tls/sys/dev/nvme/nvme_sim.c projects/nfs-over-tls/sys/dev/nvme/nvme_sysctl.c projects/nfs-over-tls/sys/dev/nvme/nvme_test.c projects/nfs-over-tls/sys/dev/ow/ow.c projects/nfs-over-tls/sys/dev/qlnx/qlnxe/bcm_osal.h projects/nfs-over-tls/sys/dev/random/random_harvestq.c projects/nfs-over-tls/sys/dev/rtwn/usb/rtwn_usb_attach.h projects/nfs-over-tls/sys/dev/safe/safe.c projects/nfs-over-tls/sys/dev/sec/sec.c projects/nfs-over-tls/sys/dev/sound/pci/hda/hdac.c projects/nfs-over-tls/sys/dev/sound/pcm/dsp.c projects/nfs-over-tls/sys/dev/usb/usb_device.c projects/nfs-over-tls/sys/dev/usb/usb_device.h projects/nfs-over-tls/sys/dev/usb/usb_hub.c projects/nfs-over-tls/sys/dev/usb/usbdevs projects/nfs-over-tls/sys/dev/usb/usbhid.h projects/nfs-over-tls/sys/dev/virtio/balloon/virtio_balloon.c projects/nfs-over-tls/sys/dev/virtio/console/virtio_console.c projects/nfs-over-tls/sys/dev/virtio/mmio/virtio_mmio.c projects/nfs-over-tls/sys/dev/virtio/mmio/virtio_mmio.h projects/nfs-over-tls/sys/dev/virtio/random/virtio_random.c projects/nfs-over-tls/sys/dev/virtio/scsi/virtio_scsi.c projects/nfs-over-tls/sys/dev/xen/evtchn/evtchn_dev.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_alloc.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_balloc.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_bmap.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_csum.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_extattr.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_extents.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_extents.h projects/nfs-over-tls/sys/fs/ext2fs/ext2_htree.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_inode.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_inode_cnv.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_lookup.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_subr.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_vfsops.c projects/nfs-over-tls/sys/fs/ext2fs/ext2_vnops.c projects/nfs-over-tls/sys/fs/ext2fs/ext2fs.h projects/nfs-over-tls/sys/fs/ext2fs/fs.h projects/nfs-over-tls/sys/fs/fuse/fuse_internal.c projects/nfs-over-tls/sys/fs/fuse/fuse_vfsops.c projects/nfs-over-tls/sys/fs/fuse/fuse_vnops.c projects/nfs-over-tls/sys/fs/nfs/nfs_commonacl.c projects/nfs-over-tls/sys/fs/nfs/nfs_commonsubs.c projects/nfs-over-tls/sys/fs/nfs/nfs_var.h projects/nfs-over-tls/sys/fs/nfs/nfsdport.h projects/nfs-over-tls/sys/fs/nfs/nfsm_subs.h projects/nfs-over-tls/sys/fs/nfs/nfsport.h projects/nfs-over-tls/sys/fs/nfsclient/nfs_clcomsubs.c projects/nfs-over-tls/sys/fs/nfsclient/nfs_clport.c projects/nfs-over-tls/sys/fs/nfsclient/nfs_clrpcops.c projects/nfs-over-tls/sys/fs/nfsclient/nfs_clstate.c projects/nfs-over-tls/sys/fs/nfsclient/nfs_clvfsops.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdcache.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdport.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdserv.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdsocket.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdstate.c projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdsubs.c projects/nfs-over-tls/sys/geom/eli/g_eli.c projects/nfs-over-tls/sys/geom/geom.h projects/nfs-over-tls/sys/geom/geom_dev.c projects/nfs-over-tls/sys/geom/geom_disk.c projects/nfs-over-tls/sys/geom/geom_dump.c projects/nfs-over-tls/sys/geom/geom_subr.c projects/nfs-over-tls/sys/geom/mirror/g_mirror_ctl.c projects/nfs-over-tls/sys/geom/mountver/g_mountver.c projects/nfs-over-tls/sys/geom/nop/g_nop.c projects/nfs-over-tls/sys/geom/part/g_part.c projects/nfs-over-tls/sys/geom/uzip/g_uzip.c projects/nfs-over-tls/sys/i386/i386/mp_machdep.c projects/nfs-over-tls/sys/i386/i386/support.s projects/nfs-over-tls/sys/kern/kern_exec.c projects/nfs-over-tls/sys/kern/kern_fork.c projects/nfs-over-tls/sys/kern/kern_linker.c projects/nfs-over-tls/sys/kern/kern_mbuf.c projects/nfs-over-tls/sys/kern/kern_mib.c projects/nfs-over-tls/sys/kern/kern_procctl.c projects/nfs-over-tls/sys/kern/kern_sendfile.c projects/nfs-over-tls/sys/kern/kern_sig.c projects/nfs-over-tls/sys/kern/link_elf.c projects/nfs-over-tls/sys/kern/subr_bus_dma.c projects/nfs-over-tls/sys/kern/subr_prf.c projects/nfs-over-tls/sys/kern/subr_sglist.c projects/nfs-over-tls/sys/kern/subr_uio.c projects/nfs-over-tls/sys/kern/tty.c projects/nfs-over-tls/sys/kern/uipc_ktls.c projects/nfs-over-tls/sys/kern/uipc_mbuf.c projects/nfs-over-tls/sys/kern/uipc_sockbuf.c projects/nfs-over-tls/sys/kern/uipc_socket.c projects/nfs-over-tls/sys/kern/vfs_export.c projects/nfs-over-tls/sys/kern/vfs_extattr.c projects/nfs-over-tls/sys/kern/vfs_subr.c projects/nfs-over-tls/sys/mips/atheros/ar531x/ar5315_machdep.c projects/nfs-over-tls/sys/mips/atheros/ar71xx_machdep.c projects/nfs-over-tls/sys/mips/atheros/if_arge.c projects/nfs-over-tls/sys/mips/atheros/if_argevar.h projects/nfs-over-tls/sys/mips/cavium/cryptocteon/cavium_crypto.c projects/nfs-over-tls/sys/mips/cavium/cryptocteon/cryptocteon.c projects/nfs-over-tls/sys/mips/cavium/cryptocteon/cryptocteonvar.h projects/nfs-over-tls/sys/mips/cavium/octeon_machdep.c projects/nfs-over-tls/sys/mips/include/md_var.h projects/nfs-over-tls/sys/mips/mediatek/mtk_soc.c projects/nfs-over-tls/sys/mips/mips/cpu.c projects/nfs-over-tls/sys/mips/mips/freebsd32_machdep.c projects/nfs-over-tls/sys/mips/mips/machdep.c projects/nfs-over-tls/sys/mips/mips/pm_machdep.c projects/nfs-over-tls/sys/mips/mips/swtch.S projects/nfs-over-tls/sys/mips/nlm/dev/sec/nlmsec.c projects/nfs-over-tls/sys/mips/nlm/dev/sec/nlmseclib.c projects/nfs-over-tls/sys/modules/Makefile projects/nfs-over-tls/sys/modules/crypto/Makefile projects/nfs-over-tls/sys/modules/hyperv/Makefile projects/nfs-over-tls/sys/modules/libalias/modules/modules.inc projects/nfs-over-tls/sys/modules/vmm/Makefile projects/nfs-over-tls/sys/net/if.c projects/nfs-over-tls/sys/net/iflib.c projects/nfs-over-tls/sys/net/mppcc.c projects/nfs-over-tls/sys/net/netisr.c projects/nfs-over-tls/sys/net/pfvar.h projects/nfs-over-tls/sys/net/route.c projects/nfs-over-tls/sys/net/route.h projects/nfs-over-tls/sys/net/route/nhop.h projects/nfs-over-tls/sys/net/route/nhop_ctl.c projects/nfs-over-tls/sys/net/route/nhop_var.h projects/nfs-over-tls/sys/net/route/route_ddb.c projects/nfs-over-tls/sys/net/route/route_helpers.c projects/nfs-over-tls/sys/net/route/route_var.h projects/nfs-over-tls/sys/net/rtsock.c projects/nfs-over-tls/sys/net80211/ieee80211_freebsd.c projects/nfs-over-tls/sys/net80211/ieee80211_output.c projects/nfs-over-tls/sys/net80211/ieee80211_proto.c projects/nfs-over-tls/sys/net80211/ieee80211_proto.h projects/nfs-over-tls/sys/netgraph/bluetooth/include/ng_hci.h projects/nfs-over-tls/sys/netinet/accf_data.c projects/nfs-over-tls/sys/netinet/accf_dns.c projects/nfs-over-tls/sys/netinet/accf_http.c projects/nfs-over-tls/sys/netinet/cc/cc_cubic.c projects/nfs-over-tls/sys/netinet/cc/cc_dctcp.c projects/nfs-over-tls/sys/netinet/in_pcb.c projects/nfs-over-tls/sys/netinet/in_pcb.h projects/nfs-over-tls/sys/netinet/in_rmx.c projects/nfs-over-tls/sys/netinet/ip_output.c projects/nfs-over-tls/sys/netinet/libalias/alias.c projects/nfs-over-tls/sys/netinet/libalias/alias_ftp.c projects/nfs-over-tls/sys/netinet/libalias/libalias.3 projects/nfs-over-tls/sys/netinet/sctp_asconf.c projects/nfs-over-tls/sys/netinet/sctp_auth.c projects/nfs-over-tls/sys/netinet/sctp_indata.c projects/nfs-over-tls/sys/netinet/sctp_indata.h projects/nfs-over-tls/sys/netinet/sctp_input.c projects/nfs-over-tls/sys/netinet/sctp_os_bsd.h projects/nfs-over-tls/sys/netinet/sctp_output.c projects/nfs-over-tls/sys/netinet/sctp_pcb.c projects/nfs-over-tls/sys/netinet/sctp_timer.c projects/nfs-over-tls/sys/netinet/sctp_usrreq.c projects/nfs-over-tls/sys/netinet/sctputil.c projects/nfs-over-tls/sys/netinet/tcp_input.c projects/nfs-over-tls/sys/netinet/tcp_output.c projects/nfs-over-tls/sys/netinet/tcp_pcap.c projects/nfs-over-tls/sys/netinet/tcp_stacks/bbr.c projects/nfs-over-tls/sys/netinet/tcp_stacks/rack.c projects/nfs-over-tls/sys/netinet/tcp_stacks/rack_bbr_common.c projects/nfs-over-tls/sys/netinet/tcp_stacks/rack_bbr_common.h projects/nfs-over-tls/sys/netinet/tcp_stacks/tcp_bbr.h projects/nfs-over-tls/sys/netinet/tcp_stacks/tcp_rack.h projects/nfs-over-tls/sys/netinet/tcp_usrreq.c projects/nfs-over-tls/sys/netinet/tcp_var.h projects/nfs-over-tls/sys/netinet6/in6_pcb.c projects/nfs-over-tls/sys/netinet6/in6_pcb.h projects/nfs-over-tls/sys/netinet6/in6_rmx.c projects/nfs-over-tls/sys/netinet6/ip6_output.c projects/nfs-over-tls/sys/netinet6/nd6.c projects/nfs-over-tls/sys/netinet6/nd6_rtr.c projects/nfs-over-tls/sys/netipsec/ipsec.c projects/nfs-over-tls/sys/netipsec/ipsec.h projects/nfs-over-tls/sys/netipsec/key.c projects/nfs-over-tls/sys/netipsec/xform_ah.c projects/nfs-over-tls/sys/netipsec/xform_esp.c projects/nfs-over-tls/sys/netpfil/ipfw/ip_fw2.c projects/nfs-over-tls/sys/netpfil/pf/pf_ioctl.c projects/nfs-over-tls/sys/netpfil/pf/pf_table.c projects/nfs-over-tls/sys/nfs/bootp_subr.c projects/nfs-over-tls/sys/opencrypto/crypto.c projects/nfs-over-tls/sys/opencrypto/cryptodev.c projects/nfs-over-tls/sys/opencrypto/cryptodev.h projects/nfs-over-tls/sys/opencrypto/cryptosoft.c projects/nfs-over-tls/sys/opencrypto/xform.c projects/nfs-over-tls/sys/opencrypto/xform.h projects/nfs-over-tls/sys/opencrypto/xform_aes_icm.c projects/nfs-over-tls/sys/opencrypto/xform_aes_xts.c projects/nfs-over-tls/sys/opencrypto/xform_auth.h projects/nfs-over-tls/sys/opencrypto/xform_cml.c projects/nfs-over-tls/sys/opencrypto/xform_comp.h projects/nfs-over-tls/sys/opencrypto/xform_enc.h projects/nfs-over-tls/sys/opencrypto/xform_gmac.c projects/nfs-over-tls/sys/opencrypto/xform_null.c projects/nfs-over-tls/sys/opencrypto/xform_rijndael.c projects/nfs-over-tls/sys/opencrypto/xform_sha1.c projects/nfs-over-tls/sys/powerpc/aim/aim_machdep.c projects/nfs-over-tls/sys/powerpc/aim/locore64.S projects/nfs-over-tls/sys/powerpc/aim/mmu_oea.c projects/nfs-over-tls/sys/powerpc/aim/mmu_oea64.c projects/nfs-over-tls/sys/powerpc/aim/trap_subr64.S projects/nfs-over-tls/sys/powerpc/booke/machdep_e500.c projects/nfs-over-tls/sys/powerpc/booke/pmap.c projects/nfs-over-tls/sys/powerpc/conf/QORIQ64 projects/nfs-over-tls/sys/powerpc/include/cpu.h projects/nfs-over-tls/sys/powerpc/include/cpufunc.h projects/nfs-over-tls/sys/powerpc/include/db_machdep.h projects/nfs-over-tls/sys/powerpc/include/mmuvar.h projects/nfs-over-tls/sys/powerpc/include/param.h projects/nfs-over-tls/sys/powerpc/include/pmap.h projects/nfs-over-tls/sys/powerpc/include/pte.h projects/nfs-over-tls/sys/powerpc/include/spr.h projects/nfs-over-tls/sys/powerpc/include/sr.h projects/nfs-over-tls/sys/powerpc/include/vmparam.h projects/nfs-over-tls/sys/powerpc/ofw/ofw_initrd.c projects/nfs-over-tls/sys/powerpc/powerpc/elf32_machdep.c projects/nfs-over-tls/sys/powerpc/powerpc/elf64_machdep.c projects/nfs-over-tls/sys/powerpc/powerpc/machdep.c projects/nfs-over-tls/sys/powerpc/powerpc/mmu_if.m projects/nfs-over-tls/sys/powerpc/powerpc/pmap_dispatch.c projects/nfs-over-tls/sys/powerpc/powerpc/trap.c projects/nfs-over-tls/sys/riscv/include/sbi.h projects/nfs-over-tls/sys/riscv/riscv/locore.S projects/nfs-over-tls/sys/riscv/riscv/machdep.c projects/nfs-over-tls/sys/riscv/riscv/mp_machdep.c projects/nfs-over-tls/sys/riscv/riscv/pmap.c projects/nfs-over-tls/sys/riscv/riscv/sbi.c projects/nfs-over-tls/sys/rpc/rpc_generic.c projects/nfs-over-tls/sys/security/audit/audit_bsm.c projects/nfs-over-tls/sys/security/mac_bsdextended/mac_bsdextended.c projects/nfs-over-tls/sys/security/mac_bsdextended/mac_bsdextended.h projects/nfs-over-tls/sys/sys/elf_common.h projects/nfs-over-tls/sys/sys/ktls.h projects/nfs-over-tls/sys/sys/link_elf.h projects/nfs-over-tls/sys/sys/mbuf.h projects/nfs-over-tls/sys/sys/mount.h projects/nfs-over-tls/sys/sys/param.h projects/nfs-over-tls/sys/sys/random.h projects/nfs-over-tls/sys/sys/sglist.h projects/nfs-over-tls/sys/sys/socket.h projects/nfs-over-tls/sys/sys/socketvar.h projects/nfs-over-tls/sys/sys/systm.h projects/nfs-over-tls/sys/sys/tree.h projects/nfs-over-tls/sys/sys/uio.h projects/nfs-over-tls/sys/ufs/ffs/ffs_alloc.c projects/nfs-over-tls/sys/vm/uma_core.c projects/nfs-over-tls/sys/vm/vm_fault.c projects/nfs-over-tls/sys/vm/vm_radix.c projects/nfs-over-tls/sys/x86/acpica/srat.c projects/nfs-over-tls/sys/x86/include/x86_var.h projects/nfs-over-tls/sys/x86/x86/cpu_machdep.c Directory Properties: projects/nfs-over-tls/sys/ (props changed) projects/nfs-over-tls/sys/cddl/contrib/opensolaris/ (props changed) projects/nfs-over-tls/sys/contrib/dev/acpica/ (props changed) Modified: projects/nfs-over-tls/sys/amd64/amd64/cpu_switch.S ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/cpu_switch.S Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/cpu_switch.S Sat May 23 15:49:07 2020 (r361416) @@ -221,6 +221,8 @@ done_load_dr: movq %rax,(%rsp) movq PCPU(CURTHREAD),%rdi call fpu_activate_sw + cmpb $0,cpu_flush_rsb_ctxsw(%rip) + jne rsb_flush ret /* Modified: projects/nfs-over-tls/sys/amd64/amd64/initcpu.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/initcpu.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/initcpu.c Sat May 23 15:49:07 2020 (r361416) @@ -238,12 +238,24 @@ initializecpu(void) cr4 |= CR4_PKE; /* + * If SMEP is present, we only need to flush RSB (by default) + * on context switches, to prevent cross-process ret2spec + * attacks. Do it automatically if ibrs_disable is set, to + * complete the mitigation. + * * Postpone enabling the SMEP on the boot CPU until the page * tables are switched from the boot loader identity mapping * to the kernel tables. The boot loader enables the U bit in * its tables. */ - if (!IS_BSP()) { + if (IS_BSP()) { + if (cpu_stdext_feature & CPUID_STDEXT_SMEP && + !TUNABLE_INT_FETCH( + "machdep.mitigations.cpu_flush_rsb_ctxsw", + &cpu_flush_rsb_ctxsw) && + hw_ibrs_disable) + cpu_flush_rsb_ctxsw = 1; + } else { if (cpu_stdext_feature & CPUID_STDEXT_SMEP) cr4 |= CR4_SMEP; if (cpu_stdext_feature & CPUID_STDEXT_SMAP) Modified: projects/nfs-over-tls/sys/amd64/amd64/machdep.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/machdep.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/machdep.c Sat May 23 15:49:07 2020 (r361416) @@ -1857,7 +1857,7 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) setidt(IDT_IO_INTS + 15, IDTVEC(spuriousint), SDT_SYSIGT, SEL_KPL, 0); #endif #else -#error "have you forgotten the isa device?"; +#error "have you forgotten the isa device?" #endif if (late_console) @@ -1867,12 +1867,13 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) fpuinit(); /* - * Set up thread0 pcb save area after fpuinit calculated fpu save - * area size. Zero out the extended state header in fpu save - * area. + * Reinitialize thread0's stack base now that the xsave area size is + * known. Set up thread0's pcb save area after fpuinit calculated fpu + * save area size. Zero out the extended state header in fpu save area. */ + set_top_of_stack_td(&thread0); thread0.td_pcb->pcb_save = get_pcb_user_save_td(&thread0); - bzero(get_pcb_user_save_td(&thread0), cpu_max_ext_state_size); + bzero(thread0.td_pcb->pcb_save, cpu_max_ext_state_size); if (use_xsave) { xhdr = (struct xstate_hdr *)(get_pcb_user_save_td(&thread0) + 1); @@ -1882,7 +1883,7 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) rsp0 = thread0.td_md.md_stack_base; /* Ensure the stack is aligned to 16 bytes */ rsp0 &= ~0xFul; - __pcpu[0].pc_common_tss.tss_rsp0 = rsp0; + PCPU_PTR(common_tss)->tss_rsp0 = rsp0; amd64_bsp_pcpu_init2(rsp0); /* transfer to user mode */ Modified: projects/nfs-over-tls/sys/amd64/amd64/mp_machdep.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/mp_machdep.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/mp_machdep.c Sat May 23 15:49:07 2020 (r361416) @@ -29,6 +29,7 @@ #include __FBSDID("$FreeBSD$"); +#include "opt_acpi.h" #include "opt_cpu.h" #include "opt_ddb.h" #include "opt_kstack_pages.h" @@ -78,8 +79,10 @@ __FBSDID("$FreeBSD$"); #include #include +#ifdef DEV_ACPI #include #include +#endif #define WARMBOOT_TARGET 0 #define WARMBOOT_OFF (KERNBASE + 0x0467) @@ -264,8 +267,11 @@ cpu_mp_start(void) init_ops.start_all_aps(); set_interrupt_apic_ids(); -} +#if defined(DEV_ACPI) && MAXMEMDOM > 1 + acpi_pxm_set_cpu_locality(); +#endif +} /* * AP CPU's call this to initialize themselves. Modified: projects/nfs-over-tls/sys/amd64/amd64/pmap.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/pmap.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/pmap.c Sat May 23 15:49:07 2020 (r361416) @@ -323,12 +323,12 @@ pmap_pku_mask_bit(pmap_t pmap) #endif #undef pa_index +#ifdef NUMA #define pa_index(pa) ({ \ KASSERT((pa) <= vm_phys_segs[vm_phys_nsegs - 1].end, \ ("address %lx beyond the last segment", (pa))); \ (pa) >> PDRSHIFT; \ }) -#ifdef NUMA #define pa_to_pmdp(pa) (&pv_table[pa_index(pa)]) #define pa_to_pvh(pa) (&(pa_to_pmdp(pa)->pv_page)) #define PHYS_TO_PV_LIST_LOCK(pa) ({ \ @@ -340,6 +340,7 @@ pmap_pku_mask_bit(pmap_t pmap) _lock; \ }) #else +#define pa_index(pa) ((pa) >> PDRSHIFT) #define pa_to_pvh(pa) (&pv_table[pa_index(pa)]) #define NPV_LIST_LOCKS MAXCPU @@ -1316,6 +1317,8 @@ pmap_pdpe_to_pde(pdp_entry_t *pdpe, vm_offset_t va) { pd_entry_t *pde; + KASSERT((*pdpe & PG_PS) == 0, + ("%s: pdpe %#lx is a leaf", __func__, *pdpe)); pde = (pd_entry_t *)PHYS_TO_DMAP(*pdpe & PG_FRAME); return (&pde[pmap_pde_index(va)]); } @@ -1340,6 +1343,8 @@ pmap_pde_to_pte(pd_entry_t *pde, vm_offset_t va) { pt_entry_t *pte; + KASSERT((*pde & PG_PS) == 0, + ("%s: pde %#lx is a leaf", __func__, *pde)); pte = (pt_entry_t *)PHYS_TO_DMAP(*pde & PG_FRAME); return (&pte[pmap_pte_index(va)]); } Modified: projects/nfs-over-tls/sys/amd64/amd64/support.S ============================================================================== --- projects/nfs-over-tls/sys/amd64/amd64/support.S Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/amd64/support.S Sat May 23 15:49:07 2020 (r361416) @@ -1613,23 +1613,27 @@ ENTRY(pmap_pti_pcid_invlrng) retq .altmacro - .macro ibrs_seq_label l -handle_ibrs_\l: + .macro rsb_seq_label l +rsb_seq_\l: .endm - .macro ibrs_call_label l - call handle_ibrs_\l + .macro rsb_call_label l + call rsb_seq_\l .endm - .macro ibrs_seq count + .macro rsb_seq count ll=1 .rept \count - ibrs_call_label %(ll) + rsb_call_label %(ll) nop - ibrs_seq_label %(ll) + rsb_seq_label %(ll) addq $8,%rsp ll=ll+1 .endr .endm +ENTRY(rsb_flush) + rsb_seq 32 + ret + /* all callers already saved %rax, %rdx, and %rcx */ ENTRY(handle_ibrs_entry) cmpb $0,hw_ibrs_ibpb_active(%rip) @@ -1641,8 +1645,7 @@ ENTRY(handle_ibrs_entry) wrmsr movb $1,PCPU(IBPB_SET) testl $CPUID_STDEXT_SMEP,cpu_stdext_feature(%rip) - jne 1f - ibrs_seq 32 + je rsb_flush 1: ret END(handle_ibrs_entry) Modified: projects/nfs-over-tls/sys/amd64/include/vmm.h ============================================================================== --- projects/nfs-over-tls/sys/amd64/include/vmm.h Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/include/vmm.h Sat May 23 15:49:07 2020 (r361416) @@ -34,6 +34,8 @@ #include #include +struct vm_snapshot_meta; + #ifdef _KERNEL SDT_PROVIDER_DECLARE(vmm); #endif @@ -152,6 +154,7 @@ struct vmspace; struct vm_object; struct vm_guest_paging; struct pmap; +enum snapshot_req; struct vm_eventinfo { void *rptr; /* rendezvous cookie */ @@ -180,6 +183,10 @@ typedef struct vmspace * (*vmi_vmspace_alloc)(vm_offse typedef void (*vmi_vmspace_free)(struct vmspace *vmspace); typedef struct vlapic * (*vmi_vlapic_init)(void *vmi, int vcpu); typedef void (*vmi_vlapic_cleanup)(void *vmi, struct vlapic *vlapic); +typedef int (*vmi_snapshot_t)(void *vmi, struct vm_snapshot_meta *meta); +typedef int (*vmi_snapshot_vmcx_t)(void *vmi, struct vm_snapshot_meta *meta, + int vcpu); +typedef int (*vmi_restore_tsc_t)(void *vmi, int vcpuid, uint64_t now); struct vmm_ops { vmm_init_func_t init; /* module wide initialization */ @@ -199,6 +206,11 @@ struct vmm_ops { vmi_vmspace_free vmspace_free; vmi_vlapic_init vlapic_init; vmi_vlapic_cleanup vlapic_cleanup; + + /* checkpoint operations */ + vmi_snapshot_t vmsnapshot; + vmi_snapshot_vmcx_t vmcx_snapshot; + vmi_restore_tsc_t vm_restore_tsc; }; extern struct vmm_ops vmm_ops_intel; @@ -272,7 +284,10 @@ void vm_exit_debug(struct vm *vm, int vcpuid, uint64_t void vm_exit_rendezvous(struct vm *vm, int vcpuid, uint64_t rip); void vm_exit_astpending(struct vm *vm, int vcpuid, uint64_t rip); void vm_exit_reqidle(struct vm *vm, int vcpuid, uint64_t rip); +int vm_snapshot_req(struct vm *vm, struct vm_snapshot_meta *meta); +int vm_restore_time(struct vm *vm); + #ifdef _SYS__CPUSET_H_ /* * Rendezvous all vcpus specified in 'dest' and execute 'func(arg)'. @@ -408,6 +423,15 @@ int vm_exit_intinfo(struct vm *vm, int vcpuid, uint64_ int vm_entry_intinfo(struct vm *vm, int vcpuid, uint64_t *info); int vm_get_intinfo(struct vm *vm, int vcpuid, uint64_t *info1, uint64_t *info2); + +/* + * Function used to keep track of the guest's TSC offset. The + * offset is used by the virutalization extensions to provide a consistent + * value for the Time Stamp Counter to the guest. + * + * Return value is 0 on success and non-zero on failure. + */ +int vm_set_tsc_offset(struct vm *vm, int vcpu_id, uint64_t offset); enum vm_reg_name vm_segment_name(int seg_encoding); Modified: projects/nfs-over-tls/sys/amd64/include/vmm_dev.h ============================================================================== --- projects/nfs-over-tls/sys/amd64/include/vmm_dev.h Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/include/vmm_dev.h Sat May 23 15:49:07 2020 (r361416) @@ -31,6 +31,8 @@ #ifndef _VMM_DEV_H_ #define _VMM_DEV_H_ +struct vm_snapshot_meta; + #ifdef _KERNEL void vmmdev_init(void); int vmmdev_cleanup(void); @@ -54,6 +56,13 @@ struct vm_memseg { char name[VM_MAX_SUFFIXLEN + 1]; }; +struct vm_memseg_fbsd12 { + int segid; + size_t len; + char name[64]; +}; +_Static_assert(sizeof(struct vm_memseg_fbsd12) == 80, "COMPAT_FREEBSD12 ABI"); + struct vm_register { int cpuid; int regnum; /* enum vm_reg_name */ @@ -233,6 +242,15 @@ struct vm_cpu_topology { uint16_t maxcpus; }; +struct vm_readwrite_kernemu_device { + int vcpuid; + unsigned access_width : 3; + unsigned _unused : 29; + uint64_t gpa; + uint64_t value; +}; +_Static_assert(sizeof(struct vm_readwrite_kernemu_device) == 24, "ABI"); + enum { /* general routines */ IOCNUM_ABIVERS = 0, @@ -260,6 +278,8 @@ enum { IOCNUM_GET_SEGMENT_DESCRIPTOR = 23, IOCNUM_SET_REGISTER_SET = 24, IOCNUM_GET_REGISTER_SET = 25, + IOCNUM_GET_KERNEMU_DEV = 26, + IOCNUM_SET_KERNEMU_DEV = 27, /* interrupt injection */ IOCNUM_GET_INTINFO = 28, @@ -312,6 +332,11 @@ enum { IOCNUM_RTC_WRITE = 101, IOCNUM_RTC_SETTIME = 102, IOCNUM_RTC_GETTIME = 103, + + /* checkpoint */ + IOCNUM_SNAPSHOT_REQ = 113, + + IOCNUM_RESTORE_TIME = 115 }; #define VM_RUN \ @@ -320,8 +345,12 @@ enum { _IOW('v', IOCNUM_SUSPEND, struct vm_suspend) #define VM_REINIT \ _IO('v', IOCNUM_REINIT) +#define VM_ALLOC_MEMSEG_FBSD12 \ + _IOW('v', IOCNUM_ALLOC_MEMSEG, struct vm_memseg_fbsd12) #define VM_ALLOC_MEMSEG \ _IOW('v', IOCNUM_ALLOC_MEMSEG, struct vm_memseg) +#define VM_GET_MEMSEG_FBSD12 \ + _IOWR('v', IOCNUM_GET_MEMSEG, struct vm_memseg_fbsd12) #define VM_GET_MEMSEG \ _IOWR('v', IOCNUM_GET_MEMSEG, struct vm_memseg) #define VM_MMAP_MEMSEG \ @@ -340,6 +369,12 @@ enum { _IOW('v', IOCNUM_SET_REGISTER_SET, struct vm_register_set) #define VM_GET_REGISTER_SET \ _IOWR('v', IOCNUM_GET_REGISTER_SET, struct vm_register_set) +#define VM_SET_KERNEMU_DEV \ + _IOW('v', IOCNUM_SET_KERNEMU_DEV, \ + struct vm_readwrite_kernemu_device) +#define VM_GET_KERNEMU_DEV \ + _IOWR('v', IOCNUM_GET_KERNEMU_DEV, \ + struct vm_readwrite_kernemu_device) #define VM_INJECT_EXCEPTION \ _IOW('v', IOCNUM_INJECT_EXCEPTION, struct vm_exception) #define VM_LAPIC_IRQ \ @@ -422,4 +457,8 @@ enum { _IOR('v', IOCNUM_RTC_GETTIME, struct vm_rtc_time) #define VM_RESTART_INSTRUCTION \ _IOW('v', IOCNUM_RESTART_INSTRUCTION, int) +#define VM_SNAPSHOT_REQ \ + _IOWR('v', IOCNUM_SNAPSHOT_REQ, struct vm_snapshot_meta) +#define VM_RESTORE_TIME \ + _IOWR('v', IOCNUM_RESTORE_TIME, int) #endif Copied: projects/nfs-over-tls/sys/amd64/include/vmm_snapshot.h (from r361415, head/sys/amd64/include/vmm_snapshot.h) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ projects/nfs-over-tls/sys/amd64/include/vmm_snapshot.h Sat May 23 15:49:07 2020 (r361416, copy of r361415, head/sys/amd64/include/vmm_snapshot.h) @@ -0,0 +1,156 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause-FreeBSD + * + * Copyright (c) 2016 Flavius Anton + * Copyright (c) 2016 Mihai Tiganus + * Copyright (c) 2016-2019 Mihai Carabas + * Copyright (c) 2017-2019 Darius Mihai + * Copyright (c) 2017-2019 Elena Mihailescu + * Copyright (c) 2018-2019 Sergiu Weisz + * All rights reserved. + * The bhyve-snapshot feature was developed under sponsorships + * from Matthew Grooms. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY NETAPP, INC ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL NETAPP, INC OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _VMM_SNAPSHOT_ +#define _VMM_SNAPSHOT_ + +#include +#include +#ifndef _KERNEL +#include +#endif + +struct vmctx; + +enum snapshot_req { + STRUCT_VMX, + STRUCT_VIOAPIC, + STRUCT_VM, + STRUCT_VLAPIC, + VM_MEM, + STRUCT_VHPET, + STRUCT_VMCX, + STRUCT_VATPIC, + STRUCT_VATPIT, + STRUCT_VPMTMR, + STRUCT_VRTC, +}; + +struct vm_snapshot_buffer { + /* + * R/O for device-specific functions; + * written by generic snapshot functions. + */ + uint8_t *const buf_start; + const size_t buf_size; + + /* + * R/W for device-specific functions used to keep track of buffer + * current position and remaining size. + */ + uint8_t *buf; + size_t buf_rem; + + /* + * Length of the snapshot is either determined as (buf_size - buf_rem) + * or (buf - buf_start) -- the second variation returns a signed value + * so it may not be appropriate. + * + * Use vm_get_snapshot_size(meta). + */ +}; + +enum vm_snapshot_op { + VM_SNAPSHOT_SAVE, + VM_SNAPSHOT_RESTORE, +}; + +struct vm_snapshot_meta { + struct vmctx *ctx; + void *dev_data; + const char *dev_name; /* identify userspace devices */ + enum snapshot_req dev_req; /* identify kernel structs */ + + struct vm_snapshot_buffer buffer; + + enum vm_snapshot_op op; +}; + + +void vm_snapshot_buf_err(const char *bufname, const enum vm_snapshot_op op); +int vm_snapshot_buf(volatile void *data, size_t data_size, + struct vm_snapshot_meta *meta); +size_t vm_get_snapshot_size(struct vm_snapshot_meta *meta); +int vm_snapshot_guest2host_addr(void **addrp, size_t len, bool restore_null, + struct vm_snapshot_meta *meta); +int vm_snapshot_buf_cmp(volatile void *data, size_t data_size, + struct vm_snapshot_meta *meta); + +#define SNAPSHOT_BUF_OR_LEAVE(DATA, LEN, META, RES, LABEL) \ +do { \ + (RES) = vm_snapshot_buf((DATA), (LEN), (META)); \ + if ((RES) != 0) { \ + vm_snapshot_buf_err(#DATA, (META)->op); \ + goto LABEL; \ + } \ +} while (0) + +#define SNAPSHOT_VAR_OR_LEAVE(DATA, META, RES, LABEL) \ + SNAPSHOT_BUF_OR_LEAVE(&(DATA), sizeof(DATA), (META), (RES), LABEL) + +/* + * Address variables are pointers to guest memory. + * + * When RNULL != 0, do not enforce invalid address checks; instead, make the + * pointer NULL at restore time. + */ +#define SNAPSHOT_GUEST2HOST_ADDR_OR_LEAVE(ADDR, LEN, RNULL, META, RES, LABEL) \ +do { \ + (RES) = vm_snapshot_guest2host_addr((void **)&(ADDR), (LEN), (RNULL), \ + (META)); \ + if ((RES) != 0) { \ + if ((RES) == EFAULT) \ + fprintf(stderr, "%s: invalid address: %s\r\n", \ + __func__, #ADDR); \ + goto LABEL; \ + } \ +} while (0) + +/* compare the value in the meta buffer with the data */ +#define SNAPSHOT_BUF_CMP_OR_LEAVE(DATA, LEN, META, RES, LABEL) \ +do { \ + (RES) = vm_snapshot_buf_cmp((DATA), (LEN), (META)); \ + if ((RES) != 0) { \ + vm_snapshot_buf_err(#DATA, (META)->op); \ + goto LABEL; \ + } \ +} while (0) + +#define SNAPSHOT_VAR_CMP_OR_LEAVE(DATA, META, RES, LABEL) \ + SNAPSHOT_BUF_CMP_OR_LEAVE(&(DATA), sizeof(DATA), (META), (RES), LABEL) + +#endif Modified: projects/nfs-over-tls/sys/amd64/vmm/amd/svm.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/vmm/amd/svm.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/vmm/amd/svm.c Sat May 23 15:49:07 2020 (r361416) @@ -29,6 +29,8 @@ #include __FBSDID("$FreeBSD$"); +#include "opt_bhyve_snapshot.h" + #include #include #include @@ -50,6 +52,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include "vmm_lapic.h" #include "vmm_stat.h" @@ -276,6 +279,25 @@ svm_restore(void) svm_enable(NULL); } +#ifdef BHYVE_SNAPSHOT +int +svm_set_tsc_offset(struct svm_softc *sc, int vcpu, uint64_t offset) +{ + int error; + struct vmcb_ctrl *ctrl; + + ctrl = svm_get_vmcb_ctrl(sc, vcpu); + ctrl->tsc_offset = offset; + + svm_set_dirty(sc, vcpu, VMCB_CACHE_I); + VCPU_CTR1(sc->vm, vcpu, "tsc offset changed to %#lx", offset); + + error = vm_set_tsc_offset(sc->vm, vcpu, offset); + + return (error); +} +#endif + /* Pentium compatible MSRs */ #define MSR_PENTIUM_START 0 #define MSR_PENTIUM_END 0x1FFF @@ -2203,7 +2225,37 @@ svm_setreg(void *arg, int vcpu, int ident, uint64_t va return (EINVAL); } +#ifdef BHYVE_SNAPSHOT static int +svm_snapshot_reg(void *arg, int vcpu, int ident, + struct vm_snapshot_meta *meta) +{ + int ret; + uint64_t val; + + if (meta->op == VM_SNAPSHOT_SAVE) { + ret = svm_getreg(arg, vcpu, ident, &val); + if (ret != 0) + goto done; + + SNAPSHOT_VAR_OR_LEAVE(val, meta, ret, done); + } else if (meta->op == VM_SNAPSHOT_RESTORE) { + SNAPSHOT_VAR_OR_LEAVE(val, meta, ret, done); + + ret = svm_setreg(arg, vcpu, ident, val); + if (ret != 0) + goto done; + } else { + ret = EINVAL; + goto done; + } + +done: + return (ret); +} +#endif + +static int svm_setcap(void *arg, int vcpu, int type, int val) { struct svm_softc *sc; @@ -2285,6 +2337,306 @@ svm_vlapic_cleanup(void *arg, struct vlapic *vlapic) free(vlapic, M_SVM_VLAPIC); } +#ifdef BHYVE_SNAPSHOT +static int +svm_snapshot_vmi(void *arg, struct vm_snapshot_meta *meta) +{ + /* struct svm_softc is AMD's representation for SVM softc */ + struct svm_softc *sc; + struct svm_vcpu *vcpu; + struct vmcb *vmcb; + uint64_t val; + int i; + int ret; + + sc = arg; + + KASSERT(sc != NULL, ("%s: arg was NULL", __func__)); + + SNAPSHOT_VAR_OR_LEAVE(sc->nptp, meta, ret, done); + + for (i = 0; i < VM_MAXCPU; i++) { + vcpu = &sc->vcpu[i]; + vmcb = &vcpu->vmcb; + + /* VMCB fields for virtual cpu i */ + SNAPSHOT_VAR_OR_LEAVE(vmcb->ctrl.v_tpr, meta, ret, done); + val = vmcb->ctrl.v_tpr; + SNAPSHOT_VAR_OR_LEAVE(val, meta, ret, done); + vmcb->ctrl.v_tpr = val; + + SNAPSHOT_VAR_OR_LEAVE(vmcb->ctrl.asid, meta, ret, done); + val = vmcb->ctrl.np_enable; + SNAPSHOT_VAR_OR_LEAVE(val, meta, ret, done); + vmcb->ctrl.np_enable = val; + + val = vmcb->ctrl.intr_shadow; + SNAPSHOT_VAR_OR_LEAVE(val, meta, ret, done); + vmcb->ctrl.intr_shadow = val; + SNAPSHOT_VAR_OR_LEAVE(vmcb->ctrl.tlb_ctrl, meta, ret, done); + + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad1, + sizeof(vmcb->state.pad1), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cpl, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad2, + sizeof(vmcb->state.pad2), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.efer, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad3, + sizeof(vmcb->state.pad3), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cr4, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cr3, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cr0, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.dr7, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.dr6, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.rflags, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.rip, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad4, + sizeof(vmcb->state.pad4), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.rsp, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad5, + sizeof(vmcb->state.pad5), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.rax, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.star, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.lstar, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cstar, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.sfmask, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.kernelgsbase, + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.sysenter_cs, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.sysenter_esp, + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.sysenter_eip, + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.cr2, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad6, + sizeof(vmcb->state.pad6), + meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.g_pat, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.dbgctl, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.br_from, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.br_to, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.int_from, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vmcb->state.int_to, meta, ret, done); + SNAPSHOT_BUF_OR_LEAVE(vmcb->state.pad7, + sizeof(vmcb->state.pad7), + meta, ret, done); + + /* Snapshot swctx for virtual cpu i */ + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rbp, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rbx, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rcx, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rdx, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rdi, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_rsi, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r8, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r9, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r10, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r11, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r12, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r13, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r14, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_r15, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_dr0, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_dr1, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_dr2, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.sctx_dr3, meta, ret, done); + + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr0, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr1, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr2, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr3, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr6, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_dr7, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->swctx.host_debugctl, meta, ret, + done); + + /* Restore other svm_vcpu struct fields */ + + /* Restore NEXTRIP field */ + SNAPSHOT_VAR_OR_LEAVE(vcpu->nextrip, meta, ret, done); + + /* Restore lastcpu field */ + SNAPSHOT_VAR_OR_LEAVE(vcpu->lastcpu, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->dirty, meta, ret, done); + + /* Restore EPTGEN field - EPT is Extended Page Tabel */ + SNAPSHOT_VAR_OR_LEAVE(vcpu->eptgen, meta, ret, done); + + SNAPSHOT_VAR_OR_LEAVE(vcpu->asid.gen, meta, ret, done); + SNAPSHOT_VAR_OR_LEAVE(vcpu->asid.num, meta, ret, done); + + /* Set all caches dirty */ + if (meta->op == VM_SNAPSHOT_RESTORE) { + svm_set_dirty(sc, i, VMCB_CACHE_ASID); + svm_set_dirty(sc, i, VMCB_CACHE_IOPM); + svm_set_dirty(sc, i, VMCB_CACHE_I); + svm_set_dirty(sc, i, VMCB_CACHE_TPR); + svm_set_dirty(sc, i, VMCB_CACHE_CR2); + svm_set_dirty(sc, i, VMCB_CACHE_CR); + svm_set_dirty(sc, i, VMCB_CACHE_DT); + svm_set_dirty(sc, i, VMCB_CACHE_SEG); + svm_set_dirty(sc, i, VMCB_CACHE_NP); + } + } + + if (meta->op == VM_SNAPSHOT_RESTORE) + flush_by_asid(); + +done: + return (ret); +} + +static int +svm_snapshot_vmcx(void *arg, struct vm_snapshot_meta *meta, int vcpu) +{ + struct vmcb *vmcb; + struct svm_softc *sc; + int err, running, hostcpu; + + sc = (struct svm_softc *)arg; + err = 0; + + KASSERT(arg != NULL, ("%s: arg was NULL", __func__)); + vmcb = svm_get_vmcb(sc, vcpu); + + running = vcpu_is_running(sc->vm, vcpu, &hostcpu); + if (running && hostcpu !=curcpu) { + printf("%s: %s%d is running", __func__, vm_name(sc->vm), vcpu); + return (EINVAL); + } + + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_CR0, meta); + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_CR2, meta); + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_CR3, meta); + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_CR4, meta); + + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_DR7, meta); + + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_RAX, meta); + + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_RSP, meta); + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_RIP, meta); + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_RFLAGS, meta); + + /* Guest segments */ + /* ES */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_ES, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_ES, meta); + + /* CS */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_CS, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_CS, meta); + + /* SS */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_SS, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_SS, meta); + + /* DS */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_DS, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_DS, meta); + + /* FS */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_FS, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_FS, meta); + + /* GS */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_GS, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_GS, meta); + + /* TR */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_TR, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_TR, meta); + + /* LDTR */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_LDTR, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_LDTR, meta); + + /* EFER */ + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_EFER, meta); + + /* IDTR and GDTR */ + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_IDTR, meta); + err += vmcb_snapshot_desc(sc, vcpu, VM_REG_GUEST_GDTR, meta); + + /* Specific AMD registers */ + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_SYSENTER_CS, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_SYSENTER_ESP, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_SYSENTER_EIP, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_NPT_BASE, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_CR_INTERCEPT, 4), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_DR_INTERCEPT, 4), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_EXC_INTERCEPT, 4), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_INST1_INTERCEPT, 4), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_INST2_INTERCEPT, 4), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_TLB_CTRL, 4), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_EXITINFO1, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_EXITINFO2, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_EXITINTINFO, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_VIRQ, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_GUEST_PAT, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_AVIC_BAR, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_AVIC_PAGE, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_AVIC_LT, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_AVIC_PT, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_IO_PERM, 8), meta); + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_MSR_PERM, 8), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_ASID, 4), meta); + + err += vmcb_snapshot_any(sc, vcpu, + VMCB_ACCESS(VMCB_OFF_EXIT_REASON, 8), meta); + + err += svm_snapshot_reg(sc, vcpu, VM_REG_GUEST_INTR_SHADOW, meta); + + return (err); +} + +static int +svm_restore_tsc(void *arg, int vcpu, uint64_t offset) +{ + int err; + + err = svm_set_tsc_offset(arg, vcpu, offset); + + return (err); +} +#endif + struct vmm_ops vmm_ops_amd = { .init = svm_init, .cleanup = svm_cleanup, @@ -2302,4 +2654,9 @@ struct vmm_ops vmm_ops_amd = { .vmspace_free = svm_npt_free, .vlapic_init = svm_vlapic_init, .vlapic_cleanup = svm_vlapic_cleanup, +#ifdef BHYVE_SNAPSHOT + .vmsnapshot = svm_snapshot_vmi, + .vmcx_snapshot = svm_snapshot_vmcx, + .vm_restore_tsc = svm_restore_tsc, +#endif }; Modified: projects/nfs-over-tls/sys/amd64/vmm/amd/svm.h ============================================================================== --- projects/nfs-over-tls/sys/amd64/vmm/amd/svm.h Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/vmm/amd/svm.h Sat May 23 15:49:07 2020 (r361416) @@ -32,6 +32,7 @@ #define _SVM_H_ struct pcpu; +struct svm_softc; /* * Guest register state that is saved outside the VMCB. @@ -66,5 +67,8 @@ struct svm_regctx { }; void svm_launch(uint64_t pa, struct svm_regctx *gctx, struct pcpu *pcpu); +#ifdef BHYVE_SNAPSHOT +int svm_set_tsc_offset(struct svm_softc *sc, int vcpu, uint64_t offset); +#endif #endif /* _SVM_H_ */ Modified: projects/nfs-over-tls/sys/amd64/vmm/amd/svm_msr.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/vmm/amd/svm_msr.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/vmm/amd/svm_msr.c Sat May 23 15:49:07 2020 (r361416) @@ -29,6 +29,8 @@ #include __FBSDID("$FreeBSD$"); +#include "opt_bhyve_snapshot.h" + #include #include #include @@ -162,6 +164,11 @@ svm_wrmsr(struct svm_softc *sc, int vcpu, u_int num, u * Ignore writes to microcode update register. */ break; +#ifdef BHYVE_SNAPSHOT + case MSR_TSC: + error = svm_set_tsc_offset(sc, vcpu, val - rdtsc()); + break; +#endif case MSR_EXTFEATURES: break; default: Modified: projects/nfs-over-tls/sys/amd64/vmm/amd/vmcb.c ============================================================================== --- projects/nfs-over-tls/sys/amd64/vmm/amd/vmcb.c Sat May 23 12:15:47 2020 (r361415) +++ projects/nfs-over-tls/sys/amd64/vmm/amd/vmcb.c Sat May 23 15:49:07 2020 (r361416) @@ -29,12 +29,15 @@ #include __FBSDID("$FreeBSD$"); +#include "opt_bhyve_snapshot.h" + #include #include #include #include #include +#include #include "vmm_ktr.h" @@ -452,3 +455,106 @@ vmcb_getdesc(void *arg, int vcpu, int reg, struct seg_ return (0); } + +#ifdef BHYVE_SNAPSHOT +int +vmcb_getany(struct svm_softc *sc, int vcpu, int ident, uint64_t *val) +{ + int error = 0; + + if (vcpu < 0 || vcpu >= VM_MAXCPU) { + error = EINVAL; + goto err; + } *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-projects@freebsd.org Sat May 23 16:42:27 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AC5182CF7CF for ; Sat, 23 May 2020 16:42:27 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Tq034Cjdz4HMl; Sat, 23 May 2020 16:42:27 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8BF8E12E07; Sat, 23 May 2020 16:42:27 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04NGgRWq002884; Sat, 23 May 2020 16:42:27 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04NGgRcJ002883; Sat, 23 May 2020 16:42:27 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005231642.04NGgRcJ002883@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 23 May 2020 16:42:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361417 - projects/nfs-over-tls/sys/fs/nfsserver X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/nfs-over-tls/sys/fs/nfsserver X-SVN-Commit-Revision: 361417 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2020 16:42:27 -0000 Author: rmacklem Date: Sat May 23 16:42:27 2020 New Revision: 361417 URL: https://svnweb.freebsd.org/changeset/base/361417 Log: Fix one case of an ext_pgs mbuf change that I missed. Modified: projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdport.c Modified: projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdport.c ============================================================================== --- projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdport.c Sat May 23 15:49:07 2020 (r361416) +++ projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdport.c Sat May 23 16:42:27 2020 (r361417) @@ -863,7 +863,7 @@ nfsrv_createiovec_extpgs(int len, int maxextsiz, struc m = mb_alloc_ext_plus_pages(siz, M_WAITOK, mb_free_mext_pgs); left -= siz; - i += m->m_ext_pgs.npgs; + i += m->m_epg_npgs; if (m3 != NULL) m2->m_next = m; else From owner-svn-src-projects@freebsd.org Sat May 23 23:38:37 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1F7172F128E for ; Sat, 23 May 2020 23:38:37 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49V0DF02wNz3cbW; Sat, 23 May 2020 23:38:37 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EC2CA17E81; Sat, 23 May 2020 23:38:36 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 04NNcaMC058665; Sat, 23 May 2020 23:38:36 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 04NNcaKw058662; Sat, 23 May 2020 23:38:36 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202005232338.04NNcaKw058662@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 23 May 2020 23:38:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r361428 - in projects/nfs-over-tls/sys: fs/nfsserver kern sys X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/nfs-over-tls/sys: fs/nfsserver kern sys X-SVN-Commit-Revision: 361428 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2020 23:38:37 -0000 Author: rmacklem Date: Sat May 23 23:38:35 2020 New Revision: 361428 URL: https://svnweb.freebsd.org/changeset/base/361428 Log: Fix a few problems caused during the code merge of the up-to-date head kernel. I think it now builds, but will it work? Modified: projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdcache.c projects/nfs-over-tls/sys/kern/uipc_ktls.c projects/nfs-over-tls/sys/kern/uipc_sockbuf.c projects/nfs-over-tls/sys/sys/socket.h Modified: projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdcache.c ============================================================================== --- projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdcache.c Sat May 23 23:10:03 2020 (r361427) +++ projects/nfs-over-tls/sys/fs/nfsserver/nfs_nfsdcache.c Sat May 23 23:38:35 2020 (r361428) @@ -1021,22 +1021,8 @@ nfsrc_getlenandcksum(struct mbuf *m1, u_int16_t *cksum len += m->m_len; m = m->m_next; } - /* - * in_cksum() doesn't work for ext_pgs mbufs, so just return a - * random checksum to avoid a false hit. - * Since NFSv4.1 and NFSv4.2 does not actually use - * the DRC, due to sessions, I think this should be ok. - * Also, most NFS over TCP implementations do not implement - * a DRC at all. Unfortunately, the DRC is used for NFSv4.0 - * for the cases where there are sequenced operations, such as - * file lock operations, so it must still be enabled for NFSv4.0. - */ - if ((m1->m_flags & M_NOMAP) == 0) { - cklen = (len > NFSRVCACHE_CHECKLEN) ? NFSRVCACHE_CHECKLEN : - len; - *cksum = in_cksum(m1, cklen); - } else - *cksum = arc4random(); + cklen = (len > NFSRVCACHE_CHECKLEN) ? NFSRVCACHE_CHECKLEN : len; + *cksum = in_cksum(m1, cklen); return (len); } Modified: projects/nfs-over-tls/sys/kern/uipc_ktls.c ============================================================================== --- projects/nfs-over-tls/sys/kern/uipc_ktls.c Sat May 23 23:10:03 2020 (r361427) +++ projects/nfs-over-tls/sys/kern/uipc_ktls.c Sat May 23 23:38:35 2020 (r361428) @@ -2070,8 +2070,7 @@ ktls_work_thread(void *ctx) { struct ktls_wq *wq = ctx; struct socket *so, *son; - struct ktls_session *tls; - struct mbuf *m; + struct mbuf *m, *n; STAILQ_HEAD(, mbuf) local_head; STAILQ_HEAD(, socket) local_so_head; @@ -2093,9 +2092,6 @@ ktls_work_thread(void *ctx) STAILQ_CONCAT(&local_so_head, &wq->so_head); mtx_unlock(&wq->mtx); - STAILQ_FOREACH_SAFE(p, &local_head, stailq, n) { - if (p->mbuf != NULL) { - ktls_encrypt(p); STAILQ_FOREACH_SAFE(m, &local_head, m_epg_stailq, n) { if (m->m_epg_flags & EPG_FLAG_2FREE) { ktls_free(m->m_epg_tls); Modified: projects/nfs-over-tls/sys/kern/uipc_sockbuf.c ============================================================================== --- projects/nfs-over-tls/sys/kern/uipc_sockbuf.c Sat May 23 23:10:03 2020 (r361427) +++ projects/nfs-over-tls/sys/kern/uipc_sockbuf.c Sat May 23 23:38:35 2020 (r361428) @@ -1373,7 +1373,7 @@ sbcompress_ktls_rx(struct sockbuf *sb, struct mbuf *m, ("TLS RX mbuf %p is not MT_DATA", m)); KASSERT((m->m_flags & M_NOTREADY) != 0, ("TLS RX mbuf %p ready", m)); - KASSERT((m->m_flags & M_NOMAP) == 0, + KASSERT((m->m_flags & M_EXTPG) == 0, ("TLS RX mbuf %p unmapped", m)); if (m->m_len == 0) { @@ -1388,7 +1388,7 @@ sbcompress_ktls_rx(struct sockbuf *sb, struct mbuf *m, if (n && M_WRITABLE(n) && ((sb->sb_flags & SB_NOCOALESCE) == 0) && - !(n->m_flags & (M_NOMAP)) && + !(n->m_flags & (M_EXTPG)) && m->m_len <= MCLBYTES / 4 && /* XXX: Don't copy too much */ m->m_len <= M_TRAILINGSPACE(n)) { m_copydata(m, 0, m->m_len, mtodo(n, n->m_len)); Modified: projects/nfs-over-tls/sys/sys/socket.h ============================================================================== --- projects/nfs-over-tls/sys/sys/socket.h Sat May 23 23:10:03 2020 (r361427) +++ projects/nfs-over-tls/sys/sys/socket.h Sat May 23 23:38:35 2020 (r361428) @@ -471,6 +471,7 @@ struct msghdr { #endif #ifdef _KERNEL #define MSG_MORETOCOME 0x00100000 /* additional data pending */ +#define MSG_TLSAPPDATA 0x00200000 /* only soreceive() app. data (TLS) */ #endif /*