From owner-svn-src-projects@freebsd.org Wed Jul 1 22:47:31 2020 Return-Path: Delivered-To: svn-src-projects@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8512335DB3E for ; Wed, 1 Jul 2020 22:47:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-to1can01on0630.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5d::630]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49xxFG2JBWz3fPR; Wed, 1 Jul 2020 22:47:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n6b9R8ofTUaaGXoHHCu7TumjjrbRoc+qcnE4OirDrW2OmPB/ewL9pbxzirYPp6y/Pl+QDzruCo/oYEKypk1MHRl+w2ycng0TsWaeLVQtQNsg9wwTPF+jq9zkg7hkWKuMLpc6gwvCa0bE752Dw+Cq+5YCqkGX1y8gmZYeI3ekOW01TzC0te8EESLGBTPn7hfjOTdIP/raEPNb0I+6dCi5UV2TZPe1KNCXtJsku7rZYy2K+hBrdZQu/6F+kAQ0f3RvTnsCM9vVU2GGN1VUcv6t/EVZaayYhknbg/hrDcOWD7FJ5Ra8fySajKCja9P2ugVOBc3QX9XgBh2Uw8tV3NLEkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2561Kt0zqN4iBHrO6VEcWZo21ogOSSvow2fiXr6FD9M=; b=W/Y11QHKgf7inllUPGPqPIIbvVXDgqnLgmH9DRhN3dXFKS6VodCA0sSNjupGM5cZEXsXPhP0dz1ybKj5raZ5u5Fue25uF4aZKpKzw6Go7ayzZknEQJ183kzlQW8Mrvkcx86/ZMacruVRE2vtx+FTZ/ahTa8wImGgntik49i4qxk9rl722HAMqQQ4H0SKzqCy/tHBSz8c8Iy5D7mwX1PCBZuwf4/hZNwWnSCokKJGMA6pzO3AE4dq4ADwL+ObUMyW7DjctQD1hKKbpx7gtjSxJ1iM246bXRS+fkeK+9M+2XY9WNQRoAf8cB9PSfeVbV4w+GhdFe87tXapw5XaAo0d9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2561Kt0zqN4iBHrO6VEcWZo21ogOSSvow2fiXr6FD9M=; b=lrVADFL41Uvp4uiFluxPuTezA5xuztv0Rsv0R4l6BEjwnzqGKnm6NuLXjhP5xbdwQuMhW/qSzldNVMCweA+MI9ssAX1z1xSApYEAYJPFybP7m1JOXoiG74eT6A3sCqZKzBQM+xw+2g5G/p1mWsSuRM3isV2+Qtplg8nqVb5namzcDd6BhuJ/6HnjF9RcXI9I6cfA/66ZsKRjJhU6hP9TJB/HUC5Pvgo1vmuE3DRH9SDpQOrWmMeCa7547jee/oKgod/xRPOAcnYcjAQolggBmoisbj6TG4cu+wlkn2hdRcCSToqPSgmXnY0It+CsSeIhgr0YdebsKsUL2JxZ9mPF4g== Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:38::14) by YQBPR0101MB1939.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:f::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.23; Wed, 1 Jul 2020 22:47:27 +0000 Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91]) by QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91%7]) with mapi id 15.20.3131.034; Wed, 1 Jul 2020 22:47:19 +0000 From: Rick Macklem To: Benjamin Kaduk CC: Benjamin Kaduk , Rick Macklem , src-committers , "svn-src-projects@freebsd.org" Subject: Re: svn commit: r362798 - in projects/nfs-over-tls/sys/rpc: . rpcsec_tls Thread-Topic: svn commit: r362798 - in projects/nfs-over-tls/sys/rpc: . rpcsec_tls Thread-Index: AQHWTu29MoFCP0YO4kaq4Sj4aYWIUajxUeOAgAAARmyAAAhnAIAAg0Q8gAANlPuAABMqAIABUrap Date: Wed, 1 Jul 2020 22:47:19 +0000 Message-ID: References: <202006301449.05UEnq2x072917@repo.freebsd.org> <20200630163340.GN58278@kduck.mit.edu> , <20200701022040.GE58278@kduck.mit.edu> In-Reply-To: <20200701022040.GE58278@kduck.mit.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3a9a8693-8ace-48f2-8d2a-08d81e10b5dc x-ms-traffictypediagnostic: YQBPR0101MB1939: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2512; x-forefront-prvs: 04519BA941 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Zgp2R3Gpjp1kOesdg6MKIIwRPXQ3F1wc2CKnZlu5+EEphx/pcgi1MjPoCOVBqMkLFobNhlZuy/XHUp+3TZhXC2+eaXmucJ031B7zk7n+YIaDEjNk03bfRiqeJGCGdqfvzdpccn2bm42GZ+JR7N8qhMFg77XQv2j4BjAaHOPIoC4H0hwqirDTaNlzeSamDmHU1/3HTH+iQye6T/uYcFuzlFi7I0yjYnyPknT3sorMYf8GqQEkWbeiIdYa+AYH6OyWoYJ7hGNhxmO5SIB9MsruXlOXSNYhEz87Vw5iYWNBZinkJ463DlY8JbshLGL8jDaNn4xgV7Mz0K7I5Qy+sD1tFA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(396003)(366004)(136003)(346002)(376002)(39860400002)(83380400001)(5660300002)(478600001)(6506007)(7696005)(66946007)(66446008)(8936002)(64756008)(66476007)(4326008)(33656002)(66556008)(8676002)(76116006)(2906002)(71200400001)(55016002)(186003)(86362001)(786003)(316002)(6916009)(52536014)(54906003)(9686003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: wvopeOYFoC6e4gPYlEJCKQcwRjcSdyDgTNRcDHCX1r+Wr+49JCq1f4bXZFW7LDLlf3rzEPph2ktaysTSD2Z/hXAT+PgsSOgsQDFOXNcuBqq/Oh/MSFQpo8frb2AmbipkYfX4O7/6h+3ImBkGpvatakUzChJ3ZkYHo1Uxo4QaaKYjunZ/EKs1l4QZDvKGmP1ImcQOwm0/98n0jBL1NU4VLDAdZMN+KuHFsoYdNsnvtRoMoeUmGBbRhi67iE6tKyq/ukTRgdWgdLo14pX6KgjuedVF9r1gl0FTRlRM8pJ5jckqpW7hrrj7M9wKwZWo8Mw/lF9n4i0KZvM/peJzTMnSDwI1ePMZ+RgYjKJ/i/3UbheEu8t7Pl5RNsFoQkXbqQmsWxwXX+TvQTi5ZAQV4l+6aH/QCrpJm9HSqA+NourDYd6PwHOUXNh602X9PwIuuOakDccS4OLowTgyQigxvDO2Ue2Ri446tyEVH26uKVW6Mz5Wj0fth4vsdTu1pRdSI2QWNLEdLT7HLU4gIwhyGFCU015ykbZQqhMn4Mlg6Im/QEMGgiE9WZ1jfiugW7n/CAAM x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 3a9a8693-8ace-48f2-8d2a-08d81e10b5dc X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2020 22:47:19.1176 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TTosy5HwhkIH2wc/nKieXvhGOw1xFe6VW/9lf62YhML88y5NeR496T3DWORjKB6m1o0xDvAzdeBWNrXhg1u1mA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1939 X-Rspamd-Queue-Id: 49xxFG2JBWz3fPR X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=lrVADFL4; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5d::630 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.83 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.003]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; NEURAL_HAM_LONG(-1.00)[-1.004]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCPT_COUNT_FIVE(0.00)[5]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.33)[-0.327]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jul 2020 22:47:31 -0000 Benjamin Kaduk wrote:=0A= >On Wed, Jul 01, 2020 at 01:23:50AM +0000, Rick Macklem wrote:=0A= >> Rick Macklem wrote:=0A= >> >Benjamin Kaduk wrote:=0A= >> >>On Tue, Jun 30, 2020 at 04:20:45PM +0000, Rick Macklem wrote:=0A= >> >>> If you happen to know how to set a timeout for SSL_connect() in the = openssl=0A= >> >>> library, I would be interested in hearing that.=0A= >> >>=0A= >> >>As it happens, I took a look before I wrote the initial note, and ther= e=0A= >> >>doesn't seem to be any intrinsic TLS (not DTLS) handshake timeouts in= =0A= >> >>libssl itself; I expect this is actually just the (kernel's!) TCP time= out.=0A= >> >>So you'd be getting the socket fd (e.g., SSL_get_fd(), if you don't ha= ve a=0A= >> >>reference already) and using setsockopt() to set the timeout(s).=0A= >> >Interesting. The test case I simulated did not close the TCP socket use= d by=0A= >> >SSL_connect(). The server just replied to the STARTTLS Null RPC, but di= d not=0A= >> >call SSL_accept(), so the server side just isn't playing "handshake".= =0A= >> >"netstat -a" showed the connection as ESTABLISHED.=0A= >> >During debugging, I also used the trick of putting:=0A= >> > while (1)=0A= >> > sleep(1);=0A= >> >right after the SSL_connect() call and, when watching it via "ps",=0A= >> >it would switch from "sbwait" to "nanoslp" after 6 minutes and=0A= >> >a syslog() call showed that SSL_connect() had returned -1.=0A= >> >=0A= >> >So, if the TCP connection was "established", what caused the SSL_connec= t()=0A= >> >to return with an error (-1) after 6 minutes?=0A= >> >=0A= >> >Now, there is a 6 minute idle timeout in the RPC code for TCP where it,= =0A= >> >by default, closes the connection when there is 6 minutes without any= =0A= >> >activity. (I have to look if waiting for a reply for the upcall implies= "no activity" and >if=0A= >> >this also happens for AF_LOCAL sockets, which is what the upcalls use.)= =0A= >> Ok, I figured out what is happening for this test.=0A= >> It is the 6 minute idle timeout, but it occurs at the server end, where = the NFS server=0A= >> end shuts down the TCP connection.=0A= >=0A= >Ah, that makes sense.=0A= >=0A= >> Now, the client cannot assume all servers will do this.=0A= >=0A= >Right.=0A= >=0A= >> I'm going to try playing around with doing a shutdown of the socket on t= he=0A= >> client end after a shorter timeout on the upcall and see if that can get= =0A= >> SSL_connect() to return with a failure in the daemon.=0A= >>=0A= >> >Now, if that happens, a SIGPIPE would be posted to the daemon, which=0A= >> >is SIG_IGN'd by the daemon. But maybe the SIGPIPE somehow causes=0A= >> >SSL_connect() to return -1 by making the syscall it is doing (read/recv= on the=0A= >> >TCP socket sitting in sbwait) return EINTR, or something like that?=0A= >> Ignore this "theory". It was bunk.=0A= >=0A= >Non-ignored signals would cause SSL_connect() to return, but ignored ones= =0A= >should be wholly ignored, yes.=0A= >=0A= >> >I can change this 6minute timeout to see if that affects it.=0A= >> Can't be changed, since it is at the server end of the TCP connection.= =0A= >=0A= >Can't you set a client-side (e.g., read) timeout, though?=0A= Well, in this case it would be the read (or recv or ??) that is done inside= the=0A= SSL_connect().=0A= =0A= The timer I can control is the one that I had set to 10minutes, which times= out=0A= the upcall RPC to the userland daemon. I had set it to 10minutes so the=0A= SSL_connect() would time out first, but now that I know that won't always h= appen..=0A= This timer is now set to 15sec and after it times out, the kernel code does= a=0A= soshutdown(so, SHUT_RD) in the client, which seems to be sufficient to get= =0A= SSL_connect() to return an error.=0A= =0A= This seems sufficient and works ok for the testing I've done.=0A= =0A= 15sec is pretty arbitrary, but I figure a timeout on the order of seconds i= s=0A= reasonable for RPC upcalls to the local daemon. (I'd guess that taking even= =0A= 1sec to do an upcall would indicate something is broken.)=0A= If others feel 15sec isn't an appropriate timeout, feel free to comment.=0A= (Note that this timeout should only happen when something is broken, like= =0A= the server that does a "STARTTLS" reply but does not do a TLS handshake.)= =0A= =0A= Thanks for the comments, rick=0A= =0A= to return an error.=0A= -Ben=0A= =0A= > (A comment in the krpc code mentions a 5minute timeout in the client,=0A= > but I don't see that in the code?)=0A= >=0A= > >When you've got upcalls and library functions both talking to sockets it= =0A= > >can get interesting.=0A= > >=0A= > >Thanks for the comments, rick=0A= >=0A= > Correcting myself, rick=0A= >=0A= > -Ben=0A= >=0A= =0A=