From owner-freebsd-current@freebsd.org Thu Apr 15 20:47:27 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 77EC35D6B3E for ; Thu, 15 Apr 2021 20:47:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0618.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::618]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLrxp1Qkqz4q9j; Thu, 15 Apr 2021 20:47:25 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IP9gSeaffl20H6x6Vw+1xttf4FvzUThfnoPhbpKuVT9w3GGUqdhOUVBXi5lsDOOyBkVw8SWuJX2eQKbzm9of/FJy6ql8iemDcs5vj/4Ai1FXZeZ0UaU2HvZ1BMMgZnb70BD26ne8AbaZPvH/gvFhMtr9PzVNAACYv4QRVpTgltQDGBy2DBe6CxwzWPMDXbVEHbBfOC4h+3fioxbTrmQIfhUg19mejWR4a/UkuMO7sAZQpen9JasVxwEU7Fr5ri7zb5p3iv6474xeRubQI/eHZu67ucuw/rTl4dpepyPVLDqjIF841No2jJyombRF44wkPjTRfOD6VmrstBNkMYMA7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TaGRxbTGHo/rTBHu+jLenxjXI2nyxZA39imRsbhINIY=; b=TnXdQYnanGnes6KICUTT4AHUCc44cNVgYRvZx01+ECwjZ/CklCb7WaZf721QodWEC4X0BwDlL2Q0TRHkn4YPnDBpts21vWR8RmDMV1DPv825Xhsuar4T+wIKcsl/PrsP+qYJMkwqoOPI6yzEcaVqtrQz5GxYAiP2rRO64ErfXRIxB1GyBn9Uw2ASfq7GoWFGwvN73z/q4BJC9EFQdVJGG4vcKwwG6UCgfQ1PPC37iBx1tad0R3PjGSm3KG5hh2PHQriI61l1rzW1X4zB3hutVT5WE+bmuQhlPl3On/dehwl2OxgbEQtxnT9d2gN5DdoQABXsrXTfp1C/Hd4w6HjjFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TaGRxbTGHo/rTBHu+jLenxjXI2nyxZA39imRsbhINIY=; b=HtIcFNIVjtk+dR64Ad9a26kz4ZxtijmVP+5PfuA3EiQ4IgCRKs7L5CP62TWYK0H+mLTeeX6siF0+h1PNx45A8/EL3UJtLLSAKnxrDZ3NYks6COTnKDjaB4eZbc3F1cpi3pAATh/mTA2G1hfFruTr9IL3PdN/Pvq9CaUF1JQUriOEoDJAEGup8lC/fvAlh7MTT0O5UrUdlmJArhIGY1QpmhVdeAOtrZjoFzPZVVPawZu6Or/Id2ek3XC9TFPsTPI8iCEYzSwdf3Ck+lIruC/PFUkwnGVjrzJybzNIIA0AMW0MhGV2QlOA8Ab8GOno7Y+0U/3S5NX7fne4tZheeUtZBw== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB4388.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:9::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.22; Thu, 15 Apr 2021 20:47:24 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.3999.037; Thu, 15 Apr 2021 20:47:23 +0000 From: Rick Macklem To: Allan Jude , "freebsd-current@freebsd.org" CC: Richard Scheffenegger , Juraj Lutter Subject: Re: NFS issues since upgrading to 13-RELEASE Thread-Topic: NFS issues since upgrading to 13-RELEASE Thread-Index: AQHXMhVsKLCb5bI16E6Ydj5qgkLmjKq16AQAgAAfNj0= Date: Thu, 15 Apr 2021 20:47:23 +0000 Message-ID: References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 38a3bc18-01e3-4c23-cbb5-08d9004fac1c x-ms-traffictypediagnostic: YQBPR0101MB4388: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: brk8jwwO7+lg3bBhJFQNHuKyY+ZYkfc4rUHW4Vcv+xEvEnm9djhr6iOvVIP6ZmCQoGdnOeaZyhPHFPUGGaXJ/N6Up0VMbfPWBfyunhaF6wov7yK369KzPSEGW7bju47/b5MDfOSA0O0YG7SPxEU+DzQlJKO1h056fEGzl7rDspqIXqatZ8Jwfh3mVEVuWUZmbsLw+mtqg0LKdGv+QHAoTPLA66VKOgxLe8O1uB6UCU2bniMXhRX6R/opZnMXLiqGwdklFMt7ZI5Q0FEJeS0Iy/WyN6LvKIHlXGQyJESvG8UdOXCFgdXKo5wtWxf0Dc3MY5YFxkk0esGEc/Y3hzKHezxS34ZHrhcuhZebeaE624EhdWuyFuZQ1WtWH+eyIKPD3UORy88RNnPFqGxpVZ2eqJz7tkXaGfGjyIgHf1wuJLGXZeQtnNWgQAiSPJRMfAyTKnuXzFYuaAsigNnru0NmeKgI/NwMM2DTfjrdwY0IN3Zm5URGKnPOVBb4eSgVbqRp/eKpP8Z3i2dPDlkpgcTLOSCDtF5Lc5fiPuNg/Q8E5Xhjo2pvGLOGtsKhNJxFCAWZRO4jaUUKWeQDlXdRcfLUrQKSU6QIduUY0CPkNNrwjS9YdwtFnMT+wNkt82gldoZ2cBd7OMAUu/Kb5gh/xWLpBvGjxpuKcRJ/NmyFx1D0IpmbVpLIfxmTfbbK54x2XbEn x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(136003)(376002)(346002)(366004)(39860400002)(396003)(91956017)(186003)(71200400001)(76116006)(64756008)(66946007)(966005)(83380400001)(66556008)(2906002)(54906003)(66616009)(66446008)(9686003)(8676002)(4326008)(66476007)(122000001)(99936003)(55016002)(450100002)(316002)(7696005)(6506007)(478600001)(786003)(8936002)(5660300002)(110136005)(86362001)(33656002)(52536014)(38100700002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?stJiAEXZSBTtKCbfIvSnRbzk2LcnIGSnwmc5dVJhLH8/oYnVodjpMjQJmz?= =?iso-8859-1?Q?F0+JU9rJa97Hi0EgCVP7xMS1Ci6zBwSLhsfZThbBHVFQ7qFTeRpjNgdQa/?= =?iso-8859-1?Q?Ce3ez3V9MJoUcxufN26EPbrG8qkqVbqrlRjORuoltOYDrCNJIij3ahO4+i?= =?iso-8859-1?Q?exKjRNacARtMQuJMiH5yGMHO48xZtIkJHsZmmSyaM6tySJr+tdzW6KBM2w?= =?iso-8859-1?Q?1UCwYFG77NxCfi7FwWrHO280Q6L321W+L3iey9/IvRM0MYOgr4f+vjac1z?= =?iso-8859-1?Q?QdjmW+WObrARH/6oX8P6vSTGtT5wvi0oc4jOqwU6lAqdOuRc0DprXJRR1P?= =?iso-8859-1?Q?m++cWeSyhohQ1hkVjfIMoTbLiuNjwAFUeRxCfvFXTvEepjvfPE02WeNHUR?= =?iso-8859-1?Q?InxQ9e2HTcR1Q0loOrTR59miaSRSkQNy22zXPHCQm+xNZvQl5QIbGMEi1p?= =?iso-8859-1?Q?YbtVLDm2DmJlfkkBDRGKzMffXoJn3SWwF1eESlaB1TT4h8VWOe5ynCnIWD?= =?iso-8859-1?Q?uVdYU3V4Anpbn0zm+XYrRBrgNBC2DHQjhXfG3oozUsMme1jVZ3W8mDCkB/?= =?iso-8859-1?Q?DVb/gStNq/96xMDiwmQ7rUaDFk394B19IMApeoWWoEgUpFVs3DhyhvrRD0?= =?iso-8859-1?Q?P+Eid1KvAzPxWp5vKCve3VTGhO+zvBJWlI8Gi2VUowm7uS4oYA5cDWdTd6?= =?iso-8859-1?Q?hrPdIOF3sZB0F1SDD2kER2TlFX0qMqYjxsW2v0LkBgE44mqyL2H8jzkr6Q?= =?iso-8859-1?Q?ZbIWNqnQAgnUJ2WfgQwVws/VdIKZa0GfgANffwYjFW3rI2xkfFhMcnmmY3?= =?iso-8859-1?Q?R8gQqgMf9HIdpVXj5NAM0xFjKFu/045WIITXmCKWgrwAaILl3dwsNKlTEI?= =?iso-8859-1?Q?RuW+6+7M1Zf7trhtqkFqWqw8oJBaqj9EqjNhpcgG8jX7x4YM4P98zHRghG?= =?iso-8859-1?Q?D1/yGKOzw3S7CFYDVhbAL1otTwykoJsGCJjPK788tTnILNup5io7sRIjDX?= =?iso-8859-1?Q?9iwRgFef36EyaIBjQDEDgRVHh7pQ2VzYjn6ScRnaiEO7/H1vEVZDF1ZhuQ?= =?iso-8859-1?Q?vlsmigBl2Dk1nYuacpIeZC/SyYRDV8+UIxBPIv43eWcL7w1p1tz7ZgZmeJ?= =?iso-8859-1?Q?S6t2sE2XEapuqkwSSTisnfGqSALv1t3/J6PW5ehGSdarS5ufYkNJakItSZ?= =?iso-8859-1?Q?EPLstRku1SThFIZSaKwBZ0qSlqJLK/wZuDFmp2c+yt2hFXDpRl6Or4LqcL?= =?iso-8859-1?Q?G5g4nRrAdNf9CIvh1e9VW4xUriYEvS93695sLLyVcZJW0gJYCx0oo2ijn1?= =?iso-8859-1?Q?ATvgQj6YbCgTaZiCeUmk0V5/vQ4x9p4K3lypRN4WXDO16+TyxSX7n0PqE3?= =?iso-8859-1?Q?so0Szr08wqe5XoiCImgKYQkoJB4OEFSQuDObiO7ozGlaTwK/VniH8=3D?= x-ms-exchange-transport-forked: True Content-Type: multipart/mixed; boundary="_002_YQXPR0101MB09681707D3F3DC10814A905BDD4D9YQXPR0101MB0968_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 38a3bc18-01e3-4c23-cbb5-08d9004fac1c X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Apr 2021 20:47:23.8721 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: bKEvRoSP+DhMFHOoyP/R++XspDLwA4mbPqX90p/n9A6ZcW4KZQ2Ph1RM9gair6t0eEsSOU97e66l8SfutL/PTA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB4388 X-Rspamd-Queue-Id: 4FLrxp1Qkqz4q9j X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=HtIcFNIV; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::618 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5c::618:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; HAS_ATTACHMENT(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5c::618:from:127.0.2.255]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 20:47:27 -0000 --_002_YQXPR0101MB09681707D3F3DC10814A905BDD4D9YQXPR0101MB0968_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Allan Jude wrote:=0A= >On 4/15/2021 9:22 AM, Chris Roose wrote:=0A= >> I posted this in -questions and someone suggested I post here as well.= =0A= >>=0A= >> I'm having NFS availability issues between my Proxmox client and FreeBSD= server (10G link) since upgrading to 13->RELEASE. And unfortunately I upgr= aded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind of stuck= .=0A= >>=0A= >> Periodically, the NFS server (I've tried both v3 and v4.2 clients) will = go unresponsive for several minutes. I never had >this problem on 12.2, and= as far as I can tell it's not a disk or network I/O issue. I'll get severa= l "nfs: server not >responding, still trying" messages on the client and a = few minutes later it usually recovers. It's not clear to me yet >what's cau= sing the block. Restarting nfsd on the server will resolve the issue if it = doesn't clear itself.=0A= >=0A= otis@ has run into a problem that sounds similar.=0A= He sees a growing Recv-Q size on the server for the TCP connection from the= client=0A= when "netstat -a" is done on the server when the "hang" occurs.=0A= In his case, he is using a Linux client and it does not recover, however ot= her client=0A= mounts continue to function.=0A= I suspect the recovery after a few minutes is the client establishing a new= TCP=0A= connection.=0A= =0A= He has been running for almost a week with r367492 reverted and has not rep= orted=0A= seeing the problem again (he had reported that it has taken up to a week to= recur, so=0A= reverting r367492 *might* have fixed the problem and I'd guess we'll know i= n another=0A= week?).=0A= =0A= - If using svn to revert the patch is inconvenient, I've attached a patch t= hat can be applied=0A= to revert it.=0A= - Alternately you can try rscheff@'s alternate proposed patch that is at=0A= https://reviews.freebsd.og/D29690.=0A= I have not yet had time to test this one, but since I cannot reproduce th= e hang, I can=0A= only do testing of it to see that it is "no worse" than reverting r367492= for my=0A= setup.=0A= =0A= Please let us know which you choose and whether or not it fixes your proble= m.=0A= =0A= >> Any pointers for troubleshooting this? I've been looking through vmstat,= gstat, top, etc. when the problem occurs, but I haven't been able to pinpo= int the issue. I can get pcap, but it would be from the hosts, because I do= n't have a 10G tap or managed switch.=0A= >>=0A= >=0A= >run `nfsstat -d 1` and try to capture a few lines from before, during,=0A= >and after the stall, and that may provide some insight.=0A= >=0A= >Specifically, does the queue length grow, suggesting it is waiting on=0A= >the I/O subsystem, or does it just stop getting traffic all together.=0A= =0A= If the revert of r367492 does not fix the problem, monitor the TCP connecti= on(s)=0A= via "netstat -a" and, if possible, capture packets via=0A= tcpdump -s 0 -w hang.pcap host =0A= or similar, run on the server.=0A= =0A= Ideally the tcpdump would be started before the "hang" occurs, but running= =0A= one while the hang is occurring (until after it recovers) could also be use= ful.=0A= =0A= Thanks for reporting this, rick=0A= =0A= --=0A= Allan Jude=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A= --_002_YQXPR0101MB09681707D3F3DC10814A905BDD4D9YQXPR0101MB0968_ Content-Type: application/octet-stream; name="r367492-revert.patch" Content-Description: r367492-revert.patch Content-Disposition: attachment; filename="r367492-revert.patch"; size=8952; creation-date="Thu, 15 Apr 2021 20:47:13 GMT"; modification-date="Thu, 15 Apr 2021 20:47:13 GMT" Content-Transfer-Encoding: base64 LS0tIHN5cy9uZXRpbmV0L3RjcF9zdGFja3MvYmJyLmMub3JpZwkyMDIxLTA0LTA5IDA4OjI0OjA1 LjM2NDIxMjAwMCAtMDcwMAorKysgc3lzL25ldGluZXQvdGNwX3N0YWNrcy9iYnIuYwkyMDIxLTA0 LTA5IDA4OjMzOjQ5Ljc5OTkwMjAwMCAtMDcwMApAQCAtNzg3Niw4ICs3ODc2LDcgQEAgYmJyX3By b2Nlc3NfYWNrKHN0cnVjdCBtYnVmICptLCBzdHJ1Y3QgdGNwaGRyICp0aCwgc3RyCiAJYWNrZWRf YW1vdW50ID0gbWluKGFja2VkLCAoaW50KXNiYXZhaWwoJnNvLT5zb19zbmQpKTsKIAl0cC0+c25k X3duZCAtPSBhY2tlZF9hbW91bnQ7CiAJbWZyZWUgPSBzYmN1dF9sb2NrZWQoJnNvLT5zb19zbmQs IGFja2VkX2Ftb3VudCk7Ci0JU09DS0JVRl9VTkxPQ0soJnNvLT5zb19zbmQpOwotCXRwLT50X2Zs YWdzIHw9IFRGX1dBS0VTT1c7CisJc293d2FrZXVwX2xvY2tlZChzbyk7CiAJbV9mcmVlbShtZnJl ZSk7CiAJaWYgKFNFUV9HVCh0aC0+dGhfYWNrLCB0cC0+c25kX3VuYSkpIHsKIAkJYmJyX2NvbGxh cHNlX3J0dCh0cCwgYmJyLCBUQ1BfUkVYTVRWQUwodHApKTsKQEAgLTgzNTMsOCArODM1Miw3IEBA IGJicl9wcm9jZXNzX2RhdGEoc3RydWN0IG1idWYgKm0sIHN0cnVjdCB0Y3BoZHIgKnRoLCBzdAog CQkJCWFwcGVuZGVkID0KICNlbmRpZgogCQkJCQlzYmFwcGVuZHN0cmVhbV9sb2NrZWQoJnNvLT5z b19yY3YsIG0sIDApOwotCQkJU09DS0JVRl9VTkxPQ0soJnNvLT5zb19yY3YpOwotCQkJdHAtPnRf ZmxhZ3MgfD0gVEZfV0FLRVNPUjsKKwkJCXNvcndha2V1cF9sb2NrZWQoc28pOwogI2lmZGVmIE5F VEZMSVhfU0JfTElNSVRTCiAJCQlpZiAoc28tPnNvX3Jjdi5zYl9zaGxpbSAmJiBhcHBlbmRlZCAh PSBtY250KQogCQkJCWNvdW50ZXJfZm9fcmVsZWFzZShzby0+c29fcmN2LnNiX3NobGltLApAQCAt ODQxNCw4ICs4NDEyLDYgQEAgYmJyX3Byb2Nlc3NfZGF0YShzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0 IHRjcGhkciAqdGgsIHN0CiAJaWYgKHRoZmxhZ3MgJiBUSF9GSU4pIHsKIAkJaWYgKFRDUFNfSEFW RVJDVkRGSU4odHAtPnRfc3RhdGUpID09IDApIHsKIAkJCXNvY2FudHJjdm1vcmUoc28pOwotCQkJ LyogVGhlIHNvY2tldCB1cGNhbGwgaXMgaGFuZGxlZCBieSBzb2NhbnRyY3Ztb3JlLiAqLwotCQkJ dHAtPnRfZmxhZ3MgJj0gflRGX1dBS0VTT1I7CiAJCQkvKgogCQkJICogSWYgY29ubmVjdGlvbiBp cyBoYWxmLXN5bmNocm9uaXplZCAoaWUgTkVFRFNZTgogCQkJICogZmxhZyBvbikgdGhlbiBkZWxh eSBBQ0ssIHNvIGl0IG1heSBiZSBwaWdneWJhY2tlZApAQCAtODYwNiw4ICs4NjAyLDcgQEAgYmJy X2RvX2Zhc3RuZXdkYXRhKHN0cnVjdCBtYnVmICptLCBzdHJ1Y3QgdGNwaGRyICp0aCwgCiAJCQlz YmFwcGVuZHN0cmVhbV9sb2NrZWQoJnNvLT5zb19yY3YsIG0sIDApOwogCQljdGZfY2FsY19yd2lu KHNvLCB0cCk7CiAJfQotCVNPQ0tCVUZfVU5MT0NLKCZzby0+c29fcmN2KTsKLQl0cC0+dF9mbGFn cyB8PSBURl9XQUtFU09SOworCXNvcndha2V1cF9sb2NrZWQoc28pOwogI2lmZGVmIE5FVEZMSVhf U0JfTElNSVRTCiAJaWYgKHNvLT5zb19yY3Yuc2Jfc2hsaW0gJiYgbWNudCAhPSBhcHBlbmRlZCkK IAkJY291bnRlcl9mb19yZWxlYXNlKHNvLT5zb19yY3Yuc2Jfc2hsaW0sIG1jbnQgLSBhcHBlbmRl ZCk7CkBAIC04Nzk4LDcgKzg3OTMsNyBAQCBiYnJfZmFzdGFjayhzdHJ1Y3QgbWJ1ZiAqbSwgc3Ry dWN0IHRjcGhkciAqdGgsIHN0cnVjdCAKIAkJICAgICZ0Y3Bfc2F2ZXRjcCwgMCk7CiAjZW5kaWYK IAkvKiBXYWtlIHVwIHRoZSBzb2NrZXQgaWYgd2UgaGF2ZSByb29tIHRvIHdyaXRlIG1vcmUgKi8K LQl0cC0+dF9mbGFncyB8PSBURl9XQUtFU09XOworCXNvd3dha2V1cChzbyk7CiAJaWYgKHRwLT5z bmRfdW5hID09IHRwLT5zbmRfbWF4KSB7CiAJCS8qIE5vdGhpbmcgbGVmdCBvdXRzdGFuZGluZyAq LwogCQliYnJfbG9nX3Byb2dyZXNzX2V2ZW50KGJiciwgdHAsIHRpY2tzLCBQUk9HUkVTU19DTEVB UiwgX19MSU5FX18pOwpAQCAtMTE3NTQsMTAgKzExNzQ5LDggQEAgYmJyX2RvX3NlZ21lbnQoc3Ry dWN0IG1idWYgKm0sIHN0cnVjdCB0Y3BoZHIgKnRoLCBzdHJ1CiAJfQogCXJldHZhbCA9IGJicl9k b19zZWdtZW50X25vdW5sb2NrKG0sIHRoLCBzbywgdHAsCiAJCQkJCSBkcm9wX2hkcmxlbiwgdGxl biwgaXB0b3MsIDAsICZ0dik7Ci0JaWYgKHJldHZhbCA9PSAwKSB7Ci0JCXRjcF9oYW5kbGVfd2Fr ZXVwKHRwLCBzbyk7CisJaWYgKHJldHZhbCA9PSAwKQogCQlJTlBfV1VOTE9DSyh0cC0+dF9pbnBj Yik7Ci0JfQogfQogCiAvKgotLS0gc3lzL25ldGluZXQvdGNwX3N0YWNrcy9yYWNrLmMub3JpZwky MDIxLTA0LTA5IDA4OjM2OjIzLjYyMjgyMTAwMCAtMDcwMAorKysgc3lzL25ldGluZXQvdGNwX3N0 YWNrcy9yYWNrLmMJMjAyMS0wNC0wOSAwODo0MToyNC4wOTY2ODcwMDAgLTA3MDAKQEAgLTgzNDQs OCArODM0NCw3IEBAIHJhY2tfcHJvY2Vzc19hY2soc3RydWN0IG1idWYgKm0sIHN0cnVjdCB0Y3Bo ZHIgKnRoLCBzdAogCQkgKi8KIAkJb3VyZmluaXNhY2tlZCA9IDE7CiAJfQotCVNPQ0tCVUZfVU5M T0NLKCZzby0+c29fc25kKTsKLQl0cC0+dF9mbGFncyB8PSBURl9XQUtFU09XOworCXNvd3dha2V1 cF9sb2NrZWQoc28pOwogCW1fZnJlZW0obWZyZWUpOwogCWlmIChyYWNrLT5yX2N0bC5yY19lYXJs eV9yZWNvdmVyeSA9PSAwKSB7CiAJCWlmIChJTl9SRUNPVkVSWSh0cC0+dF9mbGFncykpIHsKQEAg LTg2NjUsOCArODY2NCw3IEBAIHJhY2tfcHJvY2Vzc19kYXRhKHN0cnVjdCBtYnVmICptLCBzdHJ1 Y3QgdGNwaGRyICp0aCwgcwogCQkJCWFwcGVuZGVkID0KICNlbmRpZgogCQkJCQlzYmFwcGVuZHN0 cmVhbV9sb2NrZWQoJnNvLT5zb19yY3YsIG0sIDApOwotCQkJU09DS0JVRl9VTkxPQ0soJnNvLT5z b19yY3YpOwotCQkJdHAtPnRfZmxhZ3MgfD0gVEZfV0FLRVNPUjsKKwkJCXNvcndha2V1cF9sb2Nr ZWQoc28pOwogI2lmZGVmIE5FVEZMSVhfU0JfTElNSVRTCiAJCQlpZiAoc28tPnNvX3Jjdi5zYl9z aGxpbSAmJiBhcHBlbmRlZCAhPSBtY250KQogCQkJCWNvdW50ZXJfZm9fcmVsZWFzZShzby0+c29f cmN2LnNiX3NobGltLApAQCAtODczMSw4ICs4NzI5LDYgQEAgcmFja19wcm9jZXNzX2RhdGEoc3Ry dWN0IG1idWYgKm0sIHN0cnVjdCB0Y3BoZHIgKnRoLCBzCiAJaWYgKHRoZmxhZ3MgJiBUSF9GSU4p IHsKIAkJaWYgKFRDUFNfSEFWRVJDVkRGSU4odHAtPnRfc3RhdGUpID09IDApIHsKIAkJCXNvY2Fu dHJjdm1vcmUoc28pOwotCQkJLyogVGhlIHNvY2tldCB1cGNhbGwgaXMgaGFuZGxlZCBieSBzb2Nh bnRyY3Ztb3JlLiAqLwotCQkJdHAtPnRfZmxhZ3MgJj0gflRGX1dBS0VTT1I7CiAJCQkvKgogCQkJ ICogSWYgY29ubmVjdGlvbiBpcyBoYWxmLXN5bmNocm9uaXplZCAoaWUgTkVFRFNZTgogCQkJICog ZmxhZyBvbikgdGhlbiBkZWxheSBBQ0ssIHNvIGl0IG1heSBiZSBwaWdneWJhY2tlZApAQCAtODky NCw4ICs4OTIwLDcgQEAgcmFja19kb19mYXN0bmV3ZGF0YShzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0 IHRjcGhkciAqdGgsCiAJCQlzYmFwcGVuZHN0cmVhbV9sb2NrZWQoJnNvLT5zb19yY3YsIG0sIDAp OwogCQljdGZfY2FsY19yd2luKHNvLCB0cCk7CiAJfQotCVNPQ0tCVUZfVU5MT0NLKCZzby0+c29f cmN2KTsKLQl0cC0+dF9mbGFncyB8PSBURl9XQUtFU09SOworCXNvcndha2V1cF9sb2NrZWQoc28p OwogI2lmZGVmIE5FVEZMSVhfU0JfTElNSVRTCiAJaWYgKHNvLT5zb19yY3Yuc2Jfc2hsaW0gJiYg bWNudCAhPSBhcHBlbmRlZCkKIAkJY291bnRlcl9mb19yZWxlYXNlKHNvLT5zb19yY3Yuc2Jfc2hs aW0sIG1jbnQgLSBhcHBlbmRlZCk7CkBAIC05MTQyLDcgKzkxMzcsNyBAQCByYWNrX2Zhc3RhY2so c3RydWN0IG1idWYgKm0sIHN0cnVjdCB0Y3BoZHIgKnRoLCBzdHJ1Y3QKIAkJcmFja190aW1lcl9j YW5jZWwodHAsIHJhY2ssIHJhY2stPnJfY3RsLnJjX3JjdnRpbWUsIF9fTElORV9fKTsKIAl9CiAJ LyogV2FrZSB1cCB0aGUgc29ja2V0IGlmIHdlIGhhdmUgcm9vbSB0byB3cml0ZSBtb3JlICovCi0J dHAtPnRfZmxhZ3MgfD0gVEZfV0FLRVNPVzsKKwlzb3d3YWtldXAoc28pOwogCWlmIChzYmF2YWls KCZzby0+c29fc25kKSkgewogCQlyYWNrLT5yX3dhbnRlZF9vdXRwdXQgPSAxOwogCX0KQEAgLTEx MjA1LDEwICsxMTIwMCw4IEBAIHJhY2tfZG9fc2VnbWVudChzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0 IHRjcGhkciAqdGgsIHN0cgogCQl0Y3BfZ2V0X3VzZWNzKCZ0dik7CiAJfQogCWlmKHJhY2tfZG9f c2VnbWVudF9ub3VubG9jayhtLCB0aCwgc28sIHRwLAotCQkJCSAgICBkcm9wX2hkcmxlbiwgdGxl biwgaXB0b3MsIDAsICZ0dikgPT0gMCkgewotCQl0Y3BfaGFuZGxlX3dha2V1cCh0cCwgc28pOwor CQkJCSAgICBkcm9wX2hkcmxlbiwgdGxlbiwgaXB0b3MsIDAsICZ0dikgPT0gMCkKIAkJSU5QX1dV TkxPQ0sodHAtPnRfaW5wY2IpOwotCX0KIH0KIAogc3RydWN0IHJhY2tfc2VuZG1hcCAqCi0tLSBz eXMvbmV0aW5ldC90Y3Bfc3RhY2tzL3JhY2tfYmJyX2NvbW1vbi5jLm9yaWcJMjAyMS0wNC0wOSAw ODo0NToyNi43MjE1MjEwMDAgLTA3MDAKKysrIHN5cy9uZXRpbmV0L3RjcF9zdGFja3MvcmFja19i YnJfY29tbW9uLmMJMjAyMS0wNC0wOSAwODo0Njo1OC41ODAyMzQwMDAgLTA3MDAKQEAgLTQ1OCw3 ICs0NTgsNiBAQCBjdGZfZG9fcXVldWVkX3NlZ21lbnRzKHN0cnVjdCBzb2NrZXQgKnNvLCBzdHJ1 Y3QgdGNwY2IKIAkJCS8qIFdlIGxvc3QgdGhlIHRjcGNiIChtYXliZSBhIFJTVCBjYW1lIGluKT8g Ki8KIAkJCXJldHVybigxKTsKIAkJfQotCQl0Y3BfaGFuZGxlX3dha2V1cCh0cCwgc28pOwogCX0K IAlyZXR1cm4gKDApOwogfQotLS0gc3lzL25ldGluZXQvdGNwX2lucHV0LmMub3JpZwkyMDIxLTA0 LTA1IDAxOjA3OjAwLjM0MjU1OTAwMCAtMDcwMAorKysgc3lzL25ldGluZXQvdGNwX2lucHV0LmMJ MjAyMS0wNC0wOSAwNzo1ODowMy4yNjI4MTUwMDAgLTA3MDAKQEAgLTE0NzIsMjkgKzE0NzIsNiBA QCB0Y3BfYXV0b3JjdmJ1ZihzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0IHRjcGhkciAqdGgsIHN0cnUK IH0KIAogdm9pZAotdGNwX2hhbmRsZV93YWtldXAoc3RydWN0IHRjcGNiICp0cCwgc3RydWN0IHNv Y2tldCAqc28pCi17Ci0JLyoKLQkgKiBTaW5jZSB0cCBtaWdodCBiZSBnb25lIGlmIHRoZSBzZXNz aW9uIGVudGVyZWQKLQkgKiB0aGUgVElNRV9XQUlUIHN0YXRlIGJlZm9yZSBjb21pbmcgaGVyZSwg d2UgbmVlZAotCSAqIHRvIGNoZWNrIGlmIHRoZSBzb2NrZXQgaXMgc3RpbGwgY29ubmVjdGVkLgot CSAqLwotCWlmICgoc28tPnNvX3N0YXRlICYgU1NfSVNDT05ORUNURUQpID09IDApCi0JCXJldHVy bjsKLQlJTlBfTE9DS19BU1NFUlQodHAtPnRfaW5wY2IpOwotCWlmICh0cC0+dF9mbGFncyAmIFRG X1dBS0VTT1IpIHsKLQkJdHAtPnRfZmxhZ3MgJj0gflRGX1dBS0VTT1I7Ci0JCVNPQ0tCVUZfVU5M T0NLX0FTU0VSVCgmc28tPnNvX3Jjdik7Ci0JCXNvcndha2V1cChzbyk7Ci0JfQotCWlmICh0cC0+ dF9mbGFncyAmIFRGX1dBS0VTT1cpIHsKLQkJdHAtPnRfZmxhZ3MgJj0gflRGX1dBS0VTT1c7Ci0J CVNPQ0tCVUZfVU5MT0NLX0FTU0VSVCgmc28tPnNvX3NuZCk7Ci0JCXNvd3dha2V1cChzbyk7Ci0J fQotfQotCi12b2lkCiB0Y3BfZG9fc2VnbWVudChzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0IHRjcGhk ciAqdGgsIHN0cnVjdCBzb2NrZXQgKnNvLAogICAgIHN0cnVjdCB0Y3BjYiAqdHAsIGludCBkcm9w X2hkcmxlbiwgaW50IHRsZW4sIHVpbnQ4X3QgaXB0b3MpCiB7CkBAIC0xODYzLDcgKzE4NDAsNyBA QCB0Y3BfZG9fc2VnbWVudChzdHJ1Y3QgbWJ1ZiAqbSwgc3RydWN0IHRjcGhkciAqdGgsIHN0cnUK IAkJCQllbHNlIGlmICghdGNwX3RpbWVyX2FjdGl2ZSh0cCwgVFRfUEVSU0lTVCkpCiAJCQkJCXRj cF90aW1lcl9hY3RpdmF0ZSh0cCwgVFRfUkVYTVQsCiAJCQkJCQkgICAgICB0cC0+dF9yeHRjdXIp OwotCQkJCXRwLT50X2ZsYWdzIHw9IFRGX1dBS0VTT1c7CisJCQkJc293d2FrZXVwKHNvKTsKIAkJ CQlpZiAoc2JhdmFpbCgmc28tPnNvX3NuZCkpCiAJCQkJCSh2b2lkKSB0cC0+dF9mYi0+dGZiX3Rj cF9vdXRwdXQodHApOwogCQkJCWdvdG8gY2hlY2tfZGVsYWNrOwpAQCAtMTkyOCw4ICsxOTA1LDgg QEAgdGNwX2RvX3NlZ21lbnQoc3RydWN0IG1idWYgKm0sIHN0cnVjdCB0Y3BoZHIgKnRoLCBzdHJ1 CiAJCQkJbV9hZGoobSwgZHJvcF9oZHJsZW4pOwkvKiBkZWxheWVkIGhlYWRlciBkcm9wICovCiAJ CQkJc2JhcHBlbmRzdHJlYW1fbG9ja2VkKCZzby0+c29fcmN2LCBtLCAwKTsKIAkJCX0KLQkJCVNP Q0tCVUZfVU5MT0NLKCZzby0+c29fcmN2KTsKLQkJCXRwLT50X2ZsYWdzIHw9IFRGX1dBS0VTT1I7 CisJCQkvKiBOQjogc29yd2FrZXVwX2xvY2tlZCgpIGRvZXMgYW4gaW1wbGljaXQgdW5sb2NrLiAq LworCQkJc29yd2FrZXVwX2xvY2tlZChzbyk7CiAJCQlpZiAoREVMQVlfQUNLKHRwLCB0bGVuKSkg ewogCQkJCXRwLT50X2ZsYWdzIHw9IFRGX0RFTEFDSzsKIAkJCX0gZWxzZSB7CkBAIC0yOTI1LDgg KzI5MDIsOCBAQCBwcm9jZXNzX0FDSzoKIAkJCQl0cC0+c25kX3duZCA9IDA7CiAJCQlvdXJmaW5p c2Fja2VkID0gMDsKIAkJfQotCQlTT0NLQlVGX1VOTE9DSygmc28tPnNvX3NuZCk7Ci0JCXRwLT50 X2ZsYWdzIHw9IFRGX1dBS0VTT1c7CisJCS8qIE5COiBzb3d3YWtldXBfbG9ja2VkKCkgZG9lcyBh biBpbXBsaWNpdCB1bmxvY2suICovCisJCXNvd3dha2V1cF9sb2NrZWQoc28pOwogCQltX2ZyZWVt KG1mcmVlKTsKIAkJLyogRGV0ZWN0IHVuYSB3cmFwYXJvdW5kLiAqLwogCQlpZiAoIUlOX1JFQ09W RVJZKHRwLT50X2ZsYWdzKSAmJgpAQCAtMzE0Nyw4ICszMTI0LDggQEAgZG9kYXRhOgkJCQkJCQkv KiBYWFggKi8KIAkJCQltX2ZyZWVtKG0pOwogCQkJZWxzZQogCQkJCXNiYXBwZW5kc3RyZWFtX2xv Y2tlZCgmc28tPnNvX3JjdiwgbSwgMCk7Ci0JCQlTT0NLQlVGX1VOTE9DSygmc28tPnNvX3Jjdik7 Ci0JCQl0cC0+dF9mbGFncyB8PSBURl9XQUtFU09SOworCQkJLyogTkI6IHNvcndha2V1cF9sb2Nr ZWQoKSBkb2VzIGFuIGltcGxpY2l0IHVubG9jay4gKi8KKwkJCXNvcndha2V1cF9sb2NrZWQoc28p OwogCQl9IGVsc2UgewogCQkJLyoKIAkJCSAqIFhYWDogRHVlIHRvIHRoZSBoZWFkZXIgZHJvcCBh Ym92ZSAidGgiIGlzCkBAIC0zMjE1LDggKzMxOTIsNiBAQCBkb2RhdGE6CQkJCQkJCS8qIFhYWCAq LwogCWlmICh0aGZsYWdzICYgVEhfRklOKSB7CiAJCWlmIChUQ1BTX0hBVkVSQ1ZERklOKHRwLT50 X3N0YXRlKSA9PSAwKSB7CiAJCQlzb2NhbnRyY3Ztb3JlKHNvKTsKLQkJCS8qIFRoZSBzb2NrZXQg dXBjYWxsIGlzIGhhbmRsZWQgYnkgc29jYW50cmN2bW9yZS4gKi8KLQkJCXRwLT50X2ZsYWdzICY9 IH5URl9XQUtFU09SOwogCQkJLyoKIAkJCSAqIElmIGNvbm5lY3Rpb24gaXMgaGFsZi1zeW5jaHJv bml6ZWQKIAkJCSAqIChpZSBORUVEU1lOIGZsYWcgb24pIHRoZW4gZGVsYXkgQUNLLApAQCAtMzI4 MCw3ICszMjU1LDYgQEAgY2hlY2tfZGVsYWNrOgogCQl0cC0+dF9mbGFncyAmPSB+VEZfREVMQUNL OwogCQl0Y3BfdGltZXJfYWN0aXZhdGUodHAsIFRUX0RFTEFDSywgdGNwX2RlbGFja3RpbWUpOwog CX0KLQl0Y3BfaGFuZGxlX3dha2V1cCh0cCwgc28pOwogCUlOUF9XVU5MT0NLKHRwLT50X2lucGNi KTsKIAlyZXR1cm47CiAKQEAgLTMzMTQsNyArMzI4OCw2IEBAIGRyb3BhZnRlcmFjazoKIAlUQ1Bf UFJPQkUzKGRlYnVnX19pbnB1dCwgdHAsIHRoLCBtKTsKIAl0cC0+dF9mbGFncyB8PSBURl9BQ0tO T1c7CiAJKHZvaWQpIHRwLT50X2ZiLT50ZmJfdGNwX291dHB1dCh0cCk7Ci0JdGNwX2hhbmRsZV93 YWtldXAodHAsIHNvKTsKIAlJTlBfV1VOTE9DSyh0cC0+dF9pbnBjYik7CiAJbV9mcmVlbShtKTsK IAlyZXR1cm47CkBAIC0zMzIyLDcgKzMyOTUsNiBAQCBkcm9wYWZ0ZXJhY2s6CiBkcm9wd2l0aHJl c2V0OgogCWlmICh0cCAhPSBOVUxMKSB7CiAJCXRjcF9kcm9wd2l0aHJlc2V0KG0sIHRoLCB0cCwg dGxlbiwgcnN0cmVhc29uKTsKLQkJdGNwX2hhbmRsZV93YWtldXAodHAsIHNvKTsKIAkJSU5QX1dV TkxPQ0sodHAtPnRfaW5wY2IpOwogCX0gZWxzZQogCQl0Y3BfZHJvcHdpdGhyZXNldChtLCB0aCwg TlVMTCwgdGxlbiwgcnN0cmVhc29uKTsKQEAgLTMzMzgsMTAgKzMzMTAsOCBAQCBkcm9wOgogCQkJ ICAmdGNwX3NhdmV0Y3AsIDApOwogI2VuZGlmCiAJVENQX1BST0JFMyhkZWJ1Z19faW5wdXQsIHRw LCB0aCwgbSk7Ci0JaWYgKHRwICE9IE5VTEwpIHsKLQkJdGNwX2hhbmRsZV93YWtldXAodHAsIHNv KTsKKwlpZiAodHAgIT0gTlVMTCkKIAkJSU5QX1dVTkxPQ0sodHAtPnRfaW5wY2IpOwotCX0KIAlt X2ZyZWVtKG0pOwogfQogCi0tLSBzeXMvbmV0aW5ldC90Y3BfcmVhc3MuYy5vcmlnCTIwMjEtMDQt MDkgMDg6MTg6MTAuNTk5MDkyMDAwIC0wNzAwCisrKyBzeXMvbmV0aW5ldC90Y3BfcmVhc3MuYwky MDIxLTA0LTA5IDA4OjE5OjU0LjkxMjM3ODAwMCAtMDcwMApAQCAtOTU5LDggKzk1OSw3IEBAIG5l d19lbnRyeToKIAkJfSBlbHNlIHsKIAkJCXNiYXBwZW5kc3RyZWFtX2xvY2tlZCgmc28tPnNvX3Jj diwgbSwgMCk7CiAJCX0KLQkJU09DS0JVRl9VTkxPQ0soJnNvLT5zb19yY3YpOwotCQl0cC0+dF9m bGFncyB8PSBURl9XQUtFU09SOworCQlzb3J3YWtldXBfbG9ja2VkKHNvKTsKIAkJcmV0dXJuIChm bGFncyk7CiAJfQogCWlmICh0Y3BfbmV3X2xpbWl0cykgewpAQCAtMTEwOCw3ICsxMTA3LDYgQEAg cHJlc2VudDoKICNpZmRlZiBUQ1BfUkVBU1NfTE9HR0lORwogCXRjcF9yZWFzc19sb2dfZHVtcCh0 cCk7CiAjZW5kaWYKLQlTT0NLQlVGX1VOTE9DSygmc28tPnNvX3Jjdik7Ci0JdHAtPnRfZmxhZ3Mg fD0gVEZfV0FLRVNPUjsKKwlzb3J3YWtldXBfbG9ja2VkKHNvKTsKIAlyZXR1cm4gKGZsYWdzKTsK IH0K --_002_YQXPR0101MB09681707D3F3DC10814A905BDD4D9YQXPR0101MB0968_--