From owner-freebsd-net@freebsd.org Sun Mar 21 22:21:36 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 802805AF55B for ; Sun, 21 Mar 2021 22:21:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660041.outbound.protection.outlook.com [40.107.66.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F3XCz2MBDz4Wrw for ; Sun, 21 Mar 2021 22:21:34 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m8CBMeaA4IkALi7/gE8J+2sihTWekpojviUTWveobPIqmF9zv7o6+VfPnFbkEVq2WqUk3PZ03inFrZeWR0T5Zfh6MZoPLeVuZGtFw2IeqdUIReRQ3aD5xyYR9tH0wR5x79eCsZBDW8vtGjj8S8JjEcKk4EFU0dwY5MLBactgrgFL3Gzpgj2XHVgHm8eBAFbN25wnE9FvUddwjoR5M1o2riiarOdIPQw7/KW60wPlamiRs6Bw+jUX60Qd0ymv2jzmx+73++HSzeNXZREonOrwxuvoXdEm4EwA4dlgwmD2TmjCMCL/g9FQQfFsXYOKxxPEPw1cj18RC3uUCnF43Gc7Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FO2SeuUyDJGwqrwBNE+/rrYSPm+FczE3wGhMPBS1MQY=; b=KRhAJb7A42iMo1MlAUHfVq21yUf2aGvxleJVg3gJiPjeTuELkOdKg9qE49GmB2Ai/9dJArVkLAiflRmYKW9/jEA2uQVfUWE9S5wpmD4iBSdKbQQbvd8AHn2jIBkl1ts7ntI3wsTE0edQG3o1Kq780fK+/6eg7YkaVtb1x39ZkPk6do4KXdEub/oR4ObpuTL+sHzuWS2BsnVVGPVjq7aCxx1p2udc7m1VKt4saqxX8qQYvd66dZxWNVVkbXvr5XA+rru3WnyzcCNu7nCu+MFg4kvIRikGAD2pW7m5wQJgp6qslf52p5u0EMNVBhl+LiU9WW+N39Z5oo+sd2Sxp8B2Dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FO2SeuUyDJGwqrwBNE+/rrYSPm+FczE3wGhMPBS1MQY=; b=U0YhAWxGzNKZPySm0tfxfMr5MoT0uvKUd/FdOZJSOi8hOZEQSejQpJaspyVVEBExqD7+xie4ErtnoRU28NU7rm3UqTQw1woEORx5yjlGz4/jCbuPq/jKeZU+xhf98yef5fJ/5zWvGCnx+ukmat5HIvfZ6uuhq0osYHP5OQK5Nui9WDlsRlEMK0vW/lQhRaqWyfvSDqpXeDEG68HDIDk/KbqAln8BZDwOaRuzefOgrT5L5YfGJyzfDwSQGPq4xNoynVeBuVarOrghlzR8TPdJhxcmjG8VwQysZE/X5QaYmUe1ValAFrDgXvdaYPKr0N0wec9wsW1ZvPj798GKgaCKGQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by QB1PR01MB2867.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.25; Sun, 21 Mar 2021 22:21:33 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3955.025; Sun, 21 Mar 2021 22:21:33 +0000 From: Rick Macklem To: "Youssef GHORBAL" , Jason Breitman CC: "freebsd-net@freebsd.org" Subject: Re: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1G2D7AHBwtmAkS1jBAqNNo2I6qMDIgAgALy8kw= Date: Sun, 21 Mar 2021 22:21:33 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: dd9733bf-8961-44e2-bee3-08d8ecb7af15 x-ms-traffictypediagnostic: QB1PR01MB2867: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3826; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TG/Cz+j3nRJfkNj81hTHOdx7NlYwnaZAKcxLqtIqHG1ZjjCMkEXwGg3QwvTuvT4c/qF5hPyovTm4MwLInYnOLliYmbgsReYtwPVQN7jeqncyh48P48W8qJQU+NPkIOklR0UHWwi6g63N/RD9efLv3f6AAsPcS2dltcUj6g9/BLYd/4iCWT+SFFje7Q51k6SYRiCK91Q/6jlu3XmpAl/pOGxw90cq7jb6ncadWbjwS7VBsdSyeB5567B0Bj688QH+bO2+/+dGtQQS+SS2UBueVtd/+sEsEoao4+ldmTS4M3Kz1jcIlSfBMX933W3B+Klcr5/Obj7F9suZr41FCiGzi8RSf43VAKsHPviD+dE4yB1UShAuz2ayOLoiDXaSb9H0ZuM8pxNG7glIXJbcMe6sxJVIdK/rRTAw3+WX45BX/YeJmbSgM49EmmFwAeB7XAuYB7jlQTxfR/vOsVi+Z9n4fLCnafQDxmUL0EzjjtwlBdi2bUXQGkzsjFwaE+rzeDUzi2p6Kn8ubNuVaZK8zNMNnjqcRquEwWxINp5uE6uC+tnksWJFik4MCpwP15w5YfM14oGx/oItZAVsB5TZvaV9y8BjUxXG38GDPhvhhsJH71GKvB6sPzhLPca3HzUTfyhgVw+W9lZG0w0qVbN6b3hCDnIcg/0y32LF8U5zC2ADb3/A/qZm93eSaF+KByM7cAjgWyR1AZnzZuS7g1/2HWW76954c7x+CqR0DXLIU5O4+uQ= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(346002)(39850400004)(396003)(376002)(136003)(366004)(4326008)(66446008)(3480700007)(66476007)(99936003)(76116006)(66946007)(66616009)(71200400001)(7696005)(66556008)(2906002)(86362001)(186003)(53546011)(52536014)(5660300002)(6506007)(8676002)(83380400001)(64756008)(91956017)(55016002)(110136005)(9686003)(8936002)(478600001)(7116003)(33656002)(966005)(316002)(38100700001)(786003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?hMtrnXNKdfTvsZtVypsr7s4sTFMJyHfxwMUolVaJYLs/0z0Xa64itcPF?= =?Windows-1252?Q?fOsALRs9TV1U0o1U+6rTqGQjQMqtXFl11VG+vJa6cMVNWGj4J7gkyUZt?= =?Windows-1252?Q?VJx4VjmX8mp2ZtNuabD5kX10mFoL/bi6sLzgVJ8J4+4oR1dpYayUZ4ZZ?= =?Windows-1252?Q?O/nSU1rHVF188qSBva5/eQy3iGU1O+3ykJTqo4A5rPhAhgbVK/S5gR45?= =?Windows-1252?Q?1sPsZc5HvX/a7Wyh60VHI/6nCMzlYlC8+7hcR7fhy8vhWGUc+jTO7QnP?= =?Windows-1252?Q?w8MRp7R/f4L4oMMIMr5qskzwvMiCL6vMVxEN5/UYmjgB32NhBvK1o3lh?= =?Windows-1252?Q?4p67n8V6bhQGjLZny4qWP/DBaoHCkIFnjuJCiavrBzVOxOtPJVS1V+UZ?= =?Windows-1252?Q?FSM78UHpKsVkObzDTr3kKXa6c0fs/JFZ7Y/IJE9yQ4mZIHTXWS7kWnlP?= =?Windows-1252?Q?P+L5INY/yMuddGcTrN5bsXSJbkja7WFcuiSWHFkLAJaU3jPU4R28wmrl?= =?Windows-1252?Q?HgSnLzIwIHhLQxNp6OWGCn+fykJQUmWE/jqRREyAmnyHWLneCVwcLqnH?= =?Windows-1252?Q?kBjgsnzsLoKkWvhHI8ORUWJJzjNnVqlmij42QmvW8iKX9lcyai+Ax2ZG?= =?Windows-1252?Q?QM+d+GMcGTQ1lcc9jpixHBBS0Twzl9yLod1qUGeidgvcbPj7XkojUzwh?= =?Windows-1252?Q?iCCFQuGTbWPdO3CkngyHKtT4VabcqT8oyfuBxiRYzcjw/NgARqwsqB+J?= =?Windows-1252?Q?IjZ31cBaorqEn3DulPJxQLh0BHt5nTSO6mpaR2li7w/hGwSsnN3hwe+E?= =?Windows-1252?Q?Hv/kuqB/SAEF0b2vCRDWrKPVRa/qbklxBqtDGdELOIWthXYB4AyEG3yT?= =?Windows-1252?Q?I3qB5+davY/L26QCwHIU88p35M759GdLZaM/Smx7prw+2GpyL4SBwcsB?= =?Windows-1252?Q?mZk0Sbkzim16xVxXzbo/55vIfw80z7VKozaZ3Tybj3U+j6H1Q/+Vs9X7?= =?Windows-1252?Q?qnWMgjZfZM7/Ph7SRDqMyoIbrdouS92RHlySKlE8DhHx+qDourMtKQaw?= =?Windows-1252?Q?KO8BrOkT35mI6jDcn0KXHNlQPaeAzUy4JmI1Q2DKHBoWL/LkwRyjvLpp?= =?Windows-1252?Q?sMyDhht2CmhdmOEtPkaa24CsUYr5y7PzZaUgAZQJJw+G6pXlXME+DCYs?= =?Windows-1252?Q?GoZdUlgh4N98wvGQKGixm+NEqub4lQqgaNSMIZDrntuyhz11OydX+sKu?= =?Windows-1252?Q?Wdy8VK8IbNzxsxDz8jMlOapIU5AwwluJx2Ahaj+5+p4P4bR8pJEU9/ah?= =?Windows-1252?Q?EcsnA2RtAs4inruj0CzanI0IkKdPNA+mEUEdPmJ0EEK+Yyl5Jrmw0BVC?= =?Windows-1252?Q?jjNmjaZC3E8R6PVEGP02N0O2I29FpK9I1UGPjeQYLa1FqkNY4APk0/fg?= =?Windows-1252?Q?RHYiLpEmcsmZE27Ch1FqiyaR0XSFvXjsvRRb2BS9jWc=3D?= x-ms-exchange-transport-forked: True Content-Type: multipart/mixed; boundary="_002_YQXPR0101MB09684AB7BEFA911213604467DD669YQXPR0101MB0968_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: dd9733bf-8961-44e2-bee3-08d8ecb7af15 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2021 22:21:33.2583 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: j7ZidkdUJ+h7bck7BOAcIRBlmfZn01pq//l7Th4aG4d2PedfFtEKou3WCESs2yp+Ht1cacDxc4Jwvt7U2Xsqeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2867 X-Rspamd-Queue-Id: 4F3XCz2MBDz4Wrw X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=U0YhAWxG; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.41 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-0.995]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.41:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; SPAMHAUS_ZRD(0.00)[40.107.66.41:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.41:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.41:from]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Mar 2021 22:21:36 -0000 --_002_YQXPR0101MB09684AB7BEFA911213604467DD669YQXPR0101MB0968_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Youssef GHORBAL wrote:=0A= >Hi Jason,=0A= >=0A= >> On 17 Mar 2021, at 18:17, Jason Breitman wrote:=0A= >>=0A= >> Please review the details below and let me know if there is a setting th= at I should apply to my FreeBSD NFS Server or if there is a bug fix that I = can apply to resolve my issue.=0A= >> I shared this information with the linux-nfs mailing list and they belie= ve the issue is on the server side.=0A= >>=0A= >> Issue=0A= >> NFSv4 mounts periodically hang on the NFS Client.=0A= >>=0A= >> During this time, it is possible to manually mount from another NFS Serv= er on the NFS Client having issues.=0A= >> Also, other NFS Clients are successfully mounting from the NFS Server in= question.=0A= >> Rebooting the NFS Client appears to be the only solution.=0A= >=0A= >I had experienced a similar weird situation with periodically stuck Linux = NFS clients >mounting Isilon NFS servers (Isilon is FreeBSD based but they = seem to have there >own nfsd)=0A= Yes, my understanding is that Isilon uses a proprietary user space nfsd and= =0A= not the kernel based RPC and nfsd in FreeBSD.=0A= =0A= >We=92ve had better luck and we did manage to have packet captures on both = sides >during the issue. The gist of it goes like follows:=0A= >=0A= >- Data flows correctly between SERVER and the CLIENT=0A= >- At some point SERVER starts decreasing it's TCP Receive Window until it = reachs 0=0A= >- The client (eager to send data) can only ack data sent by SERVER.=0A= >- When SERVER was done sending data, the client starts sending TCP Window = >Probes hoping that the TCP Window opens again so he can flush its buffers.= =0A= >- SERVER responds with a TCP Zero Window to those probes.=0A= Having the window size drop to zero is not necessarily incorrect.=0A= If the server is overloaded (has a backlog of NFS requests), it can stop do= ing=0A= soreceive() on the socket (so the socket rcv buffer can fill up and the TCP= window=0A= closes). This results in "backpressure" to stop the NFS client from floodin= g the=0A= NFS server with requests.=0A= --> However, once the backlog is handled, the nfsd should start to soreceiv= e()=0A= again and this shouls cause the window to open back up.=0A= --> Maybe this is broken in the socket/TCP code. I quickly got lost i= n=0A= tcp_output() when it decides what to do about the rcvwin.=0A= =0A= >- After 6 minutes (the NFS server default Idle timeout) SERVER racefully c= loses the >TCP connection sending a FIN Packet (and still a TCP Window 0)= =0A= This probably does not happen for Jason's case, since the 6minute timeout= =0A= is disabled when the TCP connection is assigned as a backchannel (most like= ly=0A= the case for NFSv4.1).=0A= =0A= >- CLIENT ACK that FIN.=0A= >- SERVER goes in FIN_WAIT_2 state=0A= >- CLIENT closes its half part part of the socket and goes in LAST_ACK stat= e.=0A= >- FIN is never sent by the client since there still data in its SendQ and = receiver TCP >Window is still 0. At this stage the client starts sending TC= P Window Probes again >and again hoping that the server opens its TCP Windo= w so it can flush it's buffers >and terminate its side of the socket.=0A= >- SERVER keeps responding with a TCP Zero Window to those probes.=0A= >=3D> The last two steps goes on and on for hours/days freezing the NFS mou= nt bound >to that TCP session.=0A= >=0A= >If we had a situation where CLIENT was responsible for closing the TCP Win= dow (and >initiating the TCP FIN first) and server wanting to send data we= =92ll end up in the same >state as you I think.=0A= >=0A= >We=92ve never had the root cause of why the SERVER decided to close the TC= P >Window and no more acccept data, the fix on the Isilon part was to recyc= le more >aggressively the FIN_WAIT_2 sockets (net.inet.tcp.fast_finwait2_re= cycle=3D1 & >net.inet.tcp.finwait2_timeout=3D5000). Once the socket recycle= d and at the next >occurence of CLIENT TCP Window probe, SERVER sends a RST= , triggering the >teardown of the session on the client side, a new TCP han= dchake, etc and traffic >flows again (NFS starts responding)=0A= >=0A= >To avoid rebooting the client (and before the aggressive FIN_WAIT_2 was >= implemented on the Isilon side) we=92ve added a check script on the client = that detects >LAST_ACK sockets on the client and through iptables rule enfo= rces a TCP RST, >Something like: -A OUTPUT -p tcp -d $nfs_server_addr --spo= rt $local_port -j REJECT >--reject-with tcp-reset (the script removes this = iptables rule as soon as the LAST_ACK >disappears)=0A= >=0A= >The bottom line would be to have a packet capture during the outage (clien= t and/or >server side), it will show you at least the shape of the TCP exch= ange when NFS is >stuck.=0A= Interesting story and good work w.r.t. sluething, Youssef, thanks.=0A= =0A= I looked at Jason's log and it shows everything is ok w.r.t the nfsd thread= s.=0A= (They're just waiting for RPC requests.)=0A= However, I do now think I know why the soclose() does not happen.=0A= When the TCP connection is assigned as a backchannel, that takes a referenc= e=0A= cnt on the structure. This refcnt won't be released until the connection is= =0A= replaced by a BindConnectiotoSession operation from the client. But that wo= n't=0A= happen until the client creates a new TCP connection.=0A= --> No refcnt release-->no refcnt of 0-->no soclose().=0A= =0A= I've created the attached patch (completely different from the previous one= )=0A= that adds soshutdown(SHUT_WR) calls in the three places where the TCP=0A= connection is going away. This seems to get it past CLOSE_WAIT without a=0A= soclose().=0A= --> I know you are not comfortable with patching your server, but I do thin= k=0A= this change will get the socket shutdown to complete.=0A= =0A= There are a couple more things you can check on the server...=0A= # nfsstat -E -s=0A= --> Look for the count under "BindConnToSes".=0A= --> If non-zero, backchannels have been assigned=0A= # sysctl -a | fgrep request_space_throttle_count=0A= --> If non-zero, the server has been overloaded at some point.=0A= =0A= I think the attached patch might work around the problem.=0A= The code that should open up the receive window needs to be checked.=0A= I am also looking at enabling the 6minute timeout when a backchannel is=0A= assigned.=0A= =0A= rick=0A= =0A= Youssef=0A= =0A= _______________________________________________=0A= freebsd-net@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A= To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A= --_002_YQXPR0101MB09684AB7BEFA911213604467DD669YQXPR0101MB0968_ Content-Type: application/octet-stream; name="xprtdied.patch" Content-Description: xprtdied.patch Content-Disposition: attachment; filename="xprtdied.patch"; size=802; creation-date="Sun, 21 Mar 2021 22:21:27 GMT"; modification-date="Sun, 21 Mar 2021 22:21:27 GMT" Content-Transfer-Encoding: base64 LS0tIHN5cy9ycGMvc3ZjLmMueHBydGRpZWQJMjAyMS0wMy0yMCAxMzozNjo1MC43OTgzMzUwMDAg LTA3MDAKKysrIHN5cy9ycGMvc3ZjLmMJMjAyMS0wMy0yMCAxNzozMDo0MS40NDgyNzAwMDAgLTA3 MDAKQEAgLTIwMyw2ICsyMDMsOCBAQCBzdmNwb29sX2NsZWFudXAoU1ZDUE9PTCAqcG9vbCkKIAkJ bXR4X3VubG9jaygmZ3JwLT5zZ19sb2NrKTsKIAl9CiAJVEFJTFFfRk9SRUFDSF9TQUZFKHhwcnQs ICZjbGVhbnVwLCB4cF9saW5rLCBueHBydCkgeworCQlpZiAoeHBydC0+eHBfc29ja2V0ICE9IE5V TEwpCisJCQlzb3NodXRkb3duKHhwcnQtPnhwX3NvY2tldCwgU0hVVF9XUik7CiAJCVNWQ19SRUxF QVNFKHhwcnQpOwogCX0KIApAQCAtMzg4LDYgKzM5MCw4IEBAIHhwcnRfdW5yZWdpc3RlcihTVkNY UFJUICp4cHJ0KQogCXhwcnRfdW5yZWdpc3Rlcl9sb2NrZWQoeHBydCk7CiAJbXR4X3VubG9jaygm Z3JwLT5zZ19sb2NrKTsKIAorCWlmICh4cHJ0LT54cF9zb2NrZXQgIT0gTlVMTCkKKwkJc29zaHV0 ZG93bih4cHJ0LT54cF9zb2NrZXQsIFNIVVRfV1IpOwogCVNWQ19SRUxFQVNFKHhwcnQpOwogfQog CkBAIC0xMDc4LDYgKzEwODIsNyBAQCBzdmNfY2hlY2tpZGxlKFNWQ0dST1VQICpncnApCiAKIAlt dHhfdW5sb2NrKCZncnAtPnNnX2xvY2spOwogCVRBSUxRX0ZPUkVBQ0hfU0FGRSh4cHJ0LCAmY2xl YW51cCwgeHBfbGluaywgbnhwcnQpIHsKKwkJc29zaHV0ZG93bih4cHJ0LT54cF9zb2NrZXQsIFNI VVRfV1IpOwogCQlTVkNfUkVMRUFTRSh4cHJ0KTsKIAl9CiAJbXR4X2xvY2soJmdycC0+c2dfbG9j ayk7Cg== --_002_YQXPR0101MB09684AB7BEFA911213604467DD669YQXPR0101MB0968_--