From owner-freebsd-fs@freebsd.org Mon Dec 14 00:24:18 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 109C84A8C87 for ; Mon, 14 Dec 2020 00:24:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0601.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CvMZn0QKjz3hJ1 for ; Mon, 14 Dec 2020 00:24:16 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A7KV8dC4Pd41/NEqZ29yM0HJdLLFQXk8OTvsHBQIp3AdLlXLyUK4ItdnOsAHa/KAWRgL6QVY/n6ma8PsH+ih+EArSLiNHIFPoplctA8A1D1wvbWMAPkTM+PX9X48sXKHbkea9NA6GNns3kaUg6TixFmyJk/r7eungShMtN6ouYQ9sPRG/QokLavRSsspu6V5RTp7VQbsNa0/gQW449ApY/0kP/nUiIcm9EjD/3eidLkgVGQvEAugO7VdZW4ciRdtNXRLfgUxeEAv+FqPRDXlFoWEGDzu74I5zAqS1eHU5mofM/Mibb1uSgWZg3tEYdeiFJfNehXzTypLvCakcRQOlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XQq+7hvmAVrCCkKx43PpqujO/5fTSsfZ50vXnJd4InQ=; b=I3Ram8G1BAsJyN4TTKiJCLPC/uDlC5BaTPGS6Q/Gr0lpUccM0kCagqS/cxL5GnUGVZePR31pkIE1G1BxqGqo6hK16wrdSWJYJ0tFOqG6sKbQhFh+v2/XKJo1ThYa9V5LEzJy5shcHJOIxkaowc6Acd5xjcHn5fHyIP3Fo27JkzpAfY22wKU+seLO7KFPYMxiIahGFtdAtO1Os25pobI9P0a3L5Kfj2tzC2HnPQ0nSIljKMTS3jyVd9guoWgUlYT074CC0pqX7+DwpcrnafGSeynByROmvlujVBXF0w2vNxsRCGZ6rE2MiT9chcFSJbzvYLiqiWaVTzWLqqbSj8242w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XQq+7hvmAVrCCkKx43PpqujO/5fTSsfZ50vXnJd4InQ=; b=Bdl8FyBmzY1DRqok3yElSmgrPnfeT+Gfke94ntwUQQW4qjE3gdKBX9HXbIcnCP5gQ4W7PeKl2R96Kjolq7Jn5+1qy35iO9cG7F0p1MaZeV/TyFjxtoqZeA3whj80eS7GkIjLjezHBUvfeJfAHNdu7OsTrcEDbIbOEA5YGy/3hw3txodMMWUpE3suXzUrvylHEoMOGjTAm4ev9hR38jhLkoTyez+MNwmwdQXhm8fIhEjQiO3h5SSu6PmUGZ2toKqFiIZjm0SsaT0WXkCZcze9gxvIPc2Fggcjn/fVtL8/LzmdVjnon7AWFDo7kdYRdNf2dSVSDXnpEpLllFGrddoVbA== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB2584.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.13; Mon, 14 Dec 2020 00:24:15 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::7d6b:aa68:78f4:5d94]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::7d6b:aa68:78f4:5d94%7]) with mapi id 15.20.3654.024; Mon, 14 Dec 2020 00:24:14 +0000 From: Rick Macklem To: J David , Konstantin Belousov CC: "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Thread-Topic: Major issues with nfsv4 Thread-Index: AQHWzw/HDat+dHoH9kKG5K3Xpd53kqnxDteQgAFi0QCAABTa84AALLCAgAAVvcmAAAu0AIAA4wiAgAAI5gCAAO/IAIABCutw Date: Mon, 14 Dec 2020 00:24:14 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 85b48805-9f30-4b41-3201-08d89fc69678 x-ms-traffictypediagnostic: YQXPR01MB2584: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4Vu8lszuwdBtctWd/WmCF+RAt+L1SNQO6YdtU2xmbmu+W58ExX22Qp9lcnA+N9ALnEN0BCWGzwOzn1ZOWwCW3uuxAFFTIDnCWMqpFdUwRCgksfYhWIC4W2mLZXuwu4EUr9Yy1lP4vMxqTITBpK9A84yPnjV4bAH2qbpzwPI8DqUxL+RVj2yGdy2R2rJUB9j95mUtpBJR3xhR2WDJIlaivoZijsftMwpztnZ9s4arEt5r+UjHAfbO5YE9gl6dRDLw3AIQ2eDRPH0CewrnsrmjZBIQWbd6pT5B6j4t0oUVe6P8qWdP2A87YrkZcHdndKMjnI5H2kS6LU62cjnYXMdkKIEOWQIIuaG0tv9NrMSkCtqWWJmgCPNoc8godUDkLHYSrAyK9RxUU7nlC/6kKazumVsPMZeV2oCA9y43GPUbR6fj+fZL7TdOwvETRVBkzPM2557kKZkSXwLTf/ugB3dtgA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(346002)(136003)(376002)(366004)(8936002)(110136005)(2906002)(55016002)(8676002)(66446008)(66616009)(66946007)(786003)(66556008)(66476007)(64756008)(9686003)(4326008)(99936003)(76116006)(52536014)(91956017)(966005)(508600001)(86362001)(33656002)(6506007)(83380400001)(7696005)(5660300002)(186003)(71200400001)(10126625002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?oPXauQBwLrW2EYtjIEK+uCDfLKN5IJnm3rFzpoIXE3MA59+U8G63xpBOor?= =?iso-8859-1?Q?RACnW+e/+O9ac0XkBxhZdZlN9KWHi9bEyP49mC37D+T08+v9jmAIrdVkSG?= =?iso-8859-1?Q?TijG3RfE0vLAFPmliXqmmboE8wvIlxhArnTPKffuJZbCBcInLfKqlDNiN1?= =?iso-8859-1?Q?JQGmBt8rSggC8OO9AXbSS9aC2/eaFQ3i4dHFqZztMFdyBv7wygsVXCQUZl?= =?iso-8859-1?Q?fFmHM5SgrLwGziNCmoF3eXeidAajT1xX/YJBXnjGul/tPelLaXucRtPbUT?= =?iso-8859-1?Q?MCpCsghB1ToKz/d7J+DVRO2Nm7E4h5/kpB+vOwMb6TG6nRZfzASzCBOl/M?= =?iso-8859-1?Q?QUN7cX4VKEnVLQgcakpYTFWrJnMPsNW4XpIMWeT98bCD4WJ4dZzCDQ7lXS?= =?iso-8859-1?Q?LNB2JEr0V4MBVTNtxH1aVeEw7YNYLuFnrtMTd5BE3i7vWZMilzlKvhAFRv?= =?iso-8859-1?Q?jVpcTalNR7p+EeQReXtDODlm49qhEZnsiGYJBusKiN8SH30LiKd/vu/eFZ?= =?iso-8859-1?Q?iK09NKtj8ug4HFLl4xMfXfrXqXGamy7CYPXS8HEhTwcm8L14JEgFq8cjRW?= =?iso-8859-1?Q?i5O5qcjmjYwzrSyD25PW567+sofad32B8LpInBMsXowuyNVVpx6RrNOiwI?= =?iso-8859-1?Q?ph6Bjlu9XxgEPw5ploXfyyNCL+oyPdsPRndtbcMt7OoUb8A5X+09L4y3Ae?= =?iso-8859-1?Q?M2zJYE+3x4q/iYpq+2mTohxISZgv8JqgTwHYGIqkumBs9ro1A7hHzfGBTj?= =?iso-8859-1?Q?lDv7JwT54D+W5krz4obT1eEO2VAcwiFs5XwcQ+x+7eo+4kDVw++ybCiOj2?= =?iso-8859-1?Q?S9dQ0d5GlQuym1noPsMlDi6yp2Cgu/nyMpRi96wY5iH3dUSx2ar1rnShKI?= =?iso-8859-1?Q?nWm7n0XFwBGSADtuRoN4si39LWrWdri2cMW0fYgc5MOJa0ohZQTHwr1OKS?= =?iso-8859-1?Q?kDUhwDr9vrdULf/xAOGuSs80bXCc/K/ELaR0cvgZutIghOWIWWijXTz1Ml?= =?iso-8859-1?Q?PEPt9Te6qdJJ98Oa735f5IIgtvXky49pd9hK6wSLyf9YNWgA2sMeAM0f5N?= =?iso-8859-1?Q?Id+9ybibTsiIswmc88MQm6Q=3D?= x-ms-exchange-transport-forked: True Content-Type: multipart/mixed; boundary="_002_YQXPR0101MB096834DB16439960ADAFCE29DDC70YQXPR0101MB0968_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 85b48805-9f30-4b41-3201-08d89fc69678 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Dec 2020 00:24:14.8806 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: IUeicf0UbjtuO3ps+8TRC3AawGSKAQjpXL+THF3lFZ56dXt0T216/LFvRHDuppyj/D4+v9MsZypRWsw7g9SQLg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB2584 X-Rspamd-Queue-Id: 4CvMZn0QKjz3hJ1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Bdl8FyBm; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::601 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5c::601:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5c::601:from:127.0.2.255]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Dec 2020 00:24:18 -0000 --_002_YQXPR0101MB096834DB16439960ADAFCE29DDC70YQXPR0101MB0968_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable J David wrote:=0A= [stuff snipped]=0A= >The most common trace (91 times, or over 1/4 of all observed) is:=0A= >=0A= >__mtx_lock_sleep+0xf8 nfscl_nodeleg+0x207 nfs_lookup+0x314=0A= >VOP_LOOKUP_APV+0x75 null_lookup+0x98 VOP_LOOKUP_APV+0x75 >lookup+0x451=0A= >namei+0x414 kern_statat+0x72 freebsd11_stat+0x30 amd64_syscall+0x387=0A= >fast_syscall_common+0xf8=0A= >=0A= >This trace appears roughly uniformly (28/38/34) in 3 of the 4 processes.= =0A= nfscl_nodeleg() plus two others acquire the mutex to check for a delegation= =0A= for the file. Servers should only issue delegations when the callback=0A= path is working and some never issue delegations.=0A= =0A= So, if you are not running the nfscbd(8) daemon (there is no need=0A= unless you are using delegations or pNFS) a trivial check for the=0A= callback path being enabled should determine that no delegation=0A= exists if it is not enabled. This avoids acquiring the mutex for this=0A= common case.=0A= --> You have no "Delegs" according to your "nfsstat -c -E".=0A= =0A= So, if you set up a test system with the attached patch applied to the=0A= kernel and you do not run nfscbd(8) (ie no nfscbd_enable=3D"YES" in=0A= your /etc/rc.conf) this patch might help.=0A= --> It avoids acquiring the mutex for lookups and getattrs (stat calls).=0A= =0A= The only reason I haven't done this before is a concern w.r.t. a broken=0A= NFSv4 server that erroneously issues delegations when the callback=0A= path is not working. (I don't know of such a server and, since you do not= =0A= have delegations, it should not be an issue for your case.)=0A= =0A= Might be enough to alleviate your problem.=0A= =0A= I will work on changing the single open linked list to a hash table of=0A= linked lists, but that is probably weeks away from a test patch.=0A= =0A= Let me know if you can try the patch and if it helps, rick=0A= =0A= The full set of traces (sorted by uniq -c) is here:=0A= =0A= https://pastebin.com/HUqkeMri=0A= =0A= (This message is already long enough!)=0A= =0A= The characters stripped off the front of each line are consistently:=0A= =0A= (pid) (uid) python3 -=0A= =0A= It happens these were all Python-based jobs. Python seems predisposed=0A= to trigger this, but non-Python jobs trigger it as well. Heavy use of=0A= stat() does seem to be a common element regardless of job type.=0A= =0A= Here's the output of "nfsstat -E -c" 60 seconds after running it with -z:= =0A= =0A= Client Info:=0A= RPC Counts:=0A= Getattr Setattr Lookup Readlink Read Wr= ite=0A= 1667 111 6376 0 42 = 153=0A= Create Remove Rename Link Symlink Mk= dir=0A= 111 7 14 0 0 = 0=0A= Rmdir Readdir RdirPlus Access Mknod Fss= tat=0A= 0 0 0 2620 0 = 160=0A= FSinfo pathConf Commit SetClId SetClIdCf L= ock=0A= 0 0 113 0 0 = 48=0A= LockT LockU Open OpenCfr=0A= 0 48 320 0=0A= OpenDownGr Close=0A= 0 402=0A= RelLckOwn FreeStateID PutRootFH DelegRet GetAcl Set= Acl=0A= 0 3 0 0 0 = 0=0A= ExchangeId CreateSess DestroySess DestroyClId LayoutGet GetDevI= nfo=0A= 0 0 0 0 0 = 0=0A= LayoutCommit LayoutReturn ReclaimCompl ReadDataS WriteDataS CommitDa= taS=0A= 0 0 0 0 0 = 0=0A= OpenLayout CreateLayout=0A= 0 0=0A= OpenOwner Opens LockOwner Locks Delegs Local= Own=0A= 21175 130439 30 6 0 = 0=0A= LocalOpen LocalLown LocalLock=0A= 0 0 0=0A= Rpc Info:=0A= TimedOut Invalid X Replies Retries Requests=0A= 0 0 0 0 12247=0A= Cache Info:=0A= Attr Hits Attr Misses Lkup Hits Lkup Misses=0A= 1110054 858 1002829 6361=0A= BioR Hits BioR Misses BioW Hits BioW Misses=0A= 2000 54 292 153=0A= BioRL Hits BioRL Misses BioD Hits BioD Misses=0A= 6911 0 208 0=0A= DirE Hits DirE Misses=0A= 104 0=0A= =0A= This does reflect the whole machine, not just those four processes.=0A= =0A= Finally, when I attempted to kill those four processes with ktrace=0A= running on all of them, the system panicked:=0A= =0A= Fatal trap 12: page fault while in kernel mode=0A= cpuid =3D 2; apic id =3D 02=0A= fault virtual address =3D 0x10=0A= fault code =3D supervisor read data, page not present=0A= instruction pointer =3D 0x20:0xffffffff8272d28f=0A= stack pointer =3D 0x28:0xfffffe008a5f24c0=0A= frame pointer =3D 0x28:0xfffffe008a5f25f0=0A= code segment =3D base 0x0, limit 0xfffff, type 0x1b=0A= =3D DPL 0, pres 1, long 1, def32 0, gran 1=0A= processor eflags =3D interrupt enabled, resume, IOPL =3D 0=0A= current process =3D 5627 (node)=0A= trap number =3D 12=0A= panic: page fault=0A= cpuid =3D 2=0A= time =3D 1607845622=0A= KDB: stack backtrace:=0A= db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008a5f2= 180=0A= vpanic() at vpanic+0x17b/frame 0xfffffe008a5f21d0=0A= panic() at panic+0x43/frame 0xfffffe008a5f2230=0A= trap_fatal() at trap_fatal+0x391/frame 0xfffffe008a5f2290=0A= trap_pfault() at trap_pfault+0x4f/frame 0xfffffe008a5f22e0=0A= trap() at trap+0x286/frame 0xfffffe008a5f23f0=0A= calltrap() at calltrap+0x8/frame 0xfffffe008a5f23f0=0A= --- trap 0xc, rip =3D 0xffffffff8272d28f, rsp =3D 0xfffffe008a5f24c0, rbp= =0A= =3D 0xfffffe008a5f25f0 ---=0A= null_bypass() at null_bypass+0xaf/frame 0xfffffe008a5f25f0=0A= VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0x80/frame 0xfffffe008a5f2620=0A= closef() at closef+0x8f/frame 0xfffffe008a5f26b0=0A= fdescfree_fds() at fdescfree_fds+0x3c/frame 0xfffffe008a5f2700=0A= fdescfree() at fdescfree+0x466/frame 0xfffffe008a5f27c0=0A= exit1() at exit1+0x488/frame 0xfffffe008a5f2820=0A= sigexit() at sigexit+0x159/frame 0xfffffe008a5f2b00=0A= postsig() at postsig+0x2fd/frame 0xfffffe008a5f2bc0=0A= ast() at ast+0x317/frame 0xfffffe008a5f2bf0=0A= doreti_ast() at doreti_ast+0x1f/frame 0x7fffffff67a0=0A= =0A= The fsck ate the ktrace.out file on reboot. This system does not have=0A= kernel debug symbols. If there's a way to try to figure out what=0A= null_bypass+0xaf corresponds to using the machine that built this=0A= kernel, which has the kernel, objects, debug symbols, and source,=0A= please let me know. It's built from releng/12.2 r368515.=0A= =0A= Thanks!=0A= --_002_YQXPR0101MB096834DB16439960ADAFCE29DDC70YQXPR0101MB0968_ Content-Type: application/octet-stream; name="clstate.patch" Content-Description: clstate.patch Content-Disposition: attachment; filename="clstate.patch"; size=912; creation-date="Mon, 14 Dec 2020 00:24:08 GMT"; modification-date="Mon, 14 Dec 2020 00:24:08 GMT" Content-Transfer-Encoding: base64 LS0tIHN5cy9mcy9uZnNjbGllbnQvbmZzX2Nsc3RhdGUuYy5zYXYJMjAyMC0xMi0xMyAwNjoxNzoz MC42NDI3NTIwMDAgLTA4MDAKKysrIHN5cy9mcy9uZnNjbGllbnQvbmZzX2Nsc3RhdGUuYwkyMDIw LTEyLTEzIDA2OjM4OjEwLjgxOTkwOTAwMCAtMDgwMApAQCAtNDM3NCw3ICs0Mzc0LDcgQEAgbmZz Y2xfbm9kZWxlZyh2bm9kZV90IHZwLCBpbnQgd3JpdGVkZWxlZykKIAogCW5wID0gVlRPTkZTKHZw KTsKIAlubXAgPSBWRlNUT05GUyh2cC0+dl9tb3VudCk7Ci0JaWYgKCFORlNIQVNORlNWNChubXAp KQorCWlmICghTkZTSEFTTkZTVjQobm1wKSB8fCBuZnNjbF9lbmFibGVjYWxsYiA9PSAwKQogCQly ZXR1cm4gKDEpOwogCU5GU0xPQ0tDTFNUQVRFKCk7CiAJY2xwID0gbmZzY2xfZmluZGNsKG5tcCk7 CkBAIC00NzMwLDcgKzQ3MzAsNyBAQCBuZnNjbF9kZWxlZ21vZHRpbWUodm5vZGVfdCB2cCkKIAlz dHJ1Y3QgbmZzbW91bnQgKm5tcDsKIAogCW5tcCA9IFZGU1RPTkZTKHZwLT52X21vdW50KTsKLQlp ZiAoIU5GU0hBU05GU1Y0KG5tcCkpCisJaWYgKCFORlNIQVNORlNWNChubXApIHx8IG5mc2NsX2Vu YWJsZWNhbGxiID09IDApCiAJCXJldHVybjsKIAlORlNMT0NLQ0xTVEFURSgpOwogCWNscCA9IG5m c2NsX2ZpbmRjbChubXApOwpAQCAtNDc1OSw3ICs0NzU5LDcgQEAgbmZzY2xfZGVsZWdnZXRtb2R0 aW1lKHZub2RlX3QgdnAsIHN0cnVjdCB0aW1lc3BlYyAqbXRpCiAJc3RydWN0IG5mc21vdW50ICpu bXA7CiAKIAlubXAgPSBWRlNUT05GUyh2cC0+dl9tb3VudCk7Ci0JaWYgKCFORlNIQVNORlNWNChu bXApKQorCWlmICghTkZTSEFTTkZTVjQobm1wKSB8fCBuZnNjbF9lbmFibGVjYWxsYiA9PSAwKQog CQlyZXR1cm47CiAJTkZTTE9DS0NMU1RBVEUoKTsKIAljbHAgPSBuZnNjbF9maW5kY2wobm1wKTsK --_002_YQXPR0101MB096834DB16439960ADAFCE29DDC70YQXPR0101MB0968_--