From owner-freebsd-fs@freebsd.org Wed May 15 15:06:24 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CAC41594E62 for ; Wed, 15 May 2019 15:06:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660058.outbound.protection.outlook.com [40.107.66.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AB918F6EE for ; Wed, 15 May 2019 15:06:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQXPR0101MB2262.CANPRD01.PROD.OUTLOOK.COM (52.132.80.151) by YQXPR0101MB2102.CANPRD01.PROD.OUTLOOK.COM (52.132.77.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1878.22; Wed, 15 May 2019 15:06:19 +0000 Received: from YQXPR0101MB2262.CANPRD01.PROD.OUTLOOK.COM ([fe80::44dc:5f27:855:3fde]) by YQXPR0101MB2262.CANPRD01.PROD.OUTLOOK.COM ([fe80::44dc:5f27:855:3fde%3]) with mapi id 15.20.1900.010; Wed, 15 May 2019 15:06:19 +0000 From: Rick Macklem To: Ganbold Tsagaankhuu , "freebsd-fs@freebsd.org" Subject: Re: NFS file access log Thread-Topic: NFS file access log Thread-Index: AQHVCvMq7+IVJ0Zr30q7KplvIoFLsaZsRxZg Date: Wed, 15 May 2019 15:06:19 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ea269d1b-9259-4b57-af67-08d6d946e2ca x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(7193020); SRVR:YQXPR0101MB2102; x-ms-traffictypediagnostic: YQXPR0101MB2102: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-forefront-prvs: 0038DE95A2 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(346002)(366004)(39860400002)(136003)(376002)(199004)(189003)(102836004)(66476007)(66946007)(76116006)(73956011)(64756008)(6506007)(486006)(305945005)(66556008)(66446008)(76176011)(476003)(478600001)(99286004)(446003)(25786009)(3480700005)(110136005)(86362001)(46003)(14454004)(53936002)(81166006)(81156014)(186003)(6246003)(2906002)(8936002)(8676002)(68736007)(316002)(786003)(74482002)(71200400001)(71190400001)(229853002)(2501003)(11346002)(33656002)(7696005)(4744005)(55016002)(6436002)(52536014)(256004)(5660300002)(14444005)(9686003)(74316002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQXPR0101MB2102; H:YQXPR0101MB2262.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: uXJzuiTXONMIg/lAwmKMl9VlBrSPPPW8P8qjCdFQn7WyWHkolEEWqONnaWwHzFuft0oDyBnTrBU46iP6+TOZSSukXKA/bkhfYrNbaBnrDXKzFQBH0g04vZv796AD8nGzf9rCb9qdq2XFKN4gEd8uwPW9NCv3+PLK19ByS+qN2EVeGJffoSMHPJqvL8glRJLKE4jlnIIUOZMCq40mozg1WT9X1BHq75t4IJA4h5rS8yb1iL27oDNiRC/PpMwfePJkLmXJzCxr9oX4UYu2OkRT/Rbn9Fm46HLnNyULPcJc7QWQWvXSsDXuhIQJ3WwFX4TRap6f1cyFP4cbzj4vlZeq0L1BSBDiiszDZlA4Q/PDv+5Nrsnh+DZTzin1lsbHJqypukHZYTBbB4ntS9t6zeyHzD45eyVsV+ZpVwp0n0tl/hU= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: ea269d1b-9259-4b57-af67-08d6d946e2ca X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2019 15:06:19.4806 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB2102 X-Rspamd-Queue-Id: 2AB918F6EE X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.58 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-2.76 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.981,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-0.99)[-0.985,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[58.66.107.40.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.49)[-0.485,0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2019 15:06:24 -0000 Ganbold Tsagaankhuu wrote: >Is there anyway to log file access in NFS client/server side? >I'm experiencing following problem: > >vm_fault: pager read error, pid 75363 (php-fpm) >ncl_getpages: error 70 > >and would like to know if there is possibility to find problematic files. I haven't used DTrace, so I can't help with that. However, "70" is ESTALE and that means that the file has been deleted on th= e server, so the client can't read it. Look for a case where an application running on another client (or locally = on the server) would delete a file that would be accessed by php-fpm on this clien= t. (If you know about when it happens, you could do a raw packet capture at th= e server using "tcpdump -s 0 -w find.pcap" and then look at the packet captu= re in wireshark, looking for where file(s) get removed. It won't be easy for = a busy server.) rick