From owner-freebsd-fs@freebsd.org Wed Mar 6 21:31:57 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF99215280F2 for ; Wed, 6 Mar 2019 21:31:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B46B571A3A for ; Wed, 6 Mar 2019 21:31:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id 6D3DC15280F1; Wed, 6 Mar 2019 21:31:56 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4838A15280EF for ; Wed, 6 Mar 2019 21:31:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670041.outbound.protection.outlook.com [40.107.67.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB0A871A29; Wed, 6 Mar 2019 21:31:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from QB1PR01MB3537.CANPRD01.PROD.OUTLOOK.COM (52.132.89.15) by QB1PR01MB3410.CANPRD01.PROD.OUTLOOK.COM (52.132.87.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.19; Wed, 6 Mar 2019 21:31:50 +0000 Received: from QB1PR01MB3537.CANPRD01.PROD.OUTLOOK.COM ([fe80::89d1:c1e9:cb2:3e66]) by QB1PR01MB3537.CANPRD01.PROD.OUTLOOK.COM ([fe80::89d1:c1e9:cb2:3e66%3]) with mapi id 15.20.1665.021; Wed, 6 Mar 2019 21:31:50 +0000 From: Rick Macklem To: "bugzilla-noreply@freebsd.org" , "fs@FreeBSD.org" Subject: Re: [Bug 235774] [FUSE]: Need to evict invalidated cache contents on fuse_write_directbackend() Thread-Topic: [Bug 235774] [FUSE]: Need to evict invalidated cache contents on fuse_write_directbackend() Thread-Index: AQHU1ECgXfJCGSEOt0KosKRIiQNXUKX/HJja Date: Wed, 6 Mar 2019 21:31:50 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e6bcda7a-53d9-4278-a408-08d6a27b24d0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:QB1PR01MB3410; x-ms-traffictypediagnostic: QB1PR01MB3410: x-microsoft-exchange-diagnostics: =?Windows-1252?Q?1; QB1PR01MB3410; 23:ye5B7+dpwGG2nntxqxHyEtvYJmnw8QBH06xJa?= =?Windows-1252?Q?AB8AoHRB31xI/HcFGyr+nYuuL5Z8IKqN8L1U9/ss58j+LN4UAyVXxRiJ?= =?Windows-1252?Q?Wqj4R1wAYDRrZfVHPjuVZ2zvYTXwx1+sjZr3E2/a4Y+R47BE2P7PJZFy?= =?Windows-1252?Q?rUmsE/QYXehgpZvaJJgBK94irG3+4pK2xgaLt7vRVVAI92nwcAMutxp1?= =?Windows-1252?Q?T/KuwCHYz3Zxv5ZfKXDYRV0pgdkGzvKr21/xY6Y9zL8VEh9qc4w0fmPT?= =?Windows-1252?Q?vvNUsdPT+95M7m3owWdt29RD2yf8XisbfB7m1jnF+qqip5bdbLrVeuWV?= =?Windows-1252?Q?nBlJPnnu0kduFq09Go+e38wECA9WSER7IiR+vuew0I2lEaGtdih48Ddu?= =?Windows-1252?Q?JO/sa2Vu8qkEYhKppU2Xd63C2bc+rcYsn6NykDjw6Ex+RONvzMmntdlB?= =?Windows-1252?Q?8VKlk5vMeEcrxupJtor44VYwzxdXRUqMFJA6jbwXVm++6wLUdZ1jieEx?= =?Windows-1252?Q?w2GJvxbUUl7/SRBnYhw+6+ya9L90V+Ii542G0MMzW5qpEJ5sCJTndFbB?= =?Windows-1252?Q?V+sEZli7c1dLNkeDJGiZfXOQWr6fGIaCKrEuKlGXY+z2lwQA0jITSP1P?= =?Windows-1252?Q?cNBDSinbxrGR/9HhkMOIrMBicEOUt0bVAOkKnRZEfekQYafzp5jMrR3C?= =?Windows-1252?Q?9dKvVkISW1W3cThD6FvYdSkYX5RQCcLwPpPOxOqky1QQxuy3EZV3NwgW?= =?Windows-1252?Q?oUNLTdL67RsR5TsvpOP+OmuI0CqQ14c29evJiWQKZYuS/lRnJlXq7nZe?= =?Windows-1252?Q?yMEIAyeFOlOAGmf6fDamfC5k4PM+C+Qf4b/HaZav5cPoUrQvhgXBY6jo?= =?Windows-1252?Q?V6S0BoFT5gmsiznls4t6KvLfSEikrYFx104hURGihaAxc+XfIytyfqr5?= =?Windows-1252?Q?dKDYVuO6x0UgUyNG7Ke8xQ3BQogG81/sJdkJ3fSLE4+e4WvV+yfxWp4T?= =?Windows-1252?Q?dNxT46KREenldJPtc4coxyo+4xuNWjTXckkvFeihHMuBbERbZ2HAep0K?= =?Windows-1252?Q?wDJXQ7iPQnu99X6Ih5Yy4aSFsYJK0pSVw+gFz6eb9WC2pePGCyn+QODa?= =?Windows-1252?Q?dbDNSa4n5Qx9HXbr1FiK8BN0CF1YHdb8or3AaHRK5s1C11bss0hemGE1?= =?Windows-1252?Q?8abVt2YO2phTGCvNITQETbFMiahTm7+f5j9jCos6E7QGNpE2XPP5ZFai?= =?Windows-1252?Q?5hTppEsUdTLCaL5lzy96mkGcXo7o9HKAjiZO2o=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 0968D37274 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(39850400004)(376002)(366004)(396003)(136003)(199004)(189003)(52314003)(2906002)(99286004)(74316002)(7696005)(102836004)(9686003)(33656002)(6436002)(316002)(97736004)(305945005)(6506007)(55016002)(786003)(6246003)(76176011)(81156014)(53936002)(229853002)(14454004)(81166006)(8676002)(86362001)(186003)(46003)(486006)(71190400001)(478600001)(106356001)(446003)(105586002)(25786009)(11346002)(5660300002)(68736007)(2501003)(476003)(8936002)(71200400001)(110136005)(450100002)(74482002)(256004); DIR:OUT; SFP:1101; SCL:1; SRVR:QB1PR01MB3410; H:QB1PR01MB3537.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 0fcdidLaoqxRnPk5Z8vzhpuTCId+2h1QhHJPsRP0DbGoxf+R2dto6wuOdB5LMAknTkShP+qOKMZqvOxP6etIrbn1D/Z8VYsp/iC3FVlwylBnCkquWbN1zw6bVW8lL4pXc6l/7Sa9HLxYux1sU5L8wd0SAlp/0lKRvaBM9TPio/IHoETDng3nhfRlg++vUi5rY1dnWBlxPPYrs21e/ypCe9cT87u8n9LT7WpSDOVc4H8bLit7TiuPNKiwBvFbxSI8iXDIfXSm1/5H4T4ovRVz7XqPlMRZ6i4XJRb0Dy5jiYohhgHUkWMEpkEzUoC1NZP7lCNgh6WBbTpJB4pqo8ND+uIjk/L92q8OUZQSHJDZX6LTSkvm3IP0cbtqaD22pcgcns2xK+6cr6cwhgpR2xzuXZyIYsU/kXZ0pGiXg+Ru3RY= Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: e6bcda7a-53d9-4278-a408-08d6a27b24d0 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Mar 2019 21:31:50.0943 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3410 X-Rspamd-Queue-Id: BB0A871A29 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.972,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2019 21:31:57 -0000 --- Comment #4 from Conrad Meyer --- >I think fuse's IO_DIRECT path is a mess. Really all IO should go through = the >buffer cache, and B_DIRECT and ~B_CACHE are just flags that control the >buffer's lifetime once the operation is complete. Removing the "direct" >backends entirely (except as implementation details of strategy()) would >simplify and correct the caching logic. Hmm, I'm not sure that I agree that all I/O should go through the buffer ca= che, in general. (I won't admit to knowing the fuse code well enough to comment specifically on it.) There is a code path on the NFS client that bypasses the buffer cache for O= _DIRECT, but it isn't enabled by default, so I doubt many use it. (It is enabled via= a sysctl which defaults to 0.) I can see an argument for enabling it, since having the NFS (or FUSE) clien= t do a large amount of writing to a file can flood the buffer cache and avoiding t= his for the case where the client won't be reading the file would be nice. What I am not sure is whether O_DIRECT is a good indicator of "doing a lot = of writing that won't be read back". >Looking at UFS; it really only has a non-bufcache "rawread" path that uses >pbufs (and flushes all dirty bufs on the vnode first!). There is no equiv= alent >for O_DIRECT writes. And ffs_rawread basically duplicates the ordinary re= ad >path for extremely limited cases (single iov, must be sector sized/aligned= , >etc) =97 it's unclear to me why it exists. > >ffs_write() just uses the ordinary buf cache, paying attention to ioflag & >IO_DIRECT and using vfs_bio_set_flags(, ioflag) to propagate it to b_flags= & >B_DIRECT. (B_DIRECT causes the buffer to be released immediately when it = is >freed, instead of being cached.) > >I think we should probably learn from UFS for FUSE's IO modes: > >1. Keep and enable the direct_io option, for users who truly want to bypas= s the >buf cache entirely. Preferably this is a per-mountpoint option rather tha= n a >global, but that is an orthogonal enhancement. Confusingly, this is disti= nct >from opening a file O_DIRECT. Maybe the sysctl/option can be renamed. "r= aw >io?" > >2. Do not actually use the "direct" paths in FUSE outside of global direct= _io >mode (or a future MP-specific always-direct mode). > >3. A caveat here is: FUSE filesystems (?)don't have a native sector/block = size, >but the buf cache is in block units. And, we translate O_WRONLY opens int= o >FUSE FUFH_WRONLY opens. So there will be some trickiness in partial block >writes with a O_WRONLY handle when the block is not in cache. Today that = is >sidestepped by invoking direct mode, but shouldn't be. > >Anyway, this is all future cleanup ideas for this area. For the more limi= ted >scope of fixing just this PR, we can probably draw inspiration from >ffs_rawread_sync(). rick