Date: Sun, 17 Nov 2024 04:10:52 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: freebsd-fs <fs@freebsd.org> Subject: Re: tmpfs loses (sub-page chunks of) data? Message-ID: <ZzlQrIlOYAMkh13t@kib.kiev.ua> In-Reply-To: <140eb994-ae19-41b4-8f0e-fc4290603ce0@freebsd.org> References: <a301f26f-0c67-48c3-af08-5f36c7ce764f@FreeBSD.org> <ZzfKJhrgbwq6CawR@kib.kiev.ua> <140eb994-ae19-41b4-8f0e-fc4290603ce0@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Nov 16, 2024 at 02:31:25PM +0200, Andriy Gapon wrote: > On 16/11/2024 00:24, Konstantin Belousov wrote: > > On Fri, Nov 15, 2024 at 02:43:22PM +0200, Andriy Gapon wrote: > > > > > > We have a number of servers based on FreeSBD 13.3 that initially write some > > > data to files on tmpfs and then the files are dispatched elsewhere. The > > > writes are done by appending variable sized records to a file. There are no > > > seeks or overwrites. > > > > > > I observe that occasionally (very rarely indeed given the amount of data > > > produced) we get a corrupted file. > > > > > > In all cases so far the corruption follows the same pattern: data range from > > > the end of a record until the next page-aligned boundary is zeroed out. > > > That is, good data always continues from an offset which is multiple of 4096 > > > and the zeroed area never crosses such offsets. > > > > > > Because of the page boundary, I have a suspicion that either tmpfs or, > > > perhaps, the broader VM subsystem might have a race where writing to a page > > > does not mark it dirty. Maybe this is related to paging out of a tmpfs page > > > to the swap. > > > > > > The problem is that I have never been able to observe this happening, the > > > corruption gets detected after the fact, hours after it occurs. > > > > > > If anyone could suggest any areas / changes / techniques to explore the > > > problem, I would be much obliged. > > > > Do you have swap enabled on the problematic machines? > > Yes. > > > Are the files mapped, do you write or read through map? > > I think that mmap is not involved at all. > Files are written from kernel using kern_writev(). Why? > After they are complete, they are read from userland using whatever libssl > uses to read input files (when encrypting). Looks like that's fread(3) / > read(2). Try to reproduce it on the HEAD kernel. Then, turn off swap and see if it is still there. On HEAD, if reproducable without swap, try to add 'pgread' option to the tmpfs moount. I am interested if the issue persists after all the measures listed above.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ZzlQrIlOYAMkh13t>