From owner-freebsd-hackers@freebsd.org Mon Mar 27 16:33:52 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23F38D202A3 for ; Mon, 27 Mar 2017 16:33:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6336ECE for ; Mon, 27 Mar 2017 16:33:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22d.google.com with SMTP id w11so49850673wrc.3 for ; Mon, 27 Mar 2017 09:33:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=CKBdDx+MF0i+Gx90tMikTVPxzLLyyOCYhaDw62Oa1fc=; b=06Q0KcFNh1MsX3NsAf5rdCJGx7xoHuR/kpSEdS9LT0P1oh1gVDyEwodAGgmHn7sCup mtzCLATNnX763WObPF7YWRvrnV/n5/9ElH0jDItCRLGcgXIiXsGaxp+x56ogUjKGogQ1 2Qo9iE7ZSLvDfReJh3ItVN8E66UiZRtstJR2BrwMynS+yPUIuFFuQ/JaLb0xdDPL3v/e hvYsw2eEL/1mZOEQiusxll1v2m9fLaViWpZraMsqRZeRYcB6GYTou8VF9LoZYlzEcWuq MqnEv2MkBTwT0tKMbc23Q4NC4GT78yeWBrAR3l+BifA9UXCHbTbt7QNnl89YVOiRUcYS kvFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=CKBdDx+MF0i+Gx90tMikTVPxzLLyyOCYhaDw62Oa1fc=; b=Lo8AfL9MVQOCioPneuX32mzQrarnC3e2tv2W6AdP9m9imYGDlhYYA0TZdKI4fqNVdA PgPC05byRi22fP8CGvIAW4ZH4GEE5bUV2NJ5/hOdE0XnDGN3Dv2KPq9UPPKRO8RuHgYj +z9/qNdHB4w2CIeIHqT+LdPg6WEDz0DH4ew8MYCucP1FYdshBYK70GMBEF0YCZ6Sduc9 ygS0AhTGYryJF0W8yvDQ+ugm7JLU4Lb4tKTgP3Q1lYgSy4ESsGayQCjasyBbbEoSYS4s JvV8dSMP4f9FHhCoLmCiCxTk50fTn1LZj6KsJQJhJujCnEdXZ4TywNhO8w6WJPukrOUq /5WA== X-Gm-Message-State: AFeK/H1LNRzj2PtvQJMa265Pe2GV1WvrNv/My5vqUZGEBBi5qamRk3/qrOeqA0u1NdpU8RcE X-Received: by 10.28.101.68 with SMTP id z65mr10433251wmb.102.1490632428750; Mon, 27 Mar 2017 09:33:48 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id b13sm110026wmf.6.2017.03.27.09.33.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Mar 2017 09:33:48 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20161206125919.GQ54029@kib.kiev.ua> <8b502580-4d2d-1e1f-9e05-61d46d5ac3b1@multiplay.co.uk> <20161206143532.GR54029@kib.kiev.ua> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> Date: Mon, 27 Mar 2017 17:33:49 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170327161833.GL43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 16:33:52 -0000 On 27/03/2017 17:18, Konstantin Belousov wrote: > On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote: >> OK now the similar but unrelated issue with signal stacks is solved I've >> moved back to the initial issue. >> >> I've made some progress with a reproduction case as detailed here: >> https://github.com/golang/go/issues/15658#issuecomment-288747812 >> >> In short it seems that having a running child, while the parent runs GC, >> is some how responsible for memory corruption in the parent. >> >> The reason I believe this is if I run the same GC in the parent after >> the child exits instead of while its running, I've been unable to >> reproduce the issue. >> >> As the memory segments are COW then the issue might be in VM subsystem. > Well, it might be, but it is a strange corruption mode to believe. Indeed, but would you agree the evidence seems to indicate that this may be the case, as otherwise I would have expected that running the GC after the child process has exited would have zero impact on the issue. > >> In order to confirm / deny this I was wondering if there was a way to >> force a full copy of all segments for the child instead of using the COW >> optimisation. > No, there is no. By design, copying only occurs on faults, when VM > detects that the map entry needs copying. Doing the actual copy at fork > time would require writing a lot of new code. I noticed in vm_map_copy_entry the following: /* * We don't want to make writeable wired pages copy-on-write. * Immediately copy these pages into the new map by simulating * page faults. The new pages are pageable. */ vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry, fork_charge); I wondered if I could use vm_fault_copy_entry to force the copy on fork? > Does go have FreeBSD/i386 port ? If yes, is the issue reproducable there ? Yes it does, I don't currently have i386 machine to test with, I'm assuming testing i386 on amd64 kernel, would likely not have any effect. > Another blind experiment to try is to comment out call to > vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if > it changes anything. I'll do that shortly. > What could be quite interesting is to look at the parent and possibly > child address map after the error occured, using procstat -v. At > least for parent, this should be relatively easy to set up, just make > go runtime spin or pause on panic, instead of exiting, and then use > procstat. I've been looking at the output from procstat -v I have seen the parent FLAGS ping ping between C--- and CN--, not sure if that's relevant e.g. procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 3 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 3 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 18 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 38 1 0 C--- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 1 0 C--- df 27099 0x800c1e000 0x800c6e000 rw- 18 18 1 0 C--- df 27099 0x800c6e000 0x800cae000 rw- 2 2 1 0 C--- df 27099 0x800cae000 0x800cee000 rw- 2 2 1 0 C--- df 27099 0x800cee000 0x800dae000 rw- 5 5 1 0 C--- df 27099 0x800dae000 0x800dee000 rw- 1 1 1 0 C--- df 27099 0x800dee000 0x800e2e000 rw- 1 1 1 0 C--- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 1 0 C--- df 27099 0x800e6e000 0x800eae000 rw- 1 1 1 0 C--- df 27099 0xc000000000 0xc000001000 rw- 1 1 1 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 1 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 37 0 ---- ph procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 38 2 0 CN-- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df 27099 0x800c1e000 0x800c6e000 rw- 18 18 2 0 CN-- df 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df 27099 0x800dee000 0x800e2e000 rw- 1 1 2 0 CN-- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 255 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 CN-D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph procstat -v 27099 PID START END PRT RES PRES REF SHD FLAG TP PATH 27099 0x400000 0x70d000 r-x 309 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x70d000 0x94e000 r-- 270 635 5 1 CN-- vn /root/golang/src/test5/test5 27099 0x94e000 0x985000 rw- 55 0 1 0 C--- vn /root/golang/src/test5/test5 27099 0x985000 0x9a8000 rw- 18 0 1 0 C--- df 27099 0x80094e000 0x800b4e000 rw- 38 0 1 0 C--- df 27099 0x800b4e000 0x800c1e000 rw- 28 28 2 0 CN-- df 27099 0x800c1e000 0x800c6e000 rw- 18 0 1 0 C--- df 27099 0x800c6e000 0x800cae000 rw- 2 2 2 0 CN-- df 27099 0x800cae000 0x800cee000 rw- 2 2 2 0 CN-- df 27099 0x800cee000 0x800dae000 rw- 5 5 2 0 CN-- df 27099 0x800dae000 0x800dee000 rw- 1 1 2 0 CN-- df 27099 0x800dee000 0x800e2e000 rw- 1 0 1 0 C--- df 27099 0x800e2e000 0x800e6e000 rw- 1 1 2 0 CN-- df 27099 0x800e6e000 0x800eae000 rw- 1 1 2 0 CN-- df 27099 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df 27099 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df 27099 0xc41fff8000 0xc420200000 rw- 255 0 1 0 C--- df 27099 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 1 0 C--D df 27099 0x7ffffffff000 0x800000000000 r-x 1 1 38 0 ---- ph I'll definitely try capturing the output on fault, see what that looks like > >> Is this something that would be relatively easy to hack into the kernel, >> and if so pointers would be appreciated. > BTW, I looked some more at the go code, and I noted that > runtimemmap() implementation looks very strange. > It ignores %rflags.C bit to identify error, and instead callers > of mmap() compare the return value with 4096, assuming Linux-style > error reporting. This would certainly break if mmap(2) syscall > returns ERESTART one day. I'll look at this too, thanks for the heads up. Regards Steve