Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 27 Mar 2017 17:33:49 +0100
From:      Steven Hartland <killing@multiplay.co.uk>
To:        Konstantin Belousov <kostikbel@gmail.com>
Cc:        "K. Macy" <kmacy@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Subject:   Re: Help needed to identify golang fork / memory corruption issue on FreeBSD
Message-ID:  <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk>
In-Reply-To: <20170327161833.GL43712@kib.kiev.ua>
References:  <20161206125919.GQ54029@kib.kiev.ua> <8b502580-4d2d-1e1f-9e05-61d46d5ac3b1@multiplay.co.uk> <20161206143532.GR54029@kib.kiev.ua> <e160381c-9935-6edf-04a9-1ff78e95d818@multiplay.co.uk> <CAHM0Q_Mg662u9D0KJ9knEWWqi9Ydy38qKDnjLt6XaS0ks%2B9-iw@mail.gmail.com> <18b40a69-4460-faf2-c0ce-7491eca92782@multiplay.co.uk> <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua>

next in thread | previous in thread | raw e-mail | index | archive | help
On 27/03/2017 17:18, Konstantin Belousov wrote:
> On Mon, Mar 27, 2017 at 12:47:11PM +0100, Steven Hartland wrote:
>> OK now the similar but unrelated issue with signal stacks is solved I've
>> moved back to the initial issue.
>>
>> I've made some progress with a reproduction case as detailed here:
>> https://github.com/golang/go/issues/15658#issuecomment-288747812
>>
>> In short it seems that having a running child, while the parent runs GC,
>> is some how responsible for memory corruption in the parent.
>>
>> The reason I believe this is if I run the same GC in the parent after
>> the child exits instead of while its running, I've been unable to
>> reproduce the issue.
>>
>> As the memory segments are COW then the issue might be in VM subsystem.
> Well, it might be, but it is a strange corruption mode to believe.
Indeed, but would you agree the evidence seems to indicate that this may 
be the case, as otherwise I would have expected that running the GC 
after the child process has exited would have zero impact on the issue.
>
>> In order to confirm / deny this I was wondering if there was a way to
>> force a full copy of all segments for the child instead of using the COW
>> optimisation.
> No, there is no. By design, copying only occurs on faults, when VM
> detects that the map entry needs copying. Doing the actual copy at fork
> time would require writing a lot of new code.
I noticed in vm_map_copy_entry the following:
                 /*
                  * We don't want to make writeable wired pages 
copy-on-write.
                  * Immediately copy these pages into the new map by 
simulating
                  * page faults.  The new pages are pageable.
                  */
                 vm_fault_copy_entry(dst_map, src_map, dst_entry, src_entry,
                     fork_charge);

I wondered if I could use vm_fault_copy_entry to force the copy on fork?
> Does go have FreeBSD/i386 port ?  If yes, is the issue reproducable there ?
Yes it does, I don't currently have i386 machine to test with, I'm 
assuming testing i386 on amd64 kernel, would likely not have any effect.
> Another blind experiment to try is to comment out call to
> vm_object_collapse() in sys/vm/vm_map.c:vm_map_copy_entry() and see if
> it changes anything.
I'll do that shortly.
> What could be quite interesting is to look at the parent and possibly
> child address map after the error occured, using procstat -v. At
> least for parent, this should be relatively easy to set up, just make
> go runtime spin or pause on panic, instead of exiting, and then use
> procstat.
I've been looking at the output from procstat -v I have seen the parent 
FLAGS ping ping between C--- and CN--, not sure if that's relevant e.g.
procstat -v 27099
   PID              START                END PRT  RES PRES REF SHD FLAG 
TP PATH
27099           0x400000           0x70d000 r-x  309  635 3   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x70d000           0x94e000 r--  270  635 3   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x94e000           0x985000 rw-   55    0 1   0 C--- vn 
/root/golang/src/test5/test5
27099           0x985000           0x9a8000 rw-   18   18 1   0 C--- df
27099        0x80094e000        0x800b4e000 rw-   38   38 1   0 C--- df
27099        0x800b4e000        0x800c1e000 rw-   28   28 1   0 C--- df
27099        0x800c1e000        0x800c6e000 rw-   18   18 1   0 C--- df
27099        0x800c6e000        0x800cae000 rw-    2    2 1   0 C--- df
27099        0x800cae000        0x800cee000 rw-    2    2 1   0 C--- df
27099        0x800cee000        0x800dae000 rw-    5    5 1   0 C--- df
27099        0x800dae000        0x800dee000 rw-    1    1 1   0 C--- df
27099        0x800dee000        0x800e2e000 rw-    1    1 1   0 C--- df
27099        0x800e2e000        0x800e6e000 rw-    1    1 1   0 C--- df
27099        0x800e6e000        0x800eae000 rw-    1    1 1   0 C--- df
27099       0xc000000000       0xc000001000 rw-    1    1 1   0 CN-- df
27099       0xc41fff0000       0xc41fff8000 rw-    3    3 1   0 CN-- df
27099       0xc41fff8000       0xc420200000 rw-  255  255 1   0 C--- df
27099     0x7ffffffdf000     0x7ffffffff000 rwx    2    2 1   0 C--D df
27099     0x7ffffffff000     0x800000000000 r-x    1    1 37   0 ---- ph

procstat -v 27099
   PID              START                END PRT  RES PRES REF SHD FLAG 
TP PATH
27099           0x400000           0x70d000 r-x  309  635 5   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x70d000           0x94e000 r--  270  635 5   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x94e000           0x985000 rw-   55    0 1   0 C--- vn 
/root/golang/src/test5/test5
27099           0x985000           0x9a8000 rw-   18    0 1   0 C--- df
27099        0x80094e000        0x800b4e000 rw-   38   38 2   0 CN-- df
27099        0x800b4e000        0x800c1e000 rw-   28   28 2   0 CN-- df
27099        0x800c1e000        0x800c6e000 rw-   18   18 2   0 CN-- df
27099        0x800c6e000        0x800cae000 rw-    2    2 2   0 CN-- df
27099        0x800cae000        0x800cee000 rw-    2    2 2   0 CN-- df
27099        0x800cee000        0x800dae000 rw-    5    5 2   0 CN-- df
27099        0x800dae000        0x800dee000 rw-    1    1 2   0 CN-- df
27099        0x800dee000        0x800e2e000 rw-    1    1 2   0 CN-- df
27099        0x800e2e000        0x800e6e000 rw-    1    1 2   0 CN-- df
27099        0x800e6e000        0x800eae000 rw-    1    1 2   0 CN-- df
27099       0xc000000000       0xc000001000 rw-    1    1 2   0 CN-- df
27099       0xc41fff0000       0xc41fff8000 rw-    3    3 2   0 CN-- df
27099       0xc41fff8000       0xc420200000 rw-  255  255 1   0 C--- df
27099     0x7ffffffdf000     0x7ffffffff000 rwx    2    2 1   0 CN-D df
27099     0x7ffffffff000     0x800000000000 r-x    1    1 38   0 ---- ph

procstat -v 27099
   PID              START                END PRT  RES PRES REF SHD FLAG 
TP PATH
27099           0x400000           0x70d000 r-x  309  635 5   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x70d000           0x94e000 r--  270  635 5   1 CN-- vn 
/root/golang/src/test5/test5
27099           0x94e000           0x985000 rw-   55    0 1   0 C--- vn 
/root/golang/src/test5/test5
27099           0x985000           0x9a8000 rw-   18    0 1   0 C--- df
27099        0x80094e000        0x800b4e000 rw-   38    0 1   0 C--- df
27099        0x800b4e000        0x800c1e000 rw-   28   28 2   0 CN-- df
27099        0x800c1e000        0x800c6e000 rw-   18    0 1   0 C--- df
27099        0x800c6e000        0x800cae000 rw-    2    2 2   0 CN-- df
27099        0x800cae000        0x800cee000 rw-    2    2 2   0 CN-- df
27099        0x800cee000        0x800dae000 rw-    5    5 2   0 CN-- df
27099        0x800dae000        0x800dee000 rw-    1    1 2   0 CN-- df
27099        0x800dee000        0x800e2e000 rw-    1    0 1   0 C--- df
27099        0x800e2e000        0x800e6e000 rw-    1    1 2   0 CN-- df
27099        0x800e6e000        0x800eae000 rw-    1    1 2   0 CN-- df
27099       0xc000000000       0xc000001000 rw-    1    1 2   0 CN-- df
27099       0xc41fff0000       0xc41fff8000 rw-    3    3 2   0 CN-- df
27099       0xc41fff8000       0xc420200000 rw-  255    0 1   0 C--- df
27099     0x7ffffffdf000     0x7ffffffff000 rwx    2    2 1   0 C--D df
27099     0x7ffffffff000     0x800000000000 r-x    1    1 38   0 ---- ph

I'll definitely try capturing the output on fault, see what that looks like
>
>> Is this something that would be relatively easy to hack into the kernel,
>> and if so pointers would be appreciated.
> BTW, I looked some more at the go code, and I noted that
> runtime<stupid UTF-8 char>mmap() implementation looks very strange.
> It ignores %rflags.C bit to identify error, and instead callers
> of mmap() compare the return value with 4096, assuming Linux-style
> error reporting.  This would certainly break if mmap(2) syscall
> returns ERESTART one day.
I'll look at this too, thanks for the heads up.

     Regards
     Steve



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3ec35a46-ae70-35cd-29f8-82e7cebb0eb6>