From owner-freebsd-hackers@freebsd.org Tue Dec 6 12:59:30 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 042E1C6954F for ; Tue, 6 Dec 2016 12:59:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7694117BE for ; Tue, 6 Dec 2016 12:59:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uB6CxJ79085815 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 6 Dec 2016 14:59:20 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uB6CxJ79085815 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uB6CxJH9085814; Tue, 6 Dec 2016 14:59:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 6 Dec 2016 14:59:19 +0200 From: Konstantin Belousov To: Steven Hartland Cc: "freebsd-hackers@freebsd.org" Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD Message-ID: <20161206125919.GQ54029@kib.kiev.ua> References: <27e1a828-5cd9-0755-50ca-d7143e7df117@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <27e1a828-5cd9-0755-50ca-d7143e7df117@multiplay.co.uk> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2016 12:59:30 -0000 On Tue, Dec 06, 2016 at 12:31:47PM +0000, Steven Hartland wrote: > Hi guys I'm trying to help identify / fix an issue with golang where by > fork results in memory corruption. > > Details of the issue can be found here: > https://github.com/golang/go/issues/15658 > > In summary when a fork is done in golang is has a chance of causing > memory corruption in the parent resulting in a process crash once detected. > > Its believed that this only effects FreeBSD. > > This has similarities to other reported issues such as this one which > impacted perl during 10.x: > https://rt.perl.org/Public/Bug/Display.html?id=122199 I cannot judge about any similarilities when all the description provided is 'memory corruption'. BTW, the perl issue described, where child segfaults after the fork, is more likely to be caused by the set of problems referenced in the FreeBSD-EN-16:17.vm. > > And more recently the issue with nginx on 11.x: > https://lists.freebsd.org/pipermail/freebsd-stable/2016-September/085540.html Which does not affect anything unless aio is used on Sandy/Ivy. > > Its possible, some believe likely, that this is a kernel bug around fork > / vm that golang stresses, but I've not been able to confirm. > > I can reproduce the issue at will, takes between 5mins and 1hour using > 16 threads, and it definitely seems like an interaction between fork and > other memory operations. Which arch is the kernel and the process which demonstrates the behaviour ? I mean i386/amd64. > > I've tried reproducing the issue in C but also no joy (captured in the bug). > > For reference I'm currently testing on 11.0-RELEASE-p3 + kibs PCID fix > (#306350). Switch to HEAD kernel, for start. Show the memory map of the failed process. Are you able to take ktrace of the process while still producing the bug ? Where is the memory corruption happen ? Is it in go runtime structures, or in the application data ? Can somebody knowledgable of either the go runtime or the app, try to identify the initial corrupted userspace data ?