From owner-freebsd-hackers@freebsd.org Tue Mar 28 08:48:24 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD5CED21AB0 for ; Tue, 28 Mar 2017 08:48:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B4E1DEF for ; Tue, 28 Mar 2017 08:48:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x236.google.com with SMTP id w11so77183202wrc.3 for ; Tue, 28 Mar 2017 01:48:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=Dldxy/rK/bu7c0wneu0gg+m/xmwKaAaiyWsRGyBigOY=; b=OuPbcJk+VLR97YJZRQXdjlo6Hj5W9u8ZkpC4yKxLFkd57AI04qf3z5UvcwQL72arW5 9tHTVWOq3DyuPQUPZTaGt/4LvDuiD1VX/Y5j3J5iqtIw3IaASP1ZtrKl9j7Vignomcux gy4M/Z7hdHA0Nh9TfrkNU0mW68+ZQ4xMHEXtUpBSQpQEBMMi8Ep6er2vhQLPV+L10O5p 6ZjqpM9v5jBRuUW1F9pabMK9mXCKwqQ5wuCD4tO8WaImLvuFjuX1F+UxCz03ks2M15Vw 6PuPwOYkZMoA/FNagtMb/eM2i/vVKY6Duyu8VmKmNGzGfOg8g4uqe5mac06EzfN+sWz3 Z2eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=Dldxy/rK/bu7c0wneu0gg+m/xmwKaAaiyWsRGyBigOY=; b=B60uAJzFnYMNPk/614shierSFYqjQR+vbVtdukVaxG5e2+ajkev55ZA/5YQIXopYTd t5tPbrURLp34JrskOAwZn6HU05q07x9GypOWfLIzUqC1qCTCtEgdLdQJnQzd+9v7e9p5 ZNLUP9zpad326Qi7eo8j98m6Woi8/UOmc+l2QqqvZFnwp9yvKD4gtEe7iu5aOwyAig8i KfqalduzxTUKtskggqGtvk4ptqIuYAEi1i1ouP6NAaWbDrNiKxX4XpT0b+SVhus/9Oxl NzboofVqElsDs7IRmTRnqHuFPtoIOqaOKv+paPWvRzNCqgR5xrVdtb6ovxp/SN/jFDf7 gQKw== X-Gm-Message-State: AFeK/H0isoavqL4NIuDXkZ5gvSPtYlIfondrhHvAw4OGPaPa/zF6aMn/zInBjlH3wdVW33yZ X-Received: by 10.28.105.8 with SMTP id e8mr13616363wmc.122.1490690902436; Tue, 28 Mar 2017 01:48:22 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id e16sm3911486wra.62.2017.03.28.01.48.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Mar 2017 01:48:21 -0700 (PDT) Subject: Re: Help needed to identify golang fork / memory corruption issue on FreeBSD To: Konstantin Belousov References: <20170317082333.GP16105@kib.kiev.ua> <180a601b-5481-bb41-f7fc-67976aabe451@multiplay.co.uk> <20170317124437.GR16105@kib.kiev.ua> <5ba92447-945e-6fea-ad4f-f58ac2a0012e@multiplay.co.uk> <20170327161833.GL43712@kib.kiev.ua> <3ec35a46-ae70-35cd-29f8-82e7cebb0eb6@multiplay.co.uk> <20170327164905.GN43712@kib.kiev.ua> <17f29342-f3c0-5940-d012-1a698e59a384@multiplay.co.uk> <20170328075859.GQ43712@kib.kiev.ua> <85f86a20-948f-025a-0d09-92ee2a815136@multiplay.co.uk> <20170328083810.GR43712@kib.kiev.ua> Cc: "K. Macy" , "freebsd-hackers@freebsd.org" From: Steven Hartland Message-ID: <5aa653ba-30e1-c9de-46ce-bad74d78c40c@multiplay.co.uk> Date: Tue, 28 Mar 2017 09:48:23 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170328083810.GR43712@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2017 08:48:24 -0000 On 28/03/2017 09:38, Konstantin Belousov wrote: > On Tue, Mar 28, 2017 at 09:23:24AM +0100, Steven Hartland wrote: >> As I stopped the panic before that I couldn't tell so I've re-run with >> some debug added just before the panic to capture the addresses of the >> workbuf structure that the issue was detected in, here goes (parent: >> 62620, child: 98756): >> >> workbuf: 0x800b51800 >> fatal error: workbuf is not empty >> workbuf: 0x800a72000 >> fatal error: workbuf is empty >> workbuf: 0x800a72000 >> fatal error: workbuf is not empty > I do not understand. Why do you show several addresses ? Wouldn't the > runtime panic after detecting the discrepancy, so there could be only one > address ? There are several goroutines (threads) running each detected an error, as I'm blocking the panic by entering a sleep in the faulting goroutine to enable the capture of procstat, other routines continue and detect an error too. To be clear this is not exclusive to the addition of the sleep, as I've seen this without any changes, as the GC is multi threaded. > >> procstat -v 62620 98756 >> PID START END PRT RES PRES REF SHD FLAG >> TP PATH >> 62620 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 62620 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 62620 0x951000 0x988000 rw- 31 0 1 0 C--- vn >> /root/golang/src/test5/test5 >> 62620 0x988000 0x9ab000 rw- 18 0 1 0 C--- df >> 62620 0x800951000 0x8009f1000 rw- 36 0 1 0 C--- df >> 62620 0x8009f1000 0x800ac1000 rw- 28 0 1 0 C--- df >> 62620 0x800ac1000 0x800ad1000 rw- 16 0 1 0 C--- df >> 62620 0x800ad1000 0x800b91000 rw- 5 0 1 0 C--- df >> 62620 0x800b91000 0x800bd1000 rw- 2 0 1 0 C--- df >> 62620 0x800bd1000 0x800c11000 rw- 2 0 1 0 C--- df >> 62620 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df >> 62620 0x800c51000 0x800c91000 rw- 1 0 1 0 C--- df >> 62620 0x800c91000 0x800cd1000 rw- 1 0 1 0 C--- df >> 62620 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 62620 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 62620 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 62620 0x7ffffffdf000 0x7ffffffff000 rwx 2 0 1 0 C--D df >> 62620 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> 98756 0x400000 0x70e000 r-x 309 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x70e000 0x951000 r-- 254 595 5 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x951000 0x988000 rw- 31 0 2 1 CN-- vn >> /root/golang/src/test5/test5 >> 98756 0x988000 0x9ab000 rw- 18 18 2 1 CN-- df >> 98756 0x800951000 0x8009f1000 rw- 36 36 2 1 CN-- df >> 98756 0x8009f1000 0x800ac1000 rw- 28 28 2 1 CN-- df >> 98756 0x800ac1000 0x800ad1000 rw- 16 16 2 1 CN-- df >> 98756 0x800ad1000 0x800b91000 rw- 5 5 2 1 CN-- df >> 98756 0x800b91000 0x800bd1000 rw- 2 2 2 1 CN-- df >> 98756 0x800bd1000 0x800c11000 rw- 2 2 2 1 CN-- df >> 98756 0x800c11000 0x800c51000 rw- 1 1 2 0 CN-- df >> 98756 0x800c51000 0x800c91000 rw- 1 1 2 1 CN-- df >> 98756 0x800c91000 0x800cd1000 rw- 1 1 2 1 CN-- df >> 98756 0xc000000000 0xc000001000 rw- 1 1 2 0 CN-- df >> 98756 0xc41fff0000 0xc41fff8000 rw- 3 3 2 0 CN-- df >> 98756 0xc41fff8000 0xc420200000 rw- 251 0 1 0 C--- df >> 98756 0x7ffffffdf000 0x7ffffffff000 rwx 2 2 2 1 CN-D df >> 98756 0x7ffffffff000 0x800000000000 r-x 1 1 28 0 ---- ph >> >> Regards >> Steve