From owner-freebsd-questions@freebsd.org Thu Jan 10 19:31:20 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED06214963C5 for ; Thu, 10 Jan 2019 19:31:19 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC9416AFC7 for ; Thu, 10 Jan 2019 19:31:17 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 4E43C2260B for ; Thu, 10 Jan 2019 14:31:11 -0500 (EST) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Thu, 10 Jan 2019 14:31:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=message-id:from:to:mime-version:content-transfer-encoding :content-type:date:subject; s=fm1; bh=FDdVmFzL5HhW5AU/qbox05xAt0 JUgW7TYLywzZMpBFs=; b=cN4eipHtprxDgTWhCIAcfF17Mgo1uHIBq7W5EW5CoX +URI+nbgz74LJnhXM01jZPy9gyzHR/E3i27D30z8uqbyV5c6bUNK3FOZH+ae/QqD h/Vi17WXHXHlCKj9MRsGVeW8YXFShs1NxuqfVDeuFwoSlsLd+dsipwJx4RMtPxH/ fr9sDEBLVri7kSHWardS0Bn5zIu2nA2Um2LJIFjIpMgB8r08HMArqxvRd+voLUen au3Waqhwuw0wZ2GscS5s5lUA8MQlasOF4FseN+APnUfNIW6xjzHTSLkSgaSDYK4k QH2/BWpEuHGz1M9WxCqkLjcTobN4VXUdGGkv/2ctbd1Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=FDdVmF zL5HhW5AU/qbox05xAt0JUgW7TYLywzZMpBFs=; b=glk7NHSTGteOoaxfi8Qhei QsEBVmk3+0lSkmFaGHdfBj6qb5pSFbHyVKOQ2/0/nP+JdEpMfC1nOHQjOyB1BpEd zL2NmzP1Qa5qK/9SCbN4epF8GRx+AMLFkbXSk/fdme/7BzvVvFoy44xOMtP2Yaxe s+ZT2rla6vy9+GVd1y2mk3smtJ4bhND26rkXNvqaY7+yyLJpaDolljaL66mCGxkY N2TrZHFp/pnxjte9kNvlmrcwB7b9cKVlZyF4WDQDomYJBP8TRCDuODTVM9VIxqS4 9+tVIWEjU9RtRmKY+xXZdzc121vrPgU0MmzhBNoExRUXPXHd3GM2UXxeif6+19MA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtledrfeefgdduvdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecuogetfeejfedqtdegucdlhedtmdenucfjughrpefkhffvggfgtgfofffusehtje ertdertdejnecuhfhrohhmpeffrghvvgcuvehothhtlhgvhhhusggvrhcuoegutghhsehs khhunhhkfigvrhhkshdrrghtqeenucffohhmrghinhephhgrtghkmhgurdhiohenucfrrg hrrghmpehmrghilhhfrhhomhepuggthhesshhkuhhnkhifvghrkhhsrdgrthenucevlhhu shhtvghrufhiiigvpedt X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 419564278; Thu, 10 Jan 2019 14:31:10 -0500 (EST) Message-Id: <1547148670.2385923.1631171800.2ED4526F@webmail.messagingengine.com> From: Dave Cottlehuber To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-5ae1f753 Date: Thu, 10 Jan 2019 20:31:10 +0100 Subject: repeated segfault of daemon after ~245 minutes of OS uptime, or multiples thereof (12.0Rp1 amd64) X-Rspamd-Queue-Id: BC9416AFC7 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=skunkwerks.at header.s=fm1 header.b=cN4eipHt; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=glk7NHST; spf=pass (mx1.freebsd.org: domain of dch@skunkwerks.at designates 66.111.4.25 as permitted sender) smtp.mailfrom=dch@skunkwerks.at X-Spamd-Result: default: False [-7.13 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[skunkwerks.at:s=fm1,messagingengine.com:s=fm1]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[skunkwerks.at]; DKIM_TRACE(0.00)[skunkwerks.at:+,messagingengine.com:+]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.55)[ip: (-9.32), ipnet: 66.111.4.0/24(-4.61), asn: 11403(-3.72), country: US(-0.08)]; RCVD_IN_DNSWL_LOW(-0.10)[25.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2019 19:31:20 -0000 Does this rather unusual duration remind somebody of some periodic counter cycle? I'm stumped. AFAICT the failure is periodic based on the host OS boot time, and *not* the runtime, but I'm not 100% sure on that yet. Per subject, round about 245 minutes after host boot, and repeatedly after that at the same interval (+- a couple of minutes), a jailed erlang runtime (databases/couchdb2) segfaults, on multiple systems. All are low end 8 core non-HT x86_64 arch atom CPUs C2750 @ with 8GB RAM, and are well within normal limits for cpu, ram, disk io. I've looked at 245 minutes in hex & binary, as minutes, seconds, milli- and micro-, and none of these resemble some sort of nibble-aligned counter that might conceivably overflow. All ports are built via a custom poudriere stack, albeit not very far off standard ports - we need a handful of custom settings and packages. URL below has details. sysctls are quite a few but are largely identical to those I run elsewhere, admittedly on larger boxes. networking is internet facing BGP and a private fc00/7 IPv6 vpn for the cluster nodes to communicate. lldb stacktraces and further notes at https://hackmd.io/elgRy4IWSR-FhViJDXHbDA Also, my lldb skills are limited, if somebody can recommend a resource to bone up that would be awesome. A+ Dave