From owner-freebsd-java@freebsd.org  Thu Jul  6 21:14:03 2017
Return-Path: <owner-freebsd-java@freebsd.org>
Delivered-To: freebsd-java@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68133D92121
 for <freebsd-java@mailman.ysv.freebsd.org>;
 Thu,  6 Jul 2017 21:14:03 +0000 (UTC)
 (envelope-from kostikbel@gmail.com)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id EF2407FD2D;
 Thu,  6 Jul 2017 21:14:02 +0000 (UTC)
 (envelope-from kostikbel@gmail.com)
Received: from tom.home (kib@localhost [127.0.0.1])
 by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v66LDvfW042953
 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
 Fri, 7 Jul 2017 00:13:57 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v66LDvfW042953
Received: (from kostik@localhost)
 by tom.home (8.15.2/8.15.2/Submit) id v66LDvh4042952;
 Fri, 7 Jul 2017 00:13:57 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
 using -f
Date: Fri, 7 Jul 2017 00:13:57 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Li-Wen Hsu <lwhsu@freebsd.org>
Cc: "freebsd-java@freebsd.org" <freebsd-java@freebsd.org>
Subject: Re: Some weird errors in Jenkins on post-ino64 -CURRENT
Message-ID: <20170706211357.GJ1935@kib.kiev.ua>
References: <CAKBkRUzJpM9Dta8C9nUEjE4bq7Cc9TAqb8Dt__YH1n2F1LhJTA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAKBkRUzJpM9Dta8C9nUEjE4bq7Cc9TAqb8Dt__YH1n2F1LhJTA@mail.gmail.com>
User-Agent: Mutt/1.8.3 (2017-05-23)
X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
 DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no
 autolearn_force=no version=3.4.1
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home
X-BeenThere: freebsd-java@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Porting Java to FreeBSD <freebsd-java.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-java>,
 <mailto:freebsd-java-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-java/>
List-Post: <mailto:freebsd-java@freebsd.org>
List-Help: <mailto:freebsd-java-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-java>,
 <mailto:freebsd-java-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Jul 2017 21:14:03 -0000

On Thu, Jul 06, 2017 at 01:09:01PM +0100, Li-Wen Hsu wrote:
> Hi -java,
> 
> With FreeBSD cluster update, we migrated several Jenkins builders to
> post-ino64 -CURRENT, but since then, we encountered several mysterious
> errors.
> 
> What happens here is Jenkins (which uses SVNKit, subversion client library
> written in Java) refuses checkout codes from svn.freebsd.org.  Some logs
> like these:
> 
> https://ci.freebsd.org/job/FreeBSD-head-amd64-build/3674/console
> 
>   00:00:10.759 ERROR: Failed to check out svn://svn.freebsd.org/base/head
>   00:00:10.760 org.tmatesoft.svn.core.SVNException: svn: E200030: BUSY
> 
> https://ci.freebsd.org/job/FreeBSD-head-amd64-build/3676/console
> 
>   00:00:00.104 ERROR: Failed to check out svn://svn.freebsd.org/base/head
>   00:00:00.104 org.tmatesoft.svn.core.SVNException: svn: E155018:
> '/jenkins/workspace/FreeBSD-head-amd64-build/src' already exists and is not
> a directory
> 
> I've talked this with Ed at BSDCan, and we suspect this is related to
> ino64.  Normally, recompiling the binaries should solve the problem.  But I
> have tried to recompile everything that Jenkins depends on but it still
> doesn't work.  I also tried to use openjdk7 to build a new openjdk7-version
> bootstrap-openjdk but it still doesn't help.
> 
> This is reproducible with latest -CURRENT snapshot, installing Jenkins and
> checking out codes from svn.freebsd.org.
> 
> Any ideas, suggest what to test next?

There is no way around it other than debug the problem.  In other words,
trace the error back to the point of occurence, using sources and reasoning
about the code paths.

Sometimes you might cheat by looking for the known problematic usages in
the source, like locally redefining struct stat.  I do not expect that
JVM/JDK code does that, but who knows.  Also, it might be useful to look
around in ktrace, watching for the syscalls preceeding the failure.

But all such measures are secondary to normal debugging process.