From owner-freebsd-embedded@FreeBSD.ORG Sun Jun 14 19:13:46 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C308A57E; Sun, 14 Jun 2015 19:13:46 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FD28EEA; Sun, 14 Jun 2015 19:13:46 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iebgx4 with SMTP id gx4so50462437ieb.0; Sun, 14 Jun 2015 12:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=CrXoxBllZ0EcUYzTZqK3MjwPI/79gf+62WxgMakvwHE=; b=G8drNhsU8h6a6S4yZQrkuNZpReMpyaQy852Y5H4dydPHHBlGua7D1wJcOeIyih5eHY ACo++zMoEPVX+KAi/p6Nn0hxJW2OBhTv3EThza1GboisGDHqtR+8hAEEkfvbto8FtGzi CnrjGt7Hctf8lw4r/TOqYCe3bJrRMw3PIckKes+j3tkmxhBUAjEvt+1hah9qU7fjSnoL XxUor+wLzalMV5Y8V+hvcTJSlM7GpmOskgmMnisJ7Bhlke5hpv4rZ2QqtpWriECD/Eb6 mTCzVX0CvdGZUPLe05xu72jZFKfRothJj6rXaEavv311xW/tk+gB8sG0pXoIbyzA/jmE PuDA== MIME-Version: 1.0 X-Received: by 10.50.78.232 with SMTP id e8mr6480162igx.32.1434309225893; Sun, 14 Jun 2015 12:13:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Sun, 14 Jun 2015 12:13:45 -0700 (PDT) Date: Sun, 14 Jun 2015 12:13:45 -0700 X-Google-Sender-Auth: XpcOWo25DpF7HISY5zeSpRugXiI Message-ID: Subject: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 19:13:46 -0000 Hi! I've been hacking at cross compiling things on FreeBSD for a while, and I have made some progress. This email is less of a "how you can do it" and more of a "these are the roadblocks that prevent it from working out of the box." I'm using gcc-4.9.2 from ports - there's a cross compiler for mips/mips64, and there's mips-binutils/mips64-binutils. I'm using those to compile world and kernel. The kernel works; world requires some patching to compile. I'll write a post when I have a complete system booting/running from gcc-4.9.2. My test application is dropbear. Ie, I'm cross-compiling dropbear for mips from freebsd/i386. * There are some issues in libgcc - I have a patch that I've emailed out for review. I'll do some more testing and commit it next week when I know it works. * There are some environment variables that need to be set before I compile dropbear. Here's what it looks like: CPU_ARGS="-march=mips32 -msoft-float -Wa,-msoft-float" INCS="-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/include" export CROSS_COMPILE=mips-portbld-freebsd11.0 export CC=${CROSS_COMPILE}-gcc export CXX=${CROSS_COMPILE}-g++ #export LD=${CROSS_COMPILE}-ld #export AR=${CROSS_COMPILE}-ar #export RANLIB=${CROSS_COMPILE}-ranlib #export STRIP=${CROSS_COMPILE}-strip export LD=mips-freebsd-l export AR=mips-freebsd-ar export RANLIB=mips-freebsd-ranlib export STRIP=mips-freebsd-strip export SYSROOT=/home/adrian/work/freebsd/head-embedded-2/root/mips/ export CFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O" export CXXFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O" export CPPFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O" export LDFLAGS=--sysroot=$SYSROOT # now run: # /configure --host=mips-portbld-freebsd11.0 # gmake ... then I hit a problem with how we populate links in /usr/lib and such. TL;DR - the symlinks we use are not relative paths, they're absolute paths - and this makes cross building using sysroot / explicit library paths fail to work right. Sigh. In particular, I needed to manually undo the libgcc symlink for it to compile. compilation complained about other libraries, but the resulting binaries did run. * yes, I tested the dropbear/dbclient binaries in a mips qemu VM (not qemu-mips-user, but a full mips MALTA VM.) Both worked fine. * the sysroot setup didn't correctly set the include path stuff right - I need to chase that down. Ie, it was still searching in /usr/include/ rather than ${SYSROOT}/usr/include/ . The library code obeyed the library searching, except for that hardlink. So, the very basics of "how can we cross compile things to run inside a freebsd-mips machine" work fine. It's amusing and annoying that the sysroot thing popped up. I do wonder if this class of problem keeps popping up when people do builds from one freebsd version to another; but I currently have no clear idea how to automate detecting this. On the plus side, it did work. So: * I'll try to wrangle juli/bsdimp to review the libgcc change I have so userland compiles using gcc-4.9 with warnings disabled. * I'll try to wrangle juli/bsdimp to review the kernel changes needed to compile up the kernel with gcc-4.9. * I'd like some help / comments on what we can do about the absolute symlinks used for various things inside an installed root. Ideally we'd have an option that would let us populate only relative links - that'd instantly make this problem go away. * I'll go hit up bapt and some netbsd friends for some help with expected sysroot behaviour on gcc/binutils. It's possible there are still bugs with the dynamic sysroot (ie, when you don't compile your toolchain with an explicit sysroot to something other than '/') that need addressing. Then I'll worry about starting to convert a few ports to be native cross building. -a From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 02:15:24 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E6DBFEB; Mon, 15 Jun 2015 02:15:24 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ECAE6BED; Mon, 15 Jun 2015 02:15:23 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iebgx4 with SMTP id gx4so53762392ieb.0; Sun, 14 Jun 2015 19:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=hg5UDeIG+PlkoykmoF3i/fTMol4oDv+B38Rnydro3SI=; b=fWxqqwEZ2PcMAMaJEH6vUpbg3kgn8dpgR9zbtSu2we1pIjh1x22NShhH5xYNdby33L Onl7YiO4wWPifQm7sBIZz6SXaRhqSbFpiaHN0VGQVTD70R/fiz8vw1kq+9ljuIgnEqpr igY+8TXfqfGxNmSu/fXXRhDhlVN1JqZ9tV3kC/+CE7SJFZbHq2VelRUJ+fSn9Wt0395l iWaBPbw2vJMbPdrsGcvs26hcIfy/B/aB3a8Am59M2nA7AWsSRaK+289Ijg0ay9K0phWK 8Xet7ASGJZKDgimcR+nhtuf09xIxeModrWDbBXxhj064SMbPFrKRO9bmC+DyzZlGXStr uhmg== MIME-Version: 1.0 X-Received: by 10.50.79.228 with SMTP id m4mr17542453igx.6.1434334522998; Sun, 14 Jun 2015 19:15:22 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Sun, 14 Jun 2015 19:15:22 -0700 (PDT) In-Reply-To: References: Date: Sun, 14 Jun 2015 19:15:22 -0700 X-Google-Sender-Auth: kKjzPBZa5fMoQy2bF_JiPCwzRus Message-ID: Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 02:15:24 -0000 Hm, for some reason, doing a make installworld with an external compiler set fails with an unknown compiler type. -adrian From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 02:18:33 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E555DBB; Mon, 15 Jun 2015 02:18:33 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB2EBBF7; Mon, 15 Jun 2015 02:18:33 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igbsb11 with SMTP id sb11so41235482igb.0; Sun, 14 Jun 2015 19:18:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=TJpN8hwwMYJCE/AcWQyNXbU8MSGP9dfkC3ZqUXz7gvU=; b=ccm/h0DRDoqNwlIkhOfkbffYY0QHgLUlEDPzoZvxL40Duy30h4Vmyi+XddIdddZTxq yps+drGRYoi35Jux5uCHl8q0ga0Rcgq0LhP0+r3v0AzEwbN8xeQ/7hw7p9caxu0cBrgf O6+yaLp60XDQFetV67LTsnhMxcY5tygxnpQPK5a5PC7IdhU26kRRmQEkCM6jm+XsUsHa vmi7+NTZfb6ybIasoaGw9gDlSj0kO8rBLJfarCu3dTYv9MDIhMSYlrbP8WOPLjpiN9Vm TGy+yyMkCTvAXzBX52xLX6fGJpuqPpsnh4O/PXTpYDhcuXlRcyEg0IHt1iNvUKx8vyiP eMXw== MIME-Version: 1.0 X-Received: by 10.107.5.210 with SMTP id 201mr30838050iof.88.1434334713141; Sun, 14 Jun 2015 19:18:33 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Sun, 14 Jun 2015 19:18:33 -0700 (PDT) In-Reply-To: References: Date: Sun, 14 Jun 2015 19:18:33 -0700 X-Google-Sender-Auth: faQhvxhsTK5-hxwWarYmsZ65XVw Message-ID: Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 02:18:34 -0000 Actually, the installworld part fails: ===> gnu (install) ===> gnu/lib (install) ===> gnu/lib/csu (install) ===> gnu/lib/libgcc (install) ===> gnu/lib/libdialog (install) ===> gnu/lib/libregex (install) ===> gnu/lib/libgcov (install) install: libgcov.a: No such file or directory *** Error code 71 .. I'm guessing it's not building some gnu/gcc bits when using an external toolchain? Ah yes, it is - MK_GCC is apparently being set to NO during build, but not during intsallworld, even though I am setting things during the installworld phase. here's what i'm calling: *** Stage : installworld env MAKEOBJDIRPREFIX=/home/adrian/work/freebsd/head-embedded-2/src/../obj/mips/ make -s DB_FROM_SRC=1 TARGET=mips TARGET_ARCH=mips NO_WERROR=1 CROSS_TOOLCHAIN=mips-gcc COMPILER_TYPE=gcc TARGET_CPUTYPE=mips32 CPUTYPE=mips32 KERNCONF=MALTA KODIR=/boot/kernel.MALTA/ DESTDIR=/home/adrian/work/freebsd/head-embedded-2/src/../root/mips NO_ROOT=yes KMODDIR=/boot/kernel.MALTA/ __MAKE_CONF=/home/adrian/work/freebsd/head-embedded-2/src/../root/mips/../make.conf.mips SRCCONF=/home/adrian/work/freebsd/head-embedded-2/src/../root/mips/../src.conf.mips LOCAL_DIRS= installworld .. any ideas? -a From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 06:21:43 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AAE553B4; Mon, 15 Jun 2015 06:21:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 744E9CD6; Mon, 15 Jun 2015 06:21:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iesa3 with SMTP id a3so56093547ies.2; Sun, 14 Jun 2015 23:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=8YZH052hKcZMOwhBMubJDylBo5pWYO3GWllyj7hLkVE=; b=ck0TDjRBeY78aDW25Dy4wYyaGnqXfTY3Li1RdWlCleggg9OlVIkqvXH3+gChZ3/NRz RCugfiBrVvbEkQmYh5Xj5s/bSH1YzJtdOtPwYIy/+/4zcGX9Eewpko70XYzvZdvzKxg2 Bom/32doeAy7yPKWUx7uyUh/Ih2Jic6pmMF2+aaT2cuS2GcvKgSRYOyPmLneO50PNwnL JzkyqXTntngUxE8k9p5DjvQFzkDf8VQo/YAHhZgyje6tDNJ+RuXeVJP2PUl2LvESaSHA +/CBmXkU3lvozeyJ8KuWzIwMsCrotbmBeUCe7OK6t/Z6R3qXTfcA7C9wVprIhwE16zr7 MH1w== MIME-Version: 1.0 X-Received: by 10.107.5.210 with SMTP id 201mr31659259iof.88.1434349302899; Sun, 14 Jun 2015 23:21:42 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Sun, 14 Jun 2015 23:21:42 -0700 (PDT) In-Reply-To: References: Date: Sun, 14 Jun 2015 23:21:42 -0700 X-Google-Sender-Auth: kfwP6eqJ8PjVlsavb64W3JwJIgQ Message-ID: Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 06:21:43 -0000 So, for what it's worth - after the relevant hacks to convince the various stages to do their thing, I actually /do/ have a user/kernel cross-built from gcc-4.9.2 up on both the mips32 emulator (qemu) /and/ on real hardware (AR9331, MIPS24kc). The installworld fails because somehow the CROSS_TOOLCHAIN bits aren't being included right, so the compiler bits aren't set right. Here, COMPILER_TYPE=gcc and COMPILER_VERSION=40902 - with this, MK_GCC is set to NO and the legacy c++ library isn't built. There's no compiler built with it though, so until I cross compile a compiler package (and cross-compile pkg, grr) getting a compiler /into/ the image is going to take some time. But, it does boot and run fine, and cross-compiled binaries (eg dropbear client/server) work fine, as well as crunchgen binaries (rescue, bsdbox.) -adrian From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 17:23:11 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1E800D6F for ; Mon, 15 Jun 2015 17:23:11 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-f173.google.com (mail-ie0-f173.google.com [209.85.223.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC9DFB43 for ; Mon, 15 Jun 2015 17:23:10 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: by iesa3 with SMTP id a3so67102352ies.2 for ; Mon, 15 Jun 2015 10:23:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to; bh=Jg61YZFHp+UqzSewoJRQcqrihKrefHlpbVbiDNM4r5g=; b=VrgRY9d64IbwsGDf5Tvyt44Z6vDhZjDkFpBxBN4ToFCw5AqQLJKtkwe4bjqa7wWJw+ Za7YXJIV3xr9yLoyt+pTcpPIg8XLRwSBGJTM4IToOJXqGqH6JGkj8BrUIMds5Y6ZkSU+ Uzymr5vpzdN6zsYDqMKN/WcYIKmFOAlDcWsM5sNjxou8I8seEnes8tmCaQv1cCCLE9zZ fEWulwccCHc4TML5ZLDLFb6wI/yeoDzweLdOZ/IlRRAgK+esisx9ATSlXuqxMQieb1pj YolbjNRZrHnstQZkQ4n+yDXClKmS+6CJMX9pHrF0sPLYHcZxpUrkjj6Khk5ZaskCtreG gKNg== X-Gm-Message-State: ALoCoQkbYJYo5qZmdFW6SRPWqGRyot69PJU0DlkP34FWTHJJ/eql1fyA8sWC4WGNQ6NmzMd3hQUN X-Received: by 10.50.79.196 with SMTP id l4mr22545722igx.48.1434388984250; Mon, 15 Jun 2015 10:23:04 -0700 (PDT) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id rt5sm7993918igb.8.2015.06.15.10.23.03 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 15 Jun 2015 10:23:03 -0700 (PDT) Sender: Warner Losh Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Content-Type: multipart/signed; boundary="Apple-Mail=_2B777C26-1D06-4FE9-96C4-978352D20094"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5b6 From: Warner Losh In-Reply-To: Date: Mon, 15 Jun 2015 11:23:00 -0600 Cc: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Message-Id: References: To: Adrian Chadd X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 17:23:11 -0000 --Apple-Mail=_2B777C26-1D06-4FE9-96C4-978352D20094 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 14, 2015, at 1:13 PM, Adrian Chadd wrote: >=20 > Hi! >=20 > I've been hacking at cross compiling things on FreeBSD for a while, > and I have made some progress. This email is less of a "how you can do > it" and more of a "these are the roadblocks that prevent it from > working out of the box." >=20 > I'm using gcc-4.9.2 from ports - there's a cross compiler for > mips/mips64, and there's mips-binutils/mips64-binutils. I'm using > those to compile world and kernel. The kernel works; world requires > some patching to compile. I'll write a post when I have a complete > system booting/running from gcc-4.9.2. >=20 > My test application is dropbear. Ie, I'm cross-compiling dropbear for > mips from freebsd/i386. >=20 > * There are some issues in libgcc - I have a patch that I've emailed > out for review. I'll do some more testing and commit it next week when > I know it works. > * There are some environment variables that need to be set before I > compile dropbear. Here's what it looks like: >=20 > CPU_ARGS=3D"-march=3Dmips32 -msoft-float -Wa,-msoft-float" > = INCS=3D"-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/include"= >=20 > export CROSS_COMPILE=3Dmips-portbld-freebsd11.0 > export CC=3D${CROSS_COMPILE}-gcc > export CXX=3D${CROSS_COMPILE}-g++ > #export LD=3D${CROSS_COMPILE}-ld > #export AR=3D${CROSS_COMPILE}-ar > #export RANLIB=3D${CROSS_COMPILE}-ranlib > #export STRIP=3D${CROSS_COMPILE}-strip > export LD=3Dmips-freebsd-l > export AR=3Dmips-freebsd-ar > export RANLIB=3Dmips-freebsd-ranlib > export STRIP=3Dmips-freebsd-strip > export SYSROOT=3D/home/adrian/work/freebsd/head-embedded-2/root/mips/ > export CFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > export CXXFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > export CPPFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > export LDFLAGS=3D--sysroot=3D$SYSROOT >=20 > # now run: > # /configure --host=3Dmips-portbld-freebsd11.0 > # gmake >=20 > ... then I hit a problem with how we populate links in /usr/lib and > such. TL;DR - the symlinks we use are not relative paths, they're > absolute paths - and this makes cross building using sysroot / > explicit library paths fail to work right. Sigh. In particular, I > needed to manually undo the libgcc symlink for it to compile. > compilation complained about other libraries, but the resulting > binaries did run. That=E2=80=99s one of about 6 reasons that what we produce isn=E2=80=99t = a sysroot. We need to create a sysroot target that=E2=80=99s very close to the xdev targets with all the compiler bits stripped out. We=E2=80=99re= close though. I have about 1/2 of the patch needed to remove xdev entirely and replace it with sysroot. You shouldn=E2=80=99t need the INCS args if you have a proper sysroot. And the CPU_ARGS is an indication we=E2=80=99re building our port incorrectly. This is the same feedback I gave on the kernel make changes you suggested. > * yes, I tested the dropbear/dbclient binaries in a mips qemu VM (not > qemu-mips-user, but a full mips MALTA VM.) Both worked fine. >=20 > * the sysroot setup didn't correctly set the include path stuff right > - I need to chase that down. Ie, it was still searching in > /usr/include/ rather than ${SYSROOT}/usr/include/ . The library code > obeyed the library searching, except for that hardlink. That sounds like a bug in the default path of our gcc port. > So, the very basics of "how can we cross compile things to run inside > a freebsd-mips machine" work fine. It's amusing and annoying that the > sysroot thing popped up. I do wonder if this class of problem keeps > popping up when people do builds from one freebsd version to another; > but I currently have no clear idea how to automate detecting this. For one version to another we have a very controlled and constrained environment. Inside of that, there are enough people building in = purposely broken environments to detect regressions. Once you involve configure and foreign software things become rather unconstrained... > On the plus side, it did work. I=E2=80=99ve had hacks like this for =E2=80=9Csimple=E2=80=9D arts for = years. It=E2=80=99s why I wrote xdev in the first place. I had it working in the FreeBSD 6 time frame for cross building some ports. But once you get past a certain class of very well behaved ports, it goes to hell in a hurry. > So: >=20 > * I'll try to wrangle juli/bsdimp to review the libgcc change I have > so userland compiles using gcc-4.9 with warnings disabled. I=E2=80=99ll be happy to take a look. > * I'll try to wrangle juli/bsdimp to review the kernel changes needed > to compile up the kernel with gcc-4.9. I=E2=80=99ve already reviewed that one. gcc isn=E2=80=99t being compiled = correctly and that need to be fixed. This is the forth time I=E2=80=99ve given = this feedback. > * I'd like some help / comments on what we can do about the absolute > symlinks used for various things inside an installed root. Ideally > we'd have an option that would let us populate only relative links - > that'd instantly make this problem go away. All installworld-ish targets should use relative path symbolic links. The fact they are using absolute paths is a bug. In the FreeBSD 6 time frame, I fought this battle at work and it wasn=E2=80=99t hard to = fix. I no longer have access to those sources, and I honestly can=E2=80=99t recall how I fixed it (it may have been in a post-processing step). > * I'll go hit up bapt and some netbsd friends for some help with > expected sysroot behaviour on gcc/binutils. It's possible there are > still bugs with the dynamic sysroot (ie, when you don't compile your > toolchain with an explicit sysroot to something other than '/') that > need addressing. bapt and I have been talking about this for a while. We=E2=80=99ve both = been busy doing other things so haven=E2=80=99t focused on it other than in = off moments. > Then I'll worry about starting to convert a few ports to be native > cross building. You should talk to bapt about the rather extensive work he=E2=80=99s = done in this area, which is why I=E2=80=99ve looked at transitioning the xdev support to just sysroot support. Warner --Apple-Mail=_2B777C26-1D06-4FE9-96C4-978352D20094 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJVfwn1AAoJEGwc0Sh9sBEAMI0P/j12ZOpHoOHhBkn0crUdLTR8 FRskASw84nMM5/h0vOkVlaBVrWxj/pRSbl7ysUhMXWC0CHx3RK5tKqrDby0A7TFj UurUZE1kmD7oUacX/tAMlnZuL5di2jopmqntprY83HvZoUsXsoeoDhTPxOXjW5KB c7UhbCZPCZkxXMNWUFIAPs/DI5HfpfxQxZmtq3HyLPuQOlzQt+Q24nocQzRXxT5b PcKSd0pN+F/Km6CiPcZzdg0oGefuwLZ+oOMrJH/igiqR94WmYypOvksb/E+H5mEa 6kpGDJuzrBueVy/Rx0TRK+MhlsGcM3zFtYmpmU9H06smR9RaYizka2NLqA38gop5 gvoZvELK0050+cjUBFuoS2hM8e258mkphfr+HKAsittZtT1HX7A8OA0W3AID6Vi6 xYvGisMllvHqN9cnGj3pNnbXRqeYx6RAJylmJnnAvGjbTw4W8C6XcD/07kzBxO3M H5qgTnn/STF0TwwlA4o6JxX0XF1mKXtGP8pvBMHun77yfQ1tBTuLEUHrCuK7Uar4 UbPpu0BgOapEhDsVZUK+MiMcUy88n1kxRoBkgzBQXsQ9JMdd2pnOAyQh+xL5tHQg GFg2UeyPRok1EkaJJLGTEXlwp6CT9xq0FufC+b3JBM46qZQLvio9LgxM97nLUDCZ YSSgptmDj+RtNJeJdXHS =J58A -----END PGP SIGNATURE----- --Apple-Mail=_2B777C26-1D06-4FE9-96C4-978352D20094-- From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 17:31:21 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 197FF330 for ; Mon, 15 Jun 2015 17:31:21 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5668D39 for ; Mon, 15 Jun 2015 17:31:20 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: by igbos3 with SMTP id os3so24527938igb.0 for ; Mon, 15 Jun 2015 10:31:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to; bh=/NoEjnc9aFLUKhEoTnf1qvDnIa180WuflRH/QjtH4nk=; b=ds5OoptGFT/YmfUZpbB40xUM6auf3FcfZfT+unqH+tobWeZiuwl0KjNboyOWK6oynY rWP0AXAwb5W4ZgBGlvVVlG5WLZtt3TndHH3zUgu9UaGlsrKyE2zvycLM/HTNlSFKBCLv NmScp/96hli85ucIpGOK1bAs9Gzkojs0wTBQbdRlFw9cH2/BjSukoIox8GzMjWTQAUeo SjdwfOJq0eycJHgOPDbHiNi0Y7Bity8SORKgucLVAWA1fhz1uzzO8jeSGxqEG2Yyuch6 w1/3M93DKxLc+0usJQnuBMmZkgfYDYKNResmLej25/NteXolgTMNXSFnvJKXTorEHVUm BtPA== X-Gm-Message-State: ALoCoQnRsMYFLUwjMNfQRwEVnZPbvzRt0SvMwjnqNgMN+NbhYYVbJdCrr9TTn8GI6IcDKPV/ZSHD X-Received: by 10.107.136.88 with SMTP id k85mr10570035iod.63.1434389480227; Mon, 15 Jun 2015 10:31:20 -0700 (PDT) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id y124sm9260891iod.13.2015.06.15.10.31.19 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 15 Jun 2015 10:31:19 -0700 (PDT) Sender: Warner Losh Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Content-Type: multipart/signed; boundary="Apple-Mail=_A5074BF4-BC88-4BF3-9445-7A318F88F910"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5b6 From: Warner Losh In-Reply-To: Date: Mon, 15 Jun 2015 11:31:17 -0600 Cc: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Message-Id: <2ADF0848-4190-4D40-808B-DA6D539FA431@bsdimp.com> References: To: Adrian Chadd X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 17:31:21 -0000 --Apple-Mail=_A5074BF4-BC88-4BF3-9445-7A318F88F910 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 15, 2015, at 12:21 AM, Adrian Chadd wrote: >=20 > So, for what it's worth - after the relevant hacks to convince the > various stages to do their thing, I actually /do/ have a user/kernel > cross-built from gcc-4.9.2 up on both the mips32 emulator (qemu) /and/ > on real hardware (AR9331, MIPS24kc). >=20 > The installworld fails because somehow the CROSS_TOOLCHAIN bits aren't > being included right, so the compiler bits aren't set right. Here, > COMPILER_TYPE=3Dgcc and COMPILER_VERSION=3D40902 - with this, MK_GCC = is > set to NO and the legacy c++ library isn't built. That=E2=80=99s actually correct. =46rom about 4.8 and onward, clang and = gcc should be using the same c++ runtime. Maybe our defaults are a bit messed up, since libgcov likely shouldn=E2=80=99t be tied to MK_GCC at all=E2=80=A6 = With the clang introduction, we=E2=80=99ve not always had a good clear line about the different = subtle aspects of clang vs gcc, and this may be one of them. > There's no compiler built with it though, so until I cross compile a > compiler package (and cross-compile pkg, grr) getting a compiler > /into/ the image is going to take some time. gcc does support the notion of building on host X to generate binaries that run on host Y that generate code for host Z. But that might be a bit of a stretch for our system. > But, it does boot and run fine, and cross-compiled binaries (eg > dropbear client/server) work fine, as well as crunchgen binaries > (rescue, bsdbox.) That=E2=80=99s cool! Warner --Apple-Mail=_A5074BF4-BC88-4BF3-9445-7A318F88F910 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJVfwvlAAoJEGwc0Sh9sBEAqX8P/32M9k39Ixiq1eq+6FIzgP/5 8DPaKn9XUXDlpdPsYlKTjLg51NfM57wZGySm9oWFxu95TSMf8iafcxLmdloWXxhZ KKSyqnPcSLrYrsUP+pccjbNsUih6fK/Nc3pvYmNIDet/zNv0lbo7HwiSrNLPW8QK I3flnXq+iThG8gFCno9dqCKBZIc3+HIt0+0SZZqO84z7Ct0uQOsQ4DwtqE3DNMPU 57b6CgnWSNrsXd6Ob6sQNe27WHogCS1nCQrzH0Fclkpe6xTF8k8lL6SARd0hP26M zso54R3zp5kmUlLTTgXMyRHzEnWBEDh/vahSEKru2mPBJnlsfVIDcMFXoWtv/3FV BDa0WlnmUudW35beuVw8CKrc4xLnHFn5WPXXp5Y+cjb8u62KTmUr1U9n8Nxba+2K eYpqzwbkG4nAyyPMVXjcs7hkPeWaKzSg8QM26Vetk2uLQ5g/s3jYEPWxzt48HhoP LGuBAT3PNMzzdYbvka6PL0CRlHTQh+XdcMuoVHVLMB+ULO4z2fWpP8GkGuQPuENP JOTWabErI/oHo3aWWwwoejwdqdlTOa4J8b5a0VEeqinXhc4EeJkhV26Yv/LgWLdh mcCqc5D+sxLlre9n3gEzYrgYzJYugyIvHJYAKMui0a3WooXjDosC+ee9YCPF3b8p FQsuBZMNrzCSB1evqPXN =5pqU -----END PGP SIGNATURE----- --Apple-Mail=_A5074BF4-BC88-4BF3-9445-7A318F88F910-- From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 18:39:37 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 602DDC15; Mon, 15 Jun 2015 18:39:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26AF1F1B; Mon, 15 Jun 2015 18:39:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igboe5 with SMTP id oe5so37187874igb.1; Mon, 15 Jun 2015 11:39:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=KAUTJecYVNsPyEp1Xg5TZox86rwan6+wnMLNwutKGMQ=; b=NzvKfjLUD+PUuzyoqVGi5VubOU5Dc2hX4iVCknzzkyFShjo/GZL1ieJH9Aru8+vPIu 1xeV0pdm1ecdNMzHamyI1kdokBiW84DFQxhFxBQRjV4PUR+2gartJ3xwKuhf9xmJGgZT eG4hI8Z7R6DYuboocA4UMyTLnQ7h1uGfkqGWUof3Cwf1TajhXoKm4uB7/54W/xpXLA+f X4gj4UQR8Qqa5honW59cATLGC6rbRwIyoO2lfGggzutzBn2sKEZQd3r1ZNGDJeX9mLzj lujU2qQxtwOdk65Q938c+MGY/1AzcomWTXSyxXAS8hqKgFyGOhguK2c4FiAz329keOFy SHPQ== MIME-Version: 1.0 X-Received: by 10.107.5.210 with SMTP id 201mr35708374iof.88.1434393576009; Mon, 15 Jun 2015 11:39:36 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Mon, 15 Jun 2015 11:39:35 -0700 (PDT) In-Reply-To: <2ADF0848-4190-4D40-808B-DA6D539FA431@bsdimp.com> References: <2ADF0848-4190-4D40-808B-DA6D539FA431@bsdimp.com> Date: Mon, 15 Jun 2015 11:39:35 -0700 X-Google-Sender-Auth: 3xRmu-j0EPUYLUWFPcTzr2qILms Message-ID: Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: Warner Losh Cc: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 18:39:37 -0000 On 15 June 2015 at 10:31, Warner Losh wrote: > >> On Jun 15, 2015, at 12:21 AM, Adrian Chadd wrote: >> >> So, for what it's worth - after the relevant hacks to convince the >> various stages to do their thing, I actually /do/ have a user/kernel >> cross-built from gcc-4.9.2 up on both the mips32 emulator (qemu) /and/ >> on real hardware (AR9331, MIPS24kc). >> >> The installworld fails because somehow the CROSS_TOOLCHAIN bits aren't >> being included right, so the compiler bits aren't set right. Here, >> COMPILER_TYPE=3Dgcc and COMPILER_VERSION=3D40902 - with this, MK_GCC is >> set to NO and the legacy c++ library isn't built. > > That=E2=80=99s actually correct. From about 4.8 and onward, clang and gcc= should > be using the same c++ runtime. Maybe our defaults are a bit messed up, > since libgcov likely shouldn=E2=80=99t be tied to MK_GCC at all=E2=80=A6 = With the clang introduction, > we=E2=80=99ve not always had a good clear line about the different subtle= aspects > of clang vs gcc, and this may be one of them. *nod* so,, I agree with all of that, but the above is the result, not the c= ause. make buildworld uses CROSS_TOOLCHAIN for the gcc config and cascades things correctly. make installworld doesn't seem to use CROSS_TOOLCHAIN everywhere, and it defaults to assuming we're building with gcc, and thus MK_GCC=3Dyes, and it tries installing non-built things. If I manually set the COMPILER_TYPE / COMPILER_VERSION variables then things work out. I /think/ that's broken behaviour, so I'm going to check first with bapt to see what's supposed to happen and see what we can do to sort that out. -a From owner-freebsd-embedded@FreeBSD.ORG Mon Jun 15 18:45:00 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C0549E44; Mon, 15 Jun 2015 18:45:00 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88A1519F; Mon, 15 Jun 2015 18:45:00 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igboe5 with SMTP id oe5so37305620igb.1; Mon, 15 Jun 2015 11:45:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=Jh9RDx/0NBKv45sZUXCoKx85B9eily1TZMuNCY64lZY=; b=tNAyozuKEKk4SmbzTziEQDLZXRZs5QH9oBLnf6I5bD+MCIUloYhOrPYYZYS2plyZO8 my7kecCl66qHrcmUn1uQtfHTDt4OhjKm7Vu6qSM36MmPI7A7VG1dSF9gYYyUXNkO2w1T j9xbReEjvJESVZMogxRWX465bNFMjPebOsqmcLCiefQfdC1dCth10FK6K4B3duEV3pzR fEW7YdLMuSp9f3M+90I+pPjXrWcTZLyDSjTZgLtXZxwXoa4/HYThvY7K1FzjnJqBZRoP X02VK1kcHnoPrQOHOJgjqII/2Lb7p0Vm5jimcluLuyI27ttjzqM5ZJbGDf0n6gmBIGcC xZqQ== MIME-Version: 1.0 X-Received: by 10.107.164.70 with SMTP id n67mr36278843ioe.8.1434393900091; Mon, 15 Jun 2015 11:45:00 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Mon, 15 Jun 2015 11:45:00 -0700 (PDT) In-Reply-To: References: Date: Mon, 15 Jun 2015 11:45:00 -0700 X-Google-Sender-Auth: 65Bl0xlapu_pTTD_cVut7jTNYT8 Message-ID: Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like From: Adrian Chadd To: Warner Losh Cc: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 18:45:01 -0000 On 15 June 2015 at 10:23, Warner Losh wrote: > >> On Jun 14, 2015, at 1:13 PM, Adrian Chadd wrote: >> >> Hi! >> >> I've been hacking at cross compiling things on FreeBSD for a while, >> and I have made some progress. This email is less of a "how you can do >> it" and more of a "these are the roadblocks that prevent it from >> working out of the box." >> >> I'm using gcc-4.9.2 from ports - there's a cross compiler for >> mips/mips64, and there's mips-binutils/mips64-binutils. I'm using >> those to compile world and kernel. The kernel works; world requires >> some patching to compile. I'll write a post when I have a complete >> system booting/running from gcc-4.9.2. >> >> My test application is dropbear. Ie, I'm cross-compiling dropbear for >> mips from freebsd/i386. >> >> * There are some issues in libgcc - I have a patch that I've emailed >> out for review. I'll do some more testing and commit it next week when >> I know it works. >> * There are some environment variables that need to be set before I >> compile dropbear. Here's what it looks like: >> >> CPU_ARGS=3D"-march=3Dmips32 -msoft-float -Wa,-msoft-float" >> INCS=3D"-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/includ= e" >> >> export CROSS_COMPILE=3Dmips-portbld-freebsd11.0 >> export CC=3D${CROSS_COMPILE}-gcc >> export CXX=3D${CROSS_COMPILE}-g++ >> #export LD=3D${CROSS_COMPILE}-ld >> #export AR=3D${CROSS_COMPILE}-ar >> #export RANLIB=3D${CROSS_COMPILE}-ranlib >> #export STRIP=3D${CROSS_COMPILE}-strip >> export LD=3Dmips-freebsd-l >> export AR=3Dmips-freebsd-ar >> export RANLIB=3Dmips-freebsd-ranlib >> export STRIP=3Dmips-freebsd-strip >> export SYSROOT=3D/home/adrian/work/freebsd/head-embedded-2/root/mips/ >> export CFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" >> export CXXFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" >> export CPPFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" >> export LDFLAGS=3D--sysroot=3D$SYSROOT >> >> # now run: >> # /configure --host=3Dmips-portbld-freebsd11.0 >> # gmake >> >> ... then I hit a problem with how we populate links in /usr/lib and >> such. TL;DR - the symlinks we use are not relative paths, they're >> absolute paths - and this makes cross building using sysroot / >> explicit library paths fail to work right. Sigh. In particular, I >> needed to manually undo the libgcc symlink for it to compile. >> compilation complained about other libraries, but the resulting >> binaries did run. > > That=E2=80=99s one of about 6 reasons that what we produce isn=E2=80=99t = a > sysroot. We need to create a sysroot target that=E2=80=99s very close > to the xdev targets with all the compiler bits stripped out. We=E2=80=99r= e > close though. I have about 1/2 of the patch needed to remove > xdev entirely and replace it with sysroot. Sweet. > You shouldn=E2=80=99t need the INCS args if you have a proper sysroot. > And the CPU_ARGS is an indication we=E2=80=99re building our port > incorrectly. This is the same feedback I gave on the kernel > make changes you suggested. It looks like a proper sysroot - I think it's just the dynamic sysroot stuff being broken. > I=E2=80=99ve had hacks like this for =E2=80=9Csimple=E2=80=9D arts for ye= ars. It=E2=80=99s why I wrote > xdev in the first place. I had it working in the FreeBSD 6 time frame > for cross building some ports. But once you get past a certain > class of very well behaved ports, it goes to hell in a hurry. >> * I'll try to wrangle juli/bsdimp to review the kernel changes needed >> to compile up the kernel with gcc-4.9. > > I=E2=80=99ve already reviewed that one. gcc isn=E2=80=99t being compiled = correctly > and that need to be fixed. This is the forth time I=E2=80=99ve given this > feedback. There are some other issues too. The softfloat/hardfloat stuff reared its head - in order to avoid assembler warnings we have to manually set it, but then the floating point register accesses in fp.S / swtch.S generate errors. I've worked around that for now, but I need to dig into why the warnings are generated in the first case. >> * I'd like some help / comments on what we can do about the absolute >> symlinks used for various things inside an installed root. Ideally >> we'd have an option that would let us populate only relative links - >> that'd instantly make this problem go away. > > All installworld-ish targets should use relative path symbolic links. > The fact they are using absolute paths is a bug. In the FreeBSD 6 > time frame, I fought this battle at work and it wasn=E2=80=99t hard to fi= x. I > no longer have access to those sources, and I honestly can=E2=80=99t > recall how I fixed it (it may have been in a post-processing > step). I'd love to get this fixed and into the tree. >> * I'll go hit up bapt and some netbsd friends for some help with >> expected sysroot behaviour on gcc/binutils. It's possible there are >> still bugs with the dynamic sysroot (ie, when you don't compile your >> toolchain with an explicit sysroot to something other than '/') that >> need addressing. > > bapt and I have been talking about this for a while. We=E2=80=99ve both b= een > busy doing other things so haven=E2=80=99t focused on it other than in of= f > moments. > >> Then I'll worry about starting to convert a few ports to be native >> cross building. > > You should talk to bapt about the rather extensive work he=E2=80=99s done > in this area, which is why I=E2=80=99ve looked at transitioning the xdev > support to just sysroot support. I have been too! fear not. That's how I got this far! -a From owner-freebsd-embedded@FreeBSD.ORG Tue Jun 16 13:19:01 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F130317A; Tue, 16 Jun 2015 13:19:00 +0000 (UTC) (envelope-from m.vale@live.com.au) Received: from COL004-OMC4S18.hotmail.com (col004-omc4s18.hotmail.com [65.55.34.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5A816E5; Tue, 16 Jun 2015 13:19:00 +0000 (UTC) (envelope-from m.vale@live.com.au) Received: from COL130-W14 ([65.55.34.201]) by COL004-OMC4S18.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751); Tue, 16 Jun 2015 06:18:54 -0700 X-TMN: [3bK4JJtp12tIl+NBHmtWfB/u6N9Q90Lx] X-Originating-Email: [m.vale@live.com.au] Message-ID: From: Michael Vale To: Adrian Chadd , Warner Losh CC: "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Subject: RE: Starting to cross-compile non-base software for MIPS - what it's like Date: Tue, 16 Jun 2015 23:48:53 +1030 Importance: Normal In-Reply-To: References: , , MIME-Version: 1.0 X-OriginalArrivalTime: 16 Jun 2015 13:18:54.0261 (UTC) FILETIME=[FE57C650:01D0A836] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jun 2015 13:19:01 -0000 Here is onething that can be added under .if defined(X_BUILD_FOR)=2C either= as an .include e.g. bsd.cross.mk. or just paste it in there. if X_BUILD_F= OR /is/ defined=2C configure --host mips-freebsd-* etc will already be done= if you just invoke a normal ports build rather than running configure manu= ally from port/work/. =20 I know it's not perfect=2C but the only thing that needs and overview is th= e declaration of SYSROOT. I used ${LOCALBASE} to define the right position= =2C i'm unsure if this is correct. =20 ---------------------------- bsd.cross.mk ---------------------------------= ------------------------------------------- =20 .if exist(/usr/${X_BUILD_FOR}/usr/bin/${X_BUILD_FOR}-portbld-freebsd11.0-gc= c) then \ MAKE_ENV+=3D CPU_ARGS=3D"-march=3D${X_BUILD_FOR}" INCS=3D"-l${SRCDIR}/../root/${X_BUILD_FOR} .else IGNORE=3D Install gcc with mips support from ports first. .if FLOAT=3D"hard" then \ MAKE_ENV+=3D CPU_ARGS=3D"-mhard-float=2C -Wa=2C-mhard-float" .else \ MAKE_ENV+=3D CPU_ARGS=3D"-msoft-float=2C -Wa=2C-msoft-float" MAKE_ENV+=3D CROSS_COMPILE=3D"${X_BUILD_FOR}-portbld-freebsd${OS_REL} \ CC=3D${CROSS_COMPILE}-gcc \ CXX=3D${CROSS_COMPILE}-g++ \ LD=3D${X_BUILD_FOR}-freebsd-ld \ AR=3D${X_BUILD_FOR}-freebsd-ar \ RANLIB=3D${X_BUILD_FOR}-freebsd-ranlib \ STRIP=3D${X_BUILD_FOR}-freebsd-strip \ SYSROOT=3D${LOCALBASE}/${X_BUILD_FOR}/ \ CFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" CXXFLAGS=3D"--sysroot=3D$SYSROOM ${CPU_ARGS} ${INCS} -O" =20 > Date: Mon=2C 15 Jun 2015 11:45:00 -0700 > Subject: Re: Starting to cross-compile non-base software for MIPS - what = it's like > From: adrian@freebsd.org > To: imp@bsdimp.com > CC: embedded@freebsd.org=3B freebsd-mips@freebsd.org >=20 > On 15 June 2015 at 10:23=2C Warner Losh wrote: > > > >> On Jun 14=2C 2015=2C at 1:13 PM=2C Adrian Chadd w= rote: > >> > >> Hi! > >> > >> I've been hacking at cross compiling things on FreeBSD for a while=2C > >> and I have made some progress. This email is less of a "how you can do > >> it" and more of a "these are the roadblocks that prevent it from > >> working out of the box." > >> > >> I'm using gcc-4.9.2 from ports - there's a cross compiler for > >> mips/mips64=2C and there's mips-binutils/mips64-binutils. I'm using > >> those to compile world and kernel. The kernel works=3B world requires > >> some patching to compile. I'll write a post when I have a complete > >> system booting/running from gcc-4.9.2. > >> > >> My test application is dropbear. Ie=2C I'm cross-compiling dropbear fo= r > >> mips from freebsd/i386. > >> > >> * There are some issues in libgcc - I have a patch that I've emailed > >> out for review. I'll do some more testing and commit it next week when > >> I know it works. > >> * There are some environment variables that need to be set before I > >> compile dropbear. Here's what it looks like: > >> > >> CPU_ARGS=3D"-march=3Dmips32 -msoft-float -Wa=2C-msoft-float" > >> INCS=3D"-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/incl= ude" > >> > >> export CROSS_COMPILE=3Dmips-portbld-freebsd11.0 > >> export CC=3D${CROSS_COMPILE}-gcc > >> export CXX=3D${CROSS_COMPILE}-g++ > >> #export LD=3D${CROSS_COMPILE}-ld > >> #export AR=3D${CROSS_COMPILE}-ar > >> #export RANLIB=3D${CROSS_COMPILE}-ranlib > >> #export STRIP=3D${CROSS_COMPILE}-strip > >> export LD=3Dmips-freebsd-l > >> export AR=3Dmips-freebsd-ar > >> export RANLIB=3Dmips-freebsd-ranlib > >> export STRIP=3Dmips-freebsd-strip > >> export SYSROOT=3D/home/adrian/work/freebsd/head-embedded-2/root/mips/ > >> export CFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > >> export CXXFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > >> export CPPFLAGS=3D"--sysroot=3D$SYSROOT ${CPU_ARGS} ${INCS} -O" > >> export LDFLAGS=3D--sysroot=3D$SYSROOT > >> > >> # now run: > >> # /configure --host=3Dmips-portbld-freebsd11.0 > >> # gmake > >> > >> ... then I hit a problem with how we populate links in /usr/lib and > >> such. TL=3BDR - the symlinks we use are not relative paths=2C they're > >> absolute paths - and this makes cross building using sysroot / > >> explicit library paths fail to work right. Sigh. In particular=2C I > >> needed to manually undo the libgcc symlink for it to compile. > >> compilation complained about other libraries=2C but the resulting > >> binaries did run. > > > > That=92s one of about 6 reasons that what we produce isn=92t a > > sysroot. We need to create a sysroot target that=92s very close > > to the xdev targets with all the compiler bits stripped out. We=92re > > close though. I have about 1/2 of the patch needed to remove > > xdev entirely and replace it with sysroot. >=20 > Sweet. >=20 > > You shouldn=92t need the INCS args if you have a proper sysroot. > > And the CPU_ARGS is an indication we=92re building our port > > incorrectly. This is the same feedback I gave on the kernel > > make changes you suggested. >=20 > It looks like a proper sysroot - I think it's just the dynamic sysroot > stuff being broken. >=20 > > I=92ve had hacks like this for =93simple=94 arts for years. It=92s why = I wrote > > xdev in the first place. I had it working in the FreeBSD 6 time frame > > for cross building some ports. But once you get past a certain > > class of very well behaved ports=2C it goes to hell in a hurry. >=20 >=20 > >> * I'll try to wrangle juli/bsdimp to review the kernel changes needed > >> to compile up the kernel with gcc-4.9. > > > > I=92ve already reviewed that one. gcc isn=92t being compiled correctly > > and that need to be fixed. This is the forth time I=92ve given this > > feedback. >=20 > There are some other issues too. The softfloat/hardfloat stuff reared > its head - in order to avoid assembler warnings we have to manually > set it=2C but then the floating point register accesses in fp.S / > swtch.S generate errors. I've worked around that for now=2C but I need > to dig into why the warnings are generated in the first case. >=20 > >> * I'd like some help / comments on what we can do about the absolute > >> symlinks used for various things inside an installed root. Ideally > >> we'd have an option that would let us populate only relative links - > >> that'd instantly make this problem go away. > > > > All installworld-ish targets should use relative path symbolic links. > > The fact they are using absolute paths is a bug. In the FreeBSD 6 > > time frame=2C I fought this battle at work and it wasn=92t hard to fix.= I > > no longer have access to those sources=2C and I honestly can=92t > > recall how I fixed it (it may have been in a post-processing > > step). >=20 > I'd love to get this fixed and into the tree. >=20 > >> * I'll go hit up bapt and some netbsd friends for some help with > >> expected sysroot behaviour on gcc/binutils. It's possible there are > >> still bugs with the dynamic sysroot (ie=2C when you don't compile your > >> toolchain with an explicit sysroot to something other than '/') that > >> need addressing. > > > > bapt and I have been talking about this for a while. We=92ve both been > > busy doing other things so haven=92t focused on it other than in off > > moments. > > > >> Then I'll worry about starting to convert a few ports to be native > >> cross building. > > > > You should talk to bapt about the rather extensive work he=92s done > > in this area=2C which is why I=92ve looked at transitioning the xdev > > support to just sysroot support. >=20 > I have been too! fear not. That's how I got this far! >=20 >=20 > -a > _______________________________________________ > freebsd-embedded@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-embedded > To unsubscribe=2C send any mail to "freebsd-embedded-unsubscribe@freebsd.= org" = From owner-freebsd-embedded@FreeBSD.ORG Wed Jun 17 15:18:41 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 477BEEBD; Wed, 17 Jun 2015 15:18:41 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (mail.soaustin.net [66.135.54.68]) by mx1.freebsd.org (Postfix) with ESMTP id 29C1C321; Wed, 17 Jun 2015 15:18:40 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: by mail.soaustin.net (Postfix, from userid 502) id F085D561CE; Wed, 17 Jun 2015 10:18:33 -0500 (CDT) Date: Wed, 17 Jun 2015 10:18:33 -0500 From: Mark Linimon To: Warner Losh Cc: Adrian Chadd , "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like Message-ID: <20150617151833.GA13147@lonesome.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 15:18:41 -0000 On Mon, Jun 15, 2015 at 11:23:00AM -0600, Warner Losh wrote: > This is the forth time I’ve given this feedback. I think you've been staring at the loader too much :-) mcl From owner-freebsd-embedded@FreeBSD.ORG Wed Jun 17 15:26:30 2015 Return-Path: Delivered-To: embedded@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2073B35B for ; Wed, 17 Jun 2015 15:26:30 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB7F97CB for ; Wed, 17 Jun 2015 15:26:29 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: by paceq1 with SMTP id eq1so13938351pac.3 for ; Wed, 17 Jun 2015 08:26:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to; bh=7B29e0c2qpI9MVda6bi8pnLPc30+W42AuVHfv0QR4Fk=; b=Y21aD4h8fR+4o1HyoqJZN3NHd9AOZRIXVxAXbsv9ii6f+QhjKufIsxOW68o6L9lIb7 SDmLrZETkI2dve+A68JuMxtoGLXsaXA9akMtrt+0WbhWwBqlygiOIAoFlFiJ1QK+49VZ EX28bYIWXPe12KNIkJc2hYuaPIOepsUJPi+79JQQXnDup7QN1HyNquUu5AT1LzPtUSvM DDNtQ2nGdFsmUu3l9nZhBglkXrxLoKhINGw1EhrIykSDEFyd9MHo/apedayAe0LpoU/N QZsqQ/ZTtHmCwQPsw0TAwYUFxCT2X/q1/sOYz77/rsgeF5C/DpiV2yb4ZdrOenGnmyvo uayA== X-Gm-Message-State: ALoCoQlkF6YZ96egEUBh59PkrEEf5FuEbK2xk7nldmJS9WZdEk75HO7qQLMwUR7SDSDVdINLyVhs X-Received: by 10.68.190.8 with SMTP id gm8mr11998144pbc.30.1434554788771; Wed, 17 Jun 2015 08:26:28 -0700 (PDT) Received: from [10.64.24.56] ([69.53.236.236]) by mx.google.com with ESMTPSA id uk6sm5119023pac.27.2015.06.17.08.26.27 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 17 Jun 2015 08:26:27 -0700 (PDT) Sender: Warner Losh Subject: Re: Starting to cross-compile non-base software for MIPS - what it's like Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Content-Type: multipart/signed; boundary="Apple-Mail=_275D63B5-3A5A-4B86-946A-94363B013DA2"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5 From: Warner Losh In-Reply-To: <20150617151833.GA13147@lonesome.com> Date: Wed, 17 Jun 2015 09:26:27 -0600 Cc: Adrian Chadd , "freebsd-embedded@freebsd.org" , "freebsd-mips@freebsd.org" Message-Id: References: <20150617151833.GA13147@lonesome.com> To: Mark Linimon X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2015 15:26:30 -0000 --Apple-Mail=_275D63B5-3A5A-4B86-946A-94363B013DA2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 17, 2015, at 9:18 AM, Mark Linimon = wrote: >=20 > On Mon, Jun 15, 2015 at 11:23:00AM -0600, Warner Losh wrote: >> This is the forth time I=E2=80=99ve given this feedback. >=20 > I think you've been staring at the loader too much :-) only forth also builtin definitions feedback 1 + dup feedback ! .=E2=80=9D Make sure the compiler=E2=80=99s defaults are correct=E2=80=9D = cr 4 - 0=3D if .=E2=80=9D This is the forth time I=E2=80=99ve given this = feedback.=E2=80=9D cr \ Bah! Who needs lua! --Apple-Mail=_275D63B5-3A5A-4B86-946A-94363B013DA2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJVgZGjAAoJEGwc0Sh9sBEAG5MP/RKPfWW5tklgCGdI4au7npG5 9D4Tr/V2PbDZ7ZOny30KS/9AF/ABJvOMX5LMtTeOcLYSJUlTUKEnI1U7SU6cW92O JWC8rntgD9uFAaDzBAYXuy5GFlr9K6JWc4xCF9fhV2NspMCCPmEfYv+j8q01HV38 EifO8+MKTKjCdASNtoYUDRb//CCVAEOxcul2k0ITA1Q/wOZnmjFcZaVzJdBIx3gX aRrrsIVgI1mdS5onpG/XAbwG7+hn2BsomaQ/+3mTEth7h9RLZea+YW4W1jCPPJg7 2I25wC4eeUUcx0fN4/1El2AeRmMul2BjRGCWS+ZBW89qbW2SbzXTttgXYHTyxU1f lUrpxv6ywDHsQOEZpLa+yWtxMnbb/gkRz+O9OcAFxoY2atfbJpRXjzrPPESricqr JnuxayJndfE7Y8qFZ4zxPVjf6z7r2Arx/hHEWR4YMLZFkRkDZRLQ/FMiIXLA6YK8 ZnyC7KqdHXLRSM2rCAJyNDWdn/joabUoTvVkRqADN93pbrflRZlLr9OXr8W03ntt L6zLKdrMCk+akBwaJzTgT6/I849ipKwkfGFRltpN9vz91KtSSgl6JQer/f8VShdu seDgOZW3tHWfMZ087vmY+nTpLB/Ipr8ia0Vw0mYQxS+Cx1WM/MbsTab94JHOzNwz S5TXJmKIjqC0Lh8XkKXF =K9PL -----END PGP SIGNATURE----- --Apple-Mail=_275D63B5-3A5A-4B86-946A-94363B013DA2--