From owner-freebsd-virtualization@freebsd.org Tue Mar 10 16:21:38 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8E4C12690E8 for ; Tue, 10 Mar 2020 16:21:38 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cL293NFtz3Dyw for ; Tue, 10 Mar 2020 16:21:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 009952690D7; Tue, 10 Mar 2020 16:21:37 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EB1922690D6; Tue, 10 Mar 2020 16:21:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cL2664VHz3Dtn; Tue, 10 Mar 2020 16:21:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f54.google.com with SMTP id g15so7613862otr.0; Tue, 10 Mar 2020 09:21:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FxExWztqzhA7Okl24KY/uH5VFrSEi9pTSZV2EDfdEuM=; b=f0FQcpo8ue7pNlJQu2sy+Xzw+xRT9QGIe/AVGAYa6NO97igSQGWMNgmgZm/gQnlxSE sToDW5+4DN97h3HH+HLLQRXz9kZd6+tCY3jt+ho1HC/EMCbduZzJ7uIEQFJu/MUjQDt1 lwcEGxOu/MdGikLcnLDO3XctuRxrvF/IxDWtnlWu/RGlw004paIzYaR1v52zlYZojvzy E6MZv2C+LKQZccomxNV2uLXr/327nUj7ifAFb4p82/QwA0OD9z8/MshhAE36dw1qzYRH rHyBgBT7L54fK/LbhN2I3/EfMYsEDHQ2nybLMcPfrk0OxDX0Qbw4Ha5KXxRnDDbFM8Zm 4+Gg== X-Gm-Message-State: ANhLgQ133mGcBLWkamUw04IQteMnj+RqXi8EdKuVVy8B6P04DekLXqfd 8z0NvIq5Wwm9kZ31KljFw14xPgb23518bKO+DRhPWV3k X-Google-Smtp-Source: ADFU+vst+kjLRXZAi1FlxUbWxEzB0D06jyjBAZHlLiyV3Ats4t4KLg6B54e1E+NwBBIljEbE/hHDPIUmbh0SQSeZStk= X-Received: by 2002:a9d:b8f:: with SMTP id 15mr15417282oth.256.1583857293687; Tue, 10 Mar 2020 09:21:33 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> In-Reply-To: <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> From: Alan Somers Date: Tue, 10 Mar 2020 10:21:21 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers X-Rspamd-Queue-Id: 48cL2664VHz3Dtn X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.54 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[54.210.85.209.rep.mailspike.net : 127.0.0.18]; URI_COUNT_ODD(1.00)[13]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[54.210.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.99)[ip: (-0.27), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:21:38 -0000 On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen wrote: > On 10-3-2020 16:15, Alan Somers wrote: > > On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen > wrote: > >> On 9-3-2020 14:46, Alan Somers wrote: >> >> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >> wrote: >> >>> Hi all, >>> >>> And sorry for crosspoing three groups, but the answer can/could be a mix >>> of things to do in these three areas. >>> >>> I have a prototype of bhyve running on Rados/Ceph working: >>> https://github.com/freebsd/freebsd/pull/426 >>> >>> .......... >> >> 4) Create a bhyve-blockrbd port. >>> This is much like 3) but instead of building a bhyve-rbd executable, >>> it delivers a libblockrbd.so that is dynamically loadable by the >>> standaard bhyve that comes with base. >>> >> ............ >> >> > Great work! I also agree that option 4 sounds like the best. There's >> precedent for ports that >> > require the FreeBSD Sources. For example, see devel/py-libzfs or >> emulators/virtualbox-ose. >> > You just need to define the SRC_BASE variable. >> Hi Alan, >> >> Thanx for the hint, and it made me check what is actually available >> within the poudriere jail >> And that does have full source, so the Makefile code is mainly for those >> that build in a different way. >> >> I've got a proto version working when compiling stuff with `make >> buildworld`, but run in the >> problem that libblock_rbd.so is stripped in such a way that the symbol I >> need is removed. >> Using the unstripped version does work. >> >> Is there an incantation for the SRC Makefiles that builds a dynamical >> loadable lib?? >> And I'm still looking for a PORTS example of building a dynamical >> loadable lib. >> Or is there no generic code for that in the PORTS Mk files? >> >> --WjW >> >> BTW: Still haven't worked in your AIO code :( >> > > There are plenty of dynamic libraries built with the SRC makefiles. For > example, > https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup > . > > > That looks dangerously close to what I have for libblock_rbd. > === > > cat Makefile-librbd > # > # $FreeBSD$ > # > > PACKAGE=lib${LIB} > > .include > > LIB= block_rbd > SHLIB_MAJOR= 1 > > SRCS= block_rbd.c > > CFLAGS+=-I${SRCTOP}/sys > CFLAGS+=-g -O0 -fPIC -rdynamic > LDFLAGS+=-Wl,-export-dynamic,-Bdynamic > CFLAGS+=-DWITHOUT_CAPSICUM > > LOCALBASE?= /usr/local > CFLAGS+= -I${LOCALBASE}/include > LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd > > WARNS?= 2 > > === > > This is the code that mk.lib.bsd runs: > objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug > objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug > libblock_rbd.so.1.full libblock_rbd.so.1 > > So still I get a stripped lib in /usr/lib. And then the one and only > symbol I need to load > is not found. Copying libblock_rbd.so.1.full actually works for me. > > So either I'm doing it the wrong way, like special options on the symbols > oid. > Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? > > And there are plenty of ports that build shared libraries too, just look > at /usr/local/lib/*.so. However, the ports framework doesn't have much > special code just to support building libraries. Instead the hard work is > always done by the ports themselves. Some use autotools, some cmake, etc > etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB > is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . > > > Oke thanx, I'll have a look at it, and given that I can see most of the > compile build stuff > in the SRC_BASE version I'll get it to work. > > --WjW > Try setting "STRIP= " in your makefile. That should prevent the stripping. However, I think there's something wrong with your library, too. The library should be usable even if it's stripped.