From owner-freebsd-virtualization@freebsd.org Tue Mar 10 16:49:08 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA76F269EC8 for ; Tue, 10 Mar 2020 16:49:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cLdw1MsVz3xLR for ; Tue, 10 Mar 2020 16:49:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id DA404269EC5; Tue, 10 Mar 2020 16:49:07 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9872269EC4; Tue, 10 Mar 2020 16:49:07 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cLdt3qgGz3xGk; Tue, 10 Mar 2020 16:49:06 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f177.google.com with SMTP id c1so14561020oiy.2; Tue, 10 Mar 2020 09:49:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gUmYw482+yc1LN/ofvq/PgpZURXsZ+DQY0p98X7Nvmo=; b=MmRpxEAHaL9mvbzWU/dNrUFefFFeC21fojPUXFEtVE0BtmrdXCbOnrjJhCXrfljSuy QYQZin0NrM5lp7eLPBFJL2HE9QGH3dNwlp07z72hi+h7lFUTlPVYrf4dTM2a/M+oo9H0 NBFhpYZBkdT7UdC5Ie/tpp6FuOM/edLwfIjS2bseuUiTEmFcJ0VrPGD84srMzXkO3+Dy FaSd5m8F2Sak92aKjgs0mgghBgPzgTZwAdB+LFjmj6sqWQ9dq8GNgA2saF3PciVsV9Jz DYuehAFdIvx51fbHm0AHPsd5ZXm4hLL3IrQ8iNASWygkirPgEWm8t0peTW40NYnC4LKR YCjw== X-Gm-Message-State: ANhLgQ37/exH7nqT+0rFw6x541TL6sqvFSFKi8SgO0PqtfoJz03Tv+mv MnTEhXfjKalkoFoswd3trpUJ6REowmKFKMbODG1TtI/VJQs= X-Google-Smtp-Source: ADFU+vvswGKrYZRywpblAPI0740yoJPo8JA0imSIhoF+nfEMWd2E0JJoyJv3BJ02C1w6xzseU8OM906D2oNugda2WOA= X-Received: by 2002:aca:3f09:: with SMTP id m9mr1778196oia.55.1583858944935; Tue, 10 Mar 2020 09:49:04 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> In-Reply-To: From: Alan Somers Date: Tue, 10 Mar 2020 10:48:53 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers X-Rspamd-Queue-Id: 48cLdt3qgGz3xGk X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; URI_COUNT_ODD(1.00)[13]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[177.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.97)[ip: (-0.19), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[177.167.85.209.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:49:09 -0000 On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen wrote: > On 10-3-2020 17:21, Alan Somers wrote: > > On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen > wrote: > >> On 10-3-2020 16:15, Alan Somers wrote: >> >> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen >> wrote: >> >>> On 9-3-2020 14:46, Alan Somers wrote: >>> >>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >>> wrote: >>> >>>> Hi all, >>>> >>>> And sorry for crosspoing three groups, but the answer can/could be a mix >>>> of things to do in these three areas. >>>> >>>> I have a prototype of bhyve running on Rados/Ceph working: >>>> https://github.com/freebsd/freebsd/pull/426 >>>> >>>> .......... >>> >>> 4) Create a bhyve-blockrbd port. >>>> This is much like 3) but instead of building a bhyve-rbd >>>> executable, >>>> it delivers a libblockrbd.so that is dynamically loadable by the >>>> standaard bhyve that comes with base. >>>> >>> ............ >>> >>> > Great work! I also agree that option 4 sounds like the best. There's >>> precedent for ports that >>> > require the FreeBSD Sources. For example, see devel/py-libzfs or >>> emulators/virtualbox-ose. >>> > You just need to define the SRC_BASE variable. >>> Hi Alan, >>> >>> Thanx for the hint, and it made me check what is actually available >>> within the poudriere jail >>> And that does have full source, so the Makefile code is mainly for those >>> that build in a different way. >>> >>> I've got a proto version working when compiling stuff with `make >>> buildworld`, but run in the >>> problem that libblock_rbd.so is stripped in such a way that the symbol I >>> need is removed. >>> Using the unstripped version does work. >>> >>> Is there an incantation for the SRC Makefiles that builds a dynamical >>> loadable lib?? >>> And I'm still looking for a PORTS example of building a dynamical >>> loadable lib. >>> Or is there no generic code for that in the PORTS Mk files? >>> >>> --WjW >>> >>> BTW: Still haven't worked in your AIO code :( >>> >> >> There are plenty of dynamic libraries built with the SRC makefiles. For >> example, >> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >> . >> >> >> That looks dangerously close to what I have for libblock_rbd. >> === >> > cat Makefile-librbd >> # >> # $FreeBSD$ >> # >> >> PACKAGE=lib${LIB} >> >> .include >> >> LIB= block_rbd >> SHLIB_MAJOR= 1 >> >> SRCS= block_rbd.c >> >> CFLAGS+=-I${SRCTOP}/sys >> CFLAGS+=-g -O0 -fPIC -rdynamic >> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic >> CFLAGS+=-DWITHOUT_CAPSICUM >> >> LOCALBASE?= /usr/local >> CFLAGS+= -I${LOCALBASE}/include >> LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd >> >> WARNS?= 2 >> >> === >> >> This is the code that mk.lib.bsd runs: >> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug >> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug >> libblock_rbd.so.1.full libblock_rbd.so.1 >> >> So still I get a stripped lib in /usr/lib. And then the one and only >> symbol I need to load >> is not found. Copying libblock_rbd.so.1.full actually works for me. >> >> So either I'm doing it the wrong way, like special options on the symbols >> oid. >> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? >> >> And there are plenty of ports that build shared libraries too, just look >> at /usr/local/lib/*.so. However, the ports framework doesn't have much >> special code just to support building libraries. Instead the hard work is >> always done by the ports themselves. Some use autotools, some cmake, etc >> etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB >> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . >> >> Oke thanx, I'll have a look at it, and given that I can see most of the >> compile build stuff >> in the SRC_BASE version I'll get it to work. >> >> --WjW >> > > Try setting "STRIP= " in your makefile. That should prevent the > stripping. However, I think there's something wrong with your library, > too. The library should be usable even if it's stripped. > > I checked with objdump, and the symbol that I need is definitly not > present in the stripped version. > And it does not really matter if I declare it static or not. > But I'll give it a few more itterations to try it out. > Including 'STRIP= ' > > Thanx, > --WjW > What does "nm --dynamic libblock_rbd.so.1" show?