From owner-freebsd-hackers@freebsd.org Sun Dec 23 21:45:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E56E31349F58 for ; Sun, 23 Dec 2018 21:45:46 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C99A9021D; Sun, 23 Dec 2018 21:45:46 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1354) id 7CF3412140; Sun, 23 Dec 2018 21:45:46 +0000 (UTC) From: Jan Beich To: "D. Ebdrup" Cc: freebsd-hackers@freebsd.org Subject: Re: Cirrus-CI: Free FreeBSD CI testing for open-source projects References: Date: Sun, 23 Dec 2018 22:45:40 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: 8C99A9021D X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-0.99)[-0.993,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2018 21:45:47 -0000 "D. Ebdrup" writes: > On 12/19/18, Jan Beich wrote: > >> Li-Wen Hsu writes: >> >>> On Wed, Dec 5, 2018 at 03:03 Alan Somers wrote: >>> >>>> Cirrus Labs has just released support for FreeBSD on their CI service. >>>> And >>>> they've made it free for OSS! Cirrus-CI is a cloud-based CI system for >>>> cloud-hosted software, much like Travis-CI, Appveyor, Circle-CI, etc. >>>> But >>>> it's the first* such system to support FreeBSD with no weird hacks >>>> required. It also runs each test in a full VM, so you can mount >>>> filesystems, create jails, etc. The free tier supports runs on a dual >>>> CPU >>>> VM with 4GB of RAM. But if that's not enough, you can cheaply configure >>>> Cirrus to use a custom VM in Google Cloud (gcp account required; cheap >>>> but >>>> not free). >>>> >>>> https://cirrus-ci.org/guide/FreeBSD/ >>> >>> >>> This is really an exciting news. Ed and I started a wiki page for >>> tracking >>> the efforts we put or wanted to add FreeBSD CI for the software widely >>> used: >>> >>> https://wiki.freebsd.org/HostedCI >> >> Why Chromium? Before hooking CI for FreeBSD it needs to build without >> patches but there was no upstreaming activity for years. >> >> https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-reviews/freebsd|sort:date >> https://cs.chromium.org/search/?q=OS_FREEBSD >> > > Another way to see the result of upstream making a practice out of not > accepting patches is to look at the files directory of the Chromium > port as seen on [1], especially when put up against an upstream > project which does accept patches as seen on [2]. > > [1]: https://svnweb.freebsd.org/ports/head/www/chromium/files > [2]: https://svnweb.freebsd.org/ports/head/www/firefox/files/ I'd err on chromium@ folks not having enough time rather than upstream being unreceptive. Over the years various Chrome developers tried to engage FreeBSD community but few have stepped up. Upstreaming is just that hard/time-consuming and the process never ends. However, there're benefits like timely/easy major updates, less bugs and feature parity due to code sharing with other BSD systems and upstream sharing their expertise in analyzing issues. OTOH, Firefox is no better with CI targeting FreeBSD. flo@ maintained FreeBSD Buildbot slave for mozilla-central which built daily rather than per push (i.e., not a real CI) but it has been offline for more than a year. From owner-freebsd-hackers@freebsd.org Mon Dec 24 04:09:43 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 293F013556A0 for ; Mon, 24 Dec 2018 04:09:43 +0000 (UTC) (envelope-from kevinz5000@gmail.com) Received: from mail-it1-x134.google.com (mail-it1-x134.google.com [IPv6:2607:f8b0:4864:20::134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0BB326DEA2 for ; Mon, 24 Dec 2018 04:09:42 +0000 (UTC) (envelope-from kevinz5000@gmail.com) Received: by mail-it1-x134.google.com with SMTP id b5so13974327iti.2 for ; Sun, 23 Dec 2018 20:09:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:openpgp:autocrypt:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=rEWQNfi5ucUT60M41AJjlorKoQ2+6+3ivKqtPwJ1GmM=; b=jm6zPSYVrlrzOMOnIZJ0kxJob7k79zqqBxZ7t5/mq05pVCw4RthB3zTEHOx5MMszmN bPonUvs0pZBvaBXBDqSG0/MWO5XrY3cuXUs2kHL+8ECOGuPtPA6gqYmncbXGTc0VLZh3 qF5ch7ag15NCt99x8qGT85z68n7pfviiYqOi4/vVPcS26C0UsSwmcuZxkhN7EOKXDy/s A/m/lmHQWILz81kSXqbj1nLYTWWn1NX6BU5wntYUn1B8woIzju8xDRG0VLsX3/bOrhXf 335gGhJ5voqPVjNE6NpT2+o0xaVyXgMxLy4l857KRdYcsvRZH4zWR3uw29WBqnNZBSvO SOpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:openpgp:autocrypt:message-id :date:user-agent:mime-version:content-language :content-transfer-encoding; bh=rEWQNfi5ucUT60M41AJjlorKoQ2+6+3ivKqtPwJ1GmM=; b=JfLxDc2PNvr4tQ07wbXrsUiue7DqxkNhXbwNwiKOhr0Ex0WHnsFeykf2oN0492Txyh UcRHrIyWsehrUV/iKLFpiTAu/Gn8RatDahzmKPXzV0u2ZAOD6wa8EC1fNQMiYXTZSJWn tZNCBLikXsvduseR+h2mFUAbFe7e6tQmtdSsunLoSlEcMVu7X0DTMGV+cpobupP1u23S BPMfRvLSoGz5SUyLZyg+v81YywNhb0QFuCnsVo3giPZbn9UxivuO6e95X09yM+HTlUIz RhopAzMNcyuAMFAJSyLND5MPnaDa0NGZfhXn5689N/5np05Yc6BuiVBY5bLd13tvVe2z v3dA== X-Gm-Message-State: AA+aEWYFxu0354Ju2/saTQNrlgGnx7V6o0CCpolRTzXLWy/aDe71zmCs h2tJt9fh2d0K9jjbEpzbNoSUfQBR X-Google-Smtp-Source: ALg8bN6aRnVwNpwd9iVHfEywIij/Cj8oH0jmQWJqcoO4iqGrnfQXO9rgSOdr8eHPclFCm9jvnQsMVA== X-Received: by 2002:a24:1c04:: with SMTP id c4mr8074193itc.79.1545624581084; Sun, 23 Dec 2018 20:09:41 -0800 (PST) Received: from [10.0.0.22] (172-221-159-029.dhcp.chtrptr.net. [172.221.159.29]) by smtp.gmail.com with ESMTPSA id q197sm9876902itb.22.2018.12.23.20.09.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Dec 2018 20:09:40 -0800 (PST) To: freebsd-hackers@freebsd.org From: Kevin Zheng Subject: mem_range_attr_set on amd64 returns ENOSPC Openpgp: preference=signencrypt Autocrypt: addr=kevinz5000@gmail.com; keydata= mQENBFHZd78BCADQmyhI3CO+MicLkkH5prkRnHI0ymafQKt40QyjCBrywkRi6ZCKihJa0d7u X7xVEx9qW2vj5xlK10bWoSz8CnHw/zVaw1t/ihtGfiW+PLXwCMmf0nIUSPFydMqPWLc2YAJG 2hp1cjYCFe3VPqf1NcXkekXVzHDADiV4uecaRP8WKZbW78gTxKSTVY1TIoIfU/j6GkQakyPa flUuMChbzCRcgXzV6HBtRy+D1yTZTGFMuipRej6MGEIMTdreyibkMGnMcQCGg3pjd7lcUhrl PlDF019Yff75gZAT7KoT9ZCYsklC4ij7d0frL0qr4fFKWZpFmBCZhYJly/+KxpY4QlGnABEB AAG0IktldmluIFpoZW5nIDxrZXZpbno1MDAwQGdtYWlsLmNvbT6JAVkEEwECAEMCGyMHCwkI BwMCAQYVCAIJCgsEFgIDAQIeAQIXgAIZARYhBHIErABsaPjTLF5dGurPD3bCLhCQBQJbQ4xk BQkLS0glAAoJEOrPD3bCLhCQVZYIALOrFiZNdEm8XW7XAYT9rBQGPcn2kzogE6Deob53k+6O zZo24mj+RosvSLjJmA/rojzfgIRXuc7lFJvWzMRXETNbtfQEMAQ4pEW2MpOaqDBRwlIRWCxw IfZav80IGYJHcVOCPj4WfD4WoRXBgg6heLAS2r/5UTNDE9DAFylXhy4HBEwT4MFMw/cpDELG HTckrVKptk1J0v6vpIXjFRq5zadWNWVxH6gSSkQx2XgXNVBfNYoeXpvX/1PRpZmWbWL1qSua 19KARF1jfn2MIJ3OyllzRzLIr/h20dhhcLSPWP4I6JssD0XGEdQqVEcf1OUfHztXpZD+1quu 5w3tgPYkvK65AQ0EUdl3vwEIAK3KQ4SdZtsCT/RhDqfCHKVRu92tZrpRiXvXJopxGX22s0fR g8AQs0httgtyY1H/tqzg/Z1428vA6pRHiBRyYfjVAwpKMvm6uaH0IbUNQdvI+OjXVfcn3PlR eFBsbAhJ1tW3IzQ4Enbz2aIOvjvFuMVVt2JfEj8IhRloRILkYihJRCbG9Pwx+LRBuQ8K4+0A qwLMcx+qXs2gzlBWsY0P9nyAO2A/91ikzmIj4VnumJNjeuMzduGn3egmmUo4PnulTDViK/l2 gjkhuPw6M60a6grGJ/LVYhDvI2J831lEb4tLpU6AYbAfLeyVeDH2n/vPVmOsqo/Y4v/nAHZ4 BB3zPdEAEQEAAYkBPAQYAQIAJgIbDBYhBHIErABsaPjTLF5dGurPD3bCLhCQBQJbQ4xoBQkL S0gpAAoJEOrPD3bCLhCQnZ8H/08oZq/HPEflQRlnVxYFBZaAwEUBexvtvRyJlvq+2YKNLawS 1brNe9ov8DijPIQPWBFzaVB+rWCjTxxHfnj9479RsqXYsLN0Dr2odAueZF9FSHwEyVyrvkX5 PJnVvjiVeUiyvYWUq/kaR1RcehcVJqNfNWXmHBShLAvbekzQM1GKkew5Zw0qlMn8+t4ll6K5 CSKDpxs8fXEnwooz4sat48ZoimoJzR1nVdJohmM8TYe6gU5ZeNqc1fZxkBO8UpNM6s4Ewigc pveatn5P/TfM4Q+4MaYa6Srm/hP241jqcMlcM9ci9yJoZyOuQla/0SZ+gbTL2ZkJ7jBRLQ0i /L9d4V8= Message-ID: <21c51e7a-e3c5-efed-6d13-9e981c118740@gmail.com> Date: Sun, 23 Dec 2018 22:09:39 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0BB326DEA2 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=jm6zPSYV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kevinz5000@gmail.com designates 2607:f8b0:4864:20::134 as permitted sender) smtp.mailfrom=kevinz5000@gmail.com X-Spamd-Result: default: False [-6.33 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.94)[-0.937,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.39)[ip: (-8.53), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.47), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[4.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 04:09:43 -0000 Hi, I'm going down a rabbit hole figuring out why i915 drm (12 snapshot but also earlier) can't set MTRR's on my computer: Failed to add WC MTRR for [0xe0000000-0xefffffff]: -28; performance may suffer (I haven't checked that this address range is sane yet, but even if it isn't, it should be returning something else.) That was a return code from a call to mem_range_attr_set. On amd64, that means x86_mrset is returning -28 (ENOSPC). That's sys/x86/x86/x86_mem.c returning ENOSPC. But, I'm not familiar with MTRR's on x86 and why FreeBSD does this search. There's an ominous comment at the top: /* * Modify/add a variable MTRR to satisfy the request. * * XXX needs to be updated to properly support "busy" ranges. */ Might this have something to do with that? My system: FreeBSD 12.0-RELEASE r341666 GENERIC amd64 CPU: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz (2591.64-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0xbfebfbff Features2=0x1fbae3ff AMD Features=0x28100800 AMD Features2=0x1 Structured Extended Features3=0x9c000000 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics Regards, Kevin -- Kevin Zheng kevinz5000@gmail.com | kevinz@berkeley.edu | PGP: 0xC22E1090 From owner-freebsd-hackers@freebsd.org Mon Dec 24 10:33:05 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0478E133DE1A for ; Mon, 24 Dec 2018 10:33:05 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C66F180E27 for ; Mon, 24 Dec 2018 10:33:03 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: by mail-ed1-x534.google.com with SMTP id b14so9784235edt.6 for ; Mon, 24 Dec 2018 02:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nuxi-nl.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tEioW2Tyxol1ZABex9pfz32sg/dlSYujho8T8hjusrU=; b=fV3xflZCs6zH3InTP7jgS3CDVvq//YqcSl8J01Dst+8ObkPTB3WZf8IpBexjSnXcDO bCEQogUl6YonmlgMLBKy1R0lxMtRnk36KYyIXBF+RAAb6O8TEJ0ellrZBdrdmfFRmWSl NiZU/W0roVe5GSIj/pGvsA3WuOuEx/6rRIi4JnYMVUgtFRAnjFlTHmBj1rPRojjB/z21 vjrWrcQX5Kyxlm01VS/K/P7O3DT0i4fCEL3+et8UMxoOqC0gRof9udR/p8D5j4HlJcqA j7GTfS4nmlL8KtDmyjVNdvNp+ID42PvnKzaJoeQX68o0XupuNs99mhLYYrS7Sc6+dCrM RB2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tEioW2Tyxol1ZABex9pfz32sg/dlSYujho8T8hjusrU=; b=l7ZCKM7X7O2ynsMnkUJpIXzWu/r8+HcHRGPSwz3/WgQOXVwXUmPsaeAsynkln8jGEG Pa5NfmG4ExtyVs6gSWuzMkxl3IkVx8VNW2y/raR+xPPso93S5vrNTDLBVmQ7jDtf36Mn oDlzn+zjqPcanBBRei7HLCAHm0/jkJsLmjJvI23ziDxYKi6rx+r5CejwSiVrqoroMC0X 8CFgMiW3uAHbMgfCPveegvnvmcxdu5AszKirtoWyunAu3vgFYXlGd1B/iUamFj6ZXEUw pgyURLfoSB9hxM5M/GILIn0AVYxBXaZGb3zwxH39vwcuSaCd9hiyZu5NAEPU4xUbCOwE VqKg== X-Gm-Message-State: AJcUuke8Lj79z+lTmLedEAnD/cvK1Du7xa5aXzCIealiYgCD3fmF42Hv hxrXbNMkh5dVEW9WXDKtiu5yJ9Uu8kfxrqCogh5l7NJYZAE= X-Google-Smtp-Source: ALg8bN4DuYEOOrEWvSdQWvSkoyZjWzCxbqjsuF6vNXxYfWu2vURgYdlDQ4oofsNfKNan7K6brp2t7HLWWDyQqBF31Ao= X-Received: by 2002:a17:906:4896:: with SMTP id v22-v6mr3510909ejq.85.1545647582305; Mon, 24 Dec 2018 02:33:02 -0800 (PST) MIME-Version: 1.0 References: <20181222202228.95f48408c8e2ebe9c40d9d75@gmail.com> In-Reply-To: <20181222202228.95f48408c8e2ebe9c40d9d75@gmail.com> From: Ed Schouten Date: Mon, 24 Dec 2018 11:32:36 +0100 Message-ID: Subject: Re: libsysctl: a C API for sysctl-mib-tree To: Alfonso Siciliano Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: C66F180E27 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nuxi-nl.20150623.gappssmtp.com header.s=20150623 header.b=fV3xflZC X-Spamd-Result: default: False [-5.48 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[nuxi-nl.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.76)[-0.756,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[nuxi.nl]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[nuxi-nl.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[ASPMX.L.GOOGLE.COM,ALT2.ASPMX.L.GOOGLE.COM,ASPMX5.GOOGLEMAIL.COM,ALT1.ASPMX.L.GOOGLE.COM,ASPMX4.GOOGLEMAIL.COM,ASPMX3.GOOGLEMAIL.COM,ASPMX2.GOOGLEMAIL.COM]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.41)[ip: (-8.82), ipnet: 2a00:1450::/32(-1.69), asn: 15169(-1.47), country: US(-0.08)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 10:33:05 -0000 Hi Alfonso, Op za 22 dec. 2018 om 20:26 schreef Alfonso Siciliano : > I' m currently working on a library called "libsysctl" which will > provide a C API to wrap kern_sysctl.c undocumented interface, build > mib-entry, entries-list and mib-tree in userspace and then to do > the work that /sbin/sysctl currently does. Great work! This would have been nice to have when I implemented the Prometheus metrics exporter for sysctl: https://svnweb.freebsd.org/base/head/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c?view=markup Best regards, -- Ed Schouten From owner-freebsd-hackers@freebsd.org Mon Dec 24 07:44:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C23AC1338687 for ; Mon, 24 Dec 2018 07:44:46 +0000 (UTC) (envelope-from scwsy00@gmail.com) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C830074600 for ; Mon, 24 Dec 2018 07:44:45 +0000 (UTC) (envelope-from scwsy00@gmail.com) Received: by mail-oi1-x22c.google.com with SMTP id m6so9431644oig.11 for ; Sun, 23 Dec 2018 23:44:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=fvQMOL2UEo6y9LaLeax/w3dUIr+fbRfAM8n/clukm5Y=; b=UE2gyA2AweOeZu/f3f5ts8P3rkyeIlOnyR0HDUB1RCOxoLACLLIuPEsy7lUFxnuwLo LYtZ3d6MTJ/GIaSDku1fmOL7ZvpNJB9tYoCMsBQq097Eh9t+t3yc+mXP3Sip5aBnjPgY HVs5z5bySTO8pHerocHHbQFw0WCoPVc7hD1p72cXJz05YC6BVkxF1cVWFtSvn7arqiLn X7t4BMiEKiC4mTgUH3alEMBGIK8/Cx5+FbAK6t7f4x4jO4iBvyyP3GD6tXFXeseDaMuO AzS9gK4smDg8uNFX8KxMWf8eJ4p2iFHeuALGTibPK1fUkzAc9ta98SM1g0H4M+SHFyjj Ns6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=fvQMOL2UEo6y9LaLeax/w3dUIr+fbRfAM8n/clukm5Y=; b=RlybCW06XPUfm6YWgd/5t64Bz0OFKXKcx2ZNvJxXtnL4wuyPId5UBfpIS13/+qXyyI 9zpWWC80MpUcWP2Ii5ZZ6Yn15Tdt3wMylJulaCKvBd6w2nTrHbFplxbnRiS1X+alBLFU XCJQzyhAlnIoNzDi9IZ6oftUi7VAyWChyhYhCZrbGgR0dGRzRbC+W2ecrBrnktEtRKVg 63ewd7HjvRVHLRYUBqBBj6ep2kFkfX0S/GN8yCnHuclvE1q3v+Ngn9rOsED/SieeAzqi wOonpadUjSGwZP2MJH3j8R3g/8PXiIAlGbUKPJWoClNsgAIL2jCbnLI9qsWQrAZKOfav SiTg== X-Gm-Message-State: AA+aEWavxD2f6aADqabMc7PxiEIxzZj8rASVaX7KXi1jMqtgke9qt7MS zMkJEAHXSfoZZr4F9HyrWrEyiq6XJLfMlx/BwgxQug== X-Google-Smtp-Source: AFSGD/X+RV4zVEcOpPrBUrfKee+i3KHOLRkQB5qwvW6OsLxNUpdAfKtG8Dj1tgDrYGjZ2h7GIfysY9ODC1kTpjkeaUQ= X-Received: by 2002:aca:b542:: with SMTP id e63mr7088206oif.125.1545637484900; Sun, 23 Dec 2018 23:44:44 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?5a6L6L6w5Lyf?= Date: Mon, 24 Dec 2018 15:44:33 +0800 Message-ID: Subject: Fwd: How to access physical memory To: freebsd-hackers@freebsd.org X-Rspamd-Queue-Id: C830074600 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UE2gyA2A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of scwsy00@gmail.com designates 2607:f8b0:4864:20::22c as permitted sender) smtp.mailfrom=scwsy00@gmail.com X-Spamd-Result: default: False [-6.08 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[c.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.59)[ip: (-9.53), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.47), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.48)[-0.482,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-Mailman-Approved-At: Mon, 24 Dec 2018 11:24:21 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 07:44:47 -0000 Forwarded Conversation Subject: How to access physical memory ------------------------ From: =E5=AE=8B=E8=BE=B0=E4=BC=9F Date: 2018=E5=B9=B412=E6=9C=8824=E6=97=A5=E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D= =8811:23 To: Hi I'm the new here . And I have some questions with pmap. [image: 1545542790461.png] See `*pml4` is the physical address of a page with some flag. And then updated the address to the Direct Map Space. Then We calculate the offset of the real pdp we need . How could it access to `*pdp`? it is a virtual addr in Direct Map Space. Why don't change the physical space directly since we have the physical page already. ---------- From: =E5=AE=8B=E8=BE=B0=E4=BC=9F Date: 2018=E5=B9=B412=E6=9C=8824=E6=97=A5=E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D= =8811:28 To: ``` pdp =3D (pdp_entry_t *)PHYS_TO_DMAP(*pml4 & PG_FRAME); /* Now find the pdp page */ pdp =3D &pdp[pdpindex & ((1ul << NPDPEPGSHIFT) - 1)]; *pdp =3D VM_PAGE_TO_PHYS(m) | PG_U | PG_RW | PG_V | PG_A | PG_M; ``` There is the code in _pmap_allocpte. =E5=AE=8B=E8=BE=B0=E4=BC=9F =E4=BA=8E2018=E5=B9=B412=E6= =9C=8824=E6=97=A5=E5=91=A8=E4=B8=80 =E4=B8=8A=E5=8D=8811:23=E5=86=99=E9=81= =93=EF=BC=9A From owner-freebsd-hackers@freebsd.org Tue Dec 25 01:25:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B008F135724B for ; Tue, 25 Dec 2018 01:25:30 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from mail-it1-x12d.google.com (mail-it1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 439AC75B30 for ; Tue, 25 Dec 2018 01:25:29 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: by mail-it1-x12d.google.com with SMTP id i145so17292153ita.4 for ; Mon, 24 Dec 2018 17:25:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli-net.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=QvM9tz8KZpYIXJmjnSZZot49b3tlgXvGeOq2B/du4X0=; b=QzvDH3rIs00FZX16oMK2fhAQEjhDfC73JjBRgHZJdqZPjr8ulNqYQ8gwhoiIaM4XCY BQvCvZlsa+33OQ0eofIU882HEIyAs66QNdbLKAEETtIDMR58qhgbpdMBLOg6nTUupnHo dPwDGsKYjAtmTX5vt002iCLm/kh/g9RacSr3M4Zzuoa1iNQQFc3ktE56C0byctIc6Yz7 SNoMb3gvAH2BWzNfqFKX0Mux5J/qil3QNOiws+/nQtLYeTCKymLWw9LMrwHUSawmPZLi h3O+lUybzq49WVzg8j9NX5WuC2ypQV4VXe6xqmteqtYup07fXvh69cWTU1bbTDFZ+oU3 4Gdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=QvM9tz8KZpYIXJmjnSZZot49b3tlgXvGeOq2B/du4X0=; b=ikZq3xYkm+ZYL5ex4qAOfOradpKXUnF85anqFLm7FowDqy8Wsp3+BHXlNRd2ZgXTyn 3bISBdt0JnpjBmRstBy73v/+7Tnsa31koRqQBVZeP68F/gChaHY8v0ylsjBIkLlQDCRA CkxmQRepc0hbBv9LrG7XTc4TiGOzF3Tpti1LVLP69aomb1b5QFRi+WKuhwu3FLlylxbd /HlTNg9/ENr86a7RMQdpR7H0CDIdlAyWjfljdblKfSn57an/pl8vQBKvLf1tA+VQCmKs 5VSSUdNfh58+4M9FozH113muchWV/cy4rqa5MI+yIdTXYY0U+91/3tYgwu6HqB6c6fOv 62Vw== X-Gm-Message-State: AA+aEWY3F1JjOPIOPOzMoJ4WY2qP1UDj4ER/J1N2BUZtQwxcBw0R8AzU txZG4d0yIju8UvlbUPznB9KJmYXl8kD9Vt5XPcbLuA== X-Google-Smtp-Source: AFSGD/VZkQC4VSL0VLw2Im2dxWGNK/0gUhGUAnR5Nj0qFreCWoK8ms4XL6p/s1t6rlzeF3kaCmDqzIeQqJDsRIwIjf0= X-Received: by 2002:a02:95e4:: with SMTP id b91mr8944556jai.15.1545701128028; Mon, 24 Dec 2018 17:25:28 -0800 (PST) MIME-Version: 1.0 From: Chuck Tuffli Date: Mon, 24 Dec 2018 17:24:39 -0800 Message-ID: Subject: core dumps running in bhyve To: freebsd-emulation@freebsd.org, FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 439AC75B30 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuffli-net.20150623.gappssmtp.com header.s=20150623 header.b=QzvDH3rI; spf=permerror (mx1.freebsd.org: domain of chuck@tuffli.net uses mechanism not recognized by this client) smtp.mailfrom=chuck@tuffli.net X-Spamd-Result: default: False [-5.60 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[tuffli-net.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[tuffli.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx3.googlemail.com,aspmx2.googlemail.com,aspmx5.googlemail.com,alt2.aspmx.l.google.com,aspmx4.googlemail.com]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[tuffli-net.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[d.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.97)[-0.974,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.31)[ip: (-8.15), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.48), country: US(-0.08)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2018 01:25:31 -0000 Using the latest bhyve, I'm seeing core dumps in the guest when running: nvmecontrol identify nvme0 against the emulated NVMe drive. The location of the core dump changes from run to run, but I suspect the root cause is a memory corruption caused by the transfer of the Identify data (4KB) back to the guest. This transfer of data is actually a memcpy to an address returned from vm_map_gpa() based on the physical address provided by the guest. Based on the signature of one of the core dumps, I modified nvmecontrol to always pass a 4KB aligned buffer to the driver instead of the (typically) unaligned address of the structure on the stack. With this change, nvmecontrol in the guest no longer core dumps. What I don't understand is why this changes the behavior. Do the addresses passed to vm_map_gpa() need to be page aligned? Or did moving the memory location from the stack to the heap merely mitigate what is corrupted? Thoughts? --chuck From owner-freebsd-hackers@freebsd.org Tue Dec 25 20:39:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 312EF135B60C for ; Tue, 25 Dec 2018 20:39:18 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1FFA6890FD for ; Tue, 25 Dec 2018 20:39:17 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id D2BCB135B60A; Tue, 25 Dec 2018 20:39:16 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48BD6135B608; Tue, 25 Dec 2018 20:39:16 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10150890F9; Tue, 25 Dec 2018 20:39:15 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-wr1-x430.google.com with SMTP id v13so14270028wrw.5; Tue, 25 Dec 2018 12:39:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :mime-version:content-disposition:user-agent; bh=p8j7FGytMgce/A838Q9WfXxBPVOhCaK8zg5FPlZSgak=; b=Y0U7EIp2G1lOzI70TKIfFGiZlJS1sdyACrqfcQ4dLtmzTOOTBLD15y0xB33PjUSIec shofDNlqAHDnBsxwh4VmcxTYF5qDeq7YCQGspY2jRreQ0D9IR0Nti4GF8ILH7v7BHBwj NVtzR4TgueMKknPryNz6NO7WHS1mHF7JCB2/1i0RFgpAtB0jrDPzSSeW16JJJjKCVm2x PBOM5lwTjGRf8f1PyuCy5ZIxpWsstOIyUz8QEKBbpdDAgYsm0/T9eb1QG8zv7SOA2sGN ZWhQxWJYJSzyoEWEksg10uaw+rffCvr+sj5u3svf3ltv/RcR8p5YgZpeeOk1/5y+Ull/ JWFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:mime-version:content-disposition:user-agent; bh=p8j7FGytMgce/A838Q9WfXxBPVOhCaK8zg5FPlZSgak=; b=QlnkEMpRIDU5VVSwYB1GzGCvwzkGP6PtPRx59DvufnwdTwUduavI1ctCcIuS3YUxdm Qpfc6mKmLV+BiR5hqgXF4LEgmlzt+Gic8QvhzbH1pBThWQD+/aJpROm37oBC7+hlWKZs s+iP8mEGAOt5E3XnWq7lnysU1ZpfVO4g4g4Stet9qO41REAIzmLozIeqf+pF+vMURXSv lLfU/+6pBp4gZn7cv9HD9vlHJhAjXb0oOfHyMjw/lPhP59jy5MNr4kAMPWezPZ/B0Yr2 gT3WNqIAODHfWCLkaXsqawOkWnSli2ONV7LnLVlmxBUeGnHKKMt0qM5sWaQj1iNTE7xQ 7KpQ== X-Gm-Message-State: AJcUukeuEhrcHbdjLtEpYRXQwcH8GiR0VjVfbMJSfCGmRjt3V90waZip jc/7mQdFRY9mc54RDGjzfty1ZC40 X-Google-Smtp-Source: ALg8bN4sTkbm2IIL/iJIuE6b0RZQaFLx9aRkF7oBnumqUY7Vn0CSzSgDdWKpKZP8AZN8iZLdE3FEgA== X-Received: by 2002:adf:fe8f:: with SMTP id l15mr15629237wrr.313.1545770352033; Tue, 25 Dec 2018 12:39:12 -0800 (PST) Received: from v2 (cpc92302-cmbg19-2-0-cust461.5-4.cable.virginm.net. [82.1.209.206]) by smtp.gmail.com with ESMTPSA id h16sm59549319wrb.62.2018.12.25.12.39.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 25 Dec 2018 12:39:10 -0800 (PST) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Tue, 25 Dec 2018 00:00:01 +0000 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: hackers@freebsd.org Cc: current@freebsd.org, stable@freebsd.org Subject: FreeBSD Quarterly Status Report - Third Quarter 2018 Message-ID: <20181225000001.GA16548@v2> Mail-Followup-To: hackers@freebsd.org, current@freebsd.org, stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline User-Agent: Mutt/1.11.1 (2018-12-01) X-Rspamd-Queue-Id: 10150890F9 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Y0U7EIp2; spf=pass (mx1.freebsd.org: domain of etnapierala@gmail.com designates 2a00:1450:4864:20::430 as permitted sender) smtp.mailfrom=etnapierala@gmail.com X-Spamd-Result: default: False [-7.32 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.93)[-0.929,0]; SIGNED_PGP(-2.00)[]; FORGED_SENDER(0.30)[trasz@freebsd.org,etnapierala@gmail.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[trasz@freebsd.org,etnapierala@gmail.com]; FORGED_RECIPIENTS(0.00)[hackers@freebsd.org ..,hackers@freebsd.org ...]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[freebsd.org]; IP_SCORE(-2.58)[ip: (-9.63), ipnet: 2a00:1450::/32(-1.71), asn: 15169(-1.48), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[0.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2018 20:39:18 -0000 --huq684BweRXVnRxX Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable FreeBSD Project Quarterly Status Report - 3rd Quarter 2018 With FreeBSD having gone all the way to 12, it is perhaps useful to take a look back at all the things that have been accomplished, in terms of many visible changes, as well as all the things that happen behind the scenes to ensure that FreeBSD continues to offer an alternative in both design, implementation, and execution. The things you can look forward to reading about are too numerous to summarize, but cover just about everything from finalizing releases, administrative work, optimizations and depessimizations, features added and fixed, and many areas of improvement that might just surprise you a little. Please have a cup of coffee, tea, hot cocoa, or other beverage of choice, and enjoy this culmulative set of reports covering everything that's been done since October, 2017. --Daniel Ebdrup __________________________________________________________________ FreeBSD Team Reports * Continuous Integration * Core Team * Ports Collection * Release Engineering Team * The FreeBSD Foundation Projects * 32-bit compatibility and other ABI cleanups * 4G/4G address space split for i386 * ACPI NVDIMM driver * Boot Loader * Building FreeBSD on non-FreeBSD hosts * Device Mode USB * ENA FreeBSD Driver Update * FreeBSD Graphics Team * FreeBSD/DTrace * ifuncs * Intel Work on Core Enabling and Security * Large scale package building * LLVM 7.0 - Sanitizers support improvements / Static code analysis * Performance improvements * Save/Restore/Migration support in bhyve * SMAP * String functions on the amd64 architecture * Usermode mapping of PCI BARs Architectures * Allwinner SoC Support * Armada 38x FreeBSD support * ARMv6 and ARMv7 image now use EFI loader * DTS Update * FreeBSD on POWER9 * FreeBSD on PowerNV (ppc64) * FreeBSD on RISC-V * PINE64-LTS Image * PocketBeagle Support * RPI Firmware/DTB/U-Boot Update Ports * KDE on FreeBSD * Puppet * scarab: CLI tool for Bugzilla-related workflows Documentation * Cleaning up the Wiki * Quarterly Reports Third-Party Projects * HardenedBSD 2018Q3 Update __________________________________________________________________ FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. Continuous Integration Links FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org/ FreeBSD Jenkins wiki URL: https://wiki.FreeBSD.org/Jenkins freebsd-testing Mailing List URL: https://lists.FreeBSD.org/mailman/listinfo/freebsd-testing freebsd-ci Repository URL: https://github.com/freebsd/freebsd-ci Contact: Jenkins Admin Contact: Li-Wen Hsu The FreeBSD CI team maintains continuous integration tasks for FreeBSD. The CI system regularly checks the changes committed to the project's Subversion repository can be successfully built, and performs various tests and analysis with the build results. The CI team also maintains the archive of the artifact built by the CI system, for the further testing and debugging needs. Starting from June 2018, the project is sponsored by the FreeBSD Foundation in hardware and staff. For more details of the sponsored projects, please refer to: https://www.freebsdfoundation.org/news-and-events/newsletter/freebsd-fo undation-update-september-2018/ In addition to that, we also helped checking regressions for OpenSSL 1.1.1 update and test continuously for 12-STABLE branch. We had meetings and working groups at two developer summits during 2018Q3: * BSDCam 2018 * EuroBSDCon 2018 Work in progress: * Fixing the failing test cases and builds * DTrace test * ZFS test * GCC build * Adding drm ports building test against -CURRENT * Adding tests for selected project branches, e.g.: clang700-import * Adding new hardware to the embedded testbed * Implementing automatic tests on bare metal hardware * Planning running ztest and network stack tests __________________________________________________________________ Core Team Contact: FreeBSD Core Team Much of Core's focus for the past months has been on three items: 1. Coordination between different groups to support the upcoming 12.0 release. The timing of the OpenSSL 1.1.1 release posed challenges, the new OpenSSL version included API changes, many components of the base system and ports required changes. Staying with the older OpenSSL in 12.0 was not a feasible option, because it would have meant backporting many changes to a version of OpenSSL that would be unmaintained by the upstream source. 2. Discussions with the release engineering team and Scott Long about updating the FreeBSD release process. Topics for exploration include: * having more frequent point releases * changing the support model * revising and improving the tooling used to manage the tree and releases * additional topics as they are discovered 3. Gathering information to make decisions more data-driven. For example, we are planning developer and user surveys. If there are questions that you think should be added to the survey, please discuss them on freebsd-arch@. We are exploring ways for automated user-driven hardware usage data to understand the changing ways our software is used and to target better hardware support. Here are other noteworthy events (in chronological order) since the last quarterly report. 2017 Q4 * Sean Eric Fagan's (sef@) commit bit was reactivated with a period of re-mentoring under Alexander Motin (mav@). * The MIPS architecture was promoted to tier 2 status. * Core approved changes to the Code of Conduct. * All fortune data files, except freebsd-tips, were removed in r325828. * Core approved the adoption of a policy requiring any license exceptions to be recorded alongside code. * Gordon Tetlow (gordon@) became the new security officer. * Core approved the use of SPDX tags. 2018 Q1 * Jeb Cramer (jeb@) was awarded a src commit bit under the mentorship of Sean Bruno (sbruno@) and Eric Joyner (erj@). * Members of the CoC Review Team were approved. The membership is to be reviewed once per year. * A vendor commit bit was awarded to Slava Shwartsman (slavash@) of Mellanox Technologies under the mentorship of Konstantin Belousov (kib@) and Hans Petter Selasky (hselasky@). * Walter Schwarzenfeld was awarded project membership. * Brad Davis (brd@) was awarded a src commit bit under the mentorship of Allan Jude (allanjude@) with Baptiste Daroussin (bapt@) as co-mentor. * Vincenzo Maffione (vmaffione@) was awarded a src commit bit under the mentorship of Hiroki Sato (hrs@). * Ram Kishore Vegesna (ram@) was awarded a src commit bit under the mentorship of Kenneth D. Merry (ken@) and Alexander Motin (mav@). 2018 Q2 * Tom Jones (thj@) was awarded a src commit bit under the mentorship of Jonathan T. Looney (jtl@). * Matt Macy's (mmacy@) commit bit was restored under the mentorship of Sean Bruno (sbruno@). * Breno Leitao (leitao@) was awarded a src commit bit under the mentorship of Justin Hibbits (jhibbits@) with Nathan Whitehorn (nwhitehorn@) as co-mentor. * Leandro Lupori (luporl@) was awarded a src commit bit under the mentorship of Justin Hibbits (jhibbits@) with Nathan Whitehorn (nwhitehorn@) as co-mentor. * The handover from the ninth to the tenth elected Core team took place. The tenth Core members are: Allan Jude (allanjude@), Benedict Reuschling (bcr@), Brooks Davis (brooks@), Hiroki Sato (hrs@), Warner Losh (imp@), Jeff Roberson (jeff@), John Baldwin (jhb@), Kris Moore (kmoore@), and Sean Chittenden (seanc@). * Joseph Mingrone (jrm@) was appointed the Core secretary under mentorship of the retiring Core secretary, Matthew Seaman (matthew@). * The new team liaisons were decided. portmgr: Sean, doceng: Hiroki, secteam: Brooks, re: John, clusteradm: Allan, CoC: Warner, Foundation: Benedict, bugmeister: John, CI: Sean. * David Maxwell (dwm@) was awarded project membership. * Daichi Goto's (daichi@) commit bit was reactivated with a period of re-mentoring under George Neville-Neil (gnn@). * A vendor commit bit was awarded to Ben Widawsky (bwidawsk@) of Intel under the mentorship of Ed Maste (emaste@). 2018 Q3 * Core decided to begin meeting twice per month in an attempt to catch up with many new agenda items. * Li-Wen Hsu (lwhsu@) was awarded a src commit bit under the mentorship of Mark Johnston (markj@) with Ed Maste (emaste@) as co-mentor. * Samy al Bahra was awarded project membership. * George Neville-Neil (gnn@) was approved to begin co-mentoring Vincenzo Maffione (vmaffione@). __________________________________________________________________ Ports Collection Links About FreeBSD Ports URL: https://www.FreeBSD.org/ports/ Contributing to ports URL: https://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributing/= ports-contributing.html FreeBSD ports monitoring URL: http://portsmon.FreeBSD.org/index.html Ports Management Team URL: https://www.FreeBSD.org/portmgr/index.html FreeBSD portmgr (@freebsd_portmgr) URL: https://twitter.com/freebsd_portmgr/ FreeBSD Ports Management Team (Facebook) URL: https://www.facebook.com/portmgr FreeBSD Ports Management Team (Google+) URL: https://plus.google.com/communities/108335846196454338383 Contact: Ren=C3=A9 Ladan During the first quarter of 2018, the number of ports grew to almost 32,000. In 2018Q1, there were 2,100 open PRs with fewer than 600 unassigned. There were 7,900 commits from 169 committers. Compared to last quarter, the number of commits grew by 18% and the number of PRs dropped by 25%. Those are some good numbers! During the 2018Q2 and 2018Q3 quarters, the number of ports grew to just under 34,000. The number of open PR grew to almost 2,500 with fewer than 600 of those unassigned. A total of 175 committers made almost 14,200 commits. Compared to the first quarter, the number of commits dropped by 10% and the number of PRs grew by 19%. During the last three quarters, portmgr took twelve commit bits in for safekeeping: daichi@, deichen@, ian@, junovitch@, kevlo@, maho@, nemysis@, pawel@, rea@, tabthorpe@, vg@, and wxs@. Portmgr welcomed thirteen new committers in 2018Q2 and 2018Q3: * Devin Teske (dteske@) * Eric Turgeon (ericbsd@) * Fernando Apestegu=C3=ADa (fernape@) * Fukang Chen (loader@) * Gleb Popov (arrowd@) * Jesper Schmitz Mouridsen (jsm@) * John Hixson (jhixson@) * Kevin Bowling (kbowling@) * Koichiro IWAO (meta@) * Mateusz Piotrowski (0mp@) * Matthias Fechner (mfechner@) * Sergey Kozlov (skozlov@) The following committers returned after a hiatus: * Ion-Mihai Tetcu (itetcu@) * Kevin Lo (kevlo@) * Sean Chittenden (seanc@) During the last three quarters, Antoine Brodin (antoine@) ran no fewer than 113 exp-runs against the ports tree. These runs were executed to test updates, perform cleanups, and make improvements to the framework and the base system. Most of the runs were for port upgrades, but others include LLD progress, changes to the default port versions, improved support for armv6, armv7, and RISC-V architectures, removed old base system functionality, new USES, and better matching pkg-plist with Makefile options (DOCS and EXAMPLES). Five new USES values were introduced: * apache: handle dependencies on the Apache web server and modules * eigen: automatically depend on math/eigen2 or math/eigen3 * emacs: handle dependencies on the Emacs editor and modules. * gl replaces the old USE_GL from bsd.port.mk * qt-dist, qt:4 and qt:5 replace the old USE_QT from bsd.qt.mk The EXTRA_PATCHES functionality has been extended to support directories, where it will automatically apply all patch-\* files to the port. Ports using USES=3Dphp:phpize, php:ext, php:zend, and php:pecl have been flavored and packages are now automatically built for all versions of PHP that are supported (5.6, 7.0, 7.1 or 7.2). 2018Q3 had updates of major ports: pkg 1.10.5, Chromium 65.0.3325.181, Firefox 59.0.2, Firefox-ESR 52.7.3, Ruby 2.3.7/2.5.1 and Qt5 5.9.4. The default version of PHP was changed from 5.6 to 7.1. The former version of PHP is no longer supported by the developers. The default versions of Samba and GCC are now respectively 4.7 and 7. The Xorg ports have been reorganized and there have been changes to net/openntpd. Please review the UPDATING file for relevant details. Open tasks: * The number of commits dropped somewhat over the last three quarters, leaving more PRs unresolved. If possible, please pick up some PRs and improve everyone's experience. __________________________________________________________________ Release Engineering Team Links FreeBSD 10.4-RELEASE announcement URL: https://www.FreeBSD.org/releases/10.4R/announce.html FreeBSD 11.2-RELEASE announcement URL: https://www.FreeBSD.org/releases/11.2R/announce.html FreeBSD 12.0-RELEASE schedule URL: https://www.FreeBSD.org/releases/12.0R/schedule.html FreeBSD development snapshots URL: https://download.FreeBSD.org/ftp/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team The FreeBSD Release Engineering Team responsibilities include: * setting and publishing release schedules for official project releases of FreeBSD * announcing code slushes, freezes, and thaws * maintaining the respective branches for all supported releases The FreeBSD Release Engineering Team, led by Marius Strobl, completed the 10.4-RELEASE in early October 2017. FreeBSD 10.4-RELEASE was the fifth release from the stable/10 branch, which built on the stability and reliability of 10.3-RELEASE. The FreeBSD 11.2-RELEASE cycle started April 20, 2018 with the announcement of the code slush. The first stage progress was continued throughout the rest of the quarter with the code freeze, followed by three BETA builds, three RC builds, and the final release build was announced June 27, 2018. The FreeBSD Release Engineering Team started the 12.0-RELEASE cycle August 10, 2018 with the announcement of the code slush. The code freeze followed on August 24, 2018. The tentative date for the stable/12 branch was expected to be September 21, 2018. Due to unforeseen circumstances with upstream code that was necessary to include in 12.0-RELEASE, the tentative release schedule needed to be adjusted several times. The API changes in the updated version of the upstream code required changes to be made for all base system utilities that linked with the upstream code. By the end of the 2018Q3 quarter, the stable/12 branch had not been created due to this delay. Throughout the remainder of 2018Q3, several development snapshots builds were released for the head, stable/11, and stable/10 branches. Much of this work was sponsored by the FreeBSD Foundation. __________________________________________________________________ The FreeBSD Foundation Contact: Deb Goodkin The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and release engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what the FreeBSD Foundation did to help FreeBSD last quarter: Partnerships and Commercial User Support As a 501(c)(3) non-profit, we don't directly support commercial users, but we do work with them to understand their needs and help facilitate collaboration with the community. Last quarter we met with a few key FreeBSD users and supporters, to discuss pain points, how they can contribute back to FreeBSD, and what technologies they would like to see supported, to support FreeBSD over more of their technologies and products. As many of you know, we formed a partnership with Intel around one and a half years ago. Since then the people we worked directly with left the company, but it moved us into a new relationship with their Open Source Technology Center (OTC). We are very encouraged that Intel has dedicated additional resources from the OTC to work on FreeBSD in addition to existing resources from the networking group and other technologies such as QuickAssist. Much of the work has been focused on security and OS mitigations but we're also focusing on other areas such as power management and persistent memory. In May and again in July we traveled to Intel's Hillsboro campus to meet with management and engineers from OTC and the networking team. We presented an overview of the project and Foundation and also discussed key markets and vendors who use FreeBSD in their products or services and their future requirements. Intel was also interested in learning more about who contributes to FreeBSD. Along those lines we've done some work with OTC to create scripts and organizational mappings to answer that question. Note that we do need developers to help us update and maintain the organizational mappings as we understand that developers do tend to move around and contractors are often working on behalf of multiple organizations. Fundraising Efforts Our work is 100% funded by your donations. As of September 30, we raised $328,482. Our 2018 fundraising goal is $1,250,00 and we are continuing to work hard to meet and exceed this goal! Please consider making a donation to help us continue and increase our support for FreeBSD: https://www.FreeBSDfoundation.org/donate/. We also have a new Partnership Program, to provide more benefits for our larger commercial donors. Find out more information at https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-progra m/ and share with your companies! OS Improvements The Foundation improves the FreeBSD operating system by employing technical staff to maintain and improve critical kernel subsystems, add features and functionality, and fix problems. This also includes funding separate project grants like the arm64 port, porting the blacklistd access control daemon, and the integration of VIMAGE support, to make sure that FreeBSD remains a viable solution for research, education, computing, products and more. We kicked off or continued the following projects last quarter: * OpenZFS RAID-Z Expansion project * Headless mode out-of-the-box for embedded ARM boards like the Beaglebone Black * Performance and scalability improvements Having software developers on staff has allowed us to jump in and work directly on projects to improve FreeBSD such as: * ZFS improvements * New Intel server support * kqueue(2) updates * 64-bit inode support * Stack guard * Kernel Undefined Behavior Sanitizer * Toolchain projects * i915 driver investigation * NVDIMM support in acpiconf(8) * Continuous integration dashboard (web page and physical hardware) * FAT filesystem support in makefs(8) Continuous Integration and Quality Assurance The Foundation provides a full-time staff member who is working on improving our automated testing, continuous integration, and overall quality assurance efforts. Foundation employee Li-Wen Hsu set up new CI servers to speed up amd64 build and test jobs, to reduce the latency between changes being committed and results being available. Li-Wen also set up a staging / development server in order to test changes to the CI system itself without affecting production results. We have also started a small hardware test lab, currently connected to the staging server, that tests the full boot and test cycle on physical hardware. In the near future additional hardware devices will be added, and this will migrate to the production CI server. Release Engineering The Foundation provides a full-time staff member to lead the release engineering efforts. This has provided timely and reliable releases over the last five years. Foundation employee Glen Barber continued leading the efforts on the upcoming 12.0-RELEASE. For details surrounding the work involved and progress thus far on 12.0-RELEASE, please see the FreeBSD Release Engineering Team section of this quarterly status report. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. Check out some of the advocacy and education work we did last quarter: * Organized and ran the Essen FreeBSD Hackathon in Essen, Germany * Participated in the FreeBSD Developer Summit BSDCam, in Cambridge, England * Represented FreeBSD at the ARM Partner Meeting * Presented and taught about FreeBSD at SdNOG 5 in Khartoum, Sudan * Exhibited and gave a talk at OSCON 2018 in Portland, OR * Exhibited at the 2018 Grace Hopper Celebration and sponsored as a Silver Non-Profit Sponsor * Exhibited at COCON 2018 in Taipei, Taiwan * Sponsored and gave presentations and tutorials at EuroBSDCon in Bucharest, Romania * Organized and ran the Bucharest FreeBSD Developer Summit * Sponsored the 2018 USENIX Security Symposium in Baltimore, MD as an Industry Partner * Provided FreeBSD advocacy material * Sponsored the 2018 USENIX Annual Technical Conference in Boston, MA as an Industry Partner * Sponsored the OpenZFS Developer Summit as a Silver Sponsor * Presented and taught about FreeBSD at SANOG32 in Dhaka, Bangladesh * Sponsored the SNIA Storage Developer Conference 2018 as an Association Partner * Provided 11 travel grants to FreeBSD contributors to attend many of the above events. We continued producing FreeBSD advocacy material to help people promote FreeBSD around the world. Read more about our conference adventures in the conference recaps and trip reports in our monthly newsletters: https://www.freebsdfoundation.org/news-and-events/newsletter/ We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. Last quarter we published the July/August issue that you can find at https://www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you! __________________________________________________________________ Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. 32-bit compatibility and other ABI cleanups Contact: Brooks Davis As part of maintaining an external ABI (application binary interface) compatibility layer, I've been improving FreeBSD infrastructure, primarily the 32-bit compatibility layer. One of FreeBSD's strengths is that we can easily support many ABIs. This includes support for a.out format executables (vs the standard ELF), support for i386 on amd64, the Linux emulation layer, etc. This infrastructure has existed for decades and not every design decision has stood the test of time. Support has also been incomplete in a number of areas (e.g. network management under 32-bit emulation). Committed improvements include: * Improved ioctl and sysctl support to allow ifconfig and netstat to work in 32-bit compat mode. * Migration from a model of translating ioctl commands and data structures at the kernel boundary to translating where the commands are processed. This is a correctness improvement (`ioctl` commands do not have meaning outside the specific file descriptor in question) and improves code reusability (my out-of-tree work will soon include a 64-bit compatibility layer.) * Simplifications of the generic ELF process execution path by Ed Maste, John Baldwin, and myself. A number of simplifications including minor speedups have been committed. Portions of this work were developed by SRI International and the University of Cambridge Computer Laboratory (Department of Computer Science and Technology) under DARPA/AFRL contract FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme and under DARPA contract HR0011-18-C-0016 ("ECATS"), as part of the DARPA SSITH research programme. Work in progress includes cleanups to the APIs used by the kernel when creating processes and continued ioctl improvements. Work is also underway to generate the 32-bit system call list from the "default" list. The remaining ioctl commands handled in sys/compat/freebsd32/freebsd32_ioctl.c need to be migrated to the point of implementation. Help with the latter would be appreciated. __________________________________________________________________ 4G/4G address space split for i386 Contact: Konstantin Belousov Most 32-bit FreeBSD architectures, including i386, started to suffer from the rapid growth of the size of software during the past decade. When a 32-bit address space is enough space for a given task, 32-bit mode still has an intrinsic advantage over 64-bit mode, due to less memory traffic and more economical use of caches. It has grown harder to provide the self-hosting i386 system build due to the increase in size of the build tools. The FreeBSD i386 kernel, prior to the 12.0-RELEASE version, split the 4GB address space of the platform into 3GB (minus 4MB) accessible to userspace accesses and 1GB for kernel accesses. Neither kernel nor userspace could access a full 4GB address space. Programs that require very large virtual address spaces, such as clang when compiling or lld when linking, could run out of address space: 3GB of address space was insufficient for their operation. The kernel also had trouble fitting into the traditional 1GB limitation of address space with the modern sizing for network buffers, ZFS and other KVA-hungry in-kernel subsystems. In FreeBSD 12, the i386 architecture has been changed to provide dedicated separate address spaces for userspace and kernel, giving each mode full access to 4GB (minus 8MB) of usable address space. The userspace on the i386 architecture now has access to the same amount of address space as the compat32 subsystem in the amd64 architecture kernel. The increase in kernel address space enables further growth and maintainability of the i386 architecture. The split 4GB/4GB user/kernel implementation uses two page directory entries (PDEs) shared between modes: one for mapping the page table, another for the mode switching trampoline and other required system tables. The required system tables, which must always be mapped, regardless of kernel or user mode, includes such things as the GDT/IDT/TSS entries. Significant changes were made to the locore code. The page table creation portion of the code was completely rewritten from assembly to C, improving readability and maintainability of the code. Because the user address space is no longer shared with the kernel, the copyout(9) functions were rewritten to make a transient mapping of userspace pages for the duration of any needed accesses. The initial implementation used the vm_fault_quick_hold_pages() framework, but this was later optimized by temporarily switching to user mode mappings from a trampoline, and then using hand-written assembler routines to perform a faster small block copy operation. Future plans for the ongoing maintenance of i386 include making the i386 pmap capable of runtime selection of the PAE or non-PAE page table format and supporting NX (no execute) mappings for regular i386 kernel. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ ACPI NVDIMM driver Contact: Konstantin Belousov NVDIMM is a technology which provides non-volatile memory with access characteristics similar to regular DRAM, which is the technology that implements the normal memory address space of a host. There are ACPI and UEFI specifications that define platform independent ways to detect and enumerate the presence of NVDIMMs. These specifications allow the retrieval of most of the data needed to allow proper application use of the NVDIMM storage. A new FreeBSD driver parses the ACPI NFIT table which lists NVDIMMs, their operational characteristics, and the physical address space where the NVDIMM memory is accessible. The driver presents each address region as two devices: One device allows userspace to open(2) a devfs node, which can be read/written/mapped from the application. This mapping is zero-copy. The second device is a geom disk(9), which makes it possible to use NVDIMM for the backing storage for a normal FreeBSD filesystem, such as UFS, ZFS, or msdosfs. Note that buffer cache/mapping of files from a traditional filesystem created over NVDIMM causes an unneeded double-buffering. Empirically, on typical modern hardware, NVDIMM regions are located far from the regular DRAM backed memory in the address space, and have attributes that are not compatible with regular DRAM memory. This makes it unfeasible to extend the kernel's direct map to provide the kernel mappings for the NVDIMM regions. A new pmap KPI was designed, pmap_large_map(9), which allows efficient mapping of very large physical regions into the KVA. The new code has some optimizations to the cache flushing operations over the mapped regions, which is needed to efficiently support bio flushes from a filesystems using the NVDIMM storage. The NVDIMM driver is the first user of the new KPI, but the new KPI might also be useful for the NTB driver. TODO: * Intel is currently working on extending the driver to support UEFI namespaces. * A DAX-capable filesystem is needed, which solves the issue of double-buffering. Our tmpfs already provides VM facilities which allows it to avoid double-buffering for mmap, which can be reused there. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Boot Loader Contact: Warner Losh Contact: Kyle Evans Contact: Toomas Soome The FreeBSD boot loader lives in src/stand (prior releases had it in sys/boot and lib/libstand). It covers all the code that the project provides that interacts with the hardware before the kernel starts. The LUA interpreter added earlier in 2018 was made default in 2018Q3. Due to undiagnosed booting issues, the LUA interpreter has been disabled on sparc64 and all powerpc. The LUA interpreter is scheduled to replace the FORTH interpreter entirely in FreeBSD 13, although the FORTH interpreter will remain available as a build option in FreeBSD 12. The plans are not to remove the FORTH loader for about a year after 12.0 release, or approximately January 2020. Platforms not currently working with the LUA interpreter have until that date to resolve the issues. At this point, the LUA scripts implement everything that the FORTH scripts did. Where there was ambiguity in the spec, or where the FORTH scripts were more forgiving than was strictly documented, every effort has been made to improve the documentation and follow the old FORTH behavior, or document the new behavior where the old behavior was clearly a bug. It's anticipated that no further changes to the FORTH loader or the FORTH scripts will happen. They are quite mature and bullet proof at this point and it's unlikely that an undiscovered bug will need to be fixed before retirement. Other work in progress includes Toomas Soome's port to OpenIndiana. In addition to porting, he's enhanced the code in a number of ways (both in the block layer, and UEFI). Many of his improvements have been committed to FreeBSD, though a few remain and hopefully will be entering the tree soon after the freeze lifts. UEFI booting has been greatly enhanced. There are still some machines that have issues with the default BootXXXX variables or something else in the environment that are being investigated. We hope to understand the problems well enough to provide a fix for FreeBSD 12.0. Ian Lepore has reworked the GELI support so that it is architecture independent and can be used on any architecture we support. There are also efforts underway to support booting signed images, improved crypto booting options, and implement Multiboot 2.0 support. __________________________________________________________________ Building FreeBSD on non-FreeBSD hosts Links Wiki =20 URL: https://wiki.FreeBSD.org/BuildingOnNonFreeBSD GitHub project URL: https://github.com/arichardson/freebsd/tree/crossbuild-aug2018 Contact: Alex Richardson Currently FreeBSD can only be built on a FreeBSD host. However, most free CI tools only allow building on Linux or macOS and therefore can not be used for building the FreeBSD base system. It is sometimes useful to cross-build FreeBSD for a remote machine or an emulator even if the build machine is not running FreeBSD. The goal of this project is to allow building FreeBSD on both Linux and macOS hosts and in the future it may be extended to allow compiling on a Windows host. This work originates from the CHERI project and was motivated by multiple cases of people wanting to try out CheriBSD but not being able to compile it since they did not have a FreeBSD system available for compiling. Once completed this project will also allow developers to contribute to FreeBSD even if they don't have access to a FreeBSD build system. The current set of patches for this project can be found on GitHub. With the current prototype it is possible to compile both world and kernel for architectures that use the clang compiler and for MIPS64, which uses gcc. However, some options such as LOCALES are not supported yet and require further changes before the bootstrap tools can be built on Linux/macOS. Some changes required for building on non-FreeBSD have already been merged to HEAD but there are still a rather large number of changes that need review. If you are interested in getting this into HEAD and would like to help, please try the current prototype and report any issues to arichardson@FreeBSD.org. If you can help with reviewing the changes please contact arichardson@FreeBSD.org to be added to any pending Phabricator reviews. __________________________________________________________________ Device Mode USB Links Handbook chapter URL: https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/usb-dev= ice-mode.html Contact: Edward Tomasz Napierala Many embedded boards include hardware which supports device side USB - the ability for the board to present itself to another system as a USB drive, network adapter, or a virtual serial port. The FreeBSD USB stack has supported this functionality for quite some time, but it has not been used to its fullest extent. The goal of this project was to fix that - to document the functionality, possibly fix some bugs, and to make it easy to use, automating it as much as possible. Starting with FreeBSD 12.0, this functionality is enabled out of the box. This means you can connect your BeagleBone Black's (using its USB client socket) or a Raspberry Pi 0 (using the On-The-Go (OTG) port) to your laptop, and you'll get a virtual USB serial port, which serves as a system console, with getty(8) waiting for you to log in. This means you no longer need to look for a keyboard and a screen, or mess with the console cables just to configure your system. You can also switch it to provide network interface, or present itself as a USB drive - it's all documented in the FreeBSD Handbook. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ ENA FreeBSD Driver Update Links ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/R= EADME Contact: Michal/ Krawczyk ENA (Elastic Network Adapter) is the smart NIC which is used in the virtualised environment of Amazon Web Services (AWS). It supports multiple queues and can handle up to 25 Gb/s, depending on the instance type on which it is used. Since last report, ENA versions v0.8.0 and v0.8.1 have been released, which introduced many bug fixes, new features, optimization, stability and error recovery improvements. The last is especially important on the AWS, where the instances have to be reliable as they may be running very sensitive functions and the down time of the VM should be reduced to minimum. The v0.8.0 and v0.8.1 release patches included: * Upgrade of the HAL to version v1.1.4.3 * Improvement to the reset routine - the driver is now triggering reset from more fault points and is passing the reset reason to the device, which can perform the reset adequately to the encountered error. * Device statistics (like global Tx and Rx counters) are no longer read directly from the device. The only exception is Rx drops, which are still read using the AENQ descriptor. * The RX Out Of Order completion feature was added, which enabled to cleanup the RX descriptors out of order by keeping trace of all free descriptors. * RX ring is now being monitored, to prevent the ring from stalling. * Error handling paths were reworked and fixed. * Driver was covered with branch prediction statements, to make the most of this CPU feature in the hot paths. * Fix handling of the DF flag in the IP packets. * Add dynamic logging and reduce number of messages being printed by the driver. * MTU configuration now is being verified using the device capabilities instead of a constant value. * Do not pass packet header length hint to the device, because for the chained mbufs it may be problematic to determine header length, if the header is split into multiple segments. This project was sponsored by Amazon.com Inc.. __________________________________________________________________ FreeBSD Graphics Team Links Project GitHub page URL: https://github.com/FreeBSDDesktop Contact: FreeBSD Graphics Team Contact: Niclas Zeising The FreeBSD X11/Graphics team is responsible for the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications. There have been a lot of changes since the last report. The most important one is the change of driver distribution and updates. On FreeBSD 11.2 and later modern graphics drivers using the Linux KPI subsystem are found in ports. These give much improved support for Intel and AMD graphics hardware, however, they are currently only available for amd64. The legacy drivers available in the FreeBSD base system are also available in the ports tree, since they cause issues with the new drivers. They will remain in tree for 11.2 and 12, but work is on-going to have them removed for 13, except for arm. The easiest way to install the new drivers is to install graphics/drm-kmod which will install the correct driver depending on your architecture and FreeBSD version. There have been changes to the ports as well. Most notably is the changed handling of OpenGL dependencies, which has moved to USES instead of being handled directly in bsd.port.mk. Other big infrastructure changes is the move from individual \*proto packages to xorgproto, and turning that into a build time dependency. Many thanks to portmgr for help with exp-runs for these changes. There have been updates to applications and libraries as needed. On the project management side, there is ongoing work to set up a more efficient way of working, including bi-weekly conference calls to discuss the current works in progress. Notes from these conference calls will be posted on the mailing list. Looking forward, the current major work in progress is to update the graphics driver to be on par with Linux 4.17. The code is merged, but patching and bug fixing is ongoing. There is also work to port the VMware guest graphics driver, vmwgfx, to FreeBSD and to the Linux KPI, to get better graphics support in VMware. Lastly, on the driver side is to get the new graphics drivers to work on i386 as well. Experimental support for this exists in the code repository, but is not yet merged to the FreeBSD ports tree. In userland, the biggest things happening is the update of the input stack, including libinput and supporting libraries. Work is also ongoing on updating MESA libraries. On the project management side, the most important tasks is to continue to work on the team, and how we work internally. We are also working on setting up a list of requirements for testing, so that we can be reasonably assured that updates won't cause regressions. People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our Gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. We are also available in #freebsd-xorg on EFNet. We also have a team area on GitHub where our work repositories can be found: https://github.com/FreeBSDDesktop __________________________________________________________________ FreeBSD/DTrace Contact: George Neville-Neil Contact: Domagoj Stolfa DTrace is a whole-system debugging tool in FreeBSD and is one of the actively supported projects during the past year. A research prototype of a distributed version of DTrace and a version of DTrace that can trace bhyve virtual machines from the host FreeBSD system are currently under development at the University of Cambridge as a part of the CADETS project. Recent developments include the creation of dlog, an in-kernel DTrace consumer which is able to publish to Kafka, and improvements to early boot and shutdown tracing. On the virtualisation front, improvements were made in the ability to dereference and follow pointers inside guests from the host in the probe context by implementing a nested page table walk inside DTrace for Intel architectures. The CADETS project has started formalizing DTrace in HOL4 which enables automated test generation, high assurance of DTrace implementations in terms of adherence to the specification and exploration of all allowable behaviors for a given D script. Currently, the formal model contains most of DIF instructions and a code generator for them, providing the ability to run DIF programs specified using the model using FreeBSD's DTrace implementation. As a result of all of this, a number of changes were upstreamed to the FreeBSD auditing subsystem and new variables such as jid and jailname were added to DTrace which can be accessed from D scripts. OpenDTrace Specification 1.0 has been published which covers the internal workings of DTrace in general, and its adaptation to various operating systems in particular. This work was sponsored by AFRL/DARPA through the CADETS project. Ruslan Bukin (br@) has added C-compressed ISA extension support to the RISC-V FBT provider as a part of the ECATS project. Mark Johnston (markj@) has done some work to fix interactions between FBT and ifuncs. ifuncs are a toolchain feature which allow programmers to select a function's implementation at boot-time, rather than at compile-time. For instance, on amd64, memcpy() is an ifunc and may be implemented by either memcpy_erms() or memcpy_std(). FBT created probes for the implementation functions, but we needed some extra support to ensure that fbt::memcpy:entry continues to work as expected. Similar work is needed for the PID provider, but is still pending. Microsoft showed a working demo of DTrace, which was ported from FreeBSD. Added to FreeBSD base in 11.2, dwatch is a new DTrace tool, developed by Devin Teske (dteske@), for automating complex queries for data and surgically tapping the kernel. In base there are 85 profiles for interpreting domain-specific data with another 17 available from ports making a total of over 100 different pipelines from which you can extract data in multiple formats. dwatch also simplifies observation of over 100,000 probe points available in FreeBSD, making it easy to find any process, thread, or jail triggering any probe. On top of all that, dwatch profiles can leverage higher-level languages such as python, perl, sh, and many more. __________________________________________________________________ ifuncs Contact: Konstantin Belousov Contact: Ed Maste Contact: Mark Johnston Contact: Mateusz Guzik An ifunc is a special construct in an ELF object, which allows for the selection of the implementation for the given symbol at runtime, when the ELF module gets the final relocations applied. The selection of which code to use is governed by the small piece of user provided code, attached to the symbol, the so called resolver function. Ifuncs provide a convenient way to select between different machine-specific implementations of the parts of the code, without the ugliness and unsafety of the alternative approach, which is runtime patching. Ifuncs require support both from the static linker ld(1), and from the runtime linker for the corresponding execution environment. On FreeBSD, with the switch from the ancient GPLv2 licensed BFD-based ld(1) to either in-tree LLD or external modern BFD ld, the use of ifuncs become possible. Runtime linkers for ifunc support exists for the following environments: * i386, amd64, and arm64 kernels * usermode dynamic linker ld-elf.so.1 on i386 and amd64 * static binaries startup code for i386 and amd64 The use of ifuncs were previously applied for optimization of the following areas of the amd64 kernel: * context switching code, instead of huge number of runtime checks (PTI vs non-PTI, PCID or not, is INVPCID instruction supported for PCID) now uses set of mode-specific routines, see pmap_activate_sw(). Besides removing checks at runtime, it also makes the code much more cleanly structured and readable. * TLB and cache flush implementation. * memcpy/memmove, copyin/copyout variants selection for ERMS and SMAP. * FPU state save and restore, depending on the support for AVX or not, this is also used on i386. For amd64 userspace, we currently use ifunc for optimization of the architecture dependent TLS base set and get functions. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Intel Work on Core Enabling and Security Contact: Ben Widawsky A new team has been formed within Intel to help with side channel security mitigations as well as core enabling. They are evaluating work from all areas except networking. The team is currently focusing on two areas: 1. Power Management improvements 2. NVDIMM namespace support The ultimate goal of the power management work is to get runtime power management to hit "opportunistic idle". What this means is when devices are idle, the OS will power them down, and when everything goes idle certain SoCs will allow you to hit very low power states across the platform. FreeBSD currently doesn't have any notion of runtime power management, and many devices don't properly implement suspend and resume. In addition, some preliminary work is in process as it was thought to help when eventually enabling opportunistic idle. That preliminary work has been happening and is now up for review: * https://reviews.FreeBSD.org/D17675 * https://reviews.FreeBSD.org/D17676 NVDIMM namespace support has also been put up for review. ACPI spec defines namespaces as a way of partitioning the device into separate labels. The current work will integrate with geom(4). How these are used is application dependent. This work is up for review as well: https://reviews.FreeBSD.org/D17619 The team has additionally taken on smaller tasks like porting turbostat(8), working on git svn init scripts, some small modifications to acpi tooling, and an effort to create a port PMDK. __________________________________________________________________ Large scale package building Contact: Mateusz Guzik Building packages on a 128-thread machine with poudriere exhibits some bottlenecks. See the October FreeBSD Foundation Newsletter for a short write-up: https://www.freebsdfoundation.org/news-and-events/newsletter/freebsd-fo undation-update-october-2018/ One encountered problem stems from process handling. The standard process life cycle on UNIX-like systems looks like this: * a process is created with fork(2) * it can do regular work or execve(2) a new binary * it exits, becoming a zombie * the parent collects the exit code and removes the zombie There are other variations (e.g. vfork(2) can be used instead of fork). When you type 'ls' into your shell, it will typically vfork a new process which will then execve /bin/ls. All this is guarded with several global kernel locks, but the granularity can be significantly improved. A different problem stems from pipes. Pipes are used all the time, e.g. "du -s | sort -n" creates a pipe whose one endpoint is standard output for du and another is standard input for sort. By default the pipe can hold up to 16KB before it gets filled up. The kernel dedicates part of its virtual address space to hold pipe buffers and allocates/deallocates physical pages as pipes get created/destroyed. This induces TLB invalidation requests to other CPUs, which causes an unnecessary slowdown. An easy way out is to cache a certain number of buffers. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ LLVM 7.0 - Sanitizers support improvements / Static code analysis Links Release notes URL: http://releases.llvm.org/7.0.0/docs/ReleaseNotes.html Contact: David Carlier In order to increase the FreeBSD tooling to uncover code bugs in the userland, further compiler-rt components support and static code analysis improvements had been added since the last 6.0 version. Starting with the sanitizers, Memory Sanitizer (for amd64) mainly to detect unitialized pointers. There is also a simple W^X paging requests detection available from most of sanitizers. Also libFuzzer support finally had been possible. It allows code to be tested with random values from corpus inputs. Mutation and combination algorithms of those random inputs can be overwritten. Can also be used in addition to ubsan, asan, msan and so on. At last, the X-Ray instrumentation feature is also supported. It is mainly about performance profiling purposes for a reasonable performance runtime cost. In the static code analysis department, reliable strlcpy (unfortunately strlcat did not get merged in due time for the release) wrong usage cases are now covered and W^X code detection tooling had been added. At the moment, this 7.0 version is imported by Dimitry Andric, within its own git branch available only for FreeBSD after 12 releases. __________________________________________________________________ Performance improvements Contact: Matthew Macy FreeBSD 12 saw the introduction of a number of performance improvements: * The introduction of the new synchronization primitive epoch(9) to replace the use of reader locks for providing existence guarantees for data structures. * epoch(9) was used to provide an 85+% reduction in the overhead of pcb lookup in high core count systems. * epoch(9) was used to provide an 85+% reduction in UDP send overhead on high core count systems. See the link for a bit more detail: http://scalebsd.org/blog/2018/06/16/UDP-and-epoch-for-liveness-guar antees * System call overhead is now half that of FreeBSD 11. * UNIX sockets now scale near linearly (previously maxed out at 3-4 threads). * The NUMA work has lead to a 20x-80x improvement in the scalability of page fault handling. __________________________________________________________________ Save/Restore/Migration support in bhyve Links Github repository for the save/restore and migration features URL: https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_migrati= on Github wiki - How to Save and Restore a bhyve guest URL: https://github.com/FreeBSD-UPB/freebsd/wiki/Save-and-Restore-a-vir= tual-machine-using-bhyve Github wiki - How to Migrate a bhyve guest URL: https://github.com/FreeBSD-UPB/freebsd/wiki/Virtual-Machine-Migrat= ion-using-bhyve Github wiki - Suspend/resume test matrix URL: https://github.com/FreeBSD-UPB/freebsd/wiki/Suspend-Resume-test-ma= trix Contact: Elena Mihailescu Contact: Darius Mihai Contact: Sergiu Weisz Contact: Mihai Carabas The Save/Restore feature is a facility to suspend and resume guest virtual images that has been added to the FreeBSD/amd64's hypervisor, bhyve. The bhyvectl tool is used to save the guest virtual machine into three files: * a file for the guest memory * a file for state of each device / CPU state * a file that has metadata that is used in the restore process To suspend a bhyve guest, the bhyvectl tool must be run with the --suspend option followed by the guest name. To restore a bhyve guest from a checkpoint, one simply has to add the -r option followed by the main state file (the same file that was given to the --suspend option for bhyvectl) when starting the VM. The Migration feature uses the Save/Restore implementation to migrate a bhyve guest from one FreeBSD host to another FreeBSD host. To migrate a bhyve guest, one needs to start an empty guest on the destination host from a shared guest image using the bhyve tool with the -R option followed by the source host IP and the port to listen to migration request. On the source host, the migration is started by executing the bhyvectl command with the --migrate option, followed by the destination host IP and the port to send to the messages. New features added: * Create the socket connection between source and destination hosts * Migrate the guest state via sockets * Separate the suspend/resume/migration code from the bhyverun.c and bhyvectl.c and added two new files for them: migration.c and migration.h * Added save/restore state for xhci * Added save/restore state for fbuf * Fix vhpet restore state issues (timers related) * Add partially support for suspending and resuming a Linux guest Future tasks: * Check if live migration can be implemented using the FreeBSD's Copy-on-Write mechanism * Add live migration support by using EPT (Intel) * Add live migration support by using NPT (AMD) * Add suspend/resume support for nvme * Add suspend/resume support for virtio-console * Add suspend/resume support for virtio-scsi * Fix restore timers issues * Fix suspending bhyve - threads issues * Fix suspending bhyve - mutexes issues * Add suspend/resume support for Windows guests This project was sponsored by Matthew Grooms, and iXsystems. __________________________________________________________________ SMAP Contact: Konstantin Belousov Support for SMAP (Supervisor-Mode Access Prevention), has been added to the amd64 kernel. The SMAP feature makes any access from the supervisor mode to the pages accessible to user mode cause a fault, unless the %eflags.AC bit is set at the time of the access. The SMAP implementation uses the ifunc framework to avoid checking for the SMAP capability of hardware on each call to copyout(9) and other functions. In the amd64 architecture, FreeBSD has a common address space between the kernel space and user space. Enabling SMAP virtually splits the shared address space into two disjoint address spaces, which have different access criteria. This splitting of the address space provides a relatively low-overhead way of catching direct accesses from kernel to usermode, when not using the copyout(9) family of functions. The copyout(9) family of functions are permitted direct access to user space. Any direct access from kernel mode to user address space that isn't performed through the copyout(9) family of functions indicates a potential programming error. It is interesting that very few bugs were found in the FreeBSD kernel after the SMAP feature was enabled. One issue that was identified existed in the pci(9) user driver. Enabling the SMAP feature identified at least two ports, VBox and acpi_call, which appeared to access userspace in an unsafe manner. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ String functions on the amd64 architecture Contact: Mateusz Guzik Functions like memset, memmove and memcpy are very frequently used by virtually all programs. They can be optimized in various ways, but FreeBSD uses very rudimentary implementations using rep movsq/stosq. rep prefix has high startup latency which is overly expensive when dealing with small sizes. Short term goal of this project is to implement faster variants for the kernel and import them into libc. The main speed up comes from not using rep for small sizes (< 256) and from aligning target buffers to 16 bytes when rep is used. On top of that runtime detection of the Enhanced REP MOVSB/STOSB extention can be used to only use rep movsb/stosb. Mid term goal extends userspace. SIMD extensions can be used to make these functions faster. They can't easily be used in the kernel: SIMD registers are not saved on transitions user<->kernel for performance reasons. Thus any use would have to take care of saving these registers, which can consume any advantage from using them in the first place. This is not a concern for userspace code. There is a BSD-licensed implementation in bionic: https://android.googlesource.com/platform/bionic/+/master/libc/arch-x86 _64/string/ which with some modifications can be used in libc later on. See the Intel Optimization Manual for reference: https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64 -ia-32-architectures-optimization-manual.pdf This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Usermode mapping of PCI BARs Contact: Konstantin Belousov Modern PCI(e) devices typically define memory-mapped BARs (Base Address Registers) to make a memory region available to the device. Each BAR has a separate page-aligned boundary and memory region associated with it. This is enforced by the need of hypervisors to provide the pass-through using VT-d, which operates with memory and has the granularity of one page for access control. As is, it also means that the BARs have a suitable configuration for providing access to usermode, controlling access by the normal page tables. Linux already gives a way for userspace mapping of BARs using sysfs. Of course, if a userspace program has enough privileges, it can read a BAR, determine the physical address of the mapping as seen by CPU, and use mem(4) (aka /dev/mem) to mmap() that region of memory. This is very cumbersome, and leaves many unresolved issues. For example, a BAR might be not activated, which would require involvement of the IOMMU on some architectures. Also this rude approach makes it very hard to create mappings with the correct caching attributes. The FreeBSD pci(4) driver was enhanced to support such mappings, and pciconf(8) utility was extended to use the new support. See pci(4) for PCIOCBARMMAP ioctl(2) request description for details, and pciconf(8) for the -D switch. TODO: automatically activate the BAR on mapping, this is not done yet. There is a problem with avoiding the resource conflicts on possible future attachmens of the kernel driver. This project was sponsored by The FreeBSD Foundation, and Mellanox Technologies. __________________________________________________________________ Architectures Updating platform-specific features and bringing in support for new hardware platforms. Allwinner SoC Support Contact: Emmanuel Vadot * SPI driver added for A64 SoC * Thermal driver added/fixed for A64/H3/H5 SoCs * Lot of bugs where fixed in the mmc driver, stability should be better * New driver for AXXP803 which is the power chip companion of the A64 SoC * Add overlays to use another timer controller as the default one in A64 if faulty These overlay is enabled in the PINE64/LTS images by default __________________________________________________________________ Armada 38x FreeBSD support Links PRODUCT BRIEF URL: https://www.marvell.com/documents/egrkpyqzpoebxblyeept/ Contact: Marcin Wojtas Contact: Patryk Duda Contact: Rafal/ Kozik The Marvell Armada 38x is a very poplular ARMv7-based dual core SoC. Thanks to the multiple low and high speed interfaces the platform is used in a wide range of products, such as Network-Attached Storage (NAS), Wi-Fi Access Point (WAP) and others. Since last report, remaining Armada 38x support was integrated to HEAD, which can now compile with the armv7 GENERIC config and use unmodified sys/gnu/dts device trees. The details are as follows: * GENERIC config * Introduce a vast rework of the sys/arm/mv directory for arm and armv7 platforms. * Enable PLATFORM support for Marvell ARMv7 SoCs, which can now can boot with GENERIC kernel. * Base on dynamic detection of SoC type and device tree instead of using ifdefs and enable more flexible environment for maintaining Marvell platforms. * sys/gnu/dts device trees * Improve platform code and the drivers (e.g. CESA, PCIE, GPIO) to properly work with original Linux device trees. * GPIO * Add multiple fixes and improvements to the sys/arm/mv/gpio.c * Rework driver to properly integrate with HEAD GPIO frameworks (main and gpioled) * Enable support for both old and Linux GPIO device tree bindings, so that multiple controllers can be used. This project was sponsored by Stormshield, and Semihalf. __________________________________________________________________ ARMv6 and ARMv7 image now use EFI loader Contact: Emmanuel Vadot Instead of using the ubldr version of the loader which uses the U-Boot API, all images now use loader.efi as their primary FreeBSD loader. This allow us to have a common boot path for all arm and arm64 images. __________________________________________________________________ DTS Update Contact: Emmanuel Vadot DTS files (Device Tree Sources) were updated to be on par with Linux 4.18 for the 12.0 release. The DTS are now compiled for some arm64 boards as the one present in U-Boot are now always up-to-date. __________________________________________________________________ FreeBSD on POWER9 Contact: Matthew Macy Once Justin Hibbits largely stabilized the powerpc64 port on the POWER9 based Talos II I decided to procure one. I've been slowly working towards taking powerpc64 to parity with amd64. I've been working in an out of tree GitHub project - in part to eliminate the need to continue to support the 11 year old in tree gcc4. Progress so far: * Adapted lock_delay to use POWER's SMT scheduling hints rather than using the yield hint from an older ISA * Added ifunc support * Ported the amd64 pmap so FreeBSD can use POWER9's new radix tree page tables. This will give us superpages mostly for free. * Reduced the overhead of copyinout primitives for radix. * Converted the copyinout primitives to ifuncs to switch between the old and the new at initialization time. * Converted pmap to use ifuncs to eliminate the overhead of kobj dispatch. * Hot patch out trap handling paths only needed by the older hashed page table support. Work in Progress: * NMI semantics: NMIs need to be emulated by only soft disabling interrupts, disabling interrupts blocks all interrupts except machine check exceptions and system resets. * Superpage support: Superpages work in the functional simulator running SMT4 but currently crash on real hardware due to incomplete page walk cache / TLB invalidation. * Kernel minidump - with radix MMU enabled minidump support was a fairly straightforward port but time needs to be spent on test / debug. * EARLY_AP_STARTUP - preliminary patches exist, but this is more work on !x86 architectures due to IPI setup not being tied to the CPU code. A list of the other items needed to achieve kernel feature parity with a (wishful) list of milestones can be found at: https://github.com/POWER9BSD/freebsd/projects/1 __________________________________________________________________ FreeBSD on PowerNV (ppc64) Contact: Patryk Duda Contact: Wojciech Macek Contact: Michal Stanek Contact: Nathan Whitehorn Semihalf is happy to announce that FreeBSD is now running on IBM POWER8. This project is a continuation of work done by Nathan Whitehorn who provided basic support for a PowerNV emulator. The IBM POWER8 family of CPUs offers superior performance compared to previous Power series. It provides complete NUMA support with up to 192 cores in a two socket system (up to 8 threads per core). All IO communication is handled by integrated PCIe interface equipped with multiple IOMMU engines. The support for POWER8 system running FreeBSD in Non-Virtualized environment contains: * Generic driver for OPAL hypervisor * kboot loader modifications to allow Little-Endian loader to load a Big-Endian kernel ELF * skiboot update for ELF-parser allowing it to understand FreeBSD kernel file format * Basic support for PowerNV architecture, including modes of operation, MMU, interrupt controller * SMP operation (tested with 128 CPU configuration) * PHB subsystem driver, including IOMMU mapping for external buses * PCIe host controller driver * USB-3.0 XHCI driver * Reworked drivers to be Big-Endian compatible: * Chelsio cxgbe(4) 10/25G network adapter * NVMe SSD drive All work has been merged into HEAD and will be included in FreeBSD 12.0-RELEASE. Sponsors: IBM, FreeBSD Foundation, QCM Technologies, Semihalf, Limelight Networks. The project is kindly initiated and supported by Limelight Networks (Kevin Bowling). __________________________________________________________________ FreeBSD on RISC-V Contact: Ruslan Bukin FreeBSD/RISC-V has been one of the actively supported projects during the past year. On a compiler front we have upstreamed FreeBSD OS-dependent bits for GNU toolchain. It was updated to GCC 8.1 and Binutils 2.30. FreeBSD packages are available. FreeBSD Testsuite and required dependencies were successfully built for RISC-V and we did a test run: 152 tests failed out of 5186, which demonstrates a very good result for initial run and reveals areas to work on. We have added support for compressed ISA extension to KDB debugger and DTrace FBT provider enabling C-compressed kernel and userland by default. The output of disassembling instructions in KDB is looking similar to objdump. QEMU has updated to latest privilege spec allowing us to bring up FreeBSD on it. The emulation is quite fast: it takes one second only to boot FreeBSD to single-user mode in QEMU: https://www.youtube.com/watch?v=3DFnWpRBaWF18 Platform-Level Interrupt Controller (PLIC) driver was added. Interrupt support was converted to INTRNG. PLIC is used in QEMU for virtio network and block devices. With these changes, a full FreeBSD distribution can now be booted in QEMU. Network virtualization support (VIMAGE) was fixed and enabled by default now. In order to support RocketChip and derivatives we had to work on A(accessed), D(dirty) PTE (page table entry) bits management. We have successfully tested this on a lowRISC board and it is booting to multiuser just fine. lowRISC UART driver was added. Superuser-User-Modify (SUM) bit in status register is now used: kernel can access userspace only within certain functions that explicitly handle crossing user/kernel boundary. __________________________________________________________________ PINE64-LTS Image Contact: Emmanuel Vadot We now produce an image for the PINE64-LTS. This image works on the PINE64-LTS and the Sopine with Baseboard. __________________________________________________________________ PocketBeagle Support Links Pocket Beagle URL: https://www.beagleboard.org/pocket Contact: Emmanuel Vadot Contact: Tom Jones The Pocket Beagle is the latest member of the BeagleBoard family. Support for it was added and the Beaglebone image can be used on it directly. __________________________________________________________________ RPI Firmware/DTB/U-Boot Update Contact: Emmanuel Vadot Contact: U-Boot mailing list The RaspberryPi firmware loads the DTB from the FAT partition based on the model. U-Boot now uses this DTB and passes it to the FreeBSD loader/kernel instead of using the DTS embedded in U-Boot. This allow the FreeBSD Kernel to use the RaspberryPi Foundation provided DTB overlays to enable HATs. The overlays can be obtained by installing the rpi-firmware package. A new U-Boot port for the W variant of the RPI0 was committed as u-boot-rpi-0-w. Some experiments started by Edward Tomasz Napierala (trasz@) have shown that we could possibly produce a generic image for all armv6 RPI (RPI-B, RPI0 and RPI0W). __________________________________________________________________ Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. KDE on FreeBSD Links KDE FreeBSD URL: https://FreeBSD.kde.org/ Contact: Adriaan de Groot Contact: Tobias C. Berner KDE FreeBSD is responsible for the ports of the Plasma5 and KDE4 desktops, and all associated applications. Further we also manage the Qt4 and Qt5 ports, as well as CMake. We also care for the FreeBSD builders for KDE's upstream CI on build.kde.org. Since the last status report a lot of things have changed. First and foremost, the Plasma5 Desktop and the Qt5 based KDE Applications have finally made their way into the official ports tree after lingering for multiple years in our development repository. Secondly KDE4 has been marked deprecated and will be removed at the end of the year. With Qt4 following no later than the next year (due to the exponentially increasing burden of maintenance). On a more technical side, bsd.qt.mk has been replaced by qt.mk and qt-dist.mk. The porter's handbook is being updated (with thanks to Tobias Kortkamp). Further we have been keeping CMake and Qt5 and almost every other port under our control up to date. SDDM has been updated to the next-to-latest release with backported security fixes. One big issue we have is www/qt5-webengine, which requires too much time to keep up to date, as the underlying chromium is in need of many patches, which change with every release. Another upcoming issue is the way in which FreeBSD's libinput lags behind. This blocks future updates to KDE Plasma as well as Wayland improvements. Thankfully x11@ is looking at this issue already, so it should be fixed soon -- for the meantime people who want to give the latest KDE Plasma Desktop a try can use the appropriate branch from our GitHub. People who are willing to contribute can find us on #kde-freebsd on freenode, the kde@FreeBSD.org mailing list. Further, we accept pull-requests and contributions on github.com/freebsd/freebsd-ports-kde. __________________________________________________________________ Puppet Links PuppetLab's FreeBSD slack channel URL: https://puppetcommunity.slack.com/messages/C6CK0UGB1/ BSDCan 2018: IT automation with Puppet URL: https://www.bsdcan.org/2018/schedule/events/930.en.html Contact: Puppet Team Since our last status report last year, the puppet@ team regularly updated the Puppet ports to catch on upstream releases. We have also held a Puppet talk at BSDCan. More recently, Puppet 6 was released, and a bunch of new ports appeared in the FreeBSD ports tree: sysutils/puppet6, sysutils/puppetserver6, databases/puppetdb6 being (obviously) the main ones. In this update, the Puppet language has not been heavily modified. As a consequence, upgrading from Puppet 5 to Puppet 6 is an easy task compared to the experience you may have encountered from previous major version bumps. If you are still using Puppet 4, we recommend to schedule an upgrade soon: Puppet 4 is expected to be EOL by the end of 2018. Because distributing Marionette Collective modules via Puppet is more efficient than using packages, the sysutils/mcollective-\*-{agent,client} ports have been deprecated. Marionette Collective itself being phased out by PuppetLabs, the sysutils/mcollective port is expected to be deprecated at some point in the future, but we plan to keep it until an alternative is available. This alternative, called Choria, is in active development by R.I.Pienaar the original author of Marionette Collective. We are actively working with him to support FreeBSD out of the box, and will commit sysutils/choria to the tree as soon as it is considered a drop-in replacement for Marionette Collective. __________________________________________________________________ scarab: CLI tool for Bugzilla-related workflows Links GitHub repo URL: https://github.com/gonzoua/scarab Contact: Oleksandr Tymoshenko scarab is a CLI tool that makes some of Bugzilla functionality available from the command line. Normally users interact with the bugtracker using a web browser but for certain workflows, Web UI may be more of an obstacle than help requiring to perform more steps compared to CLI tool. Bugzilla provides XML-RPC interfaces that can be used for automation/integration and there are several CLI tools like pybugz that can be used with bugs.FreeBSD.org as-is. They are generic one-size-fits-all tools which mean they can do a lot of thing at the cost of more complex CLI. scarab was created to be more specialized and less complex with following principles in mind: * Be an auxiliary tool, not a replacement for the web UI * Move complexity to a configuration file, keep arguments as simple as possible * Optimize for most common/tedious tasks Based on my experience with Bugzilla following tasks were identified as candidates for inclusion in the first release of the tool: * Downloading attachment on host machine and copying it to devbox * Creating a file on the devbox and copying it to a host machine to be attached through Web UI * Creating PRs with common fields' values First two operations were implemented as files, fetch, fetchall, attach commands of the tool. The third operation was implemented by introducing PR templates, set of predefined field/value pairs, that can be combined run-time to provide higher flexibility. More information and usage examples can be found in the config file example __________________________________________________________________ Documentation Noteworthy changes in the documentation tree or new external books/documents. Cleaning up the Wiki Links Wiki Fixit Group Website URL: https://wiki.FreeBSD.org/WikiFixitGroup/ IRC channel =20 URL: irc://freebsd-wiki@irc.freenode.net Contact: The FreeBSD Wiki used to be a scratch pad for the FreeBSD developers to organize projects, store notes and publish articles that were about to be added to the handbook. Recently, however, the FreeBSD wiki started to attract more and more people from the wider FreeBSD community, which resulted in a change of the character of the wiki. As a result we decided to discuss the future of the tools we want to use for documentation in FreeBSD (one of such discussions was held during BSDCam 2018, you may see some notes here). The general conclusion is that wiki is a great tool for what it was meant for: organizing projects and notes in the community of developers. We should not move all our documentation (especially handbooks) to Wiki as the quality and maintainability would suffer. On the other hand, the current workflow of submitting documentation patches, which involves checking out the doc tree and patching XML files is not ideal for many end users. This is why we are trying to approach the problem from various directions: 1. The wiki is being cleaned up of old content. We are trying to define a clear hierarchy of subpages and categories to make navigating the wiki easier. 2. Some articles from the wiki are going to be migrated to either the doc tree or manual pages. __________________________________________________________________ Quarterly Reports Contact: Edward Tomasz Napiera=C5=82a Contact: Mateusz Piotrowski <0mp@FreeBSD.org> The Quarterly Reports have been resurrected after almost a year long hiatus. The old workflow, which consisted of users submitting XML-formatted entries, which would then get hand-assembled into DocBook, was replaced with a new one, using Markdown instead. The XML submission form was replaced with GitHub Pull Requests. This should make submissions and editing much easier and user-friendly. __________________________________________________________________ Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. HardenedBSD 2018Q3 Update Contact: Shawn Webb Our last report was June 2017. A lot has transpired since then. In this status report, we will attempt to briefly cover all the progress we've made, including the few commits that made it upstream to FreeBSD. On 01 Jul 2018, we switched back to OpenSSL as the crypto library provider in base. We did this because we lack the resources and the documentation for properly supporting LibreSSL in base. We still maintain LibreSSL in base; however, OpenSSL is simply the default crypto library (aka, WITHOUT_LIBRESSL is the default). We look forward to building a development community around LibreSSL in HardenedBSD such that we can re-enable LibreSSL by default, providing enhanced security for our users through the rejection of software monocultures. Cross-DSO Control Flow Integrity (Cross-DSO CFI) is an exploit mitigation from llvm that provides forward-edge protections across shared library and application boundaries. With HardenedBSD 12-STABLE, we launched non-Cross-DSO CFI support in base. Meaning, CFI is only applied to applications and not shared libraries. Along with SafeStack, which provides backward-edge protections, Cross-DSO CFI requires both ASLR and W^X for effectiveness as they store crucial metadata needing protection. HardenedBSD expertly, efficiently, and robustly fulfill those requirements through its PaX ASLR and PaX NOEXEC implementations. Over the past two years, we have slowly worked on Cross-DSO CFI support in HardenedBSD. In mid-2018, we made enough progress that we could publish an alpha Call-For-Testing (CFT). We need to integrate the Cross-DSO CFI support with the RTLD such that function pointers resolved through dlopen(3)/`dlsym(3)` work properly with the cfi-icall scheme. We also need to perform experimental package builds, find breakages, and fix those breakages. We hope to officially debut Cross-DSO CFI in the latter half of 2019 with the possibility of pushing back to 2020. HardenedBSD remains the first and only enterprise operating system to use CFI across the base set of applications. On 20 Aug 2018, we launched a new tool called hbsdcontrol(8) to toggle exploit mitigations on a per-application basis. hbsdcontrol(8) uses filesystem extended attributes and is the preferred method for exploit mitigation toggling for those filesystem that support extended attributes (UFS, ZFS). Our original utility, secadm, should be used with filesystems that do not support extended attributes (NFS). In September 2018, the HardenedBSD Foundation Corp became a 501(c)(3) tax-exempt, not-for-profit organization in the USA. This means that donations by US persons are eligible for tax deductions. The creation of the HardenedBSD Foundation will ensure that HardenedBSD remains successful long-term. We look forward to working with the BSD community to provide an open source, clean-room reimplementation of the grsecurity patchset based on publicly-available documentation. We assisted Kyle Evans with the new bectl(8) utility, primarily enhancing jail support and fixing regressions. We are grateful for Kyle Evans' assistance in landing the enhancements upstream in FreeBSD and his overall responsiveness and helpfulness. Relevant commits for the bectl(8) are: * r339047 * r338221 * r337993 * r337947 We taught bhyve(8) how to live in a jailed environment, allowing users to jail the hypervisor. We hardened the virtual address space of bhyve(8) by using guard pages. This work made it upstream to FreeBSD. We are grateful to those in FreeBSD who provided insight to increase the quality and efficiency of our patches. __________________________________________________________________ --huq684BweRXVnRxX Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGmBAEBCgCQFiEEbvjBe1hu6u1NeinjJCKD+Vwk/7oFAlwhcwFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDZF RjhDMTdCNTg2RUVBRUQ0RDdBMjlFMzI0MjI4M0Y5NUMyNEZGQkESHHRyYXN6QGZy ZWVic2Qub3JnAAoJECQig/lcJP+6yekIAIrvD49Qj8sSNEfblE+V3uEOIsgypSPt mjIxBkl93FV99IzTv/YlZ8in174opyNr35lHf7+T1doy2HW0vENYmQ5ONbrFv8kd nBI8XRQsrZHCADu2/fK+WpHAbl7ZYzAWTzOr/Aje7smQqA7WViEQrnTF/8aawblE YITuuiSvGmEBbHSdIR6QLTaPafcLGge0A5XFmFxNW5XPR3o0hRzTDOclPTiJ6CVf rMpSUO/OrvRVdiJ7BVrDl/li3HXZnloQ23wu5qknSLcFJ+1eZ6wPJfgaOOmVBUCd 0XDTMnnSDWg9Gv8kXnqdsio+MtoUd+GtMZowMlVFOsTSgKoh1wdYV2k= =TpVf -----END PGP SIGNATURE----- --huq684BweRXVnRxX-- From owner-freebsd-hackers@freebsd.org Wed Dec 26 14:16:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8057B1352C6A for ; Wed, 26 Dec 2018 14:16:59 +0000 (UTC) (envelope-from alfix86@gmail.com) Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3866F84C38 for ; Wed, 26 Dec 2018 14:16:58 +0000 (UTC) (envelope-from alfix86@gmail.com) Received: by mail-wr1-x432.google.com with SMTP id l9so15742280wrt.13 for ; Wed, 26 Dec 2018 06:16:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pVQL7JaBfxPW5yi5TKcSFUCC1+QZQ0D5MEc2doDN6C4=; b=VSFM9eU0NBnhIXdeNICQD/zkZDBrSqTDwmQjNYS0FZ8fqerVwT9V4YSR/EAH/lsB2M tRpEiQMe9l9jyXBNcoQzH8Yfg3WKReaXBh90u2EPDuic+tXNMkW/ixBolrC7FtESwQqX UQkf6MW6iS7ARPWO0bvW5NCWRkTGqit08GGJ3zf/7ZlJH1BoF5nnemtZOhb2iawbSIRe LgiPlL5zC4UIfJZXpzesU8PpcA1C4abNHcpTftIxMHeGZld6HxxMk8Cdzb0wb/fdS313 s7EsZ8FKObWARLdo5GhI81hCo+1oFwCFuvLf3ioolV0trKE0OVghiFFvNnD4TBfLbBJb NhEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pVQL7JaBfxPW5yi5TKcSFUCC1+QZQ0D5MEc2doDN6C4=; b=lXeyCPBvtMKlxyMo9OUBgQS9f1NvEM19TPUi3OKTvzzAlLgW9hVbG6YmQ8QnhfyJOP nGIPL5G7n6fKPOAj6Hy2HRct1PWNx4nxrBWgQ4KIBlrJILdZtiYXfdlI2hul8pLUNI4M lLXB9dghBP9VTUCKMIe2XJ5gr/P+i7OG/m68V4dANnrIGaSdhKvkqe4+T1spFYg1/STl dh9zqN4iqgFq2bKF1DcBM2LtNeExFc35AGGKQ0HTUAlqNe7sFhKrjh/uCLyyBC65rYVj oMmATRLtw3sn/QBsWyGT6jBsbPiLqO0xSvaLTEj5roRUOF+0CVIo1x8d00wDULgntTPX sPSA== X-Gm-Message-State: AJcUukd9UkTDWvoBDIEeiW93Y0DKX1fljZlgD48oEs5BuMd90ae3mP6W YXCBkajH/CetaHncfAZhgzgRYoU7 X-Google-Smtp-Source: ALg8bN43cGRMh+DCmj1EdMABXRtVWHRh7LRB7UaOjWF/kmI8CAzFU/KQXQCCA17yknns7+X9UYe7dA== X-Received: by 2002:adf:a58a:: with SMTP id g10mr18605427wrc.3.1545833816724; Wed, 26 Dec 2018 06:16:56 -0800 (PST) Received: from alffbsd ([95.238.231.159]) by smtp.gmail.com with ESMTPSA id f14sm27356648wrv.56.2018.12.26.06.16.55 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Dec 2018 06:16:56 -0800 (PST) Date: Wed, 26 Dec 2018 15:16:49 +0100 From: Alfonso Siciliano To: Ed Schouten Cc: FreeBSD Hackers Subject: Re: libsysctl: a C API for sysctl-mib-tree Message-Id: <20181226151649.7bc24a54a53790699feb1b77@gmail.com> In-Reply-To: References: <20181222202228.95f48408c8e2ebe9c40d9d75@gmail.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd13.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 3866F84C38 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=VSFM9eU0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of alfix86@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) smtp.mailfrom=alfix86@gmail.com X-Spamd-Result: default: False [-5.84 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.73)[-0.727,0]; RECEIVED_SPAMHAUS_PBL(0.00)[159.231.238.95.zen.spamhaus.org : 127.0.0.10]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.61)[ip: (-9.72), ipnet: 2a00:1450::/32(-1.73), asn: 15169(-1.50), country: US(-0.08)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 14:16:59 -0000 Hi Ed, > Ed Schouten wrote: > > Alfonso Siciliano : > > I' m currently working on a library called "libsysctl" which will > > provide a C API to wrap kern_sysctl.c undocumented interface, build > > mib-entry, entries-list and mib-tree in userspace and then to do > > the work that /sbin/sysctl currently does. > > Great work! Thanks > This would have been nice to have when I implemented the > Prometheus metrics exporter for sysctl: > > https://svnweb.freebsd.org/base/head/usr.sbin/prometheus_sysctl_exporter/prometheus_sysctl_exporter.c?view=markup I thought libsysctl exactly to help to write programs like "prometheus_sysctl_exporter" Regards, Alfonso --- Alfonso S. Siciliano http://alfix.gitlab.io From owner-freebsd-hackers@freebsd.org Thu Dec 27 11:09:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02F18135965A for ; Thu, 27 Dec 2018 11:09:42 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D95306E3D9; Thu, 27 Dec 2018 11:09:40 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id A14A7BBC7F; Thu, 27 Dec 2018 12:09:38 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id He93D0vHDwFY; Thu, 27 Dec 2018 12:09:37 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 4AC0ABBC7E; Thu, 27 Dec 2018 12:09:37 +0100 (CET) Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Eugene Grosbein , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> From: Willem Jan Withagen Message-ID: Date: Thu, 27 Dec 2018 12:09:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D95306E3D9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-3.92 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[smtp.digiware.nl,www.digiware.nl]; NEURAL_HAM_SHORT(-0.75)[-0.752,0]; IP_SCORE(-0.66)[asn: 28878(-3.31), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 11:09:42 -0000 On 22/12/2018 19:28, Eugene Grosbein wrote: > 23.12.2018 1:22, Craig Leres wrote: > >> On 12/22/18 7:18 AM, Eugene Grosbein wrote: >>> You should not try to make it start before packet filters, that is wrong >> >> How should I handle the case where I start several openvpn tunnels and have references to them in my pf.conf? My solution was to write a rc.d script that gives a configured list of tun devices up to a minute to come up and then do a "service pf reload". > > And this is right thing to do :-) > I mean, if your filtering rules depend on ever-changing list of interfaces, > just reconfigure the filter when the list changes > or better teach the filter to catch up with changes automatically, if possible. Might want to use the ifup/ifdown scripts to add the specifics for the VPN that just came up. Tricky part is how to get things in the tables at the right place. So with IPFW I use specific line numbers reserved to insert certain rules. (using counter rules to split the fw code into blocks) But it sort of feels like going back in the 80's basic programming. --WjW From owner-freebsd-hackers@freebsd.org Thu Dec 27 11:39:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C641135AB77 for ; Thu, 27 Dec 2018 11:39:13 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 58E7F6F55E; Thu, 27 Dec 2018 11:39:02 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wBRBcrRA026832 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 27 Dec 2018 12:38:53 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wjw@digiware.nl Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id wBRBcp1m019906; Thu, 27 Dec 2018 18:38:52 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Willem Jan Withagen , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> From: Eugene Grosbein Message-ID: <5C24B9CB.1070800@grosbein.net> Date: Thu, 27 Dec 2018 18:38:51 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 58E7F6F55E X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-4.11 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.87)[-0.875,0]; IP_SCORE(-1.64)[ip: (-2.93), ipnet: 2a01:4f8::/29(-2.80), asn: 24940(-2.44), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 11:39:13 -0000 On 27.12.2018 18:09, Willem Jan Withagen wrote: > Might want to use the ifup/ifdown scripts to add the specifics for the > VPN that just came up. Tricky part is how to get things in the tables at > the right place. > > So with IPFW I use specific line numbers reserved to insert certain > rules. (using counter rules to split the fw code into blocks) > > But it sort of feels like going back in the 80's basic programming. Current ipfw implementation allows you to use 'tun*' or table containing interface names: ipfw table NAME create type iface ipfw add 2000 allow ip from any to any via 'table(NAME)' ipfw table NAME add tap0 ipfw table NAME add tun0 Note you do not have to change ruleset at all; you add or delete table records only. From owner-freebsd-hackers@freebsd.org Thu Dec 27 12:31:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9DF6135D3C5 for ; Thu, 27 Dec 2018 12:31:08 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43C8F71348; Thu, 27 Dec 2018 12:31:08 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 4ED74708C6; Thu, 27 Dec 2018 13:31:04 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wuFdLnVjNi8k; Thu, 27 Dec 2018 13:31:03 +0100 (CET) Received: from [192.168.11.152] (unknown [192.168.11.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 4A894708C1; Thu, 27 Dec 2018 13:31:03 +0100 (CET) Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Eugene Grosbein , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> <5C24B9CB.1070800@grosbein.net> From: Willem Jan Withagen Message-ID: <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> Date: Thu, 27 Dec 2018 13:31:03 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <5C24B9CB.1070800@grosbein.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: nl X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 12:31:08 -0000 On 27/12/2018 12:38, Eugene Grosbein wrote: > On 27.12.2018 18:09, Willem Jan Withagen wrote: > >> Might want to use the ifup/ifdown scripts to add the specifics for the >> VPN that just came up. Tricky part is how to get things in the tables at >> the right place. >> >> So with IPFW I use specific line numbers reserved to insert certain >> rules. (using counter rules to split the fw code into blocks) >> >> But it sort of feels like going back in the 80's basic programming. > Current ipfw implementation allows you to use 'tun*' or table containing interface names: > > ipfw table NAME create type iface > ipfw add 2000 allow ip from any to any via 'table(NAME)' > > ipfw table NAME add tap0 > ipfw table NAME add tun0 > > Note you do not have to change ruleset at all; you add or delete table records only. > Nice, I was wondering about this, if tables would work for that. That is fine if all your VPNs have the same rules, but if they have different properties and are in and outgoing you will want a bit more control over whats going on. Hence my basic feeling.... :) --WjW From owner-freebsd-hackers@freebsd.org Thu Dec 27 14:13:45 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DFBA113607C8 for ; Thu, 27 Dec 2018 14:13:44 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward101p.mail.yandex.net (forward101p.mail.yandex.net [77.88.28.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3FC8E757E6; Thu, 27 Dec 2018 14:13:43 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mxback7g.mail.yandex.net (mxback7g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:168]) by forward101p.mail.yandex.net (Yandex) with ESMTP id BE0ED3280AAA; Thu, 27 Dec 2018 17:13:35 +0300 (MSK) Received: from smtp3o.mail.yandex.net (smtp3o.mail.yandex.net [2a02:6b8:0:1a2d::27]) by mxback7g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id G587uK7o3W-DZweiTBx; Thu, 27 Dec 2018 17:13:35 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1545920015; bh=HOnpuv699Q+MpZYEkfCgAfxILkvdXcXsT7H7wfGH3cU=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=eg0bbOmDnWq7m5ElNJ8lwGAiIYlEad+QBj6c8+W/BTSIlovdRWuLbrTsBAg0FWTBS Z0sOiem6+kQ1UmR4iHxWFjgwiGuRDp9Bwm47/tXi2gtzSigE7qEm9LtaqGZkA8+AmQ o77Cggs3Zlb9GijV7CAF4nnbAEjtlj21aFrWx6kw= Received: by smtp3o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id AGVgltmy3I-DXkqMd2l; Thu, 27 Dec 2018 17:13:33 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Willem Jan Withagen , Eugene Grosbein , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> <5C24B9CB.1070800@grosbein.net> <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Autocrypt: addr=bu7cher@yandex.ru; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0JUFuZHJleSBWLiBFbHN1a292IDxidTdjaGVyQHlhbmRleC5ydT6JATgEEwECACIFAkwB F1kCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAHF6gQQyKF6qmYIAI6ekfm1VA4T vqankI1ISE6ku4jV7UlpIQlEbE7/8n3Zd6teJ+pGOQhN5qk8QE7utdPdbktAzi+x7LIJVzUw 4TywZLXGrkP7VKYkfg6oyCGyzITghefQeJtr2TN4hYCkzPWpylkue8MtmqfZv/6royqwTbN+ +E09FQNvTgRUYJYTeQ1qOsxNRycwvw3dr2rOfuxShbzaHBB1pBIjGrMg8fC5pd65ACH5zuFV A0CoTNGMDrEZSfBkTW604UUHFFXeCoC3dwDZRKOWJ3GmMXns65Ai5YkA63BSHEE1Qle3VBhd cG1w0CB5FBV3pB27UVnf0jEbysrDqW4qN7XMRFSWNAy5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: Date: Thu, 27 Dec 2018 17:10:46 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="PHsUSZzRk04JJgcjY3MtaUGD8bfh48uwD" X-Rspamd-Queue-Id: 3FC8E757E6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 14:13:45 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --PHsUSZzRk04JJgcjY3MtaUGD8bfh48uwD Content-Type: multipart/mixed; boundary="3H9VgvMxzIjNKLZ4ftyKJf6zBmHksKK3H"; protected-headers="v1" From: "Andrey V. Elsukov" To: Willem Jan Withagen , Eugene Grosbein , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org Message-ID: Subject: Re: rcorder for vpn-like tunnels during early rc.d startup References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> <5C24B9CB.1070800@grosbein.net> <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> In-Reply-To: <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> --3H9VgvMxzIjNKLZ4ftyKJf6zBmHksKK3H Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 27.12.2018 15:31, Willem Jan Withagen wrote: > I was wondering about this, if tables would work for that. >=20 > That is fine if all your VPNs have the same rules, but if they have > different properties and are in and outgoing you will want a bit more > control over whats going on. > Hence my basic feeling.... :) You can use "tablearg" feature and just do "skipto" to rules specific to your special tunnel. Such rules will work only for needed tunnel. --=20 WBR, Andrey V. Elsukov --3H9VgvMxzIjNKLZ4ftyKJf6zBmHksKK3H-- --PHsUSZzRk04JJgcjY3MtaUGD8bfh48uwD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAlwk3WYACgkQAcXqBBDI oXpTGgf/bYm9FT50ln6oEk0fn0KFegq0ZbeNWi6V3XVqr4cUe9/tCTEeWOk31wxM fKu2FnwRcwawHen4Zc7hBoxEFcZm2gdDbPgJxKkkMmtk2YqQJ182oLy8VJpZHt6h zo3lHjkQ3EXAzj3zOI5Gc8RCfyGqOi3axp4GFs49rEH2j93SFQbQV+/gA+DPEZRz 43SGrYNvCozNh7kAkj70E/+GdGnt+6hWih9ahEH2AqTER5783dOg6ZGQPhBttS4d sHZPvVWUZ0Es3HeANSyIWkFCY37EVaDhWTnt6GwUbnYAi83SqrQogBuCrCj1RFUH y1NxTRPlFDjZflJIo9ffM1m98EuKOw== =uK/f -----END PGP SIGNATURE----- --PHsUSZzRk04JJgcjY3MtaUGD8bfh48uwD-- From owner-freebsd-hackers@freebsd.org Thu Dec 27 14:17:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3055E136095B for ; Thu, 27 Dec 2018 14:17:08 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AB41B75AA0; Thu, 27 Dec 2018 14:17:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wBREGxC9027973 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 27 Dec 2018 15:16:59 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wjw@digiware.nl Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id wBREGwOu021242 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 27 Dec 2018 21:16:58 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Willem Jan Withagen , Craig Leres , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> <5C24B9CB.1070800@grosbein.net> <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> From: Eugene Grosbein Message-ID: Date: Thu, 27 Dec 2018 21:16:53 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <003d8528-c72b-5861-8c7f-7032731408d5@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: AB41B75AA0 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 14:17:08 -0000 27.12.2018 19:31, Willem Jan Withagen wrote: >> Current ipfw implementation allows you to use 'tun*' or table containing interface names: >> >> ipfw table NAME create type iface >> ipfw add 2000 allow ip from any to any via 'table(NAME)' >> >> ipfw table NAME add tap0 >> ipfw table NAME add tun0 >> >> Note you do not have to change ruleset at all; you add or delete table records only. >> > Nice, > > I was wondering about this, if tables would work for that. > > That is fine if all your VPNs have the same rules, but if they have different properties and are in and outgoing you will want a bit more control over whats going on. > Hence my basic feeling.... :) You still can create several tables for different properties and process tables differently. From owner-freebsd-hackers@freebsd.org Thu Dec 27 14:20:55 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38D5D1360C4E for ; Thu, 27 Dec 2018 14:20:55 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from xse.com (xse.com [IPv6:2607:f2f8:abb8::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "xse.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B19975DD1 for ; Thu, 27 Dec 2018 14:20:54 +0000 (UTC) (envelope-from leres@freebsd.org) Received-SPF: pass (dot.xse.com: authenticated connection) receiver=dot.xse.com; client-ip=2001:558:6045:10:9084:9e0:4b6d:eb99; helo=ice.alameda.xse.com; envelope-from=leres@freebsd.org; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10; Received: from ice.alameda.xse.com (ice.xse.com [IPv6:2001:558:6045:10:9084:9e0:4b6d:eb99]) (authenticated bits=0) by dot.xse.com (8.15.2/8.15.2) with ESMTPSA id wBREKlsH082877 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 27 Dec 2018 06:20:48 -0800 (PST) (envelope-from leres@freebsd.org) X-Authentication-Warning: dot.xse.com: Host ice.xse.com [IPv6:2001:558:6045:10:9084:9e0:4b6d:eb99] claimed to be ice.alameda.xse.com Subject: Re: rcorder for vpn-like tunnels during early rc.d startup To: Willem Jan Withagen , Eugene Grosbein , Dave Cottlehuber , freebsd-hackers@freebsd.org References: <1545487265.3497867.1616158504.69E513B4@webmail.messagingengine.com> <8a8c6e8e-4781-9e03-36cf-b7974cb719bc@grosbein.net> From: Craig Leres Message-ID: <8aa1f557-aa2b-76ce-1feb-cd7451e6a3a3@freebsd.org> Date: Thu, 27 Dec 2018 06:20:47 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.100.2 at dot.xse.com X-Virus-Status: Clean X-GBUdb-Analysis: Unknown X-MessageSniffer-Rules: 0-0-0-2608-c X-Rspamd-Queue-Id: 7B19975DD1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 14:20:55 -0000 On 12/27/18 3:09 AM, Willem Jan Withagen wrote: > Might want to use the ifup/ifdown scripts to add the specifics for the > VPN that just came up. Tricky part is how to get things in the tables at > the right place. That's a pretty good idea. After I wrote the working "additional rc.d script" solution I learned about ifup/ifdown scripts which seems cleaner but never went back to try that method. > So with IPFW I use specific line numbers reserved to insert certain > rules. (using counter rules to split the fw code into blocks) (I like pf and really don't want to go back to ipfw.) Craig From owner-freebsd-hackers@freebsd.org Fri Dec 28 01:14:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C9131429D72 for ; Fri, 28 Dec 2018 01:14:32 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 348BD6D04A for ; Fri, 28 Dec 2018 01:14:30 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id B52A5AEF1C; Fri, 28 Dec 2018 02:14:28 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZG33nXwRMOuC; Fri, 28 Dec 2018 02:14:27 +0100 (CET) Received: from [192.168.11.152] (unknown [192.168.11.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id B536BAEF12 for ; Fri, 28 Dec 2018 02:14:27 +0100 (CET) To: FreeBSD Hackers From: Willem Jan Withagen Subject: Using kqueue with aio_read/write Message-ID: <8753521a-4555-ec2a-5efc-dee2660b4d9b@digiware.nl> Date: Fri, 28 Dec 2018 02:14:28 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 348BD6D04A X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-3.94 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[cached: smtp.digiware.nl]; NEURAL_HAM_SHORT(-0.76)[-0.760,0]; IP_SCORE(-0.67)[asn: 28878(-3.39), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 01:14:32 -0000 Hi, Im trying to understand why I cannot get so code to work. This is the smallest extract I can make to show my problem. I would expect the kevent() call to return every timeo tick. Even if I tell it NOT to time-out I get these spurts of errors Since there is nothing to trigger the AIO-event, I would expect kqueue to hold indefinitly. But it does not generate anything other than errors And instead it repeatedly complains that there is a permission error:   get_events_kevent: EV_Error(1) kevent(): Operation not permitted But I'm not getting where that would the case... Surely a pilot error, but I do overlook it al the time. So suggestions are welcome. Thanx, --WjW #include #include #include #include #include #include #include #include #include #define BUFFER_SIZE     512 #define MAX_EVENTS 32 #define FILENAME "/tmp/aio_test" char filename[256]; int fd; int done = 0; void get_events_kevent(int fd, int kq) {     printf("get_events function fd = %d, kq = %d\n", fd, kq);     int i = 0, errcnt = 0, err, ret, reterr, rev;     int search = 1;     int timeout_ms = 10;     struct timespec timeo = {         timeout_ms / 1000,         (timeout_ms % 1000) * 1000 * 1000     };     struct kevent filter[16];     struct kevent changed[16];     EV_SET(&filter[0], fd, EVFILT_AIO,             EV_ADD,             0, 0, 0 );     while (!done) {         printf("+");         rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);         if (rev < 0) {             perror("kevent error");         } else if (rev == 0) {             printf("T");         } else {             printf("rev(%d)\n", rev);             if (changed[0].flags == EV_ERROR) {                 errno = changed[0].data;                 printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,                     strerror(errno));                 memset(&changed[0], 0, sizeof(struct kevent));             } else {                 err = aio_error((struct aiocb*)changed[0].udata);                 ret = aio_return((struct aiocb*)changed[0].udata);                 if (ret < 0 )                     reterr = errno;                 if (err != 0) {                     printf( "%s: slot: %d, Error(%d) at aio_error(): %s\n", __func__, i, err, strerror (err));                     errcnt++;                     if (errcnt > 50) {                         exit(3);                     }                 }             }         }     } } int main() {     (void) strcpy(filename, FILENAME);     unlink(filename);     fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);     if (fd == -1) {         printf( "%s: Error at open(): %s\n", __func__, strerror(errno));         exit(1);     }     int kq = kqueue();     if (fd == -1) {         printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno));         exit(1);     }     get_events_kevent( fd, kq);     close(kq);     close(fd);     return 0; } From owner-freebsd-hackers@freebsd.org Fri Dec 28 03:21:43 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68E62142C16F for ; Fri, 28 Dec 2018 03:21:43 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59A8570ADC for ; Fri, 28 Dec 2018 03:21:42 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f194.google.com with SMTP id t9-v6so17698681ljh.6 for ; Thu, 27 Dec 2018 19:21:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+YYO6KDdIABnfPlBZ0rlJTb6afT6wOgVwm+HZiRL6aI=; b=ju7mMD34jhzVwW1tGS+/MdXPcaGXxBNXOhgQryOzD3JKS6b+Xq3HDUIOfJLx6/IKL5 evn8q855352SvNoTD9tKnezGorICmZfV5TGimBXs+Me4LIYpD9BE8Cvlgwl35R0B7yH5 LKBisbL3ESo1CW7j3G76FQ2CHNS1YkcGS1908HT2SsudwZ9GqkfDBcDFRt5mOYj3VPgx DOqwzZvBPTmvzg41sB+s8kTTbceNzXcM9Te2jgk9NhlUx8MAuki0AKSNDiy6Tfa3elgm CnsFUVyjd+mpUhwPRtqJ/Bu1rNMDdDl7ryOj2TZzrej3+tg/3+WzdcQCV6xfyQAQ2TmI pVZA== X-Gm-Message-State: AJcUuken9ZuRoDh6v5N+mVvgHPKIV5CtbplMIfvP+B0fRC32Pi/IVPV0 G6ktgIEjXLgMVTXsjo+hUNtjqZTqArlGp9V4AY5PsQ== X-Google-Smtp-Source: ALg8bN6sEKeZHM8RuWRRczwjBtVcvXJrrRsXBluCkPDnOC5RYxi5UV3krWPJQtTvpevEfuGpyGBgnHPQpKapL0MQSLw= X-Received: by 2002:a2e:9356:: with SMTP id m22-v6mr14890911ljh.135.1545961646181; Thu, 27 Dec 2018 17:47:26 -0800 (PST) MIME-Version: 1.0 References: <8753521a-4555-ec2a-5efc-dee2660b4d9b@digiware.nl> In-Reply-To: <8753521a-4555-ec2a-5efc-dee2660b4d9b@digiware.nl> From: Alan Somers Date: Thu, 27 Dec 2018 18:47:14 -0700 Message-ID: Subject: Re: Using kqueue with aio_read/write To: Willem Jan Withagen Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 59A8570ADC X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.208.194 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[194.208.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.85)[-0.852,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; IP_SCORE(-1.06)[ipnet: 209.85.128.0/17(-3.70), asn: 15169(-1.51), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 03:21:43 -0000 On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen wrote: > > Hi, > > Im trying to understand why I cannot get so code to work. > This is the smallest extract I can make to show my problem. > > I would expect the kevent() call to return every timeo tick. > Even if I tell it NOT to time-out I get these spurts of errors > > Since there is nothing to trigger the AIO-event, I would expect kqueue > to hold indefinitly. > > But it does not generate anything other than errors > And instead it repeatedly complains that there is a permission error: > get_events_kevent: EV_Error(1) kevent(): Operation not permitted > > But I'm not getting where that would the case... > > Surely a pilot error, but I do overlook it al the time. > So suggestions are welcome. > > Thanx, > --WjW > > #include > #include > #include > #include > #include > #include > #include > #include > #include > > #define BUFFER_SIZE 512 > #define MAX_EVENTS 32 > > #define FILENAME "/tmp/aio_test" > char filename[256]; > int fd; > int done = 0; > > void get_events_kevent(int fd, int kq) > { > printf("get_events function fd = %d, kq = %d\n", fd, kq); > int i = 0, errcnt = 0, err, ret, reterr, rev; > int search = 1; > > int timeout_ms = 10; > struct timespec timeo = { > timeout_ms / 1000, > (timeout_ms % 1000) * 1000 * 1000 > }; > struct kevent filter[16]; > struct kevent changed[16]; > > EV_SET(&filter[0], fd, EVFILT_AIO, > EV_ADD, > 0, 0, 0 ); This is the first problem. There's no need to explicitly set EVFILT_AIO on the kqueue. It gets set by the aio_read(2) or similar syscall. And this invocation wouldn't be correct anyway, because for AIO the ident field refers to the address of the struct aiocb, not the file descriptor. If the only events you care about are AIO, then you can pass NULL as the filter argument to kevent. I suspect this is the cause of your problem. The kernel probably thinks you're trying to register for an aiocb that's outside of your address space or something like that. > while (!done) { > printf("+"); > rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo); > if (rev < 0) { > perror("kevent error"); > } else if (rev == 0) { > printf("T"); > } else { > printf("rev(%d)\n", rev); > if (changed[0].flags == EV_ERROR) { > errno = changed[0].data; > printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno, > strerror(errno)); > memset(&changed[0], 0, sizeof(struct kevent)); > } else { > err = aio_error((struct aiocb*)changed[0].udata); No need to call aio_error(2) after kevent(2) returns. You can go straight to aio_return. aio_error shouldn't hurt, but it isn't necessary. > ret = aio_return((struct aiocb*)changed[0].udata); > if (ret < 0 ) > reterr = errno; > if (err != 0) { > printf( "%s: slot: %d, Error(%d) at aio_error(): > %s\n", __func__, i, err, strerror (err)); > errcnt++; > if (errcnt > 50) { > exit(3); > } > } > } > } > } > } > > int main() > { > (void) strcpy(filename, FILENAME); > unlink(filename); > fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); > if (fd == -1) { > printf( "%s: Error at open(): %s\n", __func__, strerror(errno)); > exit(1); > } > > int kq = kqueue(); > if (fd == -1) { > printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno)); > exit(1); > } > > get_events_kevent( fd, kq); > > close(kq); > close(fd); > return 0; > } > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Fri Dec 28 08:45:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 481BC14312B0 for ; Fri, 28 Dec 2018 08:45:25 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09631806F1; Fri, 28 Dec 2018 08:45:23 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 3C11AAB58C; Fri, 28 Dec 2018 09:45:20 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H99XVaxWhqD7; Fri, 28 Dec 2018 09:45:19 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 269DEAB586; Fri, 28 Dec 2018 09:45:19 +0100 (CET) Subject: Re: Using kqueue with aio_read/write To: Alan Somers Cc: FreeBSD Hackers References: <8753521a-4555-ec2a-5efc-dee2660b4d9b@digiware.nl> From: Willem Jan Withagen Message-ID: <70a07f49-cddd-5d13-ec94-88e06b4c3fc0@digiware.nl> Date: Fri, 28 Dec 2018 09:45:18 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: nl Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 08:45:25 -0000 On 28-12-2018 02:47, Alan Somers wrote: > On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen wrote: >> >> Hi, >> >> Im trying to understand why I cannot get so code to work. >> This is the smallest extract I can make to show my problem. >> >> I would expect the kevent() call to return every timeo tick. >> Even if I tell it NOT to time-out I get these spurts of errors >> >> Since there is nothing to trigger the AIO-event, I would expect kqueue >> to hold indefinitly. >> >> But it does not generate anything other than errors >> And instead it repeatedly complains that there is a permission error: >> get_events_kevent: EV_Error(1) kevent(): Operation not permitted >> >> But I'm not getting where that would the case... >> >> Surely a pilot error, but I do overlook it al the time. >> So suggestions are welcome. >> >> Thanx, >> --WjW >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUFFER_SIZE 512 >> #define MAX_EVENTS 32 >> >> #define FILENAME "/tmp/aio_test" >> char filename[256]; >> int fd; >> int done = 0; >> >> void get_events_kevent(int fd, int kq) >> { >> printf("get_events function fd = %d, kq = %d\n", fd, kq); >> int i = 0, errcnt = 0, err, ret, reterr, rev; >> int search = 1; >> >> int timeout_ms = 10; >> struct timespec timeo = { >> timeout_ms / 1000, >> (timeout_ms % 1000) * 1000 * 1000 >> }; >> struct kevent filter[16]; >> struct kevent changed[16]; >> >> EV_SET(&filter[0], fd, EVFILT_AIO, >> EV_ADD, >> 0, 0, 0 ); > > > This is the first problem. There's no need to explicitly set > EVFILT_AIO on the kqueue. It gets set by the aio_read(2) or similar > syscall. And this invocation wouldn't be correct anyway, because for > AIO the ident field refers to the address of the struct aiocb, not the > file descriptor. If the only events you care about are AIO, then you > can pass NULL as the filter argument to kevent. I suspect this is the > cause of your problem. The kernel probably thinks you're trying to > register for an aiocb that's outside of your address space or > something like that. Hi Alan, That at least helps against getting EPERM. And I get a nice stream of timeouts hitting kevent. It sort of makes sense when you write it like this, but then this is not clear at all from the man pages (for me that is) But it seems sort of weird to "not filter" to get AIO events. >> while (!done) { >> printf("+"); >> rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo); >> if (rev < 0) { >> perror("kevent error"); >> } else if (rev == 0) { >> printf("T"); >> } else { >> printf("rev(%d)\n", rev); >> if (changed[0].flags == EV_ERROR) { >> errno = changed[0].data; >> printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno, >> strerror(errno)); >> memset(&changed[0], 0, sizeof(struct kevent)); >> } else { >> err = aio_error((struct aiocb*)changed[0].udata); > > > No need to call aio_error(2) after kevent(2) returns. You can go > straight to aio_return. aio_error shouldn't hurt, but it isn't > necessary. This is a sort of leftover from earlier trials where I was using a "home made" Poll iterating over the aiocb blocks. Thanx for the help, now I can go on trying getting this into Ceph. Perhaps I'll try an clean this up, and post a working example somewhere online. --WjW > > >> ret = aio_return((struct aiocb*)changed[0].udata); >> if (ret < 0 ) >> reterr = errno; >> if (err != 0) { >> printf( "%s: slot: %d, Error(%d) at aio_error(): >> %s\n", __func__, i, err, strerror (err)); >> errcnt++; >> if (errcnt > 50) { >> exit(3); >> } >> } >> } >> } >> } >> } >> >> int main() >> { >> (void) strcpy(filename, FILENAME); >> unlink(filename); >> fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); >> if (fd == -1) { >> printf( "%s: Error at open(): %s\n", __func__, strerror(errno)); >> exit(1); >> } >> >> int kq = kqueue(); >> if (fd == -1) { >> printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno)); >> exit(1); >> } >> >> get_events_kevent( fd, kq); >> >> close(kq); >> close(fd); >> return 0; >> } >> >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Fri Dec 28 12:31:55 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56EE51437738; Fri, 28 Dec 2018 12:31:55 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 39F5688F8A; Fri, 28 Dec 2018 12:31:53 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.43.105] (x59cc8986.dyn.telefonica.de [89.204.137.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id EFD1A61F9D; Fri, 28 Dec 2018 11:53:06 +0000 (UTC) Subject: Re: core dumps running in bhyve To: Chuck Tuffli , freebsd-emulation@freebsd.org, FreeBSD Hackers References: Cc: freebsd-virtualization@freebsd.org From: Fabian Freyer Message-ID: <79b6eebd-2320-1888-1162-d3ca5492670c@physik.tu-berlin.de> Date: Fri, 28 Dec 2018 11:53:07 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 39F5688F8A X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [5.23 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; HFILTER_HOSTNAME_4(2.50)[mail.physik-pool.tu-berlin.de]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-berlin.de]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.76)[0.761,0]; NEURAL_SPAM_SHORT(0.70)[0.704,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[25.50.149.130.list.dnswl.org : 127.0.11.2]; MX_GOOD(-0.01)[a1861.mx.srv.dfn.de,b1861.mx.srv.dfn.de,c1861.mx.srv.dfn.de]; NEURAL_SPAM_LONG(0.59)[0.586,0]; IP_SCORE(-0.02)[asn: 680(-0.06), country: DE(-0.01)]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:130.149.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 12:31:55 -0000 CCing freebsd-virtualization@, because they might know more about this. Am 25.12.2018 um 02:24 schrieb Chuck Tuffli: > Using the latest bhyve, I'm seeing core dumps in the guest when running: > nvmecontrol identify nvme0 > against the emulated NVMe drive. The location of the core dump changes > from run to run, but I suspect the root cause is a memory corruption > caused by the transfer of the Identify data (4KB) back to the guest. > This transfer of data is actually a memcpy to an address returned from > vm_map_gpa() based on the physical address provided by the guest. > > Based on the signature of one of the core dumps, I modified > nvmecontrol to always pass a 4KB aligned buffer to the driver instead > of the (typically) unaligned address of the structure on the stack. > With this change, nvmecontrol in the guest no longer core dumps. What > I don't understand is why this changes the behavior. Do the addresses > passed to vm_map_gpa() need to be page aligned? AFAIK vm_map_gpa maps a page, so yes, it needs to be 4k-aligned. > Or did moving the > memory location from the stack to the heap merely mitigate what is > corrupted? > > Thoughts? > > --chuck > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Fri Dec 28 19:31:20 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE3AD1420066 for ; Fri, 28 Dec 2018 19:31:20 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B50A97118 for ; Fri, 28 Dec 2018 19:31:18 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-ed1-x52e.google.com with SMTP id o10so18219983edt.13 for ; Fri, 28 Dec 2018 11:31:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=XMihB5MM1r8qOIaDH/V+ctZ6FSRPWOdz6E+wKH2gUJQ=; b=0VMx6kyV+ev8JBsMFGobYiS126B0Gfl1DJAguGol5wtCmeCtaUnx8UIRNXHuBLEC3y d9nK+RdHLwYt+HNtyLmkdreEreeLyTcFZWgEmEgxSDP6CBrASnBEvumfqk/yhd0qdGKH fs1zZrrltf4tlr0VuZQe4o1lka++WHeKHTITYj3L/Ztwg1Y6+S2rDUl0vyGhP1z+xhOS J9P/uy3PcAzoba90CtcUXSOG71DuU3yg7sWdVNekUF2CHIgMgC9gONhdi+CKa9q8tyjc MuvRkxZktGbzbWuCd6U21N7IjaKVsZKCz9b59Gqc+Bupbdn5EE6HRyd/z0ezlw2PIdkH n4kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=XMihB5MM1r8qOIaDH/V+ctZ6FSRPWOdz6E+wKH2gUJQ=; b=fl8j9ls3Q5/QC9/vmKySAWEOH1HGXV5UaezgYN5zocXv3DD3ozIn9qYbVKSqN6aZeD rORRUtrlA2xi1HE50FDIgfHvdMRHK6UY0NQihcg3tiWy5V4JkhBO99kcVRGr7amyhiPT rWLyZLM4KYnyosJRdhCbVRDbxJtpDqgjy1hFOAf08xlBNL/hxrgoJhB7HOzHrip1XuvZ xyqXBwqHdgKfVdIRoDf7XCpwzYaBKI7KYoox7AiCpLeXngRDOhiBhd7k0HCTjKVBxRGB Lrk4YO1v45R0YYi2KkASDB0LEqpXB+yp4V8rmBTWaKZdB2bSXxjD/JimfgtrpUef3Dea 9PSw== X-Gm-Message-State: AA+aEWYkm/HAkFzBBQH4IGMRSO8MVrfZf1H80vuh7e88H6kxt5o9+Y22 RIDsjIyQpkSrf99ytGEebyYFH+K5+CiBb1Uo64XQqDG6ygk= X-Google-Smtp-Source: AFSGD/UI//P/UfgXUM8DhprXmTIK+lS11uK9i92hBu9/AsHwqgtnWTzXMFvUHUdEQSSWf+Lu6emG0HPZ6weTnBu13cc= X-Received: by 2002:a50:fb03:: with SMTP id d3mr22654413edq.183.1546025477553; Fri, 28 Dec 2018 11:31:17 -0800 (PST) MIME-Version: 1.0 From: Mark Saad Date: Fri, 28 Dec 2018 14:31:10 -0500 Message-ID: Subject: libxo question To: FreeBSD Hackers X-Rspamd-Queue-Id: 1B50A97118 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount-org.20150623.gappssmtp.com header.s=20150623 header.b=0VMx6kyV X-Spamd-Result: default: False [-5.53 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[longcount-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[longcount.org]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.41)[ip: (-8.68), ipnet: 2a00:1450::/32(-1.76), asn: 15169(-1.52), country: US(-0.08)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[longcount-org.20150623.gappssmtp.com:+]; MIME_BASE64_TEXT(0.10)[]; RCVD_IN_DNSWL_NONE(0.00)[e.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,aspmx4.googlemail.com,aspmx3.googlemail.com,alt2.aspmx.l.google.com,aspmx2.googlemail.com]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.91)[-0.910,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 19:31:21 -0000 All I am playing around with procstat and libxo on 12-STABLE from yesterday . I wanted to get a list of thread_id's for some processes. I wrote a quick python script to grab the data but xml output is not well formed. Here is my sample script , which should work on python 2.7 ----8<----------------------- 1 import subprocess as sp 2 import os,sys 3 import pprint as pp 4 import xml.etree.cElementTree as ET 5 6 7 FNULL = open(os.devnull, 'w') 8 cmd = "procstat --libxo xml -ta" 9 p = sp.Popen(cmd, shell=True, stdout=sp.PIPE,stderr=FNULL, executable="/bin/sh") 10 text , err = p.communicate() 11 12 root = ET.fromstring(text) 13 14 pp.pprint(root) 15 16 sys.exit(1) ------------>8----------------------- I am constantly getting this odd issue about the xml being not well formatted Traceback (most recent call last): File "/tmp/test.py", line 12, in root = ET.fromstring(text) File "", line 124, in XML cElementTree.ParseError: not well-formed (invalid token): line 1, column 32 Attached is a copy of the xml. Any guidance would be helpful. -- mark saad | nonesuch@longcount.org From owner-freebsd-hackers@freebsd.org Fri Dec 28 20:19:14 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01EBA1421184 for ; Fri, 28 Dec 2018 20:19:14 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D6F069AE4; Fri, 28 Dec 2018 20:19:13 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4C8191D86D; Fri, 28 Dec 2018 20:19:13 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k1xb3xxxgjknkb.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:fdd0:982a:91a4:374b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id B83353064B; Fri, 28 Dec 2018 21:19:10 +0100 (CET) From: "Kristof Provost" To: "Mark Saad" Cc: "FreeBSD Hackers" Subject: Re: libxo question Date: Fri, 28 Dec 2018 21:19:09 +0100 X-Mailer: MailMate (2.0BETAr6133) Message-ID: <4ADD32A9-22D6-4983-BEC2-B3881EB59C81@FreeBSD.org> In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9D6F069AE4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 20:19:14 -0000 On 28 Dec 2018, at 20:31, Mark Saad wrote: > All > I am playing around with procstat and libxo on 12-STABLE from > yesterday . I wanted to get a list of thread_id's for some processes. > I wrote a quick python script to grab the data but xml output is not > well formed. Here is my sample script , which should work on python > 2.7 > > ----8<----------------------- > 1 import subprocess as sp > 2 import os,sys > 3 import pprint as pp > 4 import xml.etree.cElementTree as ET > 5 > 6 > 7 FNULL = open(os.devnull, 'w') > 8 cmd = "procstat --libxo xml -ta" > 9 p = sp.Popen(cmd, shell=True, stdout=sp.PIPE,stderr=FNULL, > executable="/bin/sh") > 10 text , err = p.communicate() > 11 > 12 root = ET.fromstring(text) > 13 > 14 pp.pprint(root) > 15 > 16 sys.exit(1) > ------------>8----------------------- > > I am constantly getting this odd issue about the xml being not well > formatted > > Traceback (most recent call last): > File "/tmp/test.py", line 12, in > root = ET.fromstring(text) > File "", line 124, in XML > cElementTree.ParseError: not well-formed (invalid token): line 1, > column 32 > > Attached is a copy of the xml. Any guidance would be helpful. > The attachment seems to have been eaten by a grue, but I can trivially reproduce the problem. Passing the output of `procstat --libxo xml -ta` to xmllint gives us: -:1: parser error : StartTag: invalid element name <0>0kernelki_pid); + asprintf(&pidstr, "pid_%d", kipp->ki_pid); if (pidstr == NULL) xo_errc(1, ENOMEM, "Failed to allocate memory in procstat()"); xo_open_container(pidstr); diff --git a/usr.bin/procstat/procstat_rusage.c b/usr.bin/procstat/procstat_rusage.c index 3d8c76370c0..f9caef49a2f 100644 --- a/usr.bin/procstat/procstat_rusage.c +++ b/usr.bin/procstat/procstat_rusage.c @@ -126,7 +126,7 @@ print_rusage(struct kinfo_proc *kipp) format_time(&kipp->ki_rusage.ru_stime)); if ((procstat_opts & PS_OPT_PERTHREAD) != 0) { - asprintf(&threadid, "%d", kipp->ki_tid); + asprintf(&threadid, "ID_%d", kipp->ki_tid); if (threadid == NULL) xo_errc(1, ENOMEM, "Failed to allocate memory in print_rusage()"); diff --git a/usr.bin/procstat/procstat_sigs.c b/usr.bin/procstat/procstat_sigs.c index 984d5d57f95..ceb36ca0dcb 100644 --- a/usr.bin/procstat/procstat_sigs.c +++ b/usr.bin/procstat/procstat_sigs.c @@ -155,7 +155,7 @@ procstat_threads_sigs(struct procstat *procstat, struct kinfo_proc *kipp) kinfo_proc_sort(kip, count); for (i = 0; i < count; i++) { kipp = &kip[i]; - asprintf(&threadid, "%d", kipp->ki_tid); + asprintf(&threadid, "ID_%d", kipp->ki_tid); if (threadid == NULL) xo_errc(1, ENOMEM, "Failed to allocate memory in " "procstat_threads_sigs()"); diff --git a/usr.bin/procstat/procstat_threads.c b/usr.bin/procstat/procstat_threads.c index c62bb516175..17f11044021 100644 --- a/usr.bin/procstat/procstat_threads.c +++ b/usr.bin/procstat/procstat_threads.c @@ -66,7 +66,7 @@ procstat_threads(struct procstat *procstat, struct kinfo_proc *kipp) kinfo_proc_sort(kip, count); for (i = 0; i < count; i++) { kipp = &kip[i]; - asprintf(&threadid, "%d", kipp->ki_tid); + asprintf(&threadid, "ID_%d", kipp->ki_tid); if (threadid == NULL) xo_errc(1, ENOMEM, "Failed to allocate memory in " "procstat_threads()"); It’s probably not the prettiest XML, and I’m not sure how useful the tags are now, but arguably tags with dynamic names are a bad idea anyway. I think you wouldn’t see this problem with JSON, so perhaps that’s a workaround you can consider as well. Regards, Kristof From owner-freebsd-hackers@freebsd.org Fri Dec 28 22:12:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4D4C1424697 for ; Fri, 28 Dec 2018 22:12:57 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 527606DFB8 for ; Fri, 28 Dec 2018 22:12:56 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-ed1-x52d.google.com with SMTP id g22so18480402edr.7 for ; Fri, 28 Dec 2018 14:12:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Bw/sfl9IM054107QKapveTrB/x6hbbToWPXYeU4RzH0=; b=GJlFS8J6umN7t3ntSpqFy4531AMNde/x//l5ttfz8hw+ICW2YZlA3ysqyQeMskDkYD HebFG77B2EOWp1INzyyyBv1Sl0+hNBi6O7qb7RHCaPoUWT7ROtAuIbUx1NS4LmFEefw8 a+p28Hfy52Mm+l2xt7iFVht6S8jyLyVV7fcsUvkkf09Uw8pGbyI1Sy1rPHZ/fwMXtxJF JgcQjxfs1aRLeE31bNLn2pNlG64yMGhpxan7RvFhgueEx3Zj6rhKvMjg7oCXA6NWp7Is Kww5K1pNOBE6MUPr8+fpojUOcomjcV7wAYss2OSOLv5eW0AvlGEpstUWUXO/ALtpO4ZJ B3mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Bw/sfl9IM054107QKapveTrB/x6hbbToWPXYeU4RzH0=; b=LYYXk57TwRKQG2OnVj5+hju13Ww/sO5CVYK1cIa0DsuO4G97cO2a8ugVwnn9zFF3ev uGz5h5qjyDe4l208BQAeuGtPTVBxlE3yIJD0LARoKjIyv2KO7IU2Slc+3o7QI+tt2Qug DonSwtM24NI2zoNZHolhj/GWn/OWG+0/hAwkh4qtuNiKoE0rbNNzg79coW+/4/YHVLOo C7jo+RB2nJhWxrcILaekXax2kIJ7JWCxXyY/dj9MQQ9AShTtkXcFvfPALPNs1lMBC9LW d6c9paBu11FDgVtJKsobb7kZH7aKHN8ZbWO3MC/kkm1KThVSh409Wbx+Q3J62E3DnOY8 QP/Q== X-Gm-Message-State: AA+aEWb/KpqAJnnBvC9uc2Ylw5/8YRSsKY8N0yk56P/H0YPsnmJcpXlR +wgIbjan8CieWBgK1FjuxUkeqHqN0WfgfQkIu1J2OMAn354= X-Google-Smtp-Source: AFSGD/Va6wyfdUXFVbnaYxZ3v3Ek9mdh2gsJq5eKHsCJ0dMHsCbd2QSllTefhKLd1ll+QNE5xNtcVhifatwlybqD8Dg= X-Received: by 2002:a50:9063:: with SMTP id z32mr23953321edz.133.1546035174810; Fri, 28 Dec 2018 14:12:54 -0800 (PST) MIME-Version: 1.0 References: <201812282040.wBSKeWPL023999@elf.torek.net> In-Reply-To: <201812282040.wBSKeWPL023999@elf.torek.net> From: Mark Saad Date: Fri, 28 Dec 2018 17:12:43 -0500 Message-ID: Subject: Re: libxo question To: Chris Torek Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 527606DFB8 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount-org.20150623.gappssmtp.com header.s=20150623 header.b=GJlFS8J6 X-Spamd-Result: default: False [-5.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[longcount-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[longcount.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; DKIM_TRACE(0.00)[longcount-org.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[d.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.93)[-0.934,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.46)[ip: (-8.96), ipnet: 2a00:1450::/32(-1.76), asn: 15169(-1.52), country: US(-0.08)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 22:12:57 -0000 On Fri, Dec 28, 2018 at 3:40 PM Chris Torek wrote: > > >Attached is a copy of the xml. Any guidance would be helpful. > > Your attachment was stripped before it got here, but the problem > is clear enough. Procstat / libxo is generating invalid XML. > > Here's a bit of sample "procstat --libxo xml" output, which > I generated locally by running > > procstat --libxo xml -ta > > and hand massaging the result: > > > > <0> > 0 > kernel > > <100000> > 100000 > swapper > -1 > [snip] > > Valid XML tags must begin with an alphabetic character or an > underscore (see https://www.w3schools.com/xml/xml_elements.asp), > and neither <0> nor <100000> do so. > > A quick workaround is to use json instead. However, libxo > probably should "work smarter" with tags. > > (XML is a terrible data-encoding language because of all of its > special rules. If you think you've found them all, watch out for > CDATA! JSON is better but still has some issues with encoding, > requiring that arbitrary binary data be atob or base64 encoded or > similar.) > > Chris I updated the patch form kb to work on 12 https://mirrors.nycbug.org/pub/patches/procstat-libxo-12-STABLE.patch Here is the xml output as well https://mirrors.nycbug.org/pub/patches/procstat.xml This works better then before and python's xml parser, mozilla and edge think its valid xml. I think this should be fixed what should we do to make it happen ? -- mark saad | nonesuch@longcount.org From owner-freebsd-hackers@freebsd.org Sat Dec 29 12:54:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE197141BBA8 for ; Sat, 29 Dec 2018 12:54:41 +0000 (UTC) (envelope-from srs0=/ph1=pg=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BEE26D454 for ; Sat, 29 Dec 2018 12:54:40 +0000 (UTC) (envelope-from srs0=/ph1=pg=sigsegv.be=kristof@codepro.be) Received: from [10.0.2.193] (ptr-8rh08k1xb3xxxgjknkb.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:fdd0:982a:91a4:374b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 5CD8B31066; Sat, 29 Dec 2018 13:54:31 +0100 (CET) From: "Kristof Provost" To: "Mark Saad" Cc: "Chris Torek" , "FreeBSD Hackers" Subject: Re: libxo question Date: Sat, 29 Dec 2018 13:54:29 +0100 X-Mailer: MailMate (2.0BETAr6133) Message-ID: In-Reply-To: References: <201812282040.wBSKeWPL023999@elf.torek.net> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9BEE26D454 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=sigsegv.be (policy=none); spf=pass (mx1.freebsd.org: domain of srs0=/ph1=pg=sigsegv.be=kristof@codepro.be designates 2a01:4f8:162:1127::2 as permitted sender) smtp.mailfrom=srs0=/ph1=pg=sigsegv.be=kristof@codepro.be X-Spamd-Result: default: False [-5.69 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[sigsegv.be : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a01:4f8:162:1127::2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-2.66)[ip: (-8.10), ipnet: 2a01:4f8::/29(-2.79), asn: 24940(-2.42), country: DE(-0.01)]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.2.1.1.2.6.1.0.8.f.4.0.1.0.a.2.list.dnswl.org : 127.0.9.2]; MX_GOOD(-0.01)[mx2.codepro.be,mx1.codepro.be]; NEURAL_HAM_SHORT(-0.92)[-0.917,0]; FORGED_SENDER(0.30)[kristof@sigsegv.be,srs0=/ph1=pg=sigsegv.be=kristof@codepro.be]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; FROM_NEQ_ENVFROM(0.00)[kristof@sigsegv.be,srs0=/ph1=pg=sigsegv.be=kristof@codepro.be]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2018 12:54:42 -0000 On 28 Dec 2018, at 23:12, Mark Saad wrote: > On Fri, Dec 28, 2018 at 3:40 PM Chris Torek > wrote: >> >>> Attached is a copy of the xml. Any guidance would be helpful. >> >> Your attachment was stripped before it got here, but the problem >> is clear enough. Procstat / libxo is generating invalid XML. >> >> Here's a bit of sample "procstat --libxo xml" output, which >> I generated locally by running >> >> procstat --libxo xml -ta >> >> and hand massaging the result: >> >> >> >> <0> >> 0 >> kernel >> >> <100000> >> 100000 >> swapper >> -1 >> [snip] >> >> Valid XML tags must begin with an alphabetic character or an >> underscore (see https://www.w3schools.com/xml/xml_elements.asp), >> and neither <0> nor <100000> do so. >> >> A quick workaround is to use json instead. However, libxo >> probably should "work smarter" with tags. >> >> (XML is a terrible data-encoding language because of all of its >> special rules. If you think you've found them all, watch out for >> CDATA! JSON is better but still has some issues with encoding, >> requiring that arbitrary binary data be atob or base64 encoded or >> similar.) >> >> Chris > > I updated the patch form kb to work on 12 > https://mirrors.nycbug.org/pub/patches/procstat-libxo-12-STABLE.patch > > Here is the xml output as well > https://mirrors.nycbug.org/pub/patches/procstat.xml > > This works better then before and python's xml parser, mozilla and > edge think its valid xml. > > I think this should be fixed what should we do to make it happen ? > I’ve posted https://reviews.freebsd.org/D18679 as a more generic way of addressing this. It’s quite possible that there are other users of libxo with the same problem, and this will help all of them. Regards, Kristof