From owner-freebsd-xen@freebsd.org Mon Jun 24 15:58:19 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F36E115D1AF4 for ; Mon, 24 Jun 2019 15:58:18 +0000 (UTC) (envelope-from bjb@darco.dk) Received: from post1.darco.dk (post1.blichsoft.dk [104.248.143.233]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4E60D74421 for ; Mon, 24 Jun 2019 15:58:18 +0000 (UTC) (envelope-from bjb@darco.dk) Received: from mail.blichsoft.dk (per [192.168.1.135]) by c3po.blichsoft.dk (Postfix) with ESMTPS id B4440224BC8 for ; Mon, 24 Jun 2019 17:58:16 +0200 (CEST) Received: by mail.blichsoft.dk (Postfix, from userid 110) id 66B38620A4; Mon, 24 Jun 2019 17:58:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darco.dk; s=mail; t=1561391896; bh=Y+un5Img1/G2nvlkIKC3RcfB2Ay/L0GtdvhSgrl6/d4=; h=From:Subject:To:References:Date:In-Reply-To; b=rFm9Oo/ZsOpu4sjOgrnCwZUHGvQZ889c8MbU/QwHot9lpxhSPWAs/oRTYiowsGYw5 JZ58iA6jcCGAR77OxWEnAE/N3hn4O56iCGKtfZ7aObGqmoCWjTCNU6L1afqXwtTd5T O1LDnW7J0x1KB7fsSA5iypStMCLCoRAHChfNB/lA= X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on per.blichsoft.dk X-Spam-Level: X-Spam-Status: No, score=-102.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, NO_DNS_FOR_FROM,USER_IN_WHITELIST autolearn=no autolearn_force=no version=3.4.2 Received: from [192.168.1.200] (sussi.darco.dk [192.168.1.200]) by mail.blichsoft.dk (Postfix) with ESMTP id A191962099 for ; Mon, 24 Jun 2019 17:58:15 +0200 (CEST) From: Bjarne Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org References: Message-ID: <03b839d1-39fb-f85f-2781-5219f69ba9ac@darco.dk> Date: Mon, 24 Jun 2019 17:58:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Status: No X-Virus-Checker-Version: clamassassin 1.2.4 with clamdscan / ClamAV 0.101.2/25489/Sun Jun 23 10:01:03 2019 X-Rspamd-Queue-Id: 4E60D74421 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=darco.dk header.s=mail header.b=rFm9Oo/Z; spf=pass (mx1.freebsd.org: domain of bjb@darco.dk designates 104.248.143.233 as permitted sender) smtp.mailfrom=bjb@darco.dk X-Spamd-Result: default: False [-2.10 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.829,0]; R_DKIM_ALLOW(-0.20)[darco.dk:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:104.248.143.233]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.99)[-0.989,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[darco.dk]; NEURAL_SPAM_SHORT(0.02)[0.020,0]; DKIM_TRACE(0.00)[darco.dk:+]; MX_GOOD(-0.01)[cached: mail.blichsoft.dk]; IP_SCORE(0.21)[asn: 14061(1.12), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.248.128.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 15:58:19 -0000 Well, to add some datapoints,  I just did a quick test om my private homeserver, testing with iperf3 -s  / iperf3 -c I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD   3,5Gbit/sec FBSD -> Centos:  3GBit/Sec Centos -> FBSD: 12 Gbit/sec Centos <-> Centos: 12 Gbit/sec OpenSuse <-> OpenSuse 13 GBit/sec So it ssems there is some fishy when Freebsd is used as client xen running on opensuse,  11 vm's,  freebsd , openSuse and Centos. No vm has been optimized in any way,  everything is out of the box. =========================== Test 1,  Freebsd 12 -> freebsd 12 =========================== Server Freebsd 12: ccepted connection from 192.168.1.100, port 39518 [  5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec   374 MBytes  3.14 Gbits/sec [  5]   1.00-2.00   sec   444 MBytes  3.72 Gbits/sec [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec [  5]   3.00-4.00   sec   438 MBytes  3.67 Gbits/sec [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec [  5]   7.00-8.00   sec   432 MBytes  3.63 Gbits/sec [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec [  5]  10.00-10.00  sec   332 KBytes  2.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver Client, freebsd 12: Connecting to host 10.192.168.99, port 5201 [  5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec   375 MBytes  3.14 Gbits/sec  239    178 KBytes [  5]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec  834   74.1 KBytes [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec  794   38.5 KBytes [  5]   3.00-4.00   sec   438 MBytes  3.68 Gbits/sec  808    198 KBytes [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec  777   67.0 KBytes [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec  795    120 KBytes [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec  763   72.7 KBytes [  5]   7.00-8.00   sec   432 MBytes  3.62 Gbits/sec  810    329 KBytes [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec  777    219 KBytes [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec  688    304 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec 7285             sender [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver ========================== test 2, centos 7.6 -> Freebsd 12 ========================== Server Freebsd 12: Accepted connection from 192.168.1.162, port 49180 [  5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec  1.17 GBytes  10.0 Gbits/sec [  5]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec [  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec [  5]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec [  5]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec [  5]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec [  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver Client, centos 7.6: Connecting to host 192.168.1.99, port 5201 [  4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd [  4]   0.00-1.00   sec  1.17 GBytes  10.1 Gbits/sec    0    822 KBytes [  4]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec    0   1.53 MBytes [  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   1.57 MBytes [  4]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.64 MBytes [  4]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.70 MBytes [  4]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec    0   1.70 MBytes [  4]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes [  4]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec    0   1.70 MBytes [  4]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes [  4]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec    0   1.70 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth       Retr [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec 0 sender [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver ========================== test 3, Freebsd 12 -> Centos 7.6 ========================== Server, Centos 7.6: Accepted connection from 192.168.1.100, port 20309 [  5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 [ ID] Interval           Transfer     Bandwidth [  5]   0.00-1.00   sec   335 MBytes  2.81 Gbits/sec [  5]   1.00-2.00   sec   379 MBytes  3.18 Gbits/sec [  5]   2.00-3.00   sec   368 MBytes  3.08 Gbits/sec [  5]   3.00-4.00   sec   371 MBytes  3.11 Gbits/sec [  5]   4.00-5.00   sec   374 MBytes  3.14 Gbits/sec [  5]   5.00-6.00   sec   362 MBytes  3.04 Gbits/sec [  5]   6.00-7.00   sec   381 MBytes  3.19 Gbits/sec [  5]   7.00-8.00   sec   384 MBytes  3.22 Gbits/sec [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec [  5]   9.00-10.00  sec   366 MBytes  3.07 Gbits/sec [  5]  10.00-10.13  sec  56.4 MBytes  3.52 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth [  5]   0.00-10.13  sec  0.00 Bytes  0.00 bits/sec sender [  5]   0.00-10.13  sec  3.66 GBytes  3.10 Gbits/sec                  receiver Client, Freebsd 12:   5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec   387 MBytes  3.25 Gbits/sec   83    309 KBytes [  5]   1.00-2.00   sec   380 MBytes  3.19 Gbits/sec   88   11.4 KBytes [  5]   2.00-3.00   sec   365 MBytes  3.06 Gbits/sec   83   54.2 KBytes [  5]   3.00-4.00   sec   372 MBytes  3.12 Gbits/sec   86    322 KBytes [  5]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec   83   2.85 KBytes [  5]   5.00-6.00   sec   366 MBytes  3.07 Gbits/sec   84    298 KBytes [  5]   6.00-7.00   sec   382 MBytes  3.20 Gbits/sec   87    356 KBytes [  5]   7.00-8.00   sec   380 MBytes  3.19 Gbits/sec   87    369 KBytes [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec   87   29.9 KBytes [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec   86    419 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec 854 sender [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec                  receiver ========================== test 4, Centos 7.6 -> Centos 7.6 ========================== Server: Accepted connection from 192.168.1.162, port 47606 [  5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 [ ID] Interval           Transfer     Bandwidth [  5]   0.00-1.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   1.00-2.00   sec  1.59 GBytes  13.7 Gbits/sec [  5]   2.00-3.00   sec  1.60 GBytes  13.8 Gbits/sec [  5]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec [  5]   4.00-5.00   sec  1.49 GBytes  12.8 Gbits/sec [  5]   5.00-6.00   sec  1.38 GBytes  11.9 Gbits/sec [  5]   6.00-7.00   sec  1.52 GBytes  13.1 Gbits/sec [  5]   7.00-8.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   8.00-9.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec [  5]  10.00-10.03  sec  48.1 MBytes  11.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth [  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec sender [  5]   0.00-10.03  sec  15.5 GBytes  13.2 Gbits/sec                  receiver Client, Centos 7.6 Connecting to host 192.168.1.155, port 5201 [  4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd [  4]   0.00-1.00   sec  1.66 GBytes  14.3 Gbits/sec    0   1.41 MBytes [  4]   1.00-2.00   sec  1.58 GBytes  13.5 Gbits/sec    0   1.46 MBytes [  4]   2.00-3.00   sec  1.61 GBytes  13.9 Gbits/sec    0   1.91 MBytes [  4]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec    0   2.13 MBytes [  4]   4.00-5.00   sec  1.47 GBytes  12.6 Gbits/sec    0   2.49 MBytes [  4]   5.00-6.00   sec  1.39 GBytes  11.9 Gbits/sec    0   2.56 MBytes [  4]   6.00-7.00   sec  1.53 GBytes  13.1 Gbits/sec    0   2.71 MBytes [  4]   7.00-8.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.73 MBytes [  4]   8.00-9.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.81 MBytes [  4]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec    0   2.84 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth       Retr [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec 0 sender [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec                  receiver =============================== test4, Opensuse 15.1 -> OpenSuse 15.1 ================================ Server, Opensuse 15.1: [  5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec [  5]   2.00-3.00   sec  1.58 GBytes  13.6 Gbits/sec [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   6.00-7.00   sec  1.58 GBytes  13.5 Gbits/sec [  5]   7.00-8.00   sec  1.60 GBytes  13.8 Gbits/sec [  5]   8.00-9.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   9.00-10.00  sec  1.61 GBytes  13.8 Gbits/sec [  5]  10.00-10.04  sec  68.0 MBytes  14.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver Client, OpenSuse 15.1: [  5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec  1.58 GBytes  13.6 Gbits/sec    0    723 KBytes [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec    0    805 KBytes [  5]   2.00-3.00   sec  1.57 GBytes  13.5 Gbits/sec    0    926 KBytes [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec    0    926 KBytes [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec    0    970 KBytes [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec    0   1.05 MBytes [  5]   6.00-7.00   sec  1.57 GBytes  13.5 Gbits/sec    0   1.05 MBytes [  5]   7.00-8.00   sec  1.61 GBytes  13.8 Gbits/sec    0   1.05 MBytes [  5]   8.00-9.00   sec  1.59 GBytes  13.7 Gbits/sec    0   1.05 MBytes [  5]   9.00-10.00  sec  1.62 GBytes  13.9 Gbits/sec    0   1.05 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  15.9 GBytes  13.7 Gbits/sec 0 sender [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver > I've noticed very slow networking speed between VM's with FreeBSD on the > same host (XCP-ng 7.6.0) for more recent FreeBSD-versions. > > > I've made some tests that show me that something happened from 10.4-RELEASE > to 11.0-RELEASE that had a huge impact on network performance, and > 12.0-RELEASE is even slower. > > > My test setup: > > Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. > > Network: Internal Private Network on the host (not connected to a PIF). > Each VM has only one VIF connected to this network. > > VM's: > > 2 x 12.0-RELEASE > > 2 x 11.0-RELEASE > > 2 x 10.4-RELEASE > > > All clean identical installs from XenOrchestra, only installed iperf on > each VM for testing. (xe-guest-utilities makes no difference in my tests, > I've tried with and without). > > > iperf -s on first server listed below, and iperf -c -r on the second > to test speed back and forth: > > > 12.0 <-> 12.0: 50Mbit as client and server > > 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) > > 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). > > 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server > > 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server > > > As I side note, not sure if related, but I've noticed that I can't run > iperf with -r flag on 10.4-RELEASE. I get this error message: > > > iperf -c 172.31.16.122 -r > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > write failed: Broken pipe > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > > > > I can run iperf -s fine, and iperf -c from the other 10.4 VM though: > > > > iperf -c 172.31.16.122 > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec > > > > What have I tried to solve this? > > I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via > XCP-ng. Disabled basically everything without any difference in iperf > results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; > ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off > > > Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum > -rxcsum -tso -lro and no difference here either. > > > Any ideas of how to proceed now to find a solution for this? > > > Regards, > > Christian > -- mvh, Bjarne