From owner-freebsd-hackers@freebsd.org Mon Apr 15 12:39:00 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2CBB1572A26 for ; Mon, 15 Apr 2019 12:38:59 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63CD677535 for ; Mon, 15 Apr 2019 12:38:57 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from zeta.dino.sk (fw3.dino.sk [84.245.95.254]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 15 Apr 2019 14:38:49 +0200 id 00DED247.5CB47B59.0000318D Date: Mon, 15 Apr 2019 14:38:49 +0200 From: Milan Obuch To: Warner Losh Cc: freebsd-hackers@freebsd.org Subject: Re: Any WAFER-BT users? Message-ID: <20190415143849.0f09eb91@zeta.dino.sk> In-Reply-To: References: <20190409103256.5d095bb5@zeta.dino.sk> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; i386-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 63CD677535 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd-hackers@dino.sk designates 84.245.65.72 as permitted sender) smtp.mailfrom=freebsd-hackers@dino.sk X-Spamd-Result: default: False [-4.42 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dino.sk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mail.dino.sk]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[72.65.245.84.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-1.00)[-0.995,0]; IP_SCORE(-2.11)[ip: (-7.09), ipnet: 84.245.64.0/18(-3.54), asn: 16160(0.02), country: SK(0.04)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:16160, ipnet:84.245.64.0/18, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 12:39:00 -0000 On Tue, 9 Apr 2019 08:25:03 -0600 Warner Losh wrote: > On Tue, Apr 9, 2019 at 2:39 AM Milan Obuch > wrote: > > > Hi, > > > > I am testing WAFER-BT-N28071-R20 board, details at > > https://www.ieiworld.com/en/product/model.php?II=637 > > > > I tried installing from mini memstick image for both i386 and amd64 > > architectures, 12.0-RELEASE, both are failing the same way - boot > > loader starts, menu is displayed, selected/timed out, then > > > > Loading kernel... > > /boot/kernel/kernel [text= etc.] > > Loading configured modules... > > can't find '/boot/entropy' > > | > > > > and that's all. No output from kernel, no sign of life :( > > > > This symptom means one of two things: > (1) The boot loader is crashing for some reason, but not reporting it > (way too common on UEFI, sadly) > (2) The boot loader is configuring some crazy thing as console, > loading the kernel and it is dying and reporting the message into > cloud kookoo land. > > > > This board was tested with Windows and Linux, so it could not be > > something totally wrong, just some BIOS settings or somesuch... I > > tried loading without ACPI, but then it fails with 'can't find > > APIC' panic or aomething similar (just from memory). Also, trying > > run kernel manually from loader does not reveal anything, 'boot > > -vs' outputs nothing as well. > > > > If you can get to the stage where it thinks it's booting manually, it > isn't likely (1), though it's still a small possibility while loading > the kernel. It would be really helpful to have a screen shot where it > died. You might also setup a serial console too. > Thanks, that's it - for some reason loader switches to serial console after starting kernel, or is it kernel itself? Does not matter... After installing system, it behaves that way - switch to serial console. I think video is the cullprit - from dmesg: Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE r341666 GENERIC i386 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT: init without driver. CPU: Intel(R) Celeron(R) CPU N2807 @ 1.58GHz (1583.38-MHz 686-class CPU) Origin="GenuineIntel" Id=0x30679 Family=0x6 Model=0x37 Stepping=9 Features=0xbfebfbff Features2=0x41d8e3bf AMD Features=0x28100000 AMD Features2=0x101 Structured Extended Features=0x2282 Structured Extended Features3=0xc000000 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics Later: vgapci0: port 0xe080-0xe087 mem 0xd0000000-0xd03fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0 vgapci0: Boot video device As this should be a video terminal, it is kind of show stopper for me. I am going to try graphics/drm-kmod port, whether it is of any help... Regards, Milan From owner-freebsd-hackers@freebsd.org Mon Apr 15 16:29:04 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3C0E1578469 for ; Mon, 15 Apr 2019 16:29:04 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5190288D49 for ; Mon, 15 Apr 2019 16:29:03 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3FGJJAd082765 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 15 Apr 2019 18:19:20 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3FGJJDi082761; Mon, 15 Apr 2019 18:19:19 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Mon, 15 Apr 2019 18:19:19 +0200 (CEST) From: Wojciech Puchar To: Milan Obuch cc: Warner Losh , freebsd-hackers@freebsd.org Subject: Re: Any WAFER-BT users? In-Reply-To: <20190415143849.0f09eb91@zeta.dino.sk> Message-ID: References: <20190409103256.5d095bb5@zeta.dino.sk> <20190415143849.0f09eb91@zeta.dino.sk> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 5190288D49 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-5.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[puchar.net]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[puchar.net]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.55)[ip: (-9.37), ipnet: 194.1.144.0/24(-4.69), asn: 43476(-3.75), country: PL(0.07)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 16:29:05 -0000 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) > VT: init without driver. how about the line above? > CPU: Intel(R) Celeron(R) CPU N2807 @ 1.58GHz (1583.38-MHz 686-class CPU) > Origin="GenuineIntel" Id=0x30679 Family=0x6 Model=0x37 Stepping=9 > Features=0xbfebfbff > Features2=0x41d8e3bf > AMD Features=0x28100000 > AMD Features2=0x101 > Structured Extended Features=0x2282 > Structured Extended Features3=0xc000000 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > > Later: > > vgapci0: port 0xe080-0xe087 mem 0xd0000000-0xd03fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0 > vgapci0: Boot video device > > As this should be a video terminal, it is kind of show stopper for me. > I am going to try graphics/drm-kmod port, whether it is of any help... > > Regards, > Milan > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Mon Apr 15 19:02:40 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 470AF157B8E6 for ; Mon, 15 Apr 2019 19:02:40 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B08DE8E6CD for ; Mon, 15 Apr 2019 19:02:38 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from zeta.dino.sk (fw3.dino.sk [84.245.95.254]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 15 Apr 2019 21:02:36 +0200 id 00DED244.5CB4D54C.000055B8 Date: Mon, 15 Apr 2019 21:02:35 +0200 From: Milan Obuch To: Wojciech Puchar Cc: freebsd-hackers@freebsd.org Subject: Re: Any WAFER-BT users? Message-ID: <20190415210235.75bc4165@zeta.dino.sk> In-Reply-To: References: <20190409103256.5d095bb5@zeta.dino.sk> <20190415143849.0f09eb91@zeta.dino.sk> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; i386-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B08DE8E6CD X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd-hackers@dino.sk designates 84.245.65.72 as permitted sender) smtp.mailfrom=freebsd-hackers@dino.sk X-Spamd-Result: default: False [-4.31 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dino.sk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mail.dino.sk]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[72.65.245.84.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.95)[-0.949,0]; IP_SCORE(-2.05)[ip: (-6.86), ipnet: 84.245.64.0/18(-3.43), asn: 16160(0.02), country: SK(0.04)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:16160, ipnet:84.245.64.0/18, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 19:02:40 -0000 On Mon, 15 Apr 2019 18:19:19 +0200 (CEST) Wojciech Puchar wrote: > > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based > > on LLVM 6.0.1) VT: init without driver. =20 >=20 > how about the line above? > I saw it and that was actually the reason I posted it here, along with the CPU definition, in hope somebody has an idea why was VT initialized without driver, thus forcing kernel to use serial console. > > CPU: Intel(R) Celeron(R) CPU N2807 @ 1.58GHz (1583.38-MHz > > 686-class CPU) Origin=3D"GenuineIntel" Id=3D0x30679 Family=3D0x6 > > Model=3D0x37 Stepping=3D9 > > Features=3D0xbfebfbff > > Features2=3D0x41d8e3bf > > AMD Features=3D0x28100000 AMD > > Features2=3D0x101 Structured Extended > > Features=3D0x2282 Structured Extended > > Features3=3D0xc000000 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > > TSC: P-state invariant, performance statistics > > > > Later: > > > > vgapci0: port 0xe080-0xe087 mem > > 0xd0000000-0xd03fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on > > pci0 vgapci0: Boot video device > > Somehow, VGA device is detected here, question remains - how could this device be used for display? I hope somebody can give some advice. In the meantime, I am building new world/kernel to test board and prepare drm port... but ut takes some time. By the way, output given above was from 12.0-RELEASE GENERIC kernel (i386 arch=C7=90tecture). Milan From owner-freebsd-hackers@freebsd.org Tue Apr 16 06:50:39 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C727158B238 for ; Tue, 16 Apr 2019 06:50:39 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2C9C376E48 for ; Tue, 16 Apr 2019 06:50:37 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3G6oWTS092049 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 16 Apr 2019 08:50:32 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3G6oWTv092046; Tue, 16 Apr 2019 08:50:32 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 16 Apr 2019 08:50:32 +0200 (CEST) From: Wojciech Puchar To: Milan Obuch cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Any WAFER-BT users? In-Reply-To: <20190415210235.75bc4165@zeta.dino.sk> Message-ID: References: <20190409103256.5d095bb5@zeta.dino.sk> <20190415143849.0f09eb91@zeta.dino.sk> <20190415210235.75bc4165@zeta.dino.sk> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 X-Rspamd-Queue-Id: 2C9C376E48 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; MIME_TRACE(0.00)[0:+,1:+]; DMARC_NA(0.00)[puchar.net]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; CTYPE_MIXED_BOGUS(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.90)[-0.899,0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.53)[ip: (-9.33), ipnet: 194.1.144.0/24(-4.67), asn: 43476(-3.73), country: PL(0.07)] Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 06:50:39 -0000 is vt_vga in kernel config? On Mon, 15 Apr 2019, Milan Obuch wrote: > On Mon, 15 Apr 2019 18:19:19 +0200 (CEST) > Wojciech Puchar wrote: > >>> FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based >>> on LLVM 6.0.1) VT: init without driver. >> >> how about the line above? >> > > I saw it and that was actually the reason I posted it here, along with > the CPU definition, in hope somebody has an idea why was VT initialized > without driver, thus forcing kernel to use serial console. > >>> CPU: Intel(R) Celeron(R) CPU N2807 @ 1.58GHz (1583.38-MHz >>> 686-class CPU) Origin="GenuineIntel" Id=0x30679 Family=0x6 >>> Model=0x37 Stepping=9 >>> Features=0xbfebfbff >>> Features2=0x41d8e3bf >>> AMD Features=0x28100000 AMD >>> Features2=0x101 Structured Extended >>> Features=0x2282 Structured Extended >>> Features3=0xc000000 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID >>> TSC: P-state invariant, performance statistics >>> >>> Later: >>> >>> vgapci0: port 0xe080-0xe087 mem >>> 0xd0000000-0xd03fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on >>> pci0 vgapci0: Boot video device >>> > > Somehow, VGA device is detected here, question remains - how could this > device be used for display? I hope somebody can give some advice. In > the meantime, I am building new world/kernel to test board and prepare > drm port... but ut takes some time. > > By the way, output given above was from 12.0-RELEASE GENERIC kernel > (i386 archǐtecture). > > Milan > > From owner-freebsd-hackers@freebsd.org Tue Apr 16 07:27:29 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D377158BCB0 for ; Tue, 16 Apr 2019 07:27:29 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6AA677EF0 for ; Tue, 16 Apr 2019 07:27:27 +0000 (UTC) (envelope-from freebsd-hackers@dino.sk) Received: from zeta.dino.sk (fw3.dino.sk [84.245.95.254]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Tue, 16 Apr 2019 09:27:24 +0200 id 00DED243.5CB583DC.0000AA7B Date: Tue, 16 Apr 2019 09:27:24 +0200 From: Milan Obuch To: Wojciech Puchar Cc: freebsd-hackers@freebsd.org Subject: Re: Any WAFER-BT users? Message-ID: <20190416092724.242e93d3@zeta.dino.sk> In-Reply-To: References: <20190409103256.5d095bb5@zeta.dino.sk> <20190415143849.0f09eb91@zeta.dino.sk> <20190415210235.75bc4165@zeta.dino.sk> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; i386-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B6AA677EF0 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd-hackers@dino.sk designates 84.245.65.72 as permitted sender) smtp.mailfrom=freebsd-hackers@dino.sk X-Spamd-Result: default: False [-4.13 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dino.sk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mail.dino.sk]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[72.65.245.84.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.84)[-0.836,0]; IP_SCORE(-1.98)[ip: (-6.64), ipnet: 84.245.64.0/18(-3.32), asn: 16160(0.02), country: SK(0.04)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:16160, ipnet:84.245.64.0/18, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 07:27:29 -0000 On Tue, 16 Apr 2019 08:50:32 +0200 (CEST) Wojciech Puchar wrote: > is vt_vga in kernel config? > [ snip ] > > Somehow, VGA device is detected here, question remains - how could > > this device be used for display? I hope somebody can give some > > advice. In the meantime, I am building new world/kernel to test > > board and prepare drm port... but ut takes some time. > > > > By the way, output given above was from 12.0-RELEASE GENERIC kernel > > (i386 arch=C7=90tecture). > > See it is GENERIC, i. e. yes - kldstat -v shows there is nexus/vtvga module present, just to be sure. Problem is either Celeron N2807 CPU is not fully supported or boot environment is not correctly set-up, I just have no idea what that could be and where look for it. Regards, Milan From owner-freebsd-hackers@freebsd.org Tue Apr 16 20:26:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58AC5157C563 for ; Tue, 16 Apr 2019 20:26:21 +0000 (UTC) (envelope-from arichardson.kde@gmail.com) Received: from mail-yw1-f45.google.com (mail-yw1-f45.google.com [209.85.161.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C5A1769DE for ; Tue, 16 Apr 2019 20:26:20 +0000 (UTC) (envelope-from arichardson.kde@gmail.com) Received: by mail-yw1-f45.google.com with SMTP id d132so7814622ywa.2 for ; Tue, 16 Apr 2019 13:26:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Fg0PLT0alDrAoSvYNq2Ds6jnNPifgcCk7gwQZHxSEpc=; b=aAu+dHGZV+AurCDfqXMHRm4IPKV43K1tN0PJfx+h8tBt7S2flcAV/6z/6BWdjfnIQn DJr0QeSjbPkU0masKF9DL8CJW9hz3DX3UAnf7EYOih8hSjDIf/XBH1PSIccVuTciXNns dwg1+tY+/42metNMTXvcYT+RPO7qjsHpqMAGHVY968M38RXx4gvxXQsewcict+V2W1fp I2eyfOUzHMez1lJQEr8m1gpyT9q9OPxL8B744vEnRzvc91uMgo7KR/QGDwTefcB1goZo vu/pe7YsXHS1gnKovSyDfmaEfkCVnkiCj4qQhgrEbW6aOf7wAx7Hl3U5bhd7q8s0Ilwx 1kuA== X-Gm-Message-State: APjAAAWFAsg18FtwjjDGG9+ExXhou7LycXQnSUIncyM+jfXFYsS3UipG 6ZalD0rNM3hJW8sL8BMnIZGjqswR X-Google-Smtp-Source: APXvYqxjEMtUgoKnn3Qq4S1FmtGIlfAmI2X0dass6MXBrqajIm41XoLOqnGoKvgH9cS+CgVIppncSQ== X-Received: by 2002:a81:4606:: with SMTP id t6mr43882296ywa.422.1555446374071; Tue, 16 Apr 2019 13:26:14 -0700 (PDT) Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com. [209.85.219.173]) by smtp.gmail.com with ESMTPSA id b68sm22593214ywh.50.2019.04.16.13.26.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 13:26:13 -0700 (PDT) Received: by mail-yb1-f173.google.com with SMTP id e190so8277065ybf.2 for ; Tue, 16 Apr 2019 13:26:13 -0700 (PDT) X-Received: by 2002:a25:dc0f:: with SMTP id y15mr58192345ybe.185.1555446373575; Tue, 16 Apr 2019 13:26:13 -0700 (PDT) MIME-Version: 1.0 From: Alexander Richardson Date: Tue, 16 Apr 2019 16:26:02 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Clang-format config for FreeBSD style To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 7C5A1769DE X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of arichardsonkde@gmail.com designates 209.85.161.45 as permitted sender) smtp.mailfrom=arichardsonkde@gmail.com X-Spamd-Result: default: False [-4.09 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.85)[-0.845,0]; FORGED_SENDER(0.30)[arichardson@freebsd.org,arichardsonkde@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[arichardson@freebsd.org,arichardsonkde@gmail.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.23)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.21), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[45.161.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 20:26:21 -0000 Hello all, For CheriBSD I have added a .clang-format file[1] that (at least based on some simple checking) gets reasonably close to existing style. Is there any interest in adding this to the main repository so that future patches can easily be formatted using git-clang-format or clang-format-diff. I find that clang-format increases my productivity when I submit patches for LLVM since I can write my code, run git-clang-format and then upload the patch to phabricator and know that I won't have to fix lots of coding style errors before the patch can be submitted. Would there be interest in adding this config file to the src repository? I am not proposing mandatory clang-format before commit in a phabricator hook, but I do wonder if this would be possible in the future once clang-format produces correctly formatted code. The attached config file gets close for some files that I tested but there are definitely some line-wrapping changes that could cause unncessary code churn if it was applied in bulk to existing code. However, I do think that it could be useful for new patches (and makes it easier for people like me who can't remember all the coding style requirements for the different projects they contribute to). Alex [1] # Basic .clang-format --- BasedOnStyle: WebKit AlignAfterOpenBracket: DontAlign AlignConsecutiveAssignments: false AlignConsecutiveDeclarations: false AlignEscapedNewlines: Left AlignOperands: false AlignTrailingComments: false AllowAllParametersOfDeclarationOnNextLine: false AllowShortBlocksOnASingleLine: false AllowShortCaseLabelsOnASingleLine: false AllowShortFunctionsOnASingleLine: InlineOnly AllowShortIfStatementsOnASingleLine: false AllowShortLoopsOnASingleLine: false AlwaysBreakAfterDefinitionReturnType: TopLevel AlwaysBreakAfterReturnType: TopLevelDefinitions BinPackArguments: true BinPackParameters: true BreakBeforeBinaryOperators: None BreakBeforeBraces: WebKit BreakBeforeTernaryOperators: false # TODO: BreakStringLiterals can cause very strange formatting so turn it off? # BreakStringLiterals: false # Avoid breaking function calls before the first argument: PenaltyBreakBeforeFirstCallParameter: 100000 CommentPragmas: '^ CHERI CHANGES START' CompactNamespaces: true DerivePointerAlignment: false DisableFormat: false ForEachMacros: - SLIST_FOREACH - SLIST_FOREACH_SAFE - LIST_FOREACH - LIST_FOREACH_SAFE - STAILQ_FOREACH - STAILQ_FOREACH_SAFE - TAILQ_FOREACH - TAILQ_FOREACH_SAFE IndentCaseLabels: false IndentPPDirectives: None Language: Cpp NamespaceIndentation: None PointerAlignment: Right ContinuationIndentWidth: 4 IndentWidth: 8 TabWidth: 8 ColumnLimit: 80 UseTab: Always SortIncludes: false ... From owner-freebsd-hackers@freebsd.org Tue Apr 16 21:04:11 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3383157D88B for ; Tue, 16 Apr 2019 21:04:11 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-it1-f179.google.com (mail-it1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BCD380950; Tue, 16 Apr 2019 21:04:10 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-it1-f179.google.com with SMTP id s3so1093337itk.1; Tue, 16 Apr 2019 14:04:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=goseJGiIn/oEixIgy0kKkOy6CI29DIEZL+guIN9Mdpc=; b=EVxmCT7yUJGk45EpG2qYdk22nVSlTRsEtchZlsBlWpUxioc5pk8YC9DUGGO0Od29eT /yUs7h77/oVZ1gsH20KHLk//DhgIQIcLPpnLS6Tu3gWqafBlGQY6cs71tNttKylzb/oE 4yxU5174y0rAPoTf9FmmorazAuX/W+TzSgPRatNWuKJ7RE8SJApm+pU769Pb4+pqKxzh U2n80JCxJsI678VOfuCn2KKiUoqLpNVZAtuz0NI2jHVp4cskclkMhkckRMFo+mSJrGpK V/EylmFMKMDDXdaBvPpWXreapqcDIdwV20PnBuudhcQK27D0YyXJyRpNKTRIFYZ410qW h6ew== X-Gm-Message-State: APjAAAUb6ewzn5z3FCAZ15c67FwXDNU8tcyD91jIIyJSx6sFKsGwLdou sYUJt4hdBSeWrt4tOZxQTJ1k3NoGg7VA5oV7brQfog== X-Google-Smtp-Source: APXvYqxtrfL4NZrAILcgKzP8xKXLeWmy5B24cAK3xhyZYOHiYbwTflsK12wnHqIJT40nYW+ffqeh9noNmHKbYXs5f4g= X-Received: by 2002:a02:ab90:: with SMTP id t16mr58646018jan.119.1555448643553; Tue, 16 Apr 2019 14:04:03 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Tue, 16 Apr 2019 17:03:50 -0400 Message-ID: Subject: Re: Clang-format config for FreeBSD style To: Alexander Richardson Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 9BCD380950 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.179 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.55 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-2.81)[ip: (-7.90), ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.21), country: US(-0.06)]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[179.166.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.73)[-0.731,0]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 21:04:11 -0000 On Tue, 16 Apr 2019 at 16:26, Alexander Richardson wrote: > > Hello all, > > For CheriBSD I have added a .clang-format file[1] that (at least based > on some simple checking) gets reasonably close to existing style. > Is there any interest in adding this to the main repository so that > future patches can easily be formatted using git-clang-format or > clang-format-diff. I'd be happy to have a .clang-format in the repository; even if it's not 100% correct it'd be nice to have something that gets close, for those who want to use it. I'd also be interested in improvements to clang-format itself to be able to fully match style(9). From owner-freebsd-hackers@freebsd.org Wed Apr 17 15:09:04 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C640E156E973 for ; Wed, 17 Apr 2019 15:09:04 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 28AE2834DD for ; Wed, 17 Apr 2019 15:09:02 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3HF8tqI088314 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 17 Apr 2019 17:08:55 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3HF8tck088311 for ; Wed, 17 Apr 2019 17:08:55 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 17 Apr 2019 17:08:55 +0200 (CEST) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: openvpn and system overhead Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 28AE2834DD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.81 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[puchar.net]; NEURAL_HAM_SHORT(-0.97)[-0.971,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.53)[ip: (-9.33), ipnet: 194.1.144.0/24(-4.67), asn: 43476(-3.73), country: PL(0.07)]; DMARC_NA(0.00)[puchar.net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 15:09:04 -0000 i'm running openvpn server on Xeon E5 2620 server. when receiving 100Mbit/s traffic over VPN it uses 20% of single core. At least 75% of it is system time. Seems like 500Mbit/s is a max for a single openvpn process. can anything be done about that to improve performance? From owner-freebsd-hackers@freebsd.org Wed Apr 17 15:22:47 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6DCB156F48E for ; Wed, 17 Apr 2019 15:22:46 +0000 (UTC) (envelope-from debdrup@gmail.com) Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C001684283 for ; Wed, 17 Apr 2019 15:22:45 +0000 (UTC) (envelope-from debdrup@gmail.com) Received: by mail-lj1-x229.google.com with SMTP id f23so1425684ljc.0 for ; Wed, 17 Apr 2019 08:22:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=bFkKWxh9rPbXZ4O9rKmooHgJDMIEYTk5fK11umv8qa4=; b=iIoK6WXEkBpUce0sJ4RQBiIukjxpMRh579oe6BXRNd9JbprscnQoG79cQeYqztZ8w8 7EXBezsG0OpxpeM7k+2siCGK8CJ50GxHdPCScEBT0/cvngvmbDrrvkIlTEGsi1SYKTiS dgjJm/wUFc+ZgcNZxr7oZUzaqtPtzYmV7OWVWlWXIbtHyvZoEF/7d96Ekf0qys3iGfPr 1gQ0f9C+l3xMg8Y3jJevxkTA644AQUSGrTs0yODuLBVEXXh3qGcB0h3G0olYIWFmVdZR nyatQ1UGb5qTFw0hUfOZEcG2W9TgbYMT/K/Tg9e8u3jIbhUpQUUOFHiH6HcGGUukX5Yv iXGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=bFkKWxh9rPbXZ4O9rKmooHgJDMIEYTk5fK11umv8qa4=; b=lF1GV5DfGW1TPgJXSYwBPToudTih7zdxY1rt4DQKWk4egpfBJsviAMGx6OSNTKzzl5 LrjAc3Mk1UE8XFZG1QyjL0UPjWuluUESCDRhEz4ClPjHDoVi8gE2tGF4TmBRXK8xrEk8 5P/udR/InO843pYWBkOrbAWxbFLMmOImalXbPpqXsez3ycVtDWmP1+T0YV219TVl32nN UN/VWH/y3r7I0hmzrsch8S1WU94mchZBRl3+Gg6DTzRb++bobQX6/P43S8m+KPcdK92l ewu9iI1J8n7H/dots9icRdYZwy9wWZt3CHOyX803VdL3d6KKu7MOZG1nnEyDHQ7WhrGh fqRg== X-Gm-Message-State: APjAAAXY3NMlGQWWCkO3hR14ANq01zPW340ixPiBc12aiH8BotuOCNuL Cv5Vg3Cp00x21q83Nsy5kNwU+ijn5k6I9fQwjoRMShxU X-Google-Smtp-Source: APXvYqzemU9Qbso0rYnHy5TQS6Bn0biaiLrt3ejKHLd0cD/SEB2mTs9PmqZ49bhmX86iOJGQQg8UAOfTXacvWgZv4Ug= X-Received: by 2002:a2e:390c:: with SMTP id g12mr47485380lja.174.1555514564213; Wed, 17 Apr 2019 08:22:44 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: "D. Ebdrup" Date: Wed, 17 Apr 2019 17:22:31 +0200 Message-ID: Subject: Re: openvpn and system overhead To: freebsd-hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: C001684283 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=iIoK6WXE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of debdrup@gmail.com designates 2a00:1450:4864:20::229 as permitted sender) smtp.mailfrom=debdrup@gmail.com X-Spamd-Result: default: False [-6.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[9.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-2.84)[ip: (-9.56), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.22), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 15:22:47 -0000 Hi Wojciech, I believe you might've run into one of the design decisions that was made by OpenVPN, specifically related to how it exclusively favours spawning more processes rather than implement any form of threading. I believe it's currently on the roadmap [1] of things that they would like to address at one point, but I'm not aware of any work being done on it at present. [1]: https://community.openvpn.net/openvpn/wiki/RoadMap#Threading Daniel Ebdrup aka. D. Ebdrup. On Wed, Apr 17, 2019 at 5:11 PM Wojciech Puchar wrote: > > i'm running openvpn server on Xeon E5 2620 server. > > when receiving 100Mbit/s traffic over VPN it uses 20% of single core. > At least 75% of it is system time. > > Seems like 500Mbit/s is a max for a single openvpn process. > > can anything be done about that to improve performance? > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Wed Apr 17 15:37:12 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9003A156FD36 for ; Wed, 17 Apr 2019 15:37:12 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BEE4E84CD8 for ; Wed, 17 Apr 2019 15:37:11 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3HFb9Sw095027 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 17 Apr 2019 17:37:09 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3HFb9Fw095024; Wed, 17 Apr 2019 17:37:09 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 17 Apr 2019 17:37:09 +0200 (CEST) From: Wojciech Puchar To: "D. Ebdrup" cc: freebsd-hackers Subject: Re: openvpn and system overhead In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: BEE4E84CD8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.90)[-0.898,0]; IP_SCORE(-3.54)[ip: (-9.34), ipnet: 194.1.144.0/24(-4.67), asn: 43476(-3.74), country: PL(0.07)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 15:37:12 -0000 > I believe you might've run into one of the design decisions that was > made by OpenVPN, specifically related to how it exclusively favours > spawning more processes rather than implement any form of threading. well i know i can divide it to many processes but my question wasn't about that, but about it's cpu usage. most if it is system time. can this overhead be reduced? > > I believe it's currently on the roadmap [1] of things that they would > like to address at one point, but I'm not aware of any work being done > on it at present. > > [1]: https://community.openvpn.net/openvpn/wiki/RoadMap#Threading > > Daniel Ebdrup aka. D. Ebdrup. > > On Wed, Apr 17, 2019 at 5:11 PM Wojciech Puchar wrote: >> >> i'm running openvpn server on Xeon E5 2620 server. >> >> when receiving 100Mbit/s traffic over VPN it uses 20% of single core. >> At least 75% of it is system time. >> >> Seems like 500Mbit/s is a max for a single openvpn process. >> >> can anything be done about that to improve performance? >> >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Wed Apr 17 15:42:38 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5170B1570091 for ; Wed, 17 Apr 2019 15:42:38 +0000 (UTC) (envelope-from SRS0=3mPw=ST=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B78385104 for ; Wed, 17 Apr 2019 15:42:37 +0000 (UTC) (envelope-from SRS0=3mPw=ST=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 4D22428423; Wed, 17 Apr 2019 17:37:15 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2FEEB28428; Wed, 17 Apr 2019 17:37:14 +0200 (CEST) Subject: Re: openvpn and system overhead To: Wojciech Puchar , freebsd-hackers@freebsd.org References: From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <8648d069-2172-2c09-8e59-d66a8265a120@quip.cz> Date: Wed, 17 Apr 2019 17:37:15 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 9B78385104 X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [5.04 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.93)[0.935,0]; IP_SCORE(0.92)[ip: (0.42), ipnet: 94.124.104.0/21(0.21), asn: 42000(3.89), country: CZ(0.08)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(1.00)[0.995,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.998,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=3mPw=ST=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=3mPw=ST=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 15:42:38 -0000 Wojciech Puchar wrote on 2019/04/17 17:08: > i'm running openvpn server on Xeon E5 2620 server. > > when receiving 100Mbit/s traffic over VPN it uses 20% of single core. > At least 75% of it is system time. > > Seems like 500Mbit/s is a max for a single openvpn process. > > can anything be done about that to improve performance? You can play with ciphers, AES-NI etc. https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux Miroslav Lachman From owner-freebsd-hackers@freebsd.org Wed Apr 17 15:54:07 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E43A1570578 for ; Wed, 17 Apr 2019 15:54:07 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 436C185879 for ; Wed, 17 Apr 2019 15:54:06 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3HFs14f098408 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 17 Apr 2019 17:54:01 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3HFs11d098405; Wed, 17 Apr 2019 17:54:01 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 17 Apr 2019 17:54:01 +0200 (CEST) From: Wojciech Puchar To: Miroslav Lachman <000.fbsd@quip.cz> cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: openvpn and system overhead In-Reply-To: <8648d069-2172-2c09-8e59-d66a8265a120@quip.cz> Message-ID: References: <8648d069-2172-2c09-8e59-d66a8265a120@quip.cz> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 436C185879 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.78 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; NEURAL_HAM_SHORT(-0.93)[-0.935,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.54)[ip: (-9.34), ipnet: 194.1.144.0/24(-4.67), asn: 43476(-3.74), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 15:54:07 -0000 On Wed, 17 Apr 2019, Miroslav Lachman wrote: > Wojciech Puchar wrote on 2019/04/17 17:08: >> i'm running openvpn server on Xeon E5 2620 server. >> >> when receiving 100Mbit/s traffic over VPN it uses 20% of single core. >> At least 75% of it is system time. >> >> Seems like 500Mbit/s is a max for a single openvpn process. >> >> can anything be done about that to improve performance? > > You can play with ciphers, AES-NI etc. > https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux > > Miroslav Lachman > > again. it's system time mostly not user time. From owner-freebsd-hackers@freebsd.org Wed Apr 17 18:09:55 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58A021574B9D for ; Wed, 17 Apr 2019 18:09:55 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2175A8C168 for ; Wed, 17 Apr 2019 18:09:54 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mail-ot1-x341.google.com with SMTP id s24so21492160otk.13 for ; Wed, 17 Apr 2019 11:09:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/0PKlSur5GU3OW7MhUxvxkhX3sT5fJ0GqTD7yGNDwDE=; b=pYm0tabMpyTfnxTgxrzF9McZqwiEVRzMRohlbhrW4QwOUq8BBkvje42QfKqL7gh+4q tPVz3m2fv92T03CC6segNDc5W62b9SQmnX3rW8ncKeDS0RkYCilBaGKUXMsIJgev9tHf rllryy6CGSWn7XJc90SFCJiPAf+085EJmcLH0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/0PKlSur5GU3OW7MhUxvxkhX3sT5fJ0GqTD7yGNDwDE=; b=OKjUOtQALuWb5xjAsjXx8VJVjGrFHpPjy+uvB6RJ95eH0OAMBOcx8LOMxcEkQLDl+i x7NJLctsXvlKJ6LqMGPB+a+XOndV0b9XNayQh7PAYi59arKUoyYUlE1vas5xaamkIV2h kBWCNVYD0AIWMDqy+iC1eGYYZU3QpoOnLupt6fc8L8jpEjIo80JjEvZ/c34qneEVT8b4 asNWI6I5e1smhVpjRaZ0HMZd8DwbCDQbBJGsx6E/I0QSCAd3JE7OD6uvr2FiJhMQXgtK RlB7wzm7xI9/OisOJzSgJR075/1mVSY5PjYmiv+npSLVb7+VA1HWe+0it2dlJHvqEc7l tkew== X-Gm-Message-State: APjAAAVXg4QDXIYIs7Itm3FNwyNGivFfncT6+rj961asXRSgtIEypfBp v+BcnfL/kIXFFqxGIQmj4MAyvs4FoEo48Q== X-Google-Smtp-Source: APXvYqxrzQUFWYiOp8MASkG2o/eKuy7tWgKAG3CkD3OQhWL+C6ny6FUE88hOCpm+/ObtHtcKW7GvAg== X-Received: by 2002:a9d:3988:: with SMTP id y8mr57134769otb.231.1555524592680; Wed, 17 Apr 2019 11:09:52 -0700 (PDT) Received: from [10.10.10.235] ([66.196.5.190]) by smtp.gmail.com with ESMTPSA id h23sm25825272oic.10.2019.04.17.11.09.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 11:09:51 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: openvpn and system overhead From: Jim Thompson In-Reply-To: Date: Wed, 17 Apr 2019 13:09:50 -0500 Cc: Miroslav Lachman <000.fbsd@quip.cz>, Mark Millard via freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <94EA4F3F-4D78-4E08-9AF8-441B957A4749@netgate.com> References: <8648d069-2172-2c09-8e59-d66a8265a120@quip.cz> To: Wojciech Puchar X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 2175A8C168 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netgate.com header.s=google header.b=pYm0tabM; dmarc=pass (policy=none) header.from=netgate.com; spf=pass (mx1.freebsd.org: domain of jim@netgate.com designates 2607:f8b0:4864:20::341 as permitted sender) smtp.mailfrom=jim@netgate.com X-Spamd-Result: default: False [-4.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[netgate.com:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[netgate.com:+]; DMARC_POLICY_ALLOW(-0.50)[netgate.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,aspmx2.googlemail.com,aspmx3.googlemail.com,alt2.aspmx.l.google.com,aspmx4.googlemail.com]; IP_SCORE(-0.81)[ip: (1.27), ipnet: 2607:f8b0::/32(-3.04), asn: 15169(-2.22), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.97)[-0.968,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 18:09:55 -0000 > On Apr 17, 2019, at 10:54 AM, Wojciech Puchar = wrote: >=20 >=20 >=20 > On Wed, 17 Apr 2019, Miroslav Lachman wrote: >=20 >> Wojciech Puchar wrote on 2019/04/17 17:08: >>> i'm running openvpn server on Xeon E5 2620 server. >>> when receiving 100Mbit/s traffic over VPN it uses 20% of single = core. >>> At least 75% of it is system time. >>> Seems like 500Mbit/s is a max for a single openvpn process. >>> can anything be done about that to improve performance? >>=20 >> You can play with ciphers, AES-NI etc. >> https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux >>=20 >> Miroslav Lachman >>=20 >>=20 > again. it's system time mostly not user time. Yup. I=E2=80=99ve looked at this a bunch over the years for pfSense. The tun/tap device can be viewed as a simple Point-to-Point IP or = Ethernet device, which instead of receiving packets from a physical=20 media, receives them from user space program and instead of sending = packets via physical media sends them to the user space program.=20 Let's say that you configured IP on the tap0, then whenever the kernel = sends an IP packet to tap0, it is passed to the application (OpenVPN, = for example).=20 Open=10VPN encrypts, authenticates, and occasionally compresses this = packet, encapsulates it, and sends it to the other side over TCP or = (preferably) UDP. The application on the other side receives the packet, decompresses and = decrypts the data received and writes the packet to its TAP device, the = kernel on the other side handles the packet like it came from real = physical device. Each time you copy data from user to kernel or kernel to user space, you = also incur a context switch with all the associated overheads. Using a tun/tap device incurs an additional context switch in each = direction, as you=E2=80=99re basically running the program to send data = (say, =E2=80=98ping=E2=80=99 or =E2=80=99ssh=E2=80=99), and another = program is used to encrypt and encapsulate the packet before it leaves = the machine. The process is roughly the same on the other side. So = you get twice the copies, and twice the number of context switches. = Making things worse, the =E2=80=9CIP stack=E2=80=9D inside OpenVPN is = single-threaded, and processes one packet at a time, so all the = overheads accrue to each packet, rather than being amortized across = several packets. Net-net, openvpn won=E2=80=99t do close to 1Mpps. There is a = decent-enough write-up of recent actual benchmarking in a masters thesis = that compares IPsec, OpenVPN and Wireguard, on linux here: = https://www.net.in.tum.de/fileadmin/bibtex/publications/theses/2018-pudelk= o-vpn-performance.pdf Section 5.5 if you want to skip to the substance. Basically, with *no* = encryption overheads, OpenVPN still has a static overhead of around 8500 = cycles/packet on the setup they used (Xeon E5-2620 v4), which seems = quite similar to yours. Given all this, they show that OpenVPN enters = an overload condition at around 120Kpps. There is some hope if you really have to have a lower-overhead OpenVPN. = An OpenVPN session has two channels, multiplexed on the same connection. = One is a control channel, the other is a data channel. The control = channel and associated configuration code in OpenVPN is =E2=80=A6 = complex. It has close to 10 trillion configuration options, and any = re-write of this code would be a huge, huge undertaking. Nearly = unthinkable, really. The data channel, otoh, is relatively = straight-forward, especially if you don=E2=80=99t need all the crypto = options provided, and, instead, limit yourself to, say AES-GCM or = another AEAD (ChaCha20 / Poly1305) transform. (Here, if your CPU has = AES-NI or similar (e.g. ARMv8 has AES acceleration instructions) AES-GCM = will always be faster.) But, if you=E2=80=99re willing to limit yourself to one, or a few = transforms, it theory, it=E2=80=99s possible to make a specialized tun / = tap device such that the data channel is kept in-kernel, with = encryption/decryption and encapsulation/decapsulation of data packets = occurring in the kernel, but control packets passed up and down to/from = the associated user space process. A partial attempt of this idea (for linux) can be found here: = https://github.com/marywangran/OpenVPN-Linux-kernel it looks abandoned, = so maybe it didn=E2=80=99t pan out, or maybe the work just got = asymptotic. There is a bunch of work to get this right (keeping the openVPN user = process happy, counters up to date, etc), but, at the end of the day, = it=E2=80=99s all software. Netflix got enough of OpenSSL's AES-GCM = implementation into the kernel to run the transmit side. They didn=E2=80=99= t care about the receive side, and just let nginx deal with the = relatively light rx flows in their deployment, but it does show that = it=E2=80=99s possible with enough work. Even with all that work, It will probably never be as fast as a decent = IPsec implementation. Jim From owner-freebsd-hackers@freebsd.org Wed Apr 17 23:12:23 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01018157C282 for ; Wed, 17 Apr 2019 23:12:23 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A2D997C5A for ; Wed, 17 Apr 2019 23:12:11 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-hackers@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x3HNC48Z001038 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 18 Apr 2019 06:12:04 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: openvpn and system overhead To: Wojciech Puchar , freebsd-hackers@freebsd.org References: From: Eugene Grosbein Message-ID: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Date: Thu, 18 Apr 2019 06:11:57 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 8A2D997C5A X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-1.73 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.830,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.95)[-0.945,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.36)[-0.356,0]; IP_SCORE(0.00)[country: RU(0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 23:12:23 -0000 17.04.2019 22:08, Wojciech Puchar wrote: > i'm running openvpn server on Xeon E5 2620 server. > > when receiving 100Mbit/s traffic over VPN it uses 20% of single core. > At least 75% of it is system time. > > Seems like 500Mbit/s is a max for a single openvpn process. > > can anything be done about that to improve performance? Anyone concerning performance should stop using solutions processing payload traffic with userland daemon while still using common system network interfaces because of unavoidable and big overhead due to constant context switching from user land to kernel land and back. Be it openvpn or another userland daemon. You need either some netmap-based solution or kernel-side vpn like IPsec (maybe with l2tp). For me, IKE daemon plus net/mpd5 work just fine. mpd5 is userland daemon too, but it processes only signalling traffic like session establishment packets and then it setups kernel structures (netgraph nodes) so that payload traffic is processed in-kernel only. From owner-freebsd-hackers@freebsd.org Wed Apr 17 23:40:08 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53355157CC0F for ; Wed, 17 Apr 2019 23:40:08 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-oi1-x242.google.com (mail-oi1-x242.google.com [IPv6:2607:f8b0:4864:20::242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 461B869DF6 for ; Wed, 17 Apr 2019 23:40:07 +0000 (UTC) (envelope-from jim@netgate.com) Received: by mail-oi1-x242.google.com with SMTP id v7so176087oie.8 for ; Wed, 17 Apr 2019 16:40:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=RNxduuNZELwqIhgwh9acuNfU7pqfVygAKKgLE7cZCUc=; b=EgAMMAg6mt7GmrDhxu8jgBeqNdHW2Fy3MpW66F11LEJXHTOF2noiiFzI1QBRUWgi5M 2tJ8vqqm+gExskAoafwy+T1wIH/LQTafWdm+YXR2jVGwwHvWOolzbAEOXuMLkIvFNJB0 OyZg+pYsMOg1X2R6qqu3deDTedR5+O1bDlWlw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=RNxduuNZELwqIhgwh9acuNfU7pqfVygAKKgLE7cZCUc=; b=ZTfwoQTm+lKdSOOp0xEOhIOSxPhgLqo8yURzTmkpc+aOR0bVbdjONpqezKYMtRkt2D Z27XcgnccvJh/Kuo56KMcxfGIwyQPvMaCRkqnj9cSgRpStZrz4J9VuJVwoBavvlW8E0f wAmIHs3pC2PXwn53fCUwqWKhiXdy9ekdZv+GAuFG6i+WWlGGiJKk2qYcTsv7iZReYRnD Ly5QOearwn3/qL4I7/1GugaZfCgMqFoPOMtb6MYU3iBczNUkjbW79HeZWVHYplsbYrGx ZfymRWqho2ItvCg+LPhKOc4vdxAKkCRb26JBYheVd8sGp2X8/pJAUpxqXyzvyjReIcEp tAZw== X-Gm-Message-State: APjAAAUT2XWENb7+yI8ToYOh9n3YOFrWZFHYkSsQ0iTudqLL9V6WskgI jytud3V2Wa6dv4BTeitPIyUmYiB9sKtRqw== X-Google-Smtp-Source: APXvYqzuVfYjiFjxzu9zYvCUgOle9qVsDSujHjDcmx6D8pYYkfLZ9SJ6IgtfavGba9Re0r/zpAuLAg== X-Received: by 2002:aca:bb88:: with SMTP id l130mr100981oif.124.1555544406225; Wed, 17 Apr 2019 16:40:06 -0700 (PDT) Received: from [10.10.10.235] ([66.196.5.190]) by smtp.gmail.com with ESMTPSA id l49sm147278otc.19.2019.04.17.16.40.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 16:40:05 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: openvpn and system overhead From: Jim Thompson In-Reply-To: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Date: Wed, 17 Apr 2019 18:40:03 -0500 Cc: Wojciech Puchar , freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> To: Eugene Grosbein X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 461B869DF6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netgate.com header.s=google header.b=EgAMMAg6; dmarc=pass (policy=none) header.from=netgate.com; spf=pass (mx1.freebsd.org: domain of jim@netgate.com designates 2607:f8b0:4864:20::242 as permitted sender) smtp.mailfrom=jim@netgate.com X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[netgate.com:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; DKIM_TRACE(0.00)[netgate.com:+]; DMARC_POLICY_ALLOW(-0.50)[netgate.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.58)[ip: (2.44), ipnet: 2607:f8b0::/32(-3.05), asn: 15169(-2.23), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.98)[-0.976,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 23:40:08 -0000 > On Apr 17, 2019, at 6:11 PM, Eugene Grosbein = wrote: >=20 > 17.04.2019 22:08, Wojciech Puchar wrote: >=20 >> i'm running openvpn server on Xeon E5 2620 server. >>=20 >> when receiving 100Mbit/s traffic over VPN it uses 20% of single core. >> At least 75% of it is system time. >>=20 >> Seems like 500Mbit/s is a max for a single openvpn process. >>=20 >> can anything be done about that to improve performance? >=20 > Anyone concerning performance should stop using solutions processing = payload traffic > with userland daemon while still using common system network = interfaces > because of unavoidable and big overhead due to constant context = switching > from user land to kernel land and back. Be it openvpn or another = userland daemon. >=20 > You need either some netmap-based solution or kernel-side vpn like = IPsec (maybe with l2tp). > For me, IKE daemon plus net/mpd5 work just fine. mpd5 is userland = daemon too, > but it processes only signalling traffic like session establishment = packets > and then it setups kernel structures (netgraph nodes) so that payload = traffic is processed in-kernel only. Addendum to previous message to freebsd-hackers: We have (also) considered a netmap-enhanced (enabled?) OpenVPN. You = still have the problem that the =E2=80=98stack=E2=80=99 inside OpenVPN = is single-threaded/single packet at a time. Also, you=E2=80=99ll need to multiplex > 1 instance of OpenVPN, maybe = using the programability of VALE (aka =E2=80=98mswitch=E2=80=99). Linaro=E2=80=99s Open Data Plane project did a shim that would talk via = shared memory (ODP queues) between OpenVPN and ODP. ODP queues = aren=E2=80=99t too different from netmap rings,=20 so a PoC based on this code would be straight-forward. "This demo was = not intend to improve performance, it was only for basic functionality." https://github.com/repu1sion/odp_apps/tree/master/openvpn Jim= From owner-freebsd-hackers@freebsd.org Thu Apr 18 00:02:17 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2B9E157D5A2 for ; Thu, 18 Apr 2019 00:02:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 66CA76AA15 for ; Thu, 18 Apr 2019 00:02:16 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-hackers@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x3I02EYs001451 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 18 Apr 2019 07:02:14 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: openvpn and system overhead To: Wojciech Puchar , freebsd-hackers@freebsd.org References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> From: Eugene Grosbein Message-ID: <2a380079-1f72-5f1e-30ff-ddd808d2862b@grosbein.net> Date: Thu, 18 Apr 2019 07:02:07 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 66CA76AA15 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-1.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.825,0]; MX_INVALID(0.50)[cached]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.94)[-0.936,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.38)[-0.377,0]; IP_SCORE(0.00)[country: RU(0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 00:02:17 -0000 18.04.2019 6:11, Eugene Grosbein wrote: > 17.04.2019 22:08, Wojciech Puchar wrote: > >> i'm running openvpn server on Xeon E5 2620 server. >> >> when receiving 100Mbit/s traffic over VPN it uses 20% of single core. >> At least 75% of it is system time. >> >> Seems like 500Mbit/s is a max for a single openvpn process. >> >> can anything be done about that to improve performance? > > Anyone concerning performance should stop using solutions processing payload traffic > with userland daemon while still using common system network interfaces > because of unavoidable and big overhead due to constant context switching > from user land to kernel land and back. Be it openvpn or another userland daemon. > > You need either some netmap-based solution or kernel-side vpn like IPsec (maybe with l2tp). > For me, IKE daemon plus net/mpd5 work just fine. mpd5 is userland daemon too, > but it processes only signalling traffic like session establishment packets > and then it setups kernel structures (netgraph nodes) so that payload traffic is processed in-kernel only. Just to clarify: mpd5 still uses common networking stack and system interfaces ng0, ng1 etc. for p2p-tunnels but it does not process tunneled traffic leaving the job to the kernel. Back in 2011 I did some measures of my production mpd5 installation serving real PPPoE users with FreeBSD 8 and mpd5, SuperMicro SuperServer 5016T-MTFB, Intel Xeon CPU E5507 @ 2.27GHz (4 cores) and four 1GB Intel NICs (two on-board 82574L em0/em1 and single two-ports 82576 card igb0/igb1). It processes 1812 simultaneous sessions when each CPU core had 90% load (2% user time and 88% system time approx.) forwarding 1184.9Mbit/s to users plus 830.9Mbit/s from users (2015.8Mbit/s in total) and dealing with 136.2Kpps in+120.3Kpps out for lagg0 (em0+em1, IP-only uplink) and 139.0Kpps in+154.3Kpps out for lagg1 (igb0+igb1, downlink with PPPoE/vlans and no IP). It processed 102K interrupts per second then. There was no encryption involved and numbers basically describe packet forwarding/filtering/shaping ability of the kernel those days. From owner-freebsd-hackers@freebsd.org Thu Apr 18 00:53:58 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0953157E683 for ; Thu, 18 Apr 2019 00:53:57 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: from mail-oi1-x230.google.com (mail-oi1-x230.google.com [IPv6:2607:f8b0:4864:20::230]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1357B6C2AC for ; Thu, 18 Apr 2019 00:53:57 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: by mail-oi1-x230.google.com with SMTP id i21so271837oib.11 for ; Wed, 17 Apr 2019 17:53:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FdqX8mo/HemeNbQM5qfLcsbupef1DM0BxaCV43e3z8w=; b=Yey1FK7bSdiv21nCtLinIB05tb4f0dxp9dmXaqfQavHRMOD05SWTn7XZn3VS98diz4 eFdxymIf1mgnBocKGWnDw+MhQ/7XFV7UAKrSI5xwCdA9t907kRggCJNnZX2/laCeARE1 yVvt5dmDGWxJ0dIocngi/cqExtffnMbXxDNfhWk6EMQxq/YuZw8//EXDj88CcWrL5ZpP M+CUFgNEA3rUDrJ8AulxLgoHSe/li0SRKsWHKewywDyDiKE4i/A0cEDZrKbmzeLH6pG/ lkw2WHaHlGoB6AwYsCFVLQeG7voq6DXrMWzNgXv4NB4rTpKnjNSrmFQ/MqXD36bGrNJH OgvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FdqX8mo/HemeNbQM5qfLcsbupef1DM0BxaCV43e3z8w=; b=daUlOagHoBIZS6Kh/tdbEe927tUARrgCCmGzQ3jDspeS4e3ksrunE07tpFptz05kxF FC5bD0iDhAXwpaqd+SYNXhN5sUBJ8Q40JUx7oZpbGecjI5zVVQAX/IOTdWppDl8AObv+ MGtnAkJ2sF47JEgFNA1EWJcvVD7xMoxxj4m98JglWBEunSEiKCPQoQOR8pL53RIZt8Nx IQn0gOLpd6pCIAUImWQNYgbovaYzvw9jgbN51WE+WRcNgdzCjkzJG1S/byvqfWxevcWc W+BSAbvMkkAiCbNXXrmjI6rvZMzTECaayDM0VQkG7KZx5jBPisDacORIVzIrOuGoTZuF cKdg== X-Gm-Message-State: APjAAAU/LouSjD+uT+VgZ4+477m5lRiArxMsVV0P91fRufUNuB/XGOuI EZBKokHkD1MTM30BKTRgaBKBmYkESeQN3m9mDvqNrMzS X-Google-Smtp-Source: APXvYqwlg2JxKSTfL3b8yZAoUhUr5Ufg2w4gfdhCwoIPxhnn0gL8jFXLbDiIYGMFRvwruWniNEiIu8Iwn8oGidGnSbI= X-Received: by 2002:aca:aa57:: with SMTP id t84mr258350oie.149.1555548835784; Wed, 17 Apr 2019 17:53:55 -0700 (PDT) MIME-Version: 1.0 From: Lee D Date: Wed, 17 Apr 2019 20:52:50 -0400 Message-ID: Subject: What code loads kernel modules at boot? To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 1357B6C2AC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Yey1FK7b; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of embaudarm@gmail.com designates 2607:f8b0:4864:20::230 as permitted sender) smtp.mailfrom=embaudarm@gmail.com X-Spamd-Result: default: False [-5.84 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[0.3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.88)[ip: (-9.06), ipnet: 2607:f8b0::/32(-3.05), asn: 15169(-2.23), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 00:53:58 -0000 A couple of years ago I wrote a FreeBSD bootloader for Zynq (Arm32). It's been working well, but now I need to add the ability to load kernel modules at boot. This is for 11.0.1 (updating to 12 soon, hopefully). Can anyone point me to the code that actually loads a kernel module for arm? I got lost reading the source in /src/sys/boot/common, and can't quite figure out what routine is actually called. I assume I need to parse the sections out of the .ko file and place them carefully in memory, like I do with the kernel image. Also, if you're feeling loquacious, where do I put the darn thing and how do I tell the kernel how to find it (part of the MODINFO stuff I assume)? This is all in the context of loading custom real time clock and I2C drivers so they are available at boot time. Thanks, Lee From owner-freebsd-hackers@freebsd.org Thu Apr 18 03:04:04 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4AB21584D57 for ; Thu, 18 Apr 2019 03:04:03 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2m.ore.mailhop.org (outbound2m.ore.mailhop.org [54.149.155.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34FDC72416 for ; Thu, 18 Apr 2019 03:04:03 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555556641; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=MGAO7FBS0BDvaTk9uTPu+cSiWnG36g9qfpyvZY7AUhsBuMTPR9H6wTzjYHDE4DorJSHrHwkq59MGI 7NjsI1H/WjZPGRnK2DqoWdQFtf7WifhXnq+1Oe9i/gHHcovc502ENe2iXDr5ruMwmeoUI76RbL6mB7 D332EcBydDnZosMkh+PDh+ycmND9SgexilXCqk9YnXz9bzFlhS85pHn9AIB1pfVxMOFkhJSiZke9BD VW938uSkSodTFJgOFACG4nErWKlYNIjuucp6d7WzXuTAZ1U+jumwh0bxIY7esdkMcLhij1ovPLy+en H1Vlcy9dQ4Q/iAAG7ivt8VJ92XaEfVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:dkim-signature:from; bh=9XgXOcvFK2ER3w9uq6ATa03Smq7v1zVk6LMLFGja4V4=; b=ZySKjHgXA/GsAlYaVfzkcWeN6aWG3aRAKT+mihs5wmP6WZTDwvpO0X7r0XzB3rZ7vht873epYvdO0 xzcqxx2PrTc9t7TwEXWbx+kFqfELnCGrfHuF3d4cyfmDg/VX9/ukipQ2soMp9A7wweCycdSJajpnre maR1gG0sVpzPTfDi9yHWM4E9Dky+xgtAaxBu8O5wHAyNzvQAcHfrZ6mOQ80vIJvvFP7lYhCRJKS0aS gBP/g03Py1gZKCFbCROdJCVa+66peJ9Q42bbQSM5qVNA8Muhxcw4ukEnt86PSbQ+lGaKVsKqWmmpyw +yPL7IP78DyTwK5+kn9yyPMeWsvZaEw== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:from; bh=9XgXOcvFK2ER3w9uq6ATa03Smq7v1zVk6LMLFGja4V4=; b=Mg6iQrIm94o1oN8r/W4vJaacDxxKLCiGteP5bM6Kt5PQHLlIDWIKHdo5BKHyjYSxtIEaBZL1p4ycr SnVwq6LSQO8MCXM0La0uVnQ5grsxtfDtZ4KBYHFv0FUTvTgOgDCY43XSbG66Y6r9sVBBPxEUfMKPHR 54BzY7GfZMc73aN8BelkM0qTGvQsZ4QWSkxQhPVVg9g1MEl1gwga1WbGDabqWSOih0vJXrX1Sy/Tl5 Ulx89AI6jLLvxWJjmzDiBnJIEpcSJ1K4sC2U0B7QY3Y1/eBkEQGryyojDMy8YygXro/Nyid2NDatM0 oXtAzsqHx2SOxKSMrYUUkdWaveOZG/Q== X-MHO-RoutePath: aGlwcGll X-MHO-User: 9cb10092-6186-11e9-919f-112c64a8cf29 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 9cb10092-6186-11e9-919f-112c64a8cf29; Thu, 18 Apr 2019 03:04:00 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3I33xSk049129; Wed, 17 Apr 2019 21:03:59 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> Subject: Re: What code loads kernel modules at boot? From: Ian Lepore To: Lee D , FreeBSD Hackers Date: Wed, 17 Apr 2019 21:03:59 -0600 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 34FDC72416 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 03:04:04 -0000 On Wed, 2019-04-17 at 20:52 -0400, Lee D wrote: > A couple of years ago I wrote a FreeBSD bootloader for Zynq (Arm32). > It's been working well, but now I need to add the ability to load > kernel modules at boot. This is for 11.0.1 (updating to 12 soon, > hopefully). > > Can anyone point me to the code that actually loads a kernel module > for arm? I got lost reading the source in /src/sys/boot/common, and > can't quite figure out what routine is actually called. > > I assume I need to parse the sections out of the .ko file and place > them carefully in memory, like I do with the kernel image. > > Also, if you're feeling loquacious, where do I put the darn thing and > how do I tell the kernel how to find it (part of the MODINFO stuff I > assume)? > > This is all in the context of loading custom real time clock and I2C > drivers so they are available at boot time. > > Thanks, > > The bulk of the module-loading code (the arch-independent part of it) is in src/stand/common/module.c. There is also archsw.arch_loadaddr, which figures out where to put the modules (handling arch-specific things like alignment requirements). For some reason, I thought Zynq used u-boot. That would allow using either ubldr or the arm uefi loader (depending on how old the u-boot is); those are just flavors of loader(8) that would get you module handling and all the other loader goodness. -- Ian From owner-freebsd-hackers@freebsd.org Thu Apr 18 03:45:04 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECC171586434 for ; Thu, 18 Apr 2019 03:45:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E4EA073CC8 for ; Thu, 18 Apr 2019 03:45:02 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x843.google.com with SMTP id x12so749865qts.7 for ; Wed, 17 Apr 2019 20:45:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/RWuQ2vuaswiR304noCZ5QzQsIdpN+4Qi3vl8C+6LX4=; b=NL9tK4M3/2kplgKAamHqKPkNOEXvPuDRXkQiHps+GAEfodDNecelDTBHdiaayOAxvV evf1F8oCD+ZJuQjgU+w79udleinNZM1CXFnFxQYkpSH727D/xLKedcbjc4HRbzjO1m4b pf/G7RkdFnsaM6YrVW4yrfAiWLtl4IpJjJzXgTluyS2FGALYC28QPzBz6oFAozQHtoz3 ehZzBHjnIUc6Ne0j36u652FPLAicvU1FkvU1rtypNKgygqgHk/4oTbTFz+zglv4CnY+w bidcrqVNoyTdzsXJdC2MSz9hCYyfYPZgSi5Hp8kcoV3B3vD3UnjLEnzzL1Hxqht67o1i h5GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/RWuQ2vuaswiR304noCZ5QzQsIdpN+4Qi3vl8C+6LX4=; b=LM/ML+zFFIq41nRs0zwqs+qxHZwgco4eFKHy1McgFfIVkB/qnPLUpZCEVdIok02IbL +YoUzvLXNWiAXBfgOQ1LyLlYwiDGngbtXGTYPdG2M4RcFEJXYh2eAVXcWAU1roRCgOTr mJF7nNQCbU8u+3gYv8BuPXalHtuN5MFyundYetPB3lk5AdeKT9hF/Ryc9mExlkUIKL8Q LRTvCAsJUgmEDxXLoGgdN4FZeYjPSrPVdIgln3Y1FtkoIPBbk1SgYsJlY51EFZAbiF+s TmmYI5Peh7dOEdwd82Tml+h5luvMAHIvamZ638hJ7mX0uOTb3hf99UxefZlS5BHCC7Lr NVZQ== X-Gm-Message-State: APjAAAXoSh6tOWQL+EvqSUATmyDi/k+zmRajGKCTwxo/zl/9r3/YWJDk B92345gO+7rfyECa51GT2i0Nfj/17ZB830OodEeJ7w== X-Google-Smtp-Source: APXvYqy9qC5SQara+pZULXsLnJnyw6WQQAzzRw+rqfpj2+cV/bBOYwL1fKZ3FEZ9k0xP0G1ZZYb5N60vhiqnWt4GOSs= X-Received: by 2002:a0c:d2fc:: with SMTP id x57mr74830327qvh.214.1555559102299; Wed, 17 Apr 2019 20:45:02 -0700 (PDT) MIME-Version: 1.0 References: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> In-Reply-To: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> From: Warner Losh Date: Wed, 17 Apr 2019 21:44:51 -0600 Message-ID: Subject: Re: What code loads kernel modules at boot? To: Ian Lepore Cc: Lee D , FreeBSD Hackers X-Rspamd-Queue-Id: E4EA073CC8 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=NL9tK4M3 X-Spamd-Result: default: False [-2.42 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.71)[ip: (1.81), ipnet: 2607:f8b0::/32(-3.05), asn: 15169(-2.23), country: US(-0.06)]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; NEURAL_HAM_LONG(-1.00)[-0.996,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[3.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.71)[-0.709,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 03:45:04 -0000 On Wed, Apr 17, 2019 at 9:07 PM Ian Lepore wrote: > On Wed, 2019-04-17 at 20:52 -0400, Lee D wrote: > > A couple of years ago I wrote a FreeBSD bootloader for Zynq (Arm32). > > It's been working well, but now I need to add the ability to load > > kernel modules at boot. This is for 11.0.1 (updating to 12 soon, > > hopefully). > > > > Can anyone point me to the code that actually loads a kernel module > > for arm? I got lost reading the source in /src/sys/boot/common, and > > can't quite figure out what routine is actually called. > > > > I assume I need to parse the sections out of the .ko file and place > > them carefully in memory, like I do with the kernel image. > > > > Also, if you're feeling loquacious, where do I put the darn thing and > > how do I tell the kernel how to find it (part of the MODINFO stuff I > > assume)? > > > > This is all in the context of loading custom real time clock and I2C > > drivers so they are available at boot time. > > > > Thanks, > > > > > > The bulk of the module-loading code (the arch-independent part of it) > is in src/stand/common/module.c. There is also archsw.arch_loadaddr, > which figures out where to put the modules (handling arch-specific > things like alignment requirements). > That code gets called from the command line commands, as well as indirectly in the .conf file parsing each of the interpretive languages have. > For some reason, I thought Zynq used u-boot. That would allow using > either ubldr or the arm uefi loader (depending on how old the u-boot > is); those are just flavors of loader(8) that would get you module > handling and all the other loader goodness. > Yea, if the loader that he's written loads /boot/loader, he doesn't need to do anything. if it loads the kernel and modules, he'll need to do what the code in src/stand/common/module does in terms of laying out memory and passing the proper meta-data to the kernel. Warner From owner-freebsd-hackers@freebsd.org Thu Apr 18 03:51:13 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BF4F1586636 for ; Thu, 18 Apr 2019 03:51:13 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound3d.ore.mailhop.org (outbound3d.ore.mailhop.org [54.186.57.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D5C7743E0 for ; Thu, 18 Apr 2019 03:51:12 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555559471; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=nu1ycIaJXMNJbSm/oL9o0ZlZMEh5TyVXJgiPVfXBm/kyHxjrCKX6zfXWN9Qo8fBhRaa4d4GqFzOIc Zu4g7NqPu1TxjH1/f3pTgoxDs6xuiRBkYEm1tybVFpi5oMwpINQotZ5vYl3/PgvFIYfmgtwEMHgM+d +xeR8/nBw0vPB3JCJRSyuSsAJ2N70mo4ByRJM7z9TTqV+aFDF5u2fgPRyuucQrNonewyxst54Z6zbp E/zQ7fXwp/Jp8GQkP8ffm/BVU47+U2ZvrjIo2w2RzwNvzod6C9tgPP43RdkC6NUQvDPEO31wNqAsnW yhpbmfpF/QgfDwKZo5TW4uqxacNjm8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=t/Riosij/44oO9Ps2jyWO0QeuCNVGpggXpx2ZR1ysGQ=; b=DSDMSL8FdZUKRfHVKXlv3XBqLN7S4FXlZ03CAfAw1hZD3WqVO1AuUi6dlNWv9a8FolP0bfkZq+/8S Lrm2Qaf95TokTcCGYux4LDo3qU6cykRivlyS9Qj3FyNxZAgqZ3Ox6QIL8PpihKNcZDsWhTzmT9akc8 sCFgy0GloHsVRf0O9MLPa1GuB5w8oC4ByHKpeLJauzsBengmuZaIKWfJ6VkbwyHA5O/oT98EtWWihm NWUR5HuB4H8K4/Gm9N+CBtWimcfooZVNngv33AJQhsy0QfwZYP3C5ZViWSuoKIFItXHrPQRoQ2TVCM qB/3WZ9+g7Pu9vswNp9X7uLnIw0KSmw== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=t/Riosij/44oO9Ps2jyWO0QeuCNVGpggXpx2ZR1ysGQ=; b=DXz07/4jrLP/P/cV+oBMuDWDs/prabqtGRN5Z3e0QkJ0vcYaDLQ84aP2Htvc5jh4ulPGnzE4koLQA q6kXexbzbTnkdzdqRitEon3Cyo75XMEVid9R1fVRD/hPgMH38Kig+jMOrI3LVzy+nkA+CgLJlND462 u6Ml5HyTyijvCkj5Ab40mb/4G7W5OL2XlWCglknM4nlVh1+ngIb7VlRhVO2cO38WQHPbr/v4Dp0oV8 e9TQossYgS0Io1OigHg4h5EaHuVqop8kva4tOvUlpB08rbx7Ihp3BYDHnQI+BxyD1I7lMgCCv+EY5o YVkzlZemYYTqZTLX1ohpr+ZHwJj3c2A== X-MHO-RoutePath: aGlwcGll X-MHO-User: 332cd770-618d-11e9-990e-673a89bc4518 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id 332cd770-618d-11e9-990e-673a89bc4518; Thu, 18 Apr 2019 03:51:10 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3I3p8KI049227; Wed, 17 Apr 2019 21:51:08 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <9e1fd08ab9f7c9d82a55186e95ff4d910c62dcc0.camel@freebsd.org> Subject: Re: What code loads kernel modules at boot? From: Ian Lepore To: Warner Losh Cc: FreeBSD Hackers , Lee D Date: Wed, 17 Apr 2019 21:51:08 -0600 In-Reply-To: References: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 7D5C7743E0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; ASN(0.00)[asn:16509, ipnet:54.186.0.0/15, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 03:51:13 -0000 On Wed, 2019-04-17 at 21:44 -0600, Warner Losh wrote: > On Wed, Apr 17, 2019 at 9:07 PM Ian Lepore wrote: > > > On Wed, 2019-04-17 at 20:52 -0400, Lee D wrote: > > > A couple of years ago I wrote a FreeBSD bootloader for Zynq > > > (Arm32). > > > It's been working well, but now I need to add the ability to load > > > kernel modules at boot. This is for 11.0.1 (updating to 12 soon, > > > hopefully). > > > > > > Can anyone point me to the code that actually loads a kernel > > > module > > > for arm? I got lost reading the source in /src/sys/boot/common, > > > and > > > can't quite figure out what routine is actually called. > > > > > > I assume I need to parse the sections out of the .ko file and > > > place > > > them carefully in memory, like I do with the kernel image. > > > > > > Also, if you're feeling loquacious, where do I put the darn thing > > > and > > > how do I tell the kernel how to find it (part of the MODINFO > > > stuff I > > > assume)? > > > > > > This is all in the context of loading custom real time clock and > > > I2C > > > drivers so they are available at boot time. > > > > > > Thanks, > > > > > > > > > > The bulk of the module-loading code (the arch-independent part of > > it) > > is in src/stand/common/module.c. There is also > > archsw.arch_loadaddr, > > which figures out where to put the modules (handling arch-specific > > things like alignment requirements). > > > > That code gets called from the command line commands, as well as > indirectly > in the .conf file parsing each of the interpretive languages have. > > > > For some reason, I thought Zynq used u-boot. That would allow > > using > > either ubldr or the arm uefi loader (depending on how old the u- > > boot > > is); those are just flavors of loader(8) that would get you module > > handling and all the other loader goodness. > > > > Yea, if the loader that he's written loads /boot/loader, he doesn't need to > do anything. if it loads the kernel and modules, he'll need to do what the > code in src/stand/common/module does in terms of laying out memory and > passing the proper meta-data to the kernel. > > It would have to do more than just load /boot/loader (or ubldr)... it would have to provide some sort of bios-like services for loader to do I/O. I almost mentioned that in my original post... if you've got a loader that can do disk I/O, it wouldn't be too hard to make it provide the same API services that u-boot does, then ubldr would work. What you need to implement for the api isn't much more than some simple disk read/write stuff, console read/write, and env var support (which could be mostly stubbed out). -- Ian From owner-freebsd-hackers@freebsd.org Fri Apr 19 04:36:31 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D91A11585168 for ; Fri, 19 Apr 2019 04:36:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-8.consmr.mail.gq1.yahoo.com (sonic308-8.consmr.mail.gq1.yahoo.com [98.137.68.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84C4C8D65C for ; Fri, 19 Apr 2019 04:36:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: zifi6gMVM1maUKrXsjrazmqUDrou5z2vRtV7EQuyIMLvYXmqCPKykO6i11jAByj mFTo9VwqsmxbBS7vxcr33WTGlxP3J7xXOVEWlEQMXITRudWlyV_04qBa0feY.fBgn3rk27lImQcv YgVQqngEwkN2jeHwIp.V937XqSN9dRKAvw2UQ59fUgp5ssCdE0q7fxp33DmjVjgZpbL0BgV66.7C KznQ6FrQjBhBdpA1tutj4_wKYOQKnN3GRlNeHJl7a93LUEPw1_v2x21.0FMxvqYd0ocpxf0dx8m8 vw.OkUt1qRYl5KqSG8betcNMyDBZO4KfFxgbeD4bHNwoXA3fGp16FUgc8s0fBuRMgtb6XSGI_qjN 1b3BHq.M5AV.tEcXZ5kgRnpv08fRKkfDkPfA1HoBcjmoYNtpqKJNDEQYqvCCF9ui.TZLY4F8RDwX _MxuBC1ROihnwo4Aczr06WlfAP46vEh7Z37SvVbG2nHhcP8NSElQcWh9_q_3js.z3vtVr8JOTD3r APuzNwY6kytxXCup6cHJuri.gGRuPK3ehTLFToLn.QMHKw5YN4rRSEMk3X9EAH.87BJQS_hi9P7P bhfwek6EhwZjJLSQ7tFKm6tm_LeyO.pAU2yS6M8.HJPDExkcEF3zRCbuxvE7ey0muh7FeG1GZNoj t9Sh3_B3DdigsfaStvTlFNTg_DRIOf5WilcYH_n_oI0G5rHrPLOsfdcx67RZHHQ7xQpysnzoFeSR 2u6kGkk9GLoQexA4tjEKKTkym.7WL5ahIcQN.6VwjgSsLVbJGEIusYmzTDauNrWdJXuGYLQnBh1Y ZUkQ_KXm.H2qTSE9Zho4P23uK3EzV7h7tDNRrDrT4pfS2McTEHoehIGKMiczVFLvYtpnIRnybjfs xvA.gIxA2jMsO4DfyeILaGDjSZ9SSKOcmINMFPLbl9DSWvKdbudnNasZG9whV_6Z6eZvF_9vME_g QRkxIdHTlF88X2.MQPTkWmbEBy.Ql9u.6FwGSle0arqpWFMOsbeNVO7QxQFs3VMmPV3nTuYI3gzQ 4FK6CamSjLHK3AVIicx6DM72XVKLE7zSC.fRNEe3Bke67rzFVAxZWbTJFGUln6hSoP7_3vP0COPk mO7tFef8t5DXya.F4g_YbizkPUZfPFFob Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Fri, 19 Apr 2019 04:36:22 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp412.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID cb8240f959e44ad1296b1c03198b64c7; Fri, 19 Apr 2019 04:36:20 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: powerpc64 or 32-bit power context: FreeBSD lwsync use vs. th->th_generation handling (and related th-> fields) looks broken to me Message-Id: <50CFD7F1-6892-4375-967B-4713517C2520@yahoo.com> Date: Thu, 18 Apr 2019 21:36:19 -0700 Cc: Bruce Evans , Konstantin Belousov To: FreeBSD PowerPC ML , freebsd-hackers Hackers X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 84C4C8D65C X-Spamd-Bar: + X-Spamd-Result: default: False [1.59 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.97)[0.972,0]; NEURAL_HAM_LONG(-0.31)[-0.314,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.42)[0.415,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[32.68.137.98.list.dnswl.org : 127.0.5.0]; IP_SCORE(1.03)[ip: (3.52), ipnet: 98.137.64.0/21(0.93), asn: 36647(0.74), country: US(-0.06)]; FREEMAIL_CC(0.00)[optusnet.com.au] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 04:36:31 -0000 First I review below lwsync behavior. It is based on a = comparison/contrast paper for the powerpc vs. arm memory models. It sets context for later material specific to powerpc64 or 32-bit powerpc FreeBSD. "For a write before a read, separated by a lwsync, the barrier will = ensure that the write is committed before the read is satisfied but lets the read be satisfied = before the write has been propagated to any other thread." (By contrast, sync, guarantees that the write has propagated to all = threads before the read in question is satisfied, the read having been separated from the = write by the sync.) Another wording in case it helps (from the same paper): "The POWER lwsync does *not* ensure that writes before the barrier have = propagated to any other thread before sequent actions, though it does keep writes = before and after an lwsync in order as far as [each thread is] concerned". (Original used = plural form: "all threads are". I tired to avoid any potential implication of cross = (hardware) "thread" ordering constraints for seeing the updates when lwsync is = used.) Next I note FreeBSD powerpc64 and 32-bit powerpc details that happen to involve lwsync, though lwsync is not the only issue: atomic_store_rel_int(&th->th_generation, ogen); and: gen =3D atomic_load_acq_int(&th->th_generation); with: static __inline void \ atomic_store_rel_##TYPE(volatile u_##TYPE *p, u_##TYPE v) \ { \ \ powerpc_lwsync(); \ *p =3D v; \ } and: static __inline u_##TYPE \ atomic_load_acq_##TYPE(volatile u_##TYPE *p) \ { \ u_##TYPE v; \ \ v =3D *p; \ powerpc_lwsync(); \ return (v); \ } \ also: static __inline void atomic_thread_fence_acq(void) { powerpc_lwsync(); } First I list a simpler-than-full-context example to try to make things clearer . . . Here is a sequence, listing in an overall time order, omitting other activity, despite the distinct cpus, (N!=3DM): (Presume th->th_generation=3D=3Dogen-1 initially, then:) cpu N: atomic_store_rel_int(&th->th_generation, ogen); (same th value as for cpu M below) cpu M: gen =3D atomic_load_acq_int(&th->th_generation); For the above sequence: There is no barrier between the store and the later load at all. This is important below. So, if I have that much right . . . Now for more actual "load side" context: (Presume, for simplicity, that there is only one=20 timehands instance instead of 2 or more timehands. So th does not vary below and is the same on both cpu's in the later example sequence of activity.) do { th =3D timehands; gen =3D atomic_load_acq_int(&th->th_generation); *bt =3D th->th_offset; bintime_addx(bt, th->th_scale * tc_delta(th)); atomic_thread_fence_acq(); } while (gen =3D=3D 0 || gen !=3D th->th_generation); For simplicity of referring to things: I again show a specific sequence in time. I only show the &th->th_generation activity from cpu N, again for simplicity. (Presume timehands->th_generation=3D=3Dogen-1 initially and that M!=3DN:) cpu M: th =3D timehands; (Could be after the "cpu N" lines.) cpu N: atomic_store_rel_int(&th->th_generation, ogen); (same th value as for cpu M) cpu M: gen =3D atomic_load_acq_int(&th->th_generation); cpu M: *bt =3D th->th_offset; cpu M: bintime_addx(bt, th->th_scale * tc_delta(th)); cpu M: atomic_thread_fence_acq(); cpu M: gen !=3D th->th_generation (evaluated to false or to true) So here: A) gen ends up with: gen=3D=3Dogen-1 || gen=3D=3Dogen (either is allowed because of the lack of any barrier between the store and the involved load). B) When gen=3D=3Dogen: there was no barrier before the assignment to gen to guarantee other th-> field-value staging relationships. C) When gen=3D=3Dogen: gen!=3Dth->th_generation false does not guarantee the *bt=3D. . . and bintime_addx(. . .) activities were based on a coherent set of th-> field-values. If I'm correct about (C) then the likes of the binuptime and sbinuptime implementations appear to be broken on powerpc64 and 32-bit powerpc unless there are extra guarantees always present. So have I found at least a powerpc64/32-bit-powerpc FreeBSD implementation problem? Note: While I'm still testing, I've seen problems on the two 970MP based 2-socket/2-cores-each G5 PowerMac11,2's that I've so far not seen on three 2-socket/1-core-each PowerMacs, two such 7455 G4 PowerMac3,6's and one such 970 G5 PowerMac7,2. The two PowerMac11,2's are far more tested at this point. But proving that any test-failure is specifically because of (C) is problematical. Note: arm apparently has no equivalent of lwsync, just of sync (aka. hwsync and sync 0). If I understand correctly, PowerPC/Power has the weakest memory model of the modern tier-1/tier-2 architectures and, so, they might be broken for memory model handling when everything else is working. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Fri Apr 19 05:17:54 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D843C1585B2A for ; Fri, 19 Apr 2019 05:17:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-28.consmr.mail.bf2.yahoo.com (sonic317-28.consmr.mail.bf2.yahoo.com [74.6.129.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C8E808E3E0 for ; Fri, 19 Apr 2019 05:17:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ONTDAVYVM1mCwjts_5U0JptmTjyOc4J_qbeARDcUcxcO1jZ7zvOlGDXgcuHQl2q iyzkuZBgM1qm0pV.XLKsWLreRKw6hJvtWVRqGqMvvuBTclku3hngm.p6C58JXD4hUT4q5Rtzl99t 6qSOkLZPoxhGSz0wyAc0NlyXwLA1SiHiUZBrLLUGbd4XU4f8Kt8FWDeKx0aVNPkuvPXiMwLr..oj X_7ZmCDnCkAIDChRulJfb0MWJQg0fmg6wP5xxzMje4dfHloAor5KaqlJWk_dKfvRRukF7_Bntg04 CyNmO.UKwdPasuFD3C43Vlti4hBW8EYgLYlU1e5Idlyu4BYkUV80QwJrIBKRwdnDU_FaaTUJzzed 5HWT2UadxgAvUiEiP70yddNsSMDinQI3bdB7BZnDGKs__LKmHcEgsGBjjsJ0BuXYH4_3x0jMEB1w S96lLpwWxbm2mpoG7kf_3KtDfJ00VzcXPHVPMdv33og_uKBhy2xfEQXXF949KC68uBkMVflPdQ4v lepOlP2hhgWttdAtunnFmOagtHQ4O9nTH.F3r9aiNdYZoWzoLSC0iie8WMZ.ya4Ut2Xn5oXbW9OF 4_xWq2BMBL2XEa1jsUYGN_mwlfAFK_SYOvNpmFktivnQJJr84ddNrXH7FaiMvwfSV3DfHVPd851z 5QSd_z9X4HAXkQQnuMmS0sf6zNjrFxTXaZ3syerWsB_iRe7.pBV4oJz_DPeVTX93flWdIafbrp5Z oikdp6FllL8zGsK_IjtoUJ0njk3s39L9IFzxjnM9r3jB3rn7pkV7BJr_ycmh8GkVBvjmXWjeC9zJ g857.dEsOJ_UZXuRDmm1cDMMsxMnjFXWGgRCjhEE1d1GpiuPrr1mCBT5sKlcBEd4iW2PuwhLmus8 TXjdh6oQOXbYuLnxtvQ8iE1rehxD4OCUP9cXLmiFrPRogbiKThAaJFxOxIH029iGQp4u0G67Q7gH Zs1nL5Uo9waevRefWRYsTHx4JmaKqGvA4EaT7OsZckrfeyGpZg2m1Vbn6_95S9Yl1aCmNf0wBp5y 9_goAa_J7TH1rxJGG4QoF456jqqMfUI9q8dxjBm1ygyHKCTouucR2k1jrrgzEMXR_552ZHv6lWi4 lrhd6YDRsB8Pj5xro_cwIgmFnXYUu2hHjkhd7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.bf2.yahoo.com with HTTP; Fri, 19 Apr 2019 05:17:50 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp408.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID bb6fba2b6d368dc95857f40b5c5be28e; Fri, 19 Apr 2019 05:17:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 or 32-bit power context: FreeBSD lwsync use vs. th->th_generation handling (and related th-> fields) [Correction] From: Mark Millard In-Reply-To: <50CFD7F1-6892-4375-967B-4713517C2520@yahoo.com> Date: Thu, 18 Apr 2019 22:17:46 -0700 Cc: Bruce Evans , Konstantin Belousov Content-Transfer-Encoding: quoted-printable Message-Id: <0FD9ED28-EF4B-4A1C-9FCE-81C4D5BAEBF1@yahoo.com> References: <50CFD7F1-6892-4375-967B-4713517C2520@yahoo.com> To: FreeBSD PowerPC ML , freebsd-hackers Hackers X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: C8E808E3E0 X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.46 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(1.67)[ip: (5.54), ipnet: 74.6.128.0/21(1.60), asn: 26101(1.28), country: US(-0.06)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:26101, ipnet:74.6.128.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.99)[0.985,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.82)[0.819,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.49)[0.493,0]; RCVD_IN_DNSWL_NONE(0.00)[83.129.6.74.list.dnswl.org : 127.0.5.0]; FREEMAIL_CC(0.00)[optusnet.com.au] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 05:17:54 -0000 [I caught my mental mistake.] On 2019-Apr-18, at 21:36, Mark Millard wrote: > First I review below lwsync behavior. It is based on a = comparison/contrast > paper for the powerpc vs. arm memory models. It sets context for later > material specific to powerpc64 or 32-bit powerpc FreeBSD. >=20 > "For a write before a read, separated by a lwsync, the barrier will = ensure that the write is > committed before the read is satisfied but lets the read be satisfied = before the write has > been propagated to any other thread." >=20 > (By contrast, sync, guarantees that the write has propagated to all = threads before the > read in question is satisfied, the read having been separated from the = write by the > sync.) >=20 > Another wording in case it helps (from the same paper): >=20 > "The POWER lwsync does *not* ensure that writes before the barrier = have propagated to > any other thread before sequent actions, though it does keep writes = before and after > an lwsync in order as far as [each thread is] concerned". (Original = used plural form: > "all threads are". I tired to avoid any potential implication of cross = (hardware) > "thread" ordering constraints for seeing the updates when lwsync is = used.) >=20 >=20 > Next I note FreeBSD powerpc64 and 32-bit powerpc details > that happen to involve lwsync, though lwsync is not the > only issue: >=20 > atomic_store_rel_int(&th->th_generation, ogen); >=20 > and: >=20 > gen =3D atomic_load_acq_int(&th->th_generation); >=20 > with: >=20 > static __inline void \ > atomic_store_rel_##TYPE(volatile u_##TYPE *p, u_##TYPE v) \ > { \ > \ > powerpc_lwsync(); \ > *p =3D v; \ > } >=20 > and: >=20 > static __inline u_##TYPE \ > atomic_load_acq_##TYPE(volatile u_##TYPE *p) \ > { \ > u_##TYPE v; \ > \ > v =3D *p; \ > powerpc_lwsync(); \ > return (v); \ > } \ >=20 > also: >=20 > static __inline void > atomic_thread_fence_acq(void) > { >=20 > powerpc_lwsync(); > } >=20 >=20 >=20 > First I list a simpler-than-full-context example to > try to make things clearer . . . >=20 > Here is a sequence, listing in an overall time > order, omitting other activity, despite the distinct > cpus, (N!=3DM): >=20 >=20 > (Presume th->th_generation=3D=3Dogen-1 initially, then:) >=20 > cpu N: atomic_store_rel_int(&th->th_generation, ogen); > (same th value as for cpu M below) >=20 > cpu M: gen =3D atomic_load_acq_int(&th->th_generation); >=20 >=20 > For the above sequence: >=20 > There is no barrier between the store and the later > load at all. This is important below. >=20 >=20 > So, if I have that much right . . . >=20 > Now for more actual "load side" context: > (Presume, for simplicity, that there is only one=20 > timehands instance instead of 2 or more timehands. So > th does not vary below and is the same on both cpu's > in the later example sequence of activity.) >=20 > do { > th =3D timehands; > gen =3D atomic_load_acq_int(&th->th_generation); > *bt =3D th->th_offset; > bintime_addx(bt, th->th_scale * tc_delta(th)); > atomic_thread_fence_acq(); > } while (gen =3D=3D 0 || gen !=3D th->th_generation); >=20 > For simplicity of referring to things: I again show > a specific sequence in time. I only show the > &th->th_generation activity from cpu N, again for > simplicity. >=20 > (Presume timehands->th_generation=3D=3Dogen-1 initially > and that M!=3DN:) >=20 > cpu M: th =3D timehands; > (Could be after the "cpu N" lines.) >=20 > cpu N: atomic_store_rel_int(&th->th_generation, ogen); > (same th value as for cpu M) >=20 > cpu M: gen =3D atomic_load_acq_int(&th->th_generation); > cpu M: *bt =3D th->th_offset; > cpu M: bintime_addx(bt, th->th_scale * tc_delta(th)); > cpu M: atomic_thread_fence_acq(); > cpu M: gen !=3D th->th_generation > (evaluated to false or to true) >=20 > So here: >=20 > A) gen ends up with: gen=3D=3Dogen-1 || gen=3D=3Dogen > (either is allowed because of the lack of > any barrier between the store and the > involved load). >=20 > B) When gen=3D=3Dogen: there was no barrier > before the assignment to gen to guarantee > other th-> field-value staging relationships. (B) is just wrong: seeing the new value (ogen) does guarantee some about the other th->=20 field-value staging relationships seen, given the lwsync before the store and after the load. > C) When gen=3D=3Dogen: gen!=3Dth->th_generation false > does not guarantee the *bt=3D. . . and > bintime_addx(. . .) activities were based > on a coherent set of th-> field-values. Without (B), (C) does not follow. > If I'm correct about (C) then the likes of the > binuptime and sbinuptime implementations appear > to be broken on powerpc64 and 32-bit powerpc > unless there are extra guarantees always present. >=20 > So have I found at least a powerpc64/32-bit-powerpc > FreeBSD implementation problem? No: I did not find a problem. > Note: While I'm still testing, I've seen problems > on the two 970MP based 2-socket/2-cores-each G5 > PowerMac11,2's that I've so far not seen on three > 2-socket/1-core-each PowerMacs, two such 7455 G4 > PowerMac3,6's and one such 970 G5 PowerMac7,2. > The two PowerMac11,2's are far more tested at > this point. But proving that any test-failure is > specifically because of (C) is problematical. >=20 >=20 > Note: arm apparently has no equivalent of lwsync, > just of sync (aka. hwsync and sync 0). If I > understand correctly, PowerPC/Power has the weakest > memory model of the modern tier-1/tier-2 > architectures and, so, they might be broken for > memory model handling when everything else is > working. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Fri Apr 19 16:40:41 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EFCC15717F9 for ; Fri, 19 Apr 2019 16:40:41 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08CF586916 for ; Fri, 19 Apr 2019 16:40:39 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3JGeUZc044915 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 19 Apr 2019 18:40:30 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3JGeTNe044906; Fri, 19 Apr 2019 18:40:30 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 19 Apr 2019 18:40:29 +0200 (CEST) From: Wojciech Puchar To: Jim Thompson cc: Wojciech Puchar , Miroslav Lachman <000.fbsd@quip.cz>, Mark Millard via freebsd-hackers Subject: Re: openvpn and system overhead In-Reply-To: <94EA4F3F-4D78-4E08-9AF8-441B957A4749@netgate.com> Message-ID: References: <8648d069-2172-2c09-8e59-d66a8265a120@quip.cz> <94EA4F3F-4D78-4E08-9AF8-441B957A4749@netgate.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 X-Rspamd-Queue-Id: 08CF586916 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-5.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[puchar.net]; CTYPE_MIXED_BOGUS(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.87)[-0.868,0]; IP_SCORE(-3.52)[ip: (-9.30), ipnet: 194.1.144.0/24(-4.65), asn: 43476(-3.72), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 16:40:41 -0000 > Using a tun/tap device incurs an additional context switch in each direction, as you’re basically running the program to send data (say, ‘ping’ or ’ssh’), and another program is used to encrypt and encapsulate the packet before it leaves the machine. The process is roughly the same on the other side. So you get twice the copies, and twice the number of context switches. Making things worse, the “IP stack” inside OpenVPN is single-threaded, and processes one packet at a time, so all the overheads accrue to each packet, rather than being amortized across several packets. it would be very good for tun device to have option (switchable by ioctl) so read will receive a bunch of packets up to read size, and write can send a bunch of packets. From owner-freebsd-hackers@freebsd.org Fri Apr 19 16:42:05 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB41B157187B for ; Fri, 19 Apr 2019 16:42:05 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EB60A86AD1 for ; Fri, 19 Apr 2019 16:42:04 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3JGg092045340 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 19 Apr 2019 18:42:00 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3JGg0Ej045337; Fri, 19 Apr 2019 18:42:00 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 19 Apr 2019 18:42:00 +0200 (CEST) From: Wojciech Puchar To: Eugene Grosbein cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: openvpn and system overhead In-Reply-To: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Message-ID: References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: EB60A86AD1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; NEURAL_HAM_SHORT(-0.87)[-0.868,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.52)[ip: (-9.31), ipnet: 194.1.144.0/24(-4.65), asn: 43476(-3.72), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 16:42:05 -0000 > because of unavoidable and big overhead due to constant context switching > from user land to kernel land and back. Be it openvpn or another userland daemon. > > You need either some netmap-based solution or kernel-side vpn like IPsec (maybe with l2tp). well it has to cooperate with multitude of clients like windoze, point&click routers etc. that's why openvpn. From owner-freebsd-hackers@freebsd.org Fri Apr 19 16:45:49 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F4CE1571C20 for ; Fri, 19 Apr 2019 16:45:49 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5802986D1F for ; Fri, 19 Apr 2019 16:45:48 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3JGjhe1046266 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 19 Apr 2019 18:45:44 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3JGjhSW046263; Fri, 19 Apr 2019 18:45:43 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 19 Apr 2019 18:45:43 +0200 (CEST) From: Wojciech Puchar To: Jim Thompson cc: Eugene Grosbein , Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: openvpn and system overhead In-Reply-To: Message-ID: References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 X-Rspamd-Queue-Id: 5802986D1F X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-5.72 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; CTYPE_MIXED_BOGUS(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.91)[-0.907,0]; IP_SCORE(-3.51)[ip: (-9.27), ipnet: 194.1.144.0/24(-4.63), asn: 43476(-3.71), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 16:45:49 -0000 >> >> You need either some netmap-based solution or kernel-side vpn like IPsec (maybe with l2tp). >> For me, IKE daemon plus net/mpd5 work just fine. mpd5 is userland daemon too, >> but it processes only signalling traffic like session establishment packets >> and then it setups kernel structures (netgraph nodes) so that payload traffic is processed in-kernel only. > > > Addendum to previous message to freebsd-hackers: > > We have (also) considered a netmap-enhanced (enabled?) OpenVPN. You still have the problem that the ‘stack’ inside OpenVPN is single-threaded/single packet at a time. > > Also, you’ll need to multiplex > 1 instance of OpenVPN, maybe using the programability of VALE (aka ‘mswitch’). > there is no problem that openvpn is single threaded. i can easily divide things over multiple openvpn processes. The problem is CPU load it produces. It will not be smart to use up whole 8 core machine just to provide 3-4Gbps of VPN traffic with no spare power to do actual work. i found that most of time openvpn executes system call, encryption takes little time. if FreeBSD would be able to provide multiple packets per read/write call from/to tun device, as well as send/recv would have multipacket version - it would mean speeding it up at least 4 times. From owner-freebsd-hackers@freebsd.org Fri Apr 19 17:01:14 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 393EC1572552 for ; Fri, 19 Apr 2019 17:01:14 +0000 (UTC) (envelope-from lidl@FreeBSD.org) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254:10::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.pix.net", Issuer "Pix.Com Technologies LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E544087988 for ; Fri, 19 Apr 2019 17:01:13 +0000 (UTC) (envelope-from lidl@FreeBSD.org) Received: from torb.pix.net ([IPv6:2001:470:e254:10:fc26:6da6:2a7d:3ae5]) (authenticated bits=0) by hydra.pix.net (8.15.2/8.15.2) with ESMTPA id x3JH16iS002659; Fri, 19 Apr 2019 13:01:13 -0400 (EDT) (envelope-from lidl@FreeBSD.org) X-Authentication-Warning: hydra.pix.net: Host [IPv6:2001:470:e254:10:fc26:6da6:2a7d:3ae5] claimed to be torb.pix.net Subject: Re: openvpn and system overhead To: freebsd-hackers@freebsd.org References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Reply-To: lidl@FreeBSD.org From: Kurt Lidl Message-ID: <8e238882-1779-41ed-92fd-33abf2667d18@FreeBSD.org> Date: Fri, 19 Apr 2019 13:01:06 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 17:01:14 -0000 On 4/19/19 12:45 PM, Wojciech Puchar wrote: >>> >>> You need either some netmap-based solution or kernel-side vpn like >>> IPsec (maybe with l2tp). >>> For me, IKE daemon plus net/mpd5 work just fine. mpd5 is userland >>> daemon too, >>> but it processes only signalling traffic like session establishment >>> packets >>> and then it setups kernel structures (netgraph nodes) so that payload >>> traffic is processed in-kernel only. >> >> >> Addendum to previous message to freebsd-hackers: >> >> We have (also) considered a netmap-enhanced (enabled?) OpenVPN.  You >> still have the problem that the ‘stack’ inside OpenVPN is >> single-threaded/single packet at a time. >> >> Also, you’ll need to multiplex > 1 instance of OpenVPN, maybe using >> the programability of VALE (aka ‘mswitch’). >> > there is no problem that openvpn is single threaded. i can easily divide > things over multiple openvpn processes. > > The problem is CPU load it produces. It will not be smart to use up > whole 8 core machine just to provide 3-4Gbps of VPN traffic with no > spare power to do actual work. > > i found that most of time openvpn executes system call, encryption takes > little time. > > if FreeBSD would be able to provide multiple packets per read/write call > from/to tun device, as well as send/recv would have multipacket version > - it would mean speeding it up at least 4 times. Well, FreeBSD does have sendmmsg()/recvmmsg(), which allows for sending/receiving multiple packets per system call. I do not know if the "tun" device allows for send/recv type processing, or just read/write. Don't get me wrong -- having in-kernel processing, like ipsec does, is far superior to doing it as a userland daemon, IMHO. Just pointing out that there is a multi-packet system call that could be used, possibly, to make the userland solution less horrible. -Kurt From owner-freebsd-hackers@freebsd.org Fri Apr 19 17:15:42 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A15981572D74 for ; Fri, 19 Apr 2019 17:15:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02E7E883F8; Fri, 19 Apr 2019 17:15:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x3JHFYqU096744 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 19 Apr 2019 20:15:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x3JHFYqU096744 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x3JHFY94096743; Fri, 19 Apr 2019 20:15:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 19 Apr 2019 20:15:34 +0300 From: Konstantin Belousov To: Kurt Lidl Cc: freebsd-hackers@freebsd.org Subject: Re: openvpn and system overhead Message-ID: <20190419171534.GF12936@kib.kiev.ua> References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> <8e238882-1779-41ed-92fd-33abf2667d18@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e238882-1779-41ed-92fd-33abf2667d18@FreeBSD.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 17:15:42 -0000 On Fri, Apr 19, 2019 at 01:01:06PM -0400, Kurt Lidl wrote: > Well, FreeBSD does have sendmmsg()/recvmmsg(), which allows for > sending/receiving multiple packets per system call. I do not know if > the "tun" device allows for send/recv type processing, or just > read/write. sendmmsg and recvmmsg are userspace wrappers, they are not real syscalls. From owner-freebsd-hackers@freebsd.org Fri Apr 19 17:18:07 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C63CA1572FF9 for ; Fri, 19 Apr 2019 17:18:07 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44382888CA; Fri, 19 Apr 2019 17:18:07 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3JHI4W1053436 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 19 Apr 2019 19:18:04 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3JHI4ji053433; Fri, 19 Apr 2019 19:18:04 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 19 Apr 2019 19:18:04 +0200 (CEST) From: Wojciech Puchar To: Kurt Lidl cc: freebsd-hackers@freebsd.org Subject: Re: openvpn and system overhead In-Reply-To: <8e238882-1779-41ed-92fd-33abf2667d18@FreeBSD.org> Message-ID: References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> <8e238882-1779-41ed-92fd-33abf2667d18@FreeBSD.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 44382888CA X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 17:18:07 -0000 >> from/to tun device, as well as send/recv would have multipacket version - >> it would mean speeding it up at least 4 times. > > Well, FreeBSD does have sendmmsg()/recvmmsg(), which allows for > sending/receiving multiple packets per system call. I do not know if > the "tun" device allows for send/recv type processing, or just > read/write. > i will have a look at openvpn code next weekend. tun doesn't have this but this would bit still a bit speedup. > Don't get me wrong -- having in-kernel processing, like ipsec does, is far > superior to doing it as a userland daemon, IMHO. Just pointing out Having everything in kernel isn't superior. Having low overhead system call interface would be. Sad to say but system call overhead in FreeBSD is high. From owner-freebsd-hackers@freebsd.org Fri Apr 19 19:31:08 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61FFC1576738 for ; Fri, 19 Apr 2019 19:31:08 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: from mail-oi1-x22b.google.com (mail-oi1-x22b.google.com [IPv6:2607:f8b0:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 77B1C8E4BB; Fri, 19 Apr 2019 19:31:07 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: by mail-oi1-x22b.google.com with SMTP id v84so4651397oif.4; Fri, 19 Apr 2019 12:31:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XxTw1eN3uez754ZlON/qxieQFKedAP/sSSNGrZOHNuU=; b=AIstS+MGeNcILz00gCJV0HPz4pfIzpsLBFDkfZnpNLGMlQXaiVHrCCi7Umhw0Istb4 FK6/c9wHobWMgHfd0F6rMZAmdZb3Hof1kAn5oE47fXZ8RbFP4Vil7OzMRqhCIrzog6sj kn4gvpxPv9jUtCatuTX9o0DEdq0cdrVfwDeJt3myWzsFui+Fze3+awTvk0Mow6rMQMGu wveAYxSm2e+0A6G4rIMJUV0CNbVB7Vv5Kq51LyFOhTROdeRT957J+RwaQnGzBp/cdZWh nYXrYGTauiAl/spRMskSabF62ejQgAUKpiW+gtYmzMec/CuV3hSoC3hKRpbgN5HTfLhp 3clA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XxTw1eN3uez754ZlON/qxieQFKedAP/sSSNGrZOHNuU=; b=HieHMysnqK3ZMR4rQnviXRLi1uRYi6ACgRQuGX8wdmRBJy4Tx3J5SiYKTwDHRvdOy6 Sertag/HzmAA2xDgZgaYZyMbF7UjmHJIWOC9UAYd3FLUWitW+zz+GcDZvst8BkiIoNpc b2zX/36n8I4p1/Bk3H8mXLN31vYfn3er9NFyD9Q3bCRxu+bOiyhqqirBoibFtYlJS+5b oNyR1pRDdpYY6HMnzUkoSvOdpfG51+p3SrJXXAniGvWhIp/RPi+WLYWgw5VxsDzEGm9j I5s4S00KY3fUac0zJFO8LXjctid5r41Rsf166Xez2DIH8Tt39nyOpn2stjxIidpRrxzy N/3A== X-Gm-Message-State: APjAAAW39ceL7G/33B4InKHg4guXRd7qrj1QNa15eCQK0grhFJYUMQFI UOCfQiSGcD2vOKruM/7x0E3Ri5dzFm+l6JEzICzR36oD X-Google-Smtp-Source: APXvYqwdLz1oGd6xqTFs6aDEAEXcVZkmhybC9luP5RIYwGybWg8FeDpqGc2KhjS2mzcVJLOCrBdk4ecgmujYEyWOzgU= X-Received: by 2002:aca:ad8e:: with SMTP id w136mr2871654oie.109.1555702266332; Fri, 19 Apr 2019 12:31:06 -0700 (PDT) MIME-Version: 1.0 References: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> In-Reply-To: <173b3741db8be891cff8b7005b2058a416d43115.camel@freebsd.org> From: Lee D Date: Fri, 19 Apr 2019 15:29:57 -0400 Message-ID: Subject: Re: What code loads kernel modules at boot? To: Ian Lepore Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 77B1C8E4BB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=AIstS+MG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of embaudarm@gmail.com designates 2607:f8b0:4864:20::22b as permitted sender) smtp.mailfrom=embaudarm@gmail.com X-Spamd-Result: default: False [-5.68 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.78)[-0.780,0]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_IN_DNSWL_NONE(0.00)[b.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.89)[ip: (-9.02), ipnet: 2607:f8b0::/32(-3.12), asn: 15169(-2.25), country: US(-0.06)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 19:31:08 -0000 On Wed, Apr 17, 2019 at 11:04 PM Ian Lepore wrote: > > The bulk of the module-loading code (the arch-independent part of it) > is in src/stand/common/module.c. There is also archsw.arch_loadaddr, > which figures out where to put the modules (handling arch-specific > things like alignment requirements). > > For some reason, I thought Zynq used u-boot. That would allow using > either ubldr or the arm uefi loader (depending on how old the u-boot > is); those are just flavors of loader(8) that would get you module > handling and all the other loader goodness. > > -- Ian > I think I sent this incorrectly the first time, so here it is again. Zynq does indeed have u-boot as a booting method, but back when I started this project I made the decision to write my own boot loader for various reasons (expediency and customization). So I am not using u-boot or loader(8), and have to prepare everything for the kernel startup myself. Lee From owner-freebsd-hackers@freebsd.org Fri Apr 19 19:58:52 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FE041577256 for ; Fri, 19 Apr 2019 19:58:52 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43B548F730; Fri, 19 Apr 2019 19:58:51 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: by mail-wr1-x430.google.com with SMTP id t17so8024211wrw.13; Fri, 19 Apr 2019 12:58:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:cc:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=PE8RH0dPIgnAURvsPbbe9tqgjhKqyzZxyZ+Gxsh3H0s=; b=GN5n43hNdgNP55CDE94W17nSYKOG1x/kaAG3iCDDcVC+dJhuNnXjtJOLMAWfcZwcgg qhpr2vLJ291c+5sQVAtqpjsWF4nWJYsmfABK4pzaH1RiA9zglNM4/BWD/Dg1L5aXsj1B zNp+sXiY09BHfNc1/qRYT8Ko1iGgjqBtPiIQVOOpDOTQn+nRlPjo7M196kqAIN9/hRWW c2jKTCmWHiMXJw0dMjEnIcNylWz8zpRb3EGye+901kAeNjg4SKCEb0Nsvy/Ncw2HoVsg 5pivlfI8Uz+WCySOFHbyf6lhpcHJXmcm0QefBRs7uNh4+0IjImUT7RVUMyCM+bjkInAT zOWw== X-Gm-Message-State: APjAAAU5oOnI6bW8rofWoIWhxWeNMB2FpoWpmx+g7g5TXvklqGOu9DDO lYN6t9/PMgC6MoXsHhO5tA4oelxurkY= X-Google-Smtp-Source: APXvYqytExNq+bTW8SIChmwrH+BYCpIxof4/L9O/GoQeZcj0k8xOGytXHtWAGl+ukAfgDhZdHLe9bQ== X-Received: by 2002:adf:f58f:: with SMTP id f15mr4095110wro.183.1555703929762; Fri, 19 Apr 2019 12:58:49 -0700 (PDT) Received: from asommer-mac.local ([2a02:2455:41f:4e00:f5ea:abaa:e09a:4d33]) by smtp.googlemail.com with ESMTPSA id j9sm6358383wrr.93.2019.04.19.12.58.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Apr 2019 12:58:48 -0700 (PDT) To: freebsd-hackers@freebsd.org From: Andreas Sommer Subject: Poudriere nested jail configuration - want explanation for each setting Cc: Bryan Drewery , Baptiste Daroussin Message-ID: <1c310e2a-0f3f-f455-71d5-2d03213c4773@googlemail.com> Date: Fri, 19 Apr 2019 21:58:46 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 43B548F730 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[googlemail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[0.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.82)[ip: (-9.40), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.25), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.95)[-0.946,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 19:58:52 -0000 Hi all (main poudriere authors in CC, hoping this is etiquette-friendly), I'm currently writing a tutorial that includes setting up Poudriere within a jail (nested). While the appropriate settings are nicely listed at https://github.com/freebsd/poudriere/wiki/poudriere_in_jail, I'd like to get clarity which ones are actually needed and *why*. Of course I did some research in freebsd/poudriere code which resulted in the write-up I have until now (feel free to improve grammar/text as well): > Jail config: > ip4.addr += "lo0|127.0.0.3"; > exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0"; > allow.chflags; > allow.mount; > allow.mount.devfs; > allow.mount.nullfs; > allow.mount.procfs; > allow.mount.tmpfs; > allow.mount.zfs; # only needed if you use ZFS > allow.raw_sockets; # optional > allow.socket_af; # optional > allow.sysvipc; # optional > children.max=16; > enforce_statfs=1; > [...] > - `ip4.addr += "lo0|127.0.0.3"` adds another IPv4 address to the jail. You will later configure Poudriere's `LOIP4` variable in order to assign this loopback address to build jails which are not supposed to talk to the Internet or other machines in your network, such as during the `build` phase. If you ever have a build which requires Internet access during build, Poudriere supports a variable `ALLOW_NETWORKING_PACKAGES` as workaround. However, you should prefer the clean way, i.e. performing downloads and other Internet-facing tasks earlier, in the `fetch` phase which is permitted Internet access. > - `allow.chflags` allows Poudriere to render certain system files in the build jail immutable, since builds are not supposed to overwrite e.g. `/bin/sh` > - `allow.mount` and the other `allow.mount.*` options enable Poudriere to mount certain required filesystems into the build jails > - `allow.raw_sockets` (permit use of raw sockets) and `allow.socket_af` (use of any socket address family) are applied to the Internet-capable build jails. This is mostly only helpful so that you can run tools like `ping` in interactive mode i.e. when entering a build jail to debug problems. You could disable these permissions. > - `allow.sysvipc` is actually deprecated in favor of three separate settings `sysvmsg`/`sysvsem`/`sysvshm` to restrict jails to only see their own shared memory objects (via "SYS V" IPC primitives). However, Poudriere can only pass on `allow.sysvipc` to build jails because it cannot read the relevant sysctl information for the three separate parameters (as of FreeBSD 11.2). With this deprecated configuration, the jail could read shared memory of processes outside the jail. This is only relevant for certain software that depends on IPC features, like PostgreSQL, so chances are small for this to affect security. You can remove this configuration unless you depend on a port that requires it during build. > - `children.max=16` allows 16 sub-jails below the worker jail. You can raise this number later if you have a lot of CPUs and Poudriere tries to create more build jails than permitted. Note that each Poudriere build will try to create a reference jail and 2 build jails per "job", and its default is to use the number of CPUs (as output by `sysctl -n hw.ncpu`) as job count. > - `enforce_statfs=1` is required together with `allow.mount` in order to mount certain filesystems. Can you confirm the descriptions? Especially for the `allow.sysvipc` topic, I digged a little more: There's `security.jail.sysvipc_allowed={0,1}` which is passed on to build jails by Poudriere. And for the 3 non-obsolete jail settings `sysvmsg`/`sysvsem`/`sysvshm`, sysctl has to offer `kern.features.sysv_{shm,sem,msg}` and `security.jail.param.sysv{shm,sem,msg}.` (with a dubious dot at the end!). And here's the problem: no matter whether I configure the jail with `{sysvmsg,sysvsem,sysvshm} = "new";` or leave out any SYSVIPC setting (= disallow altogether), the sysctls stay the same at: > $ sudo jexec myjail sysctl -a | grep .sysv > kern.features.sysv_shm: 1 <---- these are always 1 > kern.features.sysv_sem: 1 > kern.features.sysv_msg: 1 > security.jail.param.sysvshm.: 0 <---- these are always 0 > security.jail.param.sysvsem.: 0 > security.jail.param.sysvmsg.: 0 > security.jail.param.allow.sysvipc: 0 <---- these are unrelated to the 3 separate settings > security.jail.sysvipc_allowed: 0 On the other hand, I confirmed the `{sysvmsg,sysvsem,sysvshm} = "new";` settings do what they should (using FreeBSD 11.2 and PostgreSQL) – prohibit the jail from seeing outer shared objects. Therefore I concluded that Poudriere simply has no visibility whether the "poudriere jail" has those features, and hence cannot pass those on to its nested build jails. It does that for `allow.sysvipc`, though, since the sysctl `security.jail.sysvipc_allowed` shows the correct value within the jail. Reference: poudriere/src/share/poudriere/include/common.sh.freebsd > if [ ${JAILED} -eq 0 ] || \ > [ $(sysctl -n security.jail.sysvipc_allowed) -eq 1 ]; then > JAIL_PARAMS="${JAIL_PARAMS:+${JAIL_PARAMS} }allow.sysvipc" > fi Thanks in advance, Andreas From owner-freebsd-hackers@freebsd.org Fri Apr 19 21:00:37 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B8191578B42 for ; Fri, 19 Apr 2019 21:00:37 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0CB186B534 for ; Fri, 19 Apr 2019 21:00:35 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3JL0XxL099019 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 19 Apr 2019 23:00:33 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3JL0X6J099016 for ; Fri, 19 Apr 2019 23:00:33 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 19 Apr 2019 23:00:33 +0200 (CEST) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: bhyve VM stopped to boot after moving virtio disks Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 0CB186B534 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.73 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; NEURAL_HAM_SHORT(-0.91)[-0.909,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.51)[ip: (-9.27), ipnet: 194.1.144.0/24(-4.64), asn: 43476(-3.71), country: PL(0.07)]; DMARC_NA(0.00)[puchar.net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 21:00:37 -0000 i have bhyve VM (linux) with 2 virtual disks. they were backed on geli encrypted partitions on SSD as it is rarely used VM i wanted to move it to HDD file. i just did dd if=/dev/partition1.eli of=file1 bs=1m dd if=/dev/partition2.eli of=file2 bs=1m there were no errors when doing dd and changed /dev/partition1.eli to file1 and same with second in virtio-blk filename. Everything else is unchanged. now it doesn't boot. on UEFI virtual framebuffer i see Boot Failed. EFI Misc Device Boot Failed. EFI Misc Device 1 . Could you give me any clues? From owner-freebsd-hackers@freebsd.org Fri Apr 19 21:37:28 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02A001579610 for ; Fri, 19 Apr 2019 21:37:27 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2m.ore.mailhop.org (outbound2m.ore.mailhop.org [54.149.155.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB3796C994 for ; Fri, 19 Apr 2019 21:37:26 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555709845; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=KXZxzJDmNZVR2HLWpt0NG1Viv4+G6a2yzaw1Oj3JcmoWG72YwrC2dNqYaTXQLLtnJueSuEKEMHgUg Iv8cOhGcbDaUg4/jE53FCgUCyXS+ef6666Ay3S56m4v1N9XK5w2yhLRdptxvkokaSAm/1hDKxsxtTR jO2wyXDPsU4Zc0J/WjLhJgAfjZo7RtOahXXm8b390BYCJTdXQgDzpgJM866Lfmtsj7q6gSf5xoZhbo JT1Nnl6wGF9PdnEQTJJ3YD2U8jKkywng1E2Zl/054XKGUD+eHOW9YdUaKWhg7RIOtIUgK30mjnI5BH eDhMxGhWzJfKF/ZIApFUAkemqscgdDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:dkim-signature:from; bh=HWCgepgS+Eno4c2xwvws6XxBhzHx7wnFUb6YdRXI+2c=; b=UiBIMxL00LqvwSNCLiQcqNrg/exgSoHJ/LieVZlBqmKqNfOAGW/V6e/UmsS4LrT0YEwGS/UIngiMn 7JqHlJFisM7UctMl9YrNZtUTL5rzBPsMZKdri5LdjpY8zPSgGlKAlPCMWwURT1/atIXK/8W2/Whytg S42ONRGvmjmwmfqgIgwM7ZK0HNnpttTcMaLTBoHrkpIKHUNY/v08c0zT2TBRDuHDb/5bOGyxBK3E7O yDqt5jU+aYxjAMHQSgCVECLq/2dwe8z3VjVFlYFwcI9vJeparAbjzPnm68L/inN72jnGiewWMexxzt EJqylGBaDid49D7TGpyUy2/cdhS7WwA== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:from; bh=HWCgepgS+Eno4c2xwvws6XxBhzHx7wnFUb6YdRXI+2c=; b=GO0X+XLlIQSMAPslr6pTiranipZ5EymrEHRuCdGtEyguEMql77j2784GQLNLpPEiU8AcEx8LaFE71 OAMgbzjLIW3bkJNQWwlCybILVGYaIRXF11tVPUQApOWhH9Hfg0TRcAO5ytaQwT9TQguni43fR1IOsn rlr9RyeRDsCUbOXzZAp5fQtku+ObcWCiXK7GSMUtZ+TNo8dyxYHoxxjJ+AJKtOOTZ9N+ujPl2jNmaB 398DTgUde7fAoPbjsORORRqqhI0sFdp87EKJgqNP9qlBkJY+MNnvXUVhGFk5mg0sdBYkUjhocAMfrn vsal/+sEDv/6djvlKvpQmvNMCorJnlg== X-MHO-RoutePath: aGlwcGll X-MHO-User: 514bf717-62eb-11e9-919f-112c64a8cf29 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 514bf717-62eb-11e9-919f-112c64a8cf29; Fri, 19 Apr 2019 21:37:24 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3JLbMd4055044; Fri, 19 Apr 2019 15:37:22 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> Subject: Re: bhyve VM stopped to boot after moving virtio disks From: Ian Lepore To: Wojciech Puchar , freebsd-hackers@freebsd.org Date: Fri, 19 Apr 2019 15:37:22 -0600 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: DB3796C994 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 21:37:28 -0000 On Fri, 2019-04-19 at 23:00 +0200, Wojciech Puchar wrote: > i have bhyve VM (linux) with 2 virtual disks. > > they were backed on geli encrypted partitions on SSD > > as it is rarely used VM i wanted to move it to HDD file. > > i just did > > dd if=/dev/partition1.eli of=file1 bs=1m > dd if=/dev/partition2.eli of=file2 bs=1m > > there were no errors when doing dd > > and changed /dev/partition1.eli to file1 and same with second in > virtio-blk filename. > > Everything else is unchanged. > > now it doesn't boot. on UEFI virtual framebuffer i see > > Boot Failed. EFI Misc Device > Boot Failed. EFI Misc Device 1 > . > > > > Could you give me any clues? > Are you absolutely certain those encrypted partition sizes are an exact multiple of 1m? If not, you should add a 'conv=sync' option to the dd. -- Ian From owner-freebsd-hackers@freebsd.org Sat Apr 20 06:13:52 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE87B1584747 for ; Sat, 20 Apr 2019 06:13:52 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D200F84C21 for ; Sat, 20 Apr 2019 06:13:41 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-hackers@freebsd.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id x3K6DWWj048152; Sat, 20 Apr 2019 13:13:32 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: openvpn and system overhead To: Wojciech Puchar References: <0cc6e0ac-a9a6-a462-3a1e-bfccfd41e138@grosbein.net> Cc: freebsd-hackers@freebsd.org From: Eugene Grosbein Message-ID: <5CBAB88C.4020402@grosbein.net> Date: Sat, 20 Apr 2019 13:13:32 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D200F84C21 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [0.33 / 15.00]; ARC_NA(0.00)[]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.25)[0.252,0]; NEURAL_HAM_LONG(-0.50)[-0.495,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_MEDIUM(0.18)[0.175,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(0.00)[country: RU(0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 06:13:53 -0000 On 19.04.2019 23:42, Wojciech Puchar wrote: >> because of unavoidable and big overhead due to constant context switching >> from user land to kernel land and back. Be it openvpn or another userland daemon. >> >> You need either some netmap-based solution or kernel-side vpn like IPsec (maybe with l2tp). > > well it has to cooperate with multitude of clients like windoze, > point&click routers etc. that's why openvpn. Windows has stock support for IPSec with and without L2TP and has no stock openvpn, so IPSec is more preferable. Cheap and slow SOHO routers generally have worst performance with openvpn that with any other kind of VPN, too. From owner-freebsd-hackers@freebsd.org Sat Apr 20 15:30:16 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C26C8159054B for ; Sat, 20 Apr 2019 15:30:16 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 82A1894CD0; Sat, 20 Apr 2019 15:30:15 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id x3KFUAcF001727 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 20 Apr 2019 17:30:10 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id x3KFUA0Q001724; Sat, 20 Apr 2019 17:30:10 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Sat, 20 Apr 2019 17:30:10 +0200 (CEST) From: Wojciech Puchar To: Ian Lepore cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: bhyve VM stopped to boot after moving virtio disks In-Reply-To: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> Message-ID: References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 82A1894CD0 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-6.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[puchar.net]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; NEURAL_HAM_SHORT(-0.84)[-0.837,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.51)[ip: (-9.28), ipnet: 194.1.144.0/24(-4.64), asn: 43476(-3.71), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 15:30:16 -0000 >> Boot Failed. EFI Misc Device >> Boot Failed. EFI Misc Device 1 >> . >> >> >> >> Could you give me any clues? >> > > Are you absolutely certain those encrypted partition sizes are an exact > multiple of 1m? If not, you should add a 'conv=sync' option to the dd. no it wasn't. but dd doesn't truncate last block. checked - it got copied till the end - i see second copy of GPT partition table that was present in VM. From owner-freebsd-hackers@freebsd.org Sat Apr 20 16:56:57 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C02C157027E for ; Sat, 20 Apr 2019 16:56:57 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 483F16AAAE for ; Sat, 20 Apr 2019 16:56:56 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555779409; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=oB2KMfe/lgBaNyo+DrlB/8STM8l3kVNQ1YWEPtpwF96saPjT+xiZsrMKoiJk6pEI8ORjieKm83PA/ BeQ+C8Fp7+313p5J7J1LLWgNWwf8Mf92dS2J5Uf1tsXz1HttVulT7ssmmgHyKTmVOqhu6Im/nXBVQa h9kr5f0A5wNEYzq4tvUUMLDMa7VYTv28sfRz4NZh66FKamYCPjhmuIKJ3zUOrMjwgcQzi7a345t2GO C8PocFHL5pH106zuniM0X0GBqx2qKAN6/UmbwzklYRDrBc3rFrEiQ2ZDURQC7S6Gu5aXiUpUd49hNs wcpemdPyzdwbJHIWkhw9O8wy5FxBFqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:dkim-signature:from; bh=obcQ1PCkVVeX4OMg6FTUtQTt7QWaFJ9egbbtBXYXkXw=; b=ki3miFZJr8r/7DoTnk5V0vE9LHhb7aQHcOt/1lpECOQdwuLD6H1WwK5MV5UCO4TaKzeHpPZLYgKcz FCh51wVxpBpzvQXIuS6aJ1bgMolNlGrD+QmWjkOKW46/S1wgzcrAG7OI0QacsRe/BG9dEvOqFLJOXt XwDcel/jK+exSFPV8c/+dTfzeIsv6LKRoaeb6khPB1OUzjbg8Ptk/xOdZdTfbdJ+vY/fUwh+dY9CFd z/JE4ffqARXEkaTu+E7ZWjBaiZ17Hh2XsYCIWcx4UxdqPaNc2ENX9RjcUxK9Zwsrj6X8ZpsgJQL3UQ 29qu0Y5S93BD7K2Ow15FG2WR+Rk57MA== ARC-Authentication-Results: i=1; outbound2.eu.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:from; bh=obcQ1PCkVVeX4OMg6FTUtQTt7QWaFJ9egbbtBXYXkXw=; b=cmrCU8j8R9FVYK0RXcNx0cu/VGC3kfMjA16ippE4KQ4E75a76sh8Rz8aN9nx9unkYVfZxSrgLfyxN o3TVBMX5C/QFrl7oOlIZc24xFLFegGi4gBn0r76A5PzcktRZ4Q2Dp1vk38+MJm+9zv929qE3RNcI/V MMRDQerIZBAtF3TOf0tyakdlWuz5zu9BoaO8KGXzu6XZdCe1Bs2decJAu29SGbE19Hxv5C2e/K6QaQ /m3ehjzSoqA0Zv0K8h2zOakLVENMQ6X3qRIogiLhghE9ndbvzqLC6PhMZiXRecMYwKOUR7IxDJ2ff2 0IhRScE9D82BZyv6bBSy0EIdXPfbKUw== X-MHO-RoutePath: aGlwcGll X-MHO-User: 47a181f6-638d-11e9-803b-31925da7267c X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.eu.mailhop.org (Halon) with ESMTPSA id 47a181f6-638d-11e9-803b-31925da7267c; Sat, 20 Apr 2019 16:56:47 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3KGuj6k057623 for ; Sat, 20 Apr 2019 10:56:45 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: Subject: Re: bhyve VM stopped to boot after moving virtio disks From: Ian Lepore To: freebsd-hackers@freebsd.org Date: Sat, 20 Apr 2019 10:56:45 -0600 In-Reply-To: References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 483F16AAAE X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; ASN(0.00)[asn:16509, ipnet:52.58.0.0/15, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 16:56:57 -0000 On Sat, 2019-04-20 at 17:30 +0200, Wojciech Puchar wrote: > > > Boot Failed. EFI Misc Device > > > Boot Failed. EFI Misc Device 1 > > > . > > > > > > > > > > > > Could you give me any clues? > > > > > > > Are you absolutely certain those encrypted partition sizes are an > > exact > > multiple of 1m? If not, you should add a 'conv=sync' option to the > > dd. > > no it wasn't. but dd doesn't truncate last block. checked - it got > copied > till the end - i see second copy of GPT partition table that was > present > in VM. > dd absolutely will fail to copy the last block of the source if it isn't exactly the blocksize and you didn't specify conv=sync, and it will return a zero status when doing so. It appears you've convinced yourself otherwise, but for anyone else reading this thread, be aware: conv=sync is required to copy the last part of the source if it's smaller than the blocksize. -- Ian From owner-freebsd-hackers@freebsd.org Sat Apr 20 17:52:28 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C418157185D for ; Sat, 20 Apr 2019 17:52:28 +0000 (UTC) (envelope-from mozolevsky@gmail.com) Received: from mail-ot1-f66.google.com (mail-ot1-f66.google.com [209.85.210.66]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB2206C7E4; Sat, 20 Apr 2019 17:52:26 +0000 (UTC) (envelope-from mozolevsky@gmail.com) Received: by mail-ot1-f66.google.com with SMTP id s24so6575720otk.13; Sat, 20 Apr 2019 10:52:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=bLEfQI8u1xqDgZt/GRXxhi4wxSxtI3daFl+3mLffxoc=; b=TvM0ktefO6VEj6ETyra3NP/5QIx2X9KZ9PRFHpWuynPcSogUT8Celq+Vb2PcOWjGFA tTE59azxDSe/GhRItwff8Mj2UvSNGwWINVsZ9jWD2naKUCw+1VOwuCRQNU0zJcgQKr2+ lGLEexlXDMCLDtBhRQAU1bWQUTL63MgkwpvKLgb5WbPK8aAWY11yzYIwhwuDiv0Wuasq mFZ8O0m3K0pUfkCFDGThl3RulJ+JbIEHrsFJieTyl0JMLWHgGQLVmQSdGxYKzEKTsGNh 8E3pzrwlG+YNxAhcLcq7WmnEFW3RELEnYVqqN3rIbLhApadh3OMgOwsU3tOcHH1+pr6w tKzA== X-Gm-Message-State: APjAAAVBIIglDnnL/G578gOG2VTdhfna4TLwJqd8Zh765tL9aM3ffpmB RJWm4twlq1t58Wo7xRSObCAfs3Z0uysD2EJoB9XaqmDh X-Google-Smtp-Source: APXvYqy65dn9JQ20lLPBdUL4ix1V625edVIIO6Ojs9uwH+rPTUwSQR6dYp4ATU3TNkz3jdMmEMxNqji6qljGn/b0ZOg= X-Received: by 2002:a9d:6ace:: with SMTP id m14mr5873219otq.296.1555782353219; Sat, 20 Apr 2019 10:45:53 -0700 (PDT) MIME-Version: 1.0 References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> In-Reply-To: From: Igor Mozolevsky Date: Sat, 20 Apr 2019 18:45:17 +0100 Message-ID: Subject: Re: bhyve VM stopped to boot after moving virtio disks To: Ian Lepore Cc: Hackers freeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: BB2206C7E4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mozolevsky@gmail.com designates 209.85.210.66 as permitted sender) smtp.mailfrom=mozolevsky@gmail.com X-Spamd-Result: default: False [-3.42 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[66.210.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[hybrid-lab.co.uk]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.24)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.26), country: US(-0.06)]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[66.210.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.17)[-0.169,0]; FORGED_SENDER(0.30)[igor@hybrid-lab.co.uk,mozolevsky@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[igor@hybrid-lab.co.uk,mozolevsky@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 17:52:28 -0000 On Sat, 20 Apr 2019 at 18:00, Ian Lepore wrote: > dd absolutely will fail to copy the last block of the source if it > isn't exactly the blocksize and you didn't specify conv=sync, and it > will return a zero status when doing so. It appears you've convinced > yourself otherwise, but for anyone else reading this thread, be aware: > conv=sync is required to copy the last part of the source if it's > smaller than the blocksize. Isn't that contrary to the POSIX Spec? Reading the manual [1], it is implied (from STDERR part of the main section) that dd will read and write partial blocks, cf. truncated blocks?.. 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html -- Igor M. From owner-freebsd-hackers@freebsd.org Sat Apr 20 18:29:23 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CF271572482 for ; Sat, 20 Apr 2019 18:29:23 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2m.ore.mailhop.org (outbound2m.ore.mailhop.org [54.149.155.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 996726D542 for ; Sat, 20 Apr 2019 18:29:22 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555784960; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=IRdS7QHtw/TFPV2QEqSqxVqFuGTE0tQiJvUtczHpfEkDbWQqxl+7JMyudBMEdQfE3yGjBuuIJmYx5 YHZwZOsk+4d2cuPgPnwTa+jFcC30ef8ZGBTXA8u1DgDrK71M5hg1eMMNwwZWZnD2ZHciBWwzwwrZVs 9ZpNTCw2IBbHsSRnx3FvtFxOODlPolOFJh5ouyxoYwSInChwzXG8i4VjxnJSvOX//nYeLJ/jBiDICo 5XRuLcCNlls8rw4r5EpNJe4Jw5T3mLVmNnVLPU2mZetFmjihvMwCtKf4m6J2PfQ+DgnKM2bnyI5QPv kZbZ0VESH1CkN7DKz4DHPoN0J8AFSvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=gLU3nWse0Pt1BgfSf+7f9OpTmRSluhjsobMODa7Q9Ng=; b=qWJPUxpIn/0nRXAyPkhdLWBAUo3VJ5p5RIcoKG3SKGJnIL6+jO/bODA6hXaRbEcr+9HqAIepEjmhL Xq9bl8W3JDdnyQy6rSNaHapyi74+eY3a36bnheLrvrJE8hLAEJdL/Vju46nSfKLAYBin4PzDspz9M1 kH2a4Vu6Cy06xWoLtlfcwkqXyLHLcwWpnVHaSHLpbB0cC9p8AIlZ0PNH80nRXbEUTkFI6L5xj4A2jq w1v7muzEVQW/TLKs/UF1/EloetvfAZK2LB8Gw6Cz4zrKbn6yO4J12eS/jphvNOyaCEXcp9Ocx7zLnx Al7ZRkJ6JSHVYjLKx7QurxlZKmQf/1A== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=gLU3nWse0Pt1BgfSf+7f9OpTmRSluhjsobMODa7Q9Ng=; b=rTCur1bTHiWWBYQEdb5+g9dL+Pxl06CX+GFAShZ57U78uRq8p2PC75aZ5BTgrpgJ+dlB5LNmQp6Xz 4jhzR78LEQOrWwdPfWbRVUS1M4wkWlRIuVe5gYfm0/QyuKu1XW/f+MOu4Lgvr+6Kr32crpfEH/+N2L 8EQVyVcbQyyZl5kM/ZZN/f7fbJMGYJSc2As5HVEHjKTS73XQ48N+reGIuFj/XT9BdrV9NCRUX274Gw hMop6wCjQ4FHsfBQPgcHitvuRqhe3ItyB6DCgB+lcIykjRKLOshIAgfgUll5S0zkmy7y8HPNwkQWR/ bjG/VRypbuYNMYwZkcWf8P/yvUTRLbw== X-MHO-RoutePath: aGlwcGll X-MHO-User: 34e99285-639a-11e9-919f-112c64a8cf29 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 34e99285-639a-11e9-919f-112c64a8cf29; Sat, 20 Apr 2019 18:29:19 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3KITHOK057815; Sat, 20 Apr 2019 12:29:17 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: Subject: Re: bhyve VM stopped to boot after moving virtio disks From: Ian Lepore To: Igor Mozolevsky Cc: Hackers freeBSD Date: Sat, 20 Apr 2019 12:29:17 -0600 In-Reply-To: References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 996726D542 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 18:29:23 -0000 On Sat, 2019-04-20 at 18:45 +0100, Igor Mozolevsky wrote: > On Sat, 20 Apr 2019 at 18:00, Ian Lepore wrote: > > > > dd absolutely will fail to copy the last block of the source if it > > isn't exactly the blocksize and you didn't specify conv=sync, and > > it > > will return a zero status when doing so. It appears you've > > convinced > > yourself otherwise, but for anyone else reading this thread, be > > aware: > > conv=sync is required to copy the last part of the source if it's > > smaller than the blocksize. > > > Isn't that contrary to the POSIX Spec? Reading the manual [1], it is > implied (from STDERR part of the main section) that dd will read and > write partial blocks, cf. truncated blocks?.. > > 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html > > Damn, why do people trim away the entire useful context of an email thread when replying? I almost don't want to reply to this because at this point all the useful info is gone. In any case, in retrospect, what I said was wrong. Using conv=sync is important when the output is a device that has fixed block sizes where you cannot write a partial block. When the destination is a file where short writes work correctly, it will copy the entire source correctly. -- Ian From owner-freebsd-hackers@freebsd.org Sat Apr 20 18:44:41 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D41931572A0C for ; Sat, 20 Apr 2019 18:44:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.ne1.yahoo.com (sonic309-21.consmr.mail.ne1.yahoo.com [66.163.184.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ADC0F6DC5F for ; Sat, 20 Apr 2019 18:44:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: cXwplIAVM1nvCCzm2Aivk61tS_B5H9MrH00LKRyTJrlTPnLPZlP7eziGGHJozHr q0WQ3eDqtkYjrFn1U5v3shHwVHAtRISfF_uYLT46Ycy8O1zX.GM6X.DUbcM1tGgO7UTxqhaQCcR5 pFaHgxoOjoyLXsaZHfjSSpA9CYB4N2Lph94IuXQF68xcp95U6qCDCkDMuqMrIjeXJtWcFht4dXI4 5LkEu3rLkkmMx4e1GgIs8IGo9kW_ikoPE0KCs8hLKoITYEDXHTa7u7fwtltnSS9IxfIMkQ3VNiEJ 19HtMOCKKmG8C4O0sAA6BO9e1JL7ZiHSsie83sH8kkHNeeB8VjUmdOMgWBGUuKJTp4ZMipnzr10I 5PZeNADKp8UywmdKEn.x1AqDkooB0.ERCHJqwPdqLWY5vRVHeDVwAIsCLy7PQw8plIMdKSsNn.5I Ue2Wahgdjk558aClGLPFF9DSZcYLXzM_Sqb0KxUUInxuSZo9fL1CcnzGFfksvcCuDFJU63RuQiyA gV9w6ZyDUn3CEEgm4u7oDQeLbnyPg8ls8EF.qT9tqgkVAz4utffoQsGBIWwXbPlyudKGRQQ6mgIg Fra5xdyBNo3gc118QW16VHIiSkM2cjVircrTbXWhYzHspKMuBHOb3tlrbLlj5SI2CnTiFh0qp.39 6zCQUf3thQ6qhz1HTXE_dVSwHj0Mu_cPA5l7lPkfR.qB29qwpmJCJb4_ZUNwXI_8ei_td6UClHj_ UqWSweCw5O3f627NYQdUlz_gZ0VkWhikalPz5FcqaPRAq1zGxZkazVYs599YQc.Rof6cTk12FNQV DR9Rh.XdkQp4KMuiHWTVbe.3rLgOg7RB7YMQKCX_6d_j4DNKnq28f_MCkbbfMiAzXvSrdQbZPjvR Dvtrbw3Kgb_cGrVzy7EoNYwEWGulcvuugRFsSmuEIrL6C6ue5IPkHGFXz2NFV5kgSKvwPwxWGQzH 13WnlCg6ZyAy94ywdnj56C0UOBiJFQFXsDf4RumJWWkF0aj6UlJpeCV6Dy9yC9WYEC3V9yd0sxMw l6W899_6HOTD6QbE.Wjm_umiR7iIweYrkHxftT2u2wbwT7jkGXEiI_9MwbDgl4jbtu3Gj9AHdMBe jgnW8nH0h_XMWH3zFb0_wKELE Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.ne1.yahoo.com with HTTP; Sat, 20 Apr 2019 18:44:38 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp417.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID b75694d0beb16a3641ccbf2498e0741b; Sat, 20 Apr 2019 18:44:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: bhyve VM stopped to boot after moving virtio disks From: Mark Millard In-Reply-To: Date: Sat, 20 Apr 2019 11:44:34 -0700 Cc: Ian Lepore , Hackers freeBSD Content-Transfer-Encoding: quoted-printable Message-Id: References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> To: Igor Mozolevsky X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: ADC0F6DC5F X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.51 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.60)[0.599,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.52)[ip: (5.11), ipnet: 66.163.184.0/21(1.43), asn: 36646(1.14), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.31)[0.313,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.58)[0.584,0]; RCVD_IN_DNSWL_NONE(0.00)[147.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 18:44:41 -0000 On 2019-Apr-20, at 10:45, Igor Mozolevsky = wrote: > On Sat, 20 Apr 2019 at 18:00, Ian Lepore wrote: >=20 > >> dd absolutely will fail to copy the last block of the source if it >> isn't exactly the blocksize and you didn't specify conv=3Dsync, and = it >> will return a zero status when doing so. It appears you've convinced >> yourself otherwise, but for anyone else reading this thread, be = aware: >> conv=3Dsync is required to copy the last part of the source if it's >> smaller than the blocksize. >=20 >=20 > Isn't that contrary to the POSIX Spec? Reading the manual [1], it is > implied (from STDERR part of the main section) that dd will read and > write partial blocks, cf. truncated blocks?.. >=20 > 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html Have you tried as Ian suggests to see if it made a difference? Ian seems to be speaking FreeBSD, not necessarily POSIX. (I'm talking actual FreeBSD behavior here, not necessarily documented behavior.) It turns out that https://www.freebsd.org/cgi/man.cgi?dd(1) does say: Normally, data resulting from input or conversion or both are = aggregated into output blocks of the specified size. After the end of input = is reached, any remaining output is written as a block. This means = that the final output block may be shorter than the output block size. where conv=3Dsync is described as: sync Pad every input block to the input buffer size. Spaces are used for pad bytes if a block oriented conversion value is specified, otherwise NUL bytes are used. (So sync changes the overall input size when it adds pad bytes, at least as documented.) It appears to me that Ian's expectations of the actual behavior and the FreeBSD documentation of such are not a match. Absent testing, I'd bet on Ian, even if it means FreeBSD dd has a bug. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sat Apr 20 19:28:25 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C8D01573C9D for ; Sat, 20 Apr 2019 19:28:25 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4EC146EF9F; Sat, 20 Apr 2019 19:28:24 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x3KJSM8w023992; Sat, 20 Apr 2019 12:28:22 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x3KJSMKW023991; Sat, 20 Apr 2019 12:28:22 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201904201928.x3KJSMKW023991@gndrsh.dnsmgr.net> Subject: Re: bhyve VM stopped to boot after moving virtio disks In-Reply-To: To: Mark Millard Date: Sat, 20 Apr 2019 12:28:22 -0700 (PDT) CC: Igor Mozolevsky , Hackers freeBSD , Ian Lepore X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4EC146EF9F X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [2.85 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.83)[0.827,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.60)[0.605,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: gndrsh.dnsmgr.net]; NEURAL_SPAM_LONG(0.50)[0.498,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.03)[ip: (0.11), ipnet: 69.59.192.0/19(0.06), asn: 13868(0.04), country: US(-0.06)] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 19:28:25 -0000 > > > On 2019-Apr-20, at 10:45, Igor Mozolevsky wrote: > > > On Sat, 20 Apr 2019 at 18:00, Ian Lepore wrote: > > > > > >> dd absolutely will fail to copy the last block of the source if it > >> isn't exactly the blocksize and you didn't specify conv=sync, and it > >> will return a zero status when doing so. It appears you've convinced > >> yourself otherwise, but for anyone else reading this thread, be aware: > >> conv=sync is required to copy the last part of the source if it's > >> smaller than the blocksize. > > > > > > Isn't that contrary to the POSIX Spec? Reading the manual [1], it is > > implied (from STDERR part of the main section) that dd will read and > > write partial blocks, cf. truncated blocks?.. > > > > 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html > > Have you tried as Ian suggests to see if it made a difference? > Ian seems to be speaking FreeBSD, not necessarily POSIX. (I'm > talking actual FreeBSD behavior here, not necessarily documented > behavior.) > > It turns out that https://www.freebsd.org/cgi/man.cgi?dd(1) does say: > > Normally, data resulting from input or conversion or both are aggregated > into output blocks of the specified size. After the end of input is > reached, any remaining output is written as a block. This means that the > final output block may be shorter than the output block size. > > where conv=sync is described as: > > sync > Pad every input block to the input buffer size. Spaces > are used for pad bytes if a block oriented conversion > value is specified, otherwise NUL bytes are used. > > (So sync changes the overall input size when it adds pad bytes, at > least as documented.) > > It appears to me that Ian's expectations of the actual behavior and > the FreeBSD documentation of such are not a match. Absent testing, > I'd bet on Ian, even if it means FreeBSD dd has a bug. So I tested: root@x230a:/tmp/ddtest # dd if=short of=long bs=1m 0+1 records in 0+1 records out 1024 bytes transferred in 0.000101 secs (10171444 bytes/sec) root@x230a:/tmp/ddtest # ls -lag total 62 drwxr-xr-x 2 root wheel 4 Apr 20 19:16 . drwxrwxrwt 44 root wheel 283 Apr 20 19:16 .. -rw-r--r-- 1 root wheel 1024 Apr 20 19:16 long -rw-r--r-- 1 root wheel 1024 Apr 20 19:16 short And on files it is as expected, however the issue Ian describes does infact occur when dealing with block devices that can not do the "truncated" size block operation, which is extremly rare as the block devices are typically either 512, 2048 or 4096 and almost everyone using dd uses block sizes that are large multiples of this so you never end up hitting that edge of trying to write a 1k record to a 4k device. The truncated input record size is garanteed to be a mutliple of the device block size, the one case I can think of that would hit this is dd'ing a 512B disk to a 4096 native disk that happened to NOT be of a size that is a multiple of 4096, something probably VERY rare. In the case of going from a device to a file the chances of lost data are nill (and I have been doing that for 30 years and never hit it.) You DO have to be carefull if your doing forinsics, conv=sync infact can corrupt your forinsics copy in that it is the wrong size and then when someone else does the analysis they get a different results do to this size change. Part of this process usually involves verifying that the size of your image exactly matches the reported size of the device. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Sat Apr 20 20:31:55 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8243157521F for ; Sat, 20 Apr 2019 20:31:55 +0000 (UTC) (envelope-from guru@unixarea.de) Received: from ms-10.1blu.de (ms-10.1blu.de [178.254.4.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3146670D7E for ; Sat, 20 Apr 2019 20:31:53 +0000 (UTC) (envelope-from guru@unixarea.de) Received: from [188.174.55.23] (helo=localhost.unixarea.de) by ms-10.1blu.de with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.86_2) (envelope-from ) id 1hHweH-0004Bd-Gb for freebsd-hackers@freebsd.org; Sat, 20 Apr 2019 22:31:45 +0200 Received: from localhost.my.domain (localhost [127.0.0.1]) by localhost.unixarea.de (8.15.2/8.14.9) with ESMTPS id x3KKViOS003495 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 20 Apr 2019 22:31:44 +0200 (CEST) (envelope-from guru@unixarea.de) Received: (from guru@localhost) by localhost.my.domain (8.15.2/8.14.9/Submit) id x3KKVi58003494 for freebsd-hackers@freebsd.org; Sat, 20 Apr 2019 22:31:44 +0200 (CEST) (envelope-from guru@unixarea.de) X-Authentication-Warning: localhost.my.domain: guru set sender to guru@unixarea.de using -f Date: Sat, 20 Apr 2019 22:31:44 +0200 From: Matthias Apitz To: freebsd-hackers@freebsd.org Subject: Re: bhyve VM stopped to boot after moving virtio disks Message-ID: <20190420203144.GA3390@c720-r342378> Reply-To: Matthias Apitz Mail-Followup-To: freebsd-hackers@freebsd.org References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Operating-System: FreeBSD 13.0-CURRENT r342378 (amd64) X-message-flag: Mails containing HTML will not be read! Please send only plain text. User-Agent: Mutt/1.11.1 (2018-12-01) X-Con-Id: 51246 X-Con-U: 0-guru X-Originating-IP: 188.174.55.23 X-Rspamd-Queue-Id: 3146670D7E X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-4.71 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[guru@unixarea.de]; HAS_XOIP(0.00)[]; IP_SCORE(-3.10)[ip: (-8.94), ipnet: 178.254.0.0/19(-3.65), asn: 42730(-2.90), country: DE(-0.01)]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[mail.unixarea.de]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; RECEIVED_SPAMHAUS_PBL(0.00)[23.55.174.188.zen.spamhaus.org : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:42730, ipnet:178.254.0.0/19, country:DE]; RCVD_IN_DNSWL_LOW(-0.10)[101.4.254.178.list.dnswl.org : 127.0.5.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[unixarea.de]; R_SPF_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 20:31:56 -0000 El día sábado, abril 20, 2019 a las 10:56:45a. m. -0600, Ian Lepore escribió: > > no it wasn't. but dd doesn't truncate last block. checked - it got > > copied > > till the end - i see second copy of GPT partition table that was > > present > > in VM. > > > > dd absolutely will fail to copy the last block of the source if it > isn't exactly the blocksize and you didn't specify conv=sync, and it > will return a zero status when doing so. It appears you've convinced > yourself otherwise, but for anyone else reading this thread, be aware: > conv=sync is required to copy the last part of the source if it's > smaller than the blocksize. Writing to a plain file or char device, will write the last block in the available size of bytes: $ truss dd if=backupKDE-20190419.tgz of=/dev/null bs=8m ... read(3,"\M^_\M^_\M^Wi\M-F\^?\n\M-J\M-%T"...,8388608) = 8388608 (0x800000) write(4,"\M^_\M^_\M^Wi\M-F\^?\n\M-J\M-%T"...,8388608) = 8388608 (0x800000) read(3,"%\M-b\M^Y!\M^Ud\M^B\\\M^BA \^^"...,8388608) = 8169189 (0x7ca6e5) ^^^^^^^ read(3,0x8015cb065,8388608) = 0 (0x0) write(4,"%\M-b\M^Y!\M^Ud\M^B\\\M^BA \^^"...,8169189) = 8169189 (0x7ca6e5) ^^^^^^^ close(4) = 0 (0x0) clock_getres(0x4,0x7fffffffe3c8) = 0 (0x0) 20+1 records in 20+1 records out write(2,"20+1 records in\n20+1 records ou"...,33) = 33 (0x21) matthias -- Matthias Apitz, ✉ guru@unixarea.de, http://www.unixarea.de/ +49-176-38902045 Public GnuPG key: http://www.unixarea.de/key.pub N € I N zur EU! "Gegen das EU-Europa der Banken, Konzerne und Kriegstreiber. Für ein soziales und friedliches Europa der Völker." DKP From owner-freebsd-hackers@freebsd.org Sat Apr 20 20:36:20 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EDAD1575457 for ; Sat, 20 Apr 2019 20:36:20 +0000 (UTC) (envelope-from mozolevsky@gmail.com) Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B103E70FFB; Sat, 20 Apr 2019 20:36:19 +0000 (UTC) (envelope-from mozolevsky@gmail.com) Received: by mail-ot1-f65.google.com with SMTP id 64so6768547otb.8; Sat, 20 Apr 2019 13:36:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KJ+//qPz+3Eyd3XANL/Jk+NqVmkj1LNs+Vn+ZuG0sGw=; b=lvPt6psDczKweKEkI7tbJh7mfjErP3jv7ngEg9DcciY1g7AqCM8XKpLpzkSHEaRwOI 6PW/ZZ37YKYH77JMmTkwad9+tFbx1NsD9cmZM5WlaVPmUw4dFp6yBa+Adlol79M70yzy i4Hd2o6MoATlNsv2d9XbsrfTcBP0EjVGVafdOAP4YPAXN8y5Zu0yXfvNoJZlRHh0ii69 TpAxIbLr6NvAJySjvRWKi1ZLR93ZfoEj3XnbrqdO12YNgN8KVb0ZrjbD8kbDvO/FdsS8 ix8yBeMFBVGxI+3tPnQc75mTntio6wVxlz3vbpSFwnZXF6gNXe41oZ0nubIgTPh2MiMc VDGw== X-Gm-Message-State: APjAAAUHCV5abhmhKyywfuOTkm8pmNQdmCXnURA4gRHc7AbJe7RWwKOg l51Sjr7F9BmY4o2bsqRnhQ8sOX1GBF6OPAQ0N8QbYQ== X-Google-Smtp-Source: APXvYqxxvAAbE4TcYJ+QFtj9f+QwStdXT2KD9wFsivL3zi+224frioZMQ7JO8m9jw6k3/OdO6LX15YFk1TM1FZIMATs= X-Received: by 2002:a9d:6292:: with SMTP id x18mr6795060otk.224.1555792573315; Sat, 20 Apr 2019 13:36:13 -0700 (PDT) MIME-Version: 1.0 References: <2006ff8da153bbe5e7f620da9260ee1518bd248f.camel@freebsd.org> In-Reply-To: From: Igor Mozolevsky Date: Sat, 20 Apr 2019 21:35:36 +0100 Message-ID: Subject: Re: bhyve VM stopped to boot after moving virtio disks To: Ian Lepore Cc: Hackers freeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: B103E70FFB X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mozolevsky@gmail.com designates 209.85.210.65 as permitted sender) smtp.mailfrom=mozolevsky@gmail.com X-Spamd-Result: default: False [-4.21 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[hybrid-lab.co.uk]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.24)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.26), country: US(-0.06)]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[65.210.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.96)[-0.960,0]; FORGED_SENDER(0.30)[igor@hybrid-lab.co.uk,mozolevsky@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[igor@hybrid-lab.co.uk,mozolevsky@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 20:36:20 -0000 On Sat, 20 Apr 2019 at 19:29, Ian Lepore wrote: > > On Sat, 2019-04-20 at 18:45 +0100, Igor Mozolevsky wrote: > > On Sat, 20 Apr 2019 at 18:00, Ian Lepore wrote: > > > > > > > dd absolutely will fail to copy the last block of the source if it > > > isn't exactly the blocksize and you didn't specify conv=sync, and > > > it > > > will return a zero status when doing so. It appears you've > > > convinced > > > yourself otherwise, but for anyone else reading this thread, be > > > aware: > > > conv=sync is required to copy the last part of the source if it's > > > smaller than the blocksize. > > > > > > Isn't that contrary to the POSIX Spec? Reading the manual [1], it is > > implied (from STDERR part of the main section) that dd will read and > > write partial blocks, cf. truncated blocks?.. > > > > 1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html > > > > > > Damn, why do people trim away the entire useful context of an email > thread when replying? I almost don't want to reply to this because at > this point all the useful info is gone. I only trimmed because you were appearing to be speaking generally and not specifically, so the trimmed part was superfluous. > In any case, in retrospect, what I said was wrong. Using conv=sync is > important when the output is a device that has fixed block sizes where > you cannot write a partial block. When the destination is a file where > short writes work correctly, it will copy the entire source correctly. Thanks for clarifying, I was starting to think I misread the POSIX standard when implementing a local version of dd. -- Igor M.