From nobody Mon May 6 04:29:14 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXpMn3pCyz5J8kQ; Mon, 06 May 2024 04:29:25 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Received: from wes1-so2.wedos.net (wes1-so2-redir.wedos.net [46.28.106.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VXpMm23dhz4V6Y; Mon, 6 May 2024 04:29:24 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shared.dkim-wes1.wedos.net header.s=bravo header.b=eRxFrfQk; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@sysctl.cz has no SPF policy when checking 46.28.106.46) smtp.mailfrom=freebsd@sysctl.cz Received: from webmail.wedos.net (wes1-wm3.wedos.net [46.28.106.84]) by wes1-so2.wedos.net (Postfix) with ESMTPSA id 4VXpMb100gz9dS; Mon, 6 May 2024 06:29:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=shared.dkim-wes1.wedos.net; s=bravo; t=1714969755; bh=BbuaOJEVB1ubre8s2RiXzNhlWh4HRPLMQWRVm++hodY=; h=Date:From:To:Subject; b=eRxFrfQkECva/BIXVFVT0PpPPJ5XLSyw5ezpBPzsToqtq5L6t0EakyOFpjoH4UrA6 0KDFKI82XwqDhiUsMWdJAvNE4V2EtXFJYnN9E9ntSVLFNHgUCABv5N4k/CCILH2n1S br0rqSpy1jB3ga3V7BEwUuBdHyfvdI9IOgr1V8iM= List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 06 May 2024 06:29:14 +0200 From: freebsd@sysctl.cz To: Freebsd hackers , freebsd-current Subject: Update browsers, new browser Message-ID: <5143f07ff441c017929328cb666c9a8a@sysctl.cz> X-Sender: freebsd@sysctl.cz User-Agent: Roundcube Webmail/1.2.4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.18 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[wedos.net:dkim]; NEURAL_HAM_SHORT(-0.98)[-0.982]; R_DKIM_ALLOW(-0.20)[shared.dkim-wes1.wedos.net:s=bravo]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; FROM_NO_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:197019, ipnet:46.28.104.0/21, country:CZ]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; DMARC_NA(0.00)[sysctl.cz]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[shared.dkim-wes1.wedos.net:+] X-Rspamd-Queue-Id: 4VXpMm23dhz4V6Y Hi, i updated tor-browser to version 13.0.14, i am working on update librewolf and new browser you can test is waterfox in bugzilla. have nice day. M.F From nobody Wed May 8 04:27:47 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZ2FK3vj1z5KClP for ; Wed, 08 May 2024 04:28:05 +0000 (UTC) (envelope-from john@sanren.ac.za) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZ2FH3QVyz4V9G for ; Wed, 8 May 2024 04:28:02 +0000 (UTC) (envelope-from john@sanren.ac.za) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sanren-ac-za.20230601.gappssmtp.com header.s=20230601 header.b=iyptFdRH; dmarc=none; spf=pass (mx1.freebsd.org: domain of john@sanren.ac.za designates 2607:f8b0:4864:20::1032 as permitted sender) smtp.mailfrom=john@sanren.ac.za Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-2b6208c88dcso284049a91.3 for ; Tue, 07 May 2024 21:28:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sanren-ac-za.20230601.gappssmtp.com; s=20230601; t=1715142479; x=1715747279; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=eztPO1vMP1efv3aXBLsUkENn1KhcbFLYJJGaQxocifg=; b=iyptFdRHy0RJp5eFxCK8I/gZ87UJjR+eakcsOPh9KcJbCSahkhbTqmfbzIMlwhCUp8 rO4LgHyAmU1ULcMsrpQVKtk4mZjVae6IBGJCzMNlVOgziqJXnG1NRNq49Jbm2gFhZrZC FKDPLTBJsWISdOXwDwKh5SWHD+5uyWxod19vVEeHwX8vrwt/QyhZKJhp5ZHAwgKRwDSm TYf2XffLJhvGxDf7rutsafimMK1RE/qCUpxIyzI0rRjG7DJ3sMf514/qu6aLisMX5uev UnoiIQO9ldVm97ogBYhqisJEnANiJ3isAfWcG3ZlxknUo9PRIePwzsvlKGGbOPip1Z1D b3Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715142479; x=1715747279; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eztPO1vMP1efv3aXBLsUkENn1KhcbFLYJJGaQxocifg=; b=WH6Akh6ncsOke2dy0xLyc4uAQkdyedsD2w98GQF+ETnaW2RCpU7tIEVcIm2cQC8VFl 4jR5Pt0DA1C5OF3JdN8fK7CiJu16DeTcal8CD7NfNyeHe5wl1/ux5CIsSjbs/iw1Sxq+ brdYuaojNK6VbK0hRbblTaip/JVs2OL8pcc2Op69hjbmX9+jtCpINC1wAYbZFrVyqE73 SJxxRFTtD56vliVN4jJQ185q7vzluRoVZMikz8GwfCjTNZC5zv4yKU/fq1CVD/K6Bzts aKn/Or6M0Au9MhXdH+FaMSWB+5zwUnU/fzAUwJml5AR1z2on+9qnB0WupwUvU+Sl9CvO 84oQ== X-Gm-Message-State: AOJu0Yxhl6EJUE+uDPwC9PfgPft8ambmAH+Z4+wSDBzOWSUl72poZb1g Jxqib89PF8Qdje9OoOJGveNnIkxSfyPRAExaw8YTFIH0Lyn/lq7L45bmArBCRUSKJkq0xxHrnA7 HBWEITo4x2jisioS/+Km6j4PhbFqTwRWcZQyUAGDESFnfTrOw X-Google-Smtp-Source: AGHT+IExVo1ZtJCUdP7+z1ljjZO2kyBeAREHizOVF9lHhwEkxEIVsqtYxfSGZpn3Ar4W6/kSr+bTYVu0mBAhZaWiXR4= X-Received: by 2002:a17:90b:1207:b0:2a4:833f:2c1b with SMTP id 98e67ed59e1d1-2b61639cb22mr1703548a91.9.1715142479606; Tue, 07 May 2024 21:27:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: John Hay Date: Wed, 8 May 2024 06:27:47 +0200 Message-ID: Subject: FreeBSD driver for the OCP TAP Time Card To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="0000000000003c906e0617e9ba58" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[sanren-ac-za.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[sanren.ac.za]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1032:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[sanren-ac-za.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4VZ2FH3QVyz4V9G --0000000000003c906e0617e9ba58 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I have been working on a FreeBSD driver for the Open Compute Project (OCP) Time Appliance Project (TAP) Time Card. The card consists of three main parts, a clock module, a GNSS module and a FPGA module. The firmware for the FPGA implements a counter that is synchronized to TAI using the GNSS module and the clock module. The counter is implemented as two 32-bit registers, seconds and nanoseconds, like struct timespec, and make that available on the pci-e bus. More about the Time Card is available at: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Ti= me-Card The driver / software consists of two main parts: The kernel module timecard(4): - Provides access to the card=E2=80=99s functionality, - Implements a timecounter, =E2=80=9CTimeCard=E2=80=9D, using the timecount= ers(4) API, - Make the PPS signal and timestamp available with the RFC 2783 Pulse Per Second (PPS) API, - Provides a timecard bus to which uart(4), iicbus(4) and spibus(4) drivers can attach. A daemon timecard(8): - Provides a shm(28) driver for ntpd(8), - Optionally synchronize the kernel time to the timecard(4) time, - Discipline or train the clock on board of the TimeCard through the iic(4) Clock interface. With the above software and ntpd running and disciplining the kernel time, the kernel time can stay within +-2ns according to ntpd. If neither ntpd nor timecard(8) is set to discipline the kernel time, it will slowly drift away because there is a small rounding error when converting from timecounter nanoseconds read to bintime and then later to timespec. When ntpd is set to discipline the kernel time and it has settled, ntpq=E2=80=99= s kerninfo command will report a pll frequency of 1.52588e-05 (ppm). The card is Precision Time Protocol (PTP) capable, but the driver does not support that. The driver is available at: https://github.com/JohnHay/TimeCard/ Regards John --0000000000003c906e0617e9ba58 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I have been working on a FreeBSD driver for the= Open Compute Project (OCP) Time Appliance Project (TAP) Time Card. The car= d consists of three main parts, a clock module, a GNSS module and a FPGA mo= dule. The firmware for the FPGA implements a counter that is synchronized t= o TAI using the GNSS module and the clock module. The counter is implemente= d as two 32-bit registers, seconds and nanoseconds, like struct timespec, a= nd make that available on the pci-e bus.

More about the Time Card is= available at:
https://github.com/opencomputeproject= /Time-Appliance-Project/tree/master/Time-Card

The driver / softw= are consists of two main parts:

The kernel module timecard(4):
- = Provides access to the card=E2=80=99s functionality,
- Implements a time= counter, =E2=80=9CTimeCard=E2=80=9D, using the timecounters(4) API,
- Ma= ke the PPS signal and timestamp available with the RFC 2783 Pulse Per Secon= d (PPS) API,
- Provides a timecard bus to which uart(4), iicbus(4) and s= pibus(4) drivers can attach.

A daemon timecard(8):
- Provides a s= hm(28) driver for ntpd(8),
- Optionally synchronize the kernel time to t= he timecard(4) time,
- Discipline or train the clock on board of the Tim= eCard through the iic(4) Clock interface.

With the above software an= d ntpd running and disciplining the kernel time, the kernel time can stay w= ithin +-2ns according to ntpd. If neither ntpd nor timecard(8) is set to di= scipline the kernel time, it will slowly drift away because there is a smal= l rounding error when converting from timecounter nanoseconds read to binti= me and then later to timespec. When ntpd is set to discipline the kernel ti= me and it has settled, ntpq=E2=80=99s kerninfo command will report a pll fr= equency of 1.52588e-05 (ppm).

The card is Precision Time Protocol (P= TP) capable, but the driver does not support that.

The driver is ava= ilable at:

Regards

John

--0000000000003c906e0617e9ba58-- From nobody Wed May 8 05:35:15 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZ3l31yVjz5KKwZ for ; Wed, 08 May 2024 05:35:27 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZ3l26p9wz4b8b for ; Wed, 8 May 2024 05:35:26 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 68EFF89287; Wed, 08 May 2024 05:35:17 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 4485ZGsT066674 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 8 May 2024 05:35:17 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 4485ZFN9066673; Wed, 8 May 2024 05:35:15 GMT (envelope-from phk) Message-Id: <202405080535.4485ZFN9066673@critter.freebsd.dk> To: John Hay cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD driver for the OCP TAP Time Card In-reply-to: From: "Poul-Henning Kamp" References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <66671.1715146515.1@critter.freebsd.dk> Date: Wed, 08 May 2024 05:35:15 +0000 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU] X-Rspamd-Queue-Id: 4VZ3l26p9wz4b8b -------- John Hay writes: > I have been working on a FreeBSD driver for the Open Compute Project (OCP) > Time Appliance Project (TAP) Time Card. The card consists of three main > parts, a clock module, a GNSS module and a FPGA module. The firmware for > the FPGA implements a counter that is synchronized to TAI using the GNSS > module and the clock module. The counter is implemented as two 32-bit > registers, seconds and nanoseconds, like struct timespec, and make that > available on the pci-e bus. That is /precisely/ the kind of hardware timecounters were designed for :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nobody Wed May 8 06:36:53 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZ56D6wnwz5KRb2 for ; Wed, 08 May 2024 06:37:08 +0000 (UTC) (envelope-from john@sanren.ac.za) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZ56D53ptz4gjc for ; Wed, 8 May 2024 06:37:08 +0000 (UTC) (envelope-from john@sanren.ac.za) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-2b433dd2566so2865924a91.2 for ; Tue, 07 May 2024 23:37:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sanren-ac-za.20230601.gappssmtp.com; s=20230601; t=1715150226; x=1715755026; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=e5wwxomo9diVvQllS93KtqgRTmsgU+YZOSEg5yWGBMM=; b=coR+rKKbYvDy4Pe2c8tDSqDXRIDidSxLaVUmOpnea6MxrcB4wiH+P4JzYyGYltMCCO G++cWLWHX5pSlHB7ElfP0n+0p7DJr2YVmZ4JHt0rlDA2Z+WCsAI/sv6OLONtdEwRQS5k x2J4UtC9/1M5GpOywllEHJ3E6E6N81FD+iRMloyaQ5k1ck9NQp4usfKFe+0ZW/5gcDmq 1kHoN+rXls1aZfjZy44v9FcvS5XEryWmvYq2vOVpNjUuYh6KmHuKo729dfXNEzH2D6jf x0cFVC4xxZw08sCm1L39tnwnxGepRSdyxKpusLg743dcsfHBpXE7c8mXcNWIvBusVcKV bavQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715150226; x=1715755026; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=e5wwxomo9diVvQllS93KtqgRTmsgU+YZOSEg5yWGBMM=; b=dPybc8xmQqF9kMWdYsxvg9YwwHW7NAWdTGi0M8j7pV47ibjtNaMB4MktJRFg4vhskU eD/cyZ9jWe4cKV5Jz1zYYvUHe+5s1yDGDpmwgLexpjkDURPyY19rxjB1qLa3zVt1B9MZ 1e1E8SzGRk/3yYkzPwOnSi2eYN98aP/sWUOa/dWiSSVMzboiTKyqzcR9jUWF24KrWtS7 I7ZuOcXX3myMIg7Q3Nl7NEpB/REs3wwTZb8P9dP3M7kaPdRVDSJ7WkTKaezg3akCmAj5 +muqn8jnupXlB+9t8FJ3nnRcg5IzGy4d9GmCzBlVKKSIvioFHYEor/S/2Z8uElC2Qq9b WRlw== X-Gm-Message-State: AOJu0YyiH3O6f3zsXMm8a+6xvm+ckQ6zQw8G0Q+68W8FLhR6Go8mQPnd u5fhjfmgastMC12oIa8+lqfVoV6XbRYm+mXHs4NJxJPiIAmZZ9iv76ymN//k0vgGCsfp2y2ZzxR qBYvT26349GFBuqmoZXqbkxp96WKGWEIex9aTheWn1XjpRKV0HkY= X-Google-Smtp-Source: AGHT+IEs/uE9TSjEdeoRqF+2/5U9+kmXhMEO0Hl5c4z42aFEGnRaa2ly1k7bX+O6mpeDz2qCEvtxHJUh/FaKHR3s81Q= X-Received: by 2002:a17:90a:c590:b0:2b2:9d08:80a1 with SMTP id 98e67ed59e1d1-2b6169d9452mr1667386a91.33.1715150225657; Tue, 07 May 2024 23:37:05 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: <202405080535.4485ZFN9066673@critter.freebsd.dk> In-Reply-To: <202405080535.4485ZFN9066673@critter.freebsd.dk> From: John Hay Date: Wed, 8 May 2024 08:36:53 +0200 Message-ID: Subject: Re: FreeBSD driver for the OCP TAP Time Card To: Poul-Henning Kamp Cc: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000efee4d0617eb87d5" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4VZ56D53ptz4gjc --000000000000efee4d0617eb87d5 Content-Type: text/plain; charset="UTF-8" Hi Poul-Henning, On Wed, 8 May 2024 at 07:35, Poul-Henning Kamp wrote: > -------- > John Hay writes: > > > I have been working on a FreeBSD driver for the Open Compute Project > (OCP) > > Time Appliance Project (TAP) Time Card. The card consists of three main > > parts, a clock module, a GNSS module and a FPGA module. The firmware for > > the FPGA implements a counter that is synchronized to TAI using the GNSS > > module and the clock module. The counter is implemented as two 32-bit > > registers, seconds and nanoseconds, like struct timespec, and make that > > available on the pci-e bus. > > That is /precisely/ the kind of hardware timecounters were designed for :-) > True, it did make it a lot easier. :-) Working close to the nanosecond does seem to push things close to the limits though. :-) There are a few things I wish could be handled differently. Maybe there are ways, and I just did not find them. :-) One is that you cannot just feed timecounters with a timespec value. It assumes the hardware counter is in binary, while in this case, the nanoseconds rolls over at 1 second, so for every tc_get_timecount(), you have to multiply the seconds register value by 10^9 and then add the nanosecond register value. Not a big deal and reading the registers over the pci-e bus is the slowest part by far. The other is that the conversion from the above value to bintime and later back to what is used elsewhere, seems to lose a little precision. The comments in sys/kern/kern_tc.c also note that. In the pps code, I wished that one could provide a timestamp with pps_capture(). It uses the time at which it handled the interrupt. The card latch the counter values when the pps happens, so it knows the correct time. Currently I hack around it by twiddling sc->sc_pps_state.capcount directly. Regards John --000000000000efee4d0617eb87d5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Poul-Henning,

On Wed, 8 May 2024 a= t 07:35, Poul-Henning Kamp <phk@ph= k.freebsd.dk> wrote:
--------
John Hay writes:

> I have been working on a FreeBSD driver for the Open Compute Project (= OCP)
> Time Appliance Project (TAP) Time Card. The card consists of three mai= n
> parts, a clock module, a GNSS module and a FPGA module. The firmware f= or
> the FPGA implements a counter that is synchronized to TAI using the GN= SS
> module and the clock module. The counter is implemented as two 32-bit<= br> > registers, seconds and nanoseconds, like struct timespec, and make tha= t
> available on the pci-e bus.

That is /precisely/ the kind of hardware timecounters were designed for :-)=

True, it did make it a lot easier. :-)= Working close to the nanosecond does seem to push things close to the limi= ts though. :-)

There are a few things I wish c= ould be handled differently. Maybe there are ways, and I just did not find = them. :-)

One is that you cannot just feed tim= ecounters with a timespec value. It assumes the hardware counter is in bina= ry, while in this case, the nanoseconds rolls over at 1 second, so for ever= y tc_get_timecount(), you have to multiply the seconds register value by 10= ^9 and then add the nanosecond register value. Not a big deal and reading t= he registers over the pci-e bus is the slowest part by far.
<= br>
The other is that the conversion from the above value to bint= ime and later back to what is used elsewhere, seems to lose a little precis= ion. The comments in sys/kern/kern_tc.c also note that.

=
In the pps code, I wished that one could provide a timestamp wit= h pps_capture(). It uses the time at which it handled the interrupt. The ca= rd latch the counter values when the pps happens, so it knows the correct t= ime. Currently I hack around it by twiddling sc->sc_pps_state.capcount d= irectly.

Regards

John
--000000000000efee4d0617eb87d5-- From nobody Wed May 8 07:05:53 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZ5lV2r5Lz5KTnw for ; Wed, 08 May 2024 07:05:58 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZ5lV0hylz4kwk for ; Wed, 8 May 2024 07:05:57 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id B329389293; Wed, 08 May 2024 07:05:55 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 44875tht067083 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 8 May 2024 07:05:55 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 44875rwT067082; Wed, 8 May 2024 07:05:53 GMT (envelope-from phk) Message-Id: <202405080705.44875rwT067082@critter.freebsd.dk> To: John Hay cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD driver for the OCP TAP Time Card In-reply-to: From: "Poul-Henning Kamp" References: <202405080535.4485ZFN9066673@critter.freebsd.dk> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <67080.1715151953.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Wed, 08 May 2024 07:05:53 +0000 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU] X-Rspamd-Queue-Id: 4VZ5lV0hylz4kwk -------- John Hay writes: > The other is that the conversion from the above value to bintime and lat= er > back to what is used elsewhere, seems to lose a little precision. The > comments in sys/kern/kern_tc.c also note that. Yes once the /resolution/ of the hardware gets into the nanosecond domain, some of that resolution is lost, because a 64x64=3D>128 multiplication would have been both surplus to requirements and beyond the pale. Getting rid of 32bit platforms moves the needle, but it may still not be warranted. I will also note that most people who think they are approaching nanosecond /precision/ are wrong about that: Only a few of the institutions listed in Circular T manage it. The important thing is to make sure the added noise is bias free. You should capture some millions of the the difference between the HW counter and the hw->timecounter->bintime->timespec result, and run that through the usual battery (Histogram, MVAR, FFT etc.) > In the pps code, I wished that one could provide a timestamp with pps_ca= pture > (). It uses the time at which it handled the interrupt. The card latch t= he > counter values when the pps happens, so it knows the correct time. Curre= ntly I > hack around it by twiddling sc->sc_pps_state.capcount directly. That is not hacking around, that is how it is done :-) See for instance i386/i386/elan-mmcr.c -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From nobody Wed May 8 12:27:04 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZDtJ3LzMz5J1qT for ; Wed, 08 May 2024 12:27:20 +0000 (UTC) (envelope-from john@sanren.ac.za) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZDtH3PXCz4L5y for ; Wed, 8 May 2024 12:27:18 +0000 (UTC) (envelope-from john@sanren.ac.za) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-2b338460546so3255041a91.1 for ; Wed, 08 May 2024 05:27:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sanren-ac-za.20230601.gappssmtp.com; s=20230601; t=1715171237; x=1715776037; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0Nac6YtuCYDUXrC3MQf9Hc2zfDL4wtPoV55a5akAw/w=; b=YFeMPYtJl1VWveicg1eALnRAWOkwZA78beSIYrXa4wBTUB+8FUP6BM73l3LHu9yr7o aG+t8NYxQKz3GmmMQrGsWv4Nf3Q/o5wiTbFc1Jc/8ViTmYicyeuBf8gMwWpjVKBMJgkg EQ/Uy3FQV+ILcR8M16gKCg8SDqD8SzSkM0bMHHB8xaX2Md0EOvNyodnO2jBKvnKZSxKR bUFEOpC7W0qIN0r2ge0zYbsTUT5nLfICoAc+bjiZSDir/9EK478l1CYIbQ+1w/5RTT2T 7wK+N2efXLmX21avmutbAzerGKgAwNiYc7LTmAfuP+hPq2Ak+f8oolrcxVxBz4wxH6Rz B6lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715171237; x=1715776037; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0Nac6YtuCYDUXrC3MQf9Hc2zfDL4wtPoV55a5akAw/w=; b=pB9ERAde7tIGVQemks4bB/stNDyWeahJHbuXoHRLjs4l1qY7ei+9JWb4TwLZYfQdXE h31xiQscfVeCZ8TE0IyW6NQS+2FGIgeeB1JysiygfbIXcH1o3c+CFhtZLc6YvmUIa9wW bpOdQvqu+e3aA/9P/zTjxlCQkaFDJnVrcgNjjaymztG6uHzS/pO3+EpNs1ezBE3awBfJ VPytdQ8tyWQNkQYbsi54GK0xipFQEJNblmZzctMZyez39iZSzUL9bL6iaduHNDxetIXN RAgasXYbtZbLo/Phuq1ihnSaySgUwZ8V5QCgo706z5I+6VV/nHaHV1lLDvcc9QyzmQsm BPqw== X-Gm-Message-State: AOJu0YwWf1OziJfjtfq5pbDryJoynkgo8yWh/HIsz11qQ9BxVFC4HNdc P8bmEJ/qLNLi+dWvF7cRT0iHqt90E/tyRRoW8GpEBae17Ge9r+Rj+sZb4qvN3gSXfj6cr77EGBk sGVI+x18/e43ZuC2hIAuqIWsCz1cuBkk4F126QEvQfsL/vs1U X-Google-Smtp-Source: AGHT+IEL6857HsHKUbRJcoHY+kf1rRqgOGf8szsvfPdZQZJpPwdQwKauSNSngwPL5c82i/CWpRPF77vqonE27zAk+fk= X-Received: by 2002:a17:90b:46d2:b0:2b0:e2cf:118b with SMTP id 98e67ed59e1d1-2b6169db57fmr2332636a91.33.1715171236804; Wed, 08 May 2024 05:27:16 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: <202405080535.4485ZFN9066673@critter.freebsd.dk> <202405080705.44875rwT067082@critter.freebsd.dk> In-Reply-To: <202405080705.44875rwT067082@critter.freebsd.dk> From: John Hay Date: Wed, 8 May 2024 14:27:04 +0200 Message-ID: Subject: Re: FreeBSD driver for the OCP TAP Time Card To: Poul-Henning Kamp Cc: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004c9a140617f06cc9" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4VZDtH3PXCz4L5y --0000000000004c9a140617f06cc9 Content-Type: text/plain; charset="UTF-8" Hi Poul-Henning, On Wed, 8 May 2024 at 09:05, Poul-Henning Kamp wrote: > -------- > John Hay writes: > > > The other is that the conversion from the above value to bintime and > later > > back to what is used elsewhere, seems to lose a little precision. The > > comments in sys/kern/kern_tc.c also note that. > > Yes once the /resolution/ of the hardware gets into the nanosecond > domain, some of that resolution is lost, because a 64x64=>128 > multiplication would have been both surplus to requirements and > beyond the pale. > Yes, resolution is what I meant. > Getting rid of 32bit platforms moves the needle, but it may still > not be warranted. > > I will also note that most people who think they are approaching > nanosecond /precision/ are wrong about that: Only a few of the > institutions listed in Circular T manage it. > > The important thing is to make sure the added noise is bias free. > I have 3 machines with Time Cards installed. If I use the TimeCard as timecounter and do not discipline the kernel time, it will slowly drift away. If I use ntpd to discipline it, all three machines settle on a pll frequency of 1.52588e-05 (ppm) according to ntpq -c kerninfo: > ntpq -c kerninfo associd=0 status=0415 leap_none, sync_uhf_radio, 1 event, clock_sync, pll offset: 0 pll frequency: 1.52588e-05 So just from that perspective, it feels that there is some bias. > You should capture some millions of the the difference between the > HW counter and the hw->timecounter->bintime->timespec result, and > run that through the usual battery (Histogram, MVAR, FFT etc.) > I'll have a look at that. > > > In the pps code, I wished that one could provide a timestamp with > pps_capture > > (). It uses the time at which it handled the interrupt. The card latch > the > > counter values when the pps happens, so it knows the correct time. > Currently I > > hack around it by twiddling sc->sc_pps_state.capcount directly. > > That is not hacking around, that is how it is done :-) > > See for instance i386/i386/elan-mmcr.c > Ah, I forgot about that one and did not look there. :-) Thanks for the reminder. Regards John --0000000000004c9a140617f06cc9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Poul-Henning,

On Wed, 8 May 2024 at 09:05, Poul= -Henning Kamp <phk@phk.freebsd.dk<= /a>> wrote:
-= -------
John Hay writes:

> The other is that the conversion from the above value to bintime and l= ater
> back to what is used elsewhere, seems to lose a little precision. The<= br> > comments in sys/kern/kern_tc.c also note that.

Yes once the /resolution/ of the hardware gets into the nanosecond
domain, some of that resolution is lost, because a 64x64=3D>128
multiplication would have been both surplus to requirements and
beyond the pale.



Getting rid of 32bit platforms moves the needle, but it may still
not be warranted.

I will also note that most people who think they are approaching
nanosecond /precision/ are wrong about that:=C2=A0 Only a few of the
institutions listed in Circular T manage it.

The important thing is to make sure the added noise is bias free.

I have 3 machines with Time Cards installed. If = I use the TimeCard as timecounter and do not discipline the kernel time, it= will slowly drift away. If I use ntpd to discipline it, all three machines= settle on a pll frequency of 1.52588e-05 (ppm) according to ntpq -c kernin= fo:

<snip>
> ntpq -c kernin= fo
associd=3D0 status=3D0415 leap_none, sync_uhf_radio, 1 event, clock_s= ync,
pll offset: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00
pll frequ= ency: =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.52588e-05
</snip>
=

So just from that perspective, it feels that= there is some bias.


You should capture some millions of the the difference between the
HW counter and the hw->timecounter->bintime->timespec result, and<= br> run that through the usual battery (Histogram, MVAR, FFT etc.)

I'll have a look at that.
=C2=A0
<= /div>

> In the pps code, I wished that one could provide a timestamp with pps_= capture
> (). It uses the time at which it handled the interrupt. The card latch= the
> counter values when the pps happens, so it knows the correct time. Cur= rently I
> hack around it by twiddling sc->sc_pps_state.capcount directly.

That is not hacking around, that is how it is done :-)

See for instance i386/i386/elan-mmcr.c

= Ah, I forgot about that one and did not look there. :-) Thanks for the remi= nder.

Regards

John

--0000000000004c9a140617f06cc9-- From nobody Wed May 8 16:57:17 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZLvm59Yvz5JWtS for ; Wed, 08 May 2024 16:59:00 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "weser.webweaving.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZLvl2lsXz4vkc for ; Wed, 8 May 2024 16:58:59 +0000 (UTC) (envelope-from dirkx@webweaving.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=webweaving.org header.s=shared header.b="ppboX8/S"; dmarc=pass (policy=none) header.from=webweaving.org; spf=pass (mx1.freebsd.org: domain of dirkx@webweaving.org designates 148.251.234.232 as permitted sender) smtp.mailfrom=dirkx@webweaving.org Received: from smtpclient.apple (77-63-65-3.mobile.kpn.net [77.63.65.3]) (authenticated bits=0) by weser.webweaving.org (8.17.1/8.17.1) with ESMTPA id 448GvWgU047450 for ; Wed, 8 May 2024 18:57:34 +0200 (CEST) (envelope-from dirkx@webweaving.org) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=webweaving.org; s=shared; t=1715187454; bh=dxVxjGy9wXmBbwd/osod85ftCxx1sGXKnyFG+I+M3sY=; h=From:Subject:Date:To; b=ppboX8/SfrjaW/fCBbMBW6mircG6fRFDQX5duEFptF5Y6SHGQYJOIIFrNRJv0xdVC XwAE455VNjP9P5hD+6cvLBArwwrzqzGXLucCuiHUDX5aosnDFanzKOs3ubJynN70Tu dBDPYAPd/5lXBPTADgCHf8gJIyZVnFxj0lCL7thI= X-Authentication-Warning: weser.webweaving.org: Host 77-63-65-3.mobile.kpn.net [77.63.65.3] claimed to be smtpclient.apple From: Dirk-Willem van Gulik Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: IPv6 and IPv4 combined rules in pf.conf Message-Id: <0C18B410-E90B-4295-B09E-43B48F9191A4@webweaving.org> Date: Wed, 8 May 2024 18:57:17 +0200 To: FreeBSD Hackers X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.4 (weser.webweaving.org [148.251.234.232]); Wed, 08 May 2024 18:57:34 +0200 (CEST) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[webweaving.org,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[webweaving.org:s=shared]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[webweaving.org:+] X-Rspamd-Queue-Id: 4VZLvl2lsXz4vkc For dual stack hosts; with both an IPv4 and IPv6 CIDR that they are = listening to - is there a recommended way to setup pf.conf to avoid = mistakes/duplication ? To avoid duplication in constructs such as: # Foo app servers foobarserver_host4=3D231.17.X.Y foobarserver_host6=3Dfe80::5246:=E2=80=A6 # Load balancers - direct or via tun0 in post/fail-back=20 bar_net=3DX.Y.Z.Z #=20 bar_net6=3Dfe80::5246:=E2=80=A6 #=20 =E2=80=A6 pass in on { tun0, $ext_if } proto udp from $bar_net to = $foobarserver_host4 port 2194 keep state pass in on { tun0, $ext_if } proto udp6 from bar_net6 $var to = $foobarserver_host6 port 2194 keep state Is there some recommended way of doing this in stock FreeBSD ? Or does = one usually end up with some sort of macro/generate style solution ? Dw From nobody Wed May 8 20:14:27 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZRFT46TKz5K6ZS for ; Wed, 08 May 2024 20:14:37 +0000 (UTC) (envelope-from lexi@le-fay.org) Received: from fuchsia.eden.le-Fay.ORG (fuchsia.eden.le-fay.org [IPv6:2001:8b0:aab5:107::11]) by mx1.freebsd.org (Postfix) with ESMTP id 4VZRFT0cLZz4Jnp for ; Wed, 8 May 2024 20:14:37 +0000 (UTC) (envelope-from lexi@le-fay.org) Authentication-Results: mx1.freebsd.org; none Received: from iris.eden.le-Fay.ORG (iris.eden.le-fay.org [IPv6:2001:8b0:aab5:106:3::6]) by fuchsia.eden.le-Fay.ORG (Postfix) with ESMTP id 24B11BB44; Wed, 08 May 2024 20:14:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=le-fay.org; s=fuchsia; t=1715199268; bh=nsBjAu/3bzctlvpky9st3o5ABpYCzJjXV1l2yl/Iyvo=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=hVzbeTZXHHHUuavE25gWvUS5HVcPSIF5gO2LbJcOgAfe2mhOZPUlHHvMG/H+sQh6E v9ymtdHA3VCCpmVO15lvAt7QF6Yoj7Qum1QUk5TjgyDuCpdwSDX+LhKoiRUp3vqGO/ l9s1yiOK1L7W9jDvKDN6DDnjzADX8u5t38UAxsGM= Received: from ilythia.eden.le-fay.org (ilythia.eden.le-fay.org [IPv6:2001:8b0:aab5:106:3::10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by iris.eden.le-Fay.ORG (Postfix) with ESMTPSA id 425572C04D7; Wed, 08 May 2024 21:14:28 +0100 (BST) Date: Wed, 8 May 2024 21:14:27 +0100 From: Lexi Winter To: Dirk-Willem van Gulik Cc: FreeBSD Hackers Subject: Re: IPv6 and IPv4 combined rules in pf.conf Message-ID: References: <0C18B410-E90B-4295-B09E-43B48F9191A4@webweaving.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="UIAKpieOOX7RD9VB" Content-Disposition: inline In-Reply-To: <0C18B410-E90B-4295-B09E-43B48F9191A4@webweaving.org> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20712, ipnet:2001:8b0::/32, country:GB] X-Rspamd-Queue-Id: 4VZRFT0cLZz4Jnp --UIAKpieOOX7RD9VB Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Dirk-Willem van Gulik: > For dual stack hosts; with both an IPv4 and IPv6 CIDR that they are > listening to - is there a recommended way to setup pf.conf to avoid > mistakes/duplication ? =20 > To avoid duplication in constructs such as: =20 > # Foo app servers > foobarserver_host4=3D231.17.X.Y > foobarserver_host6=3Dfe80::5246:=E2=80=A6 >=20 > # Load balancers - direct or via tun0 in post/fail-back=20 > bar_net=3DX.Y.Z.Z #=20 > bar_net6=3Dfe80::5246:=E2=80=A6 #=20 > =E2=80=A6 >=20 > pass in on { tun0, $ext_if } proto udp from $bar_net to $foobarserver_= host4 port 2194 keep state > pass in on { tun0, $ext_if } proto udp6 from bar_net6 $var to $foobarse= rver_host6 port 2194 keep state =20 > Is there some recommended way of doing this in stock FreeBSD ? Or does > one usually end up with some sort of macro/generate style solution ? i would suggest something like this: table { 231.17.X.Y fe80::5246:... } table { ... } pass on { tun0, $ext_if } proto udp from \ to port 2194 alternatively, if 'foobarserver' is the local host, you can simply do: pass in on { tun0, $ext_if } proto udp from \ to self port 2194 note that in either case pf doesn't need 'keep state'. --UIAKpieOOX7RD9VB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCAAdFiEEuwt6MaPcv/+Mo+ftDHqbqZ41x5kFAmY73SAACgkQDHqbqZ41 x5lCVQv/c5UQ0eY0WwkRQki/5hZfME2DFwF7Q/hVTLmeprW+IjNZf5Ufn3bJLeoz walPBYuf0iEuQiSOnDAbk93rMAZO4arts8zIN6VtlnuJ8t2hKkIdaO9hqdae5y7d X7I3Y315Goetjcuqxnn9QaHT7LKTvEGfv58CB0oFtXT4YmoFtmooPSsq6Gps8o4j Aar57QmEBUyoFoqy6x2WdJzyHiolKO1RmpKWQereZJVF/WuJ9W2ljSP9h38XfhyG jszwxmMF26XpPYb7FBhxisrSEyVq9yVOoJ4pNkAC9ysSr14mvoFMcgTyszkwIDGu qnyc2Net45ipIFfEkD3HsHPuAnK2rDIhgj9VaIq+cz6v1KiefMyB1QcmOQ3atS33 D3vclDUahXUk6rpFDqmvGiIgcGvxRNbCxBNP7pFJgRhSpcIhxqB5+oTguVXO/5Ed 6RSMQINdZQJiIqTnxdtLmnYX9inv7qS+j4I4+lRdJgvKqQOdNOwutZMwy3xROdYZ wxHo3BCm =Pijn -----END PGP SIGNATURE----- --UIAKpieOOX7RD9VB-- From nobody Wed May 8 20:41:56 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZRtp6ctdz5K9FJ for ; Wed, 08 May 2024 20:43:30 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "weser.webweaving.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZRtp3DkPz4NR0 for ; Wed, 8 May 2024 20:43:30 +0000 (UTC) (envelope-from dirkx@webweaving.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (77-63-64-246.mobile.kpn.net [77.63.64.246]) (authenticated bits=0) by weser.webweaving.org (8.17.1/8.17.1) with ESMTPA id 448Kg7l0073388; Wed, 8 May 2024 22:42:10 +0200 (CEST) (envelope-from dirkx@webweaving.org) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=webweaving.org; s=shared; t=1715200932; bh=px90PB/14+0miOwSVMd8BGePrI2RA0PdaCKyzuydXnY=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=QT1GuFoG08fQx5T769y5TKNwHYXkNF4zDHs1VdHDMUn44AeLaGhxRL9c4RvSgvHKB vIY89JVnX2KeJ+/gcFqLtt7Uo3QRP/d/7xXoMN7Rpoid2TF0j0mSoXRj50jTo3ovZq CmyJJ+zSeVEHr2olX/KKvBqhytrmEHRBSxBP1dys= X-Authentication-Warning: weser.webweaving.org: Host 77-63-64-246.mobile.kpn.net [77.63.64.246] claimed to be smtpclient.apple From: Dirk-Willem van Gulik Message-Id: <6005DECF-10AA-487F-8F95-317B4227E988@webweaving.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_B1997D92-CFD5-4B7E-81E5-34C3BDBD8C03" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: IPv6 and IPv4 combined rules in pf.conf Date: Wed, 8 May 2024 22:41:56 +0200 In-Reply-To: Cc: FreeBSD Hackers To: Lexi Winter References: <0C18B410-E90B-4295-B09E-43B48F9191A4@webweaving.org> X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.4 (weser.webweaving.org [148.251.234.232]); Wed, 08 May 2024 22:42:12 +0200 (CEST) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE] X-Rspamd-Queue-Id: 4VZRtp3DkPz4NR0 --Apple-Mail=_B1997D92-CFD5-4B7E-81E5-34C3BDBD8C03 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 8 May 2024, at 22:14, Lexi Winter wrote: >=20 > Dirk-Willem van Gulik: >> For dual stack hosts; with both an IPv4 and IPv6 CIDR that they are >> listening to - is there a recommended way to setup pf.conf to avoid >> mistakes/duplication ? >=20 >> To avoid duplication in constructs such as: >=20 >> # Foo app servers >> foobarserver_host4=3D231.17.X.Y >> foobarserver_host6=3Dfe80::5246:=E2=80=A6 >>=20 >> # Load balancers - direct or via tun0 in post/fail-back=20 >> bar_net=3DX.Y.Z.Z #=20 >> bar_net6=3Dfe80::5246:=E2=80=A6 #=20 >> =E2=80=A6 >>=20 >> pass in on { tun0, $ext_if } proto udp from $bar_net to = $foobarserver_host4 port 2194 keep state >> pass in on { tun0, $ext_if } proto udp6 from bar_net6 $var to = $foobarserver_host6 port 2194 keep state >=20 >> Is there some recommended way of doing this in stock FreeBSD ? Or = does >> one usually end up with some sort of macro/generate style solution ? >=20 > i would suggest something like this: >=20 > table { > 231.17.X.Y > fe80::5246:... > } >=20 > table { > ... > } >=20 > pass on { tun0, $ext_if } proto udp from \ > to port 2194 Ok - excellent - =C8=99o one can mix IPv4 and IPv6 in a list - and = =E2=80=98udp=E2=80=99 no longer needs to be =E2=80=98udp6=E2=80=99 (and = same for tcp6 and icmp6 v.s. tcp/icmp_=E2=80=94 pf guesses this right = based on the address ? > note that in either case pf doesn't need 'keep state=E2=80=99. Sorry :) cut and paste of a actual TCP rule edited to protect the = innocent. Dw --Apple-Mail=_B1997D92-CFD5-4B7E-81E5-34C3BDBD8C03 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 8 May 2024, at 22:14, Lexi Winter = <lexi@le-fay.org> wrote:

Dirk-Willem van Gulik:
For dual stack hosts; with both an IPv4 and IPv6 CIDR that they = are
listening to - is there a recommended way to setup pf.conf to = avoid
mistakes/duplication ?

To avoid duplication in constructs such = as:

# Foo app servers
= foobarserver_host4=3D231.17.X.Y
= foobarserver_host6=3Dfe80::5246:=E2=80=A6

# Load = balancers  - direct or via tun0 in post/fail-back 
= bar_net=3DX.Y.Z.Z # 
= bar_net6=3Dfe80::5246:=E2=80=A6 # 
= =E2=80=A6

pass in on { tun0, $ext_if } =  proto udp from $bar_net  to $foobarserver_host4 port 2194 = keep state
pass in on { tun0, $ext_if }  proto udp6 from = bar_net6 $var to $foobarserver_host6 port 2194 keep = state

Is there some recommended way of doing this in = stock FreeBSD ? Or does
one usually end up with some sort of = macro/generate style solution ?

i would = suggest something like this:

= table = <foobarserver> {
= 231.17.X.Y
fe80::5246:...
= }

table <bar-net> = {
...
= }

pass on { tun0, $ext_if = } proto udp from <bar-net> \
= to <foobarserver> = port 2194

Ok - excellent - =C8=99o one = can mix IPv4 and IPv6 in a list - and =E2=80=98udp=E2=80=99 no longer = needs to be =E2=80=98udp6=E2=80=99 (and same for tcp6 and icmp6 v.s. = tcp/icmp_=E2=80=94 pf guesses this right based on the address = ?

note that in = either case pf doesn't need 'keep = state=E2=80=99.

Sorry :) cut and paste = of a actual TCP rule edited to protect the = innocent.

Dw


=


= --Apple-Mail=_B1997D92-CFD5-4B7E-81E5-34C3BDBD8C03-- From nobody Wed May 8 21:19:59 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZSkq67Lyz5KDZT for ; Wed, 08 May 2024 21:21:39 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "weser.webweaving.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZSkp5Nflz4T5J for ; Wed, 8 May 2024 21:21:37 +0000 (UTC) (envelope-from dirkx@webweaving.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=webweaving.org header.s=shared header.b=V8D12mPx; dmarc=pass (policy=none) header.from=webweaving.org; spf=pass (mx1.freebsd.org: domain of dirkx@webweaving.org designates 148.251.234.232 as permitted sender) smtp.mailfrom=dirkx@webweaving.org Received: from smtpclient.apple (77-63-64-246.mobile.kpn.net [77.63.64.246]) (authenticated bits=0) by weser.webweaving.org (8.17.1/8.17.1) with ESMTPA id 448LKEgQ003485; Wed, 8 May 2024 23:20:17 +0200 (CEST) (envelope-from dirkx@webweaving.org) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=webweaving.org; s=shared; t=1715203219; bh=FJTAQnn329muIHbkLe+rSd9eRNneCEDyWU2CqYW7gYE=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=V8D12mPx/QuQ9qxCZ8lSHKcDlFUEgzqHeKpt05+0M5PRA9XHCDCQXPvN9/sqqtUlw k5LRPrJ0S4zxeKG+PrYHglJrsDmTk8F2OyOOvbhvnOvrdhWFd3Kz7enEpXkUdI50Sk Zza2JbfO46USJ8CbWQO6U+fBDMx7kVqySe9aW7NY= X-Authentication-Warning: weser.webweaving.org: Host 77-63-64-246.mobile.kpn.net [77.63.64.246] claimed to be smtpclient.apple From: Dirk-Willem van Gulik Message-Id: <5258A000-3483-467F-8FE9-B3F986D62BB3@webweaving.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_E7A94B88-4767-4051-AD41-387F9DEDB4CA" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: IPv6 and IPv4 combined rules in pf.conf Date: Wed, 8 May 2024 23:19:59 +0200 In-Reply-To: <6005DECF-10AA-487F-8F95-317B4227E988@webweaving.org> Cc: FreeBSD Hackers To: Lexi Winter References: <0C18B410-E90B-4295-B09E-43B48F9191A4@webweaving.org> <6005DECF-10AA-487F-8F95-317B4227E988@webweaving.org> X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.4 (weser.webweaving.org [148.251.234.232]); Wed, 08 May 2024 23:20:19 +0200 (CEST) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[webweaving.org,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[webweaving.org:s=shared]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ONCE_RECEIVED(0.10)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[webweaving.org:+] X-Rspamd-Queue-Id: 4VZSkp5Nflz4T5J --Apple-Mail=_E7A94B88-4767-4051-AD41-387F9DEDB4CA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 8 May 2024, at 22:41, Dirk-Willem van Gulik = wrote: >=20 >> On 8 May 2024, at 22:14, Lexi Winter wrote: >>=20 >> Dirk-Willem van Gulik: >>> For dual stack hosts; with both an IPv4 and IPv6 CIDR that they are >>> listening to - is there a recommended way to setup pf.conf to avoid >>> mistakes/duplication ? >>=20 >>> To avoid duplication in constructs such as: >>=20 >>> # Foo app servers >>> foobarserver_host4=3D231.17.X.Y >>> foobarserver_host6=3Dfe80::5246:=E2=80=A6 >>>=20 >>> # Load balancers - direct or via tun0 in post/fail-back=20 >>> bar_net=3DX.Y.Z.Z #=20 >>> bar_net6=3Dfe80::5246:=E2=80=A6 #=20 >>> =E2=80=A6 >>>=20 >>> pass in on { tun0, $ext_if } proto udp from $bar_net to = $foobarserver_host4 port 2194 keep state >>> pass in on { tun0, $ext_if } proto udp6 from bar_net6 $var to = $foobarserver_host6 port 2194 keep state >>=20 >>> Is there some recommended way of doing this in stock FreeBSD ? Or = does >>> one usually end up with some sort of macro/generate style solution ? >>=20 >> i would suggest something like this: >>=20 >> table { >> 231.17.X.Y >> fe80::5246:... >> } >>=20 >> table { >> ... >> } >>=20 >> pass on { tun0, $ext_if } proto udp from \ >> to port 2194 >=20 > Ok - excellent - =C8=99o one can mix IPv4 and IPv6 in a list - and = =E2=80=98udp=E2=80=99 no longer needs to be =E2=80=98udp6=E2=80=99 (and = same for tcp6 and icmp6 v.s. tcp/icmp_=E2=80=94 pf guesses this right = based on the address ? Ignore - that works perfectly - with inet/inet6 thrown in where I need = to make the distinction. Thanks ! Dw. --Apple-Mail=_E7A94B88-4767-4051-AD41-387F9DEDB4CA Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On 8 May 2024, = at 22:41, Dirk-Willem van Gulik <dirkx@webweaving.org> = wrote:

On 8 May = 2024, at 22:14, Lexi Winter <lexi@le-fay.org> wrote:

Dirk-Willem van Gulik:
For dual stack hosts; with both an IPv4 and IPv6 CIDR that they = are
listening to - is there a recommended way to setup pf.conf to = avoid
mistakes/duplication ?

To avoid duplication in constructs such as:

# Foo app = servers
= foobarserver_host4=3D231.17.X.Y
= foobarserver_host6=3Dfe80::5246:=E2=80=A6

# Load = balancers  - direct or via tun0 in post/fail-back 
= bar_net=3DX.Y.Z.Z # 
= bar_net6=3Dfe80::5246:=E2=80=A6 # 
= =E2=80=A6

pass in on { tun0, $ext_if } =  proto udp from $bar_net  to $foobarserver_host4 port 2194 = keep state
pass in on { tun0, $ext_if }  proto udp6 from = bar_net6 $var to $foobarserver_host6 port 2194 keep = state

Is there some recommended way of doing this in stock FreeBSD ? Or = does
one usually end up with some sort of macro/generate style = solution ?

i would = suggest something like this:

= table = <foobarserver> {
= 231.17.X.Y
fe80::5246:...
= }

table <bar-net> = {
...
= }

pass on { tun0, $ext_if = } proto udp from <bar-net> \
= to <foobarserver> = port 2194

Ok - excellent - =C8=99o one = can mix IPv4 and IPv6 in a list - and =E2=80=98udp=E2=80=99 no longer = needs to be =E2=80=98udp6=E2=80=99 (and same for tcp6 and icmp6 v.s. = tcp/icmp_=E2=80=94 pf guesses this right based on the address = ?

Ignore - that works perfectly - = with inet/inet6 thrown in where I need to make the = distinction.

Thanks = !

Dw.

= --Apple-Mail=_E7A94B88-4767-4051-AD41-387F9DEDB4CA-- From nobody Thu May 9 15:56:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZxTY4K1Qz5JyWL; Thu, 09 May 2024 15:56:49 +0000 (UTC) (envelope-from timp87@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZxTX4hRwz4HHq; Thu, 9 May 2024 15:56:48 +0000 (UTC) (envelope-from timp87@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xg44xeTo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of timp87@gmail.com designates 2607:f8b0:4864:20::102d as permitted sender) smtp.mailfrom=timp87@gmail.com Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-2b433dd2566so838701a91.2; Thu, 09 May 2024 08:56:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715270207; x=1715875007; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9NsyOH4IDxXlJ1qlpMWzwZEfQhONOQjDERi21aDixcQ=; b=Xg44xeToB+TVrXipNWtMrIdSFAVhzl2A8G87/iINxrPryoVzbGb1FhuUxPvaenuSq6 6VCM4tEKeqOwSO5cHGhoNwHM4RFcpjl0HmRj2diWA84Ok0PVIKOzppMNa66ZV3eR8Ug9 kJovZYrwec7BFUFjbuQutZOD4pZ/u64g9CaSVavOw722lL+sp7svYTLSoyQTxGf5/o7s B53NCVL3LvHRZwDqjPnTY5/N8fBm1mPr5i9hZ6aitH5UW2/KNERs94d5lD7KfIwu2vzu jTY+0BwnKo0iURR4PSpjEQ0OkbqZjZTTtK5qw8vMBZYGvxRlafpZl5557Xh+kzLu6J5g 3pQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715270207; x=1715875007; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9NsyOH4IDxXlJ1qlpMWzwZEfQhONOQjDERi21aDixcQ=; b=pUqDnKVu3fD8AbC9bkNTIjYAWI6oSmQVbzEwT+bVaMngdCKFPcnhjYd2ix71tvQRTE qO8rT+APU8da04GsWPJsjMvn3CbQBQfDAoX1vWidRis93mzMGEtzMUSPefIEegbcx0iS eNL4K86bZNrAJKc3U+h+cP/oJWvgJVgxcY0Fcmw6Rv3p5PpOMsGhg5m0GDj84uQE+0BQ G0ypVGS805UKnoWKhw3+oRbQDj1s9jO/1BB2x8ZZg96x8pCHTXn9M1I0PYPsXn+LQ5Jy xC4dAWVHQjVPoWkaZ0tR1HRKfdgnGq1KrY2luSt7nfUAfaCUjlUNYcNcehPdJnAw+Bgi uRgA== X-Forwarded-Encrypted: i=1; AJvYcCUQT0GIiKJWzGb9Elz22eSW1ZttT3prbOsyBr4WRIoNCPLdz2Ld4X9hpsP6YdJ5zu8+gsy8B0/R/8s2Rlz19QZEO2j28ZhBgGmA7ow= X-Gm-Message-State: AOJu0Yx8Thps2J6fhDFE1EV1xWKDV4HSr/F639Mnt0WEKyOUqzu88NsD KhvEl080fCTJv7g0t4bliiVn6EnVxyoFiib5z+SVESOj157CtUy9ThY9+kdRC6oHLdfY5yalGQI ZmcxQz8ra5sDPOPqujM1oJoonS/DlckU3 X-Google-Smtp-Source: AGHT+IErOVgq0l1nfEC1klfnFV3QEUiEcRY/sK0e0UuK9c81wZ0BAtNhox0cRC2XafPxUZHzVgi5WawqieypFXzbjBg= X-Received: by 2002:a17:90b:506:b0:2b4:abc7:d642 with SMTP id 98e67ed59e1d1-2b61639b687mr5890613a91.6.1715270206695; Thu, 09 May 2024 08:56:46 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Pavel Timofeev Date: Thu, 9 May 2024 09:56:35 -0600 Message-ID: Subject: Re: Call-for-testing: Asynchronous audio device detach To: Christos Margiolis Cc: freebsd-multimedia@freebsd.org, freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="0000000000005d3382061807779c" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102d:from]; RCPT_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-multimedia@freebsd.org,freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4VZxTX4hRwz4HHq --0000000000005d3382061807779c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello This functionality has always been in my personal list of desires. Thank you for implementing this! I don't have many USB sound devices nowadays. Only a couple of usb headphones. So I tried 15-CURRENT main-n269827-b07689d1f2a2 under gnome and it worked well with some basic things. The system survives usb headphone disconnect (unlike before) and switches to another output after a few seconds (not immediately however). This is what I experience during youtube video playback in chromium. I'll try to extend the test and play with it more soon. I'm really glad this is fixed now! Thanks to everyone involved in this! On Fri, Apr 12, 2024 at 1:40=E2=80=AFPM Christos Margiolis wrote: > Hello, > > Yesterday I committed a patch [1] which adds support for asynchronous > device detach for audio devices, something that many people with > detachable audio devices (e.g USB) have been asking for for years > [2][3][4][5][6][many more...]. I would like to ask you to give it a try > and let me know if everything works properly. Note that this patch > depends on dc831e93bad6 [7], so make sure to apply this one as well. > > Christos > > [1] > https://cgit.freebsd.org/src/commit/?id=3D44e128fe9d92c1a544b801cb56e907a= 66ef34691 > [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194727 > [3] > https://forums.freebsd.org/threads/not-entering-sleep-state-due-to-audio.= 82597/ > [4] > https://forums.freebsd.org/threads/forcing-off-the-computer-endlessly-wai= ting-for-sound-application-to-exit-at-sleep-suspend-time.80412/ > [5] https://www.davidschlachter.com/misc/freebsd-usb-audio > [6] > https://randomnixfix.wordpress.com/2021/10/23/why-the-freebsd-desktop-and= -my-linux-rant/ > [7] > https://cgit.freebsd.org/src/commit/?id=3De8c0d15a64fadb4a330f2da7244beca= ac161bb70 > > --0000000000005d3382061807779c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello

This functionality has always bee= n in my personal list of desires.
Thank you for implementing this= !

I don't have many USB sound devices nowadays= . Only a couple of usb headphones.
So I tried 15-CURRENT=C2=A0mai= n-n269827-b07689d1f2a2 under gnome and it worked well with some basic=C2=A0= things.
The system survives usb headphone disconnect (unlike befo= re) and switches to another output after a few seconds (not immediately how= ever).
This is what I experience during youtube video playback in= chromium.

I'll try to extend the test and pla= y with it more soon.

I'm really glad this is f= ixed now!
Thanks to everyone involved in this!

On Fri, Apr 1= 2, 2024 at 1:40=E2=80=AFPM Christos Margiolis <christos@freebsd.org> wrote:
Hello,

Yesterday I committed a patch [1] which adds support for asynchronous
device detach for audio devices, something that many people with
detachable audio devices (e.g USB) have been asking for for years
[2][3][4][5][6][many more...]. I would like to ask you to give it a try
and let me know if everything works properly. Note that this patch
depends on dc831e93bad6 [7], so make sure to apply this one as well.

Christos

[1] https://cgit.f= reebsd.org/src/commit/?id=3D44e128fe9d92c1a544b801cb56e907a66ef34691 [2] https://bugs.freebsd.org/bugzilla/show= _bug.cgi?id=3D194727
[3] https://forums.fr= eebsd.org/threads/not-entering-sleep-state-due-to-audio.82597/
[4] https://forums.freebsd.org/threads/= forcing-off-the-computer-endlessly-waiting-for-sound-application-to-exit-at= -sleep-suspend-time.80412/
[5] https://www.davidschlachter.com/misc/free= bsd-usb-audio
[6] https://= randomnixfix.wordpress.com/2021/10/23/why-the-freebsd-desktop-and-my-linux-= rant/
[7] https://cgit.f= reebsd.org/src/commit/?id=3De8c0d15a64fadb4a330f2da7244becaac161bb70
--0000000000005d3382061807779c-- From nobody Thu May 9 17:58:32 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vb0BG5dVqz5KBsT for ; Thu, 09 May 2024 17:58:46 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vb0BF3gK5z4Xn1 for ; Thu, 9 May 2024 17:58:45 +0000 (UTC) (envelope-from pen@lysator.liu.se) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=lysator.liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 8095F100F7 for ; Thu, 9 May 2024 19:58:42 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 747581007E; Thu, 9 May 2024 19:58:42 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Score: -1.0 Received: from smtpclient.apple (unknown [IPv6:2001:9b1:28fc:6000:58d3:48b0:3615:d77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id D59601015C for ; Thu, 9 May 2024 19:58:40 +0200 (CEST) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Trying to understand CAM (and the cciss driver) Message-Id: <087AB4C4-9CE0-4024-8E1B-4C636014C8CC@lysator.liu.se> Date: Thu, 9 May 2024 19:58:32 +0200 To: FreeBSD Hackers X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Virus-Scanned: ClamAV using ClamSMTP X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.42 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.92)[-0.924]; DMARC_POLICY_ALLOW(-0.50)[lysator.liu.se,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; RCVD_IN_DNSWL_MED(-0.20)[130.236.254.3:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4Vb0BF3gK5z4Xn1 Hi, I=E2=80=99m trying to fix a bug in the cciss driver that has been there = =E2=80=9Cforever=E2=80=9D when using it with an HP H241 SAS HBA card.=20 The driver works fine when all (SAS, spinning rust) drives are behaving = well, but when some of them are starting to go bad it often goes into = spin and either hangs the kernel or panics. I=E2=80=99ve been trying to = add instrumentation to it in order to pin-point the problem and have = been attempting some workarounds (like clearing cr_complete since = without that hack sometime the driver get many many non-busy repeated = requests with the same =E2=80=9Ctag=E2=80=9D and then It panics with:=20 > ciss1: WARNING: completing non-busy request > ciss1: WARNING: completing non-busy request > ciss1: WARNING: completing non-busy request > panic: cam_periph_done_panic: already done with ccb 0xfffff88abefc4000 > cpuid =3D 0 > time =3D 1714852260 > KDB: stack backtrace: > #0 0xffffffff80c541b5 at kdb_backtrace+0x65 > #1 0xffffffff80c06b31 at vpanic+0x151 > #2 0xffffffff80c069d3 at panic+0x43 > #3 0xffffffff8039017c at cam_periph_done_panic+0x1c > #4 0xffffffff80397193 at xpt_done_process+0x313 > #5 0xffffffff803990a5 at xpt_done_td+0xf5 > #6 0xffffffff80bc333e at fork_exit+0x7e > #7 0xffffffff8108a44e at fork_trampoline+0xe =20 but it=E2=80=99s still there=E2=80=A6=20 I think I could use some input/suggestions from someone more = knowledgeable with CAM and device drivers in general=E2=80=A6 An example (this is with extra instrumentation added so not quite a = stock kernel driver): ciss1: *** Drive failure imminent, Port=3D1E Box=3D1 Bay=3D17 = reason=3D0x37 [ts=3D4460, class=3D4, subclass=3D0, detail=3D1, tag=3D0x2] (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 51 98 00 = 00 00 10 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 52 18 00 = 00 00 10 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 65 c8 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 63 c8 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 61 c8 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f c8 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f a8 00 = 00 00 20 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f 38 00 = 00 00 70 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5d 38 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5b 38 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 59 38 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 58 48 00 = 00 00 f0 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 56 48 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 48 d0 00 = 00 00 10 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 46 e0 00 = 00 01 f0 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8f d0 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8d d0 00 = 00 02 00 00 00=20 (da122:ciss1:32:78:0): CAM status: CCB request completed with an error (da122:ciss1:32:78:0): Retrying command, 3 more tries remain ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, = ccphys=3D1791048448, tag=3D72, flags=3D0x50 (----I-C), sg_tag=3D0x3, = cr_complete=3D0xffffffff806059a0] ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, = ccphys=3D1791318368, tag=3D313, flags=3D0x50 (----I-C), sg_tag=3D0x3, = cr_complete=3D0xffffffff806059a0] ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192, = ccphys=3D1791226528, tag=3D231, flags=3D0x50 (----I-C), sg_tag=3D0x3, = cr_complete=3D0xffffffff806059a0] ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, = ccphys=3D1791313888, tag=3D309, flags=3D0x50 (----I-C), sg_tag=3D0x5, = cr_complete=3D0xffffffff806059a0] ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192, = ccphys=3D1791206368, tag=3D213, flags=3D0x50 (----I-C), sg_tag=3D0x3, = cr_complete=3D0xffffffff806059a0] ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy panic: camq_remove: Attempt to remove out-of-bounds index -1 from queue = 0xfffff80107fa9838 of size 1 cpuid =3D 7 time =3D 1715266713 KDB: stack backtrace: #0 0xffffffff80c542d5 at kdb_backtrace+0x65 #1 0xffffffff80c06c51 at vpanic+0x151 #2 0xffffffff80c06af3 at panic+0x43 #3 0xffffffff80390676 at camq_remove+0xf6 #4 0xffffffff803936c8 at xpt_run_devq+0x148 #5 0xffffffff80392b0c at xpt_action_default+0x35c #6 0xffffffff803c3544 at dastart+0x314 #7 0xffffffff80394f73 at xpt_run_allocq+0x83 #8 0xffffffff803c5152 at dastrategy+0x82 #9 0xffffffff80b438d4 at g_disk_start+0x314 #10 0xffffffff80b46b55 at g_io_request+0x1d5 #11 0xffffffff80b46b55 at g_io_request+0x1d5 #12 0xffffffff821a34dc at vdev_geom_io_start+0x23c #13 0xffffffff82347fb4 at zio_vdev_io_start+0x204 #14 0xffffffff8234204f at zio_nowait+0x12f #15 0xffffffff822a0ce7 at vdev_mirror_io_start+0x177 #16 0xffffffff82347fb4 at zio_vdev_io_start+0x204 #17 0xffffffff8234204f at zio_nowait+0x12f Uptime: 10h16m26s The drive it complains about in the first line is one that has SMART = errors: > connector 1E box 1 bay 17 HP = MB010000JWAYK 7PH8LU3G HPD5 = S.M.A.R.T. predictive failure. It is _not_ da122 though=E2=80=A6 The drive with a SMART error (da85) is = not in use so basically is just sitting there. The code where I=E2=80=99m trying to figure out what=E2=80=99s going on = is this part in sys/dev/ciss/ciss.c: static void ciss_complete(struct ciss_softc *sc, cr_qhead_t *qh) { struct ciss_request *cr; debug_called(2); /* = = =20 * Loop taking requests off the completed queue and performing = = =20 * completion processing on them. = = =20 */ for (;;) { int got_nonbusy =3D 0; if ((cr =3D ciss_dequeue_complete(sc, qh)) =3D=3D NULL) break; ciss_unmap_request(cr); if ((cr->cr_flags & CISS_REQ_BUSY) =3D=3D 0) { cr->nonbusy_request_counter++; ciss_printf(sc, "WARNING: completing non-busy request #%d [onq=3D%d,= length=3D%u, ccphys=3D%u, tag=3D%d, flags=3D0x%x (%s), sg_tag=3D0x%x, = cr_complete=3D%p]\n", cr->nonbusy_request_counter, cr->cr_onq, cr->cr_length, cr->cr_ccphys, = cr->cr_tag, cr->cr_flags, cr_flags2str(cr->cr_flags), cr->cr_sg_tag, cr->cr_complete ); got_nonbusy =3D 1; } cr->cr_flags &=3D ~CISS_REQ_BUSY; /* = = =20 * If the request has a callback, invoke it. = = =20 */ if (cr->cr_complete !=3D NULL) { cr->cr_complete(cr); /* pen: Attempt to make sure ciss_cam_complete() doesn=E2=80=99t = get called multiple times */ if (cr->cr_complete =3D=3D ciss_cam_complete && got_nonbusy) { ciss_printf(sc, "WARNING: clearing cr_complete due to = ciss_cam_complete & nonbusy\n"); cr->cr_complete =3D NULL; } continue; } =20 /* = = =20 * If someone is sleeping on this request, wake them up. = = =20 */ if (cr->cr_flags & CISS_REQ_SLEEP) { cr->cr_flags &=3D ~CISS_REQ_SLEEP; wakeup(cr); continue; } /* = = =20 * If someone is polling this request for completion, signal. = = =20 */ if (cr->cr_flags & CISS_REQ_POLL) { cr->cr_flags &=3D ~CISS_REQ_POLL; continue; } /* = = =20 * Give up and throw the request back on the free queue. This = = =20 * should never happen; resources will probably be lost. = = =20 */ ciss_printf(sc, "WARNING: completed command with no = submitter\n"); ciss_enqueue_free(cr); } } - Any suggestions? (Probably not but I thought I=E2=80=99d ask anyway = :-) - Peter From nobody Thu May 9 23:31:34 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vb7ZQ4ZKtz5KHpw for ; Thu, 09 May 2024 23:31:42 +0000 (UTC) (envelope-from jeffpc@josefsipek.net) Received: from smtp.jeffnet.31bits.net (josefsipek.net [71.174.62.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4Vb7ZP6R8Rz47L6 for ; Thu, 9 May 2024 23:31:41 +0000 (UTC) (envelope-from jeffpc@josefsipek.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jeffpc@josefsipek.net designates 71.174.62.3 as permitted sender) smtp.mailfrom=jeffpc@josefsipek.net Received: from satis (satis [172.27.0.85]) by smtp.jeffnet.31bits.net (Postfix) with ESMTPSA id 530A62E56D for ; Thu, 9 May 2024 23:31:35 +0000 (UTC) Date: Thu, 9 May 2024 19:31:34 -0400 From: Josef 'Jeff' Sipek To: freebsd-hackers@freebsd.org Subject: Precision Hardware Clocks Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_NO_TLS_LAST(0.10)[]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:701, ipnet:71.174.0.0/16, country:US]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[josefsipek.net]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4Vb7ZP6R8Rz47L6 Hello all, I've been playing with the idea of extending the kernel to expose various clock sources to userspace via a character devices. (Yesterday's thread about the OCP TAP Time Card nudged me to send this out sooner than I planned. :) ) The code is *very* hacky and full of TODOs & FIXMEs, but I thought I'd share it now. What I'm calling a 'precision hardware clock' (PHC for short) is conceptually some piece of hardware which can provide the consumer a sense of time passing. Roughly speaking, there are two types of precision hardware clocks - those that return the current time using some defined timescale (e.g., kvmclock) and those that are simple oscillators with counters (e.g., many e1000e devices). My aim is to support both. My initial goal is to provide a *read-only* access to PHCs as this is sufficient to make use of them for stabilizing the system clock. That is, an application can only query them for the current time. Eventually, I think it'd make sense to allow *setting* PHCs as well. The devices that return the current time are fairly straight forward to work with. The ioctl simply calls a device specific method and forwards the result to the caller. The counter-type devices are more complicated to support. In my code I took the approach that's very similar to the timecounter code in the kernel. My first attempt actually tried to extend timecounters but that resulted in a lot of additional computation being done in hardclock regardless of whether or not the additional clocks were in use. That didn't feel right. [1] My current code borrows the timecounter idea (and some code) of extending the hardware counter in software. The overflow check is done via a per-devices callout that's scheduled based for an interval based on the oscillator's frequency and the counter's width. (For debugging, I cap it at 10s max interval.) Regardless of which type of PHC it is, the ioctl caller gets what amounts to a reading. Ideally, the two correspond to the same instant, but there may be some error due to hardware limitations. [2] Because there is a lot of hardware that doesn't provide a way to capture these correlated timestamps, a "capture many readings" ioctl is a useful addition. This ioctl returns a set of interleaved PHC and system clock readings, which lets the application (e.g., chrony) do the appropriate filtering to remove noise. In addition to adding the PHC code to core kernel, I hacked up the if_em driver to start the 25MHz timekeeping counters on 82574 devices and register with the PHC code. Finally, I hacked up chrony's PHC refclock driver to make use of the "get timestamp pair" ioctl. I ran this code on my test box with two 82574 NICs with both registered as chrony refclocks [3] for a while. Unsurprisingly, the 82574 oscillators are not that accurate but they are reasonably stable. (I posted histograms and allan deviation plots on mastodon [4]. Since the system's oscillator is in no way special, it is a bit silly to read too much into the graphs. However, I'd argue that it still shows that the 82574 refclocks were reasonably good and would likely help in real world scenarios [5].) You can find my patches can be found at: https://www.josefsipek.net/freebsd/phc-v1/ There are 3 patches: 1. chrony.patch modifies chronyd to use the PHC ioctls 2. fbsd-phc.patch adds the generic PHC code 3. fbsd-em.patch modifies if_em to register 82574 timekeeping counter with PHC In addition to cleaning up and generally improving the existing patches, I hope to implement the bit of code that wires up KVM's KVM_HC_CLOCK_PAIRING hypercall as a PHC. While 82574 provides a counter-type PHC, this kvm PHC would be the absolute time-type PHC. Support for kvm PHC would allow FreeBSD guests to sync *very* accurately to host's system clock. I also have an incomplete patch that adds support for clock_gettime(3) using PHC fds as clockid_t values, but since it isn't complete I'll keep it to myself for now :) So, that's what I've been up to. As I said in the beginning, I wanted to get more of this done, but I think it makes sense for me to let others know about my code now. I plan to continue hacking away on this, but if people have opinions about any of this, I'd love to hear them. It really pains me that there is so much duplication between the PHC and timecounter code, but the current tc_windup code runs in a rather special context (hardclock) and having it process *all* devices regardless of use would increase its runtime quite a bit. I've been thinking about trying to move some of the timecounter and PHC code into a generic set of helpers or try to reorganize kern_tc.c to fold the PHC login into it sanely, but that's currently very far down the todo list. To summarize, the goals/non-goals for this work are: Goals: * read-only interface to various precision hardware clocks (PHCs) * support for both absolute time and counter-only PHCs * ability to use software like chrony to stabilize system clocks Non-goals/future work: * adjusting PHCs * support for cross-timestamping techniques (like Intel's ART) * support for if_em PTP packet timestamping * external pin timestamping support Thanks for reading this far. Let me know if you have any questions, suggestions, etc. Jeff. [1] I actually ran for about a week with a e1000e card in my box providing timekeeping by selecting it via the kern.timecounter sysctls. It worked and was quite amusing to see, but the additional complexity in tc_windup made it unworkable. [2] At some point, Intel added the Always Running Timer (ART) which can be used by devices to get timestamps that are easily convertible to TSC readings. Support for this is part of future work. [3] The chrony config was the following. I ran chronyd with the -x flag to prevent it from trying to set the clock. The system clock was disciplined with ptp2d, which was syncing to ptp2d running on the same server that chrony used for NTP. Note that the refclocks are marked as 'pps local', meaning that they are to be used only as a frequency source. ('pps' means that the refclock isn't reporting UTC, and 'local' means that the clock isn't aligned to UTC seconds) server iburst minpoll 0 maxpoll 4 xleave refclock PHC /dev/phc-em0 refid EM0 pps local refclock PHC /dev/phc-em1 refid EM1 pps local logdir /tmp log measurements statistics tracking refclocks selection rtc logbanner 0 [4] https://mastodon.radio/@jeffpc/112230743393202103 [5] A huge problem with NTP is that it suffers greatly from any network latency jitter and asymmetrical routing. Having a stable reference clock (even if the stability is short-term only) helps NTP software quite a bit. From nobody Fri May 10 05:37:50 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbHj41J5dz5KtFQ for ; Fri, 10 May 2024 05:38:00 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VbHj35wM7z4nKB for ; Fri, 10 May 2024 05:37:59 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 2417389287; Fri, 10 May 2024 05:37:52 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 44A5bps5076724 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 10 May 2024 05:37:51 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 44A5boZA076723; Fri, 10 May 2024 05:37:50 GMT (envelope-from phk) Message-Id: <202405100537.44A5boZA076723@critter.freebsd.dk> To: "Josef 'Jeff' Sipek" cc: freebsd-hackers@freebsd.org Subject: Re: Precision Hardware Clocks In-reply-to: From: "Poul-Henning Kamp" References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <76721.1715319470.1@critter.freebsd.dk> Date: Fri, 10 May 2024 05:37:50 +0000 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU] X-Rspamd-Queue-Id: 4VbHj35wM7z4nKB Josef 'Jeff' Sipek writes: > To summarize, the goals/non-goals for this work are: > > Goals: > * read-only interface to various precision hardware clocks (PHCs) > * support for both absolute time and counter-only PHCs > * ability to use software like chrony to stabilize system clocks I should and will be the last to discourage anybody from having fun with timekeeping. But I do feel a responsibility to point out, that you are trying to solve already solved problems, in a way that does not work nearly as well as those solutions. Chesterton's fence: Before you throw it out or bypass it, you should find out why the current timekeeping infrastructure is built like it is. Back in the mists of time, before even I got involved in it, NTPD did more or less exactly what you propose, because there were no kernel support for timekeeping, only for adding device drivers, and it did not work then, and it wont work much better today, for fundamental and inescapable reasons. For starters, exposing the hardware count though a char-dev is going to be very jittery (= time-noise). The "userland->kernel->userland" context switches are very unpredictable timewise, because it is anyones guess how many memory operations it will take, even in the best case. Worse, there is a high risk that you loose the CPU to another (kernel)thread which is going to /really/ introduce jitter. That is why the PPS-API, timecounters and kernel_pll exists: To keep the "real-time" aspect of the timekeeping firmly inside the kernel and undisturbed by userland and lower priority kernel activities. Unless you can expose the hardware directly to userland, via mmap(2), timekeeping in userland is simply not going to perform. With that said, a lot of our timekeeping is stuff I wrote 25 years old, and it is absolutely due for both a rethink and a refresh, but if you decide to throw it all out and start from fresh, you will not get to the interesting parts for years. So before you continue, at the very least, read this: https://papers.freebsd.org/2002/phk-timecounters/ And you should think a LOT about page 91 in this one too: https://www.am1.us/wp-content/uploads/Documents/U11625_VIG-TUTORIAL.pdf (The other 307 pages are also interesting :-) Poul-Henning FreeBSD TimeLord (retired) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nobody Fri May 10 14:35:28 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbWdj2Bj2z5JZF1 for ; Fri, 10 May 2024 14:35:53 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from gid2.gid.co.uk (ns0.gid.co.uk [IPv6:2001:470:94de::240]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gid2.gid.co.uk", Issuer "gid2.gid.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VbWdh6tZTz4jyx for ; Fri, 10 May 2024 14:35:52 +0000 (UTC) (envelope-from rb@gid.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by gid2.gid.co.uk (8.15.2/8.15.2) with ESMTP id 44AEZimR070715; Fri, 10 May 2024 15:35:44 +0100 (BST) (envelope-from rb@gid.co.uk) Received: from smtpclient.apple (moriarty.gid.co.uk [194.32.164.17]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 44AEZcxT066247; Fri, 10 May 2024 15:35:39 +0100 (BST) (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: Precision Hardware Clocks From: Bob Bishop In-Reply-To: <202405100537.44A5boZA076723@critter.freebsd.dk> Date: Fri, 10 May 2024 15:35:28 +0100 Cc: "freebsd-hackers@freebsd.org" , Poul-Henning Kamp Content-Transfer-Encoding: quoted-printable Message-Id: References: <202405100537.44A5boZA076723@critter.freebsd.dk> To: "Josef 'Jeff' Sipek" X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4VbWdh6tZTz4jyx Hi, > On 10 May 2024, at 06:37, Poul-Henning Kamp = wrote: >=20 > Josef 'Jeff' Sipek writes: >=20 >> To summarize, the goals/non-goals for this work are: >>=20 >> Goals: >> * read-only interface to various precision hardware clocks (PHCs) >> * support for both absolute time and counter-only PHCs >> * ability to use software like chrony to stabilize system clocks >=20 > I should and will be the last to discourage anybody from having fun > with timekeeping. >=20 > But I do feel a responsibility to point out, that you are trying > to solve already solved problems, in a way that does not work nearly > as well as those solutions. >=20 > Chesterton's fence: Before you throw it out or bypass it, you > should find out why the current timekeeping infrastructure is built > like it is. >=20 > Back in the mists of time, before even I got involved in it, NTPD > did more or less exactly what you propose, because there were no > kernel support for timekeeping, only for adding device drivers, and > it did not work then, and it wont work much better today, for > fundamental and inescapable reasons. >=20 > For starters, exposing the hardware count though a char-dev is going > to be very jittery (=3D time-noise). The "userland->kernel->userland" > context switches are very unpredictable timewise, because it is > anyones guess how many memory operations it will take, even in the > best case. Worse, there is a high risk that you loose the CPU to > another (kernel)thread which is going to /really/ introduce jitter. I can second this. Having in the past tried to do time-sensitive machine = control that way, the jitter was too bad to maintain a few tens of = milliseconds. You might do an order of magnitude better today but it=E2=80= =99s still nowhere near good enough for modern timekeeping. > That is why the PPS-API, timecounters and kernel_pll exists: To > keep the "real-time" aspect of the timekeeping firmly inside the > kernel and undisturbed by userland and lower priority kernel > activities. >=20 > Unless you can expose the hardware directly to userland, via mmap(2), > timekeeping in userland is simply not going to perform. >=20 > With that said, a lot of our timekeeping is stuff I wrote 25 years > old, and it is absolutely due for both a rethink and a refresh, but > if you decide to throw it all out and start from fresh, you will > not get to the interesting parts for years. >=20 > So before you continue, at the very least, read this: >=20 > https://papers.freebsd.org/2002/phk-timecounters/ >=20 > And you should think a LOT about page 91 in this one too: >=20 > = https://www.am1.us/wp-content/uploads/Documents/U11625_VIG-TUTORIAL.pdf >=20 > (The other 307 pages are also interesting :-) >=20 > Poul-Henning > FreeBSD TimeLord (retired) >=20 > --=20 > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe =20 > Never attribute to malice what can adequately be explained by = incompetence. >=20 -- Bob Bishop rb@gid.co.uk From nobody Fri May 10 15:00:18 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbX9x0l31z5JcKl for ; Fri, 10 May 2024 15:00:21 +0000 (UTC) (envelope-from jeffpc@josefsipek.net) Received: from smtp.jeffnet.31bits.net (josefsipek.net [71.174.62.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4VbX9w4vFqz4msb for ; Fri, 10 May 2024 15:00:20 +0000 (UTC) (envelope-from jeffpc@josefsipek.net) Authentication-Results: mx1.freebsd.org; none Received: from satis (satis [172.27.0.85]) by smtp.jeffnet.31bits.net (Postfix) with ESMTPSA id DEE4F2E999; Fri, 10 May 2024 15:00:19 +0000 (UTC) Date: Fri, 10 May 2024 11:00:18 -0400 From: Josef 'Jeff' Sipek To: Poul-Henning Kamp Cc: freebsd-hackers@freebsd.org Subject: Re: Precision Hardware Clocks Message-ID: References: <202405100537.44A5boZA076723@critter.freebsd.dk> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202405100537.44A5boZA076723@critter.freebsd.dk> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:701, ipnet:71.174.0.0/16, country:US] X-Rspamd-Queue-Id: 4VbX9w4vFqz4msb On Fri, May 10, 2024 at 05:37:50 +0000, Poul-Henning Kamp wrote: > Josef 'Jeff' Sipek writes: > > To summarize, the goals/non-goals for this work are: > > > > Goals: > > * read-only interface to various precision hardware clocks (PHCs) > > * support for both absolute time and counter-only PHCs > > * ability to use software like chrony to stabilize system clocks > > I should and will be the last to discourage anybody from having fun > with timekeeping. :) > But I do feel a responsibility to point out, that you are trying > to solve already solved problems, in a way that does not work nearly > as well as those solutions. I disagree. I'm sure there are ways of improving my approach, but I don't think this is a completely solved problem. There are a *lot* systems out there that currently have only NTP for sync, they would benefit from a PPS but adding an actual PPS source is not possible. Many of the same systems have additional oscillators with counters that can be used as reference clocks. (I'm using 82574 e1000e cards for my experiments since I have a number of them and they have a 25MHz clock & counter. There are other sources which are better, but I don't have any on hand.) Thorough benchmarking is necessary, of course. > Chesterton's fence: Before you throw it out or bypass it, you > should find out why the current timekeeping infrastructure is built > like it is. 100% agreed. What I have shared is a very work-in-progress proof-of-concept. As I said before, I don't like the code duplication, etc. If you have any pointers to docs with that context, please let me know. I've read what I could find (including timecounters & nanokernel), but there are probably more. ... > For starters, exposing the hardware count though a char-dev is going > to be very jittery (= time-noise). The "userland->kernel->userland" > context switches are very unpredictable timewise, because it is > anyones guess how many memory operations it will take, even in the > best case. Worse, there is a high risk that you loose the CPU to > another (kernel)thread which is going to /really/ introduce jitter. I'm not sure that matters. The PHC ioctl used by my hacked-up chrony returns time pair. So, from userspace's perspective, it doesn't matter how long the ioctl took to execute since the result tells the application the CLOCK_REALTIME timestamp of the PHC reading. Regardless of how long it took to get the timestamp pair, the application knows exactly how long ago (according to the system clock) the PHC reading was made and it can apply the various filtering and statistics to derive the parameters for ntp_adjtime(2) or however it decides to tweak the system clock. This (unpredictable) ioctl latency is different from the software-timestamped PPS interrupt latency, which is a problem. > That is why the PPS-API, timecounters and kernel_pll exists: To > keep the "real-time" aspect of the timekeeping firmly inside the > kernel and undisturbed by userland and lower priority kernel > activities. Sure, but having the code in the kernel has its limitations too. A huge one is that it is much harder to fuse multiple time sources into an estimate that can be used to steer the system clock. Exposing additional time sources to userspace and having it figure out what to do seems more practical than trying to implement time source combining in the kernel. To be clear, I think a hybrid approach is the way to go - userspace does fancy filtering and feeds the kernel pre-processed information so it can steer the system clock more effectively. This is already what happens, I just want to give userspace more time sources to work with. > Unless you can expose the hardware directly to userland, via mmap(2), > timekeeping in userland is simply not going to perform. Counterexample: Linux's PTP kernel API exposes time via char dev ioctls and it performs very well. With good PTP-capable hardware, you can get to tens of ns. > With that said, a lot of our timekeeping is stuff I wrote 25 years > old, and it is absolutely due for both a rethink and a refresh, Let me know if you have any ideas how the timekeeping code should change, I'm more than happy to work on it. > but > if you decide to throw it all out and start from fresh, you will > not get to the interesting parts for years. I definitely don't want to throw it all out. My current code ignores it because that's the quick & dirty way to experiment with an idea. A production-ready version must be more cohesive, of course. > So before you continue, at the very least, read this: > > https://papers.freebsd.org/2002/phk-timecounters/ I've read it a few times over the past couple of months while I was playing around with the PHC idea. :) > And you should think a LOT about page 91 in this one too: > > https://www.am1.us/wp-content/uploads/Documents/U11625_VIG-TUTORIAL.pdf I didn't see this presentation before, but that slide is pretty much the one figure in every timekeeping presentation. So, I have been thinking about it already ;) > (The other 307 pages are also interesting :-) Indeed! It looks like a really information-dense set of slides. Thanks for the link & feedback. Jeff. From nobody Fri May 10 18:07:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbcM16X0Fz5K9VT for ; Fri, 10 May 2024 18:08:29 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Received: from mail.dcrosstech.com (syn-024-097-005-251.biz.spectrum.com [24.97.5.251]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.dcrosstech.com", Issuer "DCrossTech.com LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VbcM10yFMz4Hqr for ; Fri, 10 May 2024 18:08:29 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of david@crossfamilyweb.com designates 24.97.5.251 as permitted sender) smtp.mailfrom=david@crossfamilyweb.com X-Virus-Scanned: amavisd-new at dcrosstech.com Received: from smtpclient.apple (d138.guest.wifi.crossfamilyweb.com [10.1.8.138]) (authenticated bits=0) by mail.dcrosstech.com (8.15.2/8.15.2) with ESMTPSA id 44AI8Get059347 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Fri, 10 May 2024 18:08:16 GMT (envelope-from david@crossfamilyweb.com) X-Authentication-Warning: mail.priv.dcrosstech.com: Host d138.guest.wifi.crossfamilyweb.com [10.1.8.138] claimed to be smtpclient.apple Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable From: David Cross List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (1.0) Date: Fri, 10 May 2024 14:07:35 -0400 Subject: Quick suggestion for upcoming 14.1 and future releases Message-Id: <69039247-30D2-4959-BF06-751436D55485@crossfamilyweb.com> To: FreeBSD Hackers X-Mailer: iPhone Mail (21E236) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.18 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.983]; R_SPF_ALLOW(-0.20)[+mx]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; APPLE_IOS_MAILER_COMMON(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[crossfamilyweb.com]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:11351, ipnet:24.97.0.0/16, country:US]; ARC_NA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[david]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4VbcM10yFMz4Hqr It would be really nice to capture the official freebsd pgp keys (at least f= or the release officers and security officers) into something like /usr/shar= e/pgpkeys/freebsd or something. That way on a plain freebsd install one can validate SAs/ENs/Releases withou= t having to first find them on the website, download, etc. and then of cours= e having to redo it every year as they expire. This would also necessitate putting them in source control as well, which wo= uld give some added transparency when things change. Thanks!= From nobody Fri May 10 21:29:26 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VbhqR3TDRz5KSPn for ; Fri, 10 May 2024 21:29:55 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VbhqQ4VS7z4fYT for ; Fri, 10 May 2024 21:29:54 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; none Received: from kalamity.joker.local (123-1-21-232.area1b.commufa.jp [123.1.21.232]) (authenticated bits=0) by www121.sakura.ne.jp (8.17.1/8.17.1/[SAKURA-WEB]/20201212) with ESMTPA id 44ALTQe1070625; Sat, 11 May 2024 06:29:27 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sat, 11 May 2024 06:29:26 +0900 From: Tomoaki AOKI To: David Cross Cc: FreeBSD Hackers Subject: Re: Quick suggestion for upcoming 14.1 and future releases Message-Id: <20240511062926.1616e3fffc90b6ef08a11b8d@dec.sakura.ne.jp> In-Reply-To: <69039247-30D2-4959-BF06-751436D55485@crossfamilyweb.com> References: <69039247-30D2-4959-BF06-751436D55485@crossfamilyweb.com> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP] X-Rspamd-Queue-Id: 4VbhqQ4VS7z4fYT On Fri, 10 May 2024 14:07:35 -0400 David Cross wrote: > It would be really nice to capture the official freebsd pgp keys (at least for the release officers and security officers) into something like /usr/share/pgpkeys/freebsd or something. > > That way on a plain freebsd install one can validate SAs/ENs/Releases without having to first find them on the website, download, etc. and then of course having to redo it every year as they expire. > > This would also necessitate putting them in source control as well, which would give some added transparency when things change. > > Thanks! Or place a link to the Handbook[1]? Or to another[2]? [1] https://docs.freebsd.org/en/books/handbook/pgpkeys/ [2] https://docs.freebsd.org/en/articles/pgpkeys/ -- Tomoaki AOKI From nobody Sat May 11 13:57:46 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vc6m20bJfz5KB2g for ; Sat, 11 May 2024 13:58:26 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vc6m136QVz4rVt for ; Sat, 11 May 2024 13:58:25 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=WHmc3IsW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::52b as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5d8b887bb0cso2378455a12.2 for ; Sat, 11 May 2024 06:58:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715435903; x=1716040703; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=xeEhKPG2zz/Q4CeNpyGBQH6jRr4hJU5a90KF8L7mBgw=; b=WHmc3IsWf9LTSSelhku45Ql1mJIpmwv+HxV0n/oodCfsVFbe/F6oBKQge+waSBrQ0s H/BwjNfoOI6X5BFhRcSc/xGwLIHSGydah311vHFEoTVvjDSILK7hJRiP4HnC8cisEnQ7 bpHUE3pU1hIIru5L6GT/RGQix7m9qoU6bhU7+0nA5/4chhxGoERi1TxI2RK3QvhtuKVf YvCCqkgdx1v21spdMgicHYcYrYAHomiyYrWEPJdgOU3ZtkRY4sqpQpF173hKDOdW3jS2 t46WFCrysrsAyK3y08pLK4qJ95ur63ZfeVQpUU826TV/UK6uj/KmYuN70eg1WUHlIFQG kxdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715435903; x=1716040703; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=xeEhKPG2zz/Q4CeNpyGBQH6jRr4hJU5a90KF8L7mBgw=; b=dn/PT8CcDFm5GZYDifyC9Xkz8DntC+7ay/pTVYdgMwN14OcuJTlU8fbCAXeL62OZfn 0ZVJp4j27bONGsSmag70fl+0iKOMIc57R4K7r64qHZi92WB455Y81ynoEXpXJd0z5BNg 3USxZs5vMyfqEHf7ud0GcuK3z7cqpvvNmqIdCxgeJlvZ18+VZJoy7L+R7CMVTs8CCcDx hDmP+/Y4seMsL1SYo2z+27l2ejt5qIhbnhQ9GKGagmaxFx3vkP4Hh0MrGB46MhQJaBP9 kiqbSfEHaLljffgQSOYzO+owb22G5J520Ult83Pyimo4dxzq85YY8bkJVBbJl3WsQJgO arKg== X-Gm-Message-State: AOJu0YzEGarMBpGpddOMrTErjxTQVBhZ+Cm9RRL/v1NW7eZFgCaPmVEs oJSWfGQSCBFCTF45rgGlsOdb6Ql2UO5QlPpnHt2tuTwaOXtj+JjT6T2/97RZ52q+O3KyXYKACe4 AbB/6u+lpqarCdqvsu9lkhqOHJphGBQKedcbMNo1e X-Google-Smtp-Source: AGHT+IE0dPmAIJ9IWD1Nm67fCvAn4OBB9ysvfrWW1jZROopLJ+5Q8s6LnJYmCNYfmh9MgnSNxxDEu8sfvhdxl+ktOig= X-Received: by 2002:a05:6a21:622:b0:1af:9ec6:afb9 with SMTP id adf61e73a8af0-1afde082516mr6863551637.2.1715435902860; Sat, 11 May 2024 06:58:22 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: Mario Marietto Date: Sat, 11 May 2024 15:57:46 +0200 Message-ID: Subject: bhyve can't handle ACPI when a nvidia modern GPU in passed within a Windows vm To: freebsd-hackers Content-Type: multipart/related; boundary="000000000000a0303206182e0be4" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.92 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_LONG(-0.97)[-0.969]; NEURAL_HAM_SHORT(-0.96)[-0.960]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52b:from] X-Rspamd-Queue-Id: 4Vc6m136QVz4rVt --000000000000a0303206182e0be4 Content-Type: multipart/alternative; boundary="000000000000a0303106182e0be3" --000000000000a0303106182e0be3 Content-Type: text/plain; charset="UTF-8" Hello. What I would like to understand is why, when I try to pass my RTX 2080 ti from FreeBSD to Windows 11,it won't do it,causing the error 12. It says that it generates a resource conflict and requires additional installation). Now,inside it I still see the error 12 : The device PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&61aaa01&0&48 generates a resource conflict and requires additional installation. I have 3 GPUS. The ones you see below : root@marietto-133:/usr/ports/www/chromium # lspci 00:02.0 Display controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02) 01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1) 02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1) 02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1) 02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1) What I do is to select (from the BIOS) the Intel or the Nvidia Geforce 1060 GPU, reserving the RTX 2080 ti to a guest os (Linux or Windows),by declaring this parameter inside the file /boot/loader.conf : pptdevs="2/0/0 2/0/1 2/0/2 2/0/3" Take in consideration that IT WORKS inside the Linux vm,but NOT in the Windows vm, because there are some kinds of conflicts between resources that I need to understand. I ran msinfo32 and then I checked the resource sharing and conflicts tab to see which devices are conflicting with each other and I found something really interesting (please see the picture below). [image: Istantanea_2024-05-09_17-56-58.png] I'm not a developer,but I suspect that a patch is needed for bhyve. Windows needs that ACPI is enabled within Windows itself, because when I have disabled (configuring the entries below to Start 4 - disabled) the entries below : HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\ACPI Start 0 HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\AcpiPmi Start 3 HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\acpitime Start 3 what's happened has been a BSOD : "Inaccessible Boot Device" I'm not sure if ACPI can be disabled in Windows without to break it irreversibly. I will keep trying to answer this question. What happens,instead if I keep ACPI enabled in Windows,but disabled in bhyve ? Windows crashed with the error "Video TDR failed" The error that I have shown happens in FreeBSD 13 and 14,maybe even 12. It never worked. Anyway,I'm using FreeBSD 14.0-RELEASE-p6. -- Mario. --000000000000a0303106182e0be3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 
Hello.

What I would like to understand is why,
when I try to pass my RTX 2080 t=
i from FreeBSD to 
Windows 11,it won't do it,causing the error 12. <=
br>It says that it generates a resource conflict and
requires additional=
 installation).

Now,inside it I still see the error 12 :

The device PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&6=
1aaa01&0&48 
generates a resource conflict and requires 
addi=
tional installation.

I have 3 GPUS.=20

The ones you see below :

root@marietto-133:/usr/ports/www/chromium # lspci

00:02.0 Display controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphic=
s 630] (rev 02)

01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 10=
60 3GB] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Contro=
ller (rev a1)

02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 20=
80 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Contro=
ller (rev a1)
02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (r=
ev a1)
02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Con=
troller (rev a1)

What I do is to select (from the BIOS) the Intel or the Nvidia Geforce 1060=
 GPU,
reserving the RTX 2080 ti to a guest os (Linux or Windows),by decl=
aring this parameter
inside the file /boot/loader.conf :

pptdevs=3D"2/0/0 2/0/1 2/0/2 2/0/3"

Take in consideration that IT WORKS inside the Linux vm,but NOT in the Wind=
ows vm,
because there are some kinds of conflicts between resources that=
 I need to understand.

I ran msinfo32 and then I checked the resource sharing and conflicts tab to=
 see which
devices are conflicting with each other and I found something=
 really interesting 
(please see the picture below).
3D"Istantanea_2024-05-09_17-56-=
I'm not a developer,but I suspect that a=
 patch is needed for bhyve. 
Windows needs that ACPI is enabled within W=
indows itself,
because when I have disabled (configuring the entries bel=
ow to Start 4 - disabled)
the entries below :
=20
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\ACPI Start 0
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\AcpiPmi Start 3
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\acpitime Start 3

what's happened has been a BSOD : "Inaccessible Boot Device"

I'm not sure if ACPI can be disabled in Windows without to break it irr=
eversibly.
I will keep trying to answer this question.

What happens,instead if I keep ACPI enabled in Windows,but disabled in bhyv=
e ?

Windows crashed with the error "Video TDR failed"

The error that I have shown happens in FreeBSD 13 and 14,maybe even 12.
=
It never worked. Anyway,I'm using FreeBSD 14.0-RELEASE-p6.
--
Mario= .
--000000000000a0303106182e0be3-- --000000000000a0303206182e0be4 Content-Type: image/png; name="Istantanea_2024-05-09_17-56-58.png" Content-Disposition: inline; filename="Istantanea_2024-05-09_17-56-58.png" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: ii_lw264qx60 iVBORw0KGgoAAAANSUhEUgAABj0AAAM7CAMAAAAmuKIYAAAAe1BMVEX///9JSUnz8/O2ZgBFRUUA AAAYGBjw8PAgISCCh5AAZbb/2pBMTEwaGho7OzsDOo//tma2///b//8AAGZmtv+Q2//bjzmQOgA6 kNr//9s6AAD//7ZtbWxmAAA9AD4AADri4uKVl5W7v7tUYrBnpePbqnStdz2p39NmHXnt/x/zAAAg AElEQVR42uzdDUOiShuHcRodASW0UtSk3ZnRns73/4TP3DPgS4tvbXvaY9fPEoOBimr+3gNG0tfz qpoP/LvWg2LQLwAAOCcZ+PCoXnrVPKudSl4HdwAAnJPoajQaTSajZSJSzS4BAFyQHolJfWwY28/K osh0H7gdtbJ7k5OMEnna1dIYf+dUofx27qy66xfWKJOzf/Gd+fSwJjH9sr7XXkl64JYUMTaKC9Kj zp3K88LWHcusqvt3qeurxBR9SQ+nnHU1+xe3zgTvnkk1M3x6uNpm1jqndVakpAduq/ZwxlcTYeL6 JpV+PzWhjshNag77fz8zVCk2NRIXsUGtcuNkrn/QV04WK59Grl+yd/Et02M7I9F1qTOXZ7XrD2yS 6BK4HYUyvqSoC5WGykLVpXHx3gdK7cxBY6f6fgXrl+RGFU2DQvkPS2PCUmV9wlhV+iRh3+I7MNu7 7RwvPkrCeJWkh5XwSHQG3I6+slnhu/rtpFB1Fu5zlaZKhUZl09j6D32TXBVZrvpNg1K57N5/7AMn y2QTPn2y2m8F+AbM9m47x4uPEm3zUltn80Edzrlid+GGlAfpkTkjCeHvTSYVSBlywykfF/HBYXqE BrJqSCEJDL8Jq3yzvo8U4AbDwuzCoTs99lokusxTVw70IJczdiekB26r9jC1U0WbHoWSCsLfWwmA os5Dm6I4qD0K5Wqj+k2DmB73Jk2zkB6ZkYrFKVvbgv2Lm0+PX2fE95geunSJUuHlHquK9MBt1R4u TW1bgkjXL0NORooNnxDGHjSW2kOGumxqnOo3DWJ6+GLENelRq8RvV87YJT3wrSW66g107kJ6jCrS AzfN7Z5F3b9ftj/DqvLXBgAO0mNeVS+DrCxefXZUc9IDt6vIlb2gXrF1cVCrA+hKj0ziozXXPN/C 7bLpBeFxX5o0dXfsLeBceoTXmLfYIfj27nkKBVyUHgAAXJ0e9/6pVitr3rhxu9Fb1jWvfWtu2z+F 7i2wE7lxC38nyWBwP7iXN38HYBD+FPhrAE7/nQwSdgIA4GoJAAAAAAAAAADA7VAmlQs6syMAAFcw +Tq3zhljUp8j7A8AwEXpMR6Ph8Ohm7m3N8kQwy4BAFyYHguTpo/po3kzjl0CADjL+fRYbzabN/P4 aB7fqD0A4CZKgyz+91t77Yq7NWx2opmyax8ei7Xx8THzN7dNj57/tMOrPmVvv3mlnx4uXvHypgCA i9LDBbW+Nj5su4Y9uWoq6bFeL9x6495ms7c2PZZziQL7eNh6Mn68MD1GP2YXfZmntwgA+FB25JFL 6uzadV1MDXc6d9Jc0mO4dsP1ZjObOadO9f7XpMdl5QTpAQCfr8zKQDunr145FB32TNFifOGxHg9z O167zWbq2oPmq5fpLhGq8eNE65+zZjCrksd+0WKu9XT0vB2i8m2X881cFlbbljLxc1+eHg7a+4n2 wRG32Fsk+23DFgAAH6abzvyu/kB6SNlhB2dGvOSUq3/WY7eR2mOz2aaH78wXbW2wnE+lFBnNYqXg sySZ+ATohZQYP8ZBrjY9/MyebxBqj8q3kqXL+Z2kzX7711kcHAtblPTYtW22AADfPgP2fCw9ig+l hy88zoVHOOVqLEc+gsUuPaQ68J36SmLjx0MzECV9vcyRQAllSShR2tGnkB6LOGol76GlPApzf20v 0zY9DtteOu4FAPgD6eEuSI/1+OnpKQbIcL12+y/3mMhIku/ZK0kFPW37+xiD01AxhE7/ID2m8ZiJ 9P+TMKTlm4S579rLQNVeehy2vfSYOwDg89PDnh+5Us5aZ9fjaLh2Zn+pdOKTsQl9/+jZ9++TZtAq hsVvpMfqpUkO0gMA/mh6XH3OVTzb6txRcyWnBFtPao/xu/SQcabVDxuHkWRIaRKOaMwuTI/RbjTq fXrEwyJ76XHYlvQAgM9Jj+tfA24vOmO3SRCb+xpEtJ9mtE7ikeykV8rB7WkIhlAhyGv7lvnj+fSI LX3odKWHb/LcVjKy5KAt6QEAv5Ueue/P/S3LP1Z5SDScjw9ffPj4yGufHzbdlh1ax1NxQ08uJ9gu pP4Ih0J0+OB8eoSDG4ukIz1kyfg1nM+1f8Zu25b0AIDfYZsztcqrK4/kfvefSu5Pt5Sxq3b8ypn0 /WJezwcAOJYeUuG4rvSIZ9sCAHBIGbmuh4sh8j48elf+p0QAwLeRmhAfJlEJVxcEAFxRgLALAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8Pnkgk96Gi7w1zv5H9l7P+N1m1Yvv/zv3WoYrxCYVPrp odde/fzUtobt5Hjr9rKDV3wjP7myFAD8W+mx6HrY2TvHxVXZmR7i4gsDhqvShounX/SFXWK1aTcL APib0qMMffNyfio9Luy9Y26c7ux711+basVlbQHg302PXnO5crnIePcAUG8RrlE+Gf9vGJvFq0f1 tA5z/Mpx3vZS5dNwTfTuq9uGoateXC2ZyKcMPb/ET7tS8xVNwgV6h83kxCYTLqULAF+WHpXvgLtr gt6iip26VBqVb7KcS+8v7XUTA6H2CFvxi0ez5HUWG3X082H9aWgt412jXXq0K23zLH5RzeT4Jq8Y OAMA/HZ6+Kf0zRFz/x768NCtd6SHLPT9u0+POETkH8cHvXfpsX9Ao7seaDbVpMdDM2c79CUr7dKj SYVtOBwpMQgPAPiq2iMOEOnu9AijU0N5a456/5jF7r56lx7b4kUGsI4MXS3i0RI/Xc7l8+3So1lp mx7LefgSm8nxTVYLfpoA8EXpcfxAdqgS3uazg/SID46kx+pFhrW6u/rJ2IQKJ3z60fPTQ5se25V2 R2J241YnN0l6AMBXpceJ0Z9QJdS+4/ZZMWpHrkadI1fboaaHbd//3uqHDWVLr60rQnr42Nmu1KbH JB7Ej5NTmwQAfFV6hNfnLfPHIw0n8pqP9rWBYSip+6h5Lxw1D4fDn4909b3mlC1fqUzlSIscC1+9 PD1sV2qPxLwsYh2zPQRybJOECgB8WXqEo+iLYw1X/zw0r+6ommbLudbD9yNX7Qu/faPx65E+fRIP rkil8hw2tXrRT85voF2p+Yp68VTdZnJqk6QHAAAAAAAAAAAAAAAAgD9HAQBwrSQFAOBapAcAgPQA AJAeAADSAwBAegAASA8AAK5KjxxoyK8EewHABX2FpAcvmUQU04P9AOB8X0F64JbSw3zbn51ht4P0 AOlBN0Z6gPQA6UE3RnqA9ADpQR/Kd85up68gPUB6kB7sdpAeID1ID3Y7vjA9lvPx4/VbW86Hv6z5 sS2B9KAbIz3wH0qPSns/Z38qPVYvU3Y56UE3RnrgBtPj6cHfSXx8iKTHKaNn0uMm0iM+JRg968Vn lKx/RTf2V39Ln/addz2/Iz3weemxelmQHqRHMJFiVC86u9qO35O/Kj1GP2ZNNa3l1/qS9LjuW9rt nIkeNsX14ug++3fTo+r+tkkP/Nn0CBnQG8rfQvwFlKmWfr83fvUz2tn+eZr+P3vXot02jkO9w6Oz rbJnoqSK4iZN3FeS+f8vXBEPAqQo2c5j5Cb3zpnUpkjIFClcAqCE1jHDeMdoS7mVqJ5JohndW8Nh FNfsrr5eN02cz/HfpsWYnBR78EBXR4W08wmvwJk9SIMOB86r47pkF6fj+0PYY+Ga/UvscXZDBPF4 dez6DuwBvILtEVqeaQ8X461BC6rx9gjxJtTiaEjc3rVpHabHlT20npPEtkdqSPd02H3q5a4jYtli UE6NPTZd1Zf5B7HHofPqGexBF6f78kifjT1mrtm/0/N57wHYA3hb9ghx3TLqfPEzyYSLhwKvqlxx J8YxfyNOUPZQN5VJ4n+t4WCLtEGN6QDj4zTZQw1Gtj/HYQq8wWJSzpbmeGQtW7LGHjypeYKRlazm 8ku65NnjSW6Ak2CP4FxTyXFQeAekRxUPgus6/S19D0UTsAfYQ9ij0ak2zrDbO7oNRPXHe5FmZVas izX+9tV7rqSek8S1rOHA4npzmTXswwJOiz3G0UkGY2Dt0srY18rjOv/sajVbssoeSZmHT+MM45/4 0i6lixM/0bR27DEcEGt5o55708McB7l3QHuUexDKrtPfwvdQNAF7gD2c7UEGBs2wUZvHlRgTRGIP Lb6h4GDTO5M/Yw+p5yQpe2jDgj1CnKUB7HFy7DGqDzMYbXrEMa+VZ+bjCrbkHvZQ3f7yLunFMUva 2KNbI2z+Pbf6M8dB4R2QTtQ8CK7rKQzqVEPeBOwB9vDswbNDZlfYXZW2R1682czZHlLPSfK2x0bd VYk9uBzscVrs0agzp8/mgKraSrk91LOSLbmHPegHjcde3iW9OKxLo2K2PVe7NWJCU/awmze/Q7VH NQ+C6zpvYs7v/7wJ2APskcU9aHoE3YLY69pjnDlJtVOxX1yxsZzFPaSek6Rxj22dPe6v8GT6aXqu vMFYqtpJedIra9mSVfZIjiSejqROX9olW3+LudE7z9UamHquzHGQeweMLzIPQtl1YovC95A3AXuA PTL2IAM1zrDfEhI0vyfNnlQcHb2/tOX4bVyY+ai51HOSeFqnhjl7iL8V7HGa7KHKorZQz8qzLRKn wh6214inZ+eN5+d2qWCP8b55Ogn2cEuwOdsj71FhqVjX522P+csOfGz2uL07v6QZJgZ4+pDWHmyu ZwZ6DLj/vs72XDUa9/BNemtYxD1oTwg8VyfJHmYw5qq2Uq6KejVbssYetgNKPPrnly/vUske431z Euzhgi7mOMi9A9qj3IMw6To/hpX7HvImYA+wB96xCyyyhxmMuaqtlZPuOrtazZassEcwPwvtG7JN SC/pUskecVl0EuyhT0c+esdB7h2QHhUehLLr9LfwPRRNwB5gD1wJYJE9ksFYqNpKORmRK9qSOXuQ j95v5GgHddq/tEsT9uC4yAmwBz+i0bTecVB4B6RHhQeh6Dr3NPc9lE3AHmAPAKiwx5+IZTX2nh9H /f5hTw6APQCwB9gD7AGAPQCwB9gD7AGAPQCwB9gD7AH2AMAewMdkj/cMsAewOnt8w3UBe0CHoue4 7ABsDwDsAfbAZQdgewBgD6gxsAcA2wMAewAAANsDAHsAAADbAwB7AAAAvND2OPp1qcNifXtnNgD2 AADg/dkeg2ZKA3uAPQAAAA62PegV18NzEm2CPcAeAAB8FPb4+UPx07OHy3UJ9gB7AAAAlOzxI/75 K31S9tCElpQ3gOlEkiSE84cmZpXl4pgJoDX2oHQIxBSSbIBrc5qEUSS9c4hIhtKdTSuBYP5E9gjt K7xOqnCUDsg5+YxRyBdrh/merdYaiSGBP5s9/mL++DGxPTSh5cOFTzcWNbwWx5THlH+NW8YkbJzF rY/EspXa1jDm0fl6veNsoEWlXBawGntQ4Gsm4RGnFX499pA8snG6LbMHJ31aTsPUUZ1t3ouY4ZtS QEmKWp68nYrrZMFSGNuS8ElXQ/7snX3u7HTVKv6Me2T38WsjKzQvl+rtS9hIPXwRe9DAgj2A422P zcT2CGIfRJVu0z8eCmxyuOKUWm3gWHvrdUoyZIgZ4iw/+884UeWGnlRaL00b2EOVFWmQx3rcSwb+ LdijPFKwB1UcFo3Tjqdmm/UizjT+bZ1kq419cAl4uTT7BUFoINak9Y0/u0swOM73RKe1Kv6M87J9 R+lLLpduqs/PYo8jUB9YADiSPXQFFKfj7V2a6jJJiVey4sQEgyX2JK/U+JVqczVeDG03QzuMa79Y XqmUZAHrsMeegNfK7MH6dpE9RPduvWI9u79w+jSKMvb48kgf/S8Ym/Dx5Gj1Zzf2UBfs5Ad2hc0x 2JqoItt1lDnDyz0sAAn2ANZhj1+M8dM32nyVDAyajqOG312pUk/socXO2s7Yg9I9h0QM1Jqm80gc oR9v5kqlXBawDnuETJfJeKSAlqwsJFBlsTCfO9tFsphZ1EVTO+rZI7TqtKGZkoJomQJeSptu7OFq xckXFy1JBp3LsceTWMZTzxXr8ijAnz1Pbh7bUb1qlWHCTBXZrkf82ct9FntomNFHLt04pHFLh9LA ti7BeaoGADPs8V+F91zxwkWmY9hdlbZHXpzP/eTvClPbI96s/1zc3j9d9/VKwLrs4ZWVxbosoCWD JtGsFAsTRZVCV8E8TOHT2JZW5LWjJXtYaM3OmbFHt7CVnJX2EPd0bHPFyjOTfzxJcOxxQcU1Dc+r nqjD/dmNPb5ejzRIK6x+U6/izjgv29hD3FWZ3HCIAs/Zwy6ii1zaOKRxS4faNLAt/9j4Q2x4AWCG Pb4rsrgHTaZkPPca91BVn4q3dfa414cNxc+13XDcY3N7/zmW/u/+oqxUyAJWYQ/H4S7WZdvwRMlY oIpNVaeoSDkGc9UEVenVoxwy5iBxkkAzpQiiHcweUen5lQhNs1uxfsklRKEFOmnLbQZZ609E7ars oU1pcU7zmwiqVsWdcV62i7Yrlzq5MVCyV4Nn7OEuokUuy3GIn9Ihzx5qmrnhBYCj2IMmW5x7v/V+ SOtMMhRScVy2/SrZQ1aYiWsGWuiw1+Lvrawry0q5LGBt9nD2prkkzWDMYmFeUWUGqqrOuPOpdrSw PZyV6oJoB7OHviVhwh4aN0/haG970Ew8gj3MsNgyyVFJ3TzJAuAzstPVkAVULpe8xPtUeMYe7iJa 5LIch/gpHfLskQ8mApHAM9jj9u78kuYQ34/2IQUp6Fsn92vOHux4DWapxPXVb1rMDHx3bzfTSpks YG3PlYt11dnDxcJU4WjoyrNHu1GtVTlasoeLkNXZY2ktnJT2xHMlcfNYzgIy9ogLl8M8Vz7gLovz bvdESr9WxZ/xAM+V673J3cjNeAR7mBiLXLpxsHFzh4w9uuj56/3wAkCdPabPmgMfmD3cjv/n2h62 0HXs0fkg2Qtsj8WdRElp++cWVJdu+czfRTdn7DFa2k+HRM0ldHeRUVRo1Cc0qeLPOC87/Vb+VMjd a3HtsT1I1O5qMg76EIwcMvYYf1lHIRLYHsAe9gAwI/yeqy5tc3Kxrip7+FiYxj22FfZg9Xx+WT1a sIeLkNXZY1GNOodRU5pQsptp94O/FuwxLu4r7MG/2O3YzQLu2h85V62KP+O8bHvSe1uRezx7uIto kUs/Dq1dED6Uscf4J2yz4QWABfZAdiiwR9LnvCHq0ce6TJOzZtJAVbnnKoWuPHs0eqB2tNxzZRGy KnvQBqRoWej/M+zheiHsoXFzeWq7YI/oNa08a97RrvQ+O7t/0JAU7u4HKd5qFXfGedl6XI2zTO7t P5d5j6vPg+d7ruwiusilHwcZt3TIBpa3XYtT2u+5Wjw9ANsDAHts5GmARp/eIEXiNHnnAhcWC2vd 0xrkSfeeq0EfG6gcnTzvMVbePdU8V2mn00HsYb1Q1wtLGhpd2EsMRtuEGntQtT4/uzXlOA9H+dqZ KnbGednaCxeON7mpH0ewh4UZXeTSjYMW2aE0sK3z+6XhBXsAsD2AQ9jjlfGMZ9DfZPM2x80/LgLe IAfA9gDeO3u8yYOj+ymJl/nNmzzg8JaywR4AbA8A7DH0G7xoGewB/NnsAUIBe6ygtbqmaRq8dADs AcD2AD48ewAAANsDAHsAAAD2AACwBwAAsD0AsAcAALA9ALAHAADv3/Y4u2m29J6j/DnUY59L5dy1 1sZ9nhOFR1/BHgAAnLTtMSw87KTvmSBVzi8Ppb+q2vWFa/uelwJ7gD0AAHhftsfZDSnpx/qbPivv h84eF6bjyyI2jmUAsAcAAO/D9ihS5xzLHkO7VwTYA+wBAMD7sz18YmZ6Yw+/G5SSA1Li5WZ3Je9Z JfdUP0gSuZZb9PtESKJBeS0o1w2U3JAqS+rsWkM8Qgv2AADgVG0Pbze4pA8xcQF9SraHpZlxyWY2 HUVDlkVwmghJSUC5cCRtDdkjDxeSPmLaEOwB9gCc+X7sRpV9OW9vxpuO/gDAc2wP54bSAPhoDQyc Iq3dxx70ctM9IrhM06ER1XRscGhDJZLy3GCP12WPrmnTiiG4DN40xoN7UazmzegKxTJYaovZkiVQ bcpmxGg7NU73KjBOq7F1csafapm8fXc67YnvrxclnRQL13e8s8+deylXrUrIU6Bnsk1Msr/tN3rB nO5jWcnrRXt19qDLcojXGYDtMbf+6IuPRbLrRfaQmPmiiCIVs/v/9o7vJyOS/Nxgj9dmD9bzUWNo Yrl44Zk9OIN30xp7fL3+2+sf2RvReXVzXHJTOsdAWYk0qeyharFjXdxu/B4Nzbu+ybrjEgRafxOC 0AB1PLKW77jLfzv+MN5iuKlX8WcsZbtcUH2ixrOblHfQBFP3P18eeNFeF3zzIrc5cKTtof9ld5ZM pGPYQyIWiyLOJG+oZjj9csHBEsqppvnPqucGe7wye3x5lEzf27RkHngJnzIBkq5Lo/HT6ZbqKvUZ 7MFyEkG1+zN7u0U9qd5sunFmKN8dl5nW+uua8HHOdmh5zanjxh7UMZ2AtSr+jKXsJCadJNb/xhzk BB+08HcXDewBnJTt4dZ9x9sevIbaI6KwPcY2nbuBAudehu3xr7DH03Wi+8EUq2cP2taghkG7Caa0 Qm6HcKbWxB5p00M4f4gL8PhdM6S2mSJktWvOsf4gzWjs4X5HFCKZoVx3HHu4/haieOrSfHMdd7YH 9WHL9apVhgkz2c+UOnaSeKUe0jEVfDh7uNuH7ZviemtxdrkpmXzag+K3qMSLRhtiPjUuQy0A9jhy z5X5tjX2ENdjh7GH6I5lEXyLaNwj/gluQ5UPoJfnBnu8Nntc0NgxZdAo0rI/Y49YwvouDo2p00zT xcY0osoetukhSNr0eIorqzhje/yfvathaltXor7j8QzPvCmmDSaQC3n3toX+/1/4rP2WbAcDAQc4 mykFRR+R5Ojo7K60afV7WMJf+IP0aZ3c5rtn/gihOwE9Qn9LJZh1MXbcuztsiX5zyPFNNZ0ltDiH Ht6I85Ks4nqBycgHLTiaFOOtyflw01dUfVCii4rYK8E9IK/jHqrOTWrk4Pc0jR4KBLL6W0DQw1X0 9Ow2Cgfnt/SFS4Tjl+wmc58roMfboQetRTSBzBhpIz+DHrxL3uS0MmwSqDo2Dwenh9r1QjGjL4TM HWzVGrbES4zuVEdHPkKbHD12ohm17rDNui36W1a1n0QPLUo79VTZrhwbyxJaHKGH5PFG0lelE7IS Kk4E4Kne+6A5Ry/HO0vO9GbGXIKLCtADciTuoXGZW9FH7K+qWfQQO2Qn5z1Y5fxkFbTE/DInKlF0 0VehUbvHZEGgx/HRI+1MeTV18+0MerhlK2y1yUVIOKipGaPikdY5BR3PKLsMc9CytGHeF6GHeh6N 0EPt5tadyD1CfxehhxOLLZuqKWWanmT270nu4Y2YziuvmEx/TwRE90FzR5PxeIdkp3VtFVUA5qIC 9IAci3u8XLC6fzz0cEND+snLYoYeZkYnZ1LXitsSXCf0aMRTV9HDnR6czVRWRxN8l+rcKXjYFtdP LJ9hWa5GJnD6eZl1J0OPCcPKjOYqGtxlb9/tH4leT2WJLT6pueJPMvwoKuZqLp70uXJCx44m5Xhr cj7cGXoEFxWgB+Ro3OOlggfvI6LHsBI98mpaX/yP17KIHn7Ip7AKm29ErdyjMqI42gtrhuxSG67y JrRZ0TbcFfVL0CO696o7X96dHD28v1lVI6s5e3+ZuZtL1I2aW0ZZYotz6KGNPF7LAZdtUXFgKAfR gwfNHU3K8c6T4+4uqADMRQXoAVmfe0A+InoMe11ewrr9P76F1RXSzOjqkWqrkx5wGBLM4KXoEZwe KL8W9oy2EObqFXaCXeyxy5+jdBDnD2HdKdDD+ptVxZ8seOxmBnf93NLWVJbY4hx6eCP6XlHxUvTg QXNHk3K8LXl7ED1Uhxb8soEekFW4B+RDokcyIWx5bWuC3olXSHYConMUso/tzCwhvhG1XiSz+znp c8VLecM+V5bRF0Kx8wZbt9GJ9Iv+m0WP4KMhS5/azaU7BXpYf4sVfi99jB0PBw3Jr3D/D43CZJbQ 4hx6eCNRkeYV7/58zzs9dW7SBy04mhTj7clxuDP0iFmGDx7R41DrEHAPCNDDF7VaVtNe/g83lbSW MHETx811do/H/iqc9zCnh1p1WZyziwcKVBlmR0rEY5XPkC9FD/fR0I1z3cbudKr8L/s7AlG+lTN0 3IqyCYGc1Jt2Jou3OEIPy9OFe1w6sWx7xdaRp9EjDVpwNCnG2/7KhjtHD/NKIR8WoAcE3AOyHD0+ rbjzHwQCAfeAAD2WSqHvnxDe5Tc/3gJl3rJuCATcAwL0gEAg4B4QoAcEAgH3gAA9IBAI0AOrBQTo AYFAwD0gQA8IBALuAVkLPRq88HrGCwL0APcAemA3AYFA3oJ79MFZvbe4ZH5ud4tx/STcAwJZKnha gB5LuAdfhMC40Td2PYVehfQNlxeAe0AgEHCPCe4h8RYoas3ZrVCO/Rn9cn7/L+7gBPeAgHtAwD3m uAffttZLjOmb6y3f/dC3VQ3VFbgHBAIB95jjHnR7c3+hwRc49FmKB9Dh/h5wDwi4BwTcY5Z7MHp8 l2DWfO8cB4beYGS/KvdI17O/7t7ug6VxJfjRhvIl+c9vmy3/gECOwD2+U2Cg+ytGjzrEw4R8eO7B YScm1orZHcJJoUfw/+vFT9B7NDyzGu3DE7tGY+pts96nWBfMra34rQ1Mp4HdqxA3Y0XpNbr8sdGD RqWI+Q7uAfR4knvoK6BH+r/noKPyHweyWf/rAzkS9+g4xmk7sQk9gB6ns/VW/7/zW/rl4Sr0iAJa tUU3OwnNxOjheYfukpfITl1CLKWq2Xtdqvp9RZHX10WP9Hne4kMgMi3kSNyD1okUiKz78Xi34YgJ GtUMzPaTcA8LyfoR0cP8/8IDGYLMEme+zBO7Hw/+CHsyQUNvAQ6VX0sKo4zHm1pbdUvfwrf4Eh5C D3APoMcs9xijR6dxRm+u/x5SEnro0lFDOf2JuAfNNKlsKMD2xe/mxwOHUaXAs3E/n+UAACAASURB VG2AlKbZpkfAVZepWHpOrHg/FB9SUsn0kFBO3nhcqC6I0qSgtiCkdutvNk+v0Ob/F57HHD0kSFRE j8frdoQeDAeiog0AIX9RvhBvahZa3xM9bnQi0mAxnMQZvNTkNLI2XRyZNqrpODg6d69OaoUz6BYg r+YetSwG3+WYeXq89GvTNbCbfyLuMUw4WRBoWmtaa3imu40c+/FZp7cNPejN86vKi9Mh03p/tuEF OqBHQhOuog0FQwukla/bKtR2UMz/L27DrUcSm51wxROH36hrAT2GZEaJ6E/oKV5cIWp1+uXco275 0/y+rIoZ1OTU291dG9FDJ0Kz+ByBe0BezT1oG8gPHH1ldn9k71XbwXPYzT8P9xjWHZnQtCzxHIft tULFjWzaI3rIL6F475aUXsAgKD1tw5w5XsgfpMnX9dDNb7Ni/n+RC2iPZBVkO4YndvIRHT1SMqvu CD2YdIUUQw+uKpGltbdONDK1UDvpfDmDWbL52BN66ETosC1DDwjQA4InovS5SloeXkjI0ExbbFt8 GtZAWdJN1FypASAU5+0GpfdRKyJbd0EOtxx4C2KjbmNtB8X8/zL0MK8OLk+ZPJERpw0+Vyk5w4oC T3zxdcxc22reqGKPhjLCvc1glpxZc2wiJAu4B+SI3APytbiHLy8letRpu65GhXPZewf0MDdWL74M PXw5sxbEm9TffAo93P9vrLny8mTs9kShG5uguSr1VJW6qZfo4XbzlV3WnSLWpKNq9lfjGZRkHiSl Sxl6SBZwDwi4B+QF3EPRY5J7CITMcw/d9D+be+QFqYXeNsILuYf7/8VjDCV6UNsFegxL72OGHpmN nGoeW82j3bxf12ukJyeWcPJqIEOjGcySI13ziRAaBe4BAfeAvJh7qNY8WSoCetzHE2m8w8/sHmr/ CsVL9Eh/8n2bET20oLcQlPChtgMS/f/CqcESPcTnPEeP3d1Fhh6MC9ZgLaebNEXzeYb1uQcPWK1n HzejGdTk7UH0SDPlcwTuAQH3gDyTe0SPHVpc5NRxWuvV7lH17MMbbd5sns58rnL0YN+epkQPLWgt mB9G5nOVcEX/jRVXwf9PwtDoacGIHsnYXaJHsnjklERtPw+X3E9PCegxVLX7WZ3IaUHSXSWm9kt8 kfMZ9OTkYfZzCj1iFp4jQ4+JgQf3AHqAe+CJmOAe4QYO2bnSoWw6GOCHKZK19ld+3iPl0FMFVLxA DykzQg8tqC3UTTzvobUdQo/c/49vGGknuEeqcYQebG8J1312elbC4qJ1bi0ILktk5F/7llDV2dEd dDby+QzaX108v5Gjh82ZztEh9IAAPSB4Ir7YHbvhkPgJVfXhBNwD6AHugSfii+0mCtX/iVQFgYB7 QMA9IOAeEHAPCLgHBAKBgHtAwD0g4B4QcA8IuAcEAgH3gKzOPfDCa/kLAvR4Pfd41g278BgH94BA IF+Ee8iNQ7PBy94SPdYO2wb0gEAgkOp1sQWPgx7PlLXDtgE9IBAI5CS4B9AD6AGBQL4g97DI1hws 2UJb1xo8ji9FHWcqQikXwa41srIHY+YYyxr3ZmFgawjQAwKBnCb3sLjTHO7a4iZ3FGJgf0WXNxSZ pkIp58Guz+hK7m12m6qHYV4c2BoC9IBAIKfEPZpGbzx1AKgtLDUhQ7qI8/yvu00e9DJkGodSDsGu 6b0h0YMxZ+GClga2hgA9IBDIiXIPizst1z5reLlEOfq2H0gIRxUdZRqHUi5CBqUcHhBNYyyHmHYI U3OK6MGT/xov7IOl4eB9tKF8Sf7z22HLSD8gkOXcQ18FeljcaQYGD209AEe9Ob+/JMqRZ5oJpVyE KyX08GDMEmPZY9oBPd4BPbqMYwaZ9Zw+KfQIMQWJMFM4Ku2RP18hsWs0pt42670a4jhOyDakCI/W qk7BItdLPPejo4fEBAN2QI7BPTzudME9hm/U45/L3f3j9WY6k8ksenRZMOaKYyyDe7wverhKsdyE HkCP09l6f7OY6/SLxhakHlGI77boZidL/84i03Le2w0b4nb6zFlKChiV8klVQxOrW+Q47NYbBDhE ZFrIi7jHDHpY3OmaAcBCW+/u/5NS/3t/WWaaC6Uc0IMja1o8UokXnTJkdg8oMd4FPSg86UdEj/P7 fyX67N123CMyzXE4J0/sfjxIYL5tTDYXwkzhqimMMh4Zau0xoC/oW1AEoAfkiNzDI1uzrcJDW1f1 txRuNL1VZpoJpRzQg0KIUvRoda7SGMv8nYDP1bujh2lkyOv6gVWP5oqtkMLu2iEybSqWsMeKj5yz K9lP9BeqC6pbL6gtiLpz628umP2kPaUl1IPnFughO5mIHo/i+RHRgx9NhpuroLmTvzp2BtlmO6K1 0eNGJ4J93atiBi81ecJ9PqrpVBWQupfC8u7PYiRbCNDjFec9LLK1fEEttDUzZ+b2ZabpUMpRc9Xr 6qCxlWMY5k34E/IO6DHMieO1e10Hf23ObG/b6klvnl8FuC+dsx09EppwFW0oGFpg+tou3jykB6or t+HWI176mb96YifufQE9kscfoYTVZ7gh1h/JZxC1dohB5x502Ip93YsZjC7wpfu8TIRm8TkC94C8 lnu8udQtZuaE0GNYd4KPNKsVg+ZKZ8vdtX0C5ZdQvHTOdvTINszZIyB/0H5E18MFDtvpw9P6HtVs 2iNZBdmO4YmdfERHj5TMqjtCD/H38BRDDzOJvInJ4dnoUQu1k86XM5glj93nU7oOG9ADcjzuAfT4 Ough3jvBT0HcqXXxYVfsqspcJnQC1TMrFJ9ykBDN1ffKkMNdurwFsVG3i50mqO66LdBD/ZGkfN1m iYw4bfC5SskZVhR44ouv9ro/CZ+rjXwk8XUvZzBLHrvPh5JADwi4B+SlmqvMRzqih7lie54MPXSh CcWXoYcvZ9aCeJP6m0+tYmIqMWNZ3iMtT8ZuTxS6sQmaq1JPRR91QnMldnO6YWFl9HDPE9JRNfur 8QxK8oz7fCgJ9ICAe0BegR6T3MNdsWe4R3ZDwLPQIytILfS2EV7IPXpznorHGEr0oLYL9BiW3scM PTIbOdU8tppLVbu79T0Be7omqPVvUp05v5sPgSfHb55PBGcBekA+EveAnBp6BB/piB738URacNfO TSFZ8alrBUgnlaGHFvQWghJ+mcO2VlGTDXg7hx5UTYkeAwRk6MFmcGuQfMlDiubz4VkfPXjAaj37 uBnNoCZvD6JHmimfI6DH/9k7A6W2cSAMe87jGcbMHKZNTCAHTHsF7v2f8KxdrbSSQwiQHr7k+zpt g2LZOMro96+VtLB47wFLUw8/Y0c6l7jq2KZia6eVpmsn86jh6WLOVakeOrenq9XDKqYrpCUUxZyr oCv2dz5wZVEKm+mVVwt69QjB7lo9QsSjtCQW+3m+jnGNIUeD0nHbh/UispeN0UCJtbO57lUL5uKd 0+fLmtZGST32ffCA9wDUo3hStznSedb1Kk/FNvmI07Xz0GM4wlYVSPVKPWKdmXpYRbtC2/n1Hna2 fZ1YW9gm3WGk3+E9whln6qHxllwsAXSN9Oj95JKmmLKUgy1frh5hDE3G+WyWQNGCfgr8junzja+Z 2gj1ALwHHKge54BbJL6gUwHgPQD1WDZHXNv31csEAfAegHoAAN4DUA8AwHsA6gEA9BV4D0A9AADv AagHACzVe9hEeTeTnt3SUQ8AwHu84T2Kve/GLqVrAtQDAPAeb3uPsa/2gwDUAwDwHm95D9luonU7 x/Fxoh4AgPd403sMkrLcRqzezPQGqAcA4D1sw/+UHmAgcI56AADe423vofttoh6oBwCgHnu8h/1p ioTQjFyhHiVhGvfn9u3eW5stwY/2UX7k+Mu7bqP/AHzce2iKhpS0h2yyJ6keQ8yqMePVLEiLUg83 kTzkqOgkHZXd0fQ1ttmDuXDoLKfeprh7yTPSWJ6QjStpNL+Hz7jXfvGT1GgpRo6tHjEnGNoB7/Me tXpYUs6OGbunrB6a2anf8RC6Rz2W8+j9Z8q5Li8st6DckaT47qvbHGLKp23KTKvH3q00N+vWRCKV hIRR4TifEeur1UPSbv2GoWQy08IxvEfKZKCrBUe87Omqx66Y1v9CPS4fftrk8s38jsLXOKZzyoXD 9+eYmG/ji0UawgNT6jp9iarMQ0qC/tcS1ON3WATUA47hPXIWNfHyBD1OXT2kmSU369VT9/25k59u bzTda5KUrtuUmWlDtcd1k6uPU/WpJNQMdkCOvNXMtDYW1Pa5ol0hZnzd5DcPMLtj37TShbrFSKV6 xIcgrx4vN/1MPXScTr/363hfVmLV8/PU1dMS1OPWGiJ8WConvgWvrTh8sqm5NDOtH6bT5Oh6eyEt 7+MFGxPBJ7xHevSC81CPqTeSCIIMT7Zdn7zHsGri8I0cnN5O3w5583Ld5OqjvP94sdIO2qlHUBM9 Re8quivIqHzbN+5sewk9/FA/hqc78gOwuXB6Jbfm1GMqVpVI50u6EaM/8TiVKPMii/Aeba8q8nTd VC1oxeFut/c+epkawg7JbYT3gM96j4Zv0Fmpx9TvxPGo0C1pSNiNXJlU2KaZXj3iC1d9zJGUMYpB VI/igbl4Ook/yEi+9YcHzPILv7z0736Yze7IJn/IFzkXDvFXzOoRinXoTtRDTZcrSeqxTcugFqEe bbR28ebrFiyK0y8s6mENYR8b6gHH9R5wHuoRZ+/EjkQCzfKInTqfTkegUtGtH7mymVmuujyha/no R0Xio3tUjjylK18hxqh7f7a3/XHbV+ph85H8xPNcqIrTuzlXobjQikpPcuer1wsP7kuYc7WKv9L2 3st9asGiuIjmpIaIh6AecFzvAec0cpV6jFo92vC4bkEF24LAqUexG/N71CN3Z+kKcTZpfvOtXiyG SiRmU49c5foSwMuF0W6s3MhVPU4lv+qOkSs5ldZfgPdQi9jKGFX3uJ63YCzWD8lGAQv1iIegHoD3 gE+ox07vESXkde9hD/3v9h5lRbnCmB6ED/QeY5o85Zcx1Ooh167UY+p6Xwr1KGLkcuZ51FxOJU7p lSUy/6166JBUbIj2cT1rwaLY27XcEHoI6gF4D/i4etioeYhUOPV48CvS9Am/iHvY9F1XvVaP8KOM SRXqYRXzFdwgvDvbHuwUMpTUbV5TDzlNrR7b+6tCPXQ+1ej2ky5K7Dg7YBFxD/nAWlv7uJq1oBVv 9qpHaKncRqgH4D3gnerhZ+xI5xJXHYe+3uIezahzeH3MW8PTxZyrUj10bk9Xq4dVTFdIy0iKOVdB V+zvfODKrWMddVKqrRb06hGC3bV6hIhHaUks9vN8rfeZS9ynZEsJF6EeMnYVnNqvOBe5bMFcHGaY /b1LPfwh2kZJPfZ98ID3wHugHsWT+l2MK7e2yUC30gUeeTFFiNb+Ktd7hCNsVYFUr9Qj1pmph1W0 K7SdX+9hZ9vXibWFbdIdRvod3iOccaYeGm9xKjDYWom0uGnI0YJiytJi1GMyUN9knM9mCRQtmH4a /PqNUj1Sm1kboR6A94AD1eMcyAtfl3QqALwHoB7Lphr6X8ipAPAegHoAAN4DUA8AALwHoB4AgPcA 1AMAFuw9dHPSLs/0Y5N21AMAUI8Dcwvm/GWWIIrsgqgHAOA93vIeKYdASk5LzhjUAwDwHod4j7CO N+0hsaScpIB6AMDCvUfe/npk1wLUAwDwHgd4j9ZtfnpAsjdAPQDg7L2H5ZRBPVAPAEA9XvUe9sd7 j1ZHr5J6MHKFegAA3uMN72H5y4iaox4OnUjxmeeIvbXZEvxoH+VHjr+86zb6D8An4x4yasWM3dNV j+G1NKvZcS5ZPVxOQRlplXRUdkfT19hSfOTCobOcepvi7iXPSGN5QjaupNF8Hi7j3len98greY+t HjEnGNoBn/ceMQWPrhNkteApqoc+I/Q7HkL3qMdyHr3/TDnX5YXlFpQ7khTffXWbQ0z5tE2ZafXY u5XmZrX0gbkkJIySGSSW/PDrt11wK3mPC5lp4XjeQ/KX6WPY7kdUOAH12OUq/xfqcfnwM2afvd/M 7yh8jWM6p1w4fH+OX+yNLxZpCHG91HX6ElUZPZX9+OXq8TssAuoBn/Uerw4TdIxJn6x6yJCN5Ga9 euq+P+uMu9sbTfeaJGV6hCgz04ZqQXtS9XGqPpWEmuHbIkfeamZaGwtq+1zRrpAeUNKbB1jdsW9a 6UJz8txKPWI6J68eLzGTrVcPHadTubGt3azEqufMUMtQj1triPBhqZz4Fry24vDJpubSzLR+mE6T o+vthbS8jxc+ky2gHuyxC/vUY+qNJIIgEa6265P3GFZNHL7Rxwd7O6mHvHm5bnL1Ud5/vFhpB+3U I6iJnqJ3Fd0VZFS+7Rt3tr2EHn6oH8PTHUULLbqSC6dXcmtOPaZiVYl0vqQbMfoTjzOJWpD3aONe EE/XTdWCVhzudnvfe/WwhrBDchvhPeA3eQ84UfWY+p04HpWmafuRq7aYcyc9UduX77nqY46kjFEM onoUD8ytH/+KP8hIftv7s+03w9MB0r/7YTa7o9gLahwjFw7xV8zqEYp16E7UIy5zyiVJLlJIZBnq 0UZrF2++bsGiOP3Goh7WEPaxoR6A94D3q0ecvRM7Egk0yyN26ny6NF6pRbd+5MpmZrnq8oSu5aMf FYmP7lE58pSufIUYo+792fYi5277Sj1SZgGtLwflQlWc3s25CsWFVlR6kjtfu+tlzLlaxV9pe+/l PrVgUVxEc1JDxENQD8B7wEdHrlKPUatHGx7XLaigxxTqYR2Nq36YeuTuLF0hzibNb77Vi8VQicRs 6pGrXF+C3bkw2o2VG7mqx6nkV90xcmVx82V4D7WIrYxRdY/reQvGYv2QbBSwUI94COoBeA/4hHrs 9B5RQl73HvbQ/27vUVaUK4zpQfhA7zGmyVN+GUOtHnLtSj2mrvelUI8iRi5nnkfNc9x8EeqhQ1Kx IdrH9awFi2Jv13JD6CGoB+A94OPqYaPmIVLh1OPBr0jTJ/wi7mHTd131Wj10xnetHlYxX8ENwruz 7cFO0UoMePOaeshpavXY3l8V6qG6kC44nbIoseOqH79UPfQDa23t42rWgla82aseoaVyG6EegPeA d6qHn7EjnUtcdRz6+jRPe9Q5vD7mreHpYs5VqR46t6er1cMqpiukZSTFnKugK/Z3PnBlUQqb6ZVX C3r1CMHuWj1CxKO0JBb7ebbVsUOOBqXjtoUV+WL1kLGr4NR+xbnIZQvm4jDD7O9d6uEP0TZK6rHv gwe8B6AexZP6XYwrxydXWZQtCwPyYooQrf1VrvcIR9iqAk1jXKpHrDNTD6toV2g7v97DzravE2sL 26Q7jPQ7vEc440w9NN7iZGCwtRKdLSUfcrSgmLK0GPUIK3llnM9mCRQtmH4a/PqNUj1Sm1kboR6A 94AD1eMcsGD3sk4FgPcA1GPZVEP/CzkVAN4DUA8AwHsA6gEA8J96DwJsqAcAnLf3GNOenOM7dmVH PVAPADhv7zF2aTIkOT1QDwBAPQ71Hhdxy+zh8QL1QD0AAPU40HtcPcd91jY3luFZd9wuEv/U6Wji vms+nRCgHgBwRt7jyvZ1fpEFqJZfpkr8UycUkj0minRCgHoAwFl5j28xdUJYHZXzy5SJf2YJheqU QYB6AMCZeY9vMcVzUI+cX6bcwmi2JXTc6Lkj/TnqAQDn4z3sjxmLjf2X88tU6lGno5EN8Hw6IUA9 AODMvEczfH+5XzXZe1j5fu9RphMC1AMATtx7zNTj9uYvSWC2cTvDleoxS0djOUVZNoh6AMDZeg9d Zq5pPS2/zCzxT5lQaFKPKp0QoB4AcGbeo9n+c207Ult+mTrxT5VQKMQ9qnRCgHoAwPl4D0A9AIC+ 4v3eA1APAKCvwHsA6gEAeA9APQAA7wGoBwDgPQD1AAC8B6AeAIB64D0A9QAAvAegHgCA9wDUAwDw HoB6AADeA1APAEA9APXg+wAAC/Ye5PlAPQDgpL3HGBLPhi3Xj4Zs4456oB4AcMLe4/JOOvnn9REv m7PZAuoBACegHj9/GD+TTdgc/7KoB+oBACelHj/CP3+kV43lBTQl6bpOUgdePYW0gbc3nWSZbXvJ GVgdksrCi94XyVDYSpINAuoBAKehHn+ofvyYW4+cqHwMatA+Xqz0/fZi0pDLu015yFQmr4LRkITm ZVGDeqAeAHBS3sO/cmNMtzfS24eU5TFtuRqKb00rIfWpsD4k/Kxlgy9CPVAPADgf9YgvLx/W/7J3 r7ttG1sYhoUSRLOlH5JtnRXb3XDi9P6vsJw1HB4s2U6ys9OEel6jMjkcCkUM8MU3B65QRl44FUrI o1vNhZddkiBy26iJPdgDwNTs8ZRpjj6mxVeDkaskgFfsMW8vvOyS7VHXeZ6DPdgDwFTt8WehzR6D TRlvZI/QwOrxjeyRYA/2ADBVe/xVKGNYq7qEjzKp0TjgpT1i3qNa3r7skgRx2O3Zgz0AXJs98vqq 2C04WFD1wh6pS1x40SUEEaenp2FTHg9jD/YAMF175C0coZA0hfG4mZ3bY75td3m86JIFsapz27jJ fg/2ADAZe5zvNf8aaIA9AFy3Pb4P9mAPAOzx7W9oZw/2AMAesgd7+HcA8DOyB9gDAHuAPfw7AJA9 wB4A/i17EAp7AIDsAfYAIHuAPQD8AvYAewCA7AH2ACB7gD0ATCx7dHvOBzWlXiFXPH+vF9gDwHSy R1vxI472L+0R721nD/YAIHu8bNrWUUswKeDcHn0h2nftAfYAMN3scV7yY/vhmKWxevzAHuwBgD0u ZY8oMvhHd5TssXyOIafDbp9qlsdsx6E9SkUG27KB2+XnVFOwCSh1dE9lB0MZ6WCf7JF6pcs1k7AH gKllj/9md/wxtEcOGIuHL2f2aLNH2CNJoXr8sM5lzNOF032pcx6l0KPq+Tpd2vtTsAeACWaP2Th7 3IYxqvnhTXuk2ZHFcZ7vyENVq7ubfBAe6dZoKRDCHgAmmD3O7bF42DTRY/O2PdJ4VSzBSiIpcWWT Dw5l5CoPXVl8xR4AJpc9njLN0ce4FkliX359tT3qYJ3EM7BHlZZtVezBHgCmlj3+LAyyx2x196Xx wjdnj1npUOzRdmcP9gAwtezxV2Foj8Pu491N64z06I+J77fsEV0TMYXezXtEErFtkD0ATC97XLRH 3mYeSkhrqPK+wbzXfH/RHnmp1ekp3Rn9sz2yRcx7sAeA68ges9PfN609Yo/Hp24Ma9Xv9xjZI12o Hzdd/3nJKnW9N3LFHgCmlj3O95qDPQB4VqhMC/YA8OOzB6GwBwDIHmAPALIH2AOA7AH2ACB7gD0A XHn2+O5S52APALIH2AMAZA+wB4D/a/YAewCA7AH2ACB7gD0ATC175DLm30S6RckP9gBwHdkjvXe9 Tm9iZw/2ACB7fGXT4hhP++fND7IH2APAFWSPXFHwB6mAPdgDwJVkj2FxwNN9ncoLztq6gaGCdDQv Fx83xRJ1bo2ahLlOYXdLtPVfBfYAML3sMYweq7qtdZ4/qjrKz65zBfPFsbm4aO2xWrcNvT36W1Jb /1VgDwATzB7JDrMSKCJjbO9u8lGSRj5a3d1kT4yo5gN79Lektv6r/KHYA8AUs8fAHu3h4mGTj5IC 8lHTdLof5YgYulre9vbob0lt/Vf5Q7EHgOlkj/IzGrlqn/XJHnGU7VEH67EIqjQwVY3s0d0S9ui+ yh+KPQBMMXsMdme8nj3GGaWcVLIHewC4ruwxIOa3W5HkyYrlbQ4ked5jP7o4SCnhnVixlebP+1uG 8x72DbIHgGlmj7TTPJ70z5vBQqlt83HY1fP29PTUWqZdc5UlkeY9VrnjfniLNVfsAWD62aPdmlGX vR15S0d6ecmnvN6qbtvSxbKGKvZ2RO7IHffDW6rxV4E9AEwxe4A9AMge35E9wB4AZA+wBwDIHmAP ALIH2AOA7AH2ACB7gD0AyB6yB3sAgOwB9gAge4A9AMgeYA8AsgfYA4DsAfYAgJ+ZPUYlPV7pUQp4 vN8X7AFgYtkjvVu9e/k6e7AHANnjK5oWx1DA8+bb7fF9fcEeAH7/7JFLyv6vRmAP9gBwXdmjGtQe jyqD6xxI6nq/K0UC5+ViWy0w39N8dmUEL/RVlpY9AEw3ewyjR1+MPD6qZILFcZ3LmC+OfV3zVbLI 6X7dljA/66uqOXsAmHj2SE/8lnb0aXt3k4+SCPLR6u4mVysfKqdtvNC3/yJ/JvYAMM3sMbBHe7h4 2OSjJIF81DSlpNGTVNL+d6Fv/0X+TOwBYErZo/yMRq7ap316/MdRNkIdrMcqaAJG6CTZ49W+7MEe ACabPfotG29kj3FGCeesV+m+C9lj9EX+TOwBYDrZY0TMcLciydMVy9scSPJcxn50sVDNq338mr3e dztYzQX2ADCp7JF2msez/nkzWCq1bT4Ou7qsqDo9tZZZbLqcEot309zHa32tuWIPABPOHu3mjLrs 18hbOtLLSz7lNVR125Yudquo2gGvWIj1St9H41bsAWDC2QPsAUD2+I7sAfYAIHuAPQBA9gB7AJA9 wB4AZA+wBwDZA+wBQPaQPdgDAGQPsAcA2QPsAUD2AHsAkD3AHgBkD7AHAMgeYA8A/0b2SO9TT29a j9/9K9e7S+2L1eO17eVt7fOuw94/PHsAuMLssThGXY7nzWwb4ujNkE5S0zaUkT2R6w5ui2MOO/Zg DwDXmD1yGdleFUkn+7E9okspUZ4LDX5o+6weP7AHewC4wuxR9aXGW3ucNUVR8q4xHWyXz3F62O13 7MEeAK4vewyiR2ePVV9Btssefb/UtF3mKLJ4+MIe7AHgCrNHGZB6yx4pbfT90tXt8jbql1fzA3uw B4ArzB7v2SOtuVqP+rX2WDxsmuixYQ/2AHA92aP8XB65an4fdo010gRH01R1M+ddr8YeSRvtL7AH gGvLHofd2ax5TJIPmuK8b0xDVo02Zqu7L/frGXuwB4DryR4DVvXLFbuDTBYJ/wAAGiVJREFUgavc FKNWsVJ3VlbsNvY47D6GWtiDPQBcX/bodgd2uwWrej0b2yOPXW3bfYLxmQJLHLIHewC4yuzRvoEk KSS/qWQ+UkvY43SfbBH9cjwJe5z+vmEP9gBwrdkD7AHAs8I7dsEeAGQPsAcA2QPsAUD2AHsAkD3A HgDYA2APALIH2AOA7AH2ACB7gD0AyB5gDwDsIXuwh38HALIH2APAL509ctmoxbHeX2r/JlJlqe+4 DewB4NfNHqt4Lfv+sj2GdWvZgz0AyB69PXL1wPnFb1w8bH7A/9ew1C3YA8A0skfUexpWo2UP9gDA Hl+VPbI9FsdcWbBafq7vbqr5rKrrurlw1p7rDDZXDrthMcLuLLrkqoPp3n2yR3fbsPAt2APAb509 UgXaxXHdPN7Tg75TQGSPS+3HxgyLzWy1bo/zV5Wz3h6r5Iq4L7Wt6tIE9gAwgeyRHut5fCmdViGU Yo9L7dVwKGp00l9O9sj3ZvXMywhWrpUO9gDwW2ePOg9PRcTIvqhieVSxx4X2032JDzFY1a2mKmed PfK9h3bkKp/9oMkUsAeAn549yk/JHvFQP8ba3Xp9Zo+z9u75X6WhqKrYozvr7REdO3vk29iDPQBM IHv09iiB4lL2GLWXpvy72KM/kz3YA8Aks8cle+Q1Uuf2uNBeluCGB7p9gP1Zlbep79vdhi/nPewb ZA8A08keeTHU6emlPS61x+KpxaZdidXaoD9Ldxx2qc82H1lzxR4Appo98gR6mj9/YY8L7TEZEot8 63rfzXv0Z9vmjk+RWfLRfHDbo3Er9gDw+2cPsAcAzwr1PcAeAGQPsAcA2QPsAUD2AHsAkD3AHgDY A2APALIH2AOA7AH2ACB7gD0AyB5gDwDsIXuwB3sAkD3AHgB+0eyh/AZ7AGCP94nKswNjsAd7AGCP d7PHNtXbOPyHPdgDAHt8Pbl27Egn7MEeANjjnewxsMfpPgr/bZdRIzC1pINUErBafq7vbrbNZ9Ph sGsHulJ/VWbZA8BVZo9ZVQSwOEat8tm2VCZvWtalWHlyyDbKkz9+WGflKFLOHgCuN3skNWQDROHx WZkI2c3jv1z3vIra59v0uTiGSJa3+WpuBHsAuLbskUxRNwo43bcpIs97NC5J0aPRxcNmVkVTXMjd Gmfkq3EZ7IF/2Lu3njayNAyjVpdKkzYtuTA+UhiSAOn8/184tb9dJ2xDsPoikForUmI70BeZkR+9 LpsN09ge3a9WfVgs+wyM6lGG1fl65K9XD/UAJro9GtXDut0SJ9sjsT3UA7A9zjxUpbdSzY/qsd/u 3qhHd93D+3vVA5jg9qh/LtOVj3ibVfueq64e+Q1V9eP5enjPlXoAE94e8aGNeENuXOdoqjDUo2lD fALkfD3i6x+8bqUewBS3B+oBeK5wvgfqAdgeqAdge6AegO2BegC2B+oBqAeoB2B7oB6A7YF6ALYH 6gHYHqgHoB62h/9HqAdge6AewJ+3PeIHvY9c3Za7//L9qAfwcbdHHEB78dN2fwz668/+9WF32X9T PdQD+DTbY5OO6Nj/fenT9nBs7etf4tBa9QD+1O1x8UBQD/UAbI9RPeI0wTjLvJinV7N2s/GtfAhh KkaxeC5vfpT5Xv9o/5/YLNpvKeJgwvjq6/22PcAw3x2+Ld1o7zd/FHP/I6oH8Cm2R/Mkvzqpx5fF shkXL24Np5gXkYG8PY7ONo96lHFG+qrdHvmrq3Q3/oP5bvdt+21z9/m6/UM91AP4LNsjNsLqqB7p 1PI4u7y/Fc/v48eiHqNHh3qkO/EXuR7d3+W/Lfq/ja9sX//qXgZTD/UAPsv2SE/m6VWjcT3irU/p yb+/1T3ND4/drvpn/f4CR37latneyvXIb6OKl66am3F3+Lb6EHFp/1AP9QA+9vbofrXqQ3piH133 6EvR3WoDcVyP4dG361GUbZRyPYZva6oS7/nKf6iHegCfZ3s0qof1ST36x9Kt/7Y98tcVp9sjvumh /0M91AP40NvjbD3yE3t/3SOe7Ltb3aWKLgAvr3t0H/J7tR736/azgHH35be1nztMf6iHegCfZHvU P5ftE3t+D1TZvTFqeH9V3Bq95yqe9PM7fc+95+pMPepDumDeX/cYvu3qKS+bp/HUQT2AD7894pMX Zd4CzY2n/MrVpvtsR38rfqBJvMbUXgev4vH+0TfrEV+261+5Gr6t/bO7qx7qAXyS7XHO8CTu6Vw9 ANvjjYfUQz0A9bA9UA/A9kA9gE+wPVAPwPa4fHugHoDtgXoA2B6oB2B7oB6A7YF6ALYH6gHYHraH egDYHqgH8Pu2R/xwdtQDUI+Lni3eUY/2+CbUA7A9LtAeJIt6AJPdHt+/db6rh3oAvHN7fDu5lc8D jMP/4t5wa5Ze19rFkYOlfqgHMOXtEc1YnNSjbM81nxVfmltXt7tRPWwP9QBsj/P1uLlOpWhuFelW 3FcP9QBsj1/UI953lW4Vcevqfq0e6gHYHqN6PGbNrbv4u6N6xGGC6qEegO3xoh7p6WSRfntle0Q9 qgf1UA/A9hjV42tjkX57pR5x3SO9fhWvYcX1c/VQD2Dy2yOF45836lE2v8W7r9Jv+21682592Pm3 VQ/A9vj6Rj3mm+7THenGU3rlqgmJz3uoBzDp7fGrz5rn6x6oB2B7XEI91ANQj1fcqYd6ANgeqAfw e7cH6gGoh1T4f4R6ALYH6gH8xu2hHuoBYHugHoDtgXoAH3N7oB4AtgfqAdgeqAfw6bdHHHaOegC2 x5FNOe9v7dRDPQD1eNf22JRxnOCsPQpKPdQDUI/3bI8vtzka1cMX9VAPQD3euT0WP+Jkwf12FwcJ 1ofucMGr27LcRT3SrXl87XNZPqyblVLG96AewGS3x+LqNsXi6v7fVI+q3I0ONs/nnKcvqA/z9hpJ 8fBl5axz9QAmvz2WcahHMW/WR/dK1ebmOt9K0ci3qpvr9HBqSZ4hxod6AJPeHsur+3UzPdapHnmH pHv5VipHt03WORn1YZX74h9cPYCJbY/uV65Hykb7R2pEW4+4letRhpV6qAdgewzbY1bd/Nsk4e3t 0X2teqgHMNntcVSP/fauicH4usdima+L5+seO/VQD8D2ONke+WPm+5fvudqUaY6kd1nF3fpRPdQD sD1e1KP+ed3WI65yPMTFj01Zlk/5/VZlfkw91AOwPVAPgJnzPVAPwPZAPQDbA/UAbA/UA7A9UA9A PTxboB6A7YF6ALYH6gHYHqgHYHugHoB62B6oB2B7oB6A7YF6AJ96e6Sfw97+SPZO/hnsqAdge7z6 UBzUsXmRj+EoWtQDsD0a3791vo/rkY+hVQ/1ANTj7Pb4dnIr6pGPM0/HCs7b17JW/V3UA5j89ohm LE7qEdsjTY76MO+2R38X9QBsj7P1KBbLdn9Uzd2ox3AX9QBsj5N6tK9TdYvjfp1vDXdRD8D2eMya W3fxd2l7FGlhpMscuSRtPbq7qAcw+e2Rnk4W6bfxK1fxKtXwTqvR9kA9ANujqcfXxiL99uK6R0rF frsb12O4i3oAk98eKRz/nNQjXruq0qtU9WP36Y/+LuoB2B6dl/WoD4tl04v2Z5ZUcbmjOvkRJqgH MM3tcfpZc9QD8Fzxrp9zhXoAXLQ9BEU9AGwP1AOwPVAPwPZAPQDbA/UAJr491EM9AGwP1AOwPVAP 4GNuD9QDwPZAPQDbA/UAbA/UA/gjt0cV58+eOwmqPjhdUD0A2+P8Q1U+W3B++mXOplUPwPbITs/3 iHrMqjPHQKmHegC2RxYnCv7V33pZj/pQxqGCs2LxXN78KPO9q9uynPvHVQ9gytvjW27HX8f12MS5 5rv2NPMicpG3R/q9PsiHegCT3x6zk+2RwrHfRiJSR4ooStQjP5oTg3oA090ex/VIL1Cl163ayxxX 9+tZsVh29WgHyP3aP696AFPeHo9Zc+su/q6fFW0hTuoR7+gtXUBXD2DK2+N/nZfXPWZvbw/UA5j2 9vjaOalHd92jKceoHvvtzj+segCT3x6v12P8nquoR33YzdoH6kf/uuoB2B5n6xHXOOJTg7ke6YL6 Kl9Wf3DRXD2AKW+P08+aox6A54p3/Zwr1APgou0hKOoBYHugHoDtgXoAtgfqAdgeqAcw8e3hqHP1 ALA9UA/A9kA9gI+5PVAPANsD9QBsD9QD+KO2x36bf/RuVsyPHkA9gIlvjziO9qQM6qEegO3xxkOb dJzH/u+3y1DM/YOrB2B7DPKZgr+iHuoB2B535+uRXsJKlSgWz+XNdQrGflu2D80VRD0A22M8K9KB tDkeq6YlKRN9MKpVeninHuoB2B7HDxVxoHmzM1Id0sHnRRx+3seiK4d6qAdge4zst2XTizQ9mqVx v54V8eaqiEW8dNXcVQ/1AGyPu/5Xqz4slvHG3TLtkKEeRZletVIP9QBsj7Oqh3XeHklfj/yQeqgH YHucf6ipx367O6nH/br92KB6qAdge4zVP5dtIap08bx+HNWjfQeWeqgHYHvMjj/vUbaf6WjyUZbp g+fDdY90LWTnlSv1AGwP1ANQD+d7oB6A7YF6ALYH6gHYHqgHYHugHoB6gHoAtgfqAdgeqAdge6Ae gO2BegDqYXuoB4DtgXoAn3Z7nPyAXT9tVz2AyW+PKs6j3amHegC2xwUPVTfX6SCPuXqoB2B7XLA9 Uj3S0bTqoR6A7XHZ9oh67LftKYPF4rm8uR7udvVIRw12D/R3UA9guttj0/xerf7P3r3otm2kYRjm LkEklYJKdiTqVB+S2Gjv/wqX/wxPsmjHWbS1bD2vYZlz4LioAb75ZkaacMI2jjIPLQzFWXdO7ao9 6nxcAHsAuNjsseyXzZMqklCGYv7ebWZt96MC2APARWaP2HKVVz3SXNXiuiib73Ex2yPSRpM5btfH BbAHgIvJHt3XUX4oI4CUvT2GYmePtLe3Wh0XwB4ALjJ7dPbIcaK3x6g4zh5JM+MC2APAhWSPZ+wR 01C7TW+Podite4zWRoYC2APARWePvJ2qX/cYFduV82XMUx1+PCmAPQBcdPZIaxnbYd1jKHbv91i2 6+tHBbAHgEvMHmAPALLHr2cPsAcA2QPsAQCyB9gDgOwB9gAge4A9AMgeYA8AsofswR4AIHuAPQDI HmAPALIH2AOA7AH2ACB7gD0A4J/LHnU+jLY90aM4vS5erAR7APjg2WNZbdmDPQCwx69lj93m96mn f00J7AFA9sh8/9bxva+b336/XbMHewBgj2eDxreJq3pWlGnqqlw8Vl+visNNOiKwXqSjBIt0cGA+ W7B5jUMEd5uqqmb5qMEquqSTBeO2OJt2dGPqMfOHYQ8A7z17JGcsjuxxuFm1Z9GW6Uk/3zcP/nlj jzjAPJ1V3shhGSfORs90gPmq7VW0ax75VPNte7T5cON8v8qHoYM9ALz/7HFsj2WKGxEayuyQ9mlf hzB2m3yC+eFmm3t2rd1F3Z5qnjo2xaZLfKdyrhtOSQd7APg42SNpIL/E9FQWSdGte5Sz0+88dZWa cwRpKiNmRPl2PdyY6+ZTiypgDwDvK3v8yDRXf6S2dvEiRYh47PcP+2N7NAkieSWVY5IqdW736oYp 8m1P7JGGrlb+NOwB4J1nj3icLOKlyx5pninPTGV77CezR6OOZVT0kSJ1zje/nD3AHgDef/a4b1jE S2uPdr0iyaCNE7MpezQv5XYUM1Lq6PQwWvdoavsbd5utPwp7APgQ2SPE8WVkj04AsUcqb8tNm6fm 66f2mO9jciuvoKfZq8V1L5one676G1Px8MNfhj0AfITscT+yR9nuiAoTZHuk5YrYOnVsj2GJI3XY xns/qvH7PaI2CWa4sdFHWwf2APCes8fUe83BHgDY4xWfcwX2AIBfyh6Ewh4AIHuAPQDIHmAPALIH 2AOA7AH2AHDh2YM92AMAZA+wBwDZA+wB4DyzB9gDAGQPsAcA2QPsAUD2AHsA+JjZI50fu7geDjQH ewBgj59ljzrO39j9du0cWfYAwB6vJh1o3mYQ9mAPAOzxuuzR26OOYwJXxW7T/IhjAevFY5wLGOU4 K7CcpRMFi2LoAfYAcLHZoyir1Th7LFdxtQ2bxJm0d59XWTDl58V1buh7gD0AXGz2CH1kf4xmruJM 8joOPJ/vcwy5bs8/r9tT0POp5WAPABebPdJUVDbFqi2luapQRrsPK5xRRrGY366HHmAPABeWPbqv lsPNottzVcbaRjlhjxQ2wh59D7AHgEvOHg3Lu3W2R6uQ5+zR9Bt6gD0AXFj2eN4eeWpqwh5pxaOR xtAD7AHgYrPH4a/rVgZpa9XhJnZaTa17xBasZbUa9QB7ALjY7HG4qbq3byxj81V6V8fkzFVd5c1Z fQ+wB4CLzR6vxRZd9gAge7AHewCA7AH2ACB7gD0AfJTsAfYAIHuAPQBA9gB7APg3s8end42/OHsA eJvswR7sAQCyB9gDgOzBHuwBQPZgD/YAIHuwB3sAkD0miE9t//Rpvq+2U/Ut9ej6b6KcnV498x/n L84eAN4seyzjE9qfKqLhPh7Qh5vtZP1b2+OePdgDwNtmj+XXq4gY00/q+e36xUf9W9nDzBV7AHjr 7BH2+LS8W7MHewBgj1/LHtkecWpgPLHLxWP19ap5eJdVVTUNJ/WfPsWRhE1LvXjct9NeXadM255+ Vg/JM33PbIZ5W8yDDh1HLbPRsM39MeJuU1VhrNTmL84eAN44e9TN63z/0DzEm8dymV0xa7PHVP2+ ebw/NvaIR/kynvp9p5xZ2vZlaCC1Dz2zPT43xdQrDzp0HLXMRsPW0a28+/yQ12LYgz0AnEH2iIf3 bjNri2USSmePqfr4/m9IJ/LFfdNh6NRPOv03t7Rqiu+uHB16Z6Wr3dBx1DIeth6WZ2K6jD3YA8Ab Z48qT0+lf+ZnX5RpNaOzx0T94eZhvO7R1Ayd8rzVQxtBHrrVk75na49U7AcddRy1jIdN9+dxW7Ow B3sA+LezR/c1zguxwlClxYen9jip7xfTR/boOo0X29ufU/aYHdli1HHUMh6WPdgDwHllj8EeD0fB 4Dh7HNX3VfVJzdFQL2SPdBFr9SfZY2gZD8se7AHg7bPHlD3uN9tJe+xO63fd+kU91GyP3sw3G/+M Xk/tkZdQYgUjGkYdRy3jYdmDPQCcZ/bIW54OP57aY6o+bZF6HCeKvlM71Mmeqyf2qLp78qCjPVdD y3hY9mAPAOeZPfICeqyfP7HHRH1akIidUENN12lYRMm7pNrak5mrunt/R37DYd9x1DIelj3YA8BZ ZY+34afvI/cZu+wB4KyzB3uwBwDPCtkD7AFA9mAP9gAgezhbkD0AyB7swR4AZA/2YA8A7PHK7AH2 AIBfzx5gDwCQPcAeAGQPsAcA2QPsAUD2AHsAkD3AHgDY45duqhfX8WO32Tav6STYpiIddl6t/C9l DwCyx3TV2B713bq5+q2xx9ertgT2ACB7vJw9DjfbtjLZYyiCPQDIHs9mjxN7mLpiDwCyx0+zR1F2 Sx3JHmVuAXsAkD1eyh6hj+yPtGoeBgF7AJA9fpY94keSRsoe8711D/YAIHuMqrqvE3vEasei3XNV 1NIHewCQPabJohgtkS/v1m3l0pZd9gAge0wz36+KI1MM9pA92AOA7PEcZXgjbdY9/HUdU1jdzNWy su7BHgBkj+eou08lOdzE1awofFIJewCQPcAeAPD3Zw+wBwD28LQAewCQPcAeAGQPsAcA2QPsAUD2 AHsAYA/ZA+wBQPYAewCQPcAeAGQPsAcA2QPsAYA9wB4A8I9lj3LWXqRPaP+/me99rDt7APiI2eOZ Uzt+ao/h8MEXyMeE3Phod/YA8MGyx27z+6Qdens8nype4YT57fqVPcEeAM7VHt+/dXwfPd+/367Z gz0A4Fl7fCtOr+pZUaapq3rxWFV3692mqiKMlLP5vkqTWskjUZilXm11dwBh15CIwwjjDNt0KGHT WDY/7r6knmW+O41cxC9Jd5XNL21uOBoE7AHgDO2xOLJHrEmkQ8sbHcziTPPPq7xYUX5unvTz/Tbb I+LD4WYWveKU86yNVTFqCHab5uLxKi+l5E599ohfstvcrZuXbbFM98fY1ezpIGAPAOdvj3io51Xt Ohwy33cRocxKaV5TVpi1fVOvVExOGBqKop+iypWp62CPuJr/p/lN826iLKxU9sP1g4A9AJy/PdKE 0jCrNIikTGvp8ahvWtuc0RRSrz6OjBpykEkGaCWSfNHbIyJHPatnxTJGSFNX4agoHA8C9gBwdvb4 kWmu/kibr2LBIdYm1qf2mBVje6Ru1erUHl1DkVNHDNV64NgeaYFlNb+9SpEjprbKwR5Hg4A9AJyZ PeLZsYiXLnukiai80DFpj+XdKHsUXUJ5mj3GlHfryexRLL/++dfV4fbPzaqtKI+zB9gDwLna475h ES/fuqyQl6rjIf7UHskr6Qk/S9NOk/YYGjqaEbp1j1h2H+xxuP0tBvty266GpHchlnkay9vR2QPA udvjy8ge3b/6l92c1MgesRsq7Zsqu4vDj7E9UmAZGtJwDzmtPNlzlXsW5e/bds9W2l9V9usex4OA PQCcoz3uR/Yo231OkRZOZq7qdikizWEtx6sjfdVqaMgyaq+Hi7QEknvWd+v/tW8HGQCAQABFTzD7 7n/SYYyI0bp4bxutMp9Uv9Ot9bVvrs5NUA/gsXpMf81RD4B7PXAi1ANQD9QDUA/UA1AP1ANQD9QD UA/UA0A9UA9APVAPQD1QD0A9UA9APVAPAPVAPQD1QD0A9UA9APVAPQD1QD0A1AP1ANQD9QDUA/UA 1AP1ANQD9QBQD9QDUA/UA1AP1ANQD9QDUA/UA1AP0wL1ANQD9QDUA/UA1AP1ANQD9QDUA9QDUA/U A1AP1ANQD9QDUA/UA1AP1ANAPVAPQD1QD0A9UA9APVAPQD1QDwD1QD0A9UA9APVAPQD1QD0A9UA9 ANQD9QDUA/UA1AP1AL6YFZGo7kBTGbdUswAAAABJRU5ErkJggg== --000000000000a0303206182e0be4--