From owner-freebsd-hackers@freebsd.org Sun Aug 19 10:11:31 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 275FD10889E3; Sun, 19 Aug 2018 10:11:31 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0CB98FB4B; Sun, 19 Aug 2018 10:11:30 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1frKg0-000E7j-0O; Sun, 19 Aug 2018 13:11:16 +0300 From: Daniel Braniss Message-Id: <3C2D99A1-DC1C-4915-81DE-7F9D48AAB10E@cs.huji.ac.il> Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Date: Sun, 19 Aug 2018 13:11:15 +0300 In-Reply-To: <1534523216.27158.17.camel@freebsd.org> Cc: Rajesh Kumar , freebsd-drivers@freebsd.org, freebsd-hackers@freebsd.org To: Ian Lepore References: <1534523216.27158.17.camel@freebsd.org> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 10:11:31 -0000 > On 17 Aug 2018, at 19:26, Ian Lepore wrote: >=20 > On Fri, 2018-08-17 at 11:48 +0530, Rajesh Kumar wrote: >> Hi, >>=20 >> I am trying to use the I2C designware controller driver available in >> FreeBSD (ig4_iic.c) in our boards. >>=20 >> Is there a clean way, I can set the I2C bus frequency from the = controller >> driver itself, rather than using device hints, FDT, tunables etc., >> Something like, if the driver is loaded for our boards (identified = using >> the PCI or ACPI ID's), then the frequency of the I2C bus needs to be >> hardcoded from driver itself. This is to avoid additional configs = from the >> config file. >>=20 >> I tried adding a new interface "iicbus_set_frequency" (in line with >> iicbus_get_frequency) and tried calling that from the ig4 driver = after the >> "iicbus" child is added. But, iicbus instance is created only after = ig4 >> driver is loaded. So, calling iicbus_set_frequency after child = addition >> leads to system panic (as there is no iicbus softc at this point). >>=20 >> Let me know if you need any details. >>=20 >> Thanks, >> Rajesh. >=20 > I don't really understand what you're asking for. The ig4_iic > controller driver doesn't appear to support bus frequency settings at > all, it just loads hard-coded values into the clock high/low registers > and never changes them. If you want to locally modify the driver to = run > at a different hard-coded speed for your application, just change = lines > 589-592 in ig4_iic.c and continue to ignore the speed set by the bus > driver when handling iicbus_reset. >=20 on the arm/allwinner it=E2=80=99s set at 100Khz, and though there are = sysctls to change it, it=E2=80=99s ignored :-) I could change this, but is there some hidden reason that it=E2=80=99s = so? danny From owner-freebsd-hackers@freebsd.org Sun Aug 19 10:29:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F25671089205; Sun, 19 Aug 2018 10:29:16 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C58F7070C; Sun, 19 Aug 2018 10:29:16 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 7f6d23e6; Sun, 19 Aug 2018 12:29:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=lGPm6ALSUJ/y6rVjVG1K2NpNQCY=; b=OFMOgvlMWD5qvQDAAaL1I8qHXfFs nLM0aE3HDlLVeuqZTY8IHtYgDdIr+9GGqBWpPnpGd071Mm2SuSTF/VB6UGSnYRyA a7kd8BW5kqeEqmtmo1Y7H2GyMaWVd4AyL6h3pK4gydVg9hAN2tJXj8Z52X83XVu7 cs83qoxoZLoGQIc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=OeYq0SnFXpeiKxelesQn2NMeJ/ru1d3fpdlMqqNDXCuzTyycY6dgVCvz k8nI7agCkFwjxDM8vp9PaOxQo5aXcdI6Bl6wpaewUCKulAmFS0zzhD/wzteqZV0S qoK5uc5Z7WZ12kP7zYIl4WN3ECdkQjKeL5UpnZasVmRuNb+oZzQ= Received: from skull.home.blih.net (ip-9.net-89-3-105.rev.numericable.fr [89.3.105.9]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 1250cabc TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Sun, 19 Aug 2018 12:29:07 +0200 (CEST) Date: Sun, 19 Aug 2018 12:29:07 +0200 From: Emmanuel Vadot To: Daniel Braniss Cc: Ian Lepore , Rajesh Kumar , freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Message-Id: <20180819122907.8c9306a0a307b9887443e818@bidouilliste.com> In-Reply-To: <3C2D99A1-DC1C-4915-81DE-7F9D48AAB10E@cs.huji.ac.il> References: <1534523216.27158.17.camel@freebsd.org> <3C2D99A1-DC1C-4915-81DE-7F9D48AAB10E@cs.huji.ac.il> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 10:29:17 -0000 On Sun, 19 Aug 2018 13:11:15 +0300 Daniel Braniss wrote: >=20 >=20 > > On 17 Aug 2018, at 19:26, Ian Lepore wrote: > >=20 > > On Fri, 2018-08-17 at 11:48 +0530, Rajesh Kumar wrote: > >> Hi, > >>=20 > >> I am trying to use the I2C designware controller driver available in > >> FreeBSD (ig4_iic.c) in our boards. > >>=20 > >> Is there a clean way, I can set the I2C bus frequency from the control= ler > >> driver itself, rather than using device hints, FDT, tunables etc., > >> Something like, if the driver is loaded for our boards (identified usi= ng > >> the PCI or ACPI ID's), then the frequency of the I2C bus needs to be > >> hardcoded from driver itself. This is to avoid additional configs from= the > >> config file. > >>=20 > >> I tried adding a new interface "iicbus_set_frequency" (in line with > >> iicbus_get_frequency) and tried calling that from the ig4 driver after= the > >> "iicbus" child is added. But, iicbus instance is created only after i= g4 > >> driver is loaded. So, calling iicbus_set_frequency after child addition > >> leads to system panic (as there is no iicbus softc at this point). > >>=20 > >> Let me know if you need any details. > >>=20 > >> Thanks, > >> Rajesh. > >=20 > > I don't really understand what you're asking for. The ig4_iic > > controller driver doesn't appear to support bus frequency settings at > > all, it just loads hard-coded values into the clock high/low registers > > and never changes them. If you want to locally modify the driver to run > > at a different hard-coded speed for your application, just change lines > > 589-592 in ig4_iic.c and continue to ignore the speed set by the bus > > driver when handling iicbus_reset. > >=20 >=20 > on the arm/allwinner it?s set at 100Khz, and though there are sysctls to = change it, > it?s ignored :-) It is set at whatever frequency the dts set it and if there is no frequency in the dts it fallback to 100Khz. twsi needs to support IICBUS_GET_FREQUENCY though, the main reason I never really touched twsi is because the same driver is used on Marvell SoC with hackish support for the clocks. Now that I have a Marvell board with this i2c controller I'll probably rewrite this driver. > I could change this, but is there some hidden reason that it?s so? >=20 > danny >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" --=20 Emmanuel Vadot From owner-freebsd-hackers@freebsd.org Sun Aug 19 13:53:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A98CC1066771; Sun, 19 Aug 2018 13:53:36 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B5D376EEE; Sun, 19 Aug 2018 13:53:36 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id t25-v6so11329249wmi.3; Sun, 19 Aug 2018 06:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Tl40/EUx5fzuQv/rqv+5h+VYXB9PemqtD9b3O0nk/pQ=; b=eDzoFkodc/6D893yCUxhK8ybPFMTtaU02TAAS1QlUjdn5iIGuFOwluHC0cfyN7DyI5 AyBap6QMMLM1Vfm4aURYnDX6QE1mrRNYs5RHHKTA9p7qGT4ydtFZj8oYiBGksOwhIeYw +LITdxOUJtFuv6qIMsKfPybcuE05/jyc/WrG0NOD4U9cn+626rkW7XuGFkzD8ws2uil1 uU0uo9MnkhLRPawZ85FbQomz+OMKTcQRrEGp0MNNnWd832ktpITkPd/kmex2Im5DQtJd qJIjDD9YTJZWIlQLeQanidA6oj0jKt11DsnyRNG21guuTk3ZJaqFpogjFVWl7Fun7bm7 GLAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Tl40/EUx5fzuQv/rqv+5h+VYXB9PemqtD9b3O0nk/pQ=; b=pbukGbnPspQwAOgcr2LHyQvI7SUYkxEeDYot6zHNekQEhUos2U/wNgR86/w2NTABBV psHXtSFaUadFNSSYW3SeSTvAgcA8ug0cZHvrSR4ucR22Yd3xxg3ggV2h15FR8+VPQ6Ud cOMxCQ5MnJP9T+hZdhOYLqFcY2nd9TPNNVcX1AID9wxMHuIjwm7Tczx2vc92HLPqk9Bv TR/yQTeTgeji5aWlZlkPDy+7dS3VfVjWlvrBt+MIq6yzLju8NT8ysrVTZv8t1D6ANhnK UQYFhdxi10WYOsSxtn+mQPKaO1iOXvebYAa8eaz/n/1trV/IfQ08svKX4CAhaomUtWw6 zgSw== X-Gm-Message-State: AOUpUlGt/LUvt5lEi6J1Q5Tt67EcKDMbzArTtDvwJZiZYnFef73U9kgV 8LgfsdfcPSiEYdcO1O7ZhPfCm4VH5oNwkhoOySncgRVq X-Google-Smtp-Source: AA+uWPzYXEqCZzoIki6qiLKXdUSbi58MP8lDO7JWyoS9164ezednUTwuGMiKexfalthuOwvvZjM7BmYGH8ggBjP//mQ= X-Received: by 2002:a1c:20cb:: with SMTP id g194-v6mr24514815wmg.102.1534686814907; Sun, 19 Aug 2018 06:53:34 -0700 (PDT) MIME-Version: 1.0 References: <1534523216.27158.17.camel@freebsd.org> In-Reply-To: <1534523216.27158.17.camel@freebsd.org> From: Rajesh Kumar Date: Sun, 19 Aug 2018 19:23:22 +0530 Message-ID: Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD To: ian@freebsd.org Cc: freebsd-drivers@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 13:53:36 -0000 Hi Ian, Basically, I want to set the I2C clock frequency for Designware IP in our board to 150Mhz. So, I was looking for the way in FreeBSD. So, Is this the frequency which is configured through the clock high/low registers? I see the those register are coded to 100 and 125 currently, I am not sure how that value is arrived. If it needs to be configured for 150Mhz, how to derive the appropriate values? I looked at the DW_apb_i2c databook section 3.11 to understand about it. I am still unclear. I see a comment saying "Program based on 25000 Hz clock". In my case, should they be programmed based on 150Mhz clock? On Fri, Aug 17, 2018 at 9:57 PM Ian Lepore wrote: > On Fri, 2018-08-17 at 11:48 +0530, Rajesh Kumar wrote: > > Hi, > > > > I am trying to use the I2C designware controller driver available in > > FreeBSD (ig4_iic.c) in our boards. > > > > Is there a clean way, I can set the I2C bus frequency from the controller > > driver itself, rather than using device hints, FDT, tunables etc., > > Something like, if the driver is loaded for our boards (identified using > > the PCI or ACPI ID's), then the frequency of the I2C bus needs to be > > hardcoded from driver itself. This is to avoid additional configs from > the > > config file. > > > > I tried adding a new interface "iicbus_set_frequency" (in line with > > iicbus_get_frequency) and tried calling that from the ig4 driver after > the > > "iicbus" child is added. But, iicbus instance is created only after ig4 > > driver is loaded. So, calling iicbus_set_frequency after child addition > > leads to system panic (as there is no iicbus softc at this point). > > > > Let me know if you need any details. > > > > Thanks, > > Rajesh. > > I don't really understand what you're asking for. The ig4_iic > controller driver doesn't appear to support bus frequency settings at > all, it just loads hard-coded values into the clock high/low registers > and never changes them. If you want to locally modify the driver to run > at a different hard-coded speed for your application, just change lines > 589-592 in ig4_iic.c and continue to ignore the speed set by the bus > driver when handling iicbus_reset. > > -- Ian > From owner-freebsd-hackers@freebsd.org Sun Aug 19 13:57:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68E051066EAB; Sun, 19 Aug 2018 13:57:57 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD9EB77329; Sun, 19 Aug 2018 13:57:56 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x42d.google.com with SMTP id o37-v6so1942896wrf.6; Sun, 19 Aug 2018 06:57:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FzlduWxM+dE2pirXFru4a6RmI4DSYPVhkI0/7f4dXgw=; b=Y8hADueHuZJz397yAoUWbIgZB/neK1kt5t/Myj4Z1/Rspv0xAc/kZq/ly9FG4TwxwB BCxxGzEXqj57dr6arAwSt0a8TvOOcfdarkbPN+TTdFQBvSvD3w8j4YHs1BjEFulPI5Z9 AIE7OlYdlaB0J+vXZ1251EN3KJ4l87yaJ15gQYQblVeC/xjmPtH3YCOVDpDV8pXpKXvK tXGuCDnQEUSoRe8pg/SSxPeC7Ju4hz5ALU6gXU3qVeuaZFo4NOjKY7WefC9d2LeX6cEa 9cLZJMUFUrDjkUvCcoXwjyrhEdNg6yIYR6+y5nwQuS2wbPIMIrO6C7g9lSfdg7ORmL0h qbFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FzlduWxM+dE2pirXFru4a6RmI4DSYPVhkI0/7f4dXgw=; b=AX8BzSOHaCQVDPjYx+czZGRmHBzRfZwUI73Iv+k9/YQGR7WhyEmHEF3fAjzqBeFZoy XZtnKrvpt+hiqpshefo+VBD1KkyfPCz1zGP1pkTqIrXcUkmY0KamXqPjgZYtc8lbXF2V VDO25BzajaITSHJ+D8CHicsPlY1X8zSeyOWm9VzNzWYPpPPda9SpqB0s04JcFCNOnbC3 0/6Isc2e5RA8tJwo7FsuUSECDR3+fkb42Grb3pDEGpd9OPLny050eVPpZHdU+BkPrv1j IZP3+bUoIPyJfTHs+lHA6xCTtPQCIxjUS8veoF7MLCnJ3D03QPaRB4uW2k1W5dv102J4 mITQ== X-Gm-Message-State: APzg51AohbOyuKrusUIAD6fJwsOxDJ6eDhxJsa0JaM5uF1aNxbUcxZxs zbZg2PmPcZb+An+hXC66D1upYHXE1K5ZlcwXCSk= X-Google-Smtp-Source: ANB0VdaHjQDC7f8yNsL+g563JQVEVM8agkpvkT5HY9NuPEy76ipowdvaywibeIzvXLTQC8LQ1dMhzUQOLq+UcDzKuRA= X-Received: by 2002:adf:ad34:: with SMTP id p49-v6mr6219503wrc.10.1534687075891; Sun, 19 Aug 2018 06:57:55 -0700 (PDT) MIME-Version: 1.0 References: <1534523216.27158.17.camel@freebsd.org> <3C2D99A1-DC1C-4915-81DE-7F9D48AAB10E@cs.huji.ac.il> <20180819122907.8c9306a0a307b9887443e818@bidouilliste.com> In-Reply-To: <20180819122907.8c9306a0a307b9887443e818@bidouilliste.com> From: Rajesh Kumar Date: Sun, 19 Aug 2018 19:27:44 +0530 Message-ID: Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD To: manu@bidouilliste.com Cc: danny@cs.huji.ac.il, ian@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 13:57:57 -0000 Hi Daniel/Emmanuel, Basically, I want to set the I2C clock frequency for Designware IP in our board to 150Mhz. So, I was looking for the way in FreeBSD. Is this 100Khz in arm/allwinner is same as what I am trying to configure? Or Is it something different? On Sun, Aug 19, 2018 at 3:59 PM Emmanuel Vadot wrote: > On Sun, 19 Aug 2018 13:11:15 +0300 > Daniel Braniss wrote: > > > > > > > > On 17 Aug 2018, at 19:26, Ian Lepore wrote: > > > > > > On Fri, 2018-08-17 at 11:48 +0530, Rajesh Kumar wrote: > > >> Hi, > > >> > > >> I am trying to use the I2C designware controller driver available in > > >> FreeBSD (ig4_iic.c) in our boards. > > >> > > >> Is there a clean way, I can set the I2C bus frequency from the > controller > > >> driver itself, rather than using device hints, FDT, tunables etc., > > >> Something like, if the driver is loaded for our boards (identified > using > > >> the PCI or ACPI ID's), then the frequency of the I2C bus needs to be > > >> hardcoded from driver itself. This is to avoid additional configs > from the > > >> config file. > > >> > > >> I tried adding a new interface "iicbus_set_frequency" (in line with > > >> iicbus_get_frequency) and tried calling that from the ig4 driver > after the > > >> "iicbus" child is added. But, iicbus instance is created only after > ig4 > > >> driver is loaded. So, calling iicbus_set_frequency after child > addition > > >> leads to system panic (as there is no iicbus softc at this point). > > >> > > >> Let me know if you need any details. > > >> > > >> Thanks, > > >> Rajesh. > > > > > > I don't really understand what you're asking for. The ig4_iic > > > controller driver doesn't appear to support bus frequency settings at > > > all, it just loads hard-coded values into the clock high/low registers > > > and never changes them. If you want to locally modify the driver to run > > > at a different hard-coded speed for your application, just change lines > > > 589-592 in ig4_iic.c and continue to ignore the speed set by the bus > > > driver when handling iicbus_reset. > > > > > > > on the arm/allwinner it?s set at 100Khz, and though there are sysctls to > change it, > > it?s ignored :-) > > It is set at whatever frequency the dts set it and if there is no > frequency in the dts it fallback to 100Khz. > twsi needs to support IICBUS_GET_FREQUENCY though, the main reason I > never really touched twsi is because the same driver is used on Marvell > SoC with hackish support for the clocks. Now that I have a Marvell > board with this i2c controller I'll probably rewrite this driver. > > > I could change this, but is there some hidden reason that it?s so? > > > > danny > > > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > > > -- > Emmanuel Vadot > From owner-freebsd-hackers@freebsd.org Sun Aug 19 18:21:11 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6ED4A10734E2 for ; Sun, 19 Aug 2018 18:21:11 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2r.ore.mailhop.org (outbound2r.ore.mailhop.org [54.200.129.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 01891839E9 for ; Sun, 19 Aug 2018 18:21:10 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: a0c63aca-a3dc-11e8-904b-1d2e466b3c59 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.ore.mailhop.org (Halon) with ESMTPSA id a0c63aca-a3dc-11e8-904b-1d2e466b3c59; Sun, 19 Aug 2018 18:21:02 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7JIL1Od086042; Sun, 19 Aug 2018 12:21:01 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1534702861.27158.36.camel@freebsd.org> Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD From: Ian Lepore To: Rajesh Kumar Cc: freebsd-drivers@freebsd.org, freebsd-hackers@freebsd.org Date: Sun, 19 Aug 2018 12:21:01 -0600 In-Reply-To: References: <1534523216.27158.17.camel@freebsd.org> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 18:21:11 -0000 On Sun, 2018-08-19 at 19:23 +0530, Rajesh Kumar wrote: > Hi Ian, > > Basically, I want to set the I2C clock frequency for Designware IP in our > board to 150Mhz.  So, I was looking for the way in FreeBSD. > > So, Is this the frequency which is configured through the clock high/low > registers? I see the those register are coded to 100 and 125 currently, I > am not sure how that value is arrived. If it needs to be configured for > 150Mhz, how to derive the appropriate values? I looked at the DW_apb_i2c > databook section 3.11 to understand about it.  I am still unclear.  I see a > comment saying "Program based on 25000 Hz clock". In my case, should they > be programmed based on 150Mhz clock? Rajesh, Please bottom-post when replying on freebsd mailing lists, mixed top- and bottom-posting is too confusing. What exactly do you mean when you say "the i2c clock frequency"? The datasheet appears to use a term like that to refer to the internal clock used to drive the IP block in the chip. That base clock is then divided down to create the i2c bus frequency on the I2C_SCL line. The IG4_REG_SS_SCL_HCNT and IG4_REG_SS_SCL_LCNT registers are the duration in base clock ticks that the SCL line is held high and low for standard speed. The registers with FS in the name are for high speed mode. The comment block and the values our driver programs into those registers appear to be wildly wrong. There is no way a base clock running at 25KHz can be divided down to create i2c bus speeds of 100KHz and 400KHz for standard and fast modes. If the base clock really is 25KHz then the driver currently sets the i2c bus to run at 111Hz. The hardware default values for the HCNT/LCNT registers, as given in the datasheet referenced by the driver [1], would be consistant with an internal base clock speed of 1GHz. The fact that the header file defines a IG4_REG_CLK_PARMS register, but the datasheet doesn't mention it, makes me think that on some versions of the hardware the speed is fixed and the driver has to know what that is based on the version, or vendor, or something. Other versions of the hardware may have information about the base clock speed in that IG4_REG_CLK_PARMS register. What we need is for someone who has this hardware to put an oscilliscope on the SCL line and get us some real-world truth. [1] http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family-mobile-i-o-datasheet.html?wapkw=datasheets+4th+generation -- Ian From owner-freebsd-hackers@freebsd.org Mon Aug 20 06:49:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F3961086445; Mon, 20 Aug 2018 06:49:34 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 11A347E40D; Mon, 20 Aug 2018 06:49:32 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1fre0D-000FYp-EW; Mon, 20 Aug 2018 09:49:25 +0300 From: Daniel Braniss Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Date: Mon, 20 Aug 2018 09:49:25 +0300 In-Reply-To: <1534702861.27158.36.camel@freebsd.org> Cc: Rajesh Kumar , freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org To: Ian Lepore References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 06:49:34 -0000 > On 19 Aug 2018, at 21:21, Ian Lepore wrote: >=20 > On Sun, 2018-08-19 at 19:23 +0530, Rajesh Kumar wrote: >> Hi Ian, >>=20 >> Basically, I want to set the I2C clock frequency for Designware IP in = our >> board to 150Mhz. So, I was looking for the way in FreeBSD. >>=20 >> So, Is this the frequency which is configured through the clock = high/low >> registers? I see the those register are coded to 100 and 125 = currently, I >> am not sure how that value is arrived. If it needs to be configured = for >> 150Mhz, how to derive the appropriate values? I looked at the = DW_apb_i2c >> databook section 3.11 to understand about it. I am still unclear. I = see a >> comment saying "Program based on 25000 Hz clock". In my case, should = they >> be programmed based on 150Mhz clock? >=20 > Rajesh, >=20 > Please bottom-post when replying on freebsd mailing lists, mixed top- > and bottom-posting is too confusing. >=20 > What exactly do you mean when you say "the i2c clock frequency"? >=20 > The datasheet appears to use a term like that to refer to the internal > clock used to drive the IP block in the chip. That base clock is then > divided down to create the i2c bus frequency on the I2C_SCL line. >=20 > The IG4_REG_SS_SCL_HCNT and IG4_REG_SS_SCL_LCNT registers are the > duration in base clock ticks that the SCL line is held high and low = for > standard speed. The registers with FS in the name are for high speed > mode. >=20 > The comment block and the values our driver programs into those > registers appear to be wildly wrong. There is no way a base clock > running at 25KHz can be divided down to create i2c bus speeds of = 100KHz > and 400KHz for standard and fast modes. If the base clock really is > 25KHz then the driver currently sets the i2c bus to run at 111Hz. >=20 > The hardware default values for the HCNT/LCNT registers, as given in > the datasheet referenced by the driver [1], would be consistant with = an > internal base clock speed of 1GHz. The fact that the header file > defines a IG4_REG_CLK_PARMS register, but the datasheet doesn't = mention > it, makes me think that on some versions of the hardware the speed is > fixed and the driver has to know what that is based on the version, or > vendor, or something. Other versions of the hardware may have > information about the base clock speed in that IG4_REG_CLK_PARMS > register. >=20 > What we need is for someone who has this hardware to put an > oscilliscope on the SCL line and get us some real-world truth. >=20 > [1] = http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family= -mobile-i-o-datasheet.html?wapkw=3Ddatasheets+4th+generation = >=20 > -- Ian hi, I have similar issues with the allwinner/twsi but I do have a Saleae = Logic and here is a nice picture: danny From owner-freebsd-hackers@freebsd.org Mon Aug 20 08:13:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41DA6108853E; Mon, 20 Aug 2018 08:13:17 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B30E68119A; Mon, 20 Aug 2018 08:13:16 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1frfJE-000OzS-LO; Mon, 20 Aug 2018 11:13:08 +0300 From: Daniel Braniss Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Date: Mon, 20 Aug 2018 11:13:08 +0300 In-Reply-To: Cc: Rajesh Kumar , freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org To: Ian Lepore References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 08:13:17 -0000 > On 20 Aug 2018, at 09:49, Daniel Braniss wrote: >=20 >=20 >=20 >> On 19 Aug 2018, at 21:21, Ian Lepore > wrote: >>=20 >> On Sun, 2018-08-19 at 19:23 +0530, Rajesh Kumar wrote: >>> Hi Ian, >>>=20 >>> Basically, I want to set the I2C clock frequency for Designware IP = in our >>> board to 150Mhz. So, I was looking for the way in FreeBSD. >>>=20 >>> So, Is this the frequency which is configured through the clock = high/low >>> registers? I see the those register are coded to 100 and 125 = currently, I >>> am not sure how that value is arrived. If it needs to be configured = for >>> 150Mhz, how to derive the appropriate values? I looked at the = DW_apb_i2c >>> databook section 3.11 to understand about it. I am still unclear. = I see a >>> comment saying "Program based on 25000 Hz clock". In my case, should = they >>> be programmed based on 150Mhz clock? >>=20 >> Rajesh, >>=20 >> Please bottom-post when replying on freebsd mailing lists, mixed top- >> and bottom-posting is too confusing. >>=20 >> What exactly do you mean when you say "the i2c clock frequency"? >>=20 >> The datasheet appears to use a term like that to refer to the = internal >> clock used to drive the IP block in the chip. That base clock is then >> divided down to create the i2c bus frequency on the I2C_SCL line. >>=20 >> The IG4_REG_SS_SCL_HCNT and IG4_REG_SS_SCL_LCNT registers are the >> duration in base clock ticks that the SCL line is held high and low = for >> standard speed. The registers with FS in the name are for high speed >> mode. >>=20 >> The comment block and the values our driver programs into those >> registers appear to be wildly wrong. There is no way a base clock >> running at 25KHz can be divided down to create i2c bus speeds of = 100KHz >> and 400KHz for standard and fast modes. If the base clock really is >> 25KHz then the driver currently sets the i2c bus to run at 111Hz. >>=20 >> The hardware default values for the HCNT/LCNT registers, as given in >> the datasheet referenced by the driver [1], would be consistant with = an >> internal base clock speed of 1GHz. The fact that the header file >> defines a IG4_REG_CLK_PARMS register, but the datasheet doesn't = mention >> it, makes me think that on some versions of the hardware the speed is >> fixed and the driver has to know what that is based on the version, = or >> vendor, or something. Other versions of the hardware may have >> information about the base clock speed in that IG4_REG_CLK_PARMS >> register. >>=20 >> What we need is for someone who has this hardware to put an >> oscilliscope on the SCL line and get us some real-world truth. >>=20 >> [1] = http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family= -mobile-i-o-datasheet.html?wapkw=3Ddatasheets+4th+generation = > >>=20 >> -- Ian >=20 >=20 > hi, > I have similar issues with the allwinner/twsi but I do have a Saleae = Logic and here is a nice picture: ah, maybe this is better: = https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.pn= g >=20 >=20 >=20 > danny >=20 > _______________________________________________ > freebsd-hackers@freebsd.org = mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers = > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org = " From owner-freebsd-hackers@freebsd.org Mon Aug 20 09:36:40 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5562E108A238; Mon, 20 Aug 2018 09:36:40 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A95D283BD9; Mon, 20 Aug 2018 09:36:39 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x432.google.com with SMTP id w11-v6so9827202wrc.5; Mon, 20 Aug 2018 02:36:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zJ9dgvfPjlgToL3w7dMe1B+r0GyzSNSc6AasKP49K9M=; b=tSCdRKYr4FujycVqYGmM8Xq7xDquGKwnxfMP2z58c9UBIpTSQg159h2w2s8Zg8D35h kUY2c+kfOra4KA+WqUi773BGZEXF6K1Fgv7dlzv1yRCJvt+hrcAzimt88xCZuXoYSchh AsdrC/8HWWXblLQfeY3SwYfZ1wNWP2iDNSU2ot/6U+yo/wTgSWhQh3RyQxd80FeVJGFA qZAE4u2p+bYbOsZqhthgIIfvUC2AUs4cQ+LXVsLcYGWmxDVdiWWifBTYFaouem9Fjiky jhtfV2rTuyOqZ1zc/P7pvljaY146raaJshOXyBV2yt+bXT0+L2J0g9sg5/3/0ophAXGj 3Q9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zJ9dgvfPjlgToL3w7dMe1B+r0GyzSNSc6AasKP49K9M=; b=RTd3iq8+SuKmu1yrvDq79pwW3IZIbicmw/CbJMjWdVWcvAteT1e8RXlzAzkJl+eG9V 7xlSGFW6nGtHISkmuJnS5oLKq2tznx4IuN0KtcJXKRWU3yZExQ9JzpQvw+dyc2zgtLjM nSYuory5UbXgf2TZSoTqkBSFrpsqbyOjJIG0AJyKtJTjE+rZb9KXTahYYB1wA0e/rTg1 5FGv86eXUg+PI4Nt7BUsyVFiopXiBZi+0NRLhvEj1QdSpJWjVNZHaijP9GZ3Qt7oHlfI pAZ74jspnqFHN/JL8vqBoeLTwZj2cDhC196+LwZHU/gxJS1q1qc+Z+tTVZzc/3rlQtVx 2bqw== X-Gm-Message-State: AOUpUlGEV3u2eNS6dcdNp8doPYBd5ud4LE9zrg7nxKEPrfiwkxd+WKd8 36zfgzHzuCY54U6+j/ZmQ0ISzsutDaZb6uxNZwLYU7dlttk= X-Google-Smtp-Source: AA+uWPzvzByWn93L0gHXE5ADN1KhxIsv9oxLAYDo1iuau/cBrDWDN0jk6t5tw4fP2ug3BJOC2BLG1o/OmZ3z1Eu1alU= X-Received: by 2002:adf:cc83:: with SMTP id p3-v6mr10754968wrj.226.1534757798586; Mon, 20 Aug 2018 02:36:38 -0700 (PDT) MIME-Version: 1.0 References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> In-Reply-To: From: Rajesh Kumar Date: Mon, 20 Aug 2018 15:06:26 +0530 Message-ID: Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD To: danny@cs.huji.ac.il Cc: ian@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 09:36:40 -0000 Hi Ian/Daniel, Sorry about the mixed-posting. By "i2c clock frequency", I mean the internal base frequency only, which drives the chip. I thought data will be transferred on bus based on the base frequency. So, thought both bus and base frequency are same. But from what you said, seems both are different. So, based on the setting in *_HCNT/LCNT register, the bus frequency (which is the rate at which data is transferred) will change for a particular base frequency. Is that right? So, few questions here 1) As you said, we need to have a base frequency of 150 Mhz in our case. So, do we need to program that IG4_REG_CLK_PARMS to 150 Mhz (0x8F0D180)? And can this be done at the same time when programming the HCNT/LNCT registers? 2) Not sure how that 111Hz value is arrived. Can you please explain this calculation. So, that I can derive the appropriate values for HCNT/LCNT for different speeds at 150Mhz base clock. 3) "Default HCNT/LCNT register values would be consistent with an internal base clock speed of 1GHz", Does it mean with those values, all speeds can be achieved until 1GHz clock? 4) I am quite unfamiliar with the oscilloscope outputs. So, it would be good if you give some idea about what is shown in that pic? On Mon, Aug 20, 2018 at 1:43 PM Daniel Braniss wrote: > > > On 20 Aug 2018, at 09:49, Daniel Braniss wrote: > > > > On 19 Aug 2018, at 21:21, Ian Lepore wrote: > > On Sun, 2018-08-19 at 19:23 +0530, Rajesh Kumar wrote: > > Hi Ian, > > Basically, I want to set the I2C clock frequency for Designware IP in our > board to 150Mhz. So, I was looking for the way in FreeBSD. > > So, Is this the frequency which is configured through the clock high/low > registers? I see the those register are coded to 100 and 125 currently, I > am not sure how that value is arrived. If it needs to be configured for > 150Mhz, how to derive the appropriate values? I looked at the DW_apb_i2c > databook section 3.11 to understand about it. I am still unclear. I see a > comment saying "Program based on 25000 Hz clock". In my case, should they > be programmed based on 150Mhz clock? > > > Rajesh, > > Please bottom-post when replying on freebsd mailing lists, mixed top- > and bottom-posting is too confusing. > > What exactly do you mean when you say "the i2c clock frequency"? > > The datasheet appears to use a term like that to refer to the internal > clock used to drive the IP block in the chip. That base clock is then > divided down to create the i2c bus frequency on the I2C_SCL line. > > The IG4_REG_SS_SCL_HCNT and IG4_REG_SS_SCL_LCNT registers are the > duration in base clock ticks that the SCL line is held high and low for > standard speed. The registers with FS in the name are for high speed > mode. > > The comment block and the values our driver programs into those > registers appear to be wildly wrong. There is no way a base clock > running at 25KHz can be divided down to create i2c bus speeds of 100KHz > and 400KHz for standard and fast modes. If the base clock really is > 25KHz then the driver currently sets the i2c bus to run at 111Hz. > > The hardware default values for the HCNT/LCNT registers, as given in > the datasheet referenced by the driver [1], would be consistant with an > internal base clock speed of 1GHz. The fact that the header file > defines a IG4_REG_CLK_PARMS register, but the datasheet doesn't mention > it, makes me think that on some versions of the hardware the speed is > fixed and the driver has to know what that is based on the version, or > vendor, or something. Other versions of the hardware may have > information about the base clock speed in that IG4_REG_CLK_PARMS > register. > > What we need is for someone who has this hardware to put an > oscilliscope on the SCL line and get us some real-world truth. > > [1] > http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family-mobile-i-o-datasheet.html?wapkw=datasheets+4th+generation > < > http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family-mobile-i-o-datasheet.html?wapkw=datasheets+4th+generation > > > > -- Ian > > > > hi, > I have similar issues with the allwinner/twsi but I do have a Saleae Logic > and here is a nice picture: > > > ah, maybe this is better: > https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.png > > > > > danny > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > From owner-freebsd-hackers@freebsd.org Mon Aug 20 11:27:46 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 797B9108D234 for ; Mon, 20 Aug 2018 11:27:46 +0000 (UTC) (envelope-from aliovx@gmail.com) Received: from mail-oi0-x244.google.com (mail-oi0-x244.google.com [IPv6:2607:f8b0:4003:c06::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 082AE88B47 for ; Mon, 20 Aug 2018 11:27:46 +0000 (UTC) (envelope-from aliovx@gmail.com) Received: by mail-oi0-x244.google.com with SMTP id k12-v6so25130065oiw.8 for ; Mon, 20 Aug 2018 04:27:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=uLlkO/2DWLjmaOPxjF2UqH1zCZepBjVWp1uH8mnj2aI=; b=mPObUrUboRDVnqN5kl6W5/DNAhuzedGxnSDDxOrwwb3gK/ROQ8Tiu62pK8mUu4m/GI NvWdz0nr+k5mHePVvvLd1JHvwTQwjPmnnUy6FOIREv+DW0QqZG1GmGWkXk/FQGLXdahE TMp/bPIKzb/au9u2aERS8vCZqp4uz5A2T0By4V5KK1oWWIh0RNDQaethba9/0KtUuuzN C2dRNA9FhwffT3O10EB608sv1u4jYTsQyZLQXqXMW022jt3IRRttrKFD/qusF9yZBVgQ Eqbf2zFGfuOb25wAA8qyemK2fvS9+gO186TjZWv4IsPxjVFqGlcHW9gdQwma8gJ7DH5Q PzuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uLlkO/2DWLjmaOPxjF2UqH1zCZepBjVWp1uH8mnj2aI=; b=tkk1XbxHC4km/HLXq5XNM4o4X3mxY3uvfKbP0ZwLkmqLAm+Cd04mgr75EzMlkxzrhb m+q2HxSeZYI31w/vC8QgFGTCFsRTNbNuAZJkXEV4+hQSqgSqXbN2+RpOvJuXa3iIsFLN 2Tp98iHH0UPd7g62C3sXZvTEJ3Ow54Rli0uZAG8QV5+Ov1EO+Q2Yps+Kvr0hXOyDkBNv xRJJpHlNBDU52LcNt/wcEVCNFYVw9pI4+gXpGg2trKnltfYpyT/WRx8ahboQKqLEYI6g awx08asQCKeiO93rlcfKv2JYWxZiQXG2bqmjIHsuquJXCBYNZDGHiVm6WXTCGX5lVQ7k pKeg== X-Gm-Message-State: AOUpUlEaoPU4Gc78XW+V5WHYhVKjHtLQMDe1WihCeyBcXuUZ8HhoJzgk D3A4M2kbBtrbfSh9KVbQbTucGdbplVJJyIJRJyjmksch X-Google-Smtp-Source: AA+uWPyKuKZ3PlpGQoikIfAqLw1iSTAvSlSQCDJC8yStDKciBH/OczE+k2bI//M+4Ss9VFYXTmCtSGiNEVVUnMmN4AY= X-Received: by 2002:aca:5a45:: with SMTP id o66-v6mr13802807oib.155.1534764465074; Mon, 20 Aug 2018 04:27:45 -0700 (PDT) MIME-Version: 1.0 From: Ali Abdallah Date: Mon, 20 Aug 2018 13:27:33 +0200 Message-ID: Subject: rand_harvestq high cpu usage when /dev/urandom is used To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 11:27:46 -0000 Hello, I was just sorting randomly some jpg image files using: ls *.jpg | sort -R --random-source=/dev/urandom The above command never exited. Later I noticed that one of my CPU is always running 100%. top -S reveals that it is rand_harvestq kernel service. Is this is a bug? This occurs on 12-ALPHA1 and 11.2 Also, I read on https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html someone saying that sysctls to turn off harvesting is documented in random(6) . I had a look at that document, but wasn't clear to me how to turn it off. I tried to play with the mask, but nothing. Cheers, Ali From owner-freebsd-hackers@freebsd.org Mon Aug 20 11:54:10 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DAD50108DC47 for ; Mon, 20 Aug 2018 11:54:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-11.consmr.mail.bf2.yahoo.com (sonic304-11.consmr.mail.bf2.yahoo.com [74.6.128.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8375D898F0 for ; Mon, 20 Aug 2018 11:54:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: pTygc6QVM1kZ4YekDYjjV_fAAIlbo0kMccMDN_z8A5hP9Ibbba2QvjtjdwZUm_c WUYSddcEVo0twiigxuEXhTpwvV9bkE5vWPFHqUx6KE6kNchlyyyIsX0EgWaAmMNr9IqIRDaH3BCm _p9IRo6QMfaYj5Kr_T49oFw7SQjQx4r.PesDX5CnWIznun.NjiYYBj7uEbYJPxWZZf7rtFdZkOEe RNRlaBlCEx0GRImtiwnuYL2Ch0zO7NBVZDf16Ledp.owgRxw52ip1A6ajk07cvJXh2rq7lehyBAe T81aI.UfEv1PphcubU1e5uj6CReyzIYq2HRXrRwkI0J_yBbh6t4o8x27Mynfwe8LMdVaScUvk31_ kBEU9M23znRfpv7y2Y4vPj1d8YuG9_RatinGemE0AsalZt6F7FmbjzTLag97RCCZsgWPBeTlKwdV uxkFAhJa4E5ACa7Vanu2XRcgM1yPBBBYiLFt6KUUZTRtxfXRJHUnBDZtab35hybpaohCiJAAqKxw 4.FZPlZ.4AnFsC93M29ZLNiFXkm00ugV855Y3h2un_ww_N6C0QH1u7zgCrRh_3HXl4ynDinytf73 2Dotnouj0XKDK2Sx4Se1kUFl63h0lF7O.1pS.MI2pJjBLGs05d7WUj3acx1wj0mtX2zSffrPyW38 lDcvv_QialTRzHVjjOnB4Hf7x5sJtntit0_UoKQqdOAUjuCK6y7vYodPvOILTmQUDLuo437FEzgJ DZsa9WRwnDlZwos6RY1RcipweUPgtPPJujc8shwHPFkQyyk.9n9vVS0oem3h7WiqndltiJv.H6Rb Goxzrg7q_1kHtzAjFaMBbIklv7YZPPVdkL1edTD2Wg5aRZSQtQxpjWExcpGzDBiMOC2j4x2Mc_0l ThNbn.J_gstD8LBjs9GA0zomTOboNihVViImnvN62v1aFoM1asvYC_mUX61UA6nDNcF5ghUY9z2S n0zvOBaRHsfv9Hb7nBNFil9rNgDEYaDl4LGwHAZCKS_IVma6Kgb.nWKt0GvwZwfrY_Td4b6DTUI0 cYfBeEszJ0lL. Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.bf2.yahoo.com with HTTP; Mon, 20 Aug 2018 11:54:09 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp422.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 760fb20e0f8d8f3c216a13641a615789; Mon, 20 Aug 2018 11:54:07 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used From: Mark Millard In-Reply-To: Date: Mon, 20 Aug 2018 04:54:05 -0700 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <95C34771-8D0B-489A-88DA-09AF0D244A1C@yahoo.com> References: To: Ali Abdallah X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 11:54:11 -0000 On 2018-Aug-20, at 4:27 AM, Ali Abdallah wrote: > . . . > Also, I read on > = https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.h= tml > someone saying that sysctls to turn off harvesting is documented in > random(6)=20 It says random(4), so: = . > . . . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Mon Aug 20 13:18:20 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6686D106D84D for ; Mon, 20 Aug 2018 13:18:20 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2r.ore.mailhop.org (outbound2r.ore.mailhop.org [54.200.129.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E159B8C9D7 for ; Mon, 20 Aug 2018 13:18:19 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 7f1764d8-a47b-11e8-904b-1d2e466b3c59 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.ore.mailhop.org (Halon) with ESMTPSA id 7f1764d8-a47b-11e8-904b-1d2e466b3c59; Mon, 20 Aug 2018 13:18:17 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7KDIFgW087893; Mon, 20 Aug 2018 07:18:15 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1534771095.27158.46.camel@freebsd.org> Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD From: Ian Lepore To: Daniel Braniss Cc: Rajesh Kumar , freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Date: Mon, 20 Aug 2018 07:18:15 -0600 In-Reply-To: References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 13:18:20 -0000 On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: > > > > > On 20 Aug 2018, at 09:49, Daniel Braniss wrote: > > > > > > > > > > > > On 19 Aug 2018, at 21:21, Ian Lepore wrote: > > > > > > On Sun, 2018-08-19 at 19:23 +0530, Rajesh Kumar wrote: > > > > > > > > Hi Ian, > > > > > > > > Basically, I want to set the I2C clock frequency for Designware IP in our > > > > board to 150Mhz.  So, I was looking for the way in FreeBSD. > > > > > > > > So, Is this the frequency which is configured through the clock high/low > > > > registers? I see the those register are coded to 100 and 125 currently, I > > > > am not sure how that value is arrived. If it needs to be configured for > > > > 150Mhz, how to derive the appropriate values? I looked at the DW_apb_i2c > > > > databook section 3.11 to understand about it.  I am still unclear.  I see a > > > > comment saying "Program based on 25000 Hz clock". In my case, should they > > > > be programmed based on 150Mhz clock? > > > Rajesh, > > > > > > Please bottom-post when replying on freebsd mailing lists, mixed top- > > > and bottom-posting is too confusing. > > > > > > What exactly do you mean when you say "the i2c clock frequency"? > > > > > > The datasheet appears to use a term like that to refer to the internal > > > clock used to drive the IP block in the chip. That base clock is then > > > divided down to create the i2c bus frequency on the I2C_SCL line. > > > > > > The IG4_REG_SS_SCL_HCNT and IG4_REG_SS_SCL_LCNT registers are the > > > duration in base clock ticks that the SCL line is held high and low for > > > standard speed. The registers with FS in the name are for high speed > > > mode. > > > > > > The comment block and the values our driver programs into those > > > registers appear to be wildly wrong. There is no way a base clock > > > running at 25KHz can be divided down to create i2c bus speeds of 100KHz > > > and 400KHz for standard and fast modes. If the base clock really is > > > 25KHz then the driver currently sets the i2c bus to run at 111Hz. > > > > > > The hardware default values for the HCNT/LCNT registers, as given in > > > the datasheet referenced by the driver [1], would be consistant with an > > > internal base clock speed of 1GHz. The fact that the header file > > > defines a IG4_REG_CLK_PARMS register, but the datasheet doesn't mention > > > it, makes me think that on some versions of the hardware the speed is > > > fixed and the driver has to know what that is based on the version, or > > > vendor, or something. Other versions of the hardware may have > > > information about the base clock speed in that IG4_REG_CLK_PARMS > > > register. > > > > > > What we need is for someone who has this hardware to put an > > > oscilliscope on the SCL line and get us some real-world truth. > > > > > > [1] http://www.intel.com/content/www/us/en/processors/core/4th-gen-core-family-mobile-i-o-datasheet.html?wapkw=datasheets+4th+generation > > > > > > > -- Ian > > > > hi, > > I have similar issues with the allwinner/twsi but I do have a Saleae Logic and here is a nice picture: > ah, maybe this is better: > https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.png > > > > > > > > > > danny This has nothing to do with the twsi driver, this is about the ig4 driver (found in sys/dev/ichiic). That screenshot seems to show a bus running at 100KHz like it should (although the 62:38 duty cycle is a bit suspicious). -- Ian From owner-freebsd-hackers@freebsd.org Mon Aug 20 14:16:29 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CDFC106F02A for ; Mon, 20 Aug 2018 14:16:29 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 215408EB90 for ; Mon, 20 Aug 2018 14:16:28 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: VGq812IVM1lkSSyCICS7JNpmc08qdmmUfRq9G_L.SPS2B9B.jtpgvnkryc5ddAp JvYr3TraqG1AP3Q8V0JxHlcltE1JQPQubTmqx5gREabq3vEiG_TKLWLvvS2cOpQkkkNAbvzXOxSg zIX4gzbJnCwauF5z8dFaDMm_BacpoQsFOrPb4VTeflr8tjgD3TrPGJf1_XHEUFu71t0d.V_HhSCv 9H7M55XBSyfoReBDvA2wTcAVNC.5xwnjOYxfcfp7m1iWAjv_nt63aAd.NoiVqRMxbxKDWZz.mmRW cVayd3nA4PQ7blrOU.3RBpydF8H_jYVCXPvSFtho_mGDc0TYCj6nuxf8kluYYvW_ZyKCI1vbE5D1 lWKAxvBUHIzf1n5phUAkVGNMMndCR7rcGXVeN8rex2YGZu7zjhZUU9uoN4Ox9uM3MfHNo9vYgMxP icNEqi4PDCDPnCMLY6DYMKMAR3AmLwwIiTUr.Fd0CbQCtMUxUCKiW1uOsYrTL7tt2QbndGTsR7Ku hQBX1LrcT5Do95TD_0LVG_RXxi4KQ2atypjUbidNGY_VUu15IVxPfFAGMfHkQ1xK4jYyoVLab_f9 5jWBPqyoScfQzPkGcq4dyuqpD.xgzgyZeOmNxTMJZVohZ3pDAMS0jgqafiNLtb4ONBBSRT9ABinp f3q03VQehflY3GLsvosamJ_0.n19KRF48CfTwVeQQmWymENjMB7E2kBLv9yai584Qn_YinvzQXmi k0.hIboCNSEqqptBzKQZc1Xcl6oa5mbiJMxHmaf8q27UXVFZHUwOIOC8S8HcCc8aGeVnz6YnpMFu gTGMFtbKF7D_kv.ZpzMjGzyDOjGbdqV21aqRVLuiwNCft78vmcdjeuCvnwdBDtsJiLcC4skzuulW tqJS7hQuH3nB4bVTDxXWkAb.Aa5EWbQqdUqOHqHdhrPIE5U.H.mC07CUpcQ5q6wWyZaYgQ2io7xK ssHgeN.HaEncVIHb_nXj5AT_CTOCdCZwp3qJdWOKGhkqbbzYkSFfYpSMjhFO.ILU3Dwf8esUOcjZ ZKGiKurJ8 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Mon, 20 Aug 2018 14:16:21 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp410.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID d257edfd13b2bed4706d68af7d4e1a91; Mon, 20 Aug 2018 14:16:18 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD From: Mark Millard In-Reply-To: <1534771095.27158.46.camel@freebsd.org> Date: Mon, 20 Aug 2018 07:16:15 -0700 Cc: Ian Lepore , Rajesh Kumar , Toomas Soome via freebsd-hackers , freebsd-drivers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> To: Daniel Braniss X-Mailer: Apple Mail (2.3445.9.1) X-Mailman-Approved-At: Mon, 20 Aug 2018 15:03:29 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 14:16:29 -0000 On 2018-Aug-20, at 6:18 AM, Ian Lepore wrote: > On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: >>=20 >>>=20 >>> On 20 Aug 2018, at 09:49, Daniel Braniss = wrote: >>>=20 >>>> . . . >>>=20 >>> hi, >>> I have similar issues with the allwinner/twsi but I do have a Saleae = Logic and here is a nice picture: >> ah, maybe this is better: >> = https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.pn= g > . . . > This has nothing to do with the twsi driver, this is about the ig4 > driver (found in sys/dev/ichiic). >=20 > That screenshot seems to show a bus running at 100KHz like it should > (although the 62:38 duty cycle is a bit suspicious). Being a logic analyzer display, it my just be that the threshold was off from the optimal value. The waveform shape is not really visible. The logic analyzer output also shows a thick "rising" edge without the uparrow symbol. My guess would be that is a rising/falling/rising sequence that on the scale in use does not show space between edges. In other words: a glitch on the leading edge side of the intended pulse. This too might be tied to the threshold used vs . the actual signal properties: no way to tell from what is shown. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Mon Aug 20 16:00:23 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CE4110734D1 for ; Mon, 20 Aug 2018 16:00:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-13.consmr.mail.bf2.yahoo.com (sonic315-13.consmr.mail.bf2.yahoo.com [74.6.134.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 150E073D84 for ; Mon, 20 Aug 2018 16:00:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: uwn8u18VM1leMvAyyZrxR2bwiVUsTeA6Otq1iyo30byDwzlYi_UyJSeTgxIV3NZ diRU85xcQeM.Ucnr6k9rRkvr9RE9UUd5PxZ21cZ8B08YsHogLVkjFZl404F1mgY2SIUJoBnuTXuo f5FrS7zpuIbBf_HyYOzLcAGix7eRuJHt6119Gh4C1w24_.Srg2oBRKcVl.4JJkNE2q0X7SWSvhGi G9rvZp6f4rxo_bE527YIOwgZHe6G0b7DBZAQH4OBhM09IufiaChH9oc8j4D1j8KwRKdnsq_Rf.QB p9vyMGX91lRZqbPZ1C5J1Kw7sulMHAhFTgv.7t_eFGQmOtpOY5VNlplvFD.PEa5tvc6kkYIwebQa o0tBSwlWunrc4aodrQIfBAd1WsSGVlBQ8h_OLKn._91GNwz1iGLnyOJlzpg1C03OQg.q3UEu9CCn ds7zMWQ.PKREWnhLcfu6wnt.Y_sF_TUZufRb4SVbipkfyvaWPxe4UN9iwUFm0yaqaC9fB1D0c36I QA3Yg1zWUwdhtJKqCvwKUlc9qV1NFNTc0s5RQccp66kzoIV9BYtdFPiTvz3ueBWngxxgkef_1sOy 9NpsiY_B4uxsxg7gKlD48z2RxmnAp2Trx9bKXFI68rPY.mrEptRhVx_3L474rjG5FloChoyMMLiL 49506nsWA7ySz_.R_IyRBHSbImMsoMV63KIK0SnYRUz0niPr_A0U4C4Hfn0k__NkyQAP2pF8ExuH SKd4XEvy561VP3dir0L0.QcVB44nlQFVnCkCX.cGGWEY4XMikOrRnP4bAUwAsbRpOW2GCQcpCHG7 mZqKUy5g5.KoZd5PHnHmcIyKuJMNJ1OpX1sZ8OuDv24eCdwEGb9nDwjbep5fVQlq4l2AxOxDE1FZ lCKz3NOA0wz0COlNjunTCBUqaz849szXY_3fKOO1JymTQeVwQxZ.Pebbn23cluXaEIRnxy1wvl00 jaAEluIiGEQCSCiAwcdXKrvOoRchGAABBNIuoZonXWWd_N3tyqNgJG1acwRXFNIWnjGoNhrK3wNc DF1.bW3U6YuJ1.w-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.bf2.yahoo.com with HTTP; Mon, 20 Aug 2018 16:00:22 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp422.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 218f368d65d3a390e5ff3d4a5d591849; Mon, 20 Aug 2018 16:00:19 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Fwd: Need a clarification regarding I2C bus frequency in FreeBSD Message-Id: References: <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> To: Toomas Soome via freebsd-hackers , freebsd-drivers@freebsd.org Date: Mon, 20 Aug 2018 09:00:17 -0700 X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 16:00:23 -0000 [Resend to (just the list) from the right account.] On 2018-Aug-20, at 6:18 AM, Ian Lepore wrote: > On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: >>=20 >>>=20 >>> On 20 Aug 2018, at 09:49, Daniel Braniss = wrote: >>>=20 >>>> . . . >>>=20 >>> hi, >>> I have similar issues with the allwinner/twsi but I do have a Saleae = Logic and here is a nice picture: >> ah, maybe this is better: >> = https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.pn= g > . . . > This has nothing to do with the twsi driver, this is about the ig4 > driver (found in sys/dev/ichiic). >=20 > That screenshot seems to show a bus running at 100KHz like it should > (although the 62:38 duty cycle is a bit suspicious). Being a logic analyzer display, it my just be that the threshold was off from the optimal value. The waveform shape is not really visible. The logic analyzer output also shows a thick "rising" edge without the uparrow symbol. My guess would be that is a rising/falling/rising sequence that on the scale in use does not show space between edges. In other words: a glitch on the leading edge side of the intended pulse. This too might be tied to the threshold used vs . the actual signal properties: no way to tell from what is shown. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Mon Aug 20 15:22:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6663D107220B for ; Mon, 20 Aug 2018 15:22:59 +0000 (UTC) (envelope-from danilogondolfo@gmail.com) Received: from mail-ua1-x936.google.com (mail-ua1-x936.google.com [IPv6:2607:f8b0:4864:20::936]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0198E72878 for ; Mon, 20 Aug 2018 15:22:59 +0000 (UTC) (envelope-from danilogondolfo@gmail.com) Received: by mail-ua1-x936.google.com with SMTP id o11-v6so9960758uak.5 for ; Mon, 20 Aug 2018 08:22:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HCD1EPZlFJzDN5nIEVBMaLiYos+YkeUQf+4BjYjs+/Y=; b=kx0swjshTHGTDA4r6cKeK313WVdobhBuhcxsvTyv/0+cxbWK5UtSaj05W4kePPndoV lr9Qcm4ZPgwRci38473wjep4XiG6pP8jy3xJR4LAou/WGX4X+SjtDuukaX0ejwtNBm2J 2/Rlb8E3AqqhTinwq46Z2ckIV7UPcLSG/NXE7q1NvpeZ03kQNP9G5JDsO4MZ0lu/YFZB 0XV1FwyGOiWOzBtRAiiS9Pt/SEJCNQiyWzl/my7CKKNU7//0oCg0GS4eMwl8mov+7TyI Hvk2jtiSD/5FnXmaW+LSlPFrV9iCZ/NfqFJQUUCjHlCmnrNNa2SLfcYtPrEjsuY6wjkR D7Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HCD1EPZlFJzDN5nIEVBMaLiYos+YkeUQf+4BjYjs+/Y=; b=t2tAFHtPSSy8MeU8MgThUBsfMHCGVYMILJ9nepQDwBEBROdeeKigD8TBjT4GdUEblJ ZD37xH1Vdf5BTCWF/TeIO6PHxq6X7Er8ez4yZOcVTnAr9GCksNQeuCBGr9tHhgIeeOTo ZYWDKtCRM1/tvJheZhPiaJTu4+sBr2vVLn2r/VO+ULT2CleTs2O7s9t4G2foGWcsFbFT kXOxysL0MvFJ/0zLumKmPnSuud7000mruWgNISMYcHjQ6j6J4KEwkLq0rGlGqtAymADK WskhwA0md+BU/MGd1guZwDol0Tx/4/gqrJ5LoVngWE+uCLs/dffEwMg19hsm4WkofWyu IanQ== X-Gm-Message-State: AOUpUlGfLQRwRGoYKYAYc+474+4S3d+zuGCtOwZ6V5y+OYWA12viuYmZ +0ya9gkjZirkLFZzkIGHuRJdlKupik09xH3PC5/HvtO5 X-Google-Smtp-Source: AA+uWPwCgwBOFupidyU/NIyRe9DWKkyojebTSBXoySEWjvvob/O+TcD5jQ4tvan+YAnDctAVYXSxRgaHiaTq48RZGdo= X-Received: by 2002:ab0:1544:: with SMTP id p4-v6mr30452747uae.81.1534778578359; Mon, 20 Aug 2018 08:22:58 -0700 (PDT) MIME-Version: 1.0 References: <95C34771-8D0B-489A-88DA-09AF0D244A1C@yahoo.com> In-Reply-To: <95C34771-8D0B-489A-88DA-09AF0D244A1C@yahoo.com> From: =?UTF-8?Q?Danilo_Eg=C3=AAa_Gondolfo?= Date: Mon, 20 Aug 2018 12:23:46 -0300 Message-ID: Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used To: marklmi@yahoo.com Cc: aliovx@gmail.com, freebsd-hackers@freebsd.org X-Mailman-Approved-At: Mon, 20 Aug 2018 16:12:25 +0000 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 15:22:59 -0000 On Mon, Aug 20, 2018 at 8:54 AM Mark Millard via freebsd-hackers < freebsd-hackers@freebsd.org> wrote: > > > On 2018-Aug-20, at 4:27 AM, Ali Abdallah wrote: > > > . . . > > Also, I read on > > > https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html > > someone saying that sysctls to turn off harvesting is documented in > > random(6) > > > It says random(4), so: > > < > https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=freebsd-release-ports > >. > > > . . . > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > I can confirm this is happening. It's easily reproducible: dd if=/dev/urandom of=/dev/null rand_harvestq eat 100% of one CPU for minutes, in my case the system became unresponsive (it freezes for few seconds). I'm running FreeBSD 12.0-ALPHA2 #14 r337973M -- Danilo. @daniloegea http://people.freebsd.org/~danilo/ From owner-freebsd-hackers@freebsd.org Mon Aug 20 16:13:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 146C510740C0; Mon, 20 Aug 2018 16:13:26 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7FB1C74CA2; Mon, 20 Aug 2018 16:13:25 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wr1-x433.google.com with SMTP id 20-v6so7478363wrb.12; Mon, 20 Aug 2018 09:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=EhIAyQi9LHZZPtsr7XHJC6XfyavYw2vT5mKQJMunLlY=; b=lneuZs+1G7Q9chxqt73PHcBKL2d8Me5p5AHJJ2u30Zsb+SbGpC+kwwZohod04DEIpH mtbV9+yxrP/0VTfUzoPrxqEhbu8SSARTnGXFp2wZbRV/TmOoviHnTG96YWVBdref6jkX gpDNBDERGK3jYKmPlosXvCbk3YZ/13ddYXXNffuxRcYSngM75Op60VMNL/58DlX6imYK hqtlf1JixgibVAcHWK898QPl1Pcne+Ym3Q3+aHzNu3DGeFwDgbT8stBca1CfQ3eMLBp0 7ciXxaBAdYpfN11Yr131sRqUrj6YYmgXS5YooXdPuqY0sJBkvhbp9dIWrtgYAvW9bRUq rFLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=EhIAyQi9LHZZPtsr7XHJC6XfyavYw2vT5mKQJMunLlY=; b=ocWU2+pxazlVY8gBjkatiMBKU3OSRO5Y2WhHBhhtxQICwSJQFZb1A0nyG23VOq6bOV jLnKuVK3Y3hB+5pQQ2F5gBx6JxDe7YGBgMEYqFIhBhpZ16POcPaIwO2xLcK3I1ZmJ1hY a/+dXMrMnBtYg7FiZ0EBkAp/Qs5oodZvd25+XgECa1Db0Jjaqwb3Dlc2PqjI3AL266cp t7Y0l9jSrEYXx/cNFheBh96UEEojM/YXUc1ow9ZULtS2h3L4kHwhlHKwZLzYOiGFCORa 2GHnkHxOBWLvAWaWj6WMbEf81lmx3BYkYUD5F3KmdhA16hCeOI2TZWzqgiWg2RFvaWc/ Ef2w== X-Gm-Message-State: APzg51Dqxup3DuBO5LP+ib8+0/sRhxcHedseAtzV7ktd8aTkAq7HaHuH HxOjgS6rJOJVNkaqHtS6SLLaS3W0 X-Google-Smtp-Source: ANB0Vdbg8CMI1qWrwS7hxKfJrdKK03e28zbYVCrqJ+rIAGvcUR4lF7KMdm4E3hNJAgx5LrBWgMoy3A== X-Received: by 2002:adf:f7c4:: with SMTP id a4-v6mr5620416wrq.86.1534781604370; Mon, 20 Aug 2018 09:13:24 -0700 (PDT) Received: from ernst.home (p5B0234D5.dip0.t-ipconnect.de. [91.2.52.213]) by smtp.gmail.com with ESMTPSA id y128-v6sm12203wmy.26.2018.08.20.09.13.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 20 Aug 2018 09:13:23 -0700 (PDT) Date: Mon, 20 Aug 2018 18:13:22 +0200 From: Gary Jennejohn To: Mark Millard via freebsd-hackers Cc: Mark Millard , Daniel Braniss , Rajesh Kumar , freebsd-drivers@freebsd.org, Ian Lepore Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Message-ID: <20180820181322.71607854@ernst.home> In-Reply-To: <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 16:13:26 -0000 On Mon, 20 Aug 2018 07:16:15 -0700 Mark Millard via freebsd-hackers wrote: > On 2018-Aug-20, at 6:18 AM, Ian Lepore wrote: > > > On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: > >> > >>> > >>> On 20 Aug 2018, at 09:49, Daniel Braniss wrote: > >>> > >>>> . . . > >>> > >>> hi, > >>> I have similar issues with the allwinner/twsi but I do have a Saleae Logic and here is a nice picture: > >> ah, maybe this is better: > >> https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.png > > . . . > > This has nothing to do with the twsi driver, this is about the ig4 > > driver (found in sys/dev/ichiic). > > > > That screenshot seems to show a bus running at 100KHz like it should > > (although the 62:38 duty cycle is a bit suspicious). > > Being a logic analyzer display, it my just be that the threshold > was off from the optimal value. The waveform shape is not really > visible. > > The logic analyzer output also shows a thick "rising" edge without the > uparrow symbol. My guess would be that is a rising/falling/rising > sequence that on the scale in use does not show space between edges. In > other words: a glitch on the leading edge side of the intended pulse. > This too might be tied to the threshold used vs . the actual signal > properties: no way to tell from what is shown. > I have two of these logic analyzers and they definitely do a major clean up of the signals displayed. Things like overshoot and ringing, which can be seen on an oscilloscope, do not appear on what the logic analyzer displays. I suspect the purpose of the trace was simply to show the 100KHz SCL. -- Gary Jennejohn From owner-freebsd-hackers@freebsd.org Mon Aug 20 16:24:00 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52D90107470A for ; Mon, 20 Aug 2018 16:24:00 +0000 (UTC) (envelope-from danilogondolfo@gmail.com) Received: from mail-vk0-f66.google.com (mail-vk0-f66.google.com [209.85.213.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF6CF75492 for ; Mon, 20 Aug 2018 16:23:59 +0000 (UTC) (envelope-from danilogondolfo@gmail.com) Received: by mail-vk0-f66.google.com with SMTP id w193-v6so3016987vke.2 for ; Mon, 20 Aug 2018 09:23:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CslWmNn6B016rJLi+3/ZkdkIHSHCF4kezSGNIKJFW4g=; b=Fp9FFLtjuCz/x71VFKTaf5eS9uG4zZt/vP0fmKX054mFDsGaU4xyd5WyQPJw+ZdCeC agY0z0SyZ2B+/6VZ4fm9jpQEaDFs00ufdi6+z9WXH5FcB/dTCjbOFXJ3jLv4QyX9XJYO +GSlkUI6+x98R7tBnHU7M91DVFUO8OwzsXsFerJXTSKMwvyoAQoFUZ84G/n/juvUw3Nm KNNc09/yjSUbfnm85FdjHDdo9hllmQVUyXVJhNFCphZwGqCv5rhUz3yDjR2piI9Bu6A5 k0rRGnWR7JVzIXyWtGw0fKCfWUs1/v31DBfVeMLxJ1/xjfsfhMeC9vamExKj3RnimPbp SNiw== X-Gm-Message-State: APzg51A7PVjdYOCe3N3P00BDGSW3NGBwPjtefdvu7413P5+FUBDhJFKl J9sJ15fhQnW3qDzPNcbBsPmvii02Z/pVNT5JRj7Ex2N7 X-Google-Smtp-Source: ANB0VdbDRPxEylsgmHRV8yznRwDh1pAGl9q/mh/TE5R/OC0tc+T7FeI3mkKgjOBurTaW0yqfGQQYYXbCpayV+pEZ9E4= X-Received: by 2002:a1f:41ce:: with SMTP id o197-v6mr1514344vka.63.1534778915630; Mon, 20 Aug 2018 08:28:35 -0700 (PDT) MIME-Version: 1.0 References: <95C34771-8D0B-489A-88DA-09AF0D244A1C@yahoo.com> In-Reply-To: <95C34771-8D0B-489A-88DA-09AF0D244A1C@yahoo.com> From: =?UTF-8?Q?Danilo_Eg=C3=AAa_Gondolfo?= Date: Mon, 20 Aug 2018 12:29:24 -0300 Message-ID: Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used To: marklmi@yahoo.com Cc: aliovx@gmail.com, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 16:24:00 -0000 On Mon, Aug 20, 2018 at 8:54 AM Mark Millard via freebsd-hackers < freebsd-hackers@freebsd.org> wrote: > > > On 2018-Aug-20, at 4:27 AM, Ali Abdallah wrote: > > > . . . > > Also, I read on > > > https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html > > someone saying that sysctls to turn off harvesting is documented in > > random(6) > > > It says random(4), so: > > < > https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=freebsd-release-ports > >. > > > . . . > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > I can confirm this is happening. It's easily reproducible: dd if=/dev/urandom of=/dev/null rand_harvestq eat 100% of one CPU for minutes, in my case the system became unresponsive (at least the graphical interface, it freezes for few seconds). I'm running FreeBSD 12.0-ALPHA2 #14 r337973M (sorry if I sent it twice) From owner-freebsd-hackers@freebsd.org Mon Aug 20 17:43:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D25F107699E for ; Mon, 20 Aug 2018 17:43:42 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wm0-x244.google.com (mail-wm0-x244.google.com [IPv6:2a00:1450:400c:c09::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E65B7903C for ; Mon, 20 Aug 2018 17:43:41 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wm0-x244.google.com with SMTP id s9-v6so333263wmh.3 for ; Mon, 20 Aug 2018 10:43:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mbuh5ng3DBhsFcupkjJ5g67gcEoQHuoIW9Tfy6ug6zM=; b=XX5PgfOihZwerq2N4lTqwGUP6wLGka4o37xn7gWNunsjOrF9YchrsWN1gKQxLFw8wJ qkKXYEtKhH1VCx6OmQLHOfVjPZlzMz97PWKsBn74GS+IhdsR5jemyHmhr4yIpcRkvK1N SGqKcnU8XgQxdL64KlQlj1aQnLzn/01UWmllzED3wWza3nLDeyNZoQayxe62xKHnv2iU l3f51pqEtV9aHU+VJFvMw2wP1xodd89YafdD2rT5fguHvpo7aVmve5ePWSQcTkR/DQK7 SUmD+9yKEEDmXaGpu21dr+MfyShY6HkwUrs+6sjW16IXZtKcrP4l6cDJfxFPc/yx3SMd lExQ== X-Gm-Message-State: AOUpUlG4zq1xtJ7EYJkFe1jbMTRZeNz2gk5RTJO92QC3f78u3JmQxLBY jU4TG4Cojk11Zmcs8S+vOTYZ3wgG0Xk= X-Google-Smtp-Source: AA+uWPwH2rGtdKxpwnb/5/w2VH2hTT5END7gD1+Z3Rzp1leNllFy/4V9E6A/llxJobbBkKx/TNxzPg== X-Received: by 2002:a1c:1609:: with SMTP id 9-v6mr9036204wmw.12.1534787020798; Mon, 20 Aug 2018 10:43:40 -0700 (PDT) Received: from gumby.homeunix.com ([90.210.182.138]) by smtp.gmail.com with ESMTPSA id q135-v6sm359655wmd.4.2018.08.20.10.43.39 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 20 Aug 2018 10:43:40 -0700 (PDT) Date: Mon, 20 Aug 2018 18:43:37 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used Message-ID: <20180820184337.6e07e951@gumby.homeunix.com> In-Reply-To: References: X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 17:43:42 -0000 On Mon, 20 Aug 2018 13:27:33 +0200 Ali Abdallah wrote: > Hello, > > I was just sorting randomly some jpg image files using: > > ls *.jpg | sort -R --random-source=/dev/urandom urandom is a sim-link to random, so --random-source=/dev/urandom does nothing useful > The above command never exited. Later I noticed that > one of my CPU is always running 100%. top -S reveals that it is > rand_harvestq kernel service. > > Is this is a bug? This occurs on 12-ALPHA1 and 11.2 It's a bit excessive > Also, I read on > https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html > someone saying that sysctls to turn off harvesting is documented in > random(6) > . > I had a look at that document, but wasn't clear to me how to turn it > off. I tried to play with the mask, > but nothing. what happens if you set the mask to zero? From owner-freebsd-hackers@freebsd.org Mon Aug 20 17:55:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 958DB1077780; Mon, 20 Aug 2018 17:55:47 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E916179C90; Mon, 20 Aug 2018 17:55:46 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x429.google.com with SMTP id g1-v6so13707153wru.2; Mon, 20 Aug 2018 10:55:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wwOZv/9aHCETh455mFE3Lwr8S/NgRDPtK1ctk83xZkA=; b=RKU1d00zOIz2kVOKdWYy//8BIhYYWbn+Qh1JKzkTeqQVf17j4q7jkc5PgkNnjmUiHG 3JPODSKGlWwcMD+nrnSl7o9oWL8W6/4cVoGwDrfCDKFr9EGsHalr3nOX1kZrnQ7mI+Tm uLak9QQJk1sRjn59BshOpMaJKNGiFIxvShgKSFmE4LIXzMksvLQIdHv27fOTGHFgJM4t 2lkhZ3lq001nJbAVYT8Y9DVlTANsgHhFF/C+X17fTumzjgJCogh4sUiq22wMHokH86A8 pvlbHIJ+u+dPr7N6d9WLdE6c1pXirUAvWlykjvmh5CkfgCMQesPGCgPb/ZnJct9z7B1P 5liQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wwOZv/9aHCETh455mFE3Lwr8S/NgRDPtK1ctk83xZkA=; b=MqFi8po3Jdl6zHhVLcoBEbbIcn+1q8qINduQuiffJZZ9WpLiJ5TqVXP1NlTPmWPbZ0 0LaZ3puLZXggQ0PShkE47n4iEDCndEAYn7whH7QHSeT+X6ovpA7jYSKa/uswK4/OQP5B JR3v3RLcuJQRw0Dk063kcszlisULu523u2ofDqYjo0+ZiYYKJKbQJCifDyDuO0yDIW24 Bb12TB8dcmdWpvJfmfW2jdLWDRHcXDjNR/fdxGubp/yzOqENPT9jx7BSESNJOwTTS5dV lmLxL6iVNc5h99OdA9qutaXzNy8hvVXTS4Vot0xKtPWE/ZNL+GuyyPhMl5xXIjWh36Bc zAeA== X-Gm-Message-State: APzg51A8g5H7oaoL0hJ74T8AB8FOQp+GMXi7GRTiR9Tqp4W353d47G/2 Vr0+wb7bbzX06Uh45lBYgowIjy+88UBeuuv0FhA= X-Google-Smtp-Source: ANB0Vda51o0Gqizdmb5km/8PzXw6gWpRxDQCWrfnJHizNqAjRurhH3iYCzcClMYJUBuvi5lhSzPa79Cj8Ihh7Y46IXg= X-Received: by 2002:a5d:4152:: with SMTP id c18-v6mr5119203wrq.61.1534787745172; Mon, 20 Aug 2018 10:55:45 -0700 (PDT) MIME-Version: 1.0 References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> <20180820181322.71607854@ernst.home> In-Reply-To: <20180820181322.71607854@ernst.home> From: Rajesh Kumar Date: Mon, 20 Aug 2018 23:25:33 +0530 Message-ID: Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD To: gljennjohn@gmail.com Cc: freebsd-hackers@freebsd.org, marklmi26-fbsd@yahoo.com, danny@cs.huji.ac.il, freebsd-drivers@freebsd.org, ian@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 17:55:47 -0000 Hi, Re-posting the questions, just in case if its missed in other conversation. By "i2c clock frequency", I mean the internal base frequency only, which drives the chip. I thought data will be transferred on bus based on the base frequency. So, thought both bus and base frequency are same. But from what you said, seems both are different. So, based on the setting in *_HCNT/LCNT register, the bus frequency (which is the rate at which data is transferred) will change for a particular base frequency. Is that right? So, few questions here 1) As you said, we need to have a base frequency of 150 Mhz in our case. So, do we need to program that IG4_REG_CLK_PARMS to 150 Mhz (0x8F0D180)? And can this be done at the same time when programming the HCNT/LNCT registers? 2) Not sure how that 111Hz value is arrived. Can you please explain this calculation. So, that I can derive the appropriate values for HCNT/LCNT for different speeds at 150Mhz base clock. 3) "Default HCNT/LCNT register values would be consistent with an internal base clock speed of 1GHz", Does it mean with those values, all speeds can be achieved until 1GHz clock? On Mon, Aug 20, 2018 at 9:43 PM Gary Jennejohn wrote: > On Mon, 20 Aug 2018 07:16:15 -0700 > Mark Millard via freebsd-hackers wrote: > > > On 2018-Aug-20, at 6:18 AM, Ian Lepore wrote: > > > > > On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: > > >> > > >>> > > >>> On 20 Aug 2018, at 09:49, Daniel Braniss > wrote: > > >>> > > >>>> . . . > > >>> > > >>> hi, > > >>> I have similar issues with the allwinner/twsi but I do have a Saleae > Logic and here is a nice picture: > > >> ah, maybe this is better: > > >> > https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.png > > > > . . . > > > This has nothing to do with the twsi driver, this is about the ig4 > > > driver (found in sys/dev/ichiic). > > > > > > That screenshot seems to show a bus running at 100KHz like it should > > > (although the 62:38 duty cycle is a bit suspicious). > > > > Being a logic analyzer display, it my just be that the threshold > > was off from the optimal value. The waveform shape is not really > > visible. > > > > The logic analyzer output also shows a thick "rising" edge without the > > uparrow symbol. My guess would be that is a rising/falling/rising > > sequence that on the scale in use does not show space between edges. In > > other words: a glitch on the leading edge side of the intended pulse. > > This too might be tied to the threshold used vs . the actual signal > > properties: no way to tell from what is shown. > > > > I have two of these logic analyzers and they definitely do a > major clean up of the signals displayed. > > Things like overshoot and ringing, which can be seen on an > oscilloscope, do not appear on what the logic analyzer displays. > > I suspect the purpose of the trace was simply to show the 100KHz > SCL. > > -- > Gary Jennejohn > From owner-freebsd-hackers@freebsd.org Mon Aug 20 18:35:14 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB33B107873D for ; Mon, 20 Aug 2018 18:35:14 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BF937B4FA for ; Mon, 20 Aug 2018 18:35:14 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wm0-x241.google.com with SMTP id t25-v6so478416wmi.3 for ; Mon, 20 Aug 2018 11:35:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d4pW4lr1C6jbxGCGvmpd2uCmfgFXAarFkutDBnmX6tw=; b=AWgSLBnSjsGhZCFeZajrr9KchdUR4KzrjQcty4t7csUwT8J+sUpgywzWNZZH6SjwFE R4YXnDQuQIyNLV3ak4CgkREL06mNmeXRf6Aqqn7bIL1uMtnrmqGu5FNiwmndEQndoKyL Xt5qmKurEL98m1lvkM8xdGEBxlkO5ZG5Nt/vQM757DYV3Ajnd2RzrsuJ7/rIiUVtka27 OiQk7hoVBIf759FhdEk0wcP0KLowVLm0oVtcQmj9tFcx5mh1J5Ev59mm/TTpPpm1Q1Fg 3gOczobg+WroN7bpQYyOMbsBz/rUsfAGDlYhuGWiz7YCFuSZQSu+8RDJBcIyg7gJ/gj+ HT5w== X-Gm-Message-State: AOUpUlGCaT/keiljPiB8KX1lRr6ETae2zg68wF8nmtjioYIk+7Wxaxpc TJ7qefj5kt5Y3PYTojUlyxSnVXA9u48= X-Google-Smtp-Source: AA+uWPxY3O3LrRZKCSXLvJfrLDFyNPyoKQAInb2csysBcbxSB84+55Y+fw/MfiiRbVsNqErBYlly9g== X-Received: by 2002:a1c:1943:: with SMTP id 64-v6mr25565846wmz.89.1534790113044; Mon, 20 Aug 2018 11:35:13 -0700 (PDT) Received: from gumby.homeunix.com ([90.210.182.138]) by smtp.gmail.com with ESMTPSA id g133-v6sm332965wmf.44.2018.08.20.11.35.11 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 20 Aug 2018 11:35:12 -0700 (PDT) Date: Mon, 20 Aug 2018 19:35:09 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used Message-ID: <20180820193509.19545f40@gumby.homeunix.com> In-Reply-To: <20180820184337.6e07e951@gumby.homeunix.com> References: <20180820184337.6e07e951@gumby.homeunix.com> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 18:35:14 -0000 On Mon, 20 Aug 2018 18:43:37 +0100 RW wrote: > > Also, I read on > > https://lists.freebsd.org/pipermail/freebsd-current/2013-November/046683.html > > someone saying that sysctls to turn off harvesting is documented in > > random(6) > > . > > I had a look at that document, but wasn't clear to me how to turn it > > off. I tried to play with the mask, > > but nothing. > > what happens if you set the mask to zero? I just tried that and it didn't make any difference, which is strange. From owner-freebsd-hackers@freebsd.org Mon Aug 20 19:25:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51AC9107A18B for ; Mon, 20 Aug 2018 19:25:42 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f48.google.com (mail-it0-f48.google.com [209.85.214.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1E837D792 for ; Mon, 20 Aug 2018 19:25:41 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f48.google.com with SMTP id p81-v6so953643itp.1 for ; Mon, 20 Aug 2018 12:25:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=JcmqHHUvjOnEaywohfzsNz2hytgnwR3n3FqGMSitU2E=; b=QM3JQYFuppBHVmfba3mRjWBxfmmVgr73MxdUnyTUJaG/6RjuXL6neBiOzOlpnpMwke x9G34k287YPVOdaMpA3V8tFow3jtR3CoOpoW3G5C2TQ6oHglOOc04fvJ07RVcgDgNXCt imY4D5B98qWvxlpIYjudZawOjOS59FMSYU8MlY09VKYboF8C5YTNMXqwuNkExUsQqpvj 7HCRf0scg088sR4Dg545d79DAff/b53MQ/BPINCt5UmbOOIXICjfeqQ7TnxTfwzuSn0l DPnPY4ieAsfAypfkNoHX9Lrz6AyeBTYG+hPKNCKj48Y+lZ4S9owSfEitOQIGXSk2/80e csHA== X-Gm-Message-State: AOUpUlEqfJT1IYnl6w2JqMam7AAY0cOi6N1SMz4MuPnfpMKIv0Ib+tcQ qcTGxKa+9DCdSWOaqnXf396o5Y/S X-Google-Smtp-Source: AA+uWPxboQYpD7mLsSsiMM5CJUNxXAnTpFwdRkmS5QJW1FB7u0PGL7kqHpx7HyqgdvohD3tXcJ0GLA== X-Received: by 2002:a24:e408:: with SMTP id o8-v6mr14843962ith.77.1534791449972; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) Received: from mail-it0-f49.google.com (mail-it0-f49.google.com. [209.85.214.49]) by smtp.gmail.com with ESMTPSA id p13-v6sm286721itp.3.2018.08.20.11.57.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 11:57:29 -0700 (PDT) Received: by mail-it0-f49.google.com with SMTP id 139-v6so839940itf.0 for ; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) X-Received: by 2002:a24:144:: with SMTP id 65-v6mr13220367itk.62.1534791449552; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:b472:0:0:0:0:0 with HTTP; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) In-Reply-To: References: From: Conrad Meyer Date: Mon, 20 Aug 2018 11:57:29 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used To: Ali Abdallah Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 19:25:42 -0000 Hi Ali, On Mon, Aug 20, 2018 at 4:27 AM, Ali Abdallah wrote: > Hello, > > I was just sorting randomly some jpg image files using: > > ls *.jpg | sort -R --random-source=/dev/urandom > > The above command never exited. Later I noticed that > one of my CPU is always running 100%. top -S reveals that it is > rand_harvestq kernel service. > > Is this is a bug? This occurs on 12-ALPHA1 and 11.2 There is probably at least one bug in sort(1). sort has special behavior if the --random-source matches its default (/dev/random), but otherwise doesn't understand device files or pipes very well. Since urandom isn't exactly the same path as /dev/random, sort fails pretty hard. sort attempts to seed its internal RNG by MD5ing the provided random source path. For its default path, /dev/random, it grabs at most MAX_DEFAULT_RANDOM_SEED_DATA_SIZE (or 1024) bytes. This is hugely excessive and MD5ing it is totally unnecessary, but still mostly harmless. For non-default files, it just passes the path to MD5File, which will read() until EOF. Since /dev/urandom will never return EOF, sort --random-source=/dev/urandom will get stuck in MD5File forever. This is totally stupid. I'm not sure why rand_harvestq would take 100% of a CPU core even as a result of excessive consumption of /dev/urandom, but it is certainly possible that sort(1) is consuming 100% of a CPU core reading from urandom and MD5ing the result. All the best, Conrad From owner-freebsd-hackers@freebsd.org Tue Aug 21 05:14:29 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63FE51089305; Tue, 21 Aug 2018 05:14:29 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C57B873104; Tue, 21 Aug 2018 05:14:28 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1fryzc-000NY9-Sb; Tue, 21 Aug 2018 08:14:12 +0300 From: Daniel Braniss Message-Id: <9F8E2C3D-61D6-487E-A19D-6B91FBAD930B@cs.huji.ac.il> Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Date: Tue, 21 Aug 2018 08:14:11 +0300 In-Reply-To: <20180820181322.71607854@ernst.home> Cc: Mark Millard via freebsd-hackers , Mark Millard , Rajesh Kumar , freebsd-drivers@freebsd.org, Ian Lepore To: gljennjohn@gmail.com References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> <20180820181322.71607854@ernst.home> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 05:14:29 -0000 > On 20 Aug 2018, at 19:13, Gary Jennejohn wrote: >=20 > On Mon, 20 Aug 2018 07:16:15 -0700 > Mark Millard via freebsd-hackers > wrote: >=20 >> On 2018-Aug-20, at 6:18 AM, Ian Lepore wrote: >>=20 >>> On Mon, 2018-08-20 at 11:13 +0300, Daniel Braniss wrote: =20 >>>>=20 >>>>>=20 >>>>> On 20 Aug 2018, at 09:49, Daniel Braniss = wrote: >>>>>=20 >>>>>> . . . =20 >>>>>=20 >>>>> hi, >>>>> I have similar issues with the allwinner/twsi but I do have a = Saleae Logic and here is a nice picture: =20 >>>> ah, maybe this is better: >>>> = https://cs.huji.ac.il/~danny/Screen%20Shot%202018-08-20%20at%2011.06.43.pn= g =20 >>> . . . >>> This has nothing to do with the twsi driver, this is about the ig4 >>> driver (found in sys/dev/ichiic). >>>=20 >>> That screenshot seems to show a bus running at 100KHz like it should >>> (although the 62:38 duty cycle is a bit suspicious). =20 >>=20 >> Being a logic analyzer display, it my just be that the threshold >> was off from the optimal value. The waveform shape is not really >> visible. >>=20 >> The logic analyzer output also shows a thick "rising" edge without = the >> uparrow symbol. My guess would be that is a rising/falling/rising >> sequence that on the scale in use does not show space between edges. = In >> other words: a glitch on the leading edge side of the intended pulse. >> This too might be tied to the threshold used vs . the actual signal >> properties: no way to tell from what is shown. >>=20 >=20 > I have two of these logic analyzers and they definitely do a > major clean up of the signals displayed. >=20 > Things like overshoot and ringing, which can be seen on an > oscilloscope, do not appear on what the logic analyzer displays. >=20 > I suspect the purpose of the trace was simply to show the 100KHz > SCL. >=20 yup, I connected the logic analyzer to check the frequency, which was not changing, later I confirmed by looking at the source that it=E2=80= =99s set at a constant 100KHz. > --=20 > Gary Jennejohn From owner-freebsd-hackers@freebsd.org Tue Aug 21 12:33:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D8ED1070905 for ; Tue, 21 Aug 2018 12:33:13 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BA43581767 for ; Tue, 21 Aug 2018 12:33:12 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.3] (helo=sslproxy06.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fs5qI-0004hM-Lq for freebsd-hackers@freebsd.org; Tue, 21 Aug 2018 14:33:02 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy06.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fs5qI-000Abu-BO for freebsd-hackers@freebsd.org; Tue, 21 Aug 2018 14:33:02 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id DF90C2A165C for ; Tue, 21 Aug 2018 14:33:02 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id mjpjQjUMq6OE for ; Tue, 21 Aug 2018 14:33:02 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 9140B2A167F for ; Tue, 21 Aug 2018 14:33:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id FABSBTHfrg71 for ; Tue, 21 Aug 2018 14:33:02 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 65CA32A165C for ; Tue, 21 Aug 2018 14:33:02 +0200 (CEST) To: FreeBSD From: Sebastian Huber Subject: epoch(9) background information? Message-ID: Date: Tue, 21 Aug 2018 14:33:00 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24862/Tue Aug 21 10:47:00 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 12:33:13 -0000 Hello, I update currently a port of the FreeBSD network stack, etc. to the=20 real-time operating system RTEMS from the head version at 2017-04-04 to=20 the head version of today. I noticed that some read-write locks are=20 replaced by a relatively new stuff called EPOCH(9). Is there some=20 background information available for this? The man page is a bit vague=20 and searching for something named epoch on the internet is not really=20 great. For example, what is the motivation for this change? How is this=20 related to read-copy-update (RCU)? --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Tue Aug 21 13:38:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DD9F1072909 for ; Tue, 21 Aug 2018 13:38:04 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: from mail-io0-x233.google.com (mail-io0-x233.google.com [IPv6:2607:f8b0:4001:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2718283B99 for ; Tue, 21 Aug 2018 13:38:04 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: by mail-io0-x233.google.com with SMTP id l7-v6so15385547iok.6 for ; Tue, 21 Aug 2018 06:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=slagwyBMQO89dIeDSC73D4kUZvviZrNpN0FYVpoF+Cc=; b=tpKeQD1559WFrtA3NlMaWRzd9C+bCifu272Bw8u+BMayTCMKU7UJD7vPwdAu6Pt3Fn QyRM3VV4d2VMt7cMCAVFUOolza7H4oxKvB0ZRYezskujiXy7QuCpwdsb4tdYh9AbXRP/ Zdnj8ffloj4+74EzWAPLwTa2nXXY3y5MLBjHmMUlQVrVkUfvrsKdwcbLUzr8PIxt0Xkf TFjh5tX1X039mm1Tk1VOfdUfLJsOD/oSyWZFiXrKpZDXhH3JtTaFh711rMkQdziLwzwa DpDwKXCZm4/KYPpa+X/v3Ua58gStHaHQE1QDnFk3q37pQXulkRG31Uv23dQt24oX7txj 8mkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=slagwyBMQO89dIeDSC73D4kUZvviZrNpN0FYVpoF+Cc=; b=k7d9AErqZPufYSRiOpoENjDiqJtSvY6LUkKNLVhTPhq0tfCpWsdkGB42IqKGQItfrM 2PUQRdgyPY6rPJSReaTG1laAi4dRUwvQtJ37pVrLCi/KkDItuhGH5dCw/NxkWWOig/+s zHLyWUoQFbQ9ivoLckd0Dl9Wq5ASv4Vzs7NmD8x7fU5GzChekA7wFiQvEirslUeK1Fq/ hLaVbJwIsmzmX1u/E9QueS4YECfCXb5z5vnmMb8B4a1lh9brlZAZxmIYlnEmaYChwZc1 taUT07XhpD1q2Vq+9+1zvcVEOHhtDgJAL5GHkdYiHLwXSVdZRUGVLhFrS6SAA5LG4QD1 I90g== X-Gm-Message-State: APzg51BV5av7BBP0yBQZFRfCf+dULP/u8WJXrGHqvERNk+aarOSjeen6 Yvk9e10W6Nuzi4ev+Ke1CmegputvdLiO4FsFR1ngNmsO X-Google-Smtp-Source: ANB0VdYjLTDHxmoYiIbjZTzaAWJg6jeTEU73s36b6z4alMgkAqj0tWM4oQawQ7yCx/0WMweenQBKXyjmQXj1gfJSsak= X-Received: by 2002:a6b:b7c7:: with SMTP id h190-v6mr5822687iof.164.1534858683585; Tue, 21 Aug 2018 06:38:03 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a6b:d611:0:0:0:0:0 with HTTP; Tue, 21 Aug 2018 06:38:03 -0700 (PDT) In-Reply-To: References: From: Jacques Fourie Date: Tue, 21 Aug 2018 09:38:03 -0400 Message-ID: Subject: Re: epoch(9) background information? To: Sebastian Huber Cc: FreeBSD Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 13:38:04 -0000 On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber < sebastian.huber@embedded-brains.de> wrote: > Hello, > > I update currently a port of the FreeBSD network stack, etc. to the > real-time operating system RTEMS from the head version at 2017-04-04 to t= he > head version of today. I noticed that some read-write locks are replaced = by > a relatively new stuff called EPOCH(9). Is there some background > information available for this? The man page is a bit vague and searching > for something named epoch on the internet is not really great. For exampl= e, > what is the motivation for this change? How is this related to > read-copy-update (RCU)? > > -- > Sebastian Huber, embedded brains GmbH > > Address : Dornierstr. 4, D-82178 Puchheim, Germany > > Phone : +49 89 189 47 41-16 > Fax : +49 89 189 47 41-09 > E-Mail : sebastian.huber@embedded-brains.de > PGP : Public key available on request. > > Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > Additional information is available here : http://concurrencykit.org/pr esentations/ebr.pdf. The way I understand it is that it is mostly used in place of read locks to provide liveness guarantees without using atomics. Additional detail is available in the commit messages. As an example see r333813 for some performance data. From owner-freebsd-hackers@freebsd.org Tue Aug 21 14:08:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E83A110737F7 for ; Tue, 21 Aug 2018 14:08:05 +0000 (UTC) (envelope-from ish@amail.plala.or.jp) Received: from msc11.plala.or.jp (msc11.plala.or.jp [60.36.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 32DC284FFB for ; Tue, 21 Aug 2018 14:08:04 +0000 (UTC) (envelope-from ish@amail.plala.or.jp) Received: from localhost ([2400:4050:9320:7a00::8]) by msc11.plala.or.jp with ESMTP id <20180821140755.BUHD19935.msc11.plala.or.jp@localhost> for ; Tue, 21 Aug 2018 23:07:55 +0900 Date: Tue, 21 Aug 2018 23:07:45 +0900 (JST) Message-Id: <20180821.230745.1541080428454062053.ish@amail.plala.or.jp> To: freebsd-hackers@freebsd.org Subject: what is 'ef_read_entry failed' of freebsd-update From: Masachika ISHIZUKA X-Mailer: Mew version 6.8 on Emacs 26.1 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-VirusScan: Outbound; mvir-ac11; Tue, 21 Aug 2018 23:07:55 +0900 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 14:08:06 -0000 I want to update from 11.2-RELEASE-p1 to 11.2-RELEASE-p2 by freebsd-update. onion# uname -a FreeBSD onion.ish.org 11.2-RELEASE-p1 FreeBSD 11.2-RELEASE-p1 #0: Sun Aug 5 12:04:13 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 onion# freebsd-version -ku 11.2-RELEASE-p1 11.2-RELEASE-p1 onion# freebsd-update fetch Looking up update.FreeBSD.org mirrors... 2 mirrors found. Fetching metadata signature for 11.2-RELEASE from update5.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. Preparing to download files... done. The following files will be updated as part of updating to 11.2-RELEASE-p2: /bin/freebsd-version /boot/kernel/kernel /boot/kernel/vmm.ko /usr/bin/gprof /usr/include/netinet6/in6.h /usr/include/netinet6/ip6_var.h /usr/include/x86/specialreg.h /usr/sbin/wpa_supplicant /usr/share/man/man4/inet.4.gz /usr/share/man/man4/inet6.4.gz /usr/share/man/man4/tcp.4.gz /usr/share/man/mandoc.db /usr/src/contrib/wpa/src/rsn_supp/wpa.c /usr/src/share/man/man4/inet.4 /usr/src/share/man/man4/inet6.4 /usr/src/share/man/man4/tcp.4 /usr/src/sys/amd64/amd64/pmap.c /usr/src/sys/amd64/vmm/intel/vmx.c /usr/src/sys/amd64/vmm/intel/vmx_genassym.c /usr/src/sys/amd64/vmm/intel/vmx_support.S /usr/src/sys/conf/newvers.sh /usr/src/sys/netinet/ip_reass.c /usr/src/sys/netinet6/frag6.c /usr/src/sys/netinet6/in6.h /usr/src/sys/netinet6/in6_proto.c /usr/src/sys/netinet6/ip6_var.h /usr/src/sys/x86/include/specialreg.h onion# freebsd-update install Installing updates...ef_read_entry failed ef_read_entry failed done. onion# freebsd-version -ku 11.2-RELEASE-p2 11.2-RELEASE-p2 onion# freebsd-update rollback Uninstalling updates...ef_read_entry failed ef_read_entry failed done. onion# freebsd-version -ku 11.2-RELEASE-p1 11.2-RELEASE-p1 How can I update to 11.2-RELEASE-p2 by freebsd-update ? P.S. The new /var/db/freebsd-update had the same error. i.e. onion# mv -i /var/db/freebsd-update /var/db/freebsd-update.old onion# mkdir /var/db/freebsd-update onion# chmod 700 /var/db/freebsd-update onion# ls -ld /var/db/freebsd-update* drwx------ 2 root wheel 512 Aug 21 22:58 /var/db/freebsd-update drwx------ 74 root wheel 2048 Aug 21 22:55 /var/db/freebsd-update.old onion# freebsd-update fetch Looking up update.FreeBSD.org mirrors... 2 mirrors found. Fetching public key from update4.freebsd.org... done. Fetching metadata signature for 11.2-RELEASE from update4.freebsd.org... done. Fetching metadata index... done. Fetching 2 metadata files... done. Inspecting system... done. Preparing to download files... done. Fetching 24 patches.....10....20.. done. Applying patches... done. The following files will be updated as part of updating to 11.2-RELEASE-p2: /bin/freebsd-version ...(snip)... /usr/src/sys/x86/include/specialreg.h onion# freebsd-update install Installing updates...ef_read_entry failed ef_read_entry failed done. onion# freebsd-update rollback Uninstalling updates...ef_read_entry failed ef_read_entry failed done. -- Masachika ISHIZUKA From owner-freebsd-hackers@freebsd.org Tue Aug 21 14:33:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 964DE1074816 for ; Tue, 21 Aug 2018 14:33:09 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2r.ore.mailhop.org (outbound2r.ore.mailhop.org [54.200.129.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14D32866E9 for ; Tue, 21 Aug 2018 14:33:09 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 1aa19205-a54f-11e8-904b-1d2e466b3c59 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.ore.mailhop.org (Halon) with ESMTPSA id 1aa19205-a54f-11e8-904b-1d2e466b3c59; Tue, 21 Aug 2018 14:33:01 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7LEX0Vo090677; Tue, 21 Aug 2018 08:33:00 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1534861980.27158.145.camel@freebsd.org> Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD From: Ian Lepore To: Rajesh Kumar , gljennjohn@gmail.com Cc: freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Date: Tue, 21 Aug 2018 08:33:00 -0600 In-Reply-To: References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> <20180820181322.71607854@ernst.home> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 14:33:09 -0000 On Mon, 2018-08-20 at 23:25 +0530, Rajesh Kumar wrote: > Hi, > > Re-posting the questions, just in case if its missed in other conversation. > > By "i2c clock frequency", I mean the internal base frequency only, which > drives the chip.  I thought data will be transferred on bus based on the > base frequency. So, thought both bus and base frequency are same. But from > what you said, seems both are different. So, based on the setting in > *_HCNT/LCNT register, the bus frequency (which is the rate at which data is > transferred) will change for a particular base frequency. Is that right? > > So, few questions here > > 1)  As you said, we need to have a base frequency of 150 Mhz in our case. > So, do we need to program that IG4_REG_CLK_PARMS to 150 Mhz (0x8F0D180)? > And can this be done at the same time when programming the HCNT/LNCT > registers? I don't have this hardware, and I don't have a datasheet that describes the IG4_REG_CLK_PARMS register mentioned in the driver, so I don't really have an answer. I suspect the hardware should set that register with information that lets the driver know what the base clock speed is. Using that information, the driver could calculate the proper values for HCNT/LCNT. Right now the driver lacks *any* support for changing bus speeds. That will be easy to fix, once we figure out:  1. What is in the IG4_REG_CLK_PARMS register?  2. What do we do about versions of the hardware that don't support that register?  > 2)  Not sure how that 111Hz value is arrived.  Can you please explain this > calculation. So, that I can derive the appropriate values for HCNT/LCNT for > different speeds at 150Mhz base clock. It's based on the comment (which I feel certain must be wrong) that the base clock is 25,000 Hz. With HCNT,LCNT set to 100,125, one cycle of the SCL line will last 225 base clock cycles. 1/25000 = 0.000040, that times 225 is 0.009 seconds per SCL cycle. 1/.009 = 111.111 Hz. FWIW, an i2c bus will run fine at 111Hz, it'll just take forever to get anything done. But I don't think the bus is really running at 111Hz, because I don't think the base clock is really 25KHz, I think the comment block is just wrong about all of that. > 3)  "Default HCNT/LCNT register values would be consistent with an internal > base clock speed of 1GHz",  Does it mean with those values, all speeds can > be achieved until 1GHz clock? > Well, the defaults I mentioned are from the datasheet cited in the driver code. Those defaults made me think the base clock was 1GHz on that particular hardware. I just realized that I was off by an order of magnitude, I think I was mixing numbers from the driver and numbers from the datasheet in my head.  The default HCNT,LCNT in that datasheet are 612+706=1318 base cycles to give an SCL rate of 100KHz. So 1318*100000 is 131.8MHz, a very reasonable number. In your world you want the 150MHz base clock to generate the 100Hz SCL, so 150000000/100000 = 1500. My inclination would be to split that in half and have HCNT,LCNT be 750,750, but for some reason there seems to be a bias on this hardware for having HCNT be slightly longer than LCNT, so maybe 775,725. Maybe that's an attempt to compensate for the fact that high levels on the clock line are accomplished with a pullup resistor, and it takes slightly longer for the line to "drift up" to a high state via the pullup, compared to being driven low which would happen quickly. I would expect the default values of the HCNT/LCNT registers would be right for any given hardware... whoever configures the IP block in a SoC would configure the base clock value and the default HCNT/LCNT values to match each other. Putting it that way makes me think that maybe the right thing to do in the driver is just stop setting HCNT/LCNT at all, and rely on the hardware to be configured correctly by default. It's worth a try. It would also be interesting to just print out those values. In the driver on lines 571-575 the code reads all those registers and does nothing with them. If you look in the dragonflybsd version of the driver it printed out all those values after reading them; whoever imported the driver to freebsd just deleted all the kprintf() lines and left the register reads. -- Ian From owner-freebsd-hackers@freebsd.org Tue Aug 21 15:16:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58EAB1075AE9; Tue, 21 Aug 2018 15:16:30 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD28088872; Tue, 21 Aug 2018 15:16:29 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wm0-x234.google.com with SMTP id n11-v6so3194432wmc.2; Tue, 21 Aug 2018 08:16:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=pRJfUfmnb9jB351GNr/CKbKrfjnL2ToAhiTDO25n1vo=; b=VKnEqPCx3rTBDENrBFoF2Ar9miTutqIk4FD0+537jy0BxRAtcrHNtoNtdO2Tb5nf8G 90HdTQ83Ax9NoMCHvsEIR/L/i9/Fzz170P+udfmzghkCl7k62fysU7aZfsAMZV+uYg57 dfs7CYUcTsxbryXVt3S1FrrxoElIyPnhb7lV1jYYBiOsSPrSRFOQkvpLcBGSDjWTAhPm hgyABvtpoAqpAoJWVbkpArHJTs/Rgo06miAx6LPm4KLwGw0nGgB2ztalgwpA7vaJcj1X mQWBbt7gUqxdqFonJMKo2E46jo/UYdhSGQYkSZ8ZAG6WWeDq20MNXIEyBqw8Z91u51i8 LyZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=pRJfUfmnb9jB351GNr/CKbKrfjnL2ToAhiTDO25n1vo=; b=Hn97J0xcTzFuxGSuNcKJlVO41I7kcUpvZq/IQVrsWMllQYEOghD9Oew3dH5dRMOQ/c Sedf2ckeURsxlFKJE3g7q32YaPSuPtK4h2+lWWqwRzNDY2OF7ELWT/pcchrsuh9EbmL+ vLy++6cS6MT7CqyydFFpDBKs98gUKRv4fIEIwf0wCNCLoaezUgH82I+UaJMReZ5CCuxT Udosm89yozszdzV2WFQCEb1UT+qY7nhirJMed6g0TpNooJcxT2YfTnNf6kz1nJVI2tT7 i76+2COztMOygcywpID4ernxNmsi6YqDOZJqHNZlM2eCQR6WXNQ7ft/P4fj1HT8b9yFn RjEA== X-Gm-Message-State: APzg51BhKMBzeIGZc4ReFdNgygtxtlDG6lYMJkT56ciwv/g9BFhbiyOe lQuo614vrbgMEDXV3vS7ZkfqLMvb X-Google-Smtp-Source: ANB0VdaPMuFxKxikOZR3bPowFK6gUMBlJpiYc89H17u8j3K37lXFyKTbSVH6nYz5jtFYxhX1ksvzlg== X-Received: by 2002:a1c:ea91:: with SMTP id g17-v6mr2519377wmi.65.1534864588525; Tue, 21 Aug 2018 08:16:28 -0700 (PDT) Received: from ernst.home (p5B02336E.dip0.t-ipconnect.de. [91.2.51.110]) by smtp.gmail.com with ESMTPSA id g133-v6sm3142828wmf.44.2018.08.21.08.16.27 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Aug 2018 08:16:27 -0700 (PDT) Date: Tue, 21 Aug 2018 17:16:26 +0200 From: Gary Jennejohn To: Ian Lepore Cc: Rajesh Kumar , freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD Message-ID: <20180821171626.49951728@ernst.home> In-Reply-To: <1534861980.27158.145.camel@freebsd.org> References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> <20180820181322.71607854@ernst.home> <1534861980.27158.145.camel@freebsd.org> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 15:16:30 -0000 On Tue, 21 Aug 2018 08:33:00 -0600 Ian Lepore wrote: > On Mon, 2018-08-20 at 23:25 +0530, Rajesh Kumar wrote: > > Hi, > > > > Re-posting the questions, just in case if its missed in other conversation. > > > > By "i2c clock frequency", I mean the internal base frequency only, which > > drives the chip.____I thought data will be transferred on bus based on the > > base frequency. So, thought both bus and base frequency are same. But from > > what you said, seems both are different. So, based on the setting in > > *_HCNT/LCNT register, the bus frequency (which is the rate at which data is > > transferred) will change for a particular base frequency. Is that right? > > > > So, few questions here > > > > 1)____As you said, we need to have a base frequency of 150 Mhz in our case. > > So, do we need to program that IG4_REG_CLK_PARMS to 150 Mhz (0x8F0D180)? > > And can this be done at the same time when programming the HCNT/LNCT > > registers? > > I don't have this hardware, and I don't have a datasheet that describes > the__IG4_REG_CLK_PARMS register mentioned in the driver, so I don't > really have an answer. I suspect the hardware should set that register > with information that lets the driver know what the base clock speed > is. Using that information, the driver could calculate the proper > values for HCNT/LCNT. > > Right now the driver lacks *any* support for changing bus speeds. That > will be easy to fix, once we figure out: > > __1. What is in the IG4_REG_CLK_PARMS register? > __2. What do we do about versions of the hardware that don't support > that register?__ > > > 2)____Not sure how that 111Hz value is arrived.____Can you please explain this > > calculation. So, that I can derive the appropriate values for HCNT/LCNT for > > different speeds at 150Mhz base clock. > > It's based on the comment (which I feel certain must be wrong) that the > base clock is 25,000 Hz. With HCNT,LCNT set to 100,125, one cycle of > the SCL line will last 225 base clock cycles. 1/25000 = 0.000040, that > times 225 is 0.009 seconds per SCL cycle. 1/.009 = 111.111 Hz. > > FWIW, an i2c bus will run fine at 111Hz, it'll just take forever to get > anything done. But I don't think the bus is really running at 111Hz, > because I don't think the base clock is really 25KHz, I think the > comment block is just wrong about all of that. > > > 3)____"Default HCNT/LCNT register values would be consistent with an internal > > base clock speed of 1GHz",____Does it mean with those values, all speeds can > > be achieved until 1GHz clock? > > > > Well, the defaults I mentioned are from the datasheet cited in the > driver code. Those defaults made me think the base clock was 1GHz on > that particular hardware. I just realized that I was off by an order of > magnitude, I think I was mixing numbers from the driver and numbers > from the datasheet in my head. __The default HCNT,LCNT in that datasheet > are 612+706=1318 base cycles to give an SCL rate of 100KHz. So > 1318*100000 is 131.8MHz, a very reasonable number. > > In your world you want the 150MHz base clock to generate the 100Hz SCL, > so 150000000/100000 = 1500. My inclination would be to split that in > half and have HCNT,LCNT be 750,750, but for some reason there seems to > be a bias on this hardware for having HCNT be slightly longer than > LCNT, so maybe 775,725. Maybe that's an attempt to compensate for the > fact that high levels on the clock line are accomplished with a pullup > resistor, and it takes slightly longer for the line to "drift up" to a > high state via the pullup, compared to being driven low which would > happen quickly. > Just a remark. According to Table 10 in Chapter 6 of the I2C standard from NXP, SCL low must satisfy a minimum of 4.7 uS and SCL high must satisfy a minimum of 4.0 uS when running SCL at 100KHz. At 400KHz the values are respectively 1.3uS and 0.6uS. 725 results in 4.83uS, which would be OK for the low phase if it gets pulled low quite quickly. 775 results in 5.17uS, which is also acceptable, especially if it takes rather long for it to be pulled up. 750 results in 5.0uS for each, of course. This may theoreticalluy be a safer value, but it doesn't reflect any odd behavior which the real hardware may exhibit. > I would expect the default values of the HCNT/LCNT registers would be > right for any given hardware... whoever configures the IP block in a > SoC would configure the base clock value and the default HCNT/LCNT > values to match each other. Putting it that way makes me think that > maybe the right thing to do in the driver is just stop setting > HCNT/LCNT at all, and rely on the hardware to be configured correctly > by default. It's worth a try. > > It would also be interesting to just print out those values. In the > driver on lines 571-575 the code reads all those registers and does > nothing with them. If you look in the dragonflybsd version of the > driver it printed out all those values after reading them; whoever > imported the driver to freebsd just deleted all the kprintf() lines and > left the register reads. > -- Gary Jennejohn From owner-freebsd-hackers@freebsd.org Wed Aug 22 00:19:07 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41CCA10844AF for ; Wed, 22 Aug 2018 00:19:07 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B679176373 for ; Wed, 22 Aug 2018 00:19:06 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wr1-x42b.google.com with SMTP id u12-v6so132236wrr.4 for ; Tue, 21 Aug 2018 17:19:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jGzFuB72oOCMHdEDG357l+BI08Y19KIsGiXsjOMMhxQ=; b=I9rQ7O3g/mDosFeCdylMoyd4doaYHHCRsq+c1za85KOGMfAQJk54OCvb75ZYbMQMb/ 1xSCD76UlNIrJYTmTS0l82Z0YAbf6q3OwOelPI1kKnKZ14sPmFH57Qo1pXz671vtNcnl OjxPPzCFYJky2HIyPzh5ypGFRj+XpPP7dzMYmMoB0/3+5Fyogmp1RwQY5ExcYMmQaYkd ix69gGmQM1vjXV/sVdUpP0k0oBacFkglYFhrtfB9yu2x6WUmYrtZkDDiWCeQD3s88KZn j5F/R579Pb/fhjqixp8w7XCkY8u6ql0QWw/abpgjGg3iJyP10VaWpZTZcOKf3OsDijfO q+/w== X-Gm-Message-State: AOUpUlF9SAB2DxJjDNGNqSB1UgrDb6OdT5/tHBsnMUdwgHHABYx5MyNV Ok+qIYYr6UdumYr0u+CGXdwT5cki6j0= X-Google-Smtp-Source: AA+uWPz+0z2OdqPE+Rfe8h6BC6iT6ydgQA+esU9FdSDGlDSXHoH6g3CvCdvY+IyqyjPgin/0WgTo0w== X-Received: by 2002:adf:b642:: with SMTP id i2-v6mr32211873wre.54.1534897145565; Tue, 21 Aug 2018 17:19:05 -0700 (PDT) Received: from gumby.homeunix.com ([90.210.182.138]) by smtp.gmail.com with ESMTPSA id l18-v6sm147837wru.75.2018.08.21.17.19.03 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Aug 2018 17:19:04 -0700 (PDT) Date: Wed, 22 Aug 2018 01:19:01 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used Message-ID: <20180822011901.6eb678cb@gumby.homeunix.com> In-Reply-To: <20180820184337.6e07e951@gumby.homeunix.com> References: <20180820184337.6e07e951@gumby.homeunix.com> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 00:19:07 -0000 On Mon, 20 Aug 2018 18:43:37 +0100 RW wrote: > On Mon, 20 Aug 2018 13:27:33 +0200 > Ali Abdallah wrote: > > > Hello, > > > > I was just sorting randomly some jpg image files using: > > > > ls *.jpg | sort -R --random-source=/dev/urandom > > urandom is a sim-link to random, so --random-source=/dev/urandom > does nothing useful > > > The above command never exited. Later I noticed that > > one of my CPU is always running 100%. top -S reveals that it is > > rand_harvestq kernel service. > > > > Is this is a bug? This occurs on 12-ALPHA1 and 11.2 > > It's a bit excessive I think I see what is going on. If you have a hardware entropy source then when you read N bytes out of /dev/random, random_sources_feed() tries to put at least that amount into each of the entropy pools (32 for fortuna). So if you are reading at 100MB/s, you are trying to feed 3.2GB/s into the pools. Overwriting a slow drive from /dev/random seems to be enough to waste a CPU core my PC. Fortuna is only allowed to resend after 100ms, and anything more than 1kB/reseed (pools*keysize) is a waste of CPU cycles. IMO random_sources_feed() should limit itself to RANDOM_KEYSIZE bytes per call for each pool/source combination - even that's overkill. From owner-freebsd-hackers@freebsd.org Wed Aug 22 00:56:48 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47DD3108559B for ; Wed, 22 Aug 2018 00:56:48 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f52.google.com (mail-it0-f52.google.com [209.85.214.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C102677692 for ; Wed, 22 Aug 2018 00:56:47 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f52.google.com with SMTP id 72-v6so875212itw.3 for ; Tue, 21 Aug 2018 17:56:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc:content-transfer-encoding; bh=YFcXOiEIcuEco9+DYN/0mRuL76wFWE7gN/pgSFulvj4=; b=m0RL+DrN3doXdyY20+/yB7rFsmEC6aymANISQSwrhAGl8+oVY1y5nK7Zcx6DhC0g4j NFlTUrR5FJEltFtAfabpmR7JooPFIZrZck7hNOZgwI7GOc3awDwM81uQSwAnvkf/7MZH kBx1Vti/CoOBbpHFxOvLipTRHOFdG2hyRttVp/ist7s6MWhCE/ofwkWAd1ZVctPBCHr8 UI4oXdKkxMdOomqzJVpgz7NOQM0ukCHiyovxM21BUyFv7Z9Pwu0Bc2m+Z1tG9L4f/9Fo JmlVYVrvjPWtCHdjYhkaRCBuyF9FlBUzAAVpr6ce/88xIYKtUNHu7n7F0f23gCF5MuTV uzfA== X-Gm-Message-State: APzg51ALhcUl+AsAzDabgdpVg496/Fq9bkUc49xQwBDbCsXdsEiim59V Aq2f1Cu3OcZs/ZAVZIZbdN0+l3DP X-Google-Smtp-Source: ANB0VdaRElDbI+Oqkhu2nNWYZewLLA6MYRDXMrcoL0S8uAGfSv6fU8KlC11FGogRbMoq860q2Q22wQ== X-Received: by 2002:a02:b5c4:: with SMTP id y4-v6mr3719590jaj.138.1534899401024; Tue, 21 Aug 2018 17:56:41 -0700 (PDT) Received: from mail-it0-f47.google.com (mail-it0-f47.google.com. [209.85.214.47]) by smtp.gmail.com with ESMTPSA id x8-v6sm114202iog.13.2018.08.21.17.56.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Aug 2018 17:56:40 -0700 (PDT) Received: by mail-it0-f47.google.com with SMTP id t69-v6so172757itb.4 for ; Tue, 21 Aug 2018 17:56:40 -0700 (PDT) X-Received: by 2002:a24:f945:: with SMTP id l66-v6mr1463961ith.6.1534899400775; Tue, 21 Aug 2018 17:56:40 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:b472:0:0:0:0:0 with HTTP; Tue, 21 Aug 2018 17:56:40 -0700 (PDT) In-Reply-To: <20180822011901.6eb678cb@gumby.homeunix.com> References: <20180820184337.6e07e951@gumby.homeunix.com> <20180822011901.6eb678cb@gumby.homeunix.com> From: Conrad Meyer Date: Tue, 21 Aug 2018 17:56:40 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used To: RW Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 00:56:48 -0000 On Tue, Aug 21, 2018 at 5:19 PM, RW via freebsd-hackers wrote: > > I think I see what is going on. If you have a hardware entropy source > then when you read N bytes out of /dev/random, random_sources_feed() > tries to put at least that amount into each of the entropy pools (32 > for fortuna). So if you are reading at 100MB/s, you are trying to feed > 3.2GB/s into the pools. Overwriting a slow drive from /dev/random seems > to be enough to waste a CPU core my PC. Yep, I came to a similar conclusion[1]. I think you're off by a factor of two, though =E2=80=94 it's even worse than that! It tries to res= eed 64x as many bytes from the configured random sources as data read out of the random device. > Fortuna is only allowed to resend after 100ms, and anything more than > 1kB/reseed (pools*keysize) is a waste of CPU cycles. IMO > random_sources_feed() should limit itself to RANDOM_KEYSIZE bytes per > call for each pool/source combination - even that's overkill. I am less familiar on what Fortuna permits, but yeah, clearly what we have now is excessive. Best, Conrad [1]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230808#c1 From owner-freebsd-hackers@freebsd.org Wed Aug 22 06:34:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 677A9108E63F for ; Wed, 22 Aug 2018 06:34:41 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F21E184A24; Wed, 22 Aug 2018 06:34:40 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.2] (helo=sslproxy05.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsMj1-00055O-A3; Wed, 22 Aug 2018 08:34:39 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsMj0-000MsY-Ub; Wed, 22 Aug 2018 08:34:39 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 9F9BC2A165C; Wed, 22 Aug 2018 08:34:40 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id zwduocX7XopA; Wed, 22 Aug 2018 08:34:38 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 355BA2A167E; Wed, 22 Aug 2018 08:34:38 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id zfAf8mP6bkTu; Wed, 22 Aug 2018 08:34:38 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id ED0922A165C; Wed, 22 Aug 2018 08:34:37 +0200 (CEST) Subject: Re: epoch(9) background information? To: FreeBSD Cc: Matthew Macy References: From: Sebastian Huber Message-ID: Date: Wed, 22 Aug 2018 08:34:35 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24863/Tue Aug 21 18:48:32 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 06:34:41 -0000 On 21/08/18 15:38, Jacques Fourie wrote: > > > On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber=20 > > wrote: > > Hello, > > I update currently a port of the FreeBSD network stack, etc. to > the real-time operating system RTEMS from the head version at > 2017-04-04 to the head version of today. I noticed that some > read-write locks are replaced by a relatively new stuff called > EPOCH(9). Is there some background information available for this? > The man page is a bit vague and searching for something named > epoch on the internet is not really great. For example, what is > the motivation for this change? How is this related to > read-copy-update (RCU)? > > --=20 > Sebastian Huber, embedded brains GmbH > > Address : Dornierstr. 4, D-82178 Puchheim, Germany > > Phone=C2=A0 =C2=A0: +49 89 189 47 41-16 > Fax=C2=A0 =C2=A0 =C2=A0: +49 89 189 47 41-09 > E-Mail=C2=A0 : sebastian.huber@embedded-brains.de > > PGP=C2=A0 =C2=A0 =C2=A0: Public key available on request. > > Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne de= s EHUG. > > _______________________________________________ > freebsd-hackers@freebsd.org > mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org > " > > > Additional information is available here :=20 > http://concurrencykit.org/presentations/ebr.pdf=20 > . The way I=20 > understand it is that it is mostly used in place of read locks to=20 > provide liveness guarantees without using atomics. Additional detail=20 > is available in the commit messages. As an example see r333813 for=20 > some performance data. > Thanks, for the reference. The "epoch reclamation" are good keywords to=20 find more information. What is the right mailing list to ask questions about the epoch=20 implementation of the FreeBSD kernel? To support this machinery in RTEMS is a bit difficult (in particular=20 EPOCH_LOCKED). Since RTEMS is supposed to be a real-time operating=20 system it supports only fixed-priority and job-level fixed priority=20 (EDF) schedulers. To allow some scaling to larger SMP systems it=20 supports clustered scheduling together with the mutual exclusion locking=20 protocols MrsP (http://www-users.cs.york.ac.uk/~burns/MRSPpaper.pdf) and=20 OMIP (http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf). This makes=20 the thread pinning hard to implement (which is very easy to support in=20 FreeBSD). The locking protocols may temporarily move a thread which owns=20 a mutex to a foreign scheduler instance, e.g. a thread which wants to=20 obtain the mutex helps the owner to make progress if it was pre-empted=20 in its home scheduler instance. Due to a timeout of the helper the owner=20 may loose the right to execute in the foreign scheduler instance. This=20 would make it impossible to fulfil the processor pinning constraint=20 (e.g. the thread priority in the foreign scheduler instance is undefined)= . It would save me a lot of trouble if I could assume that EPOCH_LOCKED is=20 an exotic feature which is unlikely to get used in FreeBSD. --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Wed Aug 22 06:42:56 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C92FC108E9B2 for ; Wed, 22 Aug 2018 06:42:55 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5FF1985055; Wed, 22 Aug 2018 06:42:55 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.2] (helo=sslproxy05.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsMqz-0005Hs-FL; Wed, 22 Aug 2018 08:42:53 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsMqz-000PPQ-8d; Wed, 22 Aug 2018 08:42:53 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id C73982A165C; Wed, 22 Aug 2018 08:42:54 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id OSXoiEVZger6; Wed, 22 Aug 2018 08:42:52 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 4AE9C2A167E; Wed, 22 Aug 2018 08:42:52 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id t92RFRz0cIzN; Wed, 22 Aug 2018 08:42:52 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 176D12A165C; Wed, 22 Aug 2018 08:42:52 +0200 (CEST) Subject: Re: epoch(9) background information? From: Sebastian Huber To: FreeBSD Cc: Matthew Macy References: Message-ID: <15e3f080-2f82-a243-80e9-f0a916445828@embedded-brains.de> Date: Wed, 22 Aug 2018 08:42:49 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24863/Tue Aug 21 18:48:32 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 06:42:56 -0000 On 22/08/18 08:34, Sebastian Huber wrote: > On 21/08/18 15:38, Jacques Fourie wrote: >> >> >> On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber=20 >> > > wrote: >> >> =C2=A0=C2=A0=C2=A0 Hello, >> >> =C2=A0=C2=A0=C2=A0 I update currently a port of the FreeBSD network st= ack, etc. to >> =C2=A0=C2=A0=C2=A0 the real-time operating system RTEMS from the head = version at >> =C2=A0=C2=A0=C2=A0 2017-04-04 to the head version of today. I noticed = that some >> =C2=A0=C2=A0=C2=A0 read-write locks are replaced by a relatively new s= tuff called >> =C2=A0=C2=A0=C2=A0 EPOCH(9). Is there some background information avai= lable for this? >> =C2=A0=C2=A0=C2=A0 The man page is a bit vague and searching for somet= hing named >> =C2=A0=C2=A0=C2=A0 epoch on the internet is not really great. For exam= ple, what is >> =C2=A0=C2=A0=C2=A0 the motivation for this change? How is this related= to >> =C2=A0=C2=A0=C2=A0 read-copy-update (RCU)? >> >> =C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 Sebastian Huber, embedded bra= ins GmbH >> >> =C2=A0=C2=A0=C2=A0 Address : Dornierstr. 4, D-82178 Puchheim, Germany >> >> =C2=A0=C2=A0=C2=A0 Phone=C2=A0 =C2=A0: +49 89 189 47 41-16 >> =C2=A0=C2=A0=C2=A0 Fax=C2=A0 =C2=A0 =C2=A0: +49 89 189 47 41-09 >> =C2=A0=C2=A0=C2=A0 E-Mail=C2=A0 : sebastian.huber@embedded-brains.de >> =C2=A0=C2=A0=C2=A0 >> =C2=A0=C2=A0=C2=A0 PGP=C2=A0 =C2=A0 =C2=A0: Public key available on re= quest. >> >> =C2=A0=C2=A0=C2=A0 Diese Nachricht ist keine gesch=C3=A4ftliche Mittei= lung im Sinne des=20 >> EHUG. >> >> =C2=A0=C2=A0=C2=A0 _______________________________________________ >> =C2=A0=C2=A0=C2=A0 freebsd-hackers@freebsd.org >> =C2=A0=C2=A0=C2=A0 mailing list >> =C2=A0=C2=A0=C2=A0 https://lists.freebsd.org/mailman/listinfo/freebsd-= hackers >> >> =C2=A0=C2=A0=C2=A0 To unsubscribe, send any mail to >> =C2=A0=C2=A0=C2=A0 "freebsd-hackers-unsubscribe@freebsd.org >> =C2=A0=C2=A0=C2=A0 " >> >> >> Additional information is available here :=20 >> http://concurrencykit.org/presentations/ebr.pdf=20 >> . The way I=20 >> understand it is that it is mostly used in place of read locks to=20 >> provide liveness guarantees without using atomics. Additional detail=20 >> is available in the commit messages. As an example see r333813 for=20 >> some performance data. >> > > Thanks, for the reference. The "epoch reclamation" are good keywords=20 > to find more information. > > What is the right mailing list to ask questions about the epoch=20 > implementation of the FreeBSD kernel? > > To support this machinery in RTEMS is a bit difficult (in particular=20 > EPOCH_LOCKED). Since RTEMS is supposed to be a real-time operating=20 > system it supports only fixed-priority and job-level fixed priority=20 > (EDF) schedulers. To allow some scaling to larger SMP systems it=20 > supports clustered scheduling together with the mutual exclusion=20 > locking protocols MrsP=20 > (http://www-users.cs.york.ac.uk/~burns/MRSPpaper.pdf) and OMIP=20 > (http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf). This makes the=20 > thread pinning hard to implement (which is very easy to support in=20 > FreeBSD). The locking protocols may temporarily move a thread which=20 > owns a mutex to a foreign scheduler instance, e.g. a thread which=20 > wants to obtain the mutex helps the owner to make progress if it was=20 > pre-empted in its home scheduler instance. Due to a timeout of the=20 > helper the owner may loose the right to execute in the foreign=20 > scheduler instance. This would make it impossible to fulfil the=20 > processor pinning constraint (e.g. the thread priority in the foreign=20 > scheduler instance is undefined). > > It would save me a lot of trouble if I could assume that EPOCH_LOCKED=20 > is an exotic feature which is unlikely to get used in FreeBSD. > Another question, is it a common use case to call epoch_enter_preempt()=20 and epoch_exit_preempt() while owning a mutex? --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Wed Aug 22 06:44:31 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5DDA108EA66 for ; Wed, 22 Aug 2018 06:44:31 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 743A885138 for ; Wed, 22 Aug 2018 06:44:31 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 2FFA811F1E for ; Wed, 22 Aug 2018 06:44:31 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f44.google.com with SMTP id 72-v6so1628808itw.3 for ; Tue, 21 Aug 2018 23:44:31 -0700 (PDT) X-Gm-Message-State: AOUpUlGGpIRfljIKmaMaDTX5G0p/tIPY+Y5Z3HFY5movxWFAoy8fzfzy EiARleuJw7WAxuZ5TRDXOu9LgdOCYk6qREeVpu4= X-Google-Smtp-Source: AA+uWPzCrc+mBQBav2NV8mgJKK8rhPy7dmBetcfWqH1nbCbk500Di+cq3BuaROBy1O6AD0KDXp9HssnzJ+6c6z+xJrE= X-Received: by 2002:a02:2b12:: with SMTP id h18-v6mr47599150jaa.10.1534920270523; Tue, 21 Aug 2018 23:44:30 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Tue, 21 Aug 2018 23:44:19 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: epoch(9) background information? To: sebastian.huber@embedded-brains.de Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 06:44:32 -0000 On Tue, Aug 21, 2018 at 11:34 PM Sebastian Huber < sebastian.huber@embedded-brains.de> wrote: > On 21/08/18 15:38, Jacques Fourie wrote: > > > > > > On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber > > > > wrote: > > > > Hello, > > > > I update currently a port of the FreeBSD network stack, etc. to > > the real-time operating system RTEMS from the head version at > > 2017-04-04 to the head version of today. I noticed that some > > read-write locks are replaced by a relatively new stuff called > > EPOCH(9). Is there some background information available for this? > > The man page is a bit vague and searching for something named > > epoch on the internet is not really great. For example, what is > > the motivation for this change? How is this related to > > read-copy-update (RCU)? > > > > -- > > Sebastian Huber, embedded brains GmbH > > > > Address : Dornierstr. 4, D-82178 Puchheim, Germany > > < > https://maps.google.com/?q=3DDornierstr.+4,+D-82178+Puchheim,+Germany&ent= ry=3Dgmail&source=3Dg > > > > Phone : +49 89 189 47 41-16 > > Fax : +49 89 189 47 41-09 > > E-Mail : sebastian.huber@embedded-brains.de > > > > PGP : Public key available on request. > > > > Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne de= s EHUG. > > > > _______________________________________________ > > freebsd-hackers@freebsd.org > > mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > > To unsubscribe, send any mail to > > "freebsd-hackers-unsubscribe@freebsd.org > > " > > > > > > Additional information is available here : > > http://concurrencykit.org/presentations/ebr.pdf > > . The way I > > understand it is that it is mostly used in place of read locks to > > provide liveness guarantees without using atomics. Additional detail > > is available in the commit messages. As an example see r333813 for > > some performance data. > > > > Thanks, for the reference. The "epoch reclamation" are good keywords to > find more information. > > What is the right mailing list to ask questions about the epoch > implementation of the FreeBSD kernel? > > -hackers is probably as good as any. Your questions are at a high enough level that they might be appropriate for -arch. To support this machinery in RTEMS is a bit difficult (in particular > EPOCH_LOCKED). Since RTEMS is supposed to be a real-time operating > system it supports only fixed-priority and job-level fixed priority > (EDF) schedulers. To allow some scaling to larger SMP systems it > supports clustered scheduling together with the mutual exclusion locking > protocols MrsP (http://www-users.cs.york.ac.uk/~burns/MRSPpaper.pdf) and > OMIP (http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf). This makes > the thread pinning hard to implement (which is very easy to support in > FreeBSD). The locking protocols may temporarily move a thread which owns > a mutex to a foreign scheduler instance, e.g. a thread which wants to > obtain the mutex helps the owner to make progress if it was pre-empted > in its home scheduler instance. Due to a timeout of the helper the owner > may loose the right to execute in the foreign scheduler instance. This > would make it impossible to fulfil the processor pinning constraint > (e.g. the thread priority in the foreign scheduler instance is undefined)= . > > It would save me a lot of trouble if I could assume that EPOCH_LOCKED is > an exotic feature which is unlikely to get used in FreeBSD. > EPOCH_LOCKED is something that one would only want to use in a fairly narrow set of circumstances. The only place it's being discussed currently is in pmap: https://reviews.freebsd.org/D15983 There it would conceivably replace a global mutex that currently serializes all munmap operations. -M From owner-freebsd-hackers@freebsd.org Wed Aug 22 06:49:39 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB455108ECA4 for ; Wed, 22 Aug 2018 06:49:38 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8AAA8853A0 for ; Wed, 22 Aug 2018 06:49:38 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f53.google.com (mail-it0-f53.google.com [209.85.214.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 4874B11F1F for ; Wed, 22 Aug 2018 06:49:38 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f53.google.com with SMTP id s7-v6so1633565itb.4 for ; Tue, 21 Aug 2018 23:49:38 -0700 (PDT) X-Gm-Message-State: APzg51A/b6j4RO/g5WRSQJAhUon+pa+9J5cZ65hFPi67lGMDG5dnXSdR CHKUKQb7hf4eYXj96d7vo7cbDz387xJByBLpsqo= X-Google-Smtp-Source: ANB0VdbYfHKactwVG1eNyNUJaH28bdPTgzP/kX/F0kp3zigr7DycKWuEIBUAVcZEyJHqVoDj5XvJ0MCruDp8REsEpMM= X-Received: by 2002:a24:704f:: with SMTP id f76-v6mr2120864itc.30.1534920577728; Tue, 21 Aug 2018 23:49:37 -0700 (PDT) MIME-Version: 1.0 References: <15e3f080-2f82-a243-80e9-f0a916445828@embedded-brains.de> In-Reply-To: <15e3f080-2f82-a243-80e9-f0a916445828@embedded-brains.de> From: Matthew Macy Date: Tue, 21 Aug 2018 23:49:26 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: epoch(9) background information? To: sebastian.huber@embedded-brains.de Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 06:49:39 -0000 On Tue, Aug 21, 2018 at 11:42 PM Sebastian Huber < sebastian.huber@embedded-brains.de> wrote: > On 22/08/18 08:34, Sebastian Huber wrote: > > On 21/08/18 15:38, Jacques Fourie wrote: > >> > >> > >> On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber > >> >> > wrote: > >> > >> Hello, > >> > >> I update currently a port of the FreeBSD network stack, etc. to > >> the real-time operating system RTEMS from the head version at > >> 2017-04-04 to the head version of today. I noticed that some > >> read-write locks are replaced by a relatively new stuff called > >> EPOCH(9). Is there some background information available for this? > >> The man page is a bit vague and searching for something named > >> epoch on the internet is not really great. For example, what is > >> the motivation for this change? How is this related to > >> read-copy-update (RCU)? > >> > >> -- Sebastian Huber, embedded brains GmbH > >> > >> Address : Dornierstr. 4, D-82178 Puchheim, Germany > >> < > https://maps.google.com/?q=3DDornierstr.+4,+D-82178+Puchheim,+Germany&ent= ry=3Dgmail&source=3Dg > > > >> Phone : +49 89 189 47 41-16 > >> Fax : +49 89 189 47 41-09 > >> E-Mail : sebastian.huber@embedded-brains.de > >> > >> PGP : Public key available on request. > >> > >> Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne d= es > >> EHUG. > >> > >> _______________________________________________ > >> freebsd-hackers@freebsd.org > >> mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >> > >> To unsubscribe, send any mail to > >> "freebsd-hackers-unsubscribe@freebsd.org > >> " > >> > >> > >> Additional information is available here : > >> http://concurrencykit.org/presentations/ebr.pdf > >> . The way I > >> understand it is that it is mostly used in place of read locks to > >> provide liveness guarantees without using atomics. Additional detail > >> is available in the commit messages. As an example see r333813 for > >> some performance data. > >> > > > > Thanks, for the reference. The "epoch reclamation" are good keywords > > to find more information. > > > > What is the right mailing list to ask questions about the epoch > > implementation of the FreeBSD kernel? > > > > To support this machinery in RTEMS is a bit difficult (in particular > > EPOCH_LOCKED). Since RTEMS is supposed to be a real-time operating > > system it supports only fixed-priority and job-level fixed priority > > (EDF) schedulers. To allow some scaling to larger SMP systems it > > supports clustered scheduling together with the mutual exclusion > > locking protocols MrsP > > (http://www-users.cs.york.ac.uk/~burns/MRSPpaper.pdf) and OMIP > > (http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf). This makes the > > thread pinning hard to implement (which is very easy to support in > > FreeBSD). The locking protocols may temporarily move a thread which > > owns a mutex to a foreign scheduler instance, e.g. a thread which > > wants to obtain the mutex helps the owner to make progress if it was > > pre-empted in its home scheduler instance. Due to a timeout of the > > helper the owner may loose the right to execute in the foreign > > scheduler instance. This would make it impossible to fulfil the > > processor pinning constraint (e.g. the thread priority in the foreign > > scheduler instance is undefined). > > > > It would save me a lot of trouble if I could assume that EPOCH_LOCKED > > is an exotic feature which is unlikely to get used in FreeBSD. > > > > Another question, is it a common use case to call epoch_enter_preempt() > and epoch_exit_preempt() while owning a mutex? > Yes. Very. It is generally not permitted to hold a mutex across epoch_wait() that's why there's the special flag EPOCH_LOCKED. If you have a very limited number of threads, you might want to have each thread have its own record registered with the epoch. Then you wouldn't need the CPU pinning. The pinning is just away of providing a limited number of records to an unbounded number of threads. -M From owner-freebsd-hackers@freebsd.org Wed Aug 22 07:01:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6766108F081 for ; Wed, 22 Aug 2018 07:01:08 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3704B85CAD; Wed, 22 Aug 2018 07:01:08 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.2] (helo=sslproxy05.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsN8c-0005k6-8i; Wed, 22 Aug 2018 09:01:06 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsN8c-000DC3-0Q; Wed, 22 Aug 2018 09:01:06 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id A5F732A165C; Wed, 22 Aug 2018 09:01:07 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id LQ1PIPsURJE7; Wed, 22 Aug 2018 09:01:05 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 5608C2A167E; Wed, 22 Aug 2018 09:01:05 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 3G8LUxfj_9uF; Wed, 22 Aug 2018 09:01:05 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 214682A165C; Wed, 22 Aug 2018 09:01:05 +0200 (CEST) Subject: Re: epoch(9) background information? To: Matthew Macy Cc: freebsd-hackers@freebsd.org References: <15e3f080-2f82-a243-80e9-f0a916445828@embedded-brains.de> From: Sebastian Huber Message-ID: <26445c95-17c5-1a05-d290-0741d91b7721@embedded-brains.de> Date: Wed, 22 Aug 2018 09:01:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24863/Tue Aug 21 18:48:32 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 07:01:09 -0000 On 22/08/18 08:49, Matthew Macy wrote: > > > On Tue, Aug 21, 2018 at 11:42 PM Sebastian Huber=20 > > wrote: > > On 22/08/18 08:34, Sebastian Huber wrote: > > On 21/08/18 15:38, Jacques Fourie wrote: > >> > >> > >> On Tue, Aug 21, 2018 at 8:33 AM, Sebastian Huber > >> > >> >> wrote: > >> > >> =C2=A0=C2=A0=C2=A0 Hello, > >> > >> =C2=A0=C2=A0=C2=A0 I update currently a port of the FreeBSD netw= ork stack, etc. to > >> =C2=A0=C2=A0=C2=A0 the real-time operating system RTEMS from the= head version at > >> =C2=A0=C2=A0=C2=A0 2017-04-04 to the head version of today. I no= ticed that some > >> =C2=A0=C2=A0=C2=A0 read-write locks are replaced by a relatively= new stuff called > >> =C2=A0=C2=A0=C2=A0 EPOCH(9). Is there some background informatio= n available > for this? > >> =C2=A0=C2=A0=C2=A0 The man page is a bit vague and searching for= something named > >> =C2=A0=C2=A0=C2=A0 epoch on the internet is not really great. Fo= r example, what is > >> =C2=A0=C2=A0=C2=A0 the motivation for this change? How is this r= elated to > >> =C2=A0=C2=A0=C2=A0 read-copy-update (RCU)? > >> > >> =C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 Sebastian Huber, embedd= ed brains GmbH > >> > >> =C2=A0=C2=A0=C2=A0 Address : Dornierstr. 4, D-82178 Puchheim, Ge= rmany > >> > > >> =C2=A0=C2=A0=C2=A0 Phone=C2=A0 =C2=A0: +49 89 189 47 41-16 > >> =C2=A0=C2=A0=C2=A0 Fax=C2=A0 =C2=A0 =C2=A0: +49 89 189 47 41-09 > >> =C2=A0=C2=A0=C2=A0 E-Mail=C2=A0 : sebastian.huber@embedded-brain= s.de > > >> =C2=A0=C2=A0=C2=A0 > > >> =C2=A0=C2=A0=C2=A0 PGP=C2=A0 =C2=A0 =C2=A0: Public key available= on request. > >> > >> =C2=A0=C2=A0=C2=A0 Diese Nachricht ist keine gesch=C3=A4ftliche = Mitteilung im Sinne > des > >> EHUG. > >> > >> =C2=A0=C2=A0=C2=A0 _____________________________________________= __ > >> freebsd-hackers@freebsd.org > > > > >> =C2=A0=C2=A0=C2=A0 mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >> > >> =C2=A0=C2=A0=C2=A0 To unsubscribe, send any mail to > >> =C2=A0=C2=A0=C2=A0 "freebsd-hackers-unsubscribe@freebsd.org > > >> =C2=A0=C2=A0=C2=A0 >" > >> > >> > >> Additional information is available here : > >> http://concurrencykit.org/presentations/ebr.pdf > >> . The way I > >> understand it is that it is mostly used in place of read locks t= o > >> provide liveness guarantees without using atomics. Additional > detail > >> is available in the commit messages. As an example see r333813 f= or > >> some performance data. > >> > > > > Thanks, for the reference. The "epoch reclamation" are good > keywords > > to find more information. > > > > What is the right mailing list to ask questions about the epoch > > implementation of the FreeBSD kernel? > > > > To support this machinery in RTEMS is a bit difficult (in > particular > > EPOCH_LOCKED). Since RTEMS is supposed to be a real-time operatin= g > > system it supports only fixed-priority and job-level fixed priori= ty > > (EDF) schedulers. To allow some scaling to larger SMP systems it > > supports clustered scheduling together with the mutual exclusion > > locking protocols MrsP > > (http://www-users.cs.york.ac.uk/~burns/MRSPpaper.pdf > ) and OMIP > > (http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf > ). This > makes the > > thread pinning hard to implement (which is very easy to support i= n > > FreeBSD). The locking protocols may temporarily move a thread whi= ch > > owns a mutex to a foreign scheduler instance, e.g. a thread which > > wants to obtain the mutex helps the owner to make progress if it > was > > pre-empted in its home scheduler instance. Due to a timeout of th= e > > helper the owner may loose the right to execute in the foreign > > scheduler instance. This would make it impossible to fulfil the > > processor pinning constraint (e.g. the thread priority in the > foreign > > scheduler instance is undefined). > > > > It would save me a lot of trouble if I could assume that > EPOCH_LOCKED > > is an exotic feature which is unlikely to get used in FreeBSD. > > > > Another question, is it a common use case to call > epoch_enter_preempt() > and epoch_exit_preempt() while owning a mutex? > > > Yes. Very. It is generally not permitted to hold a mutex across=20 > epoch_wait() that's why there's the special flag EPOCH_LOCKED. If you=20 > have a very limited number of threads, you might want to have each=20 > thread have its own record registered with the epoch. Then you=20 > wouldn't need the CPU pinning. The pinning is just away of providing a=20 > limited number of records to an unbounded number of threads. Thanks for the prompt answer. Do I need a record per thread and per epoch? Could I use only one (maybe=20 dependent on the nest level?) record per thread? --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Wed Aug 22 07:09:52 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B122108F5EC for ; Wed, 22 Aug 2018 07:09:52 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E067B863FB for ; Wed, 22 Aug 2018 07:09:51 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f172.google.com (mail-io0-f172.google.com [209.85.223.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id A7DFC1211E for ; Wed, 22 Aug 2018 07:09:51 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f172.google.com with SMTP id q4-v6so696666iob.8 for ; Wed, 22 Aug 2018 00:09:51 -0700 (PDT) X-Gm-Message-State: AOUpUlFpihwV9awx1Oqt2bejv/ppikSUpfZwXImWRB9afNwG8Oce9niU 1fIYIDfxgWY4F2z2jn+2xnz1dr/Q2JbxIjnso00= X-Google-Smtp-Source: AA+uWPy8TY9Ry9AOiXaZVldL4JJ9uqpg/okqHV+3WwuNvr+hKZSE90M9j/+vOfg/nZw7rJlvamFhgudMWoNxM9AFh/Y= X-Received: by 2002:a6b:ed11:: with SMTP id n17-v6mr19952603iog.132.1534921791172; Wed, 22 Aug 2018 00:09:51 -0700 (PDT) MIME-Version: 1.0 References: <15e3f080-2f82-a243-80e9-f0a916445828@embedded-brains.de> <26445c95-17c5-1a05-d290-0741d91b7721@embedded-brains.de> In-Reply-To: <26445c95-17c5-1a05-d290-0741d91b7721@embedded-brains.de> From: Matthew Macy Date: Wed, 22 Aug 2018 00:09:39 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: epoch(9) background information? To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 07:09:52 -0000 > > > Yes. Very. It is generally not permitted to hold a mutex across > > epoch_wait() that's why there's the special flag EPOCH_LOCKED. If you > > have a very limited number of threads, you might want to have each > > thread have its own record registered with the epoch. Then you > > wouldn't need the CPU pinning. The pinning is just away of providing a > > limited number of records to an unbounded number of threads. > > Thanks for the prompt answer. > > Do I need a record per thread and per epoch? Could I use only one (maybe > dependent on the nest level?) record per thread? > > A record can only be registered with one epoch. And yes you can have just one single global epoch. However, then the epoch_wait_preempt time or time until the gc task is run is determined be the longest epoch section globally. It may help to look at the ck_epoch man pages and the implementation in ck https://www.mankier.com/3/ck_epoch_register https://github.com/concurrencykit/ck/blob/master/src/ck_epoch.c https://github.com/concurrencykit/ck/blob/master/include/ck_epoch.h > -- > > From owner-freebsd-hackers@freebsd.org Wed Aug 22 17:34:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02645108FF1E for ; Wed, 22 Aug 2018 17:34:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 58E3D804F7; Wed, 22 Aug 2018 17:34:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w7MHYdj2089123 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 22 Aug 2018 20:34:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w7MHYdj2089123 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w7MHYdPV089122; Wed, 22 Aug 2018 20:34:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 22 Aug 2018 20:34:39 +0300 From: Konstantin Belousov To: Matthew Macy Cc: sebastian.huber@embedded-brains.de, freebsd-hackers@freebsd.org Subject: Re: epoch(9) background information? Message-ID: <20180822173439.GA2340@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 17:34:57 -0000 On Tue, Aug 21, 2018 at 11:44:19PM -0700, Matthew Macy wrote: > EPOCH_LOCKED is something that one would only want to use in a fairly > narrow set of circumstances. The only place it's being discussed currently > is in pmap: > > https://reviews.freebsd.org/D15983 > > There it would conceivably replace a global mutex that currently serializes > all munmap operations. I have a scetchy design which keeps the current approach of waiting for delayed invalidation in the pmap_remove_all()/pmap_remove_write(), simultaneosly eliminating the global_invl_mtx mutex at all. There was some discussion about epoch vs. fine-graining of the existing DI code, relative to the moving the sleep time to pagedaemon threads. I did not have time to work it out (yet) due to personal issues. From owner-freebsd-hackers@freebsd.org Thu Aug 23 06:14:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D4A71082271 for ; Thu, 23 Aug 2018 06:14:36 +0000 (UTC) (envelope-from ish@amail.plala.or.jp) Received: from msc13.plala.or.jp (msc13.plala.or.jp [60.36.166.23]) by mx1.freebsd.org (Postfix) with ESMTP id 69B1C828AF for ; Thu, 23 Aug 2018 06:14:34 +0000 (UTC) (envelope-from ish@amail.plala.or.jp) Received: from localhost ([2400:4050:9320:7a00::8]) by msc13.plala.or.jp with ESMTP id <20180823061134.JKKB16836.msc13.plala.or.jp@localhost> for ; Thu, 23 Aug 2018 15:11:34 +0900 Date: Thu, 23 Aug 2018 15:11:29 +0900 (JST) Message-Id: <20180823.151129.548368076335195863.ish@amail.plala.or.jp> To: freebsd-hackers@freebsd.org Subject: Re: what is 'ef_read_entry failed' of freebsd-update From: Masachika ISHIZUKA In-Reply-To: <20180821.230745.1541080428454062053.ish@amail.plala.or.jp> References: <20180821.230745.1541080428454062053.ish@amail.plala.or.jp> X-Mailer: Mew version 6.8 on Emacs 26.1 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-VirusScan: Outbound; mvir-ac13; Thu, 23 Aug 2018 15:11:34 +0900 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 06:14:36 -0000 Eventually I can not even leave it, so this time I decided to reinstall buildkernel/buildworld from src. However, it will be troubled if it recurs again in the future. -- Masachika ISHIZUKA From owner-freebsd-hackers@freebsd.org Thu Aug 23 08:39:56 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42F38108601C for ; Thu, 23 Aug 2018 08:39:56 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CEDF487651 for ; Thu, 23 Aug 2018 08:39:55 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.3] (helo=sslproxy06.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsl9f-0003A4-DB for freebsd-hackers@freebsd.org; Thu, 23 Aug 2018 10:39:47 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy06.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsl9f-000KsL-6X for freebsd-hackers@freebsd.org; Thu, 23 Aug 2018 10:39:47 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 0F8DC2A165C for ; Thu, 23 Aug 2018 10:39:50 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id invQAVaKo4Md for ; Thu, 23 Aug 2018 10:39:49 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id AC8952A167F for ; Thu, 23 Aug 2018 10:39:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id V96JebRsKgIB for ; Thu, 23 Aug 2018 10:39:49 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 87D342A165C for ; Thu, 23 Aug 2018 10:39:49 +0200 (CEST) Subject: Re: epoch(9) background information? From: Sebastian Huber To: FreeBSD References: Message-ID: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> Date: Thu, 23 Aug 2018 10:39:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24865/Thu Aug 23 07:52:53 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 08:39:56 -0000 On 21/08/18 14:33, Sebastian Huber wrote: > Hello, > > I update currently a port of the FreeBSD network stack, etc. to the=20 > real-time operating system RTEMS from the head version at 2017-04-04=20 > to the head version of today. I noticed that some read-write locks are=20 > replaced by a relatively new stuff called EPOCH(9). Is there some=20 > background information available for this? The man page is a bit vague=20 > and searching for something named epoch on the internet is not really=20 > great. For example, what is the motivation for this change? How is=20 > this related to read-copy-update (RCU)? > We used the FreeBSD network stack also on low-end targets=20 (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON,=20 etc. in current production environments (not legacy systems). The=20 introduction of lock-free data structures (Concurrency Kit) and this=20 epoch memory reclamation makes little sense on these targets (at least=20 from my point of view). However, FreeBSD has still the SMP configuration=20 option (sys/conf/options) which suggests that SMP is optional. Is a=20 uni-processor system something which is considered by the FreeBSD=20 community as a thing worth supporting or can I expect that this is an=20 exotic environment which will get less and less well supported in the=20 future? I just need some guidance so that I can better plan for future=20 FreeBSD baseline updates. --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Thu Aug 23 10:27:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8B1A1089194 for ; Thu, 23 Aug 2018 10:27:38 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 79E148C174 for ; Thu, 23 Aug 2018 10:27:37 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 2862C2600C9; Thu, 23 Aug 2018 12:27:36 +0200 (CEST) Subject: Re: epoch(9) background information? To: Sebastian Huber , Eugene Grosbein , FreeBSD References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> <5B7E7804.4030907@grosbein.net> <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> From: Hans Petter Selasky Message-ID: <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> Date: Thu, 23 Aug 2018 12:27:12 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 10:27:39 -0000 On 8/23/18 11:28 AM, Sebastian Huber wrote: > On 23/08/18 11:01, Eugene Grosbein wrote: >> On 23.08.2018 15:39, Sebastian Huber wrote: >> >>> We used the FreeBSD network stack also on low-end targets >>> (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, >>> etc. in current production environments (not legacy systems). The >>> introduction of lock-free data structures (Concurrency Kit) and this >>> epoch memory reclamation makes little sense on these targets (at least >>> from my point of view). However, FreeBSD has still the SMP configuration >>> option (sys/conf/options) which suggests that SMP is optional. Is a >>> uni-processor system something which is considered by the FreeBSD >>> community as a thing worth supporting or can I expect that this is an >>> exotic environment which will get less and less well supported in the >>> future? I just need some guidance so that I can better plan for future >>> FreeBSD baseline updates. >> FreeBSD as virtualized uniprocessor guest should be supported at full >> scale, >> as well as embedded applications using single core x86 and non-x86 CPUs. > > If something should be supported, then there must be also someone who > ensures that this is actually the case. I don't know the FreeBSD > community good enough to judge if there is sufficient > manpower/funding/interest for a well supported uni-processor FreeBSD. > From the commits it is clear that FreeBSD receives a lot of attention > from CDN providers such as Netflix and Limelight Networks. They probably > don't care about uni-processor system support at all. The use of > lock-free data structures (Concurrency Kit) and the epoch memory > reclamation are now a mandatory infrastructure. There is no FreeBSD > configuration option to avoid this. > > The Concurrency Kit in sys/contrib/ck has no explicit support for the > FreeBSD RISC-V and MIPS architectures. So, I guess the fall-back > sys/contrib/ck/include/gcc/ck_pr.h is used. The atomic support in > sys/contrib/ck partially duplicates/extends the general atomic support > of the FreeBSD kernel ATOMIC(9). To me it is a bit unclear what will be > the future direction in the FreeBSD kernel with respect to lock-free > data structures. > Hi Sebastian, Do you have something like critical_enter() to disable pre-emption in your OS? If you don't need to support SMP, the CPU pinning in the EPOCH can be replaced by a critial_enter() / critial_exit() pair. --HPS From owner-freebsd-hackers@freebsd.org Thu Aug 23 09:02:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DA541086BDF for ; Thu, 23 Aug 2018 09:02:25 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BA81988FC3 for ; Thu, 23 Aug 2018 09:02:24 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w7N920GX086159 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 23 Aug 2018 11:02:01 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: sebastian.huber@embedded-brains.de Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id w7N91uGg017091; Thu, 23 Aug 2018 16:01:56 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: epoch(9) background information? To: Sebastian Huber , FreeBSD References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> From: Eugene Grosbein Message-ID: <5B7E7804.4030907@grosbein.net> Date: Thu, 23 Aug 2018 16:01:56 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, LOCAL_FROM, SPF_PASS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 09:02:25 -0000 On 23.08.2018 15:39, Sebastian Huber wrote: > We used the FreeBSD network stack also on low-end targets > (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, > etc. in current production environments (not legacy systems). The > introduction of lock-free data structures (Concurrency Kit) and this > epoch memory reclamation makes little sense on these targets (at least > from my point of view). However, FreeBSD has still the SMP configuration > option (sys/conf/options) which suggests that SMP is optional. Is a > uni-processor system something which is considered by the FreeBSD > community as a thing worth supporting or can I expect that this is an > exotic environment which will get less and less well supported in the > future? I just need some guidance so that I can better plan for future > FreeBSD baseline updates. FreeBSD as virtualized uniprocessor guest should be supported at full scale, as well as embedded applications using single core x86 and non-x86 CPUs. Just my 2 cents. From owner-freebsd-hackers@freebsd.org Thu Aug 23 11:25:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CA95108A850 for ; Thu, 23 Aug 2018 11:25:42 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AAA168DEEF for ; Thu, 23 Aug 2018 11:25:41 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.3] (helo=sslproxy06.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsnkC-0008SK-7A for freebsd-hackers@freebsd.org; Thu, 23 Aug 2018 13:25:40 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy06.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsnkC-000FFz-2n for freebsd-hackers@freebsd.org; Thu, 23 Aug 2018 13:25:40 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 122782A165C for ; Thu, 23 Aug 2018 13:25:43 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id B5UddtjW0gSG for ; Thu, 23 Aug 2018 13:25:42 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id C32462A167F for ; Thu, 23 Aug 2018 13:25:42 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id a6XTitnRYt1c for ; Thu, 23 Aug 2018 13:25:42 +0200 (CEST) Received: from linux-diu0.suse (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTP id 9F9912A165C for ; Thu, 23 Aug 2018 13:25:42 +0200 (CEST) From: Sebastian Huber To: freebsd-hackers@freebsd.org Subject: [PATCH] Remove stray #ifdef _KERNEL in Date: Thu, 23 Aug 2018 13:25:39 +0200 Message-Id: <20180823112539.20725-1-sebastian.huber@embedded-brains.de> X-Mailer: git-send-email 2.13.7 X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24865/Thu Aug 23 07:52:53 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 11:25:42 -0000 The malloc() macro definition is aready in an #ifdef _KERNEL section. --- sys/sys/malloc.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/sys/sys/malloc.h b/sys/sys/malloc.h index 06a88822ed5..68595091a58 100644 --- a/sys/sys/malloc.h +++ b/sys/sys/malloc.h @@ -215,7 +215,6 @@ void *malloc(size_t size, struct malloc_type *type, int flags) __malloc_like * an inline function variant ended up being compiled to a mere malloc call * regardless of argument. gcc generates expected code (like the above). */ -#ifdef _KERNEL #define malloc(size, type, flags) ({ \ void *_malloc_item; \ size_t _size = (size); \ @@ -230,7 +229,6 @@ void *malloc(size_t size, struct malloc_type *type, int flags) __malloc_like } \ _malloc_item; \ }) -#endif void *malloc_domain(size_t size, struct malloc_type *type, int domain, int flags) __malloc_like __result_use_check __alloc_size(1); -- 2.13.7 From owner-freebsd-hackers@freebsd.org Thu Aug 23 11:22:50 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90E12108A7BF; Thu, 23 Aug 2018 11:22:50 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F0ABC8DE75; Thu, 23 Aug 2018 11:22:49 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x435.google.com with SMTP id a108-v6so4297687wrc.13; Thu, 23 Aug 2018 04:22:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=K0CCUDKUjPX78vxmx2ZMfgDUXgzP82QYi4td2QI4D3s=; b=QT/zidTcJB0xdkH5S67hk1RFPAOAkMdFubOMHC2Rc/JcecP/QTyYrKCi7BasiFAvZ2 M5vOcXSTdYyIqkr5qL+aiyv/q3iuVkwPJW67rhuAjWGtVjBpzXHQH9UGmbw1xKRFR3s3 sn1Zja2JdEn3vJRSpT8zf5WWhKC/hapLy9c7Js5PX3OsFcam8ed1S2c9wg+gjdCYU2lE gW1N+x+NsD/ZZjKeRuKapOEjzbH1+gcxnUu0FLRLRMtgwlj2l8VJTX9X9xf1i9zjYYCi 7/Gdhga3n6drU2KYYr36x+jkkNi2UzE2b+Ywk0SAhnpNUxlOFVP2Yox3G0/G9tRXMUUB skrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=K0CCUDKUjPX78vxmx2ZMfgDUXgzP82QYi4td2QI4D3s=; b=L6LxgZuUTWHaRnk+Zd/rsEFhJQskapHBPNgczXNMTWUXU2lqUKTYIS2CkyQ/UiwB+G BNMhjUkTgXfLJg+tbeQsknwi3jLBtbR56Y1zBkzGkgPX/4/DQAvrGKH+N2ie9p+8G9Ft gPx7Y7bcQZE+YH4CIaMkYuq0Gs5I0N98wFCmz9uYytmkqQgDdbfbF+TW4zonwHeKJYxa J27mNQ2xfFZKmJ0PrfPT8qpEfVvk8s190Zmm7nQGXHbHqv6md9Ee2SsGVuUUjw+jeZF9 pYcXNBKN8jkF9l3HGDLzkfw9ZnoMGUi2TYS5IWButBMrpbq6MSWQ2dJpsrpLdnA+87Fb xfbw== X-Gm-Message-State: APzg51BBbU/hdEzvMnj3qbw6LGx0nlevO6lFAIjQChOcJ0zwtls8eazL uWMvhcypbJftv4vPxopiqdKEfh/mXPPVoHMlb9TTDorx X-Google-Smtp-Source: ANB0VdZdfDnGzY5OnpNKqzEs69ryYd8Sg3TenDSPxJebFWJwE/8q1ijU/tvNuV3rfrL92x5OIIYYwszDpH3tSYWtZ/w= X-Received: by 2002:adf:806d:: with SMTP id 100-v6mr14504681wrk.23.1535023368497; Thu, 23 Aug 2018 04:22:48 -0700 (PDT) MIME-Version: 1.0 References: <1534523216.27158.17.camel@freebsd.org> <1534702861.27158.36.camel@freebsd.org> <1534771095.27158.46.camel@freebsd.org> <35F2C250-B4CB-4C53-BF8F-43C338022E34@yahoo.com> <20180820181322.71607854@ernst.home> <1534861980.27158.145.camel@freebsd.org> <20180821171626.49951728@ernst.home> In-Reply-To: <20180821171626.49951728@ernst.home> From: Rajesh Kumar Date: Thu, 23 Aug 2018 16:52:36 +0530 Message-ID: Subject: Re: Need a clarification regarding I2C bus frequency in FreeBSD To: gljennjohn@gmail.com Cc: ian@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 11:22:50 -0000 Hi Ian/Gary, Thanks for such a detailed explanation. It helps me understand some basics. > Right now the driver lacks *any* support for changing bus speeds. That > will be easy to fix, once we figure out: > > __1. What is in the IG4_REG_CLK_PARMS register? > __2. What do we do about versions of the hardware that don't support > that register?__ I tested with my hardware and it seems it doesn't have the IG4_REG_CLK_PARMS registers. When I try to read that offset, it's returning 0x63636363 (looks like some junk value?) Also, I don't see any register which specifies the base clock frequency directly, so that we can change the HCNT/LCNT as needed (which I belive you mean by changing the bus frequency here) > It would also be interesting to just print out those values. In the > driver on lines 571-575 the code reads all those registers and does > nothing with them. If you look in the dragonflybsd version of the > driver it printed out all those values after reading them; whoever > imported the driver to freebsd just deleted all the kprintf() lines and > left the register reads. I dumped few registers (during ig4iic_attach) and below are the details. Looks like in my case SS HCNT/LCNT default is 645/855 ( total - 1500). So, the base frequency seems to be configured to 150Mhz by default. Also, FS HCNT/LCNT default is 0x87/0xF0 (total - 375), which also proves the base frequency to be 150Mhz. So, my base requirement is satisfied. But, as I mentioned earlier, there is no direct way to get (or) configure the base frequency from the driver. IG4_REG_CTL 00000063 IG4_REG_SS_SCL_HCNT 00000285 IG4_REG_SS_SCL_LCNT 00000357 IG4_REG_FS_SCL_HCNT 00000087 IG4_REG_FS_SCL_LCNT 000000f0 IG4_REG_INTR_STAT 00000000 IG4_REG_INTR_MASK 00000000 IG4_REG_RAW_INTR_STAT 000000 IG4_REG_CLK_PARMS 63636363 IG4_REG_GENERAL 55555555 > According to Table 10 in Chapter 6 of the I2C standard from NXP, > SCL low must satisfy a minimum of 4.7 uS and SCL high must > satisfy a minimum of 4.0 uS when running SCL at 100KHz. At > 400KHz the values are respectively 1.3uS and 0.6uS. The above HCNT/LCNT numbers seems to match the minium requirements for SCL both in SS and FS case. Seems, this driver is written initially for controllers in Intel SoC's. Now, as we need this driver for other SoC's like in my case, can we generalize this driver by removing unnecessary checks for the "version" to be ATOM/HASWELL/SKYWELL/APL etc., If there is no concern regarding this, I will try to make the necessary changes. On Tue, Aug 21, 2018 at 8:46 PM Gary Jennejohn wrote: > On Tue, 21 Aug 2018 08:33:00 -0600 > Ian Lepore wrote: > > > On Mon, 2018-08-20 at 23:25 +0530, Rajesh Kumar wrote: > > > Hi, > > > > > > Re-posting the questions, just in case if its missed in other > conversation. > > > > > > By "i2c clock frequency", I mean the internal base frequency only, > which > > > drives the chip.____I thought data will be transferred on bus based on > the > > > base frequency. So, thought both bus and base frequency are same. But > from > > > what you said, seems both are different. So, based on the setting in > > > *_HCNT/LCNT register, the bus frequency (which is the rate at which > data is > > > transferred) will change for a particular base frequency. Is that > right? > > > > > > So, few questions here > > > > > > 1)____As you said, we need to have a base frequency of 150 Mhz in our > case. > > > So, do we need to program that IG4_REG_CLK_PARMS to 150 Mhz > (0x8F0D180)? > > > And can this be done at the same time when programming the HCNT/LNCT > > > registers? > > > > I don't have this hardware, and I don't have a datasheet that describes > > the__IG4_REG_CLK_PARMS register mentioned in the driver, so I don't > > really have an answer. I suspect the hardware should set that register > > with information that lets the driver know what the base clock speed > > is. Using that information, the driver could calculate the proper > > values for HCNT/LCNT. > > > > Right now the driver lacks *any* support for changing bus speeds. That > > will be easy to fix, once we figure out: > > > > __1. What is in the IG4_REG_CLK_PARMS register? > > __2. What do we do about versions of the hardware that don't support > > that register?__ > > > > > 2)____Not sure how that 111Hz value is arrived.____Can you please > explain this > > > calculation. So, that I can derive the appropriate values for > HCNT/LCNT for > > > different speeds at 150Mhz base clock. > > > > It's based on the comment (which I feel certain must be wrong) that the > > base clock is 25,000 Hz. With HCNT,LCNT set to 100,125, one cycle of > > the SCL line will last 225 base clock cycles. 1/25000 = 0.000040, that > > times 225 is 0.009 seconds per SCL cycle. 1/.009 = 111.111 Hz. > > > > FWIW, an i2c bus will run fine at 111Hz, it'll just take forever to get > > anything done. But I don't think the bus is really running at 111Hz, > > because I don't think the base clock is really 25KHz, I think the > > comment block is just wrong about all of that. > > > > > 3)____"Default HCNT/LCNT register values would be consistent with an > internal > > > base clock speed of 1GHz",____Does it mean with those values, all > speeds can > > > be achieved until 1GHz clock? > > > > > > > Well, the defaults I mentioned are from the datasheet cited in the > > driver code. Those defaults made me think the base clock was 1GHz on > > that particular hardware. I just realized that I was off by an order of > > magnitude, I think I was mixing numbers from the driver and numbers > > from the datasheet in my head. __The default HCNT,LCNT in that datasheet > > are 612+706=1318 base cycles to give an SCL rate of 100KHz. So > > 1318*100000 is 131.8MHz, a very reasonable number. > > > > In your world you want the 150MHz base clock to generate the 100Hz SCL, > > so 150000000/100000 = 1500. My inclination would be to split that in > > half and have HCNT,LCNT be 750,750, but for some reason there seems to > > be a bias on this hardware for having HCNT be slightly longer than > > LCNT, so maybe 775,725. Maybe that's an attempt to compensate for the > > fact that high levels on the clock line are accomplished with a pullup > > resistor, and it takes slightly longer for the line to "drift up" to a > > high state via the pullup, compared to being driven low which would > > happen quickly. > > > > Just a remark. > > According to Table 10 in Chapter 6 of the I2C standard from NXP, > SCL low must satisfy a minimum of 4.7 uS and SCL high must > satisfy a minimum of 4.0 uS when running SCL at 100KHz. At > 400KHz the values are respectively 1.3uS and 0.6uS. > > 725 results in 4.83uS, which would be OK for the low phase if it > gets pulled low quite quickly. 775 results in 5.17uS, which is > also acceptable, especially if it takes rather long for it to be > pulled up. > > 750 results in 5.0uS for each, of course. This may theoreticalluy > be a safer value, but it doesn't reflect any odd behavior which the > real hardware may exhibit. > > > I would expect the default values of the HCNT/LCNT registers would be > > right for any given hardware... whoever configures the IP block in a > > SoC would configure the base clock value and the default HCNT/LCNT > > values to match each other. Putting it that way makes me think that > > maybe the right thing to do in the driver is just stop setting > > HCNT/LCNT at all, and rely on the hardware to be configured correctly > > by default. It's worth a try. > > > > It would also be interesting to just print out those values. In the > > driver on lines 571-575 the code reads all those registers and does > > nothing with them. If you look in the dragonflybsd version of the > > driver it printed out all those values after reading them; whoever > > imported the driver to freebsd just deleted all the kprintf() lines and > > left the register reads. > > > > > -- > Gary Jennejohn > From owner-freebsd-hackers@freebsd.org Thu Aug 23 11:00:05 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C10F108A018 for ; Thu, 23 Aug 2018 11:00:05 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8134C8D357 for ; Thu, 23 Aug 2018 11:00:04 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.3] (helo=sslproxy06.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fsnLM-0007jA-ID; Thu, 23 Aug 2018 13:00:00 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy06.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fsnLM-000BVB-Av; Thu, 23 Aug 2018 13:00:00 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 4F3392A165C; Thu, 23 Aug 2018 13:00:03 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id W_VM5HCn3E8f; Thu, 23 Aug 2018 13:00:02 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id DBC312A167F; Thu, 23 Aug 2018 13:00:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id VqEnMbN2cYnY; Thu, 23 Aug 2018 13:00:02 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id B24C42A165C; Thu, 23 Aug 2018 13:00:02 +0200 (CEST) Subject: Re: epoch(9) background information? To: Hans Petter Selasky , Eugene Grosbein , FreeBSD References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> <5B7E7804.4030907@grosbein.net> <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> From: Sebastian Huber Message-ID: <68c80dd6-a811-06c1-27d9-25e99e9fd4ed@embedded-brains.de> Date: Thu, 23 Aug 2018 12:59:58 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24865/Thu Aug 23 07:52:53 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 11:00:05 -0000 On 23/08/18 12:27, Hans Petter Selasky wrote: > On 8/23/18 11:28 AM, Sebastian Huber wrote: >> On 23/08/18 11:01, Eugene Grosbein wrote: >>> On 23.08.2018 15:39, Sebastian Huber wrote: >>> >>>> We used the FreeBSD network stack also on low-end targets >>>> (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, >>>> etc. in current production environments (not legacy systems). The >>>> introduction of lock-free data structures (Concurrency Kit) and this >>>> epoch memory reclamation makes little sense on these targets (at lea= st >>>> from my point of view). However, FreeBSD has still the SMP=20 >>>> configuration >>>> option (sys/conf/options) which suggests that SMP is optional. Is a >>>> uni-processor system something which is considered by the FreeBSD >>>> community as a thing worth supporting or can I expect that this is a= n >>>> exotic environment which will get less and less well supported in th= e >>>> future? I just need some guidance so that I can better plan for futu= re >>>> FreeBSD baseline updates. >>> FreeBSD as virtualized uniprocessor guest should be supported at=20 >>> full scale, >>> as well as embedded applications using single core x86 and non-x86=20 >>> CPUs. >> >> If something should be supported, then there must be also someone who=20 >> ensures that this is actually the case. I don't know the FreeBSD=20 >> community good enough to judge if there is sufficient=20 >> manpower/funding/interest for a well supported uni-processor FreeBSD.=20 >> =C2=A0From the commits it is clear that FreeBSD receives a lot of=20 >> attention from CDN providers such as Netflix and Limelight Networks.=20 >> They probably don't care about uni-processor system support at all.=20 >> The use of lock-free data structures (Concurrency Kit) and the epoch=20 >> memory reclamation are now a mandatory infrastructure. There is no=20 >> FreeBSD configuration option to avoid this. >> >> The Concurrency Kit in sys/contrib/ck has no explicit support for the=20 >> FreeBSD RISC-V and MIPS architectures. So, I guess the fall-back=20 >> sys/contrib/ck/include/gcc/ck_pr.h is used. The atomic support in=20 >> sys/contrib/ck partially duplicates/extends the general atomic=20 >> support of the FreeBSD kernel ATOMIC(9). To me it is a bit unclear=20 >> what will be the future direction in the FreeBSD kernel with respect=20 >> to lock-free data structures. >> > > Hi Sebastian, > > Do you have something like critical_enter() to disable pre-emption in=20 > your OS? If you don't need to support SMP, the CPU pinning in the=20 > EPOCH can be replaced by a critial_enter() / critial_exit() pair. Yes, RTEMS has a critical_enter() to disable pre-emption (you could also=20 disable interrupts as a brute force means). You still have the lock-free data structure inside the critical section.=20 Currently, this is only ck_queue, so not a real problem. However, in=20 case some more advanced lock-less algorithms would appear with a bit of=20 spinning here and there, then this would need further adoptions for the=20 uni-processor system. Not all targets support C11 atomics in hardware,=20 some need libatomic (a GCC thing). --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Thu Aug 23 09:28:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7FE1108763A for ; Thu, 23 Aug 2018 09:28:29 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 80CB889EC4 for ; Thu, 23 Aug 2018 09:28:29 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.3] (helo=sslproxy06.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1fslum-0004bs-1w; Thu, 23 Aug 2018 11:28:28 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy06.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fslul-0000fz-Ru; Thu, 23 Aug 2018 11:28:27 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id B0E312A165C; Thu, 23 Aug 2018 11:28:30 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 7LQbvG8gc5gt; Thu, 23 Aug 2018 11:28:29 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 59FA32A167F; Thu, 23 Aug 2018 11:28:29 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id xo8OZYaN73BW; Thu, 23 Aug 2018 11:28:29 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 26E222A165C; Thu, 23 Aug 2018 11:28:29 +0200 (CEST) Subject: Re: epoch(9) background information? To: Eugene Grosbein , FreeBSD References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> <5B7E7804.4030907@grosbein.net> From: Sebastian Huber Message-ID: <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> Date: Thu, 23 Aug 2018 11:28:25 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <5B7E7804.4030907@grosbein.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24865/Thu Aug 23 07:52:53 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 09:28:30 -0000 On 23/08/18 11:01, Eugene Grosbein wrote: > On 23.08.2018 15:39, Sebastian Huber wrote: > >> We used the FreeBSD network stack also on low-end targets >> (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, >> etc. in current production environments (not legacy systems). The >> introduction of lock-free data structures (Concurrency Kit) and this >> epoch memory reclamation makes little sense on these targets (at least >> from my point of view). However, FreeBSD has still the SMP configurati= on >> option (sys/conf/options) which suggests that SMP is optional. Is a >> uni-processor system something which is considered by the FreeBSD >> community as a thing worth supporting or can I expect that this is an >> exotic environment which will get less and less well supported in the >> future? I just need some guidance so that I can better plan for future >> FreeBSD baseline updates. > FreeBSD as virtualized uniprocessor guest should be supported at full s= cale, > as well as embedded applications using single core x86 and non-x86 CPUs= . If something should be supported, then there must be also someone who=20 ensures that this is actually the case. I don't know the FreeBSD=20 community good enough to judge if there is sufficient=20 manpower/funding/interest for a well supported uni-processor FreeBSD.=20 From the commits it is clear that FreeBSD receives a lot of attention=20 from CDN providers such as Netflix and Limelight Networks. They probably=20 don't care about uni-processor system support at all. The use of=20 lock-free data structures (Concurrency Kit) and the epoch memory=20 reclamation are now a mandatory infrastructure. There is no FreeBSD=20 configuration option to avoid this. The Concurrency Kit in sys/contrib/ck has no explicit support for the=20 FreeBSD RISC-V and MIPS architectures. So, I guess the fall-back=20 sys/contrib/ck/include/gcc/ck_pr.h is used. The atomic support in=20 sys/contrib/ck partially duplicates/extends the general atomic support=20 of the FreeBSD kernel ATOMIC(9). To me it is a bit unclear what will be=20 the future direction in the FreeBSD kernel with respect to lock-free=20 data structures. --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Thu Aug 23 12:58:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 447DF108D241 for ; Thu, 23 Aug 2018 12:58:24 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2A717113F for ; Thu, 23 Aug 2018 12:58:23 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 34e82e90-a6d4-11e8-93fa-f3ebd9db2b94 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id 34e82e90-a6d4-11e8-93fa-f3ebd9db2b94; Thu, 23 Aug 2018 12:58:21 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7NCwHg3097758; Thu, 23 Aug 2018 06:58:17 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1535029097.27158.180.camel@freebsd.org> Subject: Re: epoch(9) background information? From: Ian Lepore To: Sebastian Huber , Hans Petter Selasky , Eugene Grosbein , FreeBSD Date: Thu, 23 Aug 2018 06:58:17 -0600 In-Reply-To: <68c80dd6-a811-06c1-27d9-25e99e9fd4ed@embedded-brains.de> References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> <5B7E7804.4030907@grosbein.net> <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> <68c80dd6-a811-06c1-27d9-25e99e9fd4ed@embedded-brains.de> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 12:58:24 -0000 On Thu, 2018-08-23 at 12:59 +0200, Sebastian Huber wrote: > On 23/08/18 12:27, Hans Petter Selasky wrote: > > > > On 8/23/18 11:28 AM, Sebastian Huber wrote: > > > > > > On 23/08/18 11:01, Eugene Grosbein wrote: > > > > > > > > On 23.08.2018 15:39, Sebastian Huber wrote: > > > > > > > > > > > > > > We used the FreeBSD network stack also on low-end targets > > > > > (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, > > > > > etc. in current production environments (not legacy systems). The > > > > > introduction of lock-free data structures (Concurrency Kit) and this > > > > > epoch memory reclamation makes little sense on these targets (at least > > > > > from my point of view). However, FreeBSD has still the SMP  > > > > > configuration > > > > > option (sys/conf/options) which suggests that SMP is optional. Is a > > > > > uni-processor system something which is considered by the FreeBSD > > > > > community as a thing worth supporting or can I expect that this is an > > > > > exotic environment which will get less and less well supported in the > > > > > future? I just need some guidance so that I can better plan for future > > > > > FreeBSD baseline updates. > > > > FreeBSD as virtualized uniprocessor guest should be supported at  > > > > full scale, > > > > as well as embedded applications using single core x86 and non-x86  > > > > CPUs. > > > If something should be supported, then there must be also someone who  > > > ensures that this is actually the case. I don't know the FreeBSD  > > > community good enough to judge if there is sufficient  > > > manpower/funding/interest for a well supported uni-processor FreeBSD.  > > >  From the commits it is clear that FreeBSD receives a lot of  > > > attention from CDN providers such as Netflix and Limelight Networks.  > > > They probably don't care about uni-processor system support at all.  > > > The use of lock-free data structures (Concurrency Kit) and the epoch  > > > memory reclamation are now a mandatory infrastructure. There is no  > > > FreeBSD configuration option to avoid this. > > > > > > The Concurrency Kit in sys/contrib/ck has no explicit support for the  > > > FreeBSD RISC-V and MIPS architectures. So, I guess the fall-back  > > > sys/contrib/ck/include/gcc/ck_pr.h is used. The atomic support in  > > > sys/contrib/ck partially duplicates/extends the general atomic  > > > support of the FreeBSD kernel ATOMIC(9). To me it is a bit unclear  > > > what will be the future direction in the FreeBSD kernel with respect  > > > to lock-free data structures. > > > > > Hi Sebastian, > > > > Do you have something like critical_enter() to disable pre-emption in  > > your OS? If you don't need to support SMP, the CPU pinning in the  > > EPOCH can be replaced by a critial_enter() / critial_exit() pair. > Yes, RTEMS has a critical_enter() to disable pre-emption (you could also  > disable interrupts as a brute force means). > > You still have the lock-free data structure inside the critical section.  > Currently, this is only ck_queue, so not a real problem. However, in  > case some more advanced lock-less algorithms would appear with a bit of  > spinning here and there, then this would need further adoptions for the  > uni-processor system. Not all targets support C11 atomics in hardware,  > some need libatomic (a GCC thing). > Freebsd runs on many single-processor systems on all supported architectures and that isn't going to change. -- Ian From owner-freebsd-hackers@freebsd.org Thu Aug 23 14:37:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52A3C108F5A6 for ; Thu, 23 Aug 2018 14:37:30 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5C4374D05 for ; Thu, 23 Aug 2018 14:37:29 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf1-x436.google.com with SMTP id k19-v6so2820311pfi.1 for ; Thu, 23 Aug 2018 07:37:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=/M0M/E9cFsyciIQnXCjdDoAFRWV3IuI3A2Cotbg5uHI=; b=iNrqzNgqupFmj3vtVoTTOU6Q+kof3F92DI75zYCqmYgfFR3JXjgOQvmwRLO6SvD7fS /BTZw8gDpgqA/HdvusqQ3y+CA11KDMGyk1Q9Z//OrGlSMN7CHt6KqAOyOvMVv+3BBBft ZmAd1wdYpSp7i+pYVutBjLodRn/+hDUjEXNIrxjHbZwR7nbmuKl9aPhJOCBIKbKm5u3D vmeXByclSAwJ2EbwM5UFt27dIq6Tgsiav6Ff2Ulj5wAsHCGPhDwkxqdzWKSwzP0f+QDa +GVtd9yMKbgnKCnUfAtyJJzLrXb5FM231FvBxwHHVPi1EPh7eWNDXHepxzmx0O4ZguXq 0mKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=/M0M/E9cFsyciIQnXCjdDoAFRWV3IuI3A2Cotbg5uHI=; b=j0h2cZ331Fttg38UTEiDaOEeP5GrGEZvTnN+siiJImFw9qf6ws8Fz+8nDF2b40RgLu j8qwfDySE0d1jaY1MOlbosluhaTD8Qge11B5EfByjbNNTX4C/GJFToN1Q9b7Gas4uDKN 0wfK1ERZndTWmwPSmc+YBVrtOkyLPkKYUXVhI0ZLyKwZbYyb2WjundPN9oxDxx/a7TsI endClu0csDou+ghl8pfdZNwN8KrhQWVKhBRfR4BLeSed9H5+yxVdJGdgKAeHo/yH0/A7 2AOdbgbor79UADjKooxtYYcVXjgX8DBY0QIt/KuIjfZfQsrcYR/Jp/JcMLAdJUjHZ1cs 4VfA== X-Gm-Message-State: AOUpUlGdZO/sh9gT7Iz8OfUEQIja5p4ap5qit7y76+75o2jp4oS5ty7D PNoh9bZbsXau7G0gQqYeG3O+6Hek/FyWbg== X-Google-Smtp-Source: AA+uWPznugaBCmOv3r+c9g6KLNa0E/T5PnC+cv8qH+PkQGcxqA1AyVrxl8nmQB8OskTTuiU9OWWHKg== X-Received: by 2002:a63:a053:: with SMTP id u19-v6mr28885573pgn.394.1535035048245; Thu, 23 Aug 2018 07:37:28 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id o3-v6sm5099262pgv.53.2018.08.23.07.37.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Aug 2018 07:37:27 -0700 (PDT) Sender: Mark Johnston Date: Thu, 23 Aug 2018 10:37:24 -0400 From: Mark Johnston To: Hans Petter Selasky Cc: Sebastian Huber , Eugene Grosbein , FreeBSD Subject: Re: epoch(9) background information? Message-ID: <20180823143724.GB68810@raichu> References: <3bfedcc3-0dae-7979-2bd4-da83f2c67e87@embedded-brains.de> <5B7E7804.4030907@grosbein.net> <978ae736-89b9-6d83-e2a1-d2834ca8ae55@embedded-brains.de> <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <90e16238-6e4d-5d3d-499d-2a19a49be78c@selasky.org> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 14:37:30 -0000 On Thu, Aug 23, 2018 at 12:27:12PM +0200, Hans Petter Selasky wrote: > On 8/23/18 11:28 AM, Sebastian Huber wrote: > > On 23/08/18 11:01, Eugene Grosbein wrote: > >> On 23.08.2018 15:39, Sebastian Huber wrote: > >> > >>> We used the FreeBSD network stack also on low-end targets > >>> (uni-processor) such as MCF548x ColdFire, Atmel SAM V71, SPARC LEON, > >>> etc. in current production environments (not legacy systems). The > >>> introduction of lock-free data structures (Concurrency Kit) and this > >>> epoch memory reclamation makes little sense on these targets (at least > >>> from my point of view). However, FreeBSD has still the SMP configuration > >>> option (sys/conf/options) which suggests that SMP is optional. Is a > >>> uni-processor system something which is considered by the FreeBSD > >>> community as a thing worth supporting or can I expect that this is an > >>> exotic environment which will get less and less well supported in the > >>> future? I just need some guidance so that I can better plan for future > >>> FreeBSD baseline updates. > >> FreeBSD as virtualized uniprocessor guest should be supported at full > >> scale, > >> as well as embedded applications using single core x86 and non-x86 CPUs. > > > > If something should be supported, then there must be also someone who > > ensures that this is actually the case. I don't know the FreeBSD > > community good enough to judge if there is sufficient > > manpower/funding/interest for a well supported uni-processor FreeBSD. > > From the commits it is clear that FreeBSD receives a lot of attention > > from CDN providers such as Netflix and Limelight Networks. They probably > > don't care about uni-processor system support at all. The use of > > lock-free data structures (Concurrency Kit) and the epoch memory > > reclamation are now a mandatory infrastructure. There is no FreeBSD > > configuration option to avoid this. > > > > The Concurrency Kit in sys/contrib/ck has no explicit support for the > > FreeBSD RISC-V and MIPS architectures. So, I guess the fall-back > > sys/contrib/ck/include/gcc/ck_pr.h is used. The atomic support in > > sys/contrib/ck partially duplicates/extends the general atomic support > > of the FreeBSD kernel ATOMIC(9). To me it is a bit unclear what will be > > the future direction in the FreeBSD kernel with respect to lock-free > > data structures. > > > > Hi Sebastian, > > Do you have something like critical_enter() to disable pre-emption in > your OS? If you don't need to support SMP, the CPU pinning in the EPOCH > can be replaced by a critial_enter() / critial_exit() pair. Threads in preemptible epoch sections are allowed to acquire locks, so critical_enter()/exit() isn't a suitable replacement for epoch sections. I think a fallback implementation could be written without CK for the !SMP case, but it would require some work. From owner-freebsd-hackers@freebsd.org Thu Aug 23 15:02:28 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B6C4108FD0A for ; Thu, 23 Aug 2018 15:02:28 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F46B75AF4 for ; Thu, 23 Aug 2018 15:02:28 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg1-x542.google.com with SMTP id h8-v6so2710839pgs.4 for ; Thu, 23 Aug 2018 08:02:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=13b/xUBZgZFhuZMTa9KsA+6FtHOkMRDayg8JHouSliI=; b=Z8zgVSb+Em1oswOyQN2gn63JOD41KmYY9q5+vGT6KVuZg4f8Fz+V3Z38XseZtiRfN/ u0+t9iyylhn+gGbSm/TuFmcUABxwFb7s+dZ1yy0IgFRRYkcA43At8iyCfRoDpaBCV2Wh +5mdXvMm2AbkUEKyySjscMcbYO+5RxzscrfvR2t4JCHS9uTuZucFVGXvMX3uKLuYpxy2 epkuLPp89SN5Jh0ZdNjyvdj1G0oNLgRGkCR8yglank2/A65SJwx2NnOSN8cY6rdnDFzF vvSC3csCIJCf7EllhHWVE2dTAEFBtoeQmytYH9m/4yOTcTpKY/23/DyHKRQhZ1v13l9e sJ+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=13b/xUBZgZFhuZMTa9KsA+6FtHOkMRDayg8JHouSliI=; b=dgfeaApdE2xTtiSaaXvmXIp3PySbeZYqRbAAaNbBwJWkGw3wrCQk0hVmCEAAQCZV3r krzkuV1zMBF+taBrpkw1pA8/ijqxWzjuX+Ia0nxQY2xry5e6/ulBnxvy5riYEoTvMX8/ NmjfWvILT2rqw6IWInoJCPY83sMipjtS/Il2ZbRVcar/A/WnCdbSac4u+ej9XIXewv1a ooId6R3hQCefwYpfmaR0JkUk4nqSBnc5mmDW1jQl0YldbG4SXJkLkCPWmlIzdB5LKDth ezJYI+JCBN7JP67blJowem1KWYzyaq1x39av6QOrflGiW6i3d7g0kgnw5xuIiQ6jC0D9 RoGg== X-Gm-Message-State: AOUpUlEDXVfRLzJRFuZshOpeP1ePltbRddMBg0kB9qLRPesR/4wuZWWD RQgbbcwxrHQ1f+KiqGJTrLc= X-Google-Smtp-Source: AA+uWPymTizqudhGTshSxgXm6VZ6Jb491oZcb37QOzjmvrCHeEzp0MIQK3nmjmMX2l34E+LQyGAMTw== X-Received: by 2002:a62:858c:: with SMTP id m12-v6mr63440766pfk.173.1535036546550; Thu, 23 Aug 2018 08:02:26 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id v19-v6sm17462630pgn.94.2018.08.23.08.02.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Aug 2018 08:02:25 -0700 (PDT) Sender: Mark Johnston Date: Thu, 23 Aug 2018 11:02:22 -0400 From: Mark Johnston To: Sebastian Huber Cc: freebsd-hackers@freebsd.org Subject: Re: [PATCH] Remove stray #ifdef _KERNEL in Message-ID: <20180823150222.GC68810@raichu> References: <20180823112539.20725-1-sebastian.huber@embedded-brains.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180823112539.20725-1-sebastian.huber@embedded-brains.de> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 15:02:28 -0000 On Thu, Aug 23, 2018 at 01:25:39PM +0200, Sebastian Huber wrote: > The malloc() macro definition is aready in an #ifdef _KERNEL section. > --- > sys/sys/malloc.h | 2 -- > 1 file changed, 2 deletions(-) Committed in r338252, thanks. From owner-freebsd-hackers@freebsd.org Fri Aug 24 13:37:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 357DB108A6B7 for ; Fri, 24 Aug 2018 13:37:38 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from dedi548.your-server.de (dedi548.your-server.de [85.10.215.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C15B28646A for ; Fri, 24 Aug 2018 13:37:37 +0000 (UTC) (envelope-from sebastian.huber@embedded-brains.de) Received: from [78.46.172.2] (helo=sslproxy05.your-server.de) by dedi548.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.85_2) (envelope-from ) id 1ftCHJ-0007UC-M6 for freebsd-hackers@freebsd.org; Fri, 24 Aug 2018 15:37:29 +0200 Received: from [82.135.62.35] (helo=mail.embedded-brains.de) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1ftCHJ-00034l-Ek for freebsd-hackers@freebsd.org; Fri, 24 Aug 2018 15:37:29 +0200 Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 79D0F2A165C for ; Fri, 24 Aug 2018 15:37:33 +0200 (CEST) Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id e3tnGbiBX6BF for ; Fri, 24 Aug 2018 15:37:32 +0200 (CEST) Received: from localhost (localhost.localhost [127.0.0.1]) by mail.embedded-brains.de (Postfix) with ESMTP id 8B1B32A167E for ; Fri, 24 Aug 2018 15:37:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.eb.localhost Received: from mail.embedded-brains.de ([127.0.0.1]) by localhost (zimbra.eb.localhost [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id F_UeI5raFWWK for ; Fri, 24 Aug 2018 15:37:32 +0200 (CEST) Received: from [192.168.96.149] (unknown [192.168.96.149]) by mail.embedded-brains.de (Postfix) with ESMTPSA id 62D2B2A165C for ; Fri, 24 Aug 2018 15:37:32 +0200 (CEST) Subject: Re: epoch(9) background information? From: Sebastian Huber To: FreeBSD References: Message-ID: Date: Fri, 24 Aug 2018 15:37:27 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable X-Authenticated-Sender: smtp-embedded@poldinet.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24868/Fri Aug 24 06:46:35 2018) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 13:37:38 -0000 Hello, I try currently to understand the epoch implementation in FreeBSD to=20 implement the API in RTEMS. If I understood epoch_wait() correctly, then this function just busy=20 waits until all conditions are satisfied. This is all right, since all=20 other participants operate with thread dispatching disabled, so the time=20 can be only influenced by interrupts and actual work. I have some questions to epoch_block_handler_preempt(). The caller of=20 epoch_wait_prempt() seems to depend on threads which are currently=20 blocked, so busy waiting is not a good idea. Some questions to the code: /* =C2=A0* epoch_block_handler_preempt is a callback from the ck code when=20 another thread is =C2=A0* currently in an epoch section. =C2=A0*/ static void epoch_block_handler_preempt(struct ck_epoch *global __unused,=20 ck_epoch_record_t *cr, =C2=A0=C2=A0=C2=A0 void *arg __unused) { =C2=A0=C2=A0=C2=A0 epoch_record_t record; =C2=A0=C2=A0=C2=A0 struct thread *td, *owner, *curwaittd; =C2=A0=C2=A0=C2=A0 struct epoch_thread *tdwait; =C2=A0=C2=A0=C2=A0 struct turnstile *ts; =C2=A0=C2=A0=C2=A0 struct lock_object *lock; =C2=A0=C2=A0=C2=A0 int spincount, gen; =C2=A0=C2=A0=C2=A0 int locksheld __unused; =C2=A0=C2=A0=C2=A0 record =3D __containerof(cr, struct epoch_record, er_= record); =C2=A0=C2=A0=C2=A0 td =3D curthread; =C2=A0=C2=A0=C2=A0 locksheld =3D td->td_locks; =C2=A0=C2=A0=C2=A0 spincount =3D 0; =C2=A0=C2=A0=C2=A0 counter_u64_add(block_count, 1); =C2=A0=C2=A0=C2=A0 /* =C2=A0=C2=A0=C2=A0 =C2=A0* We lost a race and there's no longer any thre= ads =C2=A0=C2=A0=C2=A0 =C2=A0* on the CPU in an epoch section. =C2=A0=C2=A0=C2=A0 =C2=A0*/ =C2=A0=C2=A0=C2=A0 if (TAILQ_EMPTY(&record->er_tdlist)) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 return; =C2=A0=C2=A0=C2=A0 if (record->er_cpuid !=3D curcpu) { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ... ### this seems clear =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 /* =C2=A0=C2=A0=C2=A0 =C2=A0* Try to find a thread in an epoch section on t= his CPU =C2=A0=C2=A0=C2=A0 =C2=A0* waiting on a turnstile. Otherwise find the lo= west =C2=A0=C2=A0=C2=A0 =C2=A0* priority thread (highest prio value) and drop= our priority =C2=A0=C2=A0=C2=A0 =C2=A0* to match to allow it to run. =C2=A0=C2=A0=C2=A0 =C2=A0*/ =C2=A0=C2=A0=C2=A0 TAILQ_FOREACH(tdwait, &record->er_tdlist, et_link) { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 /* =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0* Propagate our priority to = any other waiters to prevent us =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0* from starving them. They w= ill have their original priority =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0* restore on exit from epoch= _wait(). =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0*/ =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 curwaittd =3D tdwait->et_td; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 if (!TD_IS_INHIBITED(curwaittd) &&= curwaittd->td_priority >=20 td->td_priority) { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 critical_enter(= ); =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 thread_unlock(t= d); =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 thread_lock(cur= waittd); ### could we accidentally lower the priority here? =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 sched_prio(curw= aittd, td->td_priority); =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 thread_unlock(c= urwaittd); =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 thread_lock(td)= ; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 critical_exit()= ; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 if (TD_IS_INHIBITED(curwaittd) && = TD_ON_LOCK(curwaittd) && =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ((ts =3D curwai= ttd->td_blocked) !=3D NULL)) { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ... ### is this some sort of condition variable which wakes us up together=20 with the waiting thread? =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 } ### is this a yield operation, e.g. in case we still the highest=20 priority thread, then we run immediately again? =C2=A0=C2=A0=C2=A0 /* =C2=A0=C2=A0=C2=A0 =C2=A0* We didn't find any threads actually blocked o= n a lock =C2=A0=C2=A0=C2=A0 =C2=A0* so we have nothing to do except context switc= h away. =C2=A0=C2=A0=C2=A0 =C2=A0*/ =C2=A0=C2=A0=C2=A0 counter_u64_add(switch_count, 1); =C2=A0=C2=A0=C2=A0 mi_switch(SW_VOL | SWT_RELINQUISH, NULL); =C2=A0=C2=A0=C2=A0 /* =C2=A0=C2=A0=C2=A0 =C2=A0* Release the thread lock while yielding to =C2=A0=C2=A0=C2=A0 =C2=A0* allow other threads to acquire the lock =C2=A0=C2=A0=C2=A0 =C2=A0* pointed to by TDQ_LOCKPTR(td). Else a =C2=A0=C2=A0=C2=A0 =C2=A0* deadlock like situation might happen. (HPS) =C2=A0=C2=A0=C2=A0 =C2=A0*/ =C2=A0=C2=A0=C2=A0 thread_unlock(td); =C2=A0=C2=A0=C2=A0 thread_lock(td); } --=20 Sebastian Huber, embedded brains GmbH Address : Dornierstr. 4, D-82178 Puchheim, Germany Phone : +49 89 189 47 41-16 Fax : +49 89 189 47 41-09 E-Mail : sebastian.huber@embedded-brains.de PGP : Public key available on request. Diese Nachricht ist keine gesch=C3=A4ftliche Mitteilung im Sinne des EHUG= . From owner-freebsd-hackers@freebsd.org Fri Aug 24 14:53:58 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B990108C085 for ; Fri, 24 Aug 2018 14:53:58 +0000 (UTC) (envelope-from 6yearold@gmail.com) Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9405F894F4 for ; Fri, 24 Aug 2018 14:53:56 +0000 (UTC) (envelope-from 6yearold@gmail.com) Received: by mail-pg1-x530.google.com with SMTP id z25-v6so4412642pgu.7 for ; Fri, 24 Aug 2018 07:53:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=L1hkVHz5eDKQfe0x2eOjCKPtY6zTTF60dk/fVrSxVdg=; b=oRwXWglcZX+klMSoCZFjfF8KkkkRIXjh+vpxPJNpDQmKN25FwWEiaUJajHQYqo8aoF kw8hJgvbDzozzmYpe5I/5PzFrRBGc8f7O3gQjdUnfXThg/vswwFxkW7XFGDyLNQRaU9x 2w6R2mAOIRQPVOfSn17XsOtfKoZ5aycFUwvc5UiXiFpZdAdD/9DzZVh3wvHWSoH1PbOm eOsEDURyDvGUu6w4ynOJAJr9fqN8Cs3ZdXetVV27v50C3Ocl3bUqpFJbjXBdIo3YUpMP AJ43mNwNpMOGeTwI6/AsYWEECq2OI5CdLPtN8YJPkC+fKKHcKXwQECjCpJeaduh4NQDL z82w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=L1hkVHz5eDKQfe0x2eOjCKPtY6zTTF60dk/fVrSxVdg=; b=rPn0BV+cG/OupnOCLF17kQIXWrTzySCZHwdaWtywHmEQYJ46dWry+/OsHeLjEdKKSV 4KjB7OwZsKA64JeKU/3FGIlnA/QnHRfFtiP0hdb1ocnp2XHhkk+BMsegHZB5mpFo1tte DxqnLCEEiTQTmKtpeSldwX45etCpWHCltPD3iWCbtU6IyeCcBkgaQQV+6m/RujLAXtO4 rqFAyI+JoDGGFDInaMOTyjETBgPhFiwrooNTYqUEffUrqL1jDcfmkDsomrLzs+Xa+46C ftcZ3Xbde1YPt45Xqr4NG4fw137+F5RIVT6NyUDZjfQryTXmhUuxpihPSIg8+jonjSOn Vmrg== X-Gm-Message-State: APzg51Ai7C98/7P/5nEGTeCMQOWWJmRvG+lT0p6meC74M98HBPgCVnC1 lMWeHgtv+0Y75s5H12ze16XVM/Kkk8CC9HFxLPYxr47+908= X-Google-Smtp-Source: ANB0Vdb2Bh5XuZo5yRsifFB7QvoTWkI707ZEzKiLxGFaCrW1GBu9DlZMbFVf40TW71v5ahzKllKatDMSZDP3Pm5cnqM= X-Received: by 2002:a62:4fd9:: with SMTP id f86-v6mr2394653pfj.110.1535122435431; Fri, 24 Aug 2018 07:53:55 -0700 (PDT) MIME-Version: 1.0 From: Gleb Popov <6yearold@gmail.com> Date: Fri, 24 Aug 2018 17:53:28 +0300 Message-ID: Subject: Strange hang when calling signal() To: freebsd-hackers X-Mailman-Approved-At: Fri, 24 Aug 2018 15:01:13 +0000 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 14:53:58 -0000 I'm debugging a Qt test app that hangs when launching a QProcess. The parent does the following: QProcess p; ... p.start(); p.waitForStarted(-1); // wait indefinitely Under the hood starting the QProcess involves creating a pair of pipes and forking: qt_create_pipe(childStartedPipe); ... pid_t childPid; ::forkfd(FFD_CLOEXEC, &childPid); and waiting for it to be started is just ppoll()'ing on the pipe pollfd pfd = qt_make_pollfd(childStartedPipe[0], POLLIN); if (qt_poll_msecs(&pfd, 1, msecs) == 0) { ... On the child side the code looks like ::signal(SIGPIPE, SIG_DFL); ... qt_safe_close(childStartedPipe[0]); ... qt_safe_execv(argv[0], argv); So, the problem is that after forking the parent process hangs on polling and child process hangs inside signal call; Here is the backtrace: #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x0000000802bd9571 in __thr_rwlock_rdlock (rwlock=0x802bf3200, flags=, tsp=) at /usr/src/lib/libthr/thread/thr_umtx.c:307 #2 0x0000000802be24c0 in _thr_rwlock_rdlock (flags=0, tsp=0x0, rwlock=) at /usr/src/lib/libthr/thread/thr_umtx.h:232 #3 _thr_rtld_rlock_acquire (lock=0x802bf3200) at /usr/src/lib/libthr/thread/thr_rtld.c:125 #4 0x000000080024e63b in rlock_acquire (lock=0x80025f0a0 , lockstate=0x7fffffffc678) at /usr/src/libexec/rtld-elf/rtld_lock.c:208 #5 0x00000008002472dd in _rtld_bind (obj=0x80027b000, reloff=4416) at /usr/src/libexec/rtld-elf/rtld.c:788 #6 0x000000080024404d in _rtld_bind_start () at /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 #7 0x0000000803d31a76 in QProcessPrivate::execChild (this=0x81a9716c0, workingDir=0x0, argv=0x81fde5760, envp=0x0) at io/qprocess_unix.cpp:537 Any idea what causes signal() to not return? I haven't extracted a minimal repro yet, wanted to ask for any clues first. The code in question is here: https://github.com/qt/qtbase/blob/5.11/src/corelib/io/qprocess_unix.cpp Relevant functions are QProcessPrivate::startProcess(), QProcessPrivate::execChild(), QProcessPrivate::waitForStarted(). Thanks in advance. From owner-freebsd-hackers@freebsd.org Fri Aug 24 18:53:44 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E446A10917DC for ; Fri, 24 Aug 2018 18:53:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5500F728B2 for ; Fri, 24 Aug 2018 18:53:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w7OIrSP7069295 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 21:53:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w7OIrSP7069295 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w7OIrSeI069294; Fri, 24 Aug 2018 21:53:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 24 Aug 2018 21:53:28 +0300 From: Konstantin Belousov To: Gleb Popov <6yearold@gmail.com> Cc: freebsd-hackers Subject: Re: Strange hang when calling signal() Message-ID: <20180824185328.GE2340@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 18:53:44 -0000 On Fri, Aug 24, 2018 at 05:53:28PM +0300, Gleb Popov wrote: > I'm debugging a Qt test app that hangs when launching a QProcess. > > The parent does the following: > > QProcess p; > ... > p.start(); > p.waitForStarted(-1); // wait indefinitely > > Under the hood starting the QProcess involves creating a pair of pipes and > forking: > > qt_create_pipe(childStartedPipe); > ... > pid_t childPid; > ::forkfd(FFD_CLOEXEC, &childPid); > > and waiting for it to be started is just ppoll()'ing on the pipe > > pollfd pfd = qt_make_pollfd(childStartedPipe[0], POLLIN); > if (qt_poll_msecs(&pfd, 1, msecs) == 0) { > ... > > On the child side the code looks like > > ::signal(SIGPIPE, SIG_DFL); > ... > qt_safe_close(childStartedPipe[0]); > ... > qt_safe_execv(argv[0], argv); > > So, the problem is that after forking the parent process hangs on polling > and child process hangs inside signal call; Here is the backtrace: > > #0 _umtx_op_err () at > /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > #1 0x0000000802bd9571 in __thr_rwlock_rdlock (rwlock=0x802bf3200, > flags=, tsp=) at > /usr/src/lib/libthr/thread/thr_umtx.c:307 > #2 0x0000000802be24c0 in _thr_rwlock_rdlock (flags=0, tsp=0x0, > rwlock=) at /usr/src/lib/libthr/thread/thr_umtx.h:232 > #3 _thr_rtld_rlock_acquire (lock=0x802bf3200) at > /usr/src/lib/libthr/thread/thr_rtld.c:125 > #4 0x000000080024e63b in rlock_acquire (lock=0x80025f0a0 , > lockstate=0x7fffffffc678) at /usr/src/libexec/rtld-elf/rtld_lock.c:208 > #5 0x00000008002472dd in _rtld_bind (obj=0x80027b000, reloff=4416) at > /usr/src/libexec/rtld-elf/rtld.c:788 > #6 0x000000080024404d in _rtld_bind_start () at > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 > #7 0x0000000803d31a76 in QProcessPrivate::execChild (this=0x81a9716c0, > workingDir=0x0, argv=0x81fde5760, envp=0x0) at io/qprocess_unix.cpp:537 > > Any idea what causes signal() to not return? I haven't extracted a minimal > repro yet, wanted to ask for any clues first. Immediate and not that useful answer is that the child process is trying to acquire rtld bind lock, and the state of the lock is busy. There was a constant stream of the bugs some time ago, where multithreaded process forked and then tried to use services which require some of the C runtime internal locks. It did not helped that POSIX allow most of this breakage. Since state of the parent process is usually not determinate at the time of fork, other thread might have grabbed some of that locks (and made internal structures inconsistent), which is inherited by the child. Then there is nobody in the child to correct the damage (restore consistency and unlock). Since then, we started locking most of that locks in parent around fork(2), all the code in lib/libthr/thread/thr_fork.c. In particular, we lock rtld, malloc, and disable cancellation around fork. So if your program used fork(2) but ended with the broken rtld it is worrying. On the other hand, we do not do that for vfork(2). So yes, the minimal reproduction case, in bare libc/libthr API (i.e. without QT), would be the first step to diagnose and and might be fix. > > The code in question is here: > https://github.com/qt/qtbase/blob/5.11/src/corelib/io/qprocess_unix.cpp > Relevant functions are QProcessPrivate::startProcess(), > QProcessPrivate::execChild(), QProcessPrivate::waitForStarted(). From owner-freebsd-hackers@freebsd.org Fri Aug 24 19:10:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABA4A1091A6A for ; Fri, 24 Aug 2018 19:10:47 +0000 (UTC) (envelope-from eric@vangyzen.net) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [199.48.133.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59E7172D7F for ; Fri, 24 Aug 2018 19:10:47 +0000 (UTC) (envelope-from eric@vangyzen.net) Received: from hammy.vangyzen.net (unknown [70.97.188.230]) by smtp.vangyzen.net (Postfix) with ESMTPSA id D4B835646E; Fri, 24 Aug 2018 14:10:39 -0500 (CDT) Subject: Re: Strange hang when calling signal() To: Konstantin Belousov , Gleb Popov <6yearold@gmail.com> Cc: freebsd-hackers References: <20180824185328.GE2340@kib.kiev.ua> From: Eric van Gyzen Message-ID: <4e555da7-9384-0f22-3ef9-8b3661a48529@vangyzen.net> Date: Fri, 24 Aug 2018 14:10:36 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <20180824185328.GE2340@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 19:10:47 -0000 On 8/24/18 1:53 PM, Konstantin Belousov wrote: > Since then, we started locking most of that locks in parent around fork(2), > all the code in lib/libthr/thread/thr_fork.c. In particular, we lock rtld, > malloc, and disable cancellation around fork. So if your program used fork(2) > but ended with the broken rtld it is worrying. > > On the other hand, we do not do that for vfork(2). So yes, the minimal We also do not do that for rfork(2). I think we should, but I have not done any research into it. Eric From owner-freebsd-hackers@freebsd.org Fri Aug 24 19:23:27 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52DDF1092261 for ; Fri, 24 Aug 2018 19:23:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9703B7382B for ; Fri, 24 Aug 2018 19:23:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w7OJNF3K076045 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 22:23:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w7OJNF3K076045 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w7OJNFs3076044; Fri, 24 Aug 2018 22:23:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 24 Aug 2018 22:23:15 +0300 From: Konstantin Belousov To: Eric van Gyzen Cc: Gleb Popov <6yearold@gmail.com>, freebsd-hackers Subject: Re: Strange hang when calling signal() Message-ID: <20180824192315.GF2340@kib.kiev.ua> References: <20180824185328.GE2340@kib.kiev.ua> <4e555da7-9384-0f22-3ef9-8b3661a48529@vangyzen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4e555da7-9384-0f22-3ef9-8b3661a48529@vangyzen.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 19:23:27 -0000 On Fri, Aug 24, 2018 at 02:10:36PM -0500, Eric van Gyzen wrote: > On 8/24/18 1:53 PM, Konstantin Belousov wrote: > > Since then, we started locking most of that locks in parent around fork(2), > > all the code in lib/libthr/thread/thr_fork.c. In particular, we lock rtld, > > malloc, and disable cancellation around fork. So if your program used fork(2) > > but ended with the broken rtld it is worrying. > > > > On the other hand, we do not do that for vfork(2). So yes, the minimal > > We also do not do that for rfork(2). I think we should, but I have not > done any research into it. I do not see how would it be correct to do that locking for vfork(2) because the address space is shared between parent and child. For the similar reason, sometimes rfork(2) also leaves the address space shared and then we have the problem. In essence, rfork(2) and vfork(2) do not mix with pthreads, and if you do, you should know well what you do. From owner-freebsd-hackers@freebsd.org Fri Aug 24 19:38:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2734B1092636 for ; Fri, 24 Aug 2018 19:38:49 +0000 (UTC) (envelope-from eric@vangyzen.net) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [199.48.133.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CAB8373D80 for ; Fri, 24 Aug 2018 19:38:48 +0000 (UTC) (envelope-from eric@vangyzen.net) Received: from hammy.vangyzen.net (unknown [70.97.188.230]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 8F82A5646E; Fri, 24 Aug 2018 14:38:47 -0500 (CDT) Subject: Re: Strange hang when calling signal() To: Konstantin Belousov Cc: Gleb Popov <6yearold@gmail.com>, freebsd-hackers References: <20180824185328.GE2340@kib.kiev.ua> <4e555da7-9384-0f22-3ef9-8b3661a48529@vangyzen.net> <20180824192315.GF2340@kib.kiev.ua> From: Eric van Gyzen Message-ID: Date: Fri, 24 Aug 2018 14:38:46 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <20180824192315.GF2340@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 19:38:49 -0000 On 8/24/18 2:23 PM, Konstantin Belousov wrote: > On Fri, Aug 24, 2018 at 02:10:36PM -0500, Eric van Gyzen wrote: >> On 8/24/18 1:53 PM, Konstantin Belousov wrote: >>> Since then, we started locking most of that locks in parent around fork(2), >>> all the code in lib/libthr/thread/thr_fork.c. In particular, we lock rtld, >>> malloc, and disable cancellation around fork. So if your program used fork(2) >>> but ended with the broken rtld it is worrying. >>> >>> On the other hand, we do not do that for vfork(2). So yes, the minimal >> >> We also do not do that for rfork(2). I think we should, but I have not >> done any research into it. > > I do not see how would it be correct to do that locking for vfork(2) because > the address space is shared between parent and child. For the similar reason, > sometimes rfork(2) also leaves the address space shared and then we have > the problem. I wasn't suggesting that we do it for vfork(2). I was only suggesting that we do it for rfork(2), and obviously only if !RFMEM (and certain other flag conditions). Eric From owner-freebsd-hackers@freebsd.org Fri Aug 24 21:54:54 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B39701094B9D for ; Fri, 24 Aug 2018 21:54:54 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47D64781AF for ; Fri, 24 Aug 2018 21:54:54 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-oi0-x22e.google.com with SMTP id l202-v6so10890345oig.7 for ; Fri, 24 Aug 2018 14:54:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=/xpLAp0W+nhPb0LAp3UT3LT50kdzQ/sN/ehLDKQbquc=; b=GHRXfEaDhHaDok00ah9nk5BS22k+Run+EX1yoS5h+utw15g7ouRC5GA4GfYLlYeosg zDpAlUClkX3uGMfOU2uZDgI1e5cR7Gi9EfrTeiuSlgFX5x5Cj15qLvtJsV599wsTEuDa NwgwslGdHjYuYssigb20HNEV+uCGJgY43YfAaGaEPMsncU+VldJAzETkIvJBj7l9Srj4 NUlfNdhuveKj3ASUIQ/xuBvjP20VLkmtYZmibBpU6Ek/BtIomsgQC0Ety1CSUia+Tr2i ozaNr8Z1IHnFkYYjzw2Qyydys7eBAlF/X/240a9JmWBkLqkbn+5VxUmqelPUgkaupSR2 cggA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=/xpLAp0W+nhPb0LAp3UT3LT50kdzQ/sN/ehLDKQbquc=; b=ccdu7A/tFquAlQGLNviNfSmHxnxYJ9sXolWrQLH6xpHLXjt3xlJvJnIQqbZZp1mqWc R3qhfmUJLjg+N6TEpv7pJijWSB81f8K+pTLBpWnPsWl9N+PRU1Cz5eeYwcLLgeou1aM2 Q+R2gxETbASoykiRFkVGmDk7HzVR090ZouyJG/otBfWGNTqT6GG8aftwc10ZcK+JAFyJ H0w+WfUUWXk5aVYEY2TiN6rs84LV8DmgCvvStvtx3bPb8i35s/0C2I5YLCDkDtDsfo/g 87LtrORNg8L0BZXCDLn7FR9x2y/g44RTbYOfLxGLN7pP+hdL7NWwvCXIo/gaEU2PI/Aw L+Ig== X-Gm-Message-State: APzg51Ck8i0HB1PM9L+weNAwfOoVYzNohDmcNTEteyf/spUadtbFabkO wCZ6alpLatDLft+X3IDP1DLQJXxzFWwUkLJOEwHjbQ== X-Google-Smtp-Source: ANB0VdbdaKGwzlZbiFTlz7xtztAswSH4noajk3mToCeI8elvjjHgKCEjy/aeUqSPJt3/v0QqELiVxqf6oIjFEgdak3I= X-Received: by 2002:aca:de08:: with SMTP id v8-v6mr3317786oig.124.1535147693363; Fri, 24 Aug 2018 14:54:53 -0700 (PDT) MIME-Version: 1.0 From: David Cross Date: Fri, 24 Aug 2018 17:54:42 -0400 Message-ID: Subject: weird geli behavior To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 21:54:54 -0000 Ok, I am seeing something truely bizzare, I am sending this out as a shot across the bow since I am not even sure where or how to begin debugging this. Some background. This in on an Intel Xeon 5520 based machine, 72G ECC memory, 11.2, fully patched. Though this has been a problem since at least 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which it basically never even touches, even when problems are occuring) The first symptom was (and I think these are all aspects of the same root underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives would *randomly* error out on a corrupt entry. Upon investigating this I discovered by watching gstat that as this happened the IO on the drives would STOP. the L(q) would hover at 1 for a number of seconds, and then when it returned fsck was complaining about various corrupt structures. a ktrace of fsck shows that it got back data from the pread() that was partially corrupted (I am guessing, but I cannot confirm that 'some part' of the stack handed back a zeroed page, or otherwise 'not the right data' that geli dutifully 'decrypted'. No errors are ever logged in the kernel about da0 or da1 (the respective underlying USB disks). It *seems* this is *always* on phase 2 of fsck (files and paths), and its never the same inode. no data is *ever* corrupted when in the filesystem, no matter how hard I hit the disks (all data on these devices is fully checksummed) Devices have passed multiple SMART full diag checks, full read/write tests with no issues. Under heavy FS IO it does occasionally lock.. but recovers, and again data and filesystem are fully consistent. I was willing to live with that.. weird as it was (these are backup disks, data is fully checksummed, and I was only fscking out of extreme paranoia every reboot) Then I added an internal drive, configured with gmirror (broken mirror currently, second disk hasn't been added) and geli. On this disk I have a postgres 10 database in WAL replication. This was working fine and then the other day the system just locked for a few hours. During that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and it doing _1_ operation a second to the underlying disk... all the while geli is logging 'error 11' to the console (nothing about the underlying disk) After this happened a static file on the disk (a zip file) had bad data in the middle of a page (after reboot the file was ok.. so it was just in cache). Again, this disk fully checks ok, no corruption on the disk, no errors from the disk itself. Halp? where do I even begin with this? It really feels like there is some massive locking going on in geli in some way? Where should I even begin looking? I run geli on most of my systems and don't have any issues. From owner-freebsd-hackers@freebsd.org Sat Aug 25 01:00:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08DFC109986D for ; Sat, 25 Aug 2018 01:00:26 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CBC37F4A0 for ; Sat, 25 Aug 2018 01:00:24 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w7P10N4G078523 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 18:00:23 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w7P10NmT078522; Fri, 24 Aug 2018 18:00:23 -0700 (PDT) (envelope-from jmg) Date: Fri, 24 Aug 2018 18:00:23 -0700 From: John-Mark Gurney To: David Cross Cc: FreeBSD Hackers Subject: Re: weird geli behavior Message-ID: <20180825010023.GD45503@funkthat.com> Mail-Followup-To: David Cross , FreeBSD Hackers References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Fri, 24 Aug 2018 18:00:23 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 01:00:26 -0000 David Cross wrote this message on Fri, Aug 24, 2018 at 17:54 -0400: > Ok, I am seeing something truely bizzare, I am sending this out as a shot > across the bow since I am not even sure where or how to begin debugging > this. > > Some background. This in on an Intel Xeon 5520 based machine, 72G ECC > memory, 11.2, fully patched. Though this has been a problem since at least > 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which it > basically never even touches, even when problems are occuring) I assume you've applied the lazyfpu SA patch? If so, there's another patch you need to apply, see: https://docs.freebsd.org/cgi/mid.cgi?20180821081150.GU2340@kib.kiev.ua > The first symptom was (and I think these are all aspects of the same root > underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives > would *randomly* error out on a corrupt entry. Upon investigating this I > discovered by watching gstat that as this happened the IO on the drives > would STOP. the L(q) would hover at 1 for a number of seconds, and then > when it returned fsck was complaining about various corrupt structures. a > ktrace of fsck shows that it got back data from the pread() that was > partially corrupted (I am guessing, but I cannot confirm that 'some part' > of the stack handed back a zeroed page, or otherwise 'not the right data' > that geli dutifully 'decrypted'. No errors are ever logged in the kernel > about da0 or da1 (the respective underlying USB disks). It *seems* this is > *always* on phase 2 of fsck (files and paths), and its never the same > inode. no data is *ever* corrupted when in the filesystem, no matter how > hard I hit the disks (all data on these devices is fully checksummed) > Devices have passed multiple SMART full diag checks, full read/write tests > with no issues. Under heavy FS IO it does occasionally lock.. but > recovers, and again data and filesystem are fully consistent. > > I was willing to live with that.. weird as it was (these are backup disks, > data is fully checksummed, and I was only fscking out of extreme paranoia > every reboot) Then I added an internal drive, configured with gmirror > (broken mirror currently, second disk hasn't been added) and geli. On this > disk I have a postgres 10 database in WAL replication. This was working > fine and then the other day the system just locked for a few hours. During > that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and > it doing _1_ operation a second to the underlying disk... all the while > geli is logging 'error 11' to the console (nothing about the underlying > disk) After this happened a static file on the disk (a zip file) had bad > data in the middle of a page (after reboot the file was ok.. so it was > just in cache). Again, this disk fully checks ok, no corruption on the > disk, no errors from the disk itself. > > > Halp? where do I even begin with this? It really feels like there is > some massive locking going on in geli in some way? Where should I even > begin looking? I run geli on most of my systems and don't have any issues. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Sat Aug 25 02:43:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39A31109D843 for ; Sat, 25 Aug 2018 02:43:01 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-qt0-x234.google.com (mail-qt0-x234.google.com [IPv6:2607:f8b0:400d:c0d::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4038833CF for ; Sat, 25 Aug 2018 02:43:00 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-qt0-x234.google.com with SMTP id x7-v6so12285854qtk.5 for ; Fri, 24 Aug 2018 19:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=MFODX4jLsBrCTfvA1wNWv2Ys4qeh9f6Jcb71m0Cd8OQ=; b=AJW9btjlupVFPUw83YyhWgLy67J1dNPC7SMfg+vUYoj6F8QMejzUCFBUVqlMxYX/Aw RnTfWgH44PApOLzsILgRpTQ1ZWKMTiIudh77PSPt0PNCGnL8JJSndwksUBYsy9zwuXEb sjz/4OTrUoeaevzF19G2GgLPXAJfMpQHwoou5F4a3B2NCv7ELTnsdpN5lyv4FLGKIRiC +GSZJ5k9i7mFFLJd/5sBMHWCQurVjCfGdl2JM50O8ON9309e6XPvObqMeB9wJzfN9NmN GFE4YRzKLP2b9Mzlneuhl5JNFlD3B55oZhExAnQUcdxHHr/tuMW7aU9Vc0DtHuVbroA7 0F0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=MFODX4jLsBrCTfvA1wNWv2Ys4qeh9f6Jcb71m0Cd8OQ=; b=fTLwHhfJvMxyiR7udNBnu4OthPFMyttDGTizVo9Ni8LeDvDoY88lH1ef/kRcjRZ/Mt ottMP79HHH80iaQf2bUX6l52Js/1oXM1dph4G9xM9CV3cUnhwxn2LRQJ3eTIociKNTHM fGy1BkzbYksvq/AQ0J8t7KaSswcWhif4SDH0zTW2YN53EajEBo1qu3EpTPCYYWGj8qBO DH9a7PGHKT0iJSUPUp5GvzuvZcrJYpON3XyDWtDZQWeL7SVVR3R+bJh3VFtgcXhWq+Rb 0oG4byttlgkMCLszbd1LmkrVv478bjJmFdJzeM41hxI2/TvlRmVazYzr8qDAMYM9PT4x fY6w== X-Gm-Message-State: APzg51D5Bcmd5vsewJt1T8gOLHlBc091jpXN+EPF9FUC5n6NMfw9tb4j /hBiZxcKIa2IPUnSTXdJHseoctTh X-Google-Smtp-Source: ANB0Vdb01VROMoW2xFTs8Quw5nxK8V6PzjfaWTGSyrYOfmMKK5SEyIDOCxH6ZB4Nt6PNISrPPzZOcQ== X-Received: by 2002:a0c:e7cc:: with SMTP id c12-v6mr4704846qvo.128.1535164980036; Fri, 24 Aug 2018 19:43:00 -0700 (PDT) Received: from ?IPv6:2600:1017:b006:54dc:61db:5f69:d835:5ed0? ([2600:1017:b006:54dc:61db:5f69:d835:5ed0]) by smtp.gmail.com with ESMTPSA id x32-v6sm6289541qtb.70.2018.08.24.19.42.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Aug 2018 19:42:59 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: weird geli behavior From: David Cross X-Mailer: iPhone Mail (15G77) In-Reply-To: <20180825010023.GD45503@funkthat.com> Date: Fri, 24 Aug 2018 22:42:57 -0400 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180825010023.GD45503@funkthat.com> To: John-Mark Gurney X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 02:43:01 -0000 > On Aug 24, 2018, at 21:00, John-Mark Gurney wrote: >=20 > David Cross wrote this message on Fri, Aug 24, 2018 at 17:54 -0400: >> Ok, I am seeing something truely bizzare, I am sending this out as a shot= >> across the bow since I am not even sure where or how to begin debugging >> this. >>=20 >> Some background. This in on an Intel Xeon 5520 based machine, 72G ECC >> memory, 11.2, fully patched. Though this has been a problem since at lea= st >> 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which i= t >> basically never even touches, even when problems are occuring) >=20 > I assume you've applied the lazyfpu SA patch? >=20 > If so, there's another patch you need to apply, see: > https://docs.freebsd.org/cgi/mid.cgi?20180821081150.GU2340@kib.kiev.ua >=20 I will definitely apply this, but I don=E2=80=99t think it applies to the pr= oblem in question. This system doesn=E2=80=99t have AESNI, this problem well= preceded the lazyfpu patch, and i am not seeing any corruption on disk. >> The first symptom was (and I think these are all aspects of the same root= >> underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives >> would *randomly* error out on a corrupt entry. Upon investigating this I= >> discovered by watching gstat that as this happened the IO on the drives >> would STOP. the L(q) would hover at 1 for a number of seconds, and then >> when it returned fsck was complaining about various corrupt structures. a= >> ktrace of fsck shows that it got back data from the pread() that was >> partially corrupted (I am guessing, but I cannot confirm that 'some part'= >> of the stack handed back a zeroed page, or otherwise 'not the right data'= >> that geli dutifully 'decrypted'. No errors are ever logged in the kernel= >> about da0 or da1 (the respective underlying USB disks). It *seems* this i= s >> *always* on phase 2 of fsck (files and paths), and its never the same >> inode. no data is *ever* corrupted when in the filesystem, no matter how= >> hard I hit the disks (all data on these devices is fully checksummed) >> Devices have passed multiple SMART full diag checks, full read/write test= s >> with no issues. Under heavy FS IO it does occasionally lock.. but >> recovers, and again data and filesystem are fully consistent. >>=20 >> I was willing to live with that.. weird as it was (these are backup disks= , >> data is fully checksummed, and I was only fscking out of extreme paranoia= >> every reboot) Then I added an internal drive, configured with gmirror >> (broken mirror currently, second disk hasn't been added) and geli. On th= is >> disk I have a postgres 10 database in WAL replication. This was working >> fine and then the other day the system just locked for a few hours. Duri= ng >> that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and= >> it doing _1_ operation a second to the underlying disk... all the while >> geli is logging 'error 11' to the console (nothing about the underlying >> disk) After this happened a static file on the disk (a zip file) had bad= >> data in the middle of a page (after reboot the file was ok.. so it was >> just in cache). Again, this disk fully checks ok, no corruption on the >> disk, no errors from the disk itself. >>=20 >>=20 >> Halp? where do I even begin with this? It really feels like there is >> some massive locking going on in geli in some way? Where should I even >> begin looking? I run geli on most of my systems and don't have any issue= s. >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " >=20 > --=20 > John-Mark Gurney Voice: +1 415 225 5579 >=20 > "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Sat Aug 25 05:23:33 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D7A610A1814 for ; Sat, 25 Aug 2018 05:23:33 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A784989C2D for ; Sat, 25 Aug 2018 05:23:32 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w7P5NU1v085877 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 22:23:30 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w7P5NUL3085876; Fri, 24 Aug 2018 22:23:30 -0700 (PDT) (envelope-from jmg) Date: Fri, 24 Aug 2018 22:23:30 -0700 From: John-Mark Gurney To: David Cross Cc: FreeBSD Hackers Subject: Re: weird geli behavior Message-ID: <20180825052330.GE45503@funkthat.com> Mail-Followup-To: David Cross , FreeBSD Hackers References: <20180825010023.GD45503@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Fri, 24 Aug 2018 22:23:30 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 05:23:33 -0000 David Cross wrote this message on Fri, Aug 24, 2018 at 22:42 -0400: > > > > On Aug 24, 2018, at 21:00, John-Mark Gurney wrote: > > > > David Cross wrote this message on Fri, Aug 24, 2018 at 17:54 -0400: > >> Ok, I am seeing something truely bizzare, I am sending this out as a shot > >> across the bow since I am not even sure where or how to begin debugging > >> this. > >> > >> Some background. This in on an Intel Xeon 5520 based machine, 72G ECC > >> memory, 11.2, fully patched. Though this has been a problem since at least > >> 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which it > >> basically never even touches, even when problems are occuring) > > > > I assume you've applied the lazyfpu SA patch? > > > > If so, there's another patch you need to apply, see: > > https://docs.freebsd.org/cgi/mid.cgi?20180821081150.GU2340@kib.kiev.ua > > > I will definitely apply this, but I don???t think it applies to the problem in question. This system doesn???t have AESNI, this problem well preceded the lazyfpu patch, and i am not seeing any corruption on disk. Hmm, k... > >> The first symptom was (and I think these are all aspects of the same root > >> underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives > >> would *randomly* error out on a corrupt entry. Upon investigating this I > >> discovered by watching gstat that as this happened the IO on the drives > >> would STOP. the L(q) would hover at 1 for a number of seconds, and then > >> when it returned fsck was complaining about various corrupt structures. a > >> ktrace of fsck shows that it got back data from the pread() that was > >> partially corrupted (I am guessing, but I cannot confirm that 'some part' > >> of the stack handed back a zeroed page, or otherwise 'not the right data' > >> that geli dutifully 'decrypted'. No errors are ever logged in the kernel > >> about da0 or da1 (the respective underlying USB disks). It *seems* this is > >> *always* on phase 2 of fsck (files and paths), and its never the same > >> inode. no data is *ever* corrupted when in the filesystem, no matter how > >> hard I hit the disks (all data on these devices is fully checksummed) > >> Devices have passed multiple SMART full diag checks, full read/write tests > >> with no issues. Under heavy FS IO it does occasionally lock.. but > >> recovers, and again data and filesystem are fully consistent. > >> > >> I was willing to live with that.. weird as it was (these are backup disks, > >> data is fully checksummed, and I was only fscking out of extreme paranoia > >> every reboot) Then I added an internal drive, configured with gmirror > >> (broken mirror currently, second disk hasn't been added) and geli. On this > >> disk I have a postgres 10 database in WAL replication. This was working > >> fine and then the other day the system just locked for a few hours. During > >> that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and > >> it doing _1_ operation a second to the underlying disk... all the while > >> geli is logging 'error 11' to the console (nothing about the underlying > >> disk) After this happened a static file on the disk (a zip file) had bad > >> data in the middle of a page (after reboot the file was ok.. so it was > >> just in cache). Again, this disk fully checks ok, no corruption on the > >> disk, no errors from the disk itself. > >> > >> > >> Halp? where do I even begin with this? It really feels like there is > >> some massive locking going on in geli in some way? Where should I even > >> begin looking? I run geli on most of my systems and don't have any issues. Can you post actual log lines? geli has lots of error log lines, so w/o more info, pretty hard to say WHAT in geli is returning EAGAIN. I do see that _read_done and _write_done may not handle an EAGAIN error, which could cause this problem, but to confirm, I need the actual log lines... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Sat Aug 25 15:21:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA0A9108D641 for ; Sat, 25 Aug 2018 15:21:17 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7855B7479C for ; Sat, 25 Aug 2018 15:21:17 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id CD2CD21B6A for ; Sat, 25 Aug 2018 11:21:16 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Sat, 25 Aug 2018 11:21:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=SPHrM1otxIpkg43SGGhbs03ZoTSOCteendZSveasYko=; b=T2op22xj 46magpb4Bd+mG37iu9ARHF7VR3zJW20IRSWQViKGt376+Izpn3YjeCdqdq/uDGiD 20OB2Xdi/+wtbY67tW8/ObYkrflSbUzUS5qBO8zu5vbS9NDFBWkXwbaQEl0OKqSt e5UsU7ZARDqstua5r/pAjXOlyub5hnIn7p7Zibb4WhwBO/Dys6jsuI44FxwdMi7b serhLpOjP0e9ASlAaPHdNG9laNthNrJfuBR4hgDVUvWMwKXYlw2X6WKVXLXYO0fs 2HfOAPzC/rOIJzOMFa0pvQFyFtQ6vUmszOyyf9ePcG4DrDCWGSSrGSYoaT0oVOQK WdCU1IxWtm1AKg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; bh=SPHrM1otxIpkg43SGGhbs03ZoTSOC teendZSveasYko=; b=Jl7aLeromQu1i3vyTUmScOU6ZYKTK0Lln4FiCp6sFVe8W eLGkXmAxX+v551Hyl8KobKK3flWSbr+1fsSu9hB6UfRZmH4B6nNkC2xOI2eNFnvx jcxoOFVk8/gpPwb8HJQZTau6WVLO5sGtxliIczRQXp2DUb0WrtWNZA3xal6lsB5A eNxAlhfBFzpifuMPcoWKiovj9xI3aTuNOTPZzbsnJBY9jXBCVgWSW9ftiLz17/xv 6tv2Wm2ljwvogbZE+nlgD/rgPdO9Sd1R5o7ib+bF7QMQPr2HMN6MzfZZUWwXz3oS n6edSd+phq69nxanIzKiLz/JyadqNC+bmzj8zKeyw== X-ME-Proxy: X-ME-Sender: Received: from odin.yuripv.net (unknown [94.233.224.99]) by mail.messagingengine.com (Postfix) with ESMTPA id A6518E474A for ; Sat, 25 Aug 2018 11:21:15 -0400 (EDT) To: freebsd-hackers From: Yuri Pankov Subject: pci bus rescan from userland Message-ID: <561d2b7b-b36a-7bee-bb23-fb7d0ff9a50b@yuripv.net> Date: Sat, 25 Aug 2018 18:21:14 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 15:21:18 -0000 Hi, Trying to enable the HDMI/DP sound output on NVIDIA Quadro M2200, I have found the following workaround on NVIDIA Linux forums (https://devtalk.nvidia.com/default/topic/1024022/linux/gtx-1060-no-audio-over-hdmi-only-hda-intel-detected-azalia/post/5211273/#5211273): setpci -s 01:00.0 0x488.l=0x2000000:0x2000000 rmmod nvidia-drm nvidia-modeset nvidia echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove echo 1 > /sys/bus/pci/devices/0000:00:01.0/rescan modprobe nvidia-drm So far rmmod and modprobe steps are obvious, setpci works as well (I guess I could use pciconf? doesn't really matter). Is there a FreeBSD equivalent of doing the remove/rescan steps from userland? From owner-freebsd-hackers@freebsd.org Sat Aug 25 15:26:10 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 747A5108DA54 for ; Sat, 25 Aug 2018 15:26:10 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 061AC74E19 for ; Sat, 25 Aug 2018 15:26:09 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 2c46f361-a87b-11e8-93fa-f3ebd9db2b94 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id 2c46f361-a87b-11e8-93fa-f3ebd9db2b94; Sat, 25 Aug 2018 15:26:01 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7PFQ1qt003296; Sat, 25 Aug 2018 09:26:01 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1535210761.1488.45.camel@freebsd.org> Subject: Re: pci bus rescan from userland From: Ian Lepore To: Yuri Pankov , freebsd-hackers Date: Sat, 25 Aug 2018 09:26:01 -0600 In-Reply-To: <561d2b7b-b36a-7bee-bb23-fb7d0ff9a50b@yuripv.net> References: <561d2b7b-b36a-7bee-bb23-fb7d0ff9a50b@yuripv.net> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 15:26:10 -0000 On Sat, 2018-08-25 at 18:21 +0300, Yuri Pankov wrote: > Hi, > > Trying to enable the HDMI/DP sound output on NVIDIA Quadro M2200, I have  > found the following workaround on NVIDIA Linux forums  > (https://devtalk.nvidia.com/default/topic/1024022/linux/gtx-1060-no-audio-over-hdmi-only-hda-intel-detected-azalia/post/5211273/#5211273): > > setpci -s 01:00.0 0x488.l=0x2000000:0x2000000 > rmmod nvidia-drm nvidia-modeset nvidia > echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove > echo 1 > /sys/bus/pci/devices/0000:00:01.0/rescan > modprobe nvidia-drm > > So far rmmod and modprobe steps are obvious, setpci works as well (I  > guess I could use pciconf? doesn't really matter).  Is there a FreeBSD  > equivalent of doing the remove/rescan steps from userland? devctl(8) has a rescan function.  devinfo(8) might be useful for figuring out which device(s) to use the rescan command on. I don't know much about either of these tools beyond the fact that they exist. -- Ian From owner-freebsd-hackers@freebsd.org Sat Aug 25 15:29:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDEF1108DC14 for ; Sat, 25 Aug 2018 15:29:33 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A20B75021; Sat, 25 Aug 2018 15:29:33 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 2E91F21CB9; Sat, 25 Aug 2018 11:29:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Sat, 25 Aug 2018 11:29:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=+maz0e1x44SZF1YAJQ96f7tuAM+dR hTNUyQnKoxlvM8=; b=mn/sKb1TXMBbVdsBxQw28EADi6HbRfgh5V6w1QtabfjeR qQX36PcHGvkVgnAb87Wmt9nkUs8i4SqipL4kzY+N7bjggnJM7FyulzbJ3BgMNKlB A9RbFhwKF8XtjhWlL5hVLLzBBXRo4G4jU0s3bsBMzOqnNww6vwE6PfcJT6w8qI8q FKMlHRX9/AY8OHpwuz6H+5igomK2V5Rn3HMarImy5srHp9yOgdbc/T3b/1tjVRSX kfkQTE9g8Yx+he9F5pKd3jCPl0gqSPGKWL5EtKC/WMwFnU8r4vzygxJLy4C+P9zF 4xiawPLHYuampdXbZHW2yoG2GYINjD0PdS56NudgQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=+maz0e 1x44SZF1YAJQ96f7tuAM+dRhTNUyQnKoxlvM8=; b=Y+vMTj+dR7BKC1SjzC6VCf ijqSRogerx72V5nlOh+yD2ampRCg26RVDFkWPSN9FbE2KaWU5bP0UCR5b1WOcsZC bTSCXdCbCbNHaDEIbX6hxnthgiKCLpS5UnFMpZbV5CthzP69OsgZB2jcNZ0D9efq qwdd0xw68aNf7qx1OSBBQ9M0Km9qxNV/4t5l+aVVJQh4JCe5JJPRF4kW2blIXMU4 J/CFrRu9JScja2TTSVUHgYmhJrnWU2ISBnnAKXOJ96TAbmrWl4IkxgqiCy1ay4Xo l4ec1TvFGc2w5CH/IzI7hhz8q4I1lOSO0gpIVpapoOCoxkat6/mw06vH8HJgoIhA == X-ME-Proxy: X-ME-Sender: Received: from odin.yuripv.net (unknown [94.233.224.99]) by mail.messagingengine.com (Postfix) with ESMTPA id 472EDE47BF; Sat, 25 Aug 2018 11:29:32 -0400 (EDT) Subject: Re: pci bus rescan from userland To: Ian Lepore , freebsd-hackers References: <561d2b7b-b36a-7bee-bb23-fb7d0ff9a50b@yuripv.net> <1535210761.1488.45.camel@freebsd.org> From: Yuri Pankov Message-ID: <5441d0a0-63e7-13ae-5e3a-8ea02ee4700c@yuripv.net> Date: Sat, 25 Aug 2018 18:29:31 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <1535210761.1488.45.camel@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 15:29:34 -0000 Ian Lepore wrote: > On Sat, 2018-08-25 at 18:21 +0300, Yuri Pankov wrote: >> Hi, >> >> Trying to enable the HDMI/DP sound output on NVIDIA Quadro M2200, I have >> found the following workaround on NVIDIA Linux forums >> (https://devtalk.nvidia.com/default/topic/1024022/linux/gtx-1060-no-audio-over-hdmi-only-hda-intel-detected-azalia/post/5211273/#5211273): >> >> setpci -s 01:00.0 0x488.l=0x2000000:0x2000000 >> rmmod nvidia-drm nvidia-modeset nvidia >> echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove >> echo 1 > /sys/bus/pci/devices/0000:00:01.0/rescan >> modprobe nvidia-drm >> >> So far rmmod and modprobe steps are obvious, setpci works as well (I >> guess I could use pciconf? doesn't really matter).  Is there a FreeBSD >> equivalent of doing the remove/rescan steps from userland? > > devctl(8) has a rescan function.  devinfo(8) might be useful for > figuring out which device(s) to use the rescan command on. > > I don't know much about either of these tools beyond the fact that they > exist. Thanks, that did it (`devctl rescan pci1`), NVIDIA HDA controller now successfully attached. From owner-freebsd-hackers@freebsd.org Sat Aug 25 20:23:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2C331095607 for ; Sat, 25 Aug 2018 20:23:00 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B5038115F for ; Sat, 25 Aug 2018 20:23:00 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-oi0-x22c.google.com with SMTP id x197-v6so3299015oix.5 for ; Sat, 25 Aug 2018 13:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=rMEkf2y6D86hm0GHxHDm8t2+J+X9dW0Z11UUZ12/YYU=; b=hfevtmJJpMbt8pZtekr4x3QXCfJsW/o7VyD9smHRdUhR4H5aAS2uEE+Sh0WariNIJn Q5NRsV5kO7D1kN7m+tY2ThHFxRtpsuC+1n8zp/2tPVVV42nrDgX/umAmWd+ueno2ZWpr yUCIbdX4BqLTZ69DzcM1abycmAPF16Gvcws/8pqtGazhA7Yq85dR4DJnZf0id4SEq1+C zNDYdimJVNsxqzGM7fkP+7h0mTfyeWFrKVWt2ICk/D3L60hS8nGT7ZGB4unZLQaqgLf3 GSX+QOcHOZvJMdjjgEcW2rUQv/iqp2Xplo/JXmr1HMyxT45un4MJvmnfriM4tCnJbdWU Jpvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=rMEkf2y6D86hm0GHxHDm8t2+J+X9dW0Z11UUZ12/YYU=; b=E9xqqLGQ1jtV5vnqWysdhzKnofQofVpO+CCfKPD7aZ2nbppF+FStMz6IJkec7VbHz3 juEpJ2xaVnBeyz5yme4SOd2MBVYxk7VXDtjdGqB0+X3hnj/aaDRLpuZczt+NNlul+r/F BqBvtrvLI/sKOklZvOk8T21XG1McstEgJpTc/y3PNR+i9WgTjBkF8CFwHAspoiyJqizU IQt6XXTLlMqq8mt0+xELvRPkpIevVUvPgRHcOkp1l8/4LJjQCoua38Ggatdkvinni7Jf ZEyIg0J7vUmwv+ofkliWDIzVv4h4cIUG+Fei5ZiDXZD5g4nReIJvbVQl2QNYID6prIEg TkvA== X-Gm-Message-State: APzg51DdKJ/2cGF7lwZmC24KmydVIMQFvWEBPmLv7P1jUvHHRfNIBTcm oEm/J3UUgiRC7YCoaH2pzSNgheQtZCXUi9cjC1TpD3uF X-Google-Smtp-Source: ANB0VdYPJYk37Pnl1dXxiMmaBaSxU6+AEjxQH8o2IhyqQlofUCkuLOpHm4TDk4LFnkhC3CypdJK6oJ1p71h1I7vvX7s= X-Received: by 2002:aca:4c14:: with SMTP id z20-v6mr6182676oia.164.1535228579276; Sat, 25 Aug 2018 13:22:59 -0700 (PDT) MIME-Version: 1.0 References: <20180825010023.GD45503@funkthat.com> <20180825052330.GE45503@funkthat.com> In-Reply-To: <20180825052330.GE45503@funkthat.com> From: David Cross Date: Sat, 25 Aug 2018 16:22:47 -0400 Message-ID: Subject: Weird USB DA behavior (was: Re: weird geli behavior) To: FreeBSD Hackers Content-Type: multipart/mixed; boundary="0000000000001e1fa005744842c1" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 20:23:01 -0000 --0000000000001e1fa005744842c1 Content-Type: text/plain; charset="UTF-8" Top-posting because this just took a radical change in direction Ok, I need to take some of this back; I do still believe that there is some weird GELI-ness going on (or perhaps GEOM itself), in terms of hangs, but it may be being tripped over by some weird USB-DA behavior I noticed in trying to track this down. I turned debugging ALL the way up on geli (3) and noticed the hangs always happened when geli handed off a 20480 length read to the layer below (in this case mirror).. I used the rebugging output to create a dummy program called 'pread' to try to simulate these failures, and I was quite successful. Attached is 'pread.c' which takes on stdin a offset and a length to read. In that version I overwrote it to always ask for 32k., and that works every time. If I eliminate that line and let the input data control it, on some of the 20480 (and a different set each time) it hangs, after the hang the pread returns _0_ (and obviously no errno), no kernel logs. "pread" is run directly against /dev/da0, so no mirror or geli layers at this point. Kernel attach messages of relevance: da0 at umass-sim0 bus 0 scbus7 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number 574343344532485548465239 da0: 40.000MB/s transfers da0: 3815415MB (976746240 4096 byte sectors) da0: quirks=0x2 Relevant output of a pread run: ~ # ./pread /dev/da0 < requestdata.txt 117535768576(20480)=0(0) 189095018496(20480)=0(0) (last 0 is a trivial patch to include errno output) On Sat, Aug 25, 2018 at 1:23 AM John-Mark Gurney wrote: > David Cross wrote this message on Fri, Aug 24, 2018 at 22:42 -0400: > > > > > > > On Aug 24, 2018, at 21:00, John-Mark Gurney wrote: > > > > > > David Cross wrote this message on Fri, Aug 24, 2018 at 17:54 -0400: > > >> Ok, I am seeing something truely bizzare, I am sending this out as a > shot > > >> across the bow since I am not even sure where or how to begin > debugging > > >> this. > > >> > > >> Some background. This in on an Intel Xeon 5520 based machine, 72G ECC > > >> memory, 11.2, fully patched. Though this has been a problem since at > least > > >> 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, > which it > > >> basically never even touches, even when problems are occuring) > > > > > > I assume you've applied the lazyfpu SA patch? > > > > > > If so, there's another patch you need to apply, see: > > > https://docs.freebsd.org/cgi/mid.cgi?20180821081150.GU2340@kib.kiev.ua > > > > > I will definitely apply this, but I don???t think it applies to the > problem in question. This system doesn???t have AESNI, this problem well > preceded the lazyfpu patch, and i am not seeing any corruption on disk. > > Hmm, k... > > > >> The first symptom was (and I think these are all aspects of the same > root > > >> underlying cause) that fsck on a geli encrypted d stripe of 2 USB > drives > > >> would *randomly* error out on a corrupt entry. Upon investigating > this I > > >> discovered by watching gstat that as this happened the IO on the > drives > > >> would STOP. the L(q) would hover at 1 for a number of seconds, and > then > > >> when it returned fsck was complaining about various corrupt > structures. a > > >> ktrace of fsck shows that it got back data from the pread() that was > > >> partially corrupted (I am guessing, but I cannot confirm that 'some > part' > > >> of the stack handed back a zeroed page, or otherwise 'not the right > data' > > >> that geli dutifully 'decrypted'. No errors are ever logged in the > kernel > > >> about da0 or da1 (the respective underlying USB disks). It *seems* > this is > > >> *always* on phase 2 of fsck (files and paths), and its never the same > > >> inode. no data is *ever* corrupted when in the filesystem, no matter > how > > >> hard I hit the disks (all data on these devices is fully checksummed) > > >> Devices have passed multiple SMART full diag checks, full read/write > tests > > >> with no issues. Under heavy FS IO it does occasionally lock.. but > > >> recovers, and again data and filesystem are fully consistent. > > >> > > >> I was willing to live with that.. weird as it was (these are backup > disks, > > >> data is fully checksummed, and I was only fscking out of extreme > paranoia > > >> every reboot) Then I added an internal drive, configured with gmirror > > >> (broken mirror currently, second disk hasn't been added) and geli. > On this > > >> disk I have a postgres 10 database in WAL replication. This was > working > > >> fine and then the other day the system just locked for a few hours. > During > > >> that time I saw the L(q) of the _internal_ disk in the 10,000+ range, > and > > >> it doing _1_ operation a second to the underlying disk... all the > while > > >> geli is logging 'error 11' to the console (nothing about the > underlying > > >> disk) After this happened a static file on the disk (a zip file) had > bad > > >> data in the middle of a page (after reboot the file was ok.. so it > was > > >> just in cache). Again, this disk fully checks ok, no corruption on > the > > >> disk, no errors from the disk itself. > > >> > > >> > > >> Halp? where do I even begin with this? It really feels like there > is > > >> some massive locking going on in geli in some way? Where should I > even > > >> begin looking? I run geli on most of my systems and don't have any > issues. > > Can you post actual log lines? geli has lots of error log lines, so w/o > more info, pretty hard to say WHAT in geli is returning EAGAIN. I do see > that _read_done and _write_done may not handle an EAGAIN error, which could > cause this problem, but to confirm, I need the actual log lines... > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > --0000000000001e1fa005744842c1 Content-Type: text/plain; charset="US-ASCII"; name="requestdata.txt" Content-Disposition: attachment; filename="requestdata.txt" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_jl9v693h1 MjA5MDU5ODQgNDA5NgoyMDkxMDA4MCA0MDk2CjIwOTM4NzUyIDQwOTYKMjEwMzcwNTYgMTYzODQK MjExMzUzNjAgMTYzODQKMjEyMzM2NjQgMTYzODQKMjEzMzE5NjggMTYzODQKMjE0MzAyNzIgMTYz ODQKNjc3MzE0NTYwIDE2Mzg0CjY3NzQxMjg2NCAxNjM4NAo2Nzc1MTExNjggMTYzODQKNjc3NjA5 NDcyIDE2Mzg0CjY3NzcwNzc3NiAxNjM4NAoxMzMzODIxNDQwIDE2Mzg0CjEzMzM5MTk3NDQgMTYz ODQKMTMzNDAxODA0OCAxNjM4NAoxMzM0MTE2MzUyIDE2Mzg0CjEzMzQyMTQ2NTYgMTYzODQKMTk5 MDMyODMyMCAxNjM4NAoxOTkwNDI2NjI0IDE2Mzg0CjE5OTA1MjQ5MjggMTYzODQKMTk5MDYyMzIz MiAxNjM4NAoxOTkwNzIxNTM2IDE2Mzg0CjI2NDY4MzUyMDAgMTYzODQKMjY0NjkzMzUwNCAxNjM4 NAoyNjQ3MDMxODA4IDE2Mzg0CjI2NDcxMzAxMTIgMTYzODQKMjY0NzIyODQxNiAxNjM4NAozMzAz MzQyMDgwIDE2Mzg0CjMzMDM0NDAzODQgMTYzODQKMzMwMzUzODY4OCAxNjM4NAozMzAzNjM2OTky IDE2Mzg0CjMzMDM3MzUyOTYgMTYzODQKMzk1OTg0ODk2MCAxNjM4NAozOTU5OTQ3MjY0IDE2Mzg0 CjM5NjAwNDU1NjggMTYzODQKMzk2MDE0Mzg3MiAxNjM4NAozOTYwMjQyMTc2IDE2Mzg0CjQ2MTYz NTU4NDAgMTYzODQKNDYxNjQ1NDE0NCAxNjM4NAo0NjE2NTUyNDQ4IDE2Mzg0CjQ2MTY2NTA3NTIg MTYzODQKNDYxNjc0OTA1NiAxNjM4NAo1MjcyODYyNzIwIDE2Mzg0CjUyNzI5NjEwMjQgMTYzODQK NTI3MzA1OTMyOCAxNjM4NAo1MjczMTU3NjMyIDE2Mzg0CjUyNzMyNTU5MzYgMTYzODQKNTkyOTM2 OTYwMCAxNjM4NAo1OTI5NDY3OTA0IDE2Mzg0CjU5Mjk1NjYyMDggMTYzODQKNTkyOTY2NDUxMiAx NjM4NAo1OTI5NzYyODE2IDE2Mzg0CjY1ODU4NzY0ODAgMTYzODQKNjU4NTk3NDc4NCAxNjM4NAo2 NTg2MDczMDg4IDE2Mzg0CjY1ODYxNzEzOTIgMTYzODQKNjU4NjI2OTY5NiAxNjM4NAo3MjQyMzgz MzYwIDE2Mzg0CjcyNDI0ODE2NjQgMTYzODQKNzI0MjU3OTk2OCAxNjM4NAo3MjQyNjc4MjcyIDE2 Mzg0CjcyNDI3NzY1NzYgMTYzODQKNzg5ODg5MDI0MCAxNjM4NAo3ODk4OTg4NTQ0IDE2Mzg0Cjc4 OTkwODY4NDggMTYzODQKNzg5OTE4NTE1MiAxNjM4NAo3ODk5MjgzNDU2IDE2Mzg0Cjg1NTUzOTcx MjAgMTYzODQKODU1NTQ5NTQyNCAxNjM4NAo4NTU1NTkzNzI4IDE2Mzg0Cjg1NTU2OTIwMzIgMTYz ODQKODU1NTgyMzEwNCAyMDQ4MAo5MjExOTA0MDAwIDE2Mzg0CjkyMTIwMDIzMDQgMTYzODQKOTIx MjEwMDYwOCAxNjM4NAo5MjEyMTk4OTEyIDE2Mzg0CjkyMTIyOTcyMTYgMTYzODQKOTg2ODQxMDg4 MCAxNjM4NAo5ODY4NTA5MTg0IDE2Mzg0Cjk4Njg2MDc0ODggMTYzODQKOTg2ODcwNTc5MiAxNjM4 NAo5ODY4ODA0MDk2IDE2Mzg0CjEwNTI0OTE3NzYwIDE2Mzg0CjEwNTI1MDE2MDY0IDE2Mzg0CjEw NTI1MTE0MzY4IDE2Mzg0CjEwNTI1MjEyNjcyIDE2Mzg0CjEwNTI1MzEwOTc2IDE2Mzg0CjExMTgx NDI0NjQwIDE2Mzg0CjExMTgxNTIyOTQ0IDE2Mzg0CjExMTgxNjIxMjQ4IDE2Mzg0CjExMTgxNzE5 NTUyIDE2Mzg0CjExMTgxODE3ODU2IDE2Mzg0CjExODM3OTMxNTIwIDE2Mzg0CjExODM4MDI5ODI0 IDE2Mzg0CjExODM4MTI4MTI4IDE2Mzg0CjExODM4MjI2NDMyIDE2Mzg0CjExODM4MzI0NzM2IDE2 Mzg0CjEyNDk0NDM4NDAwIDE2Mzg0CjEyNDk0NTM2NzA0IDE2Mzg0CjEyNDk0NjM1MDA4IDE2Mzg0 CjEyNDk0NzMzMzEyIDE2Mzg0CjEyNDk0ODMxNjE2IDE2Mzg0CjEzMTUwOTQ1MjgwIDE2Mzg0CjEz MTUxMDQzNTg0IDE2Mzg0CjEzMTUxMTQxODg4IDE2Mzg0CjEzMTUxMjQwMTkyIDE2Mzg0CjEzMTUx MzM4NDk2IDE2Mzg0CjEzODA3NDUyMTYwIDE2Mzg0CjEzODA3NTUwNDY0IDE2Mzg0CjEzODA3NjQ4 NzY4IDE2Mzg0CjEzODA3NzQ3MDcyIDE2Mzg0CjEzODA3ODQ1Mzc2IDE2Mzg0CjE0NDYzOTU5MDQw IDE2Mzg0CjE0NDY0MDU3MzQ0IDE2Mzg0CjE0NDY0MTU1NjQ4IDE2Mzg0CjE0NDY0MjUzOTUyIDE2 Mzg0CjE0NDY0MzUyMjU2IDE2Mzg0CjE1MTIwNDY1OTIwIDE2Mzg0CjE1MTIwNTY0MjI0IDE2Mzg0 CjE1MTIwNjYyNTI4IDE2Mzg0CjE1MTIwNzYwODMyIDE2Mzg0CjE1MTIwODU5MTM2IDE2Mzg0CjE1 Nzc2OTcyODAwIDE2Mzg0CjE1Nzc3MDcxMTA0IDE2Mzg0CjE1Nzc3MTY5NDA4IDE2Mzg0CjE1Nzc3 MjY3NzEyIDE2Mzg0CjE1Nzc3MzY2MDE2IDE2Mzg0CjE2NDMzNDc5NjgwIDE2Mzg0CjE2NDMzNTc3 OTg0IDE2Mzg0CjE2NDMzNjc2Mjg4IDE2Mzg0CjE2NDMzNzc0NTkyIDE2Mzg0CjE2NDMzODcyODk2 IDE2Mzg0CjE3MDg5OTg2NTYwIDE2Mzg0CjE3MDkwMDg0ODY0IDE2Mzg0CjE3MDkwMTgzMTY4IDE2 Mzg0CjE3MDkwMjgxNDcyIDE2Mzg0CjE3MDkwMzc5Nzc2IDE2Mzg0CjE3NzQ2NDkzNDQwIDE2Mzg0 CjE3NzQ2NTkxNzQ0IDE2Mzg0CjE3NzQ2NjkwMDQ4IDE2Mzg0CjE3NzQ2Nzg4MzUyIDE2Mzg0CjE3 NzQ2ODg2NjU2IDE2Mzg0CjE3NzQ2OTg0OTYwIDE2Mzg0CjE4NDAzMDAwMzIwIDE2Mzg0CjE4NDAz MDk4NjI0IDE2Mzg0CjE4NDAzMTk2OTI4IDE2Mzg0CjE4NDAzMjk1MjMyIDE2Mzg0CjE4NDAzMzkz NTM2IDE2Mzg0CjE4NDAzNTI0NjA4IDIwNDgwCjE4NDAzNjIyOTEyIDE2Mzg0CjE5MDU5NTA3MjAw IDE2Mzg0CjE5MDU5NjA1NTA0IDE2Mzg0CjE5MDU5NzAzODA4IDE2Mzg0CjE5MDU5ODAyMTEyIDE2 Mzg0CjE5MDU5OTAwNDE2IDE2Mzg0CjE5MDU5OTk4NzIwIDE2Mzg0CjE5MDYwMDk3MDI0IDE2Mzg0 CjE5NzE2MDE0MDgwIDE2Mzg0CjE5NzE2MTEyMzg0IDE2Mzg0CjE5NzE2MjEwNjg4IDE2Mzg0CjE5 NzE2MzA4OTkyIDE2Mzg0CjE5NzE2NDA3Mjk2IDE2Mzg0CjIwMzcyNTIwOTYwIDE2Mzg0CjIwMzcy NjE5MjY0IDE2Mzg0CjIwMzcyNzE3NTY4IDE2Mzg0CjIwMzcyODE1ODcyIDE2Mzg0CjIwMzcyOTE0 MTc2IDE2Mzg0CjIxMDI5MDI3ODQwIDE2Mzg0CjIxMDI5MTI2MTQ0IDE2Mzg0CjIxMDI5MjI0NDQ4 IDE2Mzg0CjIxMDI5MzIyNzUyIDE2Mzg0CjIxMDI5NDIxMDU2IDE2Mzg0CjIxNjg1NTM0NzIwIDE2 Mzg0CjIxNjg1NjMzMDI0IDE2Mzg0CjIxNjg1NzMxMzI4IDE2Mzg0CjIxNjg1ODI5NjMyIDE2Mzg0 CjIxNjg1OTI3OTM2IDE2Mzg0CjIyMzQyMDQxNjAwIDE2Mzg0CjIyMzQyMTM5OTA0IDE2Mzg0CjIy MzQyMjM4MjA4IDE2Mzg0CjIyMzQyMzM2NTEyIDE2Mzg0CjIyMzQyNDM0ODE2IDE2Mzg0CjIyOTk4 NTQ4NDgwIDE2Mzg0CjIyOTk4NjQ2Nzg0IDE2Mzg0CjIyOTk4NzQ1MDg4IDE2Mzg0CjIyOTk4ODQz MzkyIDE2Mzg0CjIyOTk4OTQxNjk2IDE2Mzg0CjIzNjU1MDU1MzYwIDE2Mzg0CjIzNjU1MTUzNjY0 IDE2Mzg0CjIzNjU1MjUxOTY4IDE2Mzg0CjIzNjU1MzgzMDQwIDIwNDgwCjIzNjU1NDgxMzQ0IDE2 Mzg0CjI0MzExNTYyMjQwIDE2Mzg0CjI0MzExNjYwNTQ0IDE2Mzg0CjI0MzExNzU4ODQ4IDE2Mzg0 CjI0MzExODU3MTUyIDE2Mzg0CjI0MzExOTU1NDU2IDE2Mzg0CjI0OTY4MDY5MTIwIDE2Mzg0CjI0 OTY4MTY3NDI0IDE2Mzg0CjI0OTY4MjY1NzI4IDE2Mzg0CjI0OTY4MzY0MDMyIDE2Mzg0CjI0OTY4 NDYyMzM2IDE2Mzg0CjI1NjI0NTc2MDAwIDE2Mzg0CjI1NjI0Njc0MzA0IDE2Mzg0CjI1NjI0Nzcy NjA4IDE2Mzg0CjI1NjI0ODcwOTEyIDE2Mzg0CjI1NjI0OTY5MjE2IDE2Mzg0CjI2MjgxMDgyODgw IDE2Mzg0CjI2MjgxMTgxMTg0IDE2Mzg0CjI2MjgxMjc5NDg4IDE2Mzg0CjI2MjgxMzc3NzkyIDE2 Mzg0CjI2MjgxNDc2MDk2IDE2Mzg0CjI2OTM3NTg5NzYwIDE2Mzg0CjI2OTM3Njg4MDY0IDE2Mzg0 CjI2OTM3Nzg2MzY4IDE2Mzg0CjI2OTM3ODg0NjcyIDE2Mzg0CjI2OTM3OTgyOTc2IDE2Mzg0CjI3 NTk0MDk2NjQwIDE2Mzg0CjI3NTk0MTk0OTQ0IDE2Mzg0CjI3NTk0MjkzMjQ4IDE2Mzg0CjI3NTk0 MzkxNTUyIDE2Mzg0CjI3NTk0NDg5ODU2IDE2Mzg0CjI4MjUwNjAzNTIwIDE2Mzg0CjI4MjUwNzAx ODI0IDE2Mzg0CjI4MjUwODAwMTI4IDE2Mzg0CjI4MjUwODk4NDMyIDE2Mzg0CjI4MjUwOTk2NzM2 IDE2Mzg0CjI4OTA3MTEwNDAwIDE2Mzg0CjI4OTA3MjA4NzA0IDE2Mzg0CjI4OTA3MzA3MDA4IDE2 Mzg0CjI4OTA3NDA1MzEyIDE2Mzg0CjI4OTA3NTAzNjE2IDE2Mzg0CjI5NTYzNjE3MjgwIDE2Mzg0 CjI5NTYzNzE1NTg0IDE2Mzg0CjI5NTYzODEzODg4IDE2Mzg0CjI5NTYzOTEyMTkyIDE2Mzg0CjI5 NTY0MDEwNDk2IDE2Mzg0CjMwMjIwMTI0MTYwIDE2Mzg0CjMwMjIwMjIyNDY0IDE2Mzg0CjMwMjIw MzIwNzY4IDE2Mzg0CjMwMjIwNDE5MDcyIDE2Mzg0CjMwMjIwNTE3Mzc2IDE2Mzg0CjMwODc2NjMx MDQwIDE2Mzg0CjMwODc2NzI5MzQ0IDE2Mzg0CjMwODc2ODI3NjQ4IDE2Mzg0CjMwODc2OTI1OTUy IDE2Mzg0CjMwODc3MDI0MjU2IDE2Mzg0CjMxNTMzMTM3OTIwIDE2Mzg0CjMxNTMzMjM2MjI0IDE2 Mzg0CjMxNTMzMzM0NTI4IDE2Mzg0CjMxNTMzNDMyODMyIDE2Mzg0CjMxNTMzNTMxMTM2IDE2Mzg0 CjMyMTg5NjQ0ODAwIDE2Mzg0CjMyMTg5NzQzMTA0IDE2Mzg0CjMyMTg5ODQxNDA4IDE2Mzg0CjMy MTg5OTM5NzEyIDE2Mzg0CjMyMTkwMDM4MDE2IDE2Mzg0CjMyODQ2MTUxNjgwIDE2Mzg0CjMyODQ2 MjQ5OTg0IDE2Mzg0CjMyODQ2MzQ4Mjg4IDE2Mzg0CjMyODQ2NDQ2NTkyIDE2Mzg0CjMyODQ2NTQ0 ODk2IDE2Mzg0CjMzNTAyNjU4NTYwIDE2Mzg0CjMzNTAyNzU2ODY0IDE2Mzg0CjMzNTAyODU1MTY4 IDE2Mzg0CjMzNTAyOTUzNDcyIDE2Mzg0CjMzNTAzMDUxNzc2IDE2Mzg0CjM0MTU5MTY1NDQwIDE2 Mzg0CjM0MTU5MjYzNzQ0IDE2Mzg0CjM0MTU5MzYyMDQ4IDE2Mzg0CjM0MTU5NDYwMzUyIDE2Mzg0 CjM0MTU5NTU4NjU2IDE2Mzg0CjM0ODE1NjcyMzIwIDE2Mzg0CjM0ODE1NzcwNjI0IDE2Mzg0CjM0 ODE1ODY4OTI4IDE2Mzg0CjM0ODE1OTY3MjMyIDE2Mzg0CjM0ODE2MDY1NTM2IDE2Mzg0CjM1NDcy MTc5MjAwIDE2Mzg0CjM1NDcyMjc3NTA0IDE2Mzg0CjM1NDcyMzc1ODA4IDE2Mzg0CjM1NDcyNDc0 MTEyIDE2Mzg0CjM1NDcyNTcyNDE2IDE2Mzg0CjM2MTI4Njg2MDgwIDE2Mzg0CjM2MTI4Nzg0Mzg0 IDE2Mzg0CjM2MTI4ODgyNjg4IDE2Mzg0CjM2MTI4OTgwOTkyIDE2Mzg0CjM2MTI5MDc5Mjk2IDE2 Mzg0CjM2Nzg1MTkyOTYwIDE2Mzg0CjM2Nzg1MjkxMjY0IDE2Mzg0CjM2Nzg1Mzg5NTY4IDE2Mzg0 CjM2Nzg1NDg3ODcyIDE2Mzg0CjM2Nzg1NTg2MTc2IDE2Mzg0CjM3NDQxNjk5ODQwIDE2Mzg0CjM3 NDQxNzk4MTQ0IDE2Mzg0CjM3NDQxODk2NDQ4IDE2Mzg0CjM3NDQxOTk0NzUyIDE2Mzg0CjM3NDQy MDkzMDU2IDE2Mzg0CjM4MDk4MjA2NzIwIDE2Mzg0CjM4MDk4MzA1MDI0IDE2Mzg0CjM4MDk4NDAz MzI4IDE2Mzg0CjM4MDk4NTAxNjMyIDE2Mzg0CjM4MDk4NjMyNzA0IDIwNDgwCjM4NzU0NzEzNjAw IDE2Mzg0CjM4NzU0ODExOTA0IDE2Mzg0CjM4NzU0OTEwMjA4IDE2Mzg0CjM4NzU1MDA4NTEyIDE2 Mzg0CjM4NzU1MTA2ODE2IDE2Mzg0CjM5NDExMjIwNDgwIDE2Mzg0CjM5NDExMzE4Nzg0IDE2Mzg0 CjM5NDExNDE3MDg4IDE2Mzg0CjM5NDExNTE1MzkyIDE2Mzg0CjM5NDExNjEzNjk2IDE2Mzg0CjQw MDY3NzI3MzYwIDE2Mzg0CjQwMDY3ODI1NjY0IDE2Mzg0CjQwMDY3OTIzOTY4IDE2Mzg0CjQwMDY4 MDIyMjcyIDE2Mzg0CjQwMDY4MTIwNTc2IDE2Mzg0CjQwNzI0MjM0MjQwIDE2Mzg0CjQwNzI0MzMy NTQ0IDE2Mzg0CjQwNzI0NDMwODQ4IDE2Mzg0CjQwNzI0NTI5MTUyIDE2Mzg0CjQwNzI0NjI3NDU2 IDE2Mzg0CjQxMzgwNzQxMTIwIDE2Mzg0CjQxMzgwODM5NDI0IDE2Mzg0CjQxMzgwOTM3NzI4IDE2 Mzg0CjQxMzgxMDM2MDMyIDE2Mzg0CjQxMzgxMTM0MzM2IDE2Mzg0CjQyMDM3MjQ4MDAwIDE2Mzg0 CjQyMDM3MzQ2MzA0IDE2Mzg0CjQyMDM3NDQ0NjA4IDE2Mzg0CjQyMDM3NTQyOTEyIDE2Mzg0CjQy MDM3NjczOTg0IDIwNDgwCjQyNjkzNzU0ODgwIDE2Mzg0CjQyNjkzODUzMTg0IDE2Mzg0CjQyNjkz OTUxNDg4IDE2Mzg0CjQyNjk0MDQ5NzkyIDE2Mzg0CjQyNjk0MTQ4MDk2IDE2Mzg0CjQzMzUwMjYx NzYwIDE2Mzg0CjQzMzUwMzYwMDY0IDE2Mzg0CjQzMzUwNDU4MzY4IDE2Mzg0CjQzMzUwNTU2Njcy IDE2Mzg0CjQzMzUwNjU0OTc2IDE2Mzg0CjQ0MDA2NzY4NjQwIDE2Mzg0CjQ0MDA2ODY2OTQ0IDE2 Mzg0CjQ0MDA2OTY1MjQ4IDE2Mzg0CjQ0MDA3MDYzNTUyIDE2Mzg0CjQ0MDA3MTYxODU2IDE2Mzg0 CjQ0NjYzMjc1NTIwIDE2Mzg0CjQ0NjYzMzczODI0IDE2Mzg0CjQ0NjYzNDcyMTI4IDE2Mzg0CjQ0 NjYzNTcwNDMyIDE2Mzg0CjQ0NjYzNjY4NzM2IDE2Mzg0CjQ1MzE5NzgyNDAwIDE2Mzg0CjQ1MzE5 ODgwNzA0IDE2Mzg0CjQ1MzE5OTc5MDA4IDE2Mzg0CjQ1MzIwMDc3MzEyIDE2Mzg0CjQ1MzIwMTc1 NjE2IDE2Mzg0CjQ1OTc2Mjg5MjgwIDE2Mzg0CjQ1OTc2Mzg3NTg0IDE2Mzg0CjQ1OTc2NDg1ODg4 IDE2Mzg0CjQ1OTc2NTg0MTkyIDE2Mzg0CjQ1OTc2NjgyNDk2IDE2Mzg0CjQ2NjMyNzk2MTYwIDE2 Mzg0CjQ2NjMyODk0NDY0IDE2Mzg0CjQ2NjMyOTkyNzY4IDE2Mzg0CjQ2NjMzMDkxMDcyIDE2Mzg0 CjQ2NjMzMTg5Mzc2IDE2Mzg0CjQ3Mjg5MzAzMDQwIDE2Mzg0CjQ3Mjg5NDAxMzQ0IDE2Mzg0CjQ3 Mjg5NDk5NjQ4IDE2Mzg0CjQ3Mjg5NTk3OTUyIDE2Mzg0CjQ3Mjg5Njk2MjU2IDE2Mzg0CjQ3OTQ1 ODA5OTIwIDE2Mzg0CjQ3OTQ1OTA4MjI0IDE2Mzg0CjQ3OTQ2MDA2NTI4IDE2Mzg0CjQ3OTQ2MTA0 ODMyIDE2Mzg0CjQ3OTQ2MjM1OTA0IDIwNDgwCjQ3OTQ2MzM0MjA4IDE2Mzg0CjQ4NjAyMzE2ODAw IDE2Mzg0CjQ4NjAyNDE1MTA0IDE2Mzg0CjQ4NjAyNTEzNDA4IDE2Mzg0CjQ4NjAyNjExNzEyIDE2 Mzg0CjQ4NjAyNzEwMDE2IDE2Mzg0CjQ5MjU4ODIzNjgwIDE2Mzg0CjQ5MjU4OTIxOTg0IDE2Mzg0 CjQ5MjU5MDIwMjg4IDE2Mzg0CjQ5MjU5MTE4NTkyIDE2Mzg0CjQ5MjU5MjE2ODk2IDE2Mzg0CjQ5 OTE1MzMwNTYwIDE2Mzg0CjQ5OTE1NDI4ODY0IDE2Mzg0CjQ5OTE1NTI3MTY4IDE2Mzg0CjQ5OTE1 NjI1NDcyIDE2Mzg0CjQ5OTE1NzIzNzc2IDE2Mzg0CjUwNTcxODM3NDQwIDE2Mzg0CjUwNTcxOTM1 NzQ0IDE2Mzg0CjUwNTcyMDM0MDQ4IDE2Mzg0CjUwNTcyMTMyMzUyIDE2Mzg0CjUwNTcyMjMwNjU2 IDE2Mzg0CjUxMjI4MzQ0MzIwIDE2Mzg0CjUxMjI4NDQyNjI0IDE2Mzg0CjUxMjI4NTQwOTI4IDE2 Mzg0CjUxMjI4NjM5MjMyIDE2Mzg0CjUxMjI4NzcwMzA0IDIwNDgwCjUxODg0ODUxMjAwIDE2Mzg0 CjUxODg0OTQ5NTA0IDE2Mzg0CjUxODg1MDQ3ODA4IDE2Mzg0CjUxODg1MTQ2MTEyIDE2Mzg0CjUx ODg1MjQ0NDE2IDE2Mzg0CjUyNTQxMzU4MDgwIDE2Mzg0CjUyNTQxNDU2Mzg0IDE2Mzg0CjUyNTQx NTU0Njg4IDE2Mzg0CjUyNTQxNjUyOTkyIDE2Mzg0CjUyNTQxNzUxMjk2IDE2Mzg0CjUzMTk3ODY0 OTYwIDE2Mzg0CjUzMTk3OTYzMjY0IDE2Mzg0CjUzMTk4MDYxNTY4IDE2Mzg0CjUzMTk4MTU5ODcy IDE2Mzg0CjUzMTk4MjU4MTc2IDE2Mzg0CjUzODU0MzcxODQwIDE2Mzg0CjUzODU0NDcwMTQ0IDE2 Mzg0CjUzODU0NTY4NDQ4IDE2Mzg0CjUzODU0NjY2NzUyIDE2Mzg0CjUzODU0NzY1MDU2IDE2Mzg0 CjU0NTEwODc4NzIwIDE2Mzg0CjU0NTEwOTc3MDI0IDE2Mzg0CjU0NTExMDc1MzI4IDE2Mzg0CjU0 NTExMTczNjMyIDE2Mzg0CjU0NTExMjcxOTM2IDE2Mzg0CjU1MTY3Mzg1NjAwIDE2Mzg0CjU1MTY3 NDgzOTA0IDE2Mzg0CjU1MTY3NTgyMjA4IDE2Mzg0CjU1MTY3NjgwNTEyIDE2Mzg0CjU1MTY3Nzc4 ODE2IDE2Mzg0CjU1ODIzODkyNDgwIDE2Mzg0CjU1ODIzOTkwNzg0IDE2Mzg0CjU1ODI0MDg5MDg4 IDE2Mzg0CjU1ODI0MTg3MzkyIDE2Mzg0CjU1ODI0Mjg1Njk2IDE2Mzg0CjU2NDgwMzk5MzYwIDE2 Mzg0CjU2NDgwNDk3NjY0IDE2Mzg0CjU2NDgwNTk1OTY4IDE2Mzg0CjU2NDgwNjk0MjcyIDE2Mzg0 CjU2NDgwNzkyNTc2IDE2Mzg0CjU3MTM2OTA2MjQwIDE2Mzg0CjU3MTM3MDA0NTQ0IDE2Mzg0CjU3 MTM3MTAyODQ4IDE2Mzg0CjU3MTM3MjAxMTUyIDE2Mzg0CjU3MTM3Mjk5NDU2IDE2Mzg0CjU3Nzkz NDEzMTIwIDE2Mzg0CjU3NzkzNTExNDI0IDE2Mzg0CjU3NzkzNjA5NzI4IDE2Mzg0CjU3NzkzNzA4 MDMyIDE2Mzg0CjU3NzkzODA2MzM2IDE2Mzg0CjU4NDQ5OTIwMDAwIDE2Mzg0CjU4NDUwMDE4MzA0 IDE2Mzg0CjU4NDUwMTE2NjA4IDE2Mzg0CjU4NDUwMjE0OTEyIDE2Mzg0CjU4NDUwMzEzMjE2IDE2 Mzg0CjU5MTA2NDI2ODgwIDE2Mzg0CjU5MTA2NTI1MTg0IDE2Mzg0CjU5MTA2NjIzNDg4IDE2Mzg0 CjU5MTA2NzIxNzkyIDE2Mzg0CjU5MTA2ODUyODY0IDIwNDgwCjU5NzYyOTMzNzYwIDE2Mzg0CjU5 NzYzMDMyMDY0IDE2Mzg0CjU5NzYzMTMwMzY4IDE2Mzg0CjU5NzYzMjI4NjcyIDE2Mzg0CjU5NzYz MzI2OTc2IDE2Mzg0CjYwNDE5NDQwNjQwIDE2Mzg0CjYwNDE5NTM4OTQ0IDE2Mzg0CjYwNDE5NjM3 MjQ4IDE2Mzg0CjYwNDE5NzM1NTUyIDE2Mzg0CjYwNDE5ODY2NjI0IDIwNDgwCjYxMDc1OTQ3NTIw IDE2Mzg0CjYxMDc2MDQ1ODI0IDE2Mzg0CjYxMDc2MTQ0MTI4IDE2Mzg0CjYxMDc2MjQyNDMyIDE2 Mzg0CjYxMDc2MzQwNzM2IDE2Mzg0CjYxNzMyNDU0NDAwIDE2Mzg0CjYxNzMyNTUyNzA0IDE2Mzg0 CjYxNzMyNjUxMDA4IDE2Mzg0CjYxNzMyNzQ5MzEyIDE2Mzg0CjYxNzMyODQ3NjE2IDE2Mzg0CjYy Mzg4OTYxMjgwIDE2Mzg0CjYyMzg5MDU5NTg0IDE2Mzg0CjYyMzg5MTU3ODg4IDE2Mzg0CjYyMzg5 MjU2MTkyIDE2Mzg0CjYyMzg5MzU0NDk2IDE2Mzg0CjYzMDQ1NDY4MTYwIDE2Mzg0CjYzMDQ1NTY2 NDY0IDE2Mzg0CjYzMDQ1NjY0NzY4IDE2Mzg0CjYzMDQ1NzYzMDcyIDE2Mzg0CjYzMDQ1ODYxMzc2 IDE2Mzg0CjYzNzAxOTc1MDQwIDE2Mzg0CjYzNzAyMDczMzQ0IDE2Mzg0CjYzNzAyMTcxNjQ4IDE2 Mzg0CjYzNzAyMjY5OTUyIDE2Mzg0CjYzNzAyMzY4MjU2IDE2Mzg0CjY0MzU4NDgxOTIwIDE2Mzg0 CjY0MzU4NTgwMjI0IDE2Mzg0CjY0MzU4Njc4NTI4IDE2Mzg0CjY0MzU4Nzc2ODMyIDE2Mzg0CjY0 MzU4ODc1MTM2IDE2Mzg0CjY1MDE0OTg4ODAwIDE2Mzg0CjY1MDE1MDg3MTA0IDE2Mzg0CjY1MDE1 MTg1NDA4IDE2Mzg0CjY1MDE1MjgzNzEyIDE2Mzg0CjY1MDE1MzgyMDE2IDE2Mzg0CjY1NjcxNDk1 NjgwIDE2Mzg0CjY1NjcxNTkzOTg0IDE2Mzg0CjY1NjcxNjkyMjg4IDE2Mzg0CjY1NjcxNzkwNTky IDE2Mzg0CjY1NjcxODg4ODk2IDE2Mzg0CjY2MzI4MDAyNTYwIDE2Mzg0CjY2MzI4MTAwODY0IDE2 Mzg0CjY2MzI4MTk5MTY4IDE2Mzg0CjY2MzI4Mjk3NDcyIDE2Mzg0CjY2MzI4Mzk1Nzc2IDE2Mzg0 CjY2OTg0NTA5NDQwIDE2Mzg0CjY2OTg0NjA3NzQ0IDE2Mzg0CjY2OTg0NzA2MDQ4IDE2Mzg0CjY2 OTg0ODA0MzUyIDE2Mzg0CjY2OTg0OTAyNjU2IDE2Mzg0CjY3NjQxMDE2MzIwIDE2Mzg0CjY3NjQx MTE0NjI0IDE2Mzg0CjY3NjQxMjEyOTI4IDE2Mzg0CjY3NjQxMzExMjMyIDE2Mzg0CjY3NjQxNDA5 NTM2IDE2Mzg0CjY3NjQxNTA3ODQwIDE2Mzg0CjY4Mjk3NTIzMjAwIDE2Mzg0CjY4Mjk3NjIxNTA0 IDE2Mzg0CjY4Mjk3NzE5ODA4IDE2Mzg0CjY4Mjk3ODE4MTEyIDE2Mzg0CjY4Mjk3OTE2NDE2IDE2 Mzg0CjY4OTU0MDMwMDgwIDE2Mzg0CjY4OTU0MTI4Mzg0IDE2Mzg0CjY4OTU0MjI2Njg4IDE2Mzg0 CjY4OTU0MzI0OTkyIDE2Mzg0CjY4OTU0NDIzMjk2IDE2Mzg0CjY5NjEwNTM2OTYwIDE2Mzg0CjY5 NjEwNjM1MjY0IDE2Mzg0CjY5NjEwNzMzNTY4IDE2Mzg0CjY5NjEwODMxODcyIDE2Mzg0CjY5NjEw OTMwMTc2IDE2Mzg0CjY5NjExMDI4NDgwIDE2Mzg0CjY5NjExMTI2Nzg0IDE2Mzg0CjcwMjY3MDQz ODQwIDE2Mzg0CjcwMjY3MTQyMTQ0IDE2Mzg0CjcwMjY3MjQwNDQ4IDE2Mzg0CjcwMjY3MzM4NzUy IDE2Mzg0CjcwMjY3NDM3MDU2IDE2Mzg0CjcwOTIzNTUwNzIwIDE2Mzg0CjcwOTIzNjQ5MDI0IDE2 Mzg0CjcwOTIzNzQ3MzI4IDE2Mzg0CjcwOTIzODQ1NjMyIDE2Mzg0CjcwOTIzOTQzOTM2IDE2Mzg0 CjcxNTgwMDU3NjAwIDE2Mzg0CjcxNTgwMTU1OTA0IDE2Mzg0CjcxNTgwMjU0MjA4IDE2Mzg0Cjcx NTgwMzUyNTEyIDE2Mzg0CjcxNTgwNDUwODE2IDE2Mzg0CjcyMjM2NTY0NDgwIDE2Mzg0CjcyMjM2 NjYyNzg0IDE2Mzg0CjcyMjM2NzYxMDg4IDE2Mzg0CjcyMjM2ODU5MzkyIDE2Mzg0CjcyMjM2OTU3 Njk2IDE2Mzg0CjcyODkzMDcxMzYwIDE2Mzg0CjcyODkzMTY5NjY0IDE2Mzg0CjcyODkzMjY3OTY4 IDE2Mzg0CjcyODkzMzY2MjcyIDE2Mzg0CjcyODkzNDY0NTc2IDE2Mzg0CjczNTQ5NTc4MjQwIDE2 Mzg0CjczNTQ5Njc2NTQ0IDE2Mzg0CjczNTQ5Nzc0ODQ4IDE2Mzg0CjczNTQ5ODczMTUyIDE2Mzg0 CjczNTQ5OTcxNDU2IDE2Mzg0Cjc0MjA2MDg1MTIwIDE2Mzg0Cjc0MjA2MTgzNDI0IDE2Mzg0Cjc0 MjA2MjgxNzI4IDE2Mzg0Cjc0MjA2MzgwMDMyIDE2Mzg0Cjc0MjA2NDc4MzM2IDE2Mzg0Cjc0ODYy NTkyMDAwIDE2Mzg0Cjc0ODYyNjkwMzA0IDE2Mzg0Cjc0ODYyNzg4NjA4IDE2Mzg0Cjc0ODYyODg2 OTEyIDE2Mzg0Cjc0ODYyOTg1MjE2IDE2Mzg0Cjc1NTE5MDk4ODgwIDE2Mzg0Cjc1NTE5MTk3MTg0 IDE2Mzg0Cjc1NTE5Mjk1NDg4IDE2Mzg0Cjc1NTE5MzkzNzkyIDE2Mzg0Cjc1NTE5NDkyMDk2IDE2 Mzg0Cjc2MTc1NjA1NzYwIDE2Mzg0Cjc2MTc1NzA0MDY0IDE2Mzg0Cjc2MTc1ODAyMzY4IDE2Mzg0 Cjc2MTc1OTAwNjcyIDE2Mzg0Cjc2MTc1OTk4OTc2IDE2Mzg0Cjc2MTc2MDk3MjgwIDE2Mzg0Cjc2 MTc2MTk1NTg0IDE2Mzg0Cjc2ODMyMTEyNjQwIDE2Mzg0Cjc2ODMyMjEwOTQ0IDE2Mzg0Cjc2ODMy MzA5MjQ4IDE2Mzg0Cjc2ODMyNDA3NTUyIDE2Mzg0Cjc2ODMyNTA1ODU2IDE2Mzg0Cjc3NDg4NjE5 NTIwIDE2Mzg0Cjc3NDg4NzE3ODI0IDE2Mzg0Cjc3NDg4ODE2MTI4IDE2Mzg0Cjc3NDg4OTE0NDMy IDE2Mzg0Cjc3NDg5MDEyNzM2IDE2Mzg0Cjc4MTQ1MTI2NDAwIDE2Mzg0Cjc4MTQ1MjI0NzA0IDE2 Mzg0Cjc4MTQ1MzIzMDA4IDE2Mzg0Cjc4MTQ1NDIxMzEyIDE2Mzg0Cjc4MTQ1NTUyMzg0IDIwNDgw Cjc4ODAxNjMzMjgwIDE2Mzg0Cjc4ODAxNzMxNTg0IDE2Mzg0Cjc4ODAxODI5ODg4IDE2Mzg0Cjc4 ODAxOTI4MTkyIDE2Mzg0Cjc4ODAyMDI2NDk2IDE2Mzg0Cjc5NDU4MTQwMTYwIDE2Mzg0Cjc5NDU4 MjM4NDY0IDE2Mzg0Cjc5NDU4MzM2NzY4IDE2Mzg0Cjc5NDU4NDM1MDcyIDE2Mzg0Cjc5NDU4NTY2 MTQ0IDIwNDgwCjgwMTE0NjQ3MDQwIDE2Mzg0CjgwMTE0NzQ1MzQ0IDE2Mzg0CjgwMTE0ODQzNjQ4 IDE2Mzg0CjgwMTE0OTQxOTUyIDE2Mzg0CjgwMTE1MDQwMjU2IDE2Mzg0CjgwNzcxMTUzOTIwIDE2 Mzg0CjgwNzcxMjUyMjI0IDE2Mzg0CjgwNzcxMzUwNTI4IDE2Mzg0CjgwNzcxNDQ4ODMyIDE2Mzg0 CjgwNzcxNTQ3MTM2IDE2Mzg0CjgxNDI3NjYwODAwIDE2Mzg0CjgxNDI3NzU5MTA0IDE2Mzg0Cjgx NDI3ODU3NDA4IDE2Mzg0CjgxNDI3OTg4NDgwIDIwNDgwCjgxNDI4MDg2Nzg0IDE2Mzg0CjgyMDg0 MTY3NjgwIDE2Mzg0CjgyMDg0MjY1OTg0IDE2Mzg0CjgyMDg0MzY0Mjg4IDE2Mzg0CjgyMDg0NDYy NTkyIDE2Mzg0CjgyMDg0NTYwODk2IDE2Mzg0CjgyNzQwNjc0NTYwIDE2Mzg0CjgyNzQwNzcyODY0 IDE2Mzg0CjgyNzQwODcxMTY4IDE2Mzg0CjgyNzQwOTY5NDcyIDE2Mzg0CjgyNzQxMDY3Nzc2IDE2 Mzg0CjgzMzk3MTgxNDQwIDE2Mzg0CjgzMzk3Mjc5NzQ0IDE2Mzg0CjgzMzk3Mzc4MDQ4IDE2Mzg0 CjgzMzk3NDc2MzUyIDE2Mzg0CjgzMzk3NTc0NjU2IDE2Mzg0Cjg0MDUzNjg4MzIwIDE2Mzg0Cjg0 MDUzNzg2NjI0IDE2Mzg0Cjg0MDUzODg0OTI4IDE2Mzg0Cjg0MDUzOTgzMjMyIDE2Mzg0Cjg0MDU0 MDgxNTM2IDE2Mzg0Cjg0NzEwMTk1MjAwIDE2Mzg0Cjg0NzEwMjkzNTA0IDE2Mzg0Cjg0NzEwMzkx ODA4IDE2Mzg0Cjg0NzEwNTIyODgwIDIwNDgwCjg0NzEwNjIxMTg0IDE2Mzg0Cjg1MzY2NzAyMDgw IDE2Mzg0Cjg1MzY2ODAwMzg0IDE2Mzg0Cjg1MzY2ODk4Njg4IDE2Mzg0Cjg1MzY2OTk2OTkyIDE2 Mzg0Cjg1MzY3MDk1Mjk2IDE2Mzg0Cjg2MDIzMjA4OTYwIDE2Mzg0Cjg2MDIzMzA3MjY0IDE2Mzg0 Cjg2MDIzNDA1NTY4IDE2Mzg0Cjg2MDIzNTM2NjQwIDIwNDgwCjg2MDIzNjM0OTQ0IDE2Mzg0Cjg2 Njc5NzE1ODQwIDE2Mzg0Cjg2Njc5ODE0MTQ0IDE2Mzg0Cjg2Njc5OTEyNDQ4IDE2Mzg0Cjg2Njgw MDEwNzUyIDE2Mzg0Cjg2NjgwMTA5MDU2IDE2Mzg0Cjg3MzM2MjIyNzIwIDE2Mzg0Cjg3MzM2MzIx MDI0IDE2Mzg0Cjg3MzM2NDE5MzI4IDE2Mzg0Cjg3MzM2NTUwNDAwIDIwNDgwCjg3MzM2NjQ4NzA0 IDE2Mzg0Cjg3OTkyNzI5NjAwIDE2Mzg0Cjg3OTkyODI3OTA0IDE2Mzg0Cjg3OTkyOTI2MjA4IDE2 Mzg0Cjg3OTkzMDI0NTEyIDE2Mzg0Cjg3OTkzMTIyODE2IDE2Mzg0Cjg4NjQ5MjM2NDgwIDE2Mzg0 Cjg4NjQ5MzM0Nzg0IDE2Mzg0Cjg4NjQ5NDMzMDg4IDE2Mzg0Cjg4NjQ5NTMxMzkyIDE2Mzg0Cjg4 NjQ5NjI5Njk2IDE2Mzg0Cjg5MzA1NzQzMzYwIDE2Mzg0Cjg5MzA1ODQxNjY0IDE2Mzg0Cjg5MzA1 OTM5OTY4IDE2Mzg0Cjg5MzA2MDM4MjcyIDE2Mzg0Cjg5MzA2MTY5MzQ0IDIwNDgwCjg5OTYyMjUw MjQwIDE2Mzg0Cjg5OTYyMzQ4NTQ0IDE2Mzg0Cjg5OTYyNDQ2ODQ4IDE2Mzg0Cjg5OTYyNTQ1MTUy IDE2Mzg0Cjg5OTYyNjQzNDU2IDE2Mzg0CjkwNjE4NzU3MTIwIDE2Mzg0CjkwNjE4ODU1NDI0IDE2 Mzg0CjkwNjE4OTg2NDk2IDIwNDgwCjkwNjE5MDg0ODAwIDE2Mzg0CjkwNjE5MjE1ODcyIDIwNDgw CjkxMjc1MjY0MDAwIDE2Mzg0CjkxMjc1MzYyMzA0IDE2Mzg0CjkxMjc1NDYwNjA4IDE2Mzg0Cjkx Mjc1NTU4OTEyIDE2Mzg0CjkxMjc1NjU3MjE2IDE2Mzg0CjkxOTMxNzcwODgwIDE2Mzg0CjkxOTMx ODY5MTg0IDE2Mzg0CjkxOTMyMDAwMjU2IDIwNDgwCjkxOTMyMDk4NTYwIDE2Mzg0CjkxOTMyMTk2 ODY0IDE2Mzg0CjkyNTg4Mjc3NzYwIDE2Mzg0CjkyNTg4Mzc2MDY0IDE2Mzg0CjkyNTg4NDc0MzY4 IDE2Mzg0CjkyNTg4NjA1NDQwIDIwNDgwCjkyNTg4NzAzNzQ0IDE2Mzg0CjkzMjQ0Nzg0NjQwIDE2 Mzg0CjkzMjQ0ODgyOTQ0IDE2Mzg0CjkzMjQ1MDE0MDE2IDIwNDgwCjkzMjQ1MTEyMzIwIDE2Mzg0 CjkzMjQ1MjEwNjI0IDE2Mzg0CjkzOTAxMjkxNTIwIDE2Mzg0CjkzOTAxMzg5ODI0IDE2Mzg0Cjkz OTAxNTIwODk2IDIwNDgwCjkzOTAxNjE5MjAwIDE2Mzg0CjkzOTAxNzE3NTA0IDE2Mzg0CjkzOTAx ODE1ODA4IDE2Mzg0Cjk0NTU3Nzk4NDAwIDE2Mzg0Cjk0NTU3ODk2NzA0IDE2Mzg0Cjk0NTU3OTk1 MDA4IDE2Mzg0Cjk0NTU4MDkzMzEyIDE2Mzg0Cjk0NTU4MTkxNjE2IDE2Mzg0Cjk1MjE0MzA1Mjgw IDE2Mzg0Cjk1MjE0NDAzNTg0IDE2Mzg0Cjk1MjE0NTAxODg4IDE2Mzg0Cjk1MjE0NjAwMTkyIDE2 Mzg0Cjk1MjE0Njk4NDk2IDE2Mzg0Cjk1ODcwODEyMTYwIDE2Mzg0Cjk1ODcwOTEwNDY0IDE2Mzg0 Cjk1ODcxMDQxNTM2IDIwNDgwCjk1ODcxMTcyNjA4IDIwNDgwCjk1ODcxMjcwOTEyIDE2Mzg0Cjk2 NTI3MzE5MDQwIDE2Mzg0Cjk2NTI3NDE3MzQ0IDE2Mzg0Cjk2NTI3NTE1NjQ4IDE2Mzg0Cjk2NTI3 NjQ2NzIwIDIwNDgwCjk2NTI3NzQ1MDI0IDE2Mzg0Cjk3MTgzODI1OTIwIDE2Mzg0Cjk3MTgzOTU2 OTkyIDIwNDgwCjk3MTg0MDU1Mjk2IDE2Mzg0Cjk3MTg0MTg2MzY4IDIwNDgwCjk3MTg0Mjg0Njcy IDE2Mzg0Cjk3ODQwMzMyODAwIDE2Mzg0Cjk3ODQwNDMxMTA0IDE2Mzg0Cjk3ODQwNTI5NDA4IDE2 Mzg0Cjk3ODQwNjI3NzEyIDE2Mzg0Cjk3ODQwNzI2MDE2IDE2Mzg0Cjk4NDk2ODM5NjgwIDE2Mzg0 Cjk4NDk2OTM3OTg0IDE2Mzg0Cjk4NDk3MDM2Mjg4IDE2Mzg0Cjk4NDk3MTM0NTkyIDE2Mzg0Cjk4 NDk3MjMyODk2IDE2Mzg0Cjk5MTUzMzQ2NTYwIDE2Mzg0Cjk5MTUzNDQ0ODY0IDE2Mzg0Cjk5MTUz NTQzMTY4IDE2Mzg0Cjk5MTUzNjQxNDcyIDE2Mzg0Cjk5MTUzNzM5Nzc2IDE2Mzg0Cjk5MTUzODM4 MDgwIDE2Mzg0Cjk5ODA5ODUzNDQwIDE2Mzg0Cjk5ODA5OTUxNzQ0IDE2Mzg0Cjk5ODEwMDgyODE2 IDIwNDgwCjk5ODEwMTgxMTIwIDE2Mzg0Cjk5ODEwMjc5NDI0IDE2Mzg0CjEwMDQ2NjM2MDMyMCAx NjM4NAoxMDA0NjY0NTg2MjQgMTYzODQKMTAwNDY2NTU2OTI4IDE2Mzg0CjEwMDQ2NjY1NTIzMiAx NjM4NAoxMDA0NjY3NTM1MzYgMTYzODQKMTAwNDY2ODUxODQwIDE2Mzg0CjEwMTEyMjg2NzIwMCAx NjM4NAoxMDExMjI5NjU1MDQgMTYzODQKMTAxMTIzMDYzODA4IDE2Mzg0CjEwMTEyMzE5NDg4MCAy MDQ4MAoxMDExMjMyOTMxODQgMTYzODQKMTAxNzc5Mzc0MDgwIDE2Mzg0CjEwMTc3OTQ3MjM4NCAx NjM4NAoxMDE3Nzk1NzA2ODggMTYzODQKMTAxNzc5NjY4OTkyIDE2Mzg0CjEwMTc3OTc2NzI5NiAx NjM4NAoxMDI0MzU4ODA5NjAgMTYzODQKMTAyNDM1OTc5MjY0IDE2Mzg0CjEwMjQzNjA3NzU2OCAx NjM4NAoxMDI0MzYxNzU4NzIgMTYzODQKMTAyNDM2Mjc0MTc2IDE2Mzg0CjEwMjQzNjM3MjQ4MCAx NjM4NAoxMDMwOTIzODc4NDAgMTYzODQKMTAzMDkyNDg2MTQ0IDE2Mzg0CjEwMzA5MjU4NDQ0OCAx NjM4NAoxMDMwOTI2ODI3NTIgMTYzODQKMTAzMDkyNzgxMDU2IDE2Mzg0CjEwMzA5Mjg3OTM2MCAx NjM4NAoxMDM3NDg4OTQ3MjAgMTYzODQKMTAzNzQ4OTkzMDI0IDE2Mzg0CjEwMzc0OTA5MTMyOCAx NjM4NAoxMDM3NDkxODk2MzIgMTYzODQKMTAzNzQ5Mjg3OTM2IDE2Mzg0CjEwMzc0OTM4NjI0MCAx NjM4NAoxMDQ0MDU0MDE2MDAgMTYzODQKMTA0NDA1NDk5OTA0IDE2Mzg0CjEwNDQwNTU5ODIwOCAx NjM4NAoxMDQ0MDU2OTY1MTIgMTYzODQKMTA0NDA1Nzk0ODE2IDE2Mzg0CjEwNTA2MTkwODQ4MCAx NjM4NAoxMDUwNjIwMDY3ODQgMTYzODQKMTA1MDYyMTA1MDg4IDE2Mzg0CjEwNTA2MjIwMzM5MiAx NjM4NAoxMDUwNjIzMDE2OTYgMTYzODQKMTA1NzE4NDE1MzYwIDE2Mzg0CjEwNTcxODU0NjQzMiAy MDQ4MAoxMDU3MTg2NDQ3MzYgMTYzODQKMTA1NzE4NzQzMDQwIDE2Mzg0CjEwNTcxODg0MTM0NCAx NjM4NAoxMDU3MTg5Mzk2NDggMTYzODQKMTA2Mzc0OTIyMjQwIDE2Mzg0CjEwNjM3NTAyMDU0NCAx NjM4NAoxMDYzNzUxMTg4NDggMTYzODQKMTA2Mzc1MjE3MTUyIDE2Mzg0CjEwNjM3NTMxNTQ1NiAx NjM4NAoxMDYzNzU0MTM3NjAgMTYzODQKMTA3MDMxNDI5MTIwIDE2Mzg0CjEwNzAzMTUyNzQyNCAx NjM4NAoxMDcwMzE2MjU3MjggMTYzODQKMTA3MDMxNzI0MDMyIDE2Mzg0CjEwNzAzMTg1NTEwNCAy MDQ4MAoxMDc2ODc5MzYwMDAgMTYzODQKMTA3Njg4MDM0MzA0IDE2Mzg0CjEwNzY4ODEzMjYwOCAx NjM4NAoxMDc2ODgyMzA5MTIgMTYzODQKMTA3Njg4MzI5MjE2IDE2Mzg0CjEwODM0NDQ0Mjg4MCAx NjM4NAoxMDgzNDQ1NDExODQgMTYzODQKMTA4MzQ0NjM5NDg4IDE2Mzg0CjEwODM0NDczNzc5MiAx NjM4NAoxMDgzNDQ4Njg4NjQgMjA0ODAKMTA5MDAwOTQ5NzYwIDE2Mzg0CjEwOTAwMTA4MDgzMiAy MDQ4MAoxMDkwMDExNzkxMzYgMTYzODQKMTA5MDAxMjc3NDQwIDE2Mzg0CjEwOTAwMTM3NTc0NCAx NjM4NAoxMDk2NTc0NTY2NDAgMTYzODQKMTA5NjU3NTU0OTQ0IDE2Mzg0CjEwOTY1NzY1MzI0OCAx NjM4NAoxMDk2NTc3NTE1NTIgMTYzODQKMTA5NjU3ODQ5ODU2IDE2Mzg0CjExMDMxMzk2MzUyMCAx NjM4NAoxMTAzMTQwNjE4MjQgMTYzODQKMTEwMzE0MTkyODk2IDIwNDgwCjExMDMxNDI5MTIwMCAx NjM4NAoxMTAzMTQzODk1MDQgMTYzODQKMTEwOTcwNDcwNDAwIDE2Mzg0CjExMDk3MDYwMTQ3MiAy MDQ4MAoxMTA5NzA2OTk3NzYgMTYzODQKMTEwOTcwNzk4MDgwIDE2Mzg0CjExMDk3MDg5NjM4NCAx NjM4NAoxMTE2MjY5NzcyODAgMTYzODQKMTExNjI3MDc1NTg0IDE2Mzg0CjExMTYyNzE3Mzg4OCAx NjM4NAoxMTE2MjcyNzIxOTIgMTYzODQKMTExNjI3MzcwNDk2IDE2Mzg0CjExMjI4MzQ4NDE2MCAx NjM4NAoxMTIyODM1ODI0NjQgMTYzODQKMTEyMjgzNjgwNzY4IDE2Mzg0CjExMjI4MzgxMTg0MCAy MDQ4MAoxMTIyODM5MTAxNDQgMTYzODQKMTEyOTM5OTkxMDQwIDE2Mzg0CjExMjk0MDA4OTM0NCAx NjM4NAoxMTI5NDAxODc2NDggMTYzODQKMTEyOTQwMjg1OTUyIDE2Mzg0CjExMjk0MDM4NDI1NiAx NjM4NAoxMTM1OTY0OTc5MjAgMTYzODQKMTEzNTk2NTk2MjI0IDE2Mzg0CjExMzU5NjY5NDUyOCAx NjM4NAoxMTM1OTY3OTI4MzIgMTYzODQKMTEzNTk2ODkxMTM2IDE2Mzg0CjExNDI1MzAwNDgwMCAx NjM4NAoxMTQyNTMxMDMxMDQgMTYzODQKMTE0MjUzMjAxNDA4IDE2Mzg0CjExNDI1MzI5OTcxMiAx NjM4NAoxMTQyNTMzOTgwMTYgMTYzODQKMTE0OTA5NTExNjgwIDE2Mzg0CjExNDkwOTY0Mjc1MiAy MDQ4MAoxMTQ5MDk3NzM4MjQgMjA0ODAKMTE0OTA5ODcyMTI4IDE2Mzg0CjExNDkwOTk3MDQzMiAx NjM4NAoxMTU1NjYwMTg1NjAgMTYzODQKMTE1NTY2MTE2ODY0IDE2Mzg0CjExNTU2NjIxNTE2OCAx NjM4NAoxMTU1NjYzMTM0NzIgMTYzODQKMTE1NTY2NDExNzc2IDE2Mzg0CjExNjIyMjUyNTQ0MCAx NjM4NAoxMTYyMjI2MjM3NDQgMTYzODQKMTE2MjIyNzIyMDQ4IDE2Mzg0CjExNjIyMjgyMDM1MiAx NjM4NAoxMTYyMjI5MTg2NTYgMTYzODQKMTE2ODc5MDMyMzIwIDE2Mzg0CjExNjg3OTEzMDYyNCAx NjM4NAoxMTY4NzkyMjg5MjggMTYzODQKMTE2ODc5MzI3MjMyIDE2Mzg0CjExNjg3OTQyNTUzNiAx NjM4NAoxMTc1MzU1MzkyMDAgMTYzODQKMTE3NTM1NjM3NTA0IDE2Mzg0CjExNzUzNTc2ODU3NiAy MDQ4MAoxMTc1MzU4OTk2NDggMjA0ODAKMTE3NTM2MDMwNzIwIDIwNDgwCjExNzUzNjEyOTAyNCAx NjM4NAoxMTgxOTIwNDYwODAgMTYzODQKMTE4MTkyMTQ0Mzg0IDE2Mzg0CjExODE5MjI0MjY4OCAx NjM4NAoxMTgxOTIzNDA5OTIgMTYzODQKMTE4MTkyNDM5Mjk2IDE2Mzg0CjExODE5MjUzNzYwMCAx NjM4NAoxMTgxOTI2MzU5MDQgMTYzODQKMTE4ODQ4NTUyOTYwIDE2Mzg0CjExODg0ODY4NDAzMiAy MDQ4MAoxMTg4NDg3ODIzMzYgMTYzODQKMTE4ODQ4ODgwNjQwIDE2Mzg0CjExODg0ODk3ODk0NCAx NjM4NAoxMTk1MDUwNTk4NDAgMTYzODQKMTE5NTA1MTU4MTQ0IDE2Mzg0CjExOTUwNTI1NjQ0OCAx NjM4NAoxMTk1MDUzODc1MjAgMjA0ODAKMTE5NTA1NDg1ODI0IDE2Mzg0CjEyMDE2MTU2NjcyMCAx NjM4NAoxMjAxNjE2NjUwMjQgMTYzODQKMTIwMTYxNzYzMzI4IDE2Mzg0CjEyMDE2MTg2MTYzMiAx NjM4NAoxMjAxNjE5NTk5MzYgMTYzODQKMTIwODE4MDczNjAwIDE2Mzg0CjEyMDgxODE3MTkwNCAx NjM4NAoxMjA4MTgyNzAyMDggMTYzODQKMTIwODE4NDAxMjgwIDIwNDgwCjEyMDgxODQ5OTU4NCAx NjM4NAoxMjE0NzQ1ODA0ODAgMTYzODQKMTIxNDc0Njc4Nzg0IDE2Mzg0CjEyMTQ3NDc3NzA4OCAx NjM4NAoxMjE0NzQ4NzUzOTIgMTYzODQKMTIxNDc0OTczNjk2IDE2Mzg0CjEyMjEzMTA4NzM2MCAx NjM4NAoxMjIxMzExODU2NjQgMTYzODQKMTIyMTMxMjgzOTY4IDE2Mzg0CjEyMjEzMTM4MjI3MiAx NjM4NAoxMjIxMzE0ODA1NzYgMTYzODQKMTIyNzg3NTk0MjQwIDE2Mzg0CjEyMjc4NzY5MjU0NCAx NjM4NAoxMjI3ODc3OTA4NDggMTYzODQKMTIyNzg3ODg5MTUyIDE2Mzg0CjEyMjc4Nzk4NzQ1NiAx NjM4NAoxMjM0NDQxMDExMjAgMTYzODQKMTIzNDQ0MTk5NDI0IDE2Mzg0CjEyMzQ0NDI5NzcyOCAx NjM4NAoxMjM0NDQ0Mjg4MDAgMjA0ODAKMTIzNDQ0NTI3MTA0IDE2Mzg0CjEyNDEwMDYwODAwMCAx NjM4NAoxMjQxMDA3MDYzMDQgMTYzODQKMTI0MTAwODA0NjA4IDE2Mzg0CjEyNDEwMDkwMjkxMiAx NjM4NAoxMjQxMDEwMDEyMTYgMTYzODQKMTI0NzU3MTE0ODgwIDE2Mzg0CjEyNDc1NzIxMzE4NCAx NjM4NAoxMjQ3NTczMTE0ODggMTYzODQKMTI0NzU3NDA5NzkyIDE2Mzg0CjEyNDc1NzUwODA5NiAx NjM4NAoxMjU0MTM2MjE3NjAgMTYzODQKMTI1NDEzNzIwMDY0IDE2Mzg0CjEyNTQxMzgxODM2OCAx NjM4NAoxMjU0MTM5MTY2NzIgMTYzODQKMTI1NDE0MDE0OTc2IDE2Mzg0CjEyNjA3MDEyODY0MCAx NjM4NAoxMjYwNzAyNTk3MTIgMjA0ODAKMTI2MDcwMzU4MDE2IDE2Mzg0CjEyNjA3MDQ1NjMyMCAx NjM4NAoxMjYwNzA1NTQ2MjQgMTYzODQKMTI2NzI2NjM1NTIwIDE2Mzg0CjEyNjcyNjczMzgyNCAx NjM4NAoxMjY3MjY4MzIxMjggMTYzODQKMTI2NzI2OTMwNDMyIDE2Mzg0CjEyNjcyNzAyODczNiAx NjM4NAoxMjczODMxNDI0MDAgMTYzODQKMTI3MzgzMjQwNzA0IDE2Mzg0CjEyNzM4MzMzOTAwOCAx NjM4NAoxMjczODM0NzAwODAgMjA0ODAKMTI3MzgzNTY4Mzg0IDE2Mzg0CjEyODAzOTY0OTI4MCAx NjM4NAoxMjgwMzk3ODAzNTIgMjA0ODAKMTI4MDM5ODc4NjU2IDE2Mzg0CjEyODAzOTk3Njk2MCAx NjM4NAoxMjgwNDAxMDgwMzIgMjA0ODAKMTI4Njk2MTU2MTYwIDE2Mzg0CjEyODY5NjI4NzIzMiAy MDQ4MAoxMjg2OTYzODU1MzYgMTYzODQKMTI4Njk2NDgzODQwIDE2Mzg0CjEyODY5NjU4MjE0NCAx NjM4NAoxMjg2OTY3MTMyMTYgMjA0ODAKMTI5MzUyNjYzMDQwIDE2Mzg0CjEyOTM1Mjc2MTM0NCAx NjM4NAoxMjkzNTI4NTk2NDggMTYzODQKMTI5MzUyOTU3OTUyIDE2Mzg0CjEyOTM1MzA4OTAyNCAy MDQ4MAoxMzAwMDkxNjk5MjAgMTYzODQKMTMwMDA5MjY4MjI0IDE2Mzg0CjEzMDAwOTM2NjUyOCAx NjM4NAoxMzAwMDk0NjQ4MzIgMTYzODQKMTMwMDA5NTYzMTM2IDE2Mzg0CjEzMDY2NTY3NjgwMCAx NjM4NAoxMzA2NjU3NzUxMDQgMTYzODQKMTMwNjY1OTA2MTc2IDIwNDgwCjEzMDY2NjAzNzI0OCAy MDQ4MAoxMzA2NjYxNjgzMjAgMjA0ODAKMTMxMzIyMjE2NDQ4IDIwNDgwCjEzMTMyMjMxNDc1MiAx NjM4NAoxMzEzMjI0MTMwNTYgMTYzODQKMTMxMzIyNTExMzYwIDE2Mzg0CjEzMTMyMjYwOTY2NCAx NjM4NAoxMzE5Nzg2OTA1NjAgMTYzODQKMTMxOTc4Nzg4ODY0IDE2Mzg0CjEzMTk3ODg4NzE2OCAx NjM4NAoxMzE5Nzg5ODU0NzIgMTYzODQKMTMxOTc5MTE2NTQ0IDIwNDgwCjEzMjYzNTE5NzQ0MCAx NjM4NAoxMzI2MzUyOTU3NDQgMTYzODQKMTMyNjM1Mzk0MDQ4IDE2Mzg0CjEzMjYzNTQ5MjM1MiAx NjM4NAoxMzI2MzU1OTA2NTYgMTYzODQKMTMzMjkxNzA0MzIwIDE2Mzg0CjEzMzI5MTgwMjYyNCAx NjM4NAoxMzMyOTE5MzM2OTYgMjA0ODAKMTMzMjkyMDMyMDAwIDE2Mzg0CjEzMzI5MjEzMDMwNCAx NjM4NAoxMzM5NDgyMTEyMDAgMTYzODQKMTMzOTQ4MzA5NTA0IDE2Mzg0CjEzMzk0ODQwNzgwOCAx NjM4NAoxMzM5NDg1MDYxMTIgMTYzODQKMTMzOTQ4NjA0NDE2IDE2Mzg0CjEzNDYwNDcxODA4MCAx NjM4NAoxMzQ2MDQ4MTYzODQgMTYzODQKMTM0NjA0OTE0Njg4IDE2Mzg0CjEzNDYwNTAxMjk5MiAx NjM4NAoxMzQ2MDUxMTEyOTYgMTYzODQKMTM1MjYxMjI0OTYwIDE2Mzg0CjEzNTI2MTMyMzI2NCAx NjM4NAoxMzUyNjE0MjE1NjggMTYzODQKMTM1MjYxNTE5ODcyIDE2Mzg0CjEzNTI2MTYxODE3NiAx NjM4NAoxMzU5MTc3NjQ2MDggMjA0ODAKMTM1OTE3ODYyOTEyIDE2Mzg0CjEzNTkxNzk2MTIxNiAx NjM4NAoxMzU5MTgwNTk1MjAgMTYzODQKMTM1OTE4MTU3ODI0IDE2Mzg0CjEzNjU3NDIzODcyMCAx NjM4NAoxMzY1NzQzNjk3OTIgMjA0ODAKMTM2NTc0NDY4MDk2IDE2Mzg0CjEzNjU3NDU2NjQwMCAx NjM4NAoxMzY1NzQ2NjQ3MDQgMTYzODQKMTM3MjMwNzc4MzY4IDIwNDgwCjEzNzIzMDg3NjY3MiAx NjM4NAoxMzcyMzA5NzQ5NzYgMTYzODQKMTM3MjMxMTA2MDQ4IDIwNDgwCjEzNzIzMTIwNDM1MiAx NjM4NAoxMzc4ODcyNTI0ODAgMTYzODQKMTM3ODg3MzUwNzg0IDE2Mzg0CjEzNzg4NzQ0OTA4OCAx NjM4NAoxMzc4ODc1ODAxNjAgMjA0ODAKMTM3ODg3Njc4NDY0IDE2Mzg0CjEzODU0Mzc5MjEyOCAy MDQ4MAoxMzg1NDM4OTA0MzIgMTYzODQKMTM4NTQzOTg4NzM2IDE2Mzg0CjEzODU0NDA4NzA0MCAx NjM4NAoxMzg1NDQxODUzNDQgMTYzODQKMTM5MjAwMjY2MjQwIDE2Mzg0CjEzOTIwMDM2NDU0NCAx NjM4NAoxMzkyMDA0NjI4NDggMTYzODQKMTM5MjAwNTkzOTIwIDIwNDgwCjEzOTIwMDY5MjIyNCAx NjM4NAoxMzk4NTY3NzMxMjAgMTYzODQKMTM5ODU2OTA0MTkyIDIwNDgwCjEzOTg1NzAzNTI2NCAy MDQ4MAoxMzk4NTcxMzM1NjggMTYzODQKMTM5ODU3MjMxODcyIDE2Mzg0CjE0MDUxMzI4MDAwMCAx NjM4NAoxNDA1MTMzNzgzMDQgMTYzODQKMTQwNTEzNDc2NjA4IDE2Mzg0CjE0MDUxMzU3NDkxMiAx NjM4NAoxNDA1MTM2NzMyMTYgMTYzODQKMTQxMTY5ODE5NjQ4IDIwNDgwCjE0MTE2OTkxNzk1MiAx NjM4NAoxNDExNzAwMTYyNTYgMTYzODQKMTQxMTcwMTE0NTYwIDE2Mzg0CjE0MTE3MDI0NTYzMiAy MDQ4MAoxNDE4MjYyOTM3NjAgMTYzODQKMTQxODI2NDI0ODMyIDIwNDgwCjE0MTgyNjUyMzEzNiAx NjM4NAoxNDE4MjY2MjE0NDAgMTYzODQKMTQxODI2NzE5NzQ0IDE2Mzg0CjE0MjQ4MjgwMDY0MCAx NjM4NAoxNDI0ODI4OTg5NDQgMTYzODQKMTQyNDgyOTk3MjQ4IDE2Mzg0CjE0MjQ4MzA5NTU1MiAx NjM4NAoxNDI0ODMxOTM4NTYgMTYzODQKMTQyNDgzMjkyMTYwIDE2Mzg0CjE0MjQ4MzM5MDQ2NCAx NjM4NAoxNDMxMzkzMDc1MjAgMTYzODQKMTQzMTM5NDA1ODI0IDE2Mzg0CjE0MzEzOTUzNjg5NiAy MDQ4MAoxNDMxMzk2MzUyMDAgMTYzODQKMTQzMTM5NzMzNTA0IDE2Mzg0CjE0Mzc5NTgxNDQwMCAx NjM4NAoxNDM3OTU5MTI3MDQgMTYzODQKMTQzNzk2MDExMDA4IDE2Mzg0CjE0Mzc5NjEwOTMxMiAx NjM4NAoxNDM3OTYyMDc2MTYgMTYzODQKMTQ0NDUyMzIxMjgwIDE2Mzg0CjE0NDQ1MjQxOTU4NCAx NjM4NAoxNDQ0NTI1MTc4ODggMTYzODQKMTQ0NDUyNjE2MTkyIDE2Mzg0CjE0NDQ1MjcxNDQ5NiAx NjM4NAoxNDUxMDg4MjgxNjAgMTYzODQKMTQ1MTA4OTI2NDY0IDE2Mzg0CjE0NTEwOTAyNDc2OCAx NjM4NAoxNDUxMDkxMjMwNzIgMTYzODQKMTQ1MTA5MjIxMzc2IDE2Mzg0CjE0NTc2NTMzNTA0MCAx NjM4NAoxNDU3NjU0MzMzNDQgMTYzODQKMTQ1NzY1NTMxNjQ4IDE2Mzg0CjE0NTc2NTYyOTk1MiAx NjM4NAoxNDU3NjU3MjgyNTYgMTYzODQKMTQ2NDIxODc0Njg4IDIwNDgwCjE0NjQyMTk3Mjk5MiAx NjM4NAoxNDY0MjIwNzEyOTYgMTYzODQKMTQ2NDIyMTY5NjAwIDE2Mzg0CjE0NjQyMjI2NzkwNCAx NjM4NAoxNDcwNzgzNDg4MDAgMTYzODQKMTQ3MDc4NDQ3MTA0IDE2Mzg0CjE0NzA3ODU0NTQwOCAx NjM4NAoxNDcwNzg2NDM3MTIgMTYzODQKMTQ3MDc4NzQyMDE2IDE2Mzg0CjE0NzczNDg1NTY4MCAx NjM4NAoxNDc3MzQ5NTM5ODQgMTYzODQKMTQ3NzM1MDUyMjg4IDE2Mzg0CjE0NzczNTE1MDU5MiAx NjM4NAoxNDc3MzUyNDg4OTYgMTYzODQKMTQ3NzM1MzQ3MjAwIDE2Mzg0CjE0ODM5MTM2MjU2MCAx NjM4NAoxNDgzOTE0OTM2MzIgMjA0ODAKMTQ4MzkxNTkxOTM2IDE2Mzg0CjE0ODM5MTY5MDI0MCAx NjM4NAoxNDgzOTE3ODg1NDQgMTYzODQKMTQ5MDQ3ODY5NDQwIDE2Mzg0CjE0OTA0Nzk2Nzc0NCAx NjM4NAoxNDkwNDgwNjYwNDggMTYzODQKMTQ5MDQ4MTY0MzUyIDE2Mzg0CjE0OTA0ODI2MjY1NiAx NjM4NAoxNDk3MDQ0MDkwODggMjA0ODAKMTQ5NzA0NTA3MzkyIDE2Mzg0CjE0OTcwNDYwNTY5NiAx NjM4NAoxNDk3MDQ3MDQwMDAgMTYzODQKMTQ5NzA0ODAyMzA0IDE2Mzg0CjE1MDM2MDg4MzIwMCAx NjM4NAoxNTAzNjEwMTQyNzIgMjA0ODAKMTUwMzYxMTEyNTc2IDE2Mzg0CjE1MDM2MTIxMDg4MCAx NjM4NAoxNTAzNjEzNDE5NTIgMjA0ODAKMTUxMDE3MzkwMDgwIDE2Mzg0CjE1MTAxNzQ4ODM4NCAx NjM4NAoxNTEwMTc1ODY2ODggMTYzODQKMTUxMDE3Njg0OTkyIDE2Mzg0CjE1MTAxNzc4MzI5NiAx NjM4NAoxNTE2NzM4OTY5NjAgMTYzODQKMTUxNjczOTk1MjY0IDE2Mzg0CjE1MTY3NDA5MzU2OCAx NjM4NAoxNTE2NzQxOTE4NzIgMTYzODQKMTUxNjc0MzIyOTQ0IDIwNDgwCjE1MjMzMDQzNjYwOCAy MDQ4MAoxNTIzMzA1MzQ5MTIgMTYzODQKMTUyMzMwNjY1OTg0IDIwNDgwCjE1MjMzMDc2NDI4OCAx NjM4NAoxNTIzMzA4NjI1OTIgMTYzODQKMTUyOTg2OTEwNzIwIDE2Mzg0CjE1Mjk4NzAwOTAyNCAx NjM4NAoxNTI5ODcxMDczMjggMTYzODQKMTUyOTg3MjA1NjMyIDE2Mzg0CjE1Mjk4NzMwMzkzNiAx NjM4NAoxNTM2NDM0MTc2MDAgMTYzODQKMTUzNjQzNTE1OTA0IDE2Mzg0CjE1MzY0MzYxNDIwOCAx NjM4NAoxNTM2NDM3MTI1MTIgMTYzODQKMTUzNjQzODEwODE2IDE2Mzg0CjE1NDI5OTkyNDQ4MCAx NjM4NAoxNTQzMDAwNTU1NTIgMjA0ODAKMTU0MzAwMTUzODU2IDE2Mzg0CjE1NDMwMDI1MjE2MCAx NjM4NAoxNTQzMDAzNTA0NjQgMTYzODQKMTU0OTU2NDMxMzYwIDE2Mzg0CjE1NDk1NjU2MjQzMiAy MDQ4MAoxNTQ5NTY2NjA3MzYgMTYzODQKMTU0OTU2NzkxODA4IDIwNDgwCjE1NDk1Njg5MDExMiAx NjM4NAoxNTU2MTI5MzgyNDAgMTYzODQKMTU1NjEzMDM2NTQ0IDE2Mzg0CjE1NTYxMzE2NzYxNiAy MDQ4MAoxNTU2MTMyNjU5MjAgMTYzODQKMTU1NjEzMzY0MjI0IDE2Mzg0CjE1NjI2OTQ0NTEyMCAx NjM4NAoxNTYyNjk1NDM0MjQgMTYzODQKMTU2MjY5NjQxNzI4IDE2Mzg0CjE1NjI2OTc3MjgwMCAy MDQ4MAoxNTYyNjk4NzExMDQgMTYzODQKMTU2OTI1OTUyMDAwIDE2Mzg0CjE1NjkyNjA1MDMwNCAx NjM4NAoxNTY5MjYxNDg2MDggMTYzODQKMTU2OTI2MjQ2OTEyIDE2Mzg0CjE1NjkyNjM0NTIxNiAx NjM4NAoxNTc1ODI0NTg4ODAgMTYzODQKMTU3NTgyNTU3MTg0IDE2Mzg0CjE1NzU4MjY1NTQ4OCAx NjM4NAoxNTc1ODI3NTM3OTIgMTYzODQKMTU3NTgyODUyMDk2IDE2Mzg0CjE1ODIzODk2NTc2MCAx NjM4NAoxNTgyMzkwNjQwNjQgMTYzODQKMTU4MjM5MTYyMzY4IDE2Mzg0CjE1ODIzOTI2MDY3MiAx NjM4NAoxNTgyMzkzNTg5NzYgMTYzODQKMTU4ODk1NDcyNjQwIDE2Mzg0CjE1ODg5NTU3MDk0NCAx NjM4NAoxNTg4OTU3MDIwMTYgMjA0ODAKMTU4ODk1ODAwMzIwIDE2Mzg0CjE1ODg5NTg5ODYyNCAx NjM4NAoxNTk1NTIwMTIyODggMjA0ODAKMTU5NTUyMTEwNTkyIDE2Mzg0CjE1OTU1MjIwODg5NiAx NjM4NAoxNTk1NTIzMzk5NjggMjA0ODAKMTU5NTUyNDM4MjcyIDE2Mzg0CjE2MDIwODQ4NjQwMCAx NjM4NAoxNjAyMDg1ODQ3MDQgMTYzODQKMTYwMjA4NjgzMDA4IDE2Mzg0CjE2MDIwODc4MTMxMiAx NjM4NAoxNjAyMDg5MTIzODQgMjA0ODAKMTYwODY0OTkzMjgwIDE2Mzg0CjE2MDg2NTA5MTU4NCAx NjM4NAoxNjA4NjUxODk4ODggMTYzODQKMTYwODY1Mjg4MTkyIDE2Mzg0CjE2MDg2NTM4NjQ5NiAx NjM4NAoxNjE1MjE1MDAxNjAgMTYzODQKMTYxNTIxNTk4NDY0IDE2Mzg0CjE2MTUyMTY5Njc2OCAx NjM4NAoxNjE1MjE3OTUwNzIgMTYzODQKMTYxNTIxODkzMzc2IDE2Mzg0CjE2MjE3ODAwNzA0MCAx NjM4NAoxNjIxNzgxMDUzNDQgMTYzODQKMTYyMTc4MjAzNjQ4IDE2Mzg0CjE2MjE3ODMwMTk1MiAx NjM4NAoxNjIxNzg0MDAyNTYgMTYzODQKMTYyODM0NTEzOTIwIDE2Mzg0CjE2MjgzNDYxMjIyNCAx NjM4NAoxNjI4MzQ3NDMyOTYgMjA0ODAKMTYyODM0ODQxNjAwIDE2Mzg0CjE2MjgzNDkzOTkwNCAx NjM4NAoxNjM0OTEwNTM1NjggMjA0ODAKMTYzNDkxMTUxODcyIDE2Mzg0CjE2MzQ5MTI4Mjk0NCAy MDQ4MAoxNjM0OTEzODEyNDggMTYzODQKMTYzNDkxNDc5NTUyIDE2Mzg0CjE2NDE0NzU2MDQ0OCAy MDQ4MAoxNjQxNDc2OTE1MjAgMjA0ODAKMTY0MTQ3Nzg5ODI0IDE2Mzg0CjE2NDE0Nzg4ODEyOCAx NjM4NAoxNjQxNDc5ODY0MzIgMTYzODQKMTY0ODA0MDM0NTYwIDE2Mzg0CjE2NDgwNDEzMjg2NCAx NjM4NAoxNjQ4MDQyMzExNjggMTYzODQKMTY0ODA0MzYyMjQwIDIwNDgwCjE2NDgwNDQ2MDU0NCAx NjM4NAoxNjU0NjA1NDE0NDAgMTYzODQKMTY1NDYwNjM5NzQ0IDE2Mzg0CjE2NTQ2MDczODA0OCAx NjM4NAoxNjU0NjA4MzYzNTIgMTYzODQKMTY1NDYwOTM0NjU2IDE2Mzg0CjE2NjExNzA0ODMyMCAx NjM4NAoxNjYxMTcxNDY2MjQgMTYzODQKMTY2MTE3MjQ0OTI4IDE2Mzg0CjE2NjExNzM0MzIzMiAx NjM4NAoxNjYxMTc0NDE1MzYgMTYzODQKMTY2NzczNTU1MjAwIDE2Mzg0CjE2Njc3MzY1MzUwNCAx NjM4NAoxNjY3NzM3NTE4MDggMTYzODQKMTY2NzczODUwMTEyIDE2Mzg0CjE2Njc3Mzk4MTE4NCAy MDQ4MAoxNjc0MzAwNjIwODAgMTYzODQKMTY3NDMwMTYwMzg0IDE2Mzg0CjE2NzQzMDI1ODY4OCAx NjM4NAoxNjc0MzAzNTY5OTIgMTYzODQKMTY3NDMwNDU1Mjk2IDE2Mzg0CjE2ODA4NjU2ODk2MCAx NjM4NAoxNjgwODY2NjcyNjQgMTYzODQKMTY4MDg2Nzk4MzM2IDIwNDgwCjE2ODA4Njg5NjY0MCAx NjM4NAoxNjgwODY5OTQ5NDQgMTYzODQKMTY4NzQzMDc1ODQwIDE2Mzg0CjE2ODc0MzE3NDE0NCAx NjM4NAoxNjg3NDMyNzI0NDggMTYzODQKMTY4NzQzMzcwNzUyIDE2Mzg0CjE2ODc0MzQ2OTA1NiAx NjM4NAoxNjkzOTk1ODI3MjAgMTYzODQKMTY5Mzk5NjgxMDI0IDE2Mzg0CjE2OTM5OTgxMjA5NiAy MDQ4MAoxNjkzOTk5MTA0MDAgMTYzODQKMTY5NDAwMDA4NzA0IDE2Mzg0CjE3MDA1NjEyMjM2OCAy MDQ4MAoxNzAwNTYyMjA2NzIgMTYzODQKMTcwMDU2MzUxNzQ0IDIwNDgwCjE3MDA1NjQ1MDA0OCAx NjM4NAoxNzAwNTY1NDgzNTIgMTYzODQKMTcwNzEyNTk2NDgwIDE2Mzg0CjE3MDcxMjY5NDc4NCAx NjM4NAoxNzA3MTI4MjU4NTYgMjA0ODAKMTcwNzEyOTU2OTI4IDIwNDgwCjE3MDcxMzA1NTIzMiAx NjM4NAoxNzA3MTMxNTM1MzYgMTYzODQKMTcwNzEzMjUxODQwIDE2Mzg0CjE3MTM2OTEwMzM2MCAx NjM4NAoxNzEzNjkyMDE2NjQgMTYzODQKMTcxMzY5Mjk5OTY4IDE2Mzg0CjE3MTM2OTM5ODI3MiAx NjM4NAoxNzEzNjk0OTY1NzYgMTYzODQKMTcyMDI1NjEwMjQwIDE2Mzg0CjE3MjAyNTcwODU0NCAx NjM4NAoxNzIwMjU4MDY4NDggMTYzODQKMTcyMDI1OTM3OTIwIDIwNDgwCjE3MjAyNjAzNjIyNCAx NjM4NAoxNzI2ODIxMTcxMjAgMTYzODQKMTcyNjgyMjE1NDI0IDE2Mzg0CjE3MjY4MjMxMzcyOCAx NjM4NAoxNzI2ODI0MTIwMzIgMTYzODQKMTcyNjgyNTEwMzM2IDE2Mzg0CjE3MjY4MjYwODY0MCAx NjM4NAoxNzMzMzg2MjQwMDAgMTYzODQKMTczMzM4NzIyMzA0IDE2Mzg0CjE3MzMzODgyMDYwOCAx NjM4NAoxNzMzMzg5NTE2ODAgMjA0ODAKMTczMzM5MDQ5OTg0IDE2Mzg0CjE3Mzk5NTEzMDg4MCAx NjM4NAoxNzM5OTUyMjkxODQgMTYzODQKMTczOTk1MzI3NDg4IDE2Mzg0CjE3Mzk5NTQyNTc5MiAx NjM4NAoxNzM5OTU1MjQwOTYgMTYzODQKMTc0NjUxNjM3NzYwIDE2Mzg0CjE3NDY1MTczNjA2NCAx NjM4NAoxNzQ2NTE4NjcxMzYgMjA0ODAKMTc0NjUxOTk4MjA4IDIwNDgwCjE3NDY1MjA5NjUxMiAx NjM4NAoxNzUzMDgxNDQ2NDAgMTYzODQKMTc1MzA4MjQyOTQ0IDE2Mzg0CjE3NTMwODM0MTI0OCAx NjM4NAoxNzUzMDg0Mzk1NTIgMTYzODQKMTc1MzA4NTM3ODU2IDE2Mzg0CjE3NTk2NDY1MTUyMCAx NjM4NAoxNzU5NjQ3NDk4MjQgMTYzODQKMTc1OTY0ODQ4MTI4IDE2Mzg0CjE3NTk2NDk0NjQzMiAx NjM4NAoxNzU5NjUwNzc1MDQgMjA0ODAKMTc2NjIxMTU4NDAwIDE2Mzg0CjE3NjYyMTI1NjcwNCAx NjM4NAoxNzY2MjEzNTUwMDggMTYzODQKMTc2NjIxNDUzMzEyIDE2Mzg0CjE3NjYyMTU1MTYxNiAx NjM4NAoxNzcyNzc2NjUyODAgMTYzODQKMTc3Mjc3NzYzNTg0IDE2Mzg0CjE3NzI3Nzg2MTg4OCAx NjM4NAoxNzcyNzc5NjAxOTIgMTYzODQKMTc3Mjc4MDU4NDk2IDE2Mzg0CjE3NzkzNDE3MjE2MCAx NjM4NAoxNzc5MzQyNzA0NjQgMTYzODQKMTc3OTM0MzY4NzY4IDE2Mzg0CjE3NzkzNDQ2NzA3MiAx NjM4NAoxNzc5MzQ1NjUzNzYgMTYzODQKMTc4NTkwNjc5MDQwIDE2Mzg0CjE3ODU5MDc3NzM0NCAx NjM4NAoxNzg1OTA4NzU2NDggMTYzODQKMTc4NTkxMDA2NzIwIDIwNDgwCjE3ODU5MTEwNTAyNCAx NjM4NAoxNzkyNDcxODU5MjAgMTYzODQKMTc5MjQ3Mjg0MjI0IDE2Mzg0CjE3OTI0NzQxNTI5NiAy MDQ4MAoxNzkyNDc1MTM2MDAgMTYzODQKMTc5MjQ3NjExOTA0IDE2Mzg0CjE3OTkwMzY5MjgwMCAx NjM4NAoxNzk5MDM3OTExMDQgMTYzODQKMTc5OTAzODg5NDA4IDE2Mzg0CjE3OTkwMzk4NzcxMiAx NjM4NAoxNzk5MDQwODYwMTYgMTYzODQKMTgwNTYwMTk5NjgwIDE2Mzg0CjE4MDU2MDI5Nzk4NCAx NjM4NAoxODA1NjAzOTYyODggMTYzODQKMTgwNTYwNDk0NTkyIDE2Mzg0CjE4MDU2MDU5Mjg5NiAx NjM4NAoxODEyMTY3MDY1NjAgMTYzODQKMTgxMjE2ODA0ODY0IDE2Mzg0CjE4MTIxNjkwMzE2OCAx NjM4NAoxODEyMTcwMDE0NzIgMTYzODQKMTgxMjE3MTMyNTQ0IDIwNDgwCjE4MTIxNzIzMDg0OCAx NjM4NAoxODEyMTczMjkxNTIgMTYzODQKMTgxODczMjEzNDQwIDE2Mzg0CjE4MTg3MzMxMTc0NCAx NjM4NAoxODE4NzM0NDI4MTYgMjA0ODAKMTgxODczNTczODg4IDIwNDgwCjE4MjUyOTc1MzA4OCAy MDQ4MAoxODI1Mjk4NTEzOTIgMTYzODQKMTgyNTI5OTQ5Njk2IDE2Mzg0CjE4MjUzMDA0ODAwMCAx NjM4NAoxODI1MzAxNDYzMDQgMTYzODQKMTgzMTg2MjI3MjAwIDE2Mzg0CjE4MzE4NjMyNTUwNCAx NjM4NAoxODMxODY0MjM4MDggMTYzODQKMTgzMTg2NTIyMTEyIDE2Mzg0CjE4MzE4NjYyMDQxNiAx NjM4NAoxODM4NDI3NjY4NDggMjA0ODAKMTgzODQyODY1MTUyIDE2Mzg0CjE4Mzg0Mjk2MzQ1NiAx NjM4NAoxODM4NDMwOTQ1MjggMjA0ODAKMTgzODQzMTkyODMyIDE2Mzg0CjE4NDQ5OTI0MDk2MCAx NjM4NAoxODQ0OTkzMzkyNjQgMTYzODQKMTg0NDk5NDM3NTY4IDE2Mzg0CjE4NDQ5OTUzNTg3MiAx NjM4NAoxODQ0OTk2MzQxNzYgMTYzODQKMTg1MTU1NzQ3ODQwIDE2Mzg0CjE4NTE1NTg3ODkxMiAy MDQ4MAoxODUxNTYwMDk5ODQgMjA0ODAKMTg1MTU2MTQxMDU2IDIwNDgwCjE4NTE1NjIzOTM2MCAx NjM4NAoxODU4MTIyNTQ3MjAgMTYzODQKMTg1ODEyMzUzMDI0IDE2Mzg0CjE4NTgxMjQ1MTMyOCAx NjM4NAoxODU4MTI1NDk2MzIgMTYzODQKMTg1ODEyNjQ3OTM2IDE2Mzg0CjE4NjQ2ODc2MTYwMCAx NjM4NAoxODY0Njg4NTk5MDQgMTYzODQKMTg2NDY4OTU4MjA4IDE2Mzg0CjE4NjQ2OTA1NjUxMiAx NjM4NAoxODY0NjkxNTQ4MTYgMTYzODQKMTg3MTI1MzAxMjQ4IDIwNDgwCjE4NzEyNTM5OTU1MiAx NjM4NAoxODcxMjU0OTc4NTYgMTYzODQKMTg3MTI1NTk2MTYwIDE2Mzg0CjE4NzEyNTY5NDQ2NCAx NjM4NAoxODc3ODE3NzUzNjAgMTYzODQKMTg3NzgxODczNjY0IDE2Mzg0CjE4Nzc4MTk3MTk2OCAx NjM4NAoxODc3ODIwNzAyNzIgMTYzODQKMTg3NzgyMTY4NTc2IDE2Mzg0CjE4ODQzODI4MjI0MCAx NjM4NAoxODg0MzgzODA1NDQgMTYzODQKMTg4NDM4NDc4ODQ4IDE2Mzg0CjE4ODQzODYwOTkyMCAy MDQ4MAoxODkwOTQ3ODkxMjAgMTYzODQKMTg5MDk0ODg3NDI0IDE2Mzg0CjE4OTA5NTAxODQ5NiAy MDQ4MAoxODkwOTUxMTY4MDAgMTYzODQKMTg5MDk1MjE1MTA0IDE2Mzg0CjE4OTc1MTI5NjAwMCAx NjM4NAoxODk3NTEzOTQzMDQgMTYzODQKMTg5NzUxNDkyNjA4IDE2Mzg0CjE4OTc1MTU5MDkxMiAx NjM4NAoxODk3NTE2ODkyMTYgMTYzODQKMTkwNDA3ODAyODgwIDE2Mzg0CjE5MDQwNzkwMTE4NCAx NjM4NAoxOTA0MDc5OTk0ODggMTYzODQKMTkwNDA4MDk3NzkyIDE2Mzg0CjE5MDQwODE5NjA5NiAx NjM4NAoxOTEwNjQzMDk3NjAgMTYzODQKMTkxMDY0NDA4MDY0IDE2Mzg0CjE5MTA2NDUzOTEzNiAy MDQ4MAoxOTEwNjQ2NzAyMDggMjA0ODAKMTkxMDY0NzY4NTEyIDE2Mzg0CjE5MTcyMDgxNjY0MCAx NjM4NAoxOTE3MjA5MTQ5NDQgMTYzODQKMTkxNzIxMDEzMjQ4IDE2Mzg0CjE5MTcyMTE0NDMyMCAy MDQ4MAoxOTE3MjEyNDI2MjQgMTYzODQKMTkyMzc3MzIzNTIwIDE2Mzg0CjE5MjM3NzQyMTgyNCAx NjM4NAoxOTIzNzc1MjAxMjggMTYzODQKMTkyMzc3NjE4NDMyIDE2Mzg0CjE5MjM3NzcxNjczNiAx NjM4NAoxOTMwMzM4MzA0MDAgMTYzODQKMTkzMDMzOTI4NzA0IDE2Mzg0CjE5MzAzNDAyNzAwOCAx NjM4NAoxOTMwMzQxMjUzMTIgMTYzODQKMTkzMDM0MjIzNjE2IDE2Mzg0CjE5MzY5MDMzNzI4MCAx NjM4NAoxOTM2OTA0MzU1ODQgMTYzODQKMTkzNjkwNTMzODg4IDE2Mzg0CjE5MzY5MDYzMjE5MiAx NjM4NAoxOTM2OTA3MzA0OTYgMTYzODQKMTk0MzQ2ODc2OTI4IDIwNDgwCjE5NDM0Njk3NTIzMiAx NjM4NAoxOTQzNDcwNzM1MzYgMTYzODQKMTk0MzQ3MTcxODQwIDE2Mzg0CjE5NDM0NzI3MDE0NCAx NjM4NAoxOTUwMDMzNTEwNDAgMTYzODQKMTk1MDAzNDQ5MzQ0IDE2Mzg0CjE5NTAwMzU0NzY0OCAx NjM4NAoxOTUwMDM2NDU5NTIgMTYzODQKMTk1MDAzNzQ0MjU2IDE2Mzg0CjE5NTY1OTg1NzkyMCAx NjM4NAoxOTU2NTk5NTYyMjQgMTYzODQKMTk1NjYwMDU0NTI4IDE2Mzg0CjE5NTY2MDE1MjgzMiAx NjM4NAoxOTU2NjAyODM5MDQgMjA0ODAKMTk2MzE2MzY0ODAwIDE2Mzg0CjE5NjMxNjQ2MzEwNCAx NjM4NAoxOTYzMTY1NjE0MDggMTYzODQKMTk2MzE2NjkyNDgwIDIwNDgwCjE5NjMxNjc5MDc4NCAx NjM4NAoxOTY5NzI4NzE2ODAgMTYzODQKMTk2OTczMDAyNzUyIDIwNDgwCjE5Njk3MzEwMTA1NiAx NjM4NAoxOTY5NzMxOTkzNjAgMTYzODQKMTk2OTczMjk3NjY0IDE2Mzg0CjE5NzYyOTM3ODU2MCAx NjM4NAoxOTc2Mjk0NzY4NjQgMTYzODQKMTk3NjI5NTc1MTY4IDE2Mzg0CjE5NzYyOTY3MzQ3MiAx NjM4NAoxOTc2Mjk3NzE3NzYgMTYzODQKMTk4Mjg1ODg1NDQwIDE2Mzg0CjE5ODI4NjAxNjUxMiAy MDQ4MAoxOTgyODYxNDc1ODQgMjA0ODAKMTk4Mjg2MjQ1ODg4IDE2Mzg0CjE5ODI4NjM0NDE5MiAx NjM4NAoxOTg5NDIzOTIzMjAgMTYzODQKMTk4OTQyNDkwNjI0IDE2Mzg0CjE5ODk0MjU4ODkyOCAx NjM4NAoxOTg5NDI2ODcyMzIgMTYzODQKMTk4OTQyNzg1NTM2IDE2Mzg0CjE5OTU5ODg5OTIwMCAx NjM4NAoxOTk1OTkwMzAyNzIgMjA0ODAKMTk5NTk5MTI4NTc2IDE2Mzg0CjE5OTU5OTIyNjg4MCAx NjM4NAoxOTk1OTkzMjUxODQgMTYzODQKMjAwMjU1NDA2MDgwIDE2Mzg0CjIwMDI1NTUzNzE1MiAy MDQ4MAoyMDAyNTU2MzU0NTYgMTYzODQKMjAwMjU1NzMzNzYwIDE2Mzg0CjIwMDI1NTgzMjA2NCAx NjM4NAoyMDA5MTE5MTI5NjAgMTYzODQKMjAwOTEyMDExMjY0IDE2Mzg0CjIwMDkxMjEwOTU2OCAx NjM4NAoyMDA5MTIyMDc4NzIgMTYzODQKMjAwOTEyMzA2MTc2IDE2Mzg0CjIwMTU2ODQxOTg0MCAx NjM4NAoyMDE1Njg1MTgxNDQgMTYzODQKMjAxNTY4NjQ5MjE2IDIwNDgwCjIwMTU2ODc0NzUyMCAx NjM4NAoyMDE1Njg4NDU4MjQgMTYzODQKMjAyMjI0OTI2NzIwIDE2Mzg0CjIwMjIyNTAyNTAyNCAx NjM4NAoyMDIyMjUxMjMzMjggMTYzODQKMjAyMjI1MjIxNjMyIDE2Mzg0CjIwMjIyNTMxOTkzNiAx NjM4NAoyMDI4ODE0MzM2MDAgMTYzODQKMjAyODgxNTY0NjcyIDIwNDgwCjIwMjg4MTY5NTc0NCAy MDQ4MAoyMDI4ODE3OTQwNDggMTYzODQKMjAyODgxODkyMzUyIDE2Mzg0CjIwMzUzNzk0MDQ4MCAx NjM4NAoyMDM1MzgwNzE1NTIgMjA0ODAKMjAzNTM4MTY5ODU2IDE2Mzg0CjIwMzUzODI2ODE2MCAx NjM4NAoyMDM1MzgzNjY0NjQgMTYzODQKMjA0MTk0NDgwMTI4IDIwNDgwCjIwNDE5NDU3ODQzMiAx NjM4NAoyMDQxOTQ2NzY3MzYgMTYzODQKMjA0MTk0Nzc1MDQwIDE2Mzg0CjIwNDE5NDg3MzM0NCAx NjM4NAoyMDQ4NTA5NTQyNDAgMTYzODQKMjA0ODUxMDUyNTQ0IDE2Mzg0CjIwNDg1MTE1MDg0OCAx NjM4NAoyMDQ4NTEyNDkxNTIgMTYzODQKMjA0ODUxMzQ3NDU2IDE2Mzg0CjIwNTUwNzQ2MTEyMCAx NjM4NAoyMDU1MDc1OTIxOTIgMjA0ODAKMjA1NTA3NjkwNDk2IDE2Mzg0CjIwNTUwNzgyMTU2OCAy MDQ4MAoyMDYxNjQwMDA3NjggMjA0ODAKMjA2MTY0MDk5MDcyIDE2Mzg0CjIwNjE2NDE5NzM3NiAx NjM4NAoyMDYxNjQyOTU2ODAgMTYzODQKMjA2MTY0MzkzOTg0IDE2Mzg0CjIwNjgyMDQ3NDg4MCAx NjM4NAoyMDY4MjA1NzMxODQgMTYzODQKMjA2ODIwNjcxNDg4IDE2Mzg0CjIwNjgyMDc2OTc5MiAx NjM4NAoyMDY4MjA4NjgwOTYgMTYzODQKMjA3NDc2OTgxNzYwIDE2Mzg0CjIwNzQ3NzExMjgzMiAy MDQ4MAoyMDc0NzcyMTExMzYgMTYzODQKMjA3NDc3MzA5NDQwIDE2Mzg0CjIwNzQ3NzQwNzc0NCAx NjM4NAoyMDgxMzM0ODg2NDAgMTYzODQKMjA4MTMzNTg2OTQ0IDE2Mzg0CjIwODEzMzY4NTI0OCAx NjM4NAoyMDgxMzM3ODM1NTIgMTYzODQKMjA4MTMzODgxODU2IDE2Mzg0CjIwODEzMzk4MDE2MCAx NjM4NAoyMDg3ODk5OTU1MjAgMTYzODQKMjA4NzkwMDkzODI0IDE2Mzg0CjIwODc5MDE5MjEyOCAx NjM4NAoyMDg3OTAyOTA0MzIgMTYzODQKMjA4NzkwNDIxNTA0IDIwNDgwCjIwOTQ0NjUwMjQwMCAx NjM4NAoyMDk0NDY2MDA3MDQgMTYzODQKMjA5NDQ2Njk5MDA4IDE2Mzg0CjIwOTQ0Njc5NzMxMiAx NjM4NAoyMDk0NDY5MjgzODQgMjA0ODAKMjEwMTAzMDA5MjgwIDE2Mzg0CjIxMDEwMzEwNzU4NCAx NjM4NAoyMTAxMDMyMDU4ODggMTYzODQKMjEwMTAzMzA0MTkyIDE2Mzg0CjIxMDEwMzQzNTI2NCAy MDQ4MAoyMTA3NTk1NDg5MjggMjA0ODAKMjEwNzU5NjQ3MjMyIDE2Mzg0CjIxMDc1OTc0NTUzNiAx NjM4NAoyMTA3NTk4NDM4NDAgMTYzODQKMjEwNzU5OTQyMTQ0IDE2Mzg0CjIxMTQxNjAyMzA0MCAx NjM4NAoyMTE0MTYxMjEzNDQgMTYzODQKMjExNDE2MjE5NjQ4IDE2Mzg0CjIxMTQxNjMxNzk1MiAx NjM4NAoyMTE0MTY0MTYyNTYgMTYzODQKMjEyMDcyNTI5OTIwIDE2Mzg0CjIxMjA3MjYyODIyNCAx NjM4NAoyMTIwNzI3MjY1MjggMTYzODQKMjEyMDcyODI0ODMyIDE2Mzg0CjIxMjA3MjkyMzEzNiAx NjM4NAoyMTI3MjkwMzY4MDAgMTYzODQKMjEyNzI5MTM1MTA0IDE2Mzg0CjIxMjcyOTIzMzQwOCAx NjM4NAoyMTI3MjkzMzE3MTIgMTYzODQKMjEyNzI5NDMwMDE2IDE2Mzg0CjIxMzM4NTU0MzY4MCAx NjM4NAoyMTMzODU2NDE5ODQgMTYzODQKMjEzMzg1NzQwMjg4IDE2Mzg0CjIxMzM4NTgzODU5MiAx NjM4NAoyMTMzODU5MzY4OTYgMTYzODQKMjE0MDQyMDgzMzI4IDIwNDgwCjIxNDA0MjE4MTYzMiAx NjM4NAoyMTQwNDIyNzk5MzYgMTYzODQKMjE0MDQyMzc4MjQwIDE2Mzg0CjIxNDA0MjQ3NjU0NCAx NjM4NAoyMTQwNDI1NzQ4NDggMTYzODQKMjE0Njk4NTU3NDQwIDE2Mzg0CjIxNDY5ODY4ODUxMiAy MDQ4MAoyMTQ2OTg3ODY4MTYgMTYzODQKMjE0Njk4ODg1MTIwIDE2Mzg0CjIxNDY5ODk4MzQyNCAx NjM4NAoyMTUzNTUwNjQzMjAgMTYzODQKMjE1MzU1MTk1MzkyIDIwNDgwCjIxNTM1NTI5MzY5NiAx NjM4NAoyMTUzNTUzOTIwMDAgMTYzODQKMjE1MzU1NDkwMzA0IDE2Mzg0CjIxNjAxMTU3MTIwMCAx NjM4NAoyMTYwMTE3MDIyNzIgMjA0ODAKMjE2MDExODAwNTc2IDE2Mzg0CjIxNjAxMTg5ODg4MCAx NjM4NAoyMTYwMTE5OTcxODQgMTYzODQKMjE2MDEyMDk1NDg4IDE2Mzg0CjIxNjY2ODA3ODA4MCAx NjM4NAoyMTY2NjgxNzYzODQgMTYzODQKMjE2NjY4MzA3NDU2IDIwNDgwCjIxNjY2ODQwNTc2MCAx NjM4NAoyMTY2Njg1MDQwNjQgMTYzODQKMjE2NjY4NjM1MTM2IDIwNDgwCjIxNzMyNDU4NDk2MCAx NjM4NAoyMTczMjQ2ODMyNjQgMTYzODQKMjE3MzI0NzgxNTY4IDE2Mzg0CjIxNzMyNDg3OTg3MiAx NjM4NAoyMTczMjQ5NzgxNzYgMTYzODQKMjE3OTgxMDkxODQwIDE2Mzg0CjIxNzk4MTE5MDE0NCAx NjM4NAoyMTc5ODEzMjEyMTYgMjA0ODAKMjE3OTgxNDE5NTIwIDE2Mzg0CjIxNzk4MTUxNzgyNCAx NjM4NAoyMTg2Mzc2MzE0ODggMjA0ODAKMjE4NjM3NzI5NzkyIDE2Mzg0CjIxODYzNzgyODA5NiAx NjM4NAoyMTg2Mzc5MjY0MDAgMTYzODQKMjE4NjM4MDI0NzA0IDE2Mzg0CjIxOTI5NDEwNTYwMCAx NjM4NAoyMTkyOTQyMzY2NzIgMjA0ODAKMjE5Mjk0MzM0OTc2IDE2Mzg0CjIxOTI5NDQzMzI4MCAx NjM4NAoyMTkyOTQ1MzE1ODQgMTYzODQKMjE5Mjk0NjI5ODg4IDE2Mzg0CjIxOTI5NDcyODE5MiAx NjM4NAoyMTk5NTA2MTI0ODAgMTYzODQKMjE5OTUwNzEwNzg0IDE2Mzg0CjIxOTk1MDgwOTA4OCAx NjM4NAoyMTk5NTA5MDczOTIgMTYzODQKMjE5OTUxMDA1Njk2IDE2Mzg0CjIyMDYwNzExOTM2MCAx NjM4NAoyMjA2MDcyMTc2NjQgMTYzODQKMjIwNjA3MzE1OTY4IDE2Mzg0CjIyMDYwNzQxNDI3MiAx NjM4NAoyMjA2MDc1MTI1NzYgMTYzODQKMjIxMjYzNjI2MjQwIDE2Mzg0CjIyMTI2MzcyNDU0NCAx NjM4NAoyMjEyNjM4NTU2MTYgMjA0ODAKMjIxMjYzOTg2Njg4IDIwNDgwCjIyMTI2NDA4NDk5MiAx NjM4NAoyMjE5MjAxNjU4ODggMjA0ODAKMjIxOTIwMjY0MTkyIDE2Mzg0CjIyMTkyMDM2MjQ5NiAx NjM4NAoyMjE5MjA0NjA4MDAgMTYzODQKMjIxOTIwNTU5MTA0IDE2Mzg0CjIyMjU3NjY0MDAwMCAx NjM4NAoyMjI1NzY3MzgzMDQgMTYzODQKMjIyNTc2ODM2NjA4IDE2Mzg0CjIyMjU3NjkzNDkxMiAx NjM4NAoyMjI1NzcwMzMyMTYgMTYzODQKMjIzMjMzMTQ2ODgwIDE2Mzg0CjIyMzIzMzI0NTE4NCAx NjM4NAoyMjMyMzMzNDM0ODggMTYzODQKMjIzMjMzNDc0NTYwIDIwNDgwCjIyMzIzMzYwNTYzMiAy MDQ4MAoyMjM4ODk2NTM3NjAgMTYzODQKMjIzODg5NzUyMDY0IDE2Mzg0CjIyMzg4OTg4MzEzNiAy MDQ4MAoyMjM4ODk5ODE0NDAgMTYzODQKMjIzODkwMDc5NzQ0IDE2Mzg0CjIyNDU0NjE2MDY0MCAx NjM4NAoyMjQ1NDYyNTg5NDQgMTYzODQKMjI0NTQ2MzU3MjQ4IDE2Mzg0CjIyNDU0NjQ4ODMyMCAy MDQ4MAoyMjQ1NDY1ODY2MjQgMTYzODQKMjI1MjAyNjY3NTIwIDE2Mzg0CjIyNTIwMjc2NTgyNCAx NjM4NAoyMjUyMDI4NjQxMjggMTYzODQKMjI1MjAyOTYyNDMyIDE2Mzg0CjIyNTIwMzA2MDczNiAx NjM4NAoyMjU4NTkyMDcxNjggMjA0ODAKMjI1ODU5MzA1NDcyIDE2Mzg0CjIyNTg1OTQwMzc3NiAx NjM4NAoyMjU4NTk1MDIwODAgMTYzODQKMjI1ODU5NjAwMzg0IDE2Mzg0CjIyNjUxNTY4MTI4MCAx NjM4NAoyMjY1MTU3Nzk1ODQgMTYzODQKMjI2NTE1ODc3ODg4IDE2Mzg0CjIyNjUxNTk3NjE5MiAx NjM4NAoyMjY1MTYwNzQ0OTYgMTYzODQKMjI3MTcyMTg4MTYwIDE2Mzg0CjIyNzE3MjI4NjQ2NCAx NjM4NAoyMjcxNzIzODQ3NjggMTYzODQKMjI3MTcyNDgzMDcyIDE2Mzg0CjIyNzE3MjU4MTM3NiAx NjM4NAoyMjc4Mjg2OTUwNDAgMTYzODQKMjI3ODI4NzkzMzQ0IDE2Mzg0CjIyNzgyODg5MTY0OCAx NjM4NAoyMjc4Mjg5ODk5NTIgMTYzODQKMjI3ODI5MTIxMDI0IDIwNDgwCjIyODQ4NTIwMTkyMCAx NjM4NAoyMjg0ODUzMDAyMjQgMTYzODQKMjI4NDg1Mzk4NTI4IDE2Mzg0CjIyODQ4NTUyOTYwMCAy MDQ4MAoyMjg0ODU2Mjc5MDQgMTYzODQKMjI5MTQxNzA4ODAwIDE2Mzg0CjIyOTE0MTgzOTg3MiAy MDQ4MAoyMjkxNDE5MzgxNzYgMTYzODQKMjI5MTQyMDM2NDgwIDE2Mzg0CjIyOTE0MjEzNDc4NCAx NjM4NAoyMjkxNDIyMzMwODggMTYzODQKMjI5MTQyMzMxMzkyIDE2Mzg0CjIyOTc5ODIxNTY4MCAx NjM4NAoyMjk3OTgzNDY3NTIgMjA0ODAKMjI5Nzk4NDQ1MDU2IDE2Mzg0CjIyOTc5ODU3NjEyOCAy MDQ4MAoyMjk3OTg2NzQ0MzIgMTYzODQKMjI5Nzk4NzcyNzM2IDE2Mzg0CjIzMDQ1NDcyMjU2MCAx NjM4NAoyMzA0NTQ4MjA4NjQgMTYzODQKMjMwNDU0OTUxOTM2IDIwNDgwCjIzMDQ1NTA1MDI0MCAx NjM4NAoyMzA0NTUxNDg1NDQgMTYzODQKMjMwNDU1Mjc5NjE2IDIwNDgwCjIzMDQ1NTM3NzkyMCAx NjM4NAoyMzExMTEyMjk0NDAgMTYzODQKMjMxMTExMzI3NzQ0IDE2Mzg0CjIzMTExMTQyNjA0OCAx NjM4NAoyMzExMTE1MjQzNTIgMTYzODQKMjMxMTExNjIyNjU2IDE2Mzg0CjIzMTc2NzczNjMyMCAx NjM4NAoyMzE3Njc4MzQ2MjQgMTYzODQKMjMxNzY3OTMyOTI4IDE2Mzg0CjIzMTc2ODAzMTIzMiAx NjM4NAoyMzE3NjgxMjk1MzYgMTYzODQKMjMyNDI0MjQzMjAwIDE2Mzg0CjIzMjQyNDM0MTUwNCAx NjM4NAoyMzI0MjQ0Mzk4MDggMTYzODQKMjMyNDI0NTM4MTEyIDE2Mzg0CjIzMjQyNDYzNjQxNiAx NjM4NAoyMzMwODA3NTAwODAgMTYzODQKMjMzMDgwODgxMTUyIDIwNDgwCjIzMzA4MDk3OTQ1NiAx NjM4NAoyMzMwODEwNzc3NjAgMTYzODQKMjMzMDgxMTc2MDY0IDE2Mzg0CjIzMzczNzI1Njk2MCAx NjM4NAoyMzM3MzczNTUyNjQgMTYzODQKMjMzNzM3NDUzNTY4IDE2Mzg0CjIzMzczNzU1MTg3MiAx NjM4NAoyMzM3Mzc2NTAxNzYgMTYzODQKMjM0MzkzNzYzODQwIDE2Mzg0CjIzNDM5Mzg2MjE0NCAx NjM4NAoyMzQzOTM5NjA0NDggMTYzODQKMjM0Mzk0MDU4NzUyIDE2Mzg0CjIzNDM5NDE1NzA1NiAx NjM4NAoyMzUwNTAzMDM0ODggMjA0ODAKMjM1MDUwNDAxNzkyIDE2Mzg0CjIzNTA1MDUwMDA5NiAx NjM4NAoyMzUwNTA1OTg0MDAgMTYzODQKMjM1MDUwNjk2NzA0IDE2Mzg0CjIzNTcwNjgxMDM2OCAy MDQ4MAoyMzU3MDY5MDg2NzIgMTYzODQKMjM1NzA3MDA2OTc2IDE2Mzg0CjIzNTcwNzEwNTI4MCAx NjM4NAoyMzU3MDcyMDM1ODQgMTYzODQKMjM2MzYzMjg0NDgwIDE2Mzg0CjIzNjM2MzM4Mjc4NCAx NjM4NAoyMzYzNjM0ODEwODggMTYzODQKMjM2MzYzNTc5MzkyIDE2Mzg0CjIzNjM2MzY3NzY5NiAx NjM4NAoyMzcwMTk4MjQxMjggMjA0ODAKMjM3MDE5OTIyNDMyIDE2Mzg0CjIzNzAyMDAyMDczNiAx NjM4NAoyMzcwMjAxMTkwNDAgMTYzODQKMjM3MDIwMjE3MzQ0IDE2Mzg0CjIzNzY3NjI5ODI0MCAx NjM4NAoyMzc2NzYzOTY1NDQgMTYzODQKMjM3Njc2NDk0ODQ4IDE2Mzg0CjIzNzY3NjYyNTkyMCAy MDQ4MAoyMzc2NzY3MjQyMjQgMTYzODQKMjM4MzMyODA1MTIwIDE2Mzg0CjIzODMzMjkzNjE5MiAy MDQ4MAoyMzgzMzMwMzQ0OTYgMTYzODQKMjM4MzMzMTMyODAwIDE2Mzg0CjIzODMzMzI2Mzg3MiAy MDQ4MAoyMzg5ODkzMTIwMDAgMTYzODQKMjM4OTg5NDEwMzA0IDE2Mzg0CjIzODk4OTUwODYwOCAx NjM4NAoyMzg5ODk2Mzk2ODAgMjA0ODAKMjM4OTg5NzM3OTg0IDE2Mzg0CjIzOTY0NTgxODg4MCAx NjM4NAoyMzk2NDU5MTcxODQgMTYzODQKMjM5NjQ2MDE1NDg4IDE2Mzg0CjIzOTY0NjExMzc5MiAx NjM4NAoyMzk2NDYyMTIwOTYgMTYzODQKMjQwMzAyMzI1NzYwIDE2Mzg0CjI0MDMwMjQyNDA2NCAx NjM4NAoyNDAzMDI1MjIzNjggMTYzODQKMjQwMzAyNjIwNjcyIDE2Mzg0CjI0MDMwMjc1MTc0NCAy MDQ4MAoyNDA5NTg4NjU0MDggMjA0ODAKMjQwOTU4OTYzNzEyIDE2Mzg0CjI0MDk1OTA5NDc4NCAy MDQ4MAoyNDA5NTkxOTMwODggMTYzODQKMjQwOTU5MjkxMzkyIDE2Mzg0CjI0MTYxNTMzOTUyMCAx NjM4NAoyNDE2MTU0Mzc4MjQgMTYzODQKMjQxNjE1NTM2MTI4IDE2Mzg0CjI0MTYxNTYzNDQzMiAx NjM4NAoyNDE2MTU3NjU1MDQgMjA0ODAKMjQyMjcxODQ2NDAwIDE2Mzg0CjI0MjI3MTk0NDcwNCAx NjM4NAoyNDIyNzIwNDMwMDggMTYzODQKMjQyMjcyMTQxMzEyIDE2Mzg0CjI0MjI3MjIzOTYxNiAx NjM4NAoyNDI5MjgzNTMyODAgMTYzODQKMjQyOTI4NDUxNTg0IDE2Mzg0CjI0MjkyODU0OTg4OCAx NjM4NAoyNDI5Mjg2NDgxOTIgMTYzODQKMjQyOTI4Nzc5MjY0IDIwNDgwCjI0MzU4NDg2MDE2MCAx NjM4NAoyNDM1ODQ5NTg0NjQgMTYzODQKMjQzNTg1MDU2NzY4IDE2Mzg0CjI0MzU4NTE1NTA3MiAx NjM4NAoyNDM1ODUyNTMzNzYgMTYzODQKMjQ0MjQxMzk5ODA4IDIwNDgwCjI0NDI0MTQ5ODExMiAx NjM4NAoyNDQyNDE1OTY0MTYgMTYzODQKMjQ0MjQxNzI3NDg4IDIwNDgwCjI0NDI0MTgyNTc5MiAx NjM4NAoyNDQ4OTc4NzM5MjAgMTYzODQKMjQ0ODk3OTcyMjI0IDE2Mzg0CjI0NDg5ODA3MDUyOCAx NjM4NAoyNDQ4OTgxNjg4MzIgMTYzODQKMjQ0ODk4MjY3MTM2IDE2Mzg0CjI0NTU1NDM4MDgwMCAx NjM4NAoyNDU1NTQ0NzkxMDQgMTYzODQKMjQ1NTU0NTc3NDA4IDE2Mzg0CjI0NTU1NDY3NTcxMiAx NjM4NAoyNDU1NTQ4MDY3ODQgMjA0ODAKMjQ2MjEwODg3NjgwIDE2Mzg0CjI0NjIxMDk4NTk4NCAx NjM4NAoyNDYyMTEwODQyODggMTYzODQKMjQ2MjExMTgyNTkyIDE2Mzg0CjI0NjIxMTI4MDg5NiAx NjM4NAoyNDY4NjczOTQ1NjAgMTYzODQKMjQ2ODY3NDkyODY0IDE2Mzg0CjI0Njg2NzU5MTE2OCAx NjM4NAoyNDY4Njc2ODk0NzIgMTYzODQKMjQ2ODY3Nzg3Nzc2IDE2Mzg0CjI0NzUyMzkwMTQ0MCAx NjM4NAoyNDc1MjQwMzI1MTIgMjA0ODAKMjQ3NTI0MTMwODE2IDE2Mzg0CjI0NzUyNDIyOTEyMCAx NjM4NAoyNDc1MjQzMjc0MjQgMTYzODQKMjQ4MTgwNDA4MzIwIDE2Mzg0CjI0ODE4MDUwNjYyNCAx NjM4NAoyNDgxODA2MDQ5MjggMTYzODQKMjQ4MTgwNzAzMjMyIDE2Mzg0CjI0ODE4MDgwMTUzNiAx NjM4NAoyNDg4MzY5MTUyMDAgMTYzODQKMjQ4ODM3MDEzNTA0IDE2Mzg0CjI0ODgzNzExMTgwOCAx NjM4NAoyNDg4MzcyNDI4ODAgMjA0ODAKMjQ4ODM3MzQxMTg0IDE2Mzg0CjI0OTQ5MzQyMjA4MCAx NjM4NAoyNDk0OTM1MjAzODQgMTYzODQKMjQ5NDkzNjE4Njg4IDE2Mzg0CjI0OTQ5MzcxNjk5MiAx NjM4NAoyNDk0OTM4MTUyOTYgMTYzODQKMjUwMTQ5OTI4OTYwIDE2Mzg0CjI1MDE1MDAyNzI2NCAx NjM4NAoyNTAxNTAxMjU1NjggMTYzODQKMjUwMTUwMjIzODcyIDE2Mzg0CjI1MDE1MDMyMjE3NiAx NjM4NAoyNTA4MDY0MzU4NDAgMTYzODQKMjUwODA2NTM0MTQ0IDE2Mzg0CjI1MDgwNjYzMjQ0OCAx NjM4NAoyNTA4MDY3MzA3NTIgMTYzODQKMjUxNDYyOTQyNzIwIDE2Mzg0CjI1MTQ2MzA0MTAyNCAx NjM4NAoyNTE0NjMxMzkzMjggMTYzODQKMjUxNDYzMjM3NjMyIDE2Mzg0CjI1MTQ2MzMzNTkzNiAx NjM4NAoyNTIxMTk0NDk2MDAgMTYzODQKMjUyMTE5NTQ3OTA0IDE2Mzg0CjI1MjExOTY3ODk3NiAy MDQ4MAoyNTIxMTk3NzcyODAgMTYzODQKMjUyMTE5OTA4MzUyIDIwNDgwCjI1Mjc3NTk1NjQ4MCAx NjM4NAoyNTI3NzYwNTQ3ODQgMTYzODQKMjUyNzc2MTUzMDg4IDE2Mzg0CjI1Mjc3NjI4NDE2MCAy MDQ4MAoyNTI3NzYzODI0NjQgMTYzODQKMjUzNDMyNDk2MTI4IDIwNDgwCjI1MzQzMjU5NDQzMiAx NjM4NAoyNTM0MzI2OTI3MzYgMTYzODQKMjUzNDMyODIzODA4IDIwNDgwCjI1MzQzMjkyMjExMiAx NjM4NAoyNTQwODg5NzAyNDAgMTYzODQKMjU0MDg5MDY4NTQ0IDE2Mzg0CjI1NDA4OTE2Njg0OCAx NjM4NAoyNTQwODkyNjUxNTIgMTYzODQKMjU0MDg5MzYzNDU2IDE2Mzg0CjI1NDc0NTQ3NzEyMCAx NjM4NAoyNTQ3NDU1NzU0MjQgMTYzODQKMjU0NzQ1NjczNzI4IDE2Mzg0CjI1NDc0NTc3MjAzMiAx NjM4NAoyNTQ3NDU4NzAzMzYgMTYzODQKMjU1NDAxOTg0MDAwIDE2Mzg0CjI1NTQwMjExNTA3MiAy MDQ4MAoyNTU0MDIyMTMzNzYgMTYzODQKMjU1NDAyMzExNjgwIDE2Mzg0CjI1NjA1ODQ5MDg4MCAx NjM4NAoyNTYwNTg1ODkxODQgMTYzODQKMjU2MDU4Njg3NDg4IDE2Mzg0CjI1NjA1ODgxODU2MCAy MDQ4MAoyNTYwNTg5MTY4NjQgMTYzODQKMjU2NzE1MDMwNTI4IDIwNDgwCjI1NjcxNTE2MTYwMCAy MDQ4MAoyNTY3MTUyNTk5MDQgMTYzODQKMjU2NzE1MzU4MjA4IDE2Mzg0CjI1NjcxNTQ1NjUxMiAx NjM4NAoyNTczNzE1MDQ2NDAgMTYzODQKMjU3MzcxNjAyOTQ0IDE2Mzg0CjI1NzM3MTcwMTI0OCAx NjM4NAoyNTczNzE3OTk1NTIgMTYzODQKMjU3MzcxODk3ODU2IDE2Mzg0CjI1ODAyODAxMTUyMCAx NjM4NAoyNTgwMjgxMDk4MjQgMTYzODQKMjU4MDI4MjA4MTI4IDE2Mzg0CjI1ODAyODMwNjQzMiAx NjM4NAoyNTgwMjg0MDQ3MzYgMTYzODQKMjU4Njg0NTE4NDAwIDE2Mzg0CjI1ODY4NDYxNjcwNCAx NjM4NAoyNTg2ODQ3MTUwMDggMTYzODQKMjU4Njg0ODEzMzEyIDE2Mzg0CjI1ODY4NDk0NDM4NCAy MDQ4MAoyNTkzNDEwMjUyODAgMTYzODQKMjU5MzQxMTIzNTg0IDE2Mzg0CjI1OTM0MTIyMTg4OCAx NjM4NAoyNTkzNDEzMjAxOTIgMTYzODQKMjU5MzQxNDE4NDk2IDE2Mzg0CjI1OTk5NzU2NDkyOCAy MDQ4MAoyNTk5OTc2NjMyMzIgMTYzODQKMjU5OTk3NzYxNTM2IDE2Mzg0CjI1OTk5Nzg1OTg0MCAx NjM4NAoyNTk5OTc5NTgxNDQgMTYzODQKMjU5OTk4MDg5MjE2IDIwNDgwCjI2MDY1NDAzOTA0MCAx NjM4NAoyNjA2NTQxMzczNDQgMTYzODQKMjYwNjU0MjM1NjQ4IDE2Mzg0CjI2MDY1NDM2NjcyMCAy MDQ4MAoyNjA2NTQ0NjUwMjQgMTYzODQKMjYxMzEwNTQ1OTIwIDE2Mzg0CjI2MTMxMDY0NDIyNCAx NjM4NAoyNjEzMTA3NDI1MjggMTYzODQKMjYxMzEwODQwODMyIDE2Mzg0CjI2MTMxMDk3MTkwNCAy MDQ4MAoyNjE5NjcwNTI4MDAgMTYzODQKMjYxOTY3MTUxMTA0IDE2Mzg0CjI2MTk2NzI0OTQwOCAx NjM4NAoyNjE5NjczNDc3MTIgMTYzODQKMjYxOTY3NDQ2MDE2IDE2Mzg0CjI2MjYyMzU1OTY4MCAx NjM4NAoyNjI2MjM2OTA3NTIgMjA0ODAKMjYyNjIzNzg5MDU2IDE2Mzg0CjI2MjYyMzg4NzM2MCAx NjM4NAoyNjI2MjM5ODU2NjQgMTYzODQKMjYzMjgwMDY2NTYwIDE2Mzg0CjI2MzI4MDE5NzYzMiAy MDQ4MAoyNjMyODAzMjg3MDQgMjA0ODAKMjYzMjgwNDI3MDA4IDE2Mzg0CjI2MzI4MDUyNTMxMiAx NjM4NAoyNjM5MzY1NzM0NDAgMTYzODQKMjYzOTM2NzA0NTEyIDIwNDgwCjI2MzkzNjgwMjgxNiAx NjM4NAoyNjM5MzY5MDExMjAgMTYzODQKMjYzOTM2OTk5NDI0IDE2Mzg0CjI2NDU5MzA4MDMyMCAx NjM4NAoyNjQ1OTMxNzg2MjQgMTYzODQKMjY0NTkzMjc2OTI4IDE2Mzg0CjI2NDU5MzM3NTIzMiAx NjM4NAoyNjUyNDk2MTk5NjggMjA0ODAKMjY1MjQ5NzE4MjcyIDE2Mzg0CjI2NTI0OTg0OTM0NCAy MDQ4MAoyNjUyNDk5ODA0MTYgMjA0ODAKMjY1MjUwMDc4NzIwIDE2Mzg0CjI2NTkwNjA5NDA4MCAx NjM4NAoyNjU5MDYxOTIzODQgMTYzODQKMjY1OTA2MjkwNjg4IDE2Mzg0CjI2NTkwNjM4ODk5MiAx NjM4NAoyNjU5MDY0ODcyOTYgMTYzODQKMjY2NTYyNjAwOTYwIDE2Mzg0CjI2NjU2MjY5OTI2NCAx NjM4NAoyNjY1NjI3OTc1NjggMTYzODQKMjY2NTYyODk1ODcyIDE2Mzg0CjI2NjU2Mjk5NDE3NiAx NjM4NAoyNjcyMTkxMDc4NDAgMTYzODQKMjY3MjE5MjA2MTQ0IDE2Mzg0CjI2NzIxOTMzNzIxNiAy MDQ4MAoyNjcyMTk0MzU1MjAgMTYzODQKMjY3MjE5NTMzODI0IDE2Mzg0CjI2Nzg3NTY0NzQ4OCAy MDQ4MAoyNjc4NzU3NDU3OTIgMTYzODQKMjY3ODc1ODQ0MDk2IDE2Mzg0CjI2Nzg3NTk0MjQwMCAx NjM4NAoyNjc4NzYwNDA3MDQgMTYzODQKMjY4NTMyMTIxNjAwIDE2Mzg0CjI2ODUzMjIxOTkwNCAx NjM4NAoyNjg1MzIzMTgyMDggMTYzODQKMjY4NTMyNDE2NTEyIDE2Mzg0CjI2ODUzMjUxNDgxNiAx NjM4NAoyNjkxODg2Mjg0ODAgMTYzODQKMjY5MTg4NzU5NTUyIDIwNDgwCjI2OTE4ODg1Nzg1NiAx NjM4NAoyNjkxODg5NTYxNjAgMTYzODQKMjY5MTg5MDU0NDY0IDE2Mzg0CjI2OTg0NTEzNTM2MCAx NjM4NAoyNjk4NDUyNjY0MzIgMjA0ODAKMjY5ODQ1MzY0NzM2IDE2Mzg0CjI2OTg0NTQ5NTgwOCAy MDQ4MAoyNjk4NDU1OTQxMTIgMTYzODQKMjcwNTAxNjQyMjQwIDE2Mzg0CjI3MDUwMTc3MzMxMiAy MDQ4MAoyNzA1MDE5MDQzODQgMjA0ODAKMjcwNTAyMDAyNjg4IDE2Mzg0CjI3MDUwMjEwMDk5MiAx NjM4NAoyNzExNTgxNDkxMjAgMTYzODQKMjcxMTU4MjQ3NDI0IDE2Mzg0CjI3MTE1ODM0NTcyOCAx NjM4NAoyNzExNTg0NzY4MDAgMjA0ODAKMjcxMTU4NTc1MTA0IDE2Mzg0CjI3MTgxNDY1NjAwMCAx NjM4NAoyNzE4MTQ3ODcwNzIgMjA0ODAKMjcxODE0ODg1Mzc2IDE2Mzg0CjI3MTgxNDk4MzY4MCAx NjM4NAoyNzE4MTUwODE5ODQgMTYzODQKMjcyNDcxMTk1NjQ4IDIwNDgwCjI3MjQ3MTMyNjcyMCAy MDQ4MAoyNzI0NzE0MjUwMjQgMTYzODQKMjcyNDcxNTIzMzI4IDE2Mzg0CjI3MjQ3MTYyMTYzMiAx NjM4NAoyNzMxMjc2Njk3NjAgMTYzODQKMjczMTI3NzY4MDY0IDE2Mzg0CjI3MzEyNzg5OTEzNiAy MDQ4MAoyNzMxMjgwMzAyMDggMjA0ODAKMjczMTI4MTI4NTEyIDE2Mzg0CjI3Mzc4NDE3NjY0MCAx NjM4NAoyNzM3ODQyNzQ5NDQgMTYzODQKMjczNzg0MzczMjQ4IDE2Mzg0CjI3Mzc4NDUwNDMyMCAy MDQ4MAoyNzM3ODQ2MDI2MjQgMTYzODQKMjc0NDQwNjgzNTIwIDE2Mzg0CjI3NDQ0MDc4MTgyNCAx NjM4NAoyNzQ0NDA4ODAxMjggMTYzODQKMjc0NDQwOTc4NDMyIDE2Mzg0CjI3NDQ0MTA3NjczNiAx NjM4NAoyNzUwOTcxOTA0MDAgMTYzODQKMjc1MDk3Mjg4NzA0IDE2Mzg0CjI3NTA5NzM4NzAwOCAx NjM4NAoyNzUwOTc1MTgwODAgMjA0ODAKMjc1MDk3NjE2Mzg0IDE2Mzg0CjI3NTc1MzY5NzI4MCAx NjM4NAoyNzU3NTM3OTU1ODQgMTYzODQKMjc1NzUzODkzODg4IDE2Mzg0CjI3NTc1NDAyNDk2MCAy MDQ4MAoyNzU3NTQxMjMyNjQgMTYzODQKMjc2NDEwMjM2OTI4IDIwNDgwCjI3NjQxMDMzNTIzMiAx NjM4NAoyNzY0MTA0MzM1MzYgMTYzODQKMjc2NDEwNTMxODQwIDE2Mzg0CjI3NjQxMDY2MjkxMiAy MDQ4MAoyNzcwNjY3MTEwNDAgMTYzODQKMjc3MDY2ODA5MzQ0IDE2Mzg0CjI3NzA2NjkwNzY0OCAx NjM4NAoyNzcwNjcwMzg3MjAgMjA0ODAKMjc3MDY3MTM3MDI0IDE2Mzg0CjI3NzcyMzIxNzkyMCAx NjM4NAoyNzc3MjMzMTYyMjQgMTYzODQKMjc3NzIzNDQ3Mjk2IDIwNDgwCjI3NzcyMzU0NTYwMCAx NjM4NAoyNzc3MjM2NDM5MDQgMTYzODQKMjc4Mzc5NzI0ODAwIDE2Mzg0CjI3ODM3OTgyMzEwNCAx NjM4NAoyNzgzNzk5MjE0MDggMTYzODQKMjc4MzgwMDUyNDgwIDIwNDgwCjI3ODM4MDE1MDc4NCAx NjM4NAoyNzk5MzMzNTM5ODQgMTYzODQKMjc5OTk0MDQwMzIwIDIwNDgwCjI4MDAzNTMyODAwMCAx NjM4NAoyODAzNDkyNDU0NDAgMTYzODQKMjgwMzQ5MzQzNzQ0IDE2Mzg0CjI4MDM0OTQ0MjA0OCAx NjM4NAoyODAzNDk1NDAzNTIgMTYzODQKMjgwMzQ5NjM4NjU2IDE2Mzg0CjI4MTAwNTc1MjMyMCAx NjM4NAoyODEwMDU4NTA2MjQgMTYzODQKMjgxMDA1OTQ4OTI4IDE2Mzg0CjI4MTAwNjA0NzIzMiAx NjM4NAoyODEwMDYxNzgzMDQgMjA0ODAKMjgxNjYyMjU5MjAwIDE2Mzg0CjI4MTY2MjM1NzUwNCAx NjM4NAoyODE2NjI0ODg1NzYgMjA0ODAKMjgxNjYyNjE5NjQ4IDIwNDgwCjI4MTY2MjcxNzk1MiAx NjM4NAoyODIzMTg3NjYwODAgMTYzODQKMjgyMzE4ODk3MTUyIDIwNDgwCjI4MjMxOTAyODIyNCAy MDQ4MAoyODIzMTkxMjY1MjggMTYzODQKMjgyMzE5MjI0ODMyIDE2Mzg0CjI4Mjk3NTI3Mjk2MCAx NjM4NAoyODI5NzUzNzEyNjQgMTYzODQKMjgyOTc1NDY5NTY4IDE2Mzg0CjI4Mjk3NTU2Nzg3MiAx NjM4NAoyODI5NzU2NjYxNzYgMTYzODQKMjgzNjMxNzc5ODQwIDE2Mzg0CjI4MzYzMTg3ODE0NCAx NjM4NAoyODM2MzIwMDkyMTYgMjA0ODAKMjgzNjMyMTA3NTIwIDE2Mzg0CjI4MzYzMjIwNTgyNCAx NjM4NAoyODQyODgyODY3MjAgMTYzODQKMjg0Mjg4NDE3NzkyIDIwNDgwCjI4NDI4ODUxNjA5NiAx NjM4NAoyODQyODg2MTQ0MDAgMTYzODQKMjg0Mjg4NzEyNzA0IDE2Mzg0CjI4NDk0NDc5MzYwMCAx NjM4NAoyODQ5NDQ4OTE5MDQgMTYzODQKMjg0OTQ1MDIyOTc2IDIwNDgwCjI4NDk0NTEyMTI4MCAx NjM4NAoyODQ5NDUyMTk1ODQgMTYzODQKMjg2MzcyMjk4NzUyIDE2Mzg0CjI4NjQyMDYzMTU1MiAx NjM4NAoyODY0NjI4MDM5NjggMTYzODQKMjg2NTEzMTAyODQ4IDE2Mzg0CjI4NjU2MzEzOTU4NCAx NjM4NAoyODY2MjM1OTY1NDQgMjA0ODAKMjg2OTE0MzE0MjQwIDE2Mzg0CjI4NjkxNDQxMjU0NCAx NjM4NAoyODY5MTQ1MTA4NDggMTYzODQKMjg2OTE0NjA5MTUyIDE2Mzg0CjI4NjkxNDcwNzQ1NiAx NjM4NAoyODY5MTQ4MDU3NjAgMTYzODQKMjg3NTcwODIxMTIwIDE2Mzg0CjI4NzU3MDkxOTQyNCAx NjM4NAoyODc1NzEwMTc3MjggMTYzODQKMjg3NTcxMTE2MDMyIDE2Mzg0CjI4NzU3MTIxNDMzNiAx NjM4NAoyODgyMjczNjA3NjggMjA0ODAKMjg4MjI3NDU5MDcyIDE2Mzg0CjI4ODIyNzU1NzM3NiAx NjM4NAoyODgyMjc2ODg0NDggMjA0ODAKMjg4MjI3Nzg2NzUyIDE2Mzg0CjI4ODg4MzgzNDg4MCAx NjM4NAoyODg4ODM5MzMxODQgMTYzODQKMjg4ODg0MDMxNDg4IDE2Mzg0CjI4ODg4NDEyOTc5MiAx NjM4NAoyODg4ODQyMjgwOTYgMTYzODQKMjg4ODg0MzI2NDAwIDE2Mzg0CjI4OTU0MDM0MTc2MCAx NjM4NAoyODk1NDA0NDAwNjQgMTYzODQKMjg5NTQwNTM4MzY4IDE2Mzg0CjI4OTU0MDYzNjY3MiAx NjM4NAoyODk1NDA3Njc3NDQgMjA0ODAKMjkwMTk2ODQ4NjQwIDE2Mzg0CjI5MDE5Njk0Njk0NCAx NjM4NAoyOTAxOTcwNDUyNDggMTYzODQKMjkwMTk3MTQzNTUyIDE2Mzg0CjI5MDE5NzI0MTg1NiAx NjM4NAoyOTA4NTMzNTU1MjAgMTYzODQKMjkwODUzNDg2NTkyIDIwNDgwCjI5MDg1MzU4NDg5NiAx NjM4NAoyOTA4NTM2ODMyMDAgMTYzODQKMjkwODUzNzgxNTA0IDE2Mzg0CjI5MTUwOTg2MjQwMCAx NjM4NAoyOTE1MDk5NjA3MDQgMTYzODQKMjkxNTEwMDkxNzc2IDIwNDgwCjI5MTUxMDIyMjg0OCAy MDQ4MAoyOTE1MTAzMjExNTIgMTYzODQKMjkyMTY2MzY5MjgwIDE2Mzg0CjI5MjE2NjUwMDM1MiAy MDQ4MAoyOTIxNjY1OTg2NTYgMTYzODQKMjkyMTY2Njk2OTYwIDE2Mzg0CjI5MjE2Njc5NTI2NCAx NjM4NAoyOTI4MjI4NzYxNjAgMTYzODQKMjkyODIzMDA3MjMyIDIwNDgwCjI5MjgyMzEzODMwNCAy MDQ4MAoyOTI4MjMyMzY2MDggMTYzODQKMjkyODIzMzM0OTEyIDE2Mzg0CjI5MzQ3OTM4MzA0MCAx NjM4NAoyOTM0Nzk0ODEzNDQgMTYzODQKMjkzNDc5NTc5NjQ4IDE2Mzg0CjI5MzQ3OTY3Nzk1MiAx NjM4NAoyOTM0Nzk4MDkwMjQgMjA0ODAKMjk0MTM1OTIyNjg4IDIwNDgwCjI5NDEzNjAyMDk5MiAx NjM4NAoyOTQxMzYxMTkyOTYgMTYzODQKMjk0MTM2MjE3NjAwIDE2Mzg0CjI5NDEzNjMxNTkwNCAx NjM4NAoyOTQ3OTI0Mjk1NjggMjA0ODAKMjk0NzkyNTYwNjQwIDIwNDgwCjI5NDc5MjY1ODk0NCAx NjM4NAoyOTQ3OTI3NTcyNDggMTYzODQKMjk0NzkyODU1NTUyIDE2Mzg0CjI5NTQ0ODkwMzY4MCAx NjM4NAoyOTU0NDkwMDE5ODQgMTYzODQKMjk1NDQ5MTAwMjg4IDE2Mzg0CjI5NTQ0OTE5ODU5MiAx NjM4NAoyOTU0NDkyOTY4OTYgMTYzODQKMjk2MTA1NDEwNTYwIDE2Mzg0CjI5NjEwNTUwODg2NCAx NjM4NAoyOTYxMDU2MDcxNjggMTYzODQKMjk2MTA1NzA1NDcyIDE2Mzg0CjI5NjEwNTgwMzc3NiAx NjM4NAoyOTc1MDM0OTAwNDggMTYzODQKMjk3NTUzNzIzMzkyIDE2Mzg0CjI5NzYwMDc3ODI0MCAx NjM4NAoyOTc2NjI2MTE0NTYgMjA0ODAKMjk3Njk5NDA5OTIwIDE2Mzg0CjI5Nzc1MzE4MjIwOCAx NjM4NAoyOTc4MDA3Mjg1NzYgMTYzODQKMjk4MDc0OTYzOTY4IDIwNDgwCjI5ODA3NTA2MjI3MiAx NjM4NAoyOTgwNzUxNjA1NzYgMTYzODQKMjk4MDc1MjU4ODgwIDE2Mzg0CjI5ODA3NTM1NzE4NCAx NjM4NAoyOTg3MzE0MzgwODAgMTYzODQKMjk4NzMxNTM2Mzg0IDE2Mzg0CjI5ODczMTYzNDY4OCAx NjM4NAoyOTg3MzE3MzI5OTIgMTYzODQKMjk4NzMxODY0MDY0IDIwNDgwCjI5OTM4Nzk0NDk2MCAx NjM4NAoyOTkzODgwNzYwMzIgMjA0ODAKMjk5Mzg4MTc0MzM2IDE2Mzg0CjI5OTM4ODI3MjY0MCAx NjM4NAoyOTkzODgzNzA5NDQgMTYzODQKMzAwMDQ0NDUxODQwIDE2Mzg0CjMwMDA0NDU1MDE0NCAx NjM4NAozMDAwNDQ2NDg0NDggMTYzODQKMzAwMDQ0NzQ2NzUyIDE2Mzg0CjMwMDA0NDg0NTA1NiAx NjM4NAozMDA3MDA5NTg3MjAgMTYzODQKMzAwNzAxMDU3MDI0IDE2Mzg0CjMwMDcwMTE1NTMyOCAx NjM4NAozMDA3MDEyNTM2MzIgMTYzODQKMzAwNzAxMzUxOTM2IDE2Mzg0CjMwMTM1NzQ2NTYwMCAx NjM4NAozMDEzNTc1NjM5MDQgMTYzODQKMzAxMzU3NjYyMjA4IDE2Mzg0CjMwMTM1Nzc2MDUxMiAx NjM4NAozMDEzNTc4NTg4MTYgMTYzODQKMzAyODczNDExNTg0IDE2Mzg0CjMwMjkyMTUxNTAwOCAx NjM4NAozMDI5NzQ3MzAyNDAgMTYzODQKMzAzMDI0MDc4ODQ4IDE2Mzg0CjMwMzMyNjk4NjI0MCAx NjM4NAozMDMzMjcxMTczMTIgMjA0ODAKMzAzMzI3MjE1NjE2IDE2Mzg0CjMwMzMyNzMxMzkyMCAx NjM4NAozMDMzMjc0MTIyMjQgMTYzODQKMzAzOTgzNDkzMTIwIDE2Mzg0CjMwMzk4MzU5MTQyNCAx NjM4NAozMDM5ODM2ODk3MjggMTYzODQKMzAzOTgzNzg4MDMyIDE2Mzg0CjMwMzk4Mzg4NjMzNiAx NjM4NAozMDQ2NDAwMDAwMDAgMTYzODQKMzA0NjQwMDk4MzA0IDE2Mzg0CjMwNDY0MDIyOTM3NiAy MDQ4MAozMDQ2NDAzMjc2ODAgMTYzODQKMzA1Mjk2NTA2ODgwIDE2Mzg0CjMwNTI5NjYwNTE4NCAx NjM4NAozMDUyOTY3MzYyNTYgMjA0ODAKMzA1Mjk2ODM0NTYwIDE2Mzg0CjMwNTI5NjkzMjg2NCAx NjM4NAozMDU5NTMwMTM3NjAgMTYzODQKMzA1OTUzMTQ0ODMyIDIwNDgwCjMwNTk1MzI0MzEzNiAx NjM4NAozMDU5NTMzNDE0NDAgMTYzODQKMzA1OTUzNDcyNTEyIDIwNDgwCjMwNjYwOTU1MzQwOCAy MDQ4MAozMDY2MDk2ODQ0ODAgMjA0ODAKMzA2NTg5MDA3ODcyIDMyNzY4Cg== --0000000000001e1fa005744842c1-- From owner-freebsd-hackers@freebsd.org Sat Aug 25 20:45:03 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 043271095F18 for ; Sat, 25 Aug 2018 20:45:03 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D89281AFB for ; Sat, 25 Aug 2018 20:45:02 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w7PKir1e017756 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 25 Aug 2018 22:44:53 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: dcrosstech@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w7PKimRf051560 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 26 Aug 2018 03:44:48 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Weird USB DA behavior To: David Cross , FreeBSD Hackers References: <20180825010023.GD45503@funkthat.com> <20180825052330.GE45503@funkthat.com> From: Eugene Grosbein Message-ID: <83388164-39c8-14e7-d0e7-304cbfdcb796@grosbein.net> Date: Sun, 26 Aug 2018 03:44:43 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -0.0 SPF_PASS SPF: sender matches SPF record * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 20:45:03 -0000 26.08.2018 3:22, David Cross wrote: > I turned debugging ALL the way up on geli (3) and noticed the hangs always > happened when geli handed off a 20480 length read to the layer below (in > this case mirror).. I used the rebugging output to create a dummy program > called 'pread' to try to simulate these failures, and I was quite > successful. Attached is 'pread.c' which takes on stdin a offset and a > length to read. In that version I overwrote it to always ask for 32k., and > that works every time. If I eliminate that line and let the input data > control it, on some of the 20480 (and a different set each time) it hangs, > after the hang the pread returns _0_ (and obviously no errno), no kernel > logs. "pread" is run directly against /dev/da0, so no mirror or geli > layers at this point. The lists strips non-textual attachments, so requestdata.txt get passed but pread.c was stripped from the message. You need to re-send it changing name to something like pread.c.txt