From owner-freebsd-geom@freebsd.org Sun Jun 28 21:01:13 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4B426355547 for ; Sun, 28 Jun 2020 21:01:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 49w32070Rnz3b53 for ; Sun, 28 Jun 2020 21:01:12 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id ED35F355546; Sun, 28 Jun 2020 21:01:12 +0000 (UTC) Delivered-To: geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E7458355545 for ; Sun, 28 Jun 2020 21:01:12 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49w3205Hzdz3bRJ for ; Sun, 28 Jun 2020 21:01:12 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 95CF51C70E for ; Sun, 28 Jun 2020 21:01:12 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 05SL1CtD016684 for ; Sun, 28 Jun 2020 21:01:12 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 05SL1CMQ016683 for geom@FreeBSD.org; Sun, 28 Jun 2020 21:01:12 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202006282101.05SL1CMQ016683@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: geom@FreeBSD.org Subject: Problem reports for geom@FreeBSD.org that need special attention Date: Sun, 28 Jun 2020 21:01:12 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jun 2020 21:01:13 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 218679 | [geli] add a verify command Open | 237269 | panic in glabel (g_label_destroy) stop after resi Open | 238814 | geom: topology lock being dropped in dumpconf of Open | 242747 | geli: AMD Epyc+GELI not using Hardware AES 4 problems total for which you should take action. From owner-freebsd-geom@freebsd.org Mon Jun 29 04:35:18 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 35D5135D7F2 for ; Mon, 29 Jun 2020 04:35:18 +0000 (UTC) (envelope-from bunky@statepromote.xyz) Received: from statepromote.xyz (statepromote.xyz [185.225.39.141]) by mx1.freebsd.org (Postfix) with ESMTP id 49wF5x4vrsz4HwM for ; Mon, 29 Jun 2020 04:35:17 +0000 (UTC) (envelope-from bunky@statepromote.xyz) From: "Bye Insects" Date: Sun, 28 Jun 2020 23:32:22 -0500 MIME-Version: 1.0 Subject: How I removed all annoying insects from my home and office. To: Message-ID: X-Rspamd-Queue-Id: 49wF5x4vrsz4HwM X-Spamd-Bar: + X-Spamd-Result: default: False [1.19 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.81)[-0.809]; R_DKIM_ALLOW(-0.20)[statepromote.xyz:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:185.225.39.141:c]; HTML_SHORT_LINK_IMG_1(2.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ZERO_FONT(0.10)[1]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.50)[-0.495]; MANY_INVISIBLE_PARTS(0.05)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[statepromote.xyz:+]; DMARC_POLICY_ALLOW(-0.50)[statepromote.xyz,quarantine]; NEURAL_SPAM_SHORT(0.51)[0.515]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:202505, ipnet:185.225.39.0/24, country:TR]; MID_RHS_MATCH_FROM(0.00)[]; R_PARTS_DIFFER(0.83)[91.5%] Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2020 04:35:18 -0000 In this case, people with the fixed mindset would agree with statements such as: Your core interests will remain your core interests; they wont really change Yet Hardy says it offers a greater degree of freedom than carbon taxes 2 metres (87 inches) of snow per year, which means solar panels can end up blanketed in snow and ice during the winter months What youre looking at is effectively making carbon into a second currency Adam Hardy With customers only given a limited number of carbon rations to spend each week, everyone from supermarkets to saunas would be incentivised to reduce the carbon footprint of their offering, thereby making it more attractive to consumers The megacity, which is home to more than 10 million people, gets less than an inch of rain each year Author image By Allison Hirschlag 27th February 2020 F From the platform it looked like any other British commuter train The rickshaw, the vehicle of the poor, is where electrification began in India, says Anup Bandivad ekar, India lead with the International Council on Clean Transportation in San Francisco More than 125 Nuscale reactors could be put in a traditional reactors containment building, though the company plans to deploy them in groups of 12 Before the existence of this fatwa, there was no straightforward explanation on why animals had to be preserved and why Muslims were encouraged to protect them, he says Recently we spoke to him and asked him if he did it on purpose and he said, Actually no Our boss Deke Slayton had also smuggled three shots of brandy on board but we didnt drink that, Borman says There will be some similarities: in both, visitors (four guests with two staff) will nap in sleeping bags attached to the superstructure, the food will be freeze-dried, and all guests will have to go through a vigorous pre-launch health screening So, at least in theory, cells that could form a very different life in Titans vast methane oceans could physically exist there Though not everyone a grees it exists, the phenomenon does appear to show u p in various situations, including simulated space missions You might also like: Apollo in 50 numbers: The full list The most beautiful photos of Apollo 11 Apollo 13: The unsung heroes who prevented disaster At the launchpad, the ground was wet from storms that recently passed through the area and the sky is overcast If they are, itll mark another small step in our understanding of the human microbiome Since then, revenue at the Australian company has increased by 46%, and profits nearly tripled, says its CEO and founder Kath Blackham Im not sure how a company necessarily would be able to justify the cost other than that it has good optics for brand reputation and generating goodwill with departing employees, says Lorri Freifeld, editor-in-chief of Training Magazine Then there was a knock at the door, and in came the institutes director For that kind of role, Telus wants employees who have gaming experience themselves Here, tax subsidies through a voucher scheme have been in place fo r more than 15 years, with more than a million users in a country of 11 As we enter the final half of the Premiership season and approach the knockout stages of Europe, the loss of three players is a blow, said Leicester head coach Geordan Murphy These experiments are inspired by American psychologist Carol Dwecks pioneering research on mindset button {font-size: 25px;color: white;background-color: #7E3517;padding: 20px 50px;border-style: none;border-radius: 25px;text-align: center;text-decoration: none;margin: 20px;} Check it Out Here If you do not wish to continue receiving email newsletters CLICK HERE or send post-mail To 913 Lake Ave. New York, NY 10913 From owner-freebsd-geom@freebsd.org Mon Jun 29 07:40:20 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A47A341694 for ; Mon, 29 Jun 2020 07:40:20 +0000 (UTC) (envelope-from lemon@trailintention.xyz) Received: from trailintention.xyz (trailintention.xyz [185.225.39.152]) by mx1.freebsd.org (Postfix) with ESMTP id 49wKCR4KlWz4Shc for ; Mon, 29 Jun 2020 07:40:19 +0000 (UTC) (envelope-from lemon@trailintention.xyz) From: "Muscle-Builder" Date: Mon, 29 Jun 2020 02:32:16 -0500 MIME-Version: 1.0 Subject: Try Shark Tank Product to Melt Body Fat and Build Muscle Mass To: Message-ID: <4CIa-eKRVxBG6kcMsnAU0eVpWJucQcQm5B0sqnF572s.0DY3mF061YqShrIu77rIkzcTdTPnitC03GCwRu4Bkss@trailintention.xyz> X-Rspamd-Queue-Id: 49wKCR4KlWz4Shc X-Spamd-Bar: / X-Spamd-Result: default: False [0.51 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.81)[-0.809]; R_DKIM_ALLOW(-0.20)[trailintention.xyz:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:185.225.39.152:c]; HTML_SHORT_LINK_IMG_1(2.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ZERO_FONT(0.10)[1]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.50)[-0.498]; MANY_INVISIBLE_PARTS(0.05)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[trailintention.xyz:+]; DMARC_POLICY_ALLOW(-0.50)[trailintention.xyz,quarantine]; NEURAL_HAM_SHORT(-0.16)[-0.161]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:202505, ipnet:185.225.39.0/24, country:TR]; MID_RHS_MATCH_FROM(0.00)[]; R_PARTS_DIFFER(0.83)[91.5%] Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2020 07:40:20 -0000 So it's a good balance Yet Hardy says it offers a greater degree of freedom than carbon taxes In months like December, when its really dark, we just let the snow build up, as there isnt enough daylight to warrant snow clearing Jenn Miller Once installed, the operating costs for solar farms are minimal, another aspect that is attractive to investors and builders What youre looking at is effectively making carbon into a second currency Adam Hardy With customers only given a limited number of carbon rations to spend each week, everyone from supermarkets to saunas would be incentivised to reduce the carbon footprint of their offering, thereby making it more attractive to consumers The megacity, which is home to more than 10 million people, gets less than an inch of rain each year It was being shown off to the public in June 2019 for the first time on the tracks at the Quinton Rail Technology Centre, a test facility at Long Marston, near Stratford-upon-Avon, in England Various models pr oposed over the years by researchers including Sunita Chandel of the Women Institute of Technology Dehradun, India, and Priscilla Mulhall of the Illinois Institute of Technology in the US In 2018, the US Nuclear Regulatory Commission approved the first phase of review for the design Before the clerics introduced the concept of peatland restoration, many of the villagers had not known that they lived on peatlands for decades, he says Occasionally youre lucky enough to find one of those Its probably worth a lot of money now Aside from gazing out at the stars and back at Earth, it is expected that Aurora visitors will spend some of their stay tending micro-gravity experiments such as growing food, as is currently done by crews on the ISS Assuming any of these creatures evolve to be as social, intelligent and communicative as say cetaceans or elephants, and as manipulative, dexterous and clever as chimpanzees or orangutans, I see no reason why they could not eventually evolve more sophi sticated technological and cultural capacities This o ccurs when people realise they have as long left in their inescapable situation as they have already lived through, and lose motivation But with the rocket and crew ready to go, and US President Richard Nixon watching (for the first time) from the VIP stands, all systems were green for launch Of far more immediate concern, however, is the issue of bringing any Martian microbes back to Earth It is vindication for Blackham, who after a decade of weird and wonderful goes at flexibility had to convince her leadership team to trial the Wednesday-less week and vow to return to five days if it failed Yet these costs, while lower than other training methods, still fail to make financial sense for businesses in most cases, Fuller of Harvard University says An uninterrupted 11-hour break every 24 hours was guaranteed for all workers, bar a few exceptions That could be based on the accounts they follow or the links they post 5 million people A global survey in 2019 found 48% of respondents wo rk on their commute (Credit: Getty Images) In a survey of commuters travelling on the London to Birmingham and London to Aylesbury lines in 2016 and 2017, Jain found that 40% of those connecting to the internet via wi-fi or mobile data were using their time to do work emails You have to put yourself in other people's hands button {font-size: 25px;color: white;background-color: #7D1B7E;padding: 20px 50px;border-style: none;border-radius: 25px;text-align: center;text-decoration: none;margin: 20px;} Check it Out Here If you do not wish to continue receiving email newsletters CLICK HERE or send post-mail To 245 Ninth Ave. New York, NY 10245 From owner-freebsd-geom@freebsd.org Wed Jul 1 07:53:18 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 69839365E54 for ; Wed, 1 Jul 2020 07:53:18 +0000 (UTC) (envelope-from calif@careerup.xyz) Received: from careerup.xyz (careerup.xyz [185.225.39.63]) by mx1.freebsd.org (Postfix) with ESMTP id 49xYPV03rkz44HN for ; Wed, 1 Jul 2020 07:53:18 +0000 (UTC) (envelope-from calif@careerup.xyz) From: "iTrack Car" Date: Wed, 01 Jul 2020 02:50:39 -0500 MIME-Version: 1.0 Subject: Latest spy technology To: Message-ID: <1iE6Hl_vjbWmNC3hxFBIY97nkClX4r3WEZM1BANa5o8.aNXdr7MdD_PZibI9N8sgyh5cMOFVgBL-7Bhpgn4qryU@careerup.xyz> X-Rspamd-Queue-Id: 49xYPV03rkz44HN X-Spamd-Bar: ++++++ X-Spamd-Result: default: False [6.13 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(0.00)[careerup.xyz:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:185.225.39.63:c]; ZERO_FONT(0.10)[1]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; HTML_SHORT_LINK_IMG_2(1.00)[]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(1.03)[1.027]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; MANY_INVISIBLE_PARTS(0.05)[1]; RBL_VIRUSFREE_BOTNET(2.00)[185.225.39.63:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[careerup.xyz:+]; DMARC_POLICY_ALLOW(0.00)[careerup.xyz,quarantine]; NEURAL_SPAM_LONG(1.00)[0.997]; NEURAL_SPAM_SHORT(0.76)[0.756]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:202505, ipnet:185.225.39.0/24, country:TR]; MID_RHS_MATCH_FROM(0.00)[]; R_PARTS_DIFFER(0.20)[60.2%] X-Spam: Yes Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jul 2020 07:53:18 -0000 If the employee is willing to work on the way to work, especially on the railways, this is also working time for which payment is due Thomas Geiser Botachays arrangement is based on trust But, crucially, those who wanted to maintain high-impact lifestyles, such as regularly flying around the world, would have to buy extra carbon credits from someone else Willow averages 2 What youre looking at is effectively making carbon into a second currency Adam Hardy With customers only given a limited number of carbon rations to spend each week, everyone from supermarkets to saunas would be incentivised to reduce the carbon footprint of their offering, thereby making it more attractive to consumers The megacity, which is home to more than 10 million people, gets less than an inch of rain each year In one of its cars, passengers were encouraged to perch around four hydrogen fuel tanks, a fuel cell and two lithium batteries However, solar-assisted vehicles are not intended as a replacement for autos but rather an aid to pedal-powered rickshaws Many of the SMR designs in development simply shrink the systems of large-scale nuclear plants, using less fuel But Azmi is hoping to inspire his community by proving that it is possible to farm on peatlands without setting fires Hes deliberately taken the photographs, but hes not aware of their stereoscopic visibilities The Earth orbit insertion burn was accomplished on the far side of the Moon, out of touch of the ground had it failed, Id still be circling the Moon Aside from gazing out at the stars and back at Earth, it is expected that Aurora visitors will spend some of their stay tending micro-gravity experiments such as growing food, as is currently done by crews on the ISS There might emerge a fluorescence of new and very diverse organisms occupying a range of new niches, adds Sarah Blaffer Hrdy, a professor emerita in anthropology at the University of California, Davis Though not everyone agrees it exists, the phenomenon do es appear to show up in various situations, including simulated space missions But with the rocket and crew ready to go, and US President Richard Nixon watching (for the first time) from the VIP stands, all systems were green for launch In the meantime, microbiologists have something else to look forward to Adopting a four-day week has cut sick days and improved staff wellbeing for some firms (Credit: Getty Images) Making it work The policy was implemented in July last year For a lot of folks, they can't afford to quit work and do a training programme and still provide for kids and afford transportation' Amanda Bergson-Shilcock Heward-Mills stands at the helm of Outset, an independent outskilling company that streamlines the process for businesses all the way from employee training to new job placement Down time is taken very seriously in Europes biggest economy The company uses an automated system to flag up social media profiles that indicate a person has some connection to gaming The scheme has become so tremendously popula r, especially among the middle class, and those people are electorally very powerful, says professor Ive Marx at the University of Antwerp A controversial concept But despite the positive impact for customers and cleaners, the tax deduction system is an ongoing source of national debate People with a growth mindset believe your abilities can change over time, a perspective that helps dictate how they approach challenges (Credit: Getty Images) The long-term effects could be profound This Tiny Device Lets You Track Your Vehicles Using Your Smartphone What is a vehicle tracker? Quite simply, its a small electronic box that fits inside your vehicle, which sends signals with details of where it is and what speed it is travelling at. This super useful data comes from the GPS Satellite system, which also feeds position data on your vehicles to all your satnavs and devices. What are the benefits? In a word? Data. These modern tracking systems give you and your company huge amounts of powerful data that will let you run your fleet more efficiently which means saving a lot of money. Most importantly, you can also monitor driver behaviour data , such as harsh acceleration and braking, time using the cruise control, pedal position, rpm bands, fuel consumption, and engine diagnostics. This nifty little device will stop your workers driving like maniacs in a flash! And all of this rich telematics information is then fed back to you, directly on your PC or smart phone. GET FULL DETAILS If you do not wish to continue receiving email newsletters CLICK HERE or send post-mail To 668 First Ave. New York, NY 10668 From owner-freebsd-geom@freebsd.org Fri Jul 3 19:30:35 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 80C623575A6 for ; Fri, 3 Jul 2020 19:30:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49z4n65KLGz4dys for ; Fri, 3 Jul 2020 19:30:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f175.google.com with SMTP id l63so25864124oih.13 for ; Fri, 03 Jul 2020 12:30:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=PfgfJCgvHbQ7fy5R6aJpK009kwAg7TlzoMjYCffZT5o=; b=uGHQrdTqzPg1su1Tf4OJAtVt2JUCN0ROIlrgPUsoNlKOUhdtThFjVRD1StxxL6xWin PDiZyDMaH9CluEOU0MQ140tSdNwDHXxXo3ID0LxmSEc0zrEPm028vsgrZOL1NFlf4vYK hCwJvFdtKT3JOKxkK0nhxAUZ/KXk58GzPtbpaWqj43hISJt8T9xQOYAW3+WEBYV5d4eT 9BSvY6qL6dgyofU2c1Ka98zN68aueDa07KfTP0UeKq2MO2f+eEfJ/cIkB1CXzYmR2oWr RPAjgynIqqLwDr0fBvj19+5LfU0nwmjI193rvzaoXuNITenCJ4IGLvGeJeGex7RgTHoR /Mxg== X-Gm-Message-State: AOAM532b4sit4kKhuqZo17ztkF7WNb5ECK1uS7WSsjuBk2xanpNIj3j0 jidbocuzSA6oelMlFthMokYNgkrH/0VSBxuFM9T3cfnvfO4= X-Google-Smtp-Source: ABdhPJx2HP6Tn9PwaTp/vMoiJIlIT/eYmMHmeTOwFlHyGgpKntJATfU2ZMG7IhUXrS9lFk1z+V9IPUbvHwBdymSPnTc= X-Received: by 2002:aca:c391:: with SMTP id t139mr21208386oif.57.1593804633086; Fri, 03 Jul 2020 12:30:33 -0700 (PDT) MIME-Version: 1.0 From: Alan Somers Date: Fri, 3 Jul 2020 13:30:22 -0600 Message-ID: Subject: Single-threaded bottleneck in geli To: freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 49z4n65KLGz4dys X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.175 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-0.21 / 15.00]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.04)[-0.039]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_SHORT(0.35)[0.353]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.175:from]; NEURAL_HAM_MEDIUM(-0.52)[-0.521]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.175:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jul 2020 19:30:35 -0000 I'm using geli, gmultipath, and ZFS on a large system, with hundreds of drives. What I'm seeing is that under at least some workloads, the overall performance is limited by the single geom kernel process. procstat and kgdb aren't much help in telling exactly why this process is using so much CPU, but it certainly must be related to the fact that over 15,000 IOPs are going through that thread. What can I do to improve this situation? Would it make sense to enable direct dispatch for geli? That would hurt single-threaded performance, but probably improve performance for highly multithreaded workloads like mine. Example top output: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% geom{g_down} 13 root -8 - 0B 96K - 9 35.5H 25.32% geom{g_up} -Alan From owner-freebsd-geom@freebsd.org Fri Jul 3 20:18:23 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9AAA53582E6 for ; Fri, 3 Jul 2020 20:18:23 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 49z5rH2Cb8z3SHS; Fri, 3 Jul 2020 20:18:22 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from [192.168.250.156] (c-67-169-9-63.hsd1.ca.comcast.net [67.169.9.63]) by mail.dawidek.net (Postfix) with ESMTPSA id 22FE8445; Fri, 3 Jul 2020 22:18:16 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: Single-threaded bottleneck in geli From: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= In-Reply-To: Date: Fri, 3 Jul 2020 13:18:13 -0700 Cc: freebsd-geom@freebsd.org Message-Id: <48AB9191-5984-4BE0-BF98-7548424D06BF@dawidek.net> References: To: Alan Somers X-Mailer: iPhone Mail (17F80) X-Rspamd-Queue-Id: 49z5rH2Cb8z3SHS X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jul 2020 20:18:23 -0000 Hi Alan, why do you think it will hurt single-threaded performance? --=20 Pawe=C5=82 Jakub Dawidek > On Jul 3, 2020, at 12:30, Alan Somers wrote: >=20 > =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hundr= eds of > drives. What I'm seeing is that under at least some workloads, the overal= l > performance is limited by the single geom kernel process. procstat and > kgdb aren't much help in telling exactly why this process is using so much= > CPU, but it certainly must be related to the fact that over 15,000 IOPs ar= e > going through that thread. What can I do to improve this situation? Woul= d > it make sense to enable direct dispatch for geli? That would hurt > single-threaded performance, but probably improve performance for highly > multithreaded workloads like mine. >=20 > Example top output: > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > geom{g_down} > 13 root -8 - 0B 96K - 9 35.5H 25.32% > geom{g_up} >=20 > -Alan > _______________________________________________ > freebsd-geom@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" From owner-freebsd-geom@freebsd.org Fri Jul 3 20:22:25 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B5653588CA for ; Fri, 3 Jul 2020 20:22:25 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49z5ww1rd2z3SYk for ; Fri, 3 Jul 2020 20:22:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f65.google.com with SMTP id t18so14658156otq.5 for ; Fri, 03 Jul 2020 13:22:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rT88MxRqkdjmgZJB7dh9Ah3ZsgLrpKvmP2znfbPRyVg=; b=Dv4ou92t+O2s2MM3C+RXzugUxnLKe5ChnR6WsvTcVBjKmioRTp+0I2ItlhaT0X//qb LeMRGivZctFEkbxTOv0C8WkSzRjWScKrQby7J7b0HlyG02BZe0Pfu+INpBmeSkza8UjN eC9Odx7GGXDeaTEjopU2sPefC6FLSXX3uzVZNEOlWCc8lVyNh2SZEwMy4wtfIy41I5T1 /ebaV7vZSG9KsjgBz+0Ygi9oGYaz8fnY2Vt9QzIcj3gtHG14vF4EqpTaLIEEg1Srxv7w JlKr9vF/Uh7adQpP/gW1LhCTwg5oR58DtWmi2Iwx48/ZE0ja2g/itUMQ2FswIZddgiir GR4A== X-Gm-Message-State: AOAM531SctH82FVH7ajQFCVBssCvcI9ILmpp3HiwiRWERWDN50SCJ1UZ OqsTg6YPkHOfc0xmYW4TBwQleMOqQIRxyf8TtySyeZcW X-Google-Smtp-Source: ABdhPJx4n2F4LancOF9xXoahvM2oV4ATuV7640QiOhxdOoiqm6zLObyqAobqYT8O7CcQ64Zfhq0ybfhNX1a12JOE3vk= X-Received: by 2002:a05:6830:14c6:: with SMTP id t6mr19862133otq.18.1593807743032; Fri, 03 Jul 2020 13:22:23 -0700 (PDT) MIME-Version: 1.0 References: <48AB9191-5984-4BE0-BF98-7548424D06BF@dawidek.net> In-Reply-To: <48AB9191-5984-4BE0-BF98-7548424D06BF@dawidek.net> From: Alan Somers Date: Fri, 3 Jul 2020 14:22:11 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= Cc: freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 49z5ww1rd2z3SYk X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.65 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.06 / 15.00]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.91)[-0.911]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.65:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.20)[-0.197]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.65:from]; NEURAL_HAM_MEDIUM(-0.95)[-0.950]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jul 2020 20:22:25 -0000 I don't. What I meant was that a single thread (geom) is limiting the performance of the system overall. I'm certain, based on top, gstat, and zpool iostat, that geom is the limiting factor on this system. -Alan On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek wrote: > Hi Alan, > > why do you think it will hurt single-threaded performance? > > -- > Pawe=C5=82 Jakub Dawidek > > > > > On Jul 3, 2020, at 12:30, Alan Somers wrote: > > > > =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hu= ndreds of > > drives. What I'm seeing is that under at least some workloads, the > overall > > performance is limited by the single geom kernel process. procstat and > > kgdb aren't much help in telling exactly why this process is using so > much > > CPU, but it certainly must be related to the fact that over 15,000 IOPs > are > > going through that thread. What can I do to improve this situation? > Would > > it make sense to enable direct dispatch for geli? That would hurt > > single-threaded performance, but probably improve performance for highl= y > > multithreaded workloads like mine. > > > > Example top output: > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMA= ND > > 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > geom{g_down} > > 13 root -8 - 0B 96K - 9 35.5H 25.32% > > geom{g_up} > > > > -Alan > > _______________________________________________ > > freebsd-geom@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" > > From owner-freebsd-geom@freebsd.org Sat Jul 4 20:55:44 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F1DF6358212 for ; Sat, 4 Jul 2020 20:55:44 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 49zkcv45Pvz47X0; Sat, 4 Jul 2020 20:55:43 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from [IPv6:2607:fb90:9c84:f6f6:382a:457e:10d0:c0ea] (unknown [172.58.92.5]) by mail.dawidek.net (Postfix) with ESMTPSA id 84F24675; Sat, 4 Jul 2020 22:55:35 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: Single-threaded bottleneck in geli From: =?utf-8?Q?Pawe=C5=82_Jakub_Dawidek?= In-Reply-To: Date: Sat, 4 Jul 2020 13:55:30 -0700 Cc: freebsd-geom@freebsd.org Message-Id: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> References: To: Alan Somers X-Mailer: iPhone Mail (17F80) X-Rspamd-Queue-Id: 49zkcv45Pvz47X0 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of pawel@dawidek.net has no SPF policy when checking 91.121.88.72) smtp.mailfrom=pawel@dawidek.net X-Spamd-Result: default: False [3.25 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dawidek.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.37)[0.368]; NEURAL_SPAM_SHORT(0.22)[0.218]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.61)[0.611]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; RCVD_COUNT_TWO(0.00)[2]; R_MIXED_CHARSET(0.56)[subject]; RECEIVED_SPAMHAUS_PBL(0.00)[172.58.92.5:received] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 20:55:45 -0000 Direct dispatch would be great for geli, especially that geli can use own (m= ultiple) threads when necessary (eg. using crypto cards). With AES-NI you co= uld go straight to the disk. --=20 Pawe=C5=82 Jakub Dawidek > On Jul 3, 2020, at 13:22, Alan Somers wrote: >=20 > =EF=BB=BFI don't. What I meant was that a single thread (geom) is limitin= g the > performance of the system overall. I'm certain, based on top, gstat, and > zpool iostat, that geom is the limiting factor on this system. > -Alan >=20 >> On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek >> wrote: >>=20 >> Hi Alan, >>=20 >> why do you think it will hurt single-threaded performance? >>=20 >> -- >> Pawe=C5=82 Jakub Dawidek >>=20 >>=20 >>=20 >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: >>>=20 >>> =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with hun= dreds of >>> drives. What I'm seeing is that under at least some workloads, the >> overall >>> performance is limited by the single geom kernel process. procstat and >>> kgdb aren't much help in telling exactly why this process is using so >> much >>> CPU, but it certainly must be related to the fact that over 15,000 IOPs >> are >>> going through that thread. What can I do to improve this situation? >> Would >>> it make sense to enable direct dispatch for geli? That would hurt >>> single-threaded performance, but probably improve performance for highly= >>> multithreaded workloads like mine. >>>=20 >>> Example top output: >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND= >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% >>> geom{g_down} >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% >>> geom{g_up} >>>=20 >>> -Alan >>> _______________________________________________ >>> freebsd-geom@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom >>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" >>=20 >>=20 > _______________________________________________ > freebsd-geom@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" From owner-freebsd-geom@freebsd.org Sat Jul 4 21:59:29 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4FD55359818 for ; Sat, 4 Jul 2020 21:59:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49zm2S2CR6z4BVc for ; Sat, 4 Jul 2020 21:59:28 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f44.google.com with SMTP id k47so3282850ool.2 for ; Sat, 04 Jul 2020 14:59:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VYUFAf2y0gdOeoWRtLZch+yOXxisX//+9M4MMV6edDE=; b=hn221SQYy96DL5I7cCpsORiPQIbNhJowUkHTjHQjIkbfzcvvJLShKv+d5M9HEhxs3p fWCYCnbXtt6hqzchGqfysZN+y0Svlg1k/VWLqKIeH271mjWZKlgmTUny/9/ZvBO8368X JfngJWNukZb4QQTQa6jX0K56//TYeJFFLNudx5H1do3fHwX29r2HYp+b6iUvqR18GA9v o0+Iq8joUJ99f9XE7I/rJ12kOXTDi7G1TLfZ8dSUAusjn7UjLyAkJjCLmmc0w/I9cZ0R EOC29EIkJ5N9zDzCQ6BSQ2E0SEyGlKf14gnhHD2x/0YxPKrUld5/SJHuLbYT19ucBoGg E/Ew== X-Gm-Message-State: AOAM533k82AAhWRLcD+9nYX6ziH8LvcdB78KHn+C27OHHgjIyLaB2kfB 3J3rFJgVha9f1TbayJpjbICL3NK+34uRFv9q6VJkL1vH X-Google-Smtp-Source: ABdhPJyq/Hl0dpkGKr0nDvkybUV2LW54h21zaKA/AlHIvrgwTy1GwoHIxQb1EwuHbMdRR7U8DtqEbNfOCaK7zt+HAsg= X-Received: by 2002:a4a:a603:: with SMTP id e3mr20331195oom.61.1593899966940; Sat, 04 Jul 2020 14:59:26 -0700 (PDT) MIME-Version: 1.0 References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> In-Reply-To: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> From: Alan Somers Date: Sat, 4 Jul 2020 15:59:16 -0600 Message-ID: Subject: Re: Single-threaded bottleneck in geli To: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= Cc: freebsd-geom@freebsd.org X-Rspamd-Queue-Id: 49zm2S2CR6z4BVc X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.161.44 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.927]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-0.96)[-0.965]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-geom@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[209.85.161.44:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.08)[-0.076]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.161.44:from]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 21:59:29 -0000 I might give this a shot. What is the best way tell if geli ought to use direct dispatch? Is there a generic "are crypto instructions available" macro that would cover aesni as well as other platform-specific instructions? -Alan On Sat, Jul 4, 2020, 2:55 PM Pawe=C5=82 Jakub Dawidek w= rote: > Direct dispatch would be great for geli, especially that geli can use own > (multiple) threads when necessary (eg. using crypto cards). With AES-NI y= ou > could go straight to the disk. > > -- > Pawe=C5=82 Jakub Dawidek > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > =EF=BB=BFI don't. What I meant was that a single thread (geom) is limi= ting the > > performance of the system overall. I'm certain, based on top, gstat, a= nd > > zpool iostat, that geom is the limiting factor on this system. > > -Alan > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe=C5=82 Jakub Dawidek > >> wrote: > >> > >> Hi Alan, > >> > >> why do you think it will hurt single-threaded performance? > >> > >> -- > >> Pawe=C5=82 Jakub Dawidek > >> > >> > >> > >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: > >>> > >>> =EF=BB=BFI'm using geli, gmultipath, and ZFS on a large system, with = hundreds > of > >>> drives. What I'm seeing is that under at least some workloads, the > >> overall > >>> performance is limited by the single geom kernel process. procstat a= nd > >>> kgdb aren't much help in telling exactly why this process is using so > >> much > >>> CPU, but it certainly must be related to the fact that over 15,000 IO= Ps > >> are > >>> going through that thread. What can I do to improve this situation? > >> Would > >>> it make sense to enable direct dispatch for geli? That would hurt > >>> single-threaded performance, but probably improve performance for > highly > >>> multithreaded workloads like mine. > >>> > >>> Example top output: > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > >>> geom{g_down} > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > >>> geom{g_up} > >>> > >>> -Alan > >>> _______________________________________________ > >>> freebsd-geom@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom > >>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.or= g > " > >> > >> > > _______________________________________________ > > freebsd-geom@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-geom > > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" > > From owner-freebsd-geom@freebsd.org Sat Jul 4 23:16:55 2020 Return-Path: Delivered-To: freebsd-geom@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5E1B535AE1E for ; Sat, 4 Jul 2020 23:16:55 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49znlp6wCHz4FVn; Sat, 4 Jul 2020 23:16:54 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 064NGgMI090018 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 4 Jul 2020 16:16:42 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 064NGgoi090017; Sat, 4 Jul 2020 16:16:42 -0700 (PDT) (envelope-from jmg) Date: Sat, 4 Jul 2020 16:16:42 -0700 From: John-Mark Gurney To: Alan Somers Cc: Pawe?? Jakub Dawidek , freebsd-geom@freebsd.org Subject: Re: Single-threaded bottleneck in geli Message-ID: <20200704231642.GU4213@funkthat.com> Mail-Followup-To: Alan Somers , Pawe?? Jakub Dawidek , freebsd-geom@freebsd.org References: <80B62FE6-FCFB-42B8-A34C-B28E7DDBF45D@dawidek.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sat, 04 Jul 2020 16:16:42 -0700 (PDT) X-Rspamd-Queue-Id: 49znlp6wCHz4FVn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US] X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jul 2020 23:16:55 -0000 Alan Somers wrote this message on Sat, Jul 04, 2020 at 15:59 -0600: > I might give this a shot. What is the best way tell if geli ought to use > direct dispatch? Is there a generic "are crypto instructions available" > macro that would cover aesni as well as other platform-specific > instructions? Direct dispatch has the advantage of saving scheduling context switches... Geli has two modes, one is hardware acceleration mode, where it does a bit of work to put together the request, and then hands it off to the crypto framework (say an accelerator card), and then there is the mode where it has to be done in software, where it dispatches to a set of worker threads... In both modes, it would make sense for GELI to be able to do the work to construct those requests via direct dispatch... This would eliminate a context switch, which is always a good thing... I haven't looked at the OpenCrypto code in a while, so I don't know what the locking requirements are... The key cache is already locked by an mtx, but I believe it's a leaf lock, and so shouldn't be an issue... I'll add this to my list of things to look at... Also, if you have that many geli devices, you might also want to set: kern.geom.eli.threads=1 As it stands, geli fires up ncpu threads for EACH geli device, so likely have thousands of geli threads... > On Sat, Jul 4, 2020, 2:55 PM Pawe?? Jakub Dawidek wrote: > > > Direct dispatch would be great for geli, especially that geli can use own > > (multiple) threads when necessary (eg. using crypto cards). With AES-NI you > > could go straight to the disk. > > > > > On Jul 3, 2020, at 13:22, Alan Somers wrote: > > > > > > ???I don't. What I meant was that a single thread (geom) is limiting the > > > performance of the system overall. I'm certain, based on top, gstat, and > > > zpool iostat, that geom is the limiting factor on this system. > > > -Alan > > > > > >> On Fri, Jul 3, 2020 at 2:18 PM Pawe?? Jakub Dawidek > > >> wrote: > > >> > > >> Hi Alan, > > >> > > >> why do you think it will hurt single-threaded performance? > > >> > > >>>> On Jul 3, 2020, at 12:30, Alan Somers wrote: > > >>> > > >>> ???I'm using geli, gmultipath, and ZFS on a large system, with hundreds > > of > > >>> drives. What I'm seeing is that under at least some workloads, the > > >> overall > > >>> performance is limited by the single geom kernel process. procstat and > > >>> kgdb aren't much help in telling exactly why this process is using so > > >> much > > >>> CPU, but it certainly must be related to the fact that over 15,000 IOPs > > >> are > > >>> going through that thread. What can I do to improve this situation? > > >> Would > > >>> it make sense to enable direct dispatch for geli? That would hurt > > >>> single-threaded performance, but probably improve performance for > > highly > > >>> multithreaded workloads like mine. > > >>> > > >>> Example top output: > > >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > > COMMAND > > >>> 13 root -8 - 0B 96K CPU46 46 82.7H 70.54% > > >>> geom{g_down} > > >>> 13 root -8 - 0B 96K - 9 35.5H 25.32% > > >>> geom{g_up} -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."