From nobody Sun Jan 8 22:00:02 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nqrbj0QWzz2sH5j for ; Sun, 8 Jan 2023 22:00:17 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.proton.ch (mail-4317.proton.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nqrbh4kJcz3RFD for ; Sun, 8 Jan 2023 22:00:15 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; none Date: Sun, 08 Jan 2023 22:00:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1673215212; x=1673474412; bh=5zE3BTpAbP8DUgHExHXk3PniC2WFdjg2oLVBZWJ8m+Q=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=w/97rdqHG6xkCKf9kYWwRrndcTP2NqzsQNG6aZRk/z2kvzdU3EWE7k9pPkuCQWJs9 9puiaUEQzPSDJ8T2ftGFwJ1fbvIIq5ie5if+mYwDiWuGOqQCIXuF7R2XO3jrszYyk0 0RYBfZ8kjmfA8ca9lDNfvgwgyGjNSMCw9U6JUsAkdyR8lDVaNkRA3SNOicatnh60pR 0g/KnsXFHTbiJDdKu8KW3ELeoHzOXuaUblRrd7mI7U5YT4sAEiTrkKky809p7w/Eoc edV5PLd62ammTS5y0fMfyu1Kx+FxFmaUgAipWJITFFIDSCVJ0euPzMt97HCk+tRk1d T2Kw7jLO3p4Sw== To: Jonathan Vasquez From: Jonathan Vasquez Cc: woozle@woozle.net, mirror176@hotmail.com, stable@FreeBSD.org Subject: Re: poudriere builds failing when a port is expired Message-ID: In-Reply-To: References: Feedback-ID: 12351801:user:proton List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_hUUmInQo7bWInVKbCHypVjFI4DE04uLZ3EFgjIqJ8" X-Rspamd-Queue-Id: 4Nqrbh4kJcz3RFD X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --b1_hUUmInQo7bWInVKbCHypVjFI4DE04uLZ3EFgjIqJ8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SSBmaW5pc2hlZCB1cGdyYWRpbmcgZnJvbSBQSFAgNy40IHRvIFBIUCA4LjIgYW5kIGV2ZXJ5dGhp bmcgd2VudCB3ZWxsLiBJIG5vdGljZWQgdGhhdCB0aGUgImpzb24iIGFuZCAib3BlbnNzbCIgcGhw IG1vZHVsZXMgYXJlIG5vdyBjb21waWxlZCBpbiBieSBkZWZhdWx0IGl0IHNlZW1zLCBhbmQgdGhh dCdzIHdoeSB0aGUgb3RoZXIgdHdvIHBvcnRzIG5vIGxvbmdlciBleGlzdCA7RC4KCkpvbmF0aGFu IFZhc3F1ZXoKUEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgRDQ5RiBGQjg1IDkwQjcgQzRD QSA1Mjc5ClNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbAoKLS0tLS0tLSBPcmlnaW5h bCBNZXNzYWdlIC0tLS0tLS0KT24gU3VuZGF5LCBKYW51YXJ5IDh0aCwgMjAyMyBhdCAwOToyMSwg Sm9uYXRoYW4gVmFzcXVleiA8am9uQHh5aW5uLm9yZz4gd3JvdGU6Cgo+IFRoYW5rcyBmb3IgdGhh dCBEbWl0cnkuIEl0IHdpbGwgcHJvbGx5IGJlIGJldHRlciBmb3IgbWUgdG8gc2NoZWR1bGUgc29t ZSB0aW1lIHRvIGRvIGFuIHVwZ3JhZGUgb2YgcGhwNzQgaGFoYS4KPgo+IEpvbmF0aGFuIFZhc3F1 ZXoKPiBQR1A6IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3QSBENDlGIEZCODUgOTBCNyBDNENBIDUy NzkKPiBTZW50IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWwKPgo+IFNlbnQgZnJvbSBQcm90 b24gTWFpbCBtb2JpbGUKPgo+IC0tLS0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0tLS0KPiBP biBKYW4gOCwgMjAyMywgMDc6NTEsIERtaXRyeSBNb3Jvem92c2t5IDwgd29vemxlQHdvb3psZS5u ZXQ+IHdyb3RlOgo+Cj4+IE9uIFNhdCwgNyBKYW4gMjAyMywgSm9uYXRoYW4gVmFzcXVleiB3cm90 ZTogPiBJIGZvcmdvdCB0byBtZW50aW9uLCBJIGFsc28gaGFkIHRoZXNlIHdhcm5pbmdzOiA+ID4g WzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogYXJjaGl2ZXJzL3BocDc0LXpsaWIgcmVuYW1lZCB0 byBhcmNoaXZlcnMvcGhwODAtemxpYiA+IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGNvbnZl cnRlcnMvcGhwNzQtaWNvbnYgcmVuYW1lZCB0byBjb252ZXJ0ZXJzL3BocDgwLWljb252ID4gWzAw OjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogY29udmVydGVycy9waHA3NC1tYnN0cmluZyByZW5hbWVk IHRvIGNvbnZlcnRlcnMvcGhwODAtbWJzdHJpbmcgPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVE OiBkYXRhYmFzZXMvcGhwNzQtcGdzcWwgcmVuYW1lZCB0byBkYXRhYmFzZXMvcGhwODAtcGdzcWwg PiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBkZXZlbC9waHA3NC10b2tlbml6ZXIgcmVuYW1l ZCB0byBkZXZlbC9waHA4MC10b2tlbml6ZXIgPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBn cmFwaGljcy9waHA3NC1nZCByZW5hbWVkIHRvIGdyYXBoaWNzL3BocDgwLWdkID4gWzAwOjAwOjAz XSBXYXJuaW5nOiBNT1ZFRDogbGFuZy9waHA3NCByZW5hbWVkIHRvIGxhbmcvcGhwODAgPiBbMDA6 MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBzZWN1cml0eS9waHA3NC1maWx0ZXIgcmVuYW1lZCB0byBz ZWN1cml0eS9waHA4MC1maWx0ZXIgPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiB0ZXh0cHJv Yy9waHA3NC1kb20gcmVuYW1lZCB0byB0ZXh0cHJvYy9waHA4MC1kb20gPiBbMDA6MDA6MDNdIFdh cm5pbmc6IE1PVkVEOiB3d3cvbW9kX3BocDc0IHJlbmFtZWQgdG8gd3d3L21vZF9waHA4MCA+ID4g PiBJJ20gZ3Vlc3NpbmcgRnJlZUJTRCBidW1wZWQgdXAgdGhlIGRlZmF1bHQgdmVyc2lvbiB0byA4 LjAgb3IgaXMgZ29pbmcgdG8gYmUgPiBkZXByZWNhdGluZyA3LjQuIEVpdGhlciB3YXksIGl0IG1h eSBiZSB0aW1lIGZvciBtZSB0byB1cGdyYWRlIHRoZSBzZXJ2ZXIgPiBzb29uZXIgcmF0aGVyIHRo YW4gbGF0ZXIsIGhhaGEuIChyZXNlbmRpbmcgZnJvbSBjb3JyZWN0IG1haWwgYWRkcmVzcykgcGhw NzQgaGFkIGJlZW4gZGVwcmVjYXRlZCBzb21lIHRpbWUgYWdvLCB5ZXMgb24gdGhlIG90aGVyIGhh bmQsIG5vYm9keSBzdG9wcyB5b3UgdG8gaGF2ZSBkaXN0aW5jdCBwb3J0cyB0cmVlIGJhc2VkIG9u IGEgdGFnIChoaW50OiBwb3VkcmllcmUgcG9ydHMgLWMgLXAgMTFlb2wgLUIgMTEtZW9sKSBhbmQv b3IgbWFrZSBsb2NhbCBjaGFuZ2VzIHRvIHRoZSBwb3J0cyB0cmVlIHRvIGhhdmUgdGhpbmdzIGJ1 aWxkYWJsZSA+IC0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tID4gT24gRnJpZGF5LCBK YW51YXJ5IDZ0aCwgMjAyMyBhdCAyMzoyNCwgRWR3YXJkIFNhbmZvcmQgU3V0dG9uLCBJSUkgd3Jv dGU6ID4gPiA+ID4gT24gMS82LzIzIDIxOjE3LCBKb25hdGhhbiBWYXNxdWV6IHdyb3RlOiA+ID4g PiA+ID4gQW55b25lIGVsc2UgZ2V0dGluZyBidWlsZCBmYWlsdXJlcyB1c2luZyBwb3VkcmllcmUg ZHVlIHRvIHNvbWUgcG9ydHMgZXhwaXJpbmc/IEkgcmUtY3JlYXRlZCBwb3VkcmllcmUgamFpbCBh bmQgZGVmYXVsdCBwb3J0cyB0cmVlIHNpbmNlIEkgb3JpZ2luYWxseSB0aG91Z2h0IHNvbWV0aGlu ZyBnb3QgY29ycnVwdGVkIG9uIG15IG1hY2hpbmUuIEJ1dCBpdCBzZWVtcyB0aGF0IGV4cGlyaW5n IHBvcnRzIGRvZXMgYSBoYXJkIGZhaWx1cmUuIEkgY2FuIGFkbWlyZSB0aGUgaWRlYSBvZiBoYXZp bmcgYSBoYXJkIGZhaWx1cmUgbGlrZSB0aGF0IGluIHRoZXNlIGNhc2VzLCBidXQgbm90IHN1cmUg aWYgdGhhdCdzIHRoZSBiZXN0IHNvbHV0aW9uLiA+ID4gPiA+ID4gPiByb290QG9jdG9wdXM6fiAj IGJ1aWxkX3BhY2thZ2VzID4gPiA+IFswMDowMDowMF0gQ3JlYXRpbmcgdGhlIHJlZmVyZW5jZSBq YWlsLi4uIGRvbmUgPiA+ID4gWzAwOjAwOjAxXSBNb3VudGluZyBzeXN0ZW0gZGV2aWNlcyBmb3Ig MTMxeDY0LWRlZmF1bHQgPiA+ID4gWzAwOjAwOjAxXSBNb3VudGluZyBwb3J0cy9wYWNrYWdlcy9k aXN0ZmlsZXMgPiA+ID4gWzAwOjAwOjAxXSBVc2luZyBwYWNrYWdlcyBmcm9tIHByZXZpb3VzbHkg ZmFpbGVkIGJ1aWxkOiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhL3BhY2thZ2VzLzEzMXg2NC1k ZWZhdWx0Ly5idWlsZGluZyA+ID4gPiBbMDA6MDA6MDFdIE1vdW50aW5nIGNjYWNoZSBmcm9tOiAv dXNyL2xvY2FsL3BvdWRyaWVyZS9jY2FjaGUgPiA+ID4gWzAwOjAwOjAxXSBNb3VudGluZyBwYWNr YWdlcyBmcm9tOiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhL3BhY2thZ2VzLzEzMXg2NC1kZWZh dWx0ID4gPiA+IFswMDowMDowMV0gQ29weWluZyAvdmFyL2RiL3BvcnRzIGZyb206IC91c3IvbG9j YWwvZXRjL3BvdWRyaWVyZS5kL29wdGlvbnMgPiA+ID4gWzAwOjAwOjAxXSBBcHBlbmRpbmcgdG8g bWFrZS5jb25mOiAvdXNyL2xvY2FsL2V0Yy9wb3VkcmllcmUuZC9tYWtlLmNvbmYgPiA+ID4gL2V0 Yy9yZXNvbHYuY29uZiAtPiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhLy5tLzEzMXg2NC1kZWZh dWx0L3JlZi9ldGMvcmVzb2x2LmNvbmYgPiA+ID4gWzAwOjAwOjAxXSBTdGFydGluZyBqYWlsIDEz MXg2NC1kZWZhdWx0ID4gPiA+IFswMDowMDowMV0gV2lsbCBidWlsZCBhcyBub2JvZHk6ICg2NTUz NDo2NTUzNCkgPiA+ID4gWzAwOjAwOjAyXSBMb2dzOiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRh L2xvZ3MvYnVsay8xMzF4NjQtZGVmYXVsdC8yMDIzLTAxLTA2XzIzaDEwbTE0cyA+ID4gPiBbMDA6 MDA6MDJdIExvYWRpbmcgTU9WRUQgZm9yIC91c3IvbG9jYWwvcG91ZHJpZXJlL2RhdGEvLm0vMTMx eDY0LWRlZmF1bHQvcmVmL3Vzci9wb3J0cyA+ID4gPiBbMDA6MDA6MDNdIFBvcnRzIHN1cHBvcnRz OiBGTEFWT1JTIFNFTEVDVEVEX09QVElPTlMgPiA+ID4gWzAwOjAwOjAzXSBHYXRoZXJpbmcgcG9y dHMgbWV0YWRhdGEgPiA+ID4gWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogYXJjaGl2ZXJzL3Bo cDc0LXpsaWIgcmVuYW1lZCB0byBhcmNoaXZlcnMvcGhwODAtemxpYiA+ID4gPiBbMDA6MDA6MDNd IFdhcm5pbmc6IE1PVkVEOiBjb252ZXJ0ZXJzL3BocDc0LWljb252IHJlbmFtZWQgdG8gY29udmVy dGVycy9waHA4MC1pY29udiA+ID4gPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBjb252ZXJ0 ZXJzL3BocDc0LW1ic3RyaW5nIHJlbmFtZWQgdG8gY29udmVydGVycy9waHA4MC1tYnN0cmluZyA+ ID4gPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBkYXRhYmFzZXMvcGhwNzQtcGdzcWwgcmVu YW1lZCB0byBkYXRhYmFzZXMvcGhwODAtcGdzcWwgPiA+ID4gWzAwOjAwOjAzXSBFcnJvcjogTU9W RUQ6IGRldmVsL3BocDc0LWpzb24gMjAyMi0xMi0yNSBIYXMgZXhwaXJlZDogU2VjdXJpdHkgc3Vw cG9ydCBlbmRlZCBvbiAyMDIyLTExLTIyID4gPiA+IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6 IGRldmVsL3BocDc0LXRva2VuaXplciByZW5hbWVkIHRvIGRldmVsL3BocDgwLXRva2VuaXplciA+ ID4gPiBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBncmFwaGljcy9waHA3NC1nZCByZW5hbWVk IHRvIGdyYXBoaWNzL3BocDgwLWdkID4gPiA+IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGxh bmcvcGhwNzQgcmVuYW1lZCB0byBsYW5nL3BocDgwID4gPiA+IFswMDowMDowM10gV2FybmluZzog TU9WRUQ6IHNlY3VyaXR5L3BocDc0LWZpbHRlciByZW5hbWVkIHRvIHNlY3VyaXR5L3BocDgwLWZp bHRlciA+ID4gPiBbMDA6MDA6MDNdIEVycm9yOiBNT1ZFRDogc2VjdXJpdHkvcGhwNzQtb3BlbnNz bCAyMDIyLTEyLTI1IEhhcyBleHBpcmVkOiBTZWN1cml0eSBzdXBwb3J0IGVuZGVkIG9uIDIwMjIt MTEtMjIgPiA+ID4gWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogdGV4dHByb2MvcGhwNzQtZG9t IHJlbmFtZWQgdG8gdGV4dHByb2MvcGhwODAtZG9tID4gPiA+IFswMDowMDowM10gV2FybmluZzog TU9WRUQ6IHd3dy9tb2RfcGhwNzQgcmVuYW1lZCB0byB3d3cvbW9kX3BocDgwID4gPiA+IFswMDow MDowM10gRXJyb3I6IEZhdGFsIGVycm9ycyBlbmNvdW50ZXJlZCBnYXRoZXJpbmcgaW5pdGlhbCBw b3J0cyBtZXRhZGF0YSA+ID4gPiA+ID4gPiBKb25hdGhhbiBWYXNxdWV6ID4gPiA+IFBHUDogMzRE QSA4NThDIDE0NDcgNTA5RSBDNzdBIEQ0OUYgRkI4NSA5MEI3IEM0Q0EgNTI3OSA+ID4gPiBTZW50 IHdpdGggUHJvdG9uTWFpbCBTZWN1cmUgRW1haWwgPiA+ID4gPiA+ID4gVGhhdCBpcyB0aGUgbm9y bWFsIHJlYWN0aW9uIGluIFBvdWRyaWVyZSBhdCBwcmVzZW50LiBJIGFncmVlIHdpdGggeW91ID4g PiBhcyBJIHN0YXRlZCBvbiB0aGUgZ2l0aHViIGlzc3VlIGJ1dCBkbyBub3Qgc2VlIGl0IGFzIGEg YmlnIGRlYWwgYnV0IGl0ID4gPiB3b3VsZCBiZSBuaWNlciBpZiB0aGUgY2hlY2sgY291bGQgYmUg cGVyZm9ybWVkIGZhc3RlcjsgTXkgc3lzdGVtID4gPiBub3JtYWxseSB0YWtlcyBtdWNoIGxvbmdl ciB0byByZWFjaCBhIGZhaWxlZCBzdGF0ZSBkdWUgdG8gcHJldmlvdXMgc3RlcHMgPiA+IHRha2lu ZyBtaW51dGVzLiA+ID4gaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvcG91ZHJpZXJlL2lzc3Vl cy8xMDI2ID4gPiBodHRwczovL2J1Z3MuZnJlZWJzZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dp P2lkPTI2NzgzMCA+ID4gLS0gU2luY2VyZWx5LCBELk1hcmNrIFtETTUwMjAsIE1DSy1SSVBFLCBE TTMtUklQTl0gWyBGcmVlQlNEIGNvbW1pdHRlcjogbWFyY2tARnJlZUJTRC5vcmcgXSAtLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0gKioqIERtaXRyeSBNb3Jvem92c2t5IC0tLSBELk1hcmNrIC0tLSBXaWxkIFdv b3psZSAtLS0gd29vemxlQHdvb3psZS5uZXQgKioqIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ== --b1_hUUmInQo7bWInVKbCHypVjFI4DE04uLZ3EFgjIqJ8 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsOyBmb250LXNpemU6IDE0cHg7Ij5JIGZpbmlz aGVkIHVwZ3JhZGluZyBmcm9tIFBIUCA3LjQgdG8gUEhQIDguMiBhbmQgZXZlcnl0aGluZyB3ZW50 IHdlbGwuIEkgbm90aWNlZCB0aGF0IHRoZSAianNvbiIgYW5kICJvcGVuc3NsIiBwaHAgbW9kdWxl cyBhcmUgbm93IGNvbXBpbGVkIGluIGJ5IGRlZmF1bHQgaXQgc2VlbXMsIGFuZCB0aGF0J3Mgd2h5 IHRoZSBvdGhlciB0d28gcG9ydHMgbm8gbG9uZ2VyIGV4aXN0IDtELjwvZGl2PjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OiBBcmlhbDsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2Pg0KPGRpdiBj bGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2siIHN0eWxlPSJmb250LWZhbWlseTogQXJp YWw7IGZvbnQtc2l6ZTogMTRweDsiPg0KICAgIDxkaXYgY2xhc3M9InByb3Rvbm1haWxfc2lnbmF0 dXJlX2Jsb2NrLXVzZXIiPg0KICAgICAgICA8ZGl2PkpvbmF0aGFuIFZhc3F1ZXo8YnI+PC9kaXY+ PGRpdj5QR1A6IDM0REEgODU4QyAxNDQ3IDUwOUUgQzc3QSAgRDQ5RiBGQjg1IDkwQjcgQzRDQSA1 Mjc5PGJyPjwvZGl2PjxkaXY+U2VudCB3aXRoIFByb3Rvbk1haWwgU2VjdXJlIEVtYWlsPGJyPjwv ZGl2PjxkaXY+PGJyPjwvZGl2Pg0KICAgIDwvZGl2Pg0KICAgIA0KICAgICAgICAgICAgPGRpdiBj bGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2stcHJvdG9uIHByb3Rvbm1haWxfc2lnbmF0 dXJlX2Jsb2NrLWVtcHR5Ij4NCiAgICAgICAgDQogICAgICAgICAgICA8L2Rpdj4NCjwvZGl2Pg0K PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsOyBmb250LXNpemU6IDE0cHg7Ij48YnI+PC9k aXY+PGRpdiBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSI+DQogICAgICAgIC0tLS0tLS0gT3JpZ2lu YWwgTWVzc2FnZSAtLS0tLS0tPGJyPg0KICAgICAgICBPbiBTdW5kYXksIEphbnVhcnkgOHRoLCAy MDIzIGF0IDA5OjIxLCBKb25hdGhhbiBWYXNxdWV6ICZsdDtqb25AeHlpbm4ub3JnJmd0OyB3cm90 ZTo8YnI+PGJyPg0KICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSIg dHlwZT0iY2l0ZSI+DQogICAgICAgICAgICBUaGFua3MgZm9yIHRoYXQgRG1pdHJ5LiBJdCB3aWxs IHByb2xseSBiZSBiZXR0ZXIgZm9yIG1lIHRvIHNjaGVkdWxlIHNvbWUgdGltZSB0byBkbyBhbiB1 cGdyYWRlIG9mIHBocDc0IGhhaGEuPGJyPjxicj48YnI+PGRpdj5Kb25hdGhhbiBWYXNxdWV6PGJy PjwvZGl2PjxkaXY+UEdQOiAzNERBIDg1OEMgMTQ0NyA1MDlFIEM3N0EgIEQ0OUYgRkI4NSA5MEI3 IEM0Q0EgNTI3OTxicj48L2Rpdj48ZGl2PlNlbnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFp bDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48YnI+PGJyPlNlbnQgZnJvbSBQcm90b24gTWFpbCBt b2JpbGU8YnI+PGJyPjxicj48YnI+LS0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLS0tLTxi cj5PbiBKYW4gOCwgMjAyMywgMDc6NTEsIERtaXRyeSBNb3Jvem92c2t5ICZsdDsgd29vemxlQHdv b3psZS5uZXQmZ3Q7IHdyb3RlOjxibG9ja3F1b3RlIGNsYXNzPSJwcm90b25tYWlsX3F1b3RlIj48 YnI+T24gU2F0LCA3IEphbiAyMDIzLCBKb25hdGhhbiBWYXNxdWV6IHdyb3RlOg0KDQomZ3Q7IEkg Zm9yZ290IHRvIG1lbnRpb24sIEkgYWxzbyBoYWQgdGhlc2Ugd2FybmluZ3M6DQomZ3Q7DQomZ3Q7 IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGFyY2hpdmVycy9waHA3NC16bGliIHJlbmFtZWQg dG8gYXJjaGl2ZXJzL3BocDgwLXpsaWINCiZndDsgWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDog Y29udmVydGVycy9waHA3NC1pY29udiByZW5hbWVkIHRvIGNvbnZlcnRlcnMvcGhwODAtaWNvbnYN CiZndDsgWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogY29udmVydGVycy9waHA3NC1tYnN0cmlu ZyByZW5hbWVkIHRvIGNvbnZlcnRlcnMvcGhwODAtbWJzdHJpbmcNCiZndDsgWzAwOjAwOjAzXSBX YXJuaW5nOiBNT1ZFRDogZGF0YWJhc2VzL3BocDc0LXBnc3FsIHJlbmFtZWQgdG8gZGF0YWJhc2Vz L3BocDgwLXBnc3FsDQomZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGRldmVsL3BocDc0 LXRva2VuaXplciByZW5hbWVkIHRvIGRldmVsL3BocDgwLXRva2VuaXplcg0KJmd0OyBbMDA6MDA6 MDNdIFdhcm5pbmc6IE1PVkVEOiBncmFwaGljcy9waHA3NC1nZCByZW5hbWVkIHRvIGdyYXBoaWNz L3BocDgwLWdkDQomZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGxhbmcvcGhwNzQgcmVu YW1lZCB0byBsYW5nL3BocDgwDQomZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IHNlY3Vy aXR5L3BocDc0LWZpbHRlciByZW5hbWVkIHRvIHNlY3VyaXR5L3BocDgwLWZpbHRlcg0KJmd0OyBb MDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiB0ZXh0cHJvYy9waHA3NC1kb20gcmVuYW1lZCB0byB0 ZXh0cHJvYy9waHA4MC1kb20NCiZndDsgWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogd3d3L21v ZF9waHA3NCByZW5hbWVkIHRvIHd3dy9tb2RfcGhwODANCiZndDsNCiZndDsNCiZndDsgSSdtIGd1 ZXNzaW5nIEZyZWVCU0QgYnVtcGVkIHVwIHRoZSBkZWZhdWx0IHZlcnNpb24gdG8gOC4wIG9yIGlz IGdvaW5nIHRvIGJlDQomZ3Q7IGRlcHJlY2F0aW5nIDcuNC4gRWl0aGVyIHdheSwgaXQgbWF5IGJl IHRpbWUgZm9yIG1lIHRvIHVwZ3JhZGUgdGhlIHNlcnZlcg0KJmd0OyBzb29uZXIgcmF0aGVyIHRo YW4gbGF0ZXIsIGhhaGEuDQoNCihyZXNlbmRpbmcgZnJvbSBjb3JyZWN0IG1haWwgYWRkcmVzcykN Cg0KcGhwNzQgaGFkIGJlZW4gZGVwcmVjYXRlZCBzb21lIHRpbWUgYWdvLCB5ZXMNCg0Kb24gdGhl IG90aGVyIGhhbmQsIG5vYm9keSBzdG9wcyB5b3UgdG8gaGF2ZSBkaXN0aW5jdCBwb3J0cyB0cmVl IGJhc2VkIG9uIGEgdGFnDQooaGludDogcG91ZHJpZXJlIHBvcnRzIC1jIC1wIDExZW9sIC1CIDEx LWVvbCkNCg0KYW5kL29yIG1ha2UgbG9jYWwgY2hhbmdlcyB0byB0aGUgcG9ydHMgdHJlZSB0byBo YXZlIHRoaW5ncyBidWlsZGFibGUNCg0KDQomZ3Q7IC0tLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAt LS0tLS0tDQomZ3Q7IE9uIEZyaWRheSwgSmFudWFyeSA2dGgsIDIwMjMgYXQgMjM6MjQsIEVkd2Fy ZCBTYW5mb3JkIFN1dHRvbiwgSUlJICB3cm90ZToNCiZndDsNCiZndDsNCiZndDsgJmd0OyBPbiAx LzYvMjMgMjE6MTcsIEpvbmF0aGFuIFZhc3F1ZXogd3JvdGU6DQomZ3Q7ICZndDsNCiZndDsgJmd0 OyAmZ3Q7IEFueW9uZSBlbHNlIGdldHRpbmcgYnVpbGQgZmFpbHVyZXMgdXNpbmcgcG91ZHJpZXJl IGR1ZSB0byBzb21lIHBvcnRzIGV4cGlyaW5nPyBJIHJlLWNyZWF0ZWQgcG91ZHJpZXJlIGphaWwg YW5kIGRlZmF1bHQgcG9ydHMgdHJlZSBzaW5jZSBJIG9yaWdpbmFsbHkgdGhvdWdodCBzb21ldGhp bmcgZ290IGNvcnJ1cHRlZCBvbiBteSBtYWNoaW5lLiBCdXQgaXQgc2VlbXMgdGhhdCBleHBpcmlu ZyBwb3J0cyBkb2VzIGEgaGFyZCBmYWlsdXJlLiBJIGNhbiBhZG1pcmUgdGhlIGlkZWEgb2YgaGF2 aW5nIGEgaGFyZCBmYWlsdXJlIGxpa2UgdGhhdCBpbiB0aGVzZSBjYXNlcywgYnV0IG5vdCBzdXJl IGlmIHRoYXQncyB0aGUgYmVzdCBzb2x1dGlvbi4NCiZndDsgJmd0OyAmZ3Q7DQomZ3Q7ICZndDsg Jmd0OyByb290QG9jdG9wdXM6fiAjIGJ1aWxkX3BhY2thZ2VzDQomZ3Q7ICZndDsgJmd0OyBbMDA6 MDA6MDBdIENyZWF0aW5nIHRoZSByZWZlcmVuY2UgamFpbC4uLiBkb25lDQomZ3Q7ICZndDsgJmd0 OyBbMDA6MDA6MDFdIE1vdW50aW5nIHN5c3RlbSBkZXZpY2VzIGZvciAxMzF4NjQtZGVmYXVsdA0K Jmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAxXSBNb3VudGluZyBwb3J0cy9wYWNrYWdlcy9kaXN0Zmls ZXMNCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowMV0gVXNpbmcgcGFja2FnZXMgZnJvbSBwcmV2aW91 c2x5IGZhaWxlZCBidWlsZDogL3Vzci9sb2NhbC9wb3VkcmllcmUvZGF0YS9wYWNrYWdlcy8xMzF4 NjQtZGVmYXVsdC8uYnVpbGRpbmcNCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowMV0gTW91bnRpbmcg Y2NhY2hlIGZyb206IC91c3IvbG9jYWwvcG91ZHJpZXJlL2NjYWNoZQ0KJmd0OyAmZ3Q7ICZndDsg WzAwOjAwOjAxXSBNb3VudGluZyBwYWNrYWdlcyBmcm9tOiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9k YXRhL3BhY2thZ2VzLzEzMXg2NC1kZWZhdWx0DQomZ3Q7ICZndDsgJmd0OyBbMDA6MDA6MDFdIENv cHlpbmcgL3Zhci9kYi9wb3J0cyBmcm9tOiAvdXNyL2xvY2FsL2V0Yy9wb3VkcmllcmUuZC9vcHRp b25zDQomZ3Q7ICZndDsgJmd0OyBbMDA6MDA6MDFdIEFwcGVuZGluZyB0byBtYWtlLmNvbmY6IC91 c3IvbG9jYWwvZXRjL3BvdWRyaWVyZS5kL21ha2UuY29uZg0KJmd0OyAmZ3Q7ICZndDsgL2V0Yy9y ZXNvbHYuY29uZiAtJmd0OyAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhLy5tLzEzMXg2NC1kZWZh dWx0L3JlZi9ldGMvcmVzb2x2LmNvbmYNCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowMV0gU3RhcnRp bmcgamFpbCAxMzF4NjQtZGVmYXVsdA0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAxXSBXaWxsIGJ1 aWxkIGFzIG5vYm9keTogKDY1NTM0OjY1NTM0KQ0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAyXSBM b2dzOiAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhL2xvZ3MvYnVsay8xMzF4NjQtZGVmYXVsdC8y MDIzLTAxLTA2XzIzaDEwbTE0cw0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAyXSBMb2FkaW5nIE1P VkVEIGZvciAvdXNyL2xvY2FsL3BvdWRyaWVyZS9kYXRhLy5tLzEzMXg2NC1kZWZhdWx0L3JlZi91 c3IvcG9ydHMNCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowM10gUG9ydHMgc3VwcG9ydHM6IEZMQVZP UlMgU0VMRUNURURfT1BUSU9OUw0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAzXSBHYXRoZXJpbmcg cG9ydHMgbWV0YWRhdGENCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6 IGFyY2hpdmVycy9waHA3NC16bGliIHJlbmFtZWQgdG8gYXJjaGl2ZXJzL3BocDgwLXpsaWINCiZn dDsgJmd0OyAmZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IGNvbnZlcnRlcnMvcGhwNzQt aWNvbnYgcmVuYW1lZCB0byBjb252ZXJ0ZXJzL3BocDgwLWljb252DQomZ3Q7ICZndDsgJmd0OyBb MDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiBjb252ZXJ0ZXJzL3BocDc0LW1ic3RyaW5nIHJlbmFt ZWQgdG8gY29udmVydGVycy9waHA4MC1tYnN0cmluZw0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAz XSBXYXJuaW5nOiBNT1ZFRDogZGF0YWJhc2VzL3BocDc0LXBnc3FsIHJlbmFtZWQgdG8gZGF0YWJh c2VzL3BocDgwLXBnc3FsDQomZ3Q7ICZndDsgJmd0OyBbMDA6MDA6MDNdIEVycm9yOiBNT1ZFRDog ZGV2ZWwvcGhwNzQtanNvbiAyMDIyLTEyLTI1IEhhcyBleHBpcmVkOiBTZWN1cml0eSBzdXBwb3J0 IGVuZGVkIG9uIDIwMjItMTEtMjINCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowM10gV2FybmluZzog TU9WRUQ6IGRldmVsL3BocDc0LXRva2VuaXplciByZW5hbWVkIHRvIGRldmVsL3BocDgwLXRva2Vu aXplcg0KJmd0OyAmZ3Q7ICZndDsgWzAwOjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogZ3JhcGhpY3Mv cGhwNzQtZ2QgcmVuYW1lZCB0byBncmFwaGljcy9waHA4MC1nZA0KJmd0OyAmZ3Q7ICZndDsgWzAw OjAwOjAzXSBXYXJuaW5nOiBNT1ZFRDogbGFuZy9waHA3NCByZW5hbWVkIHRvIGxhbmcvcGhwODAN CiZndDsgJmd0OyAmZ3Q7IFswMDowMDowM10gV2FybmluZzogTU9WRUQ6IHNlY3VyaXR5L3BocDc0 LWZpbHRlciByZW5hbWVkIHRvIHNlY3VyaXR5L3BocDgwLWZpbHRlcg0KJmd0OyAmZ3Q7ICZndDsg WzAwOjAwOjAzXSBFcnJvcjogTU9WRUQ6IHNlY3VyaXR5L3BocDc0LW9wZW5zc2wgMjAyMi0xMi0y NSBIYXMgZXhwaXJlZDogU2VjdXJpdHkgc3VwcG9ydCBlbmRlZCBvbiAyMDIyLTExLTIyDQomZ3Q7 ICZndDsgJmd0OyBbMDA6MDA6MDNdIFdhcm5pbmc6IE1PVkVEOiB0ZXh0cHJvYy9waHA3NC1kb20g cmVuYW1lZCB0byB0ZXh0cHJvYy9waHA4MC1kb20NCiZndDsgJmd0OyAmZ3Q7IFswMDowMDowM10g V2FybmluZzogTU9WRUQ6IHd3dy9tb2RfcGhwNzQgcmVuYW1lZCB0byB3d3cvbW9kX3BocDgwDQom Z3Q7ICZndDsgJmd0OyBbMDA6MDA6MDNdIEVycm9yOiBGYXRhbCBlcnJvcnMgZW5jb3VudGVyZWQg Z2F0aGVyaW5nIGluaXRpYWwgcG9ydHMgbWV0YWRhdGENCiZndDsgJmd0OyAmZ3Q7DQomZ3Q7ICZn dDsgJmd0OyBKb25hdGhhbiBWYXNxdWV6DQomZ3Q7ICZndDsgJmd0OyBQR1A6IDM0REEgODU4QyAx NDQ3IDUwOUUgQzc3QSBENDlGIEZCODUgOTBCNyBDNENBIDUyNzkNCiZndDsgJmd0OyAmZ3Q7IFNl bnQgd2l0aCBQcm90b25NYWlsIFNlY3VyZSBFbWFpbA0KJmd0OyAmZ3Q7DQomZ3Q7ICZndDsNCiZn dDsgJmd0OyBUaGF0IGlzIHRoZSBub3JtYWwgcmVhY3Rpb24gaW4gUG91ZHJpZXJlIGF0IHByZXNl bnQuIEkgYWdyZWUgd2l0aCB5b3UNCiZndDsgJmd0OyBhcyBJIHN0YXRlZCBvbiB0aGUgZ2l0aHVi IGlzc3VlIGJ1dCBkbyBub3Qgc2VlIGl0IGFzIGEgYmlnIGRlYWwgYnV0IGl0DQomZ3Q7ICZndDsg d291bGQgYmUgbmljZXIgaWYgdGhlIGNoZWNrIGNvdWxkIGJlIHBlcmZvcm1lZCBmYXN0ZXI7IE15 IHN5c3RlbQ0KJmd0OyAmZ3Q7IG5vcm1hbGx5IHRha2VzIG11Y2ggbG9uZ2VyIHRvIHJlYWNoIGEg ZmFpbGVkIHN0YXRlIGR1ZSB0byBwcmV2aW91cyBzdGVwcw0KJmd0OyAmZ3Q7IHRha2luZyBtaW51 dGVzLg0KJmd0OyAmZ3Q7IGh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL3BvdWRyaWVyZS9pc3N1 ZXMvMTAyNg0KJmd0OyAmZ3Q7IGh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy9idWd6aWxsYS9zaG93 X2J1Zy5jZ2k/aWQ9MjY3ODMwDQomZ3Q7DQomZ3Q7DQoNCi0tDQpTaW5jZXJlbHksDQpELk1hcmNr ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtETTUwMjAsIE1DSy1SSVBF LCBETTMtUklQTl0NClsgRnJlZUJTRCBjb21taXR0ZXI6ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgbWFyY2tARnJlZUJTRC5vcmcgXQ0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoqKiog RG1pdHJ5IE1vcm96b3Zza3kgLS0tIEQuTWFyY2sgLS0tIFdpbGQgV29vemxlIC0tLSB3b296bGVA d29vemxlLm5ldCAqKioNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQo8L2Jsb2NrcXVvdGU+DQogICAg ICAgIDwvYmxvY2txdW90ZT48YnI+DQogICAgPC9kaXY+ --b1_hUUmInQo7bWInVKbCHypVjFI4DE04uLZ3EFgjIqJ8-- From nobody Fri Jan 20 04:11:38 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NymT05bQ0z311rv for ; Fri, 20 Jan 2023 04:18:28 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NymSz0YmGz43pY for ; Fri, 20 Jan 2023 04:18:26 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 30K4I5JJ078955 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Fri, 20 Jan 2023 05:18:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 30K4I55m078954 for freebsd-stable@freebsd.org; Fri, 20 Jan 2023 05:18:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30K4DrWQ046392 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Fri, 20 Jan 2023 05:13:53 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30K4BcDf032472 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Fri, 20 Jan 2023 05:11:39 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.16.1/8.16.1/Submit) id 30K4BcWo032471 for freebsd-stable@freebsd.org; Fri, 20 Jan 2023 05:11:38 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Fri, 20 Jan 2023 05:11:38 +0100 From: Peter To: freebsd-stable@freebsd.org Subject: close() taking minutes Message-ID: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Fri, 20 Jan 2023 05:18:08 +0100 (CET) X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[sub.org]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; HAS_XAW(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_FIVE(0.00)[5]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4NymSz0YmGz43pY X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Does this look familiar with anybody? ("truss -dD ls -l something") 0.047760369 0.000017548 open("/etc/nsswitch.conf",O_RDONLY|O_CLOEXEC,0666) = 4 (0x4) 0.047845099 0.000011290 fstat(4,{ mode=-rw-r--r-- ,inode=13886,size=272,blksize=4096 }) = 0 (0x0) 0.047932166 0.000034182 read(4,"#\n# nsswitch.conf(5) - name ser"...,4096) = 272 (0x110) 0.048007308 0.000010661 read(4,0x801848000,4096) = 0 (0x0) 45.827173802 45.779056767 close(4) = 0 (0x0) ^^^^^^^^^^^^ ... 45.866272643 0.000020608 open("/etc/localtime",O_RDONLY,077) = 6 (0x6) 45.866398048 0.000017731 fstat(6,{ mode=-r--r--r-- ,inode=41572,size=2309,blksize=4096 }) = 0 (0x0) 45.866505596 0.000054084 read(6,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) = 2309 (0x905) 49.511709215 3.645026276 close(6) = 0 (0x0) ^^^^^^^^^^^ (The longest I've seen was more than 5 minutes) There is otherwise nothing wrong with the machine, only I am running a postgres table merge for some 10 mio. rows. And only during that the effect happens, reproducibly. The concerned disk is a mirror of two consumer SATA-SSD. "gstat -pod" doesn't look bad, 1-2 items in the queue. zfs says it knows nothing of these delays, it just shows two consumer SSD usually busy with their own garbage collection: # zpool iostat -vlq im 3 capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim syncq_read syncq_write asyncq_read asyncq_write scrubq_read trimq_write pool alloc free read write read write read write read write read write read write wait wait pend activ pend activ pend activ pend activ pend activ pend activ ---------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- im 30.6G 14.9G 0 4.84K 0 111M - 2ms - 670us - - - 2ms - - 0 0 0 0 0 0 47 2 0 0 0 0 mirror-0 30.6G 14.9G 0 4.84K 0 111M - 2ms - 670us - - - 2ms - - 0 0 0 0 0 0 47 2 0 0 0 0 ada5p9.elip1 - - 0 1.65K 0 55.7M - 3ms - 927us - - - 3ms - - 0 0 0 0 0 0 0 0 0 0 0 0 ada1p9.elip1 - - 0 3.19K 0 55.8M - 2ms - 536us - - - 1ms - - 0 0 0 0 0 0 47 2 0 0 0 0 ---------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim syncq_read syncq_write asyncq_read asyncq_write scrubq_read trimq_write pool alloc free read write read write read write read write read write read write wait wait pend activ pend activ pend activ pend activ pend activ pend activ ---------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- im 30.6G 14.9G 0 5.69K 0 103M - 3ms - 544us - - - 2ms - - 0 0 0 0 0 0 84 2 0 0 0 0 mirror-0 30.6G 14.9G 0 5.69K 0 103M - 3ms - 544us - - - 2ms - - 0 0 0 0 0 0 98 4 0 0 0 0 ada5p9.elip1 - - 0 2.05K 0 51.4M - 5ms - 837us - - - 4ms - - 0 0 0 0 0 0 93 2 0 0 0 0 ada1p9.elip1 - - 0 3.64K 0 51.6M - 1ms - 379us - - - 1ms - - 0 0 0 0 0 0 0 0 0 0 0 0 ---------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- So where does it happen? (13.1-RELEASE-p5) The root, /usr and /var are on ufs, on the same(!) disk (unmirrored), and have no problem. The next 100-some filesystems are on the "im" pool, and there is the problem, so it is somehow related to ZFS. (The above truss output was in a jail, which is on the ZFS.) I had other things saturating these disks, with no such effect. It seems postgres does something different here. It does also not really look like a congestion problem, because I can have two or three tasks get stuck in this (and all of them will), and they continue at very much different times. Looks rather like some lock wait thing. Any clues, anybody? From nobody Fri Jan 20 10:18:25 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NywSR0D4mz2t7w2 for ; Fri, 20 Jan 2023 10:18:31 +0000 (UTC) (envelope-from peter@peter-dambier.de) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NywSP6Rglz3DTp for ; Fri, 20 Jan 2023 10:18:29 +0000 (UTC) (envelope-from peter@peter-dambier.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of peter@peter-dambier.de has no SPF policy when checking 212.227.15.18) smtp.mailfrom=peter@peter-dambier.de; dmarc=none Received: from cesium.dl2fba ([217.113.183.83]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M7sDg-1pMg5Z2VAw-00550f for ; Fri, 20 Jan 2023 11:18:27 +0100 Date: Fri, 20 Jan 2023 11:18:25 +0100 From: Peter Dambier To: stable@freebsd.org Subject: Re: close() taking minutes Message-ID: <20230120111825.46d0e337.peter@peter-dambier.de> In-Reply-To: References: Reply-To: peter@peter-dambier.de X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.31; i586-slackware-linux-gnu) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:6OBTQCqZa6h1hDOux5ml88NZWYiqLgPCjd6PaGas4M2lXH+F81W RCDEp2Kk+A/cLWR/spmLbBFTenDJwz23WmzF/JbQI8r2zkHTj0S98pGO2zgMH7AybwtwKAg BcxR5fNrSEo7wIzMPNwPRQuLjhtjHezPBF0lL9keThT5SCc+I48cWxWyHuH87XM+9aM3mcP bGJLMBQpzlvc7SUOvkNXA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:ukbfMVnG7ZI=;jSG6YnzOpMcf4drjgpdIepZpSNf Apj6on/BFw/jM1fz1Jo6zQ40zI+16tzBdxTc30UiongTAStcOl/vTEYPCnT83gjuPtWxHAqyf gR8mrAovp6wwVcRtvtO4ogpGdK3AwVKODF1bhNTJerWhdbb3OggcDHKNJE1ZpdQqtrpW/vfEV S3zdvPlD716snrtw+c9YsEFXUF0rADwAH5g+tm2dLLSHNoGGLItuVYCjj7tkULKise8Qw1Z0L j/Inckb1TwM14dDdE/wBpz8H7p8NmmoN53dHMg+Pnfz0/au7CnFyejVjQ2Pgjv1XfGImDKnn2 N0Aq0lbx6/1UT36BxUAWXEWg+IhZGWLABDbn5liIOfVg2Gqpe4ndUxpgrvJtxJVD8tKbIVbUK IuVAvtafiA8HQsuCrAw4tSCKJP6OyTYrZK4hfMeIla6saAHgqp0gpMgmdhGfPaKXGE5wkntSM 1K72NhpBMEK2Zf2LVjo4WdK9oouEV/07PgfuHNWgJLsNnK5BNT8M83YYPxISnQyu376PHPAkt 8xsak9ysbWA9uXlhakNEqLbrUhpfPwaHHwJkgmwbawBPQjPyh9BdsxN/tDKW9q9Xs/3QbQfvA gVRzmm4kV9JQhU8Zq6N3/SVu2DYSYqVSgc5cKYlp3+Sm0cAIAjM3vFKzh47ONjsGI3zoTUdp6 4Kensm2jFDt+7JAnC8ed/E4w+UnibsfFn6tJSP061qCAr0Nju+3xe+VQ8EUjUEQxGrxo6mIW0 EG+ZmedQMQqxkTENavdb5MDOlvVVzCny1vmH80pSg+bjiAksqGkLqiW9fpeT59xMNZB0I35BY S3FSDdipdLIhSS4DLQ7QNFp/tlaNRUf66WpinRW/aCCJMDDYTnSektkTZ13UbKktOiNbaCTN3 bN9eeQ+KbmqXP4pU83f7YwUwIdy4QiChpfttuDse63AWtBqaH90b01mw9coNG6TvNa7bKtfx9 JATYOQ== X-Spamd-Result: default: False [-1.20 / 15.00]; AUTH_NA(1.00)[]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.15.18:from]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; HAS_REPLYTO(0.00)[peter@peter-dambier.de]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DMARC_NA(0.00)[peter-dambier.de]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.15.18:from]; R_SPF_NA(0.00)[no SPF record]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[peter]; TO_DN_NONE(0.00)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4NywSP6Rglz3DTp X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Fri, 20 Jan 2023 05:11:38 +0100 Peter wrote: > Does this look familiar with anybody? ("truss -dD ls -l something") > > 0.047760369 0.000017548 > open("/etc/nsswitch.conf",O_RDONLY|O_CLOEXEC,0666) = 4 (0x4) > 0.047845099 0.000011290 > fstat(4,{ mode=-rw-r--r-- ,inode=13886,size=272,blksize=4096 }) = 0 > (0x0) 0.047932166 0.000034182 read(4,"#\n# nsswitch.conf(5) - name > ser"...,4096) = 272 (0x110) 0.048007308 0.000010661 > read(4,0x801848000,4096) = 0 (0x0) 45.827173802 45.779056767 > close(4) = 0 (0x0) ^^^^^^^^^^^^ ... 45.866272643 > 0.000020608 open("/etc/localtime",O_RDONLY,077) = 6 (0x6) > 45.866398048 0.000017731 > fstat(6,{ mode=-r--r--r-- ,inode=41572,size=2309,blksize=4096 }) = 0 > (0x0) 45.866505596 0.000054084 > read(6,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) = 2309 (0x905) > 49.511709215 3.645026276 close(6) = 0 (0x0) ^^^^^^^^^^^ > > (The longest I've seen was more than 5 minutes) > > There is otherwise nothing wrong with the machine, only I am running > a postgres table merge for some 10 mio. rows. And only during that the > effect happens, reproducibly. > > The concerned disk is a mirror of two consumer SATA-SSD. > > "gstat -pod" doesn't look bad, 1-2 items in the queue. > > zfs says it knows nothing of these delays, it just shows two > consumer SSD usually busy with their own garbage collection: > > # zpool iostat -vlq im 3 > capacity operations bandwidth > total_wait disk_wait syncq_wait asyncq_wait scrub trim > syncq_read syncq_write asyncq_read asyncq_write scrubq_read > trimq_write pool alloc free read write read > write read write read write read write read write > wait wait pend activ pend activ pend activ pend activ > pend activ pend activ ---------------- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- im 30.6G 14.9G > 0 4.84K 0 111M - 2ms - 670us - - > - 2ms - - 0 0 0 0 0 0 > 47 2 0 0 0 0 mirror-0 30.6G > 14.9G 0 4.84K 0 111M - 2ms - 670us > - - - 2ms - - 0 0 0 0 > 0 0 47 2 0 0 0 0 ada5p9.elip1 > - - 0 1.65K 0 55.7M - 3ms - 927us > - - - 3ms - - 0 0 0 0 > 0 0 0 0 0 0 0 0 ada1p9.elip1 > - - 0 3.19K 0 55.8M - 2ms - 536us > - - - 1ms - - 0 0 0 0 > 0 0 47 2 0 0 0 0 ---------------- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- capacity > operations bandwidth total_wait disk_wait syncq_wait > asyncq_wait scrub trim syncq_read syncq_write asyncq_read > asyncq_write scrubq_read trimq_write pool alloc > free read write read write read write read write read > write read write wait wait pend activ pend activ pend > activ pend activ pend activ pend activ ---------------- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- > im 30.6G 14.9G 0 5.69K 0 103M - > 3ms - 544us - - - 2ms - - > 0 0 0 0 0 0 84 2 0 0 > 0 0 mirror-0 30.6G 14.9G 0 5.69K 0 > 103M - 3ms - 544us - - - 2ms > - - 0 0 0 0 0 0 98 4 > 0 0 0 0 ada5p9.elip1 - - 0 2.05K > 0 51.4M - 5ms - 837us - - - 4ms > - - 0 0 0 0 0 0 93 2 > 0 0 0 0 ada1p9.elip1 - - 0 3.64K > 0 51.6M - 1ms - 379us - - - 1ms > - - 0 0 0 0 0 0 0 0 > 0 0 0 0 ---------------- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > ----- ----- ----- ----- > > So where does it happen? (13.1-RELEASE-p5) > > The root, /usr and /var are on ufs, on the same(!) disk (unmirrored), > and have no problem. > > The next 100-some filesystems are on the "im" pool, and there is the > problem, so it is somehow related to ZFS. (The above truss output > was in a jail, which is on the ZFS.) > > I had other things saturating these disks, with no such effect. It > seems postgres does something different here. > > It does also not really look like a congestion problem, because I > can have two or three tasks get stuck in this (and all of them will), > and they continue at very much different times. Looks rather like > some lock wait thing. > > Any clues, anybody? > Maybe unrelated or ot. Both BSD and linux, firefox in partcular or mplayer, kplayer xine ... the program has many windows or tabs open and slowly behaves like molasses. Closing or killing it takes a lot of time and cpu and sometimes never ends. "ps" will show ghosts that go away by killing "X" I have a feeling that I have seen it only on 32 bit os not on 64 bit. I am running "twm" as my window manager that is why logging out is not a good idea on my FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 cheers Peter From nobody Fri Jan 20 12:03:15 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NyynR6gYLz2v5hj for ; Fri, 20 Jan 2023 12:03:23 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [107.204.234.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NyynQ48ZTz3hWv for ; Fri, 20 Jan 2023 12:03:22 +0000 (UTC) (envelope-from david@catwhisker.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of david@catwhisker.org designates 107.204.234.170 as permitted sender) smtp.mailfrom=david@catwhisker.org; dmarc=none Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.16.1/8.15.2) with ESMTP id 30KC3F99080519 for ; Fri, 20 Jan 2023 12:03:15 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.16.1/8.16.1/Submit) id 30KC3FJs080517 for stable@freebsd.org; Fri, 20 Jan 2023 04:03:15 -0800 (PST) (envelope-from david) Date: Fri, 20 Jan 2023 04:03:15 -0800 From: David Wolfskill To: stable@freebsd.org Subject: Build failure: stable/13-n253532-5648f730c0cc -> stable/13-n253571-1852ca25dd2e Message-ID: Reply-To: stable@freebsd.org Mail-Followup-To: stable@freebsd.org List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="r007C/NtMOY6PYW8" Content-Disposition: inline X-Spamd-Result: default: False [-0.40 / 15.00]; REPLYTO_EQ_TO_ADDR(5.00)[]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip4:107.204.234.170]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DMARC_NA(0.00)[catwhisker.org]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:7018, ipnet:107.192.0.0/12, country:US]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[david]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_REPLYTO(0.00)[stable@freebsd.org] X-Rspamd-Queue-Id: 4NyynQ48ZTz3hWv X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --r007C/NtMOY6PYW8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Running in: FreeBSD freebeast.catwhisker.org 13.1-STABLE FreeBSD 13.1-STABLE #262 stabl= e/13-n253532-5648f730c0cc: Thu Jan 19 11:46:15 UTC 2023 root@freebeast.= catwhisker.org:/common/S3/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1301511= 1301511 after updating source to stable/13-n253571-1852ca25dd2e, "make buildworld" iusing meta-mode fails: =2E.. Building /common/S3/obj/usr/src/amd64.amd64/tests/sys/netpfil/pf/pfsync --- all_subdir_tests/sys/opencrypto --- Building /common/S3/obj/usr/src/amd64.amd64/tests/sys/opencrypto/poly1305_t= est --- all_subdir_tests/sys/posixshm --- *** [posixshm_test.o] Error code 1 make[6]: stopped in /usr/src/tests/sys/posixshm =2EERROR_TARGET=3D'posixshm_test.o' =2EERROR_META_FILE=3D'/common/S3/obj/usr/src/amd64.amd64/tests/sys/posixshm= /posixshm_test.o.meta' =2EMAKE.LEVEL=3D'6' MAKEFILE=3D'' =2E... Contents of the meta file: # Meta data file /common/S3/obj/usr/src/amd64.amd64/tests/sys/posixshm/posi= xshm_test.o.meta CMD cc -target x86_64-unknown-freebsd13.1 --sysroot=3D/common/S3/obj/usr/sr= c/amd64.amd64/tmp -B/common/S3/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pi= pe -fno-common -fPIE -g -std=3Dgnu99 -Wno-format-zero-length -fstack-prot= ector-strong -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-= parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn= -type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wca= st-align -Wchar-subscripts -Wnested-externs -Wredundant-decls -Wold-style-d= efinition -Wno-pointer-sign -Wmissing-variable-declarations -Wthread-safety= -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error= =3Dunused-but-set-variable -Qunused-arguments -c /usr/src/tests/sys/posix= shm/posixshm_test.c -o posixshm_test.o CMD=20 CWD /common/S3/obj/usr/src/amd64.amd64/tests/sys/posixshm TARGET posixshm_test.o OODATE /usr/src/tests/sys/posixshm/posixshm_test.c -- command output -- /usr/src/tests/sys/posixshm/posixshm_test.c:995:24: error: variable has inc= omplete type 'struct spacectl_range' struct spacectl_range range; ^ /usr/src/tests/sys/posixshm/posixshm_test.c:995:9: note: forward declaratio= n of 'struct spacectl_range' struct spacectl_range range; ^ /usr/src/tests/sys/posixshm/posixshm_test.c:1121:17: warning: variable 'psc= nt' set but not used [-Wunused-but-set-variable] int error, fd, pscnt; ^ 1 warning and 1 error generated. *** Error code 1 -- filemon acquired metadata -- # filemon version 5 # Target pid 3771 # Start 1674215754.395076 V 5 E 3792 /bin/sh R 3792 /etc/libmap.conf R 3792 /var/run/ld-elf.so.hints R 3792 /lib/libedit.so.8 R 3792 /lib/libc.so.7 R 3792 /lib/libncursesw.so.9 R 3792 /usr/share/locale/en_US.UTF-8/LC_COLLATE R 3792 /usr/share/locale/en_US.UTF-8/LC_CTYPE R 3792 /usr/share/locale/en_US.UTF-8/LC_MONETARY R 3792 /usr/share/locale/en_US.UTF-8/LC_NUMERIC R 3792 /usr/share/locale/en_US.UTF-8/LC_TIME R 3792 /usr/share/locale/en_US.UTF-8/LC_MESSAGES F 3792 3801 E 3801 /usr/bin/cc R 3801 /etc/libmap.conf R 3801 /var/run/ld-elf.so.hints R 3801 /lib/libz.so.6 R 3801 /usr/lib/libexecinfo.so.1 R 3801 /lib/libncursesw.so.9 R 3801 /lib/libthr.so.3 R 3801 /usr/lib/libc++.so.1 R 3801 /lib/libcxxrt.so.1 R 3801 /lib/libm.so.5 R 3801 /lib/libc.so.7 R 3801 /lib/libelf.so.2 R 3801 /lib/libgcc_s.so.1 R 3801 /usr/src/tests/sys/posixshm/posixshm_test.c R 3801 posixshm_test-e1014a52.o.tmp W 3801 posixshm_test-e1014a52.o.tmp R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/cdefs.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/param.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_null.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/types.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/endian.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/endian.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_types.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/_types.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/_types.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/_limits.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/_limits.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_endian.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_pthreadtypes= =2Eh R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_stdint.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/select.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_sigset.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_timeval.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/timespec.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_timespec.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/syslimits.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/signal.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/signal.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/signal.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/param.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/_align.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/_align.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/limits.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/ioctl.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/ioccom.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/filio.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/sockio.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/ttycom.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_winsize.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/mman.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/resource.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/stat.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/time.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_clock_id.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/time.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/xlocale/_time.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/syscall.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/sysctl.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/wait.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/ctype.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/_ctype.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/runetype.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/xlocale/_ctype.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/errno.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/fcntl.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/signal.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/machine/ucontext.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/x86/ucontext.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/_ucontext.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/stdlib.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/string.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/strings.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/xlocale/_strings.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/xlocale/_string.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/unistd.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/sys/unistd.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/error.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/error_fwd.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/stdbool.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/stddef.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/macros.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/defs.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/tc.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/tp.h R 3801 /common/S3/obj/usr/src/amd64.amd64/tmp/usr/include/atf-c/utils.h D 3801 posixshm_test-e1014a52.o.tmp D 3801 posixshm_test.o X 3801 1 0 X 3792 1 0 # Stop 1674215754.527076 # Bye bye Peace, david --=20 David H. Wolfskill david@catwhisker.org Putin seems to use the word "peace" in the way that Neville Chamberlain did. See https://www.catwhisker.org/~david/publickey.gpg for my public key. --r007C/NtMOY6PYW8 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQSr0Kzv+UJRY3wfOii0+6PfV4Ix1AUCY8qDA18UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0QUJE MEFDRUZGOTQyNTE2MzdDMUYzQTI4QjRGQkEzREY1NzgyMzFENAAKCRC0+6PfV4Ix 1DzOAP4tyts0UiyRTcL4rrOuOrnAHSvNyUr92aIT8btMacRYuwEAp0dhFm3oENc2 rqUDBTUAiBVWc0JqFa3aaPEetdFDews= =VOTu -----END PGP SIGNATURE----- --r007C/NtMOY6PYW8-- From nobody Fri Jan 20 14:26:39 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nz2B81LcWz2shVw for ; Fri, 20 Jan 2023 14:36:32 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nz2B656Lsz3vL3 for ; Fri, 20 Jan 2023 14:36:30 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 30KEa5Yn015293 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 20 Jan 2023 15:36:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 30KEa5wP015289; Fri, 20 Jan 2023 15:36:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30KESrf4039800 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Fri, 20 Jan 2023 15:28:54 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30KEQdLb037148 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 20 Jan 2023 15:26:40 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.16.1/8.16.1/Submit) id 30KEQdfO037147; Fri, 20 Jan 2023 15:26:39 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Fri, 20 Jan 2023 15:26:39 +0100 From: Peter To: Mateusz Guzik Cc: freebsd-stable@freebsd.org Subject: Re: close() taking minutes (ZFS related) Message-ID: References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Fri, 20 Jan 2023 15:36:08 +0100 (CET) X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DMARC_NA(0.00)[sub.org]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; RCVD_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4Nz2B656Lsz3vL3 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Fri, Jan 20, 2023 at 12:05:20PM +0100, Mateusz Guzik wrote: ! On 1/20/23, Peter wrote: ! > ! > Does this look familiar with anybody? ("truss -dD ls -l something") ! > ! > 0.047760369 0.000017548 open("/etc/nsswitch.conf",O_RDONLY|O_CLOEXEC,0666) = ! > 4 (0x4) ! > 0.047845099 0.000011290 fstat(4,{ mode=-rw-r--r-- ! > ,inode=13886,size=272,blksize=4096 }) = 0 (0x0) ! > 0.047932166 0.000034182 read(4,"#\n# nsswitch.conf(5) - name ser"...,4096) = ! > 272 (0x110) ! > 0.048007308 0.000010661 read(4,0x801848000,4096) = 0 (0x0) ! > 45.827173802 45.779056767 close(4) = 0 (0x0) ! > ^^^^^^^^^^^^ ! > ... ! > 45.866272643 0.000020608 open("/etc/localtime",O_RDONLY,077) = 6 (0x6) ! > 45.866398048 0.000017731 fstat(6,{ mode=-r--r--r-- ! > ,inode=41572,size=2309,blksize=4096 }) = 0 (0x0) ! > 45.866505596 0.000054084 read(6,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) ! > = 2309 (0x905) ! > 49.511709215 3.645026276 close(6) = 0 (0x0) ! > ^^^^^^^^^^^ ! > ! > (The longest I've seen was more than 5 minutes) ! > ! > There is otherwise nothing wrong with the machine, only I am running ! > a postgres table merge for some 10 mio. rows. And only during that the ! > effect happens, reproducibly. ! > ! ! This is most likely stalls happening on atime update. I agree, that is the most plausible location. ! Most likely the following will help: ! sysctl vfs.zfs.per_txg_dirty_frees_percent=30 ! ! see https://github.com/openzfs/zfs/issues/13932 for more information Thank You, but sadly that doesn't change it. I did now revert some file-locking and async in postgres to defaults. And I run truss from UFS, because otherwise it would already hang: root ~# truss -dD /j/admn/bin/ls -l /j/admn/etc/ And so I get things like this: ... 0.010328158 0.000009112 fstatat(AT_FDCWD,"newsyslog.conf",{ mode=-rw-r--r-- ,inode=13822,size=1708,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 0.010367043 0.000008568 fstatat(AT_FDCWD,"mail",{ mode=drwxr-xr-x ,inode=39,size=30,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 0.010405598 0.000008903 fstatat(AT_FDCWD,"apmd.conf",{ mode=-rw-r--r-- ,inode=44969,size=1240,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 16.746787754 16.736352005 fstatat(AT_FDCWD,"libmap.conf",{ mode=-rw-r--r-- ,inode=13980,size=47,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 16.746863876 0.000012441 fstatat(AT_FDCWD,"rc.conf.d",{ mode=drwxr-xr-x ,inode=51,size=2,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 16.746908593 0.000010568 fstatat(AT_FDCWD,"autofs",{ mode=drwxr-xr-x ,inode=31,size=9,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) 16.746951173 0.000010509 fstatat(AT_FDCWD,"rc.initdiskless",{ mode=-rw-r--r-- ,inode=14262,size=13680,blksize=13824 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) ... Is there an atime involved? (I tried to switch them of, but it seems that requires a remount to become effective) and 'ps' gives me the usual "zfs is busy waiting": UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 90738 90737 8 23 0 13396 3156 tx->tx_s DX+ 4 0:00.00 /j/admn/bin/ls -l /j/admn/etc/ # sysctl vfs.zfs.txg.timeout=1 vfs.zfs.txg.timeout: 15 -> 1 Doesn't change things noticeably. Do You have an idea where to look further? (dtrace should be working, but I don't know where to look) From nobody Sat Jan 21 12:53:26 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nzc0p3qnjz2v73F for ; Sat, 21 Jan 2023 13:00:26 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nzc0n1TDDz42hy for ; Sat, 21 Jan 2023 13:00:25 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 30LD05aR097103 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 21 Jan 2023 14:00:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 30LD05MV097102; Sat, 21 Jan 2023 14:00:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30LCtB0J034533 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sat, 21 Jan 2023 13:55:11 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 30LCrQ8O003897 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 21 Jan 2023 13:53:27 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.16.1/8.16.1/Submit) id 30LCrQSA003896; Sat, 21 Jan 2023 13:53:26 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sat, 21 Jan 2023 13:53:26 +0100 From: Peter To: Mateusz Guzik Cc: freebsd-stable@freebsd.org Subject: Re: close() taking minutes (ZFS related) Message-ID: References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 21 Jan 2023 14:00:08 +0100 (CET) X-Spamd-Result: default: False [-2.30 / 15.00]; URI_HIDDEN_PATH(1.00)[http://flag.daemon.contact/.well-known/acme-challenge/ZFS.busy.svg]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[sub.org]; HAS_XAW(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4Nzc0n1TDDz42hy X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Fri, Jan 20, 2023 at 03:55:43PM +0100, Mateusz Guzik wrote: ! offcpu.d: ! ! #pragma D option dynvarsize=32m ! ! /* ! * The p_flag & 0x4 test filters out kernel threads. ! */ ! ! sched:::sleep ! /(curthread->td_proc->p_flag & 0x4) == 0/ ! { ! self->ts = timestamp; ! self->mesg = curthread->td_wmesg; ! } ! ! sched:::wakeup ! /self->ts/ ! { ! @[stack(30), stringof(self->mesg)] = sum(timestamp - self->ts); ! self->ts = 0; ! self->mesg = 0; ! } ! ! dtrace:::END ! { ! printa("%k\n%s\n%@d\n\n", @); ! } ! ! dtrace -s offcpu.d -o offcpu.out ! ! git clone https://github.com/brendangregg/FlameGraph.git ! cat offcpu.out | perl FlameGraph/stackcollapse.pl | perl ! FlameGraph/flamegraph.pl --color=io > offcpu.svg Okay, got it. And now? I created two pictures, one when the machine is idle, and one when it is completely stuck. But I don't see significant differences. I drop them into the ACME directory, so You can have a look: http://flag.daemon.contact/.well-known/acme-challenge/ZFS.idle.svg http://flag.daemon.contact/.well-known/acme-challenge/ZFS.busy.svg Probably this needs more time to really understand - my brains are currently engaged with the fallout of these table migrations. I also tried to get a better view on what is exactly happening. This is a little database load: The beginning, everything normal: 770 85475 83995 0 21 0 17812 6032 select S+J 2 0:00.18 pg_restore -C -d postgres pgdump_TESTLOAD The backend starts working, and things start to hang: 770 85503 44085 4 52 0 156652 109460 zcw->zcw DsJ - 0:05.78 postgres: postgres testload [local] ALTER TABLE (post 770 85504 44090 1 20 0 580 384 tx->tx_s DJ - 0:00.00 sh -c /ext/libexec/RedoLog.copy "00000001000000290000 770 85475 83995 4 22 0 17812 6036 select S+J 2 0:00.84 pg_restore -C -d postgres pgdump_TESTLOAD It's getting bad: 0 13187 1 1 20 0 13276 768 tx->tx_s DsJ - 0:14.51 /usr/sbin/blacklistd 770 44089 44085 16 20 0 80464 12620 tx->tx_s DsJ - 0:09.90 postgres: walwriter (postgres) 770 85503 44085 15 52 0 177132 124516 tx->tx_s DsJ - 0:06.65 postgres: postgres testload [local] CREATE INDEX (pos 770 85504 44090 10 20 0 580 384 tx->tx_s DJ - 0:00.00 sh -c /ext/libexec/RedoLog.copy "00000001000000290000 770 85475 83995 19 22 0 17812 6036 select S+J 2 0:00.84 pg_restore -C -d postgres pgdump_TESTLOAD And worse: 770 44089 44085 17 20 0 80464 12620 zcw->zcw DsJ - 0:09.92 postgres: walwriter (postgres) 770 85503 44085 2 27 0 177132 124980 usem SsJ - 0:07.39 postgres: postgres testload [local] CREATE INDEX (pos 8 85660 10053 1 52 0 428 252 tx->tx_s DJ - 0:00.00 expr ## : \\(.\\).* 770 85661 85504 8 22 0 416 232 zfs DJ - 0:00.00 fsync /var/db/pg-int/arch/000000010000002900000014 770 85475 83995 19 20 0 17812 6036 select S+J 2 0:00.84 pg_restore -C -d postgres pgdump_TESTLOAD DB loaded, but things are ugly now: 8 85660 10053 16 52 0 428 252 tx->tx_s DJ - 0:00.00 expr ## : \\(.\\).* 770 85661 85504 8 22 0 416 232 zfs DJ - 0:00.00 fsync /var/db/pg-int/arch/000000010000002900000014 5001 85683 85674 16 20 0 580 380 zfs DsJ - 0:00.00 /bin/sh -c /ext/libexec/svnsync.cron /ext/SVNR svn+ss 0 85687 85677 4 20 0 580 380 zfs DsJ - 0:00.00 /bin/sh -c /usr/libexec/atrun 8 85688 85679 5 20 0 580 380 zfs DsJ - 0:00.00 /bin/sh -c /usr/local/news/bin/send-uucp 770 85666 83995 10 20 0 1020 764 zfs D+J 2 0:00.00 psql -c drop database testload Then, I tried to do that load again with sync write disabled, to see if things change (they do not). But, at that time the daily/periodic appeared, and it does a VACUUM - that is, a first VACUUM on all the migrated tables. And now I give up, because this is unuseable: 0 1430 1429 4 20 0 12860 3496 tx->tx_s D - 0:00.09 / /usr /var /home /ext /j/kerb /media /j/oper /usr/loc 0 2569 10505 14 20 0 21072 5800 tx->tx_s DJ - 0:00.03 sendmail: 30L26S6D002569 wand-n.intra.daemon.contact [ 0 2273 2269 13 30 0 13680 3480 tx->tx_s Ds - 0:00.02 /bin/sh /ext/libexec/Key.renew 770 2645 44085 2 23 0 83024 23528 tx->tx_s DsJ - 0:25.15 postgres: postgres bareos [local] VACUUM (postgres) 770 2924 89818 9 20 0 17176 5308 tx->tx_s DJ - 0:00.01 fetch --no-verify-peer -1 -T 30 -o /tmp/kurse.89801.ht 8 2936 10053 2 22 0 428 252 tx->tx_s DJ - 0:00.00 expr 29 + 1 0 2947 2845 0 20 0 18384 3736 tx->tx_s DJ - 0:00.00 sendmail: ./30L26SBv001628 from queue (sendmail) 770 2949 2668 2 21 0 424 240 zfs DJ - 0:00.00 mv /var/db/pg-int/arch/000000010000002900000023 /var/d 0 3110 3108 12 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SDj001446 from queue (sendmail) 0 3335 3334 11 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26Sjq001053 from queue (sendmail) 997 3504 4734 7 20 0 580 380 zfs DJ - 0:00.01 sh -c /sbin/ping -c2 wand-n.intra.daemon.contact > /de 997 3505 4734 15 20 0 580 380 zfs DJ - 0:00.01 sh -c sleep `jot -r 1 1 55` 997 3506 4734 12 20 0 580 380 zfs DJ - 0:00.01 sh -c /sbin/ping -c2 pole-n.intra.daemon.contact > /de 997 3507 4734 8 21 0 580 380 zfs DJ - 0:00.01 sh -c /usr/local/bin/xlsclients -display disp-e.intra. 0 3548 3547 4 21 0 18384 3696 zfs DJ - 0:00.00 sendmail: ./30L26SRV001167 from queue (sendmail) 0 3742 3740 9 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SrD001048 from queue (sendmail) 0 3955 3954 8 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26Sjs001053 from queue (sendmail) 0 4155 4154 17 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SBx001628 from queue (sendmail) 0 4375 4374 6 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SLE002237 from queue (sendmail) 0 4575 4574 17 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SDl001446 from queue (sendmail) 0 4753 4752 11 20 0 18384 3656 zfs DJ - 0:00.00 sendmail: ./30L26SrF001048 from queue (sendmail) 0 4952 4951 12 20 0 18384 3696 zfs DJ - 0:00.00 sendmail: ./30L26SRX001167 from queue (sendmail) 0 5170 5169 3 20 0 18384 3700 zfs DJ - 0:00.00 sendmail: ./30L26SWb001552 from queue (sendmail) 0 5383 5382 1 20 0 18384 3660 zfs DJ - 0:00.00 sendmail: ./30L26SYD000965 from queue (sendmail) 0 10486 1 6 20 0 13000 620 tx->tx_s DsJ - 0:13.29 /usr/sbin/cron -s 770 3514 83995 16 20 0 596 288 zfs D+J 2 0:00.00 pg_restore -C -d postgres pgdump_TESTLOAD 1000 3518 83924 18 20 0 13396 3200 zfs D+ 4 0:00.00 ls -la /j/admn/etc/ I attached truss to the VACUUM worker, and it does file reads and writes in 8k blocks continuousely, just as it should, keeping the disks busy. But meanwhile, nothing else is moving any step further. I had to wait half an hour until the VACUUM was done, then everything went back to normal, and nothing bad had happened (my scripting is usually written to cope with indefinte stalls at any place). The issue is a lot worse than I initially thought, it concerns everything: sort, reindex, vacuum, ... As it looks, ZFS allows only one single process to write to the pool, and everything else is blocked from accessing it during that time. And this seems SSD-specific; the DB was on mechanical disks before and did fine for the last 15 years, precisely saturating the SCSI and no stalls. I might think this could be related to some of the lots of pseudo-AI that went into ZFS over the years: all the knobs I used to tune have disappeared, even arc-min/arc-max seem in the process of disappearing. Initially in 2008, this DB did run on a Pentium-2 with 750 MB ram, and things did work, after proper tuning. Now it has 20 cores and 48 GB ram, and doesn't work.