From owner-freebsd-stable@FreeBSD.ORG Sun May 1 04:20:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 696D41065670 for ; Sun, 1 May 2011 04:20:30 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [76.96.27.227]) by mx1.freebsd.org (Postfix) with ESMTP id 54DB18FC15 for ; Sun, 1 May 2011 04:20:30 +0000 (UTC) Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43]) by qmta12.emeryville.ca.mail.comcast.net with comcast id e4JZ1g0050vp7WLAC4LVnM; Sun, 01 May 2011 04:20:29 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta05.emeryville.ca.mail.comcast.net with comcast id e4LU1g00g1t3BNj8R4LVjh; Sun, 01 May 2011 04:20:29 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 3ED8A9B418; Sat, 30 Apr 2011 21:20:28 -0700 (PDT) Date: Sat, 30 Apr 2011 21:20:28 -0700 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Message-ID: <20110501042028.GA87381@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: jkim@freebsd.org Subject: Is machdep.cpu_idle_hlt deprecated? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2011 04:20:30 -0000 Anyone know if machdep.cpu_idle_hlt still exists? Taken from acpi(4) on RELENG_8: hw.acpi.cpu.cx_lowest Lowest Cx state to use for idling the CPU. A scheduling algo- rithm will select states between C1 and this setting as system load dictates. To enable ACPI CPU idling control, machdep.cpu_idle_hlt must be set to 1. $ sysctl -d machdep.cpu_idle_hlt sysctl: unknown oid 'machdep.cpu_idle_hlt' I'm taking a stab in the dark here, but it looks like the variable no longer exists because it's been replaced with, effectively, the framework that drives machdep.idle and machdep.idle_available (specifically the mwait_hlt and hlt methods). Doing "grep -r cpu_idle_hlt /usr/src" turns up nothing other than the cpu_idle_hlt() functions that live within machdep.c per architecture, and those (based on the code) correlate with what's shown in machdep.idle_available. If I'm correct, I believe that means we can safely remove the last line of text in the acpi(4) man page? There's also a mention of this variable in a file called src/tools/tools/sysdoc/tunables.mdoc, but I'm not sure what that is. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Sun May 1 10:42:24 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CD72106564A; Sun, 1 May 2011 10:42:24 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 1A8078FC13; Sun, 1 May 2011 10:42:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p41AgLse089625; Sun, 1 May 2011 14:42:21 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Sun, 1 May 2011 14:42:21 +0400 (MSD) From: Dmitry Morozovsky To: "Kenneth D. Merry" In-Reply-To: <20110430211927.GA67374@nargothrond.kdm.org> Message-ID: References: <20110430211927.GA67374@nargothrond.kdm.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-834018739-1543566507-1304246541=:29081" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (woozle.rinet.ru [0.0.0.0]); Sun, 01 May 2011 14:42:21 +0400 (MSD) Cc: freebsd-stable@FreeBSD.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2011 10:42:24 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---834018739-1543566507-1304246541=:29081 Content-Type: TEXT/PLAIN; charset=US-ASCII On Sat, 30 Apr 2011, Kenneth D. Merry wrote: KDM> On Fri, Apr 29, 2011 at 11:51:21 +0400, Dmitry Morozovsky wrote: KDM> > Dear Ken, KDM> > KDM> > I have SuperMicro Server with mps driver you managed, with 24 SATA disks under KDM> > SAS x36 expander with large ZFS KDM> > KDM> > Sometimes, under random disk load such as daily find, it lost all its devices: KDM> > KDM> > [-- MARK -- Fri Apr 29 03:00:00 2011] KDM> > mps0: IOC Fault 0x40005900, Resetting^M KDM> > (pass20:mps0:0:22:0): SCSI command timeout on device handle 0x0020 SMID 442^M KDM> > mps0: IOC Fault 0x40001500, Resetting^M KDM> > (da19:mps0:0:21:0): SCSI command timeout on device handle 0x001f SMID 172^M KDM> > (da19:mps0:0:21:0): SCSI command timeout on device handle 0x001f SMID 511^M KDM> > (da20:mps0:0:20:0): SCSI command timeout on device handle 0x001e SMID 240^M KDM> > KDM> > .. KDM> > KDM> > (da4:mps0:0:0:0): SCSI command timeout on device handle 0x000a SMID 844^M KDM> > (da22:mps0:0:23:0): SCSI command timeout on device handle 0x0021 SMID 713^M KDM> > (da18:mps0:0:22:0): SCSI command timeout on device handle 0x0020 SMID 603^M KDM> > KDM> > and hangs there forever (in zio state). KDM> > KDM> > I've prepared debugging kernel with DDB and would be glad to help catch the KDM> > situation. KDM> KDM> Hmm... KDM> KDM> Can you send full dmesg output? Attached KDM> What I'm most interested in is whether KDM> there is more kernel output before the IOC Fault that might shed some light KDM> on what is going on. Nope. I use boot_verbose, but none of mps-related debug options yet KDM> KDM> Also, what brand (LSI, Maxim, etc.) and speed (3Gb, 6Gb) is the expander on KDM> the backplane? LSI 6G: KDM> What model LSI controller do you have? How many lanes are connected KDM> between the controller and the backplane? 2x4 IIR. BTW, how can investigate real SASA topology? KDM> What model disks do you have in the system? (dmesg will show that KDM> obviously.) 24 x WD RE4 2T KDM> Hopefully we can find some clues to point to the problem. /me too ;) Thank you very much! BTW, I have serial console, DDB kernel, so while this machine is in production, but not too heavy, and I can spend some time in kernel debugger if needed. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ ---834018739-1543566507-1304246541=:29081 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=dmesg.boot.txt Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename=dmesg.boot.txt VGFibGUgJ0ZBQ1AnIGF0IDB4YmY3YTAyOTANClRhYmxlICdBUElDJyBhdCAw eGJmN2EwMzkwDQpBUElDOiBGb3VuZCB0YWJsZSBhdCAweGJmN2EwMzkwDQpB UElDOiBVc2luZyB0aGUgTUFEVCBlbnVtZXJhdG9yLg0KTUFEVDogRm91bmQg Q1BVIEFQSUMgSUQgMCBBQ1BJIElEIDE6IGVuYWJsZWQNClNNUDogQWRkZWQg Q1BVIDAgKEFQKQ0KTUFEVDogRm91bmQgQ1BVIEFQSUMgSUQgMiBBQ1BJIElE IDI6IGVuYWJsZWQNClNNUDogQWRkZWQgQ1BVIDIgKEFQKQ0KTUFEVDogRm91 bmQgQ1BVIEFQSUMgSUQgNCBBQ1BJIElEIDM6IGVuYWJsZWQNClNNUDogQWRk ZWQgQ1BVIDQgKEFQKQ0KTUFEVDogRm91bmQgQ1BVIEFQSUMgSUQgNiBBQ1BJ IElEIDQ6IGVuYWJsZWQNClNNUDogQWRkZWQgQ1BVIDYgKEFQKQ0KTUFEVDog Rm91bmQgQ1BVIEFQSUMgSUQgMTMyIEFDUEkgSUQgNTogZGlzYWJsZWQNCk1B RFQ6IEZvdW5kIENQVSBBUElDIElEIDEzMyBBQ1BJIElEIDY6IGRpc2FibGVk DQpNQURUOiBGb3VuZCBDUFUgQVBJQyBJRCAxMzQgQUNQSSBJRCA3OiBkaXNh YmxlZA0KTUFEVDogRm91bmQgQ1BVIEFQSUMgSUQgMTM1IEFDUEkgSUQgODog ZGlzYWJsZWQNCkNvcHlyaWdodCAoYykgMTk5Mi0yMDExIFRoZSBGcmVlQlNE IFByb2plY3QuDQpDb3B5cmlnaHQgKGMpIDE5NzksIDE5ODAsIDE5ODMsIDE5 ODYsIDE5ODgsIDE5ODksIDE5OTEsIDE5OTIsIDE5OTMsIDE5OTQNCglUaGUg UmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwg cmlnaHRzIHJlc2VydmVkLg0KRnJlZUJTRCBpcyBhIHJlZ2lzdGVyZWQgdHJh ZGVtYXJrIG9mIFRoZSBGcmVlQlNEIEZvdW5kYXRpb24uDQpGcmVlQlNEIDgu Mi1TVEFCTEUgIzQ6IFdlZCBBcHIgMjAgMTk6Mjc6MDcgTVNEIDIwMTENCiAg ICBtYXJja0Btb29zZS5yaW5ldC5ydTovdXNyL29iai91c3Ivc3JjL3N5cy9N T09TRSBhbWQ2NA0KUHJlbG9hZGVkIGVsZiBrZXJuZWwgIi9ib290L2tlcm5l bC9rZXJuZWwiIGF0IDB4ZmZmZmZmZmY4MGEwYTAwMC4NClByZWxvYWRlZCBl bGYgb2JqIG1vZHVsZSAiL2Jvb3Qva2VybmVsL3pmcy5rbyIgYXQgMHhmZmZm ZmZmZjgwYTBhMjgwLg0KUHJlbG9hZGVkIGVsZiBvYmogbW9kdWxlICIvYm9v dC9rZXJuZWwvb3BlbnNvbGFyaXMua28iIGF0IDB4ZmZmZmZmZmY4MGEwYThl OC4NClByZWxvYWRlZCBlbGYgb2JqIG1vZHVsZSAiL2Jvb3Qva2VybmVsL2dl b21fbWlycm9yLmtvIiBhdCAweGZmZmZmZmZmODBhMGFlZDguDQpQcmVsb2Fk ZWQgL2Jvb3QvemZzL3pwb29sLmNhY2hlICIvYm9vdC96ZnMvenBvb2wuY2Fj aGUiIGF0IDB4ZmZmZmZmZmY4MGEwYjU0OC4NClRpbWVjb3VudGVyICJpODI1 NCIgZnJlcXVlbmN5IDExOTMxODIgSHogcXVhbGl0eSAwDQpDYWxpYnJhdGlu ZyBUU0MgY2xvY2sgLi4uIFRTQyBjbG9jazogMjQwMDAxNzE1NiBIeg0KQ1BV OiBJbnRlbChSKSBYZW9uKFIpIENQVSAgICAgICAgICAgWDM0MzAgIEAgMi40 MEdIeiAoMjQwMC4wMi1NSHogSzgtY2xhc3MgQ1BVKQ0KICBPcmlnaW4gPSAi R2VudWluZUludGVsIiAgSWQgPSAweDEwNmU1ICBGYW1pbHkgPSA2ICBNb2Rl bCA9IDFlICBTdGVwcGluZyA9IDUNCiAgRmVhdHVyZXM9MHhiZmViZmJmZjxG UFUsVk1FLERFLFBTRSxUU0MsTVNSLFBBRSxNQ0UsQ1g4LEFQSUMsU0VQLE1U UlIsUEdFLE1DQSxDTU9WLFBBVCxQU0UzNixDTEZMVVNILERUUyxBQ1BJLE1N WCxGWFNSLFNTRSxTU0UyLFNTLEhUVCxUTSxQQkU+DQogIEZlYXR1cmVzMj0w eDk4ZTNmZDxTU0UzLERURVM2NCxNT04sRFNfQ1BMLFZNWCxTTVgsRVNULFRN MixTU1NFMyxDWDE2LHhUUFIsUERDTSxTU0U0LjEsU1NFNC4yLFBPUENOVD4N CiAgQU1EIEZlYXR1cmVzPTB4MjgxMDA4MDA8U1lTQ0FMTCxOWCxSRFRTQ1As TE0+DQogIEFNRCBGZWF0dXJlczI9MHgxPExBSEY+DQogIFRTQzogUC1zdGF0 ZSBpbnZhcmlhbnQNCnJlYWwgbWVtb3J5ICA9IDE3MTc5ODY5MTg0ICgxNjM4 NCBNQikNClBoeXNpY2FsIG1lbW9yeSBjaHVuayhzKToNCjB4MDAwMDAwMDAw MDAwMTAwMCAtIDB4MDAwMDAwMDAwMDA5N2ZmZiwgNjE4NDk2IGJ5dGVzICgx NTEgcGFnZXMpDQoweDAwMDAwMDAwMDBhNDYwMDAgLSAweDAwMDAwMDAwYmY3 OGZmZmYsIDMyMDE2MDU2MzIgYnl0ZXMgKDc4MTY0MiBwYWdlcykNCjB4MDAw MDAwMDBiZjc5ZTAwMCAtIDB4MDAwMDAwMDBiZjc5ZmZmZiwgODE5MiBieXRl cyAoMiBwYWdlcykNCjB4MDAwMDAwMDEwMDAwMDAwMCAtIDB4MDAwMDAwMDQy MDBhZmZmZiwgMTM0MjI0OTM2OTYgYnl0ZXMgKDMyNzY5NzYgcGFnZXMpDQph dmFpbCBtZW1vcnkgPSAxNjUyMDc0OTA1NiAoMTU3NTUgTUIpDQpBQ1BJIEFQ SUMgVGFibGU6IDwwMjI1MTAgQVBJQzE5MTk+DQpJTlRSOiBBZGRpbmcgbG9j YWwgQVBJQyAyIGFzIGEgdGFyZ2V0DQpJTlRSOiBBZGRpbmcgbG9jYWwgQVBJ QyA0IGFzIGEgdGFyZ2V0DQpJTlRSOiBBZGRpbmcgbG9jYWwgQVBJQyA2IGFz IGEgdGFyZ2V0DQpGcmVlQlNEL1NNUDogTXVsdGlwcm9jZXNzb3IgU3lzdGVt IERldGVjdGVkOiA0IENQVXMNCkZyZWVCU0QvU01QOiAxIHBhY2thZ2Uocykg eCA0IGNvcmUocykNCiBjcHUwIChCU1ApOiBBUElDIElEOiAgMA0KIGNwdTEg KEFQKTogQVBJQyBJRDogIDINCiBjcHUyIChBUCk6IEFQSUMgSUQ6ICA0DQog Y3B1MyAoQVApOiBBUElDIElEOiAgNg0KeDg2YmlvczogICBJVlQgMHgwMDAw MDAtMHgwMDA0ZmYgYXQgMHhmZmZmZmYwMDAwMDAwMDAwDQp4ODZiaW9zOiAg U1NFRyAweDAxMDAwMC0weDAxZmZmZiBhdCAweGZmZmZmZjgwMDAwNDcwMDAN Cng4NmJpb3M6ICBFQkRBIDB4MDljMDAwLTB4MDlmZmZmIGF0IDB4ZmZmZmZm MDAwMDA5YzAwMA0KeDg2YmlvczogICBST00gMHgwYTAwMDAtMHgwZWZmZmYg YXQgMHhmZmZmZmYwMDAwMGEwMDAwDQpsYXBpYzA6IENNQ0kgdW5tYXNrZWQN CkFQSUM6IENQVSAwIGhhcyBBQ1BJIElEIDENCkFQSUM6IENQVSAxIGhhcyBB Q1BJIElEIDINCkFQSUM6IENQVSAyIGhhcyBBQ1BJIElEIDMNCkFQSUM6IENQ VSAzIGhhcyBBQ1BJIElEIDQNClVMRTogc2V0dXAgY3B1IDANClVMRTogc2V0 dXAgY3B1IDENClVMRTogc2V0dXAgY3B1IDINClVMRTogc2V0dXAgY3B1IDMN CkFDUEk6IFJTRFAgMHhmYTEyMCAwMDAyNCAodjAyIEFDUElBTSkNCkFDUEk6 IFhTRFQgMHhiZjdhMDEwMCAwMDA4NCAodjAxIFNNQ0kgICAgICAgICAgICAy MDEwMDIyNSBNU0ZUIDAwMDAwMDk3KQ0KQUNQSTogRkFDUCAweGJmN2EwMjkw IDAwMEY0ICh2MDMgMDIyNTEwIEZBQ1AxOTE5IDIwMTAwMjI1IE1TRlQgMDAw MDAwOTcpDQpBQ1BJOiBEU0RUIDB4YmY3YTA1ZjAgMDZDRDggKHYwMSAgMTA0 MEQgMTA0MEQwMDAgMDAwMDAwMDAgSU5UTCAyMDA1MTExNykNCkFDUEk6IEZB Q1MgMHhiZjdhZTAwMCAwMDA0MA0KQUNQSTogQVBJQyAweGJmN2EwMzkwIDAw MDkyICh2MDEgMDIyNTEwIEFQSUMxOTE5IDIwMTAwMjI1IE1TRlQgMDAwMDAw OTcpDQpBQ1BJOiBNQ0ZHIDB4YmY3YTA0MzAgMDAwM0MgKHYwMSAwMjI1MTAg T0VNTUNGRyAgMjAxMDAyMjUgTVNGVCAwMDAwMDA5NykNCkFDUEk6IE9FTUIg MHhiZjdhZTA0MCAwMDA3MyAodjAxIDAyMjUxMCBPRU1CMTkxOSAyMDEwMDIy NSBNU0ZUIDAwMDAwMDk3KQ0KQUNQSTogSFBFVCAweGJmN2FhNWYwIDAwMDM4 ICh2MDEgMDIyNTEwIE9FTUhQRVQgIDIwMTAwMjI1IE1TRlQgMDAwMDAwOTcp DQpBQ1BJOiBHU0NJIDB4YmY3YWUwYzAgMDIwMjQgKHYwMSAwMjI1MTAgR01D SFNDSSAgMjAxMDAyMjUgTVNGVCAwMDAwMDA5NykNCkFDUEk6IERNQVIgMHhi ZjdiMDBmMCAwMDA5MCAodjAxICAgIEFNSSAgT0VNRE1BUiAwMDAwMDAwMSBN U0ZUIDAwMDAwMDk3KQ0KQUNQSTogU1NEVCAweGJmN2IxNTgwIDAwMzYzICh2 MDEgRHBnUG1tICAgIENwdVBtIDAwMDAwMDEyIElOVEwgMjAwNTExMTcpDQpB Q1BJOiBFSU5KIDB4YmY3YWE2MzAgMDAxMzAgKHYwMSAgQU1JRVIgQU1JX0VJ TkogMjAxMDAyMjUgTVNGVCAwMDAwMDA5NykNCkFDUEk6IEJFUlQgMHhiZjdh YTdjMCAwMDAzMCAodjAxICBBTUlFUiBBTUlfQkVSVCAyMDEwMDIyNSBNU0ZU IDAwMDAwMDk3KQ0KQUNQSTogRVJTVCAweGJmN2FhN2YwIDAwMUIwICh2MDEg IEFNSUVSIEFNSV9FUlNUIDIwMTAwMjI1IE1TRlQgMDAwMDAwOTcpDQpBQ1BJ OiBIRVNUIDB4YmY3YWE5YTAgMDAwQTggKHYwMSAgQU1JRVIgQUJDX0hFU1Qg MjAxMDAyMjUgTVNGVCAwMDAwMDA5NykNCk1BRFQ6IEZvdW5kIElPIEFQSUMg SUQgNywgSW50ZXJydXB0IDAgYXQgMHhmZWMwMDAwMA0KaW9hcGljMDogQ2hh bmdpbmcgQVBJQyBJRCB0byA3DQppb2FwaWMwOiBSb3V0aW5nIGV4dGVybmFs IDgyNTlBJ3MgLT4gaW50cGluIDANCk1BRFQ6IEludGVycnVwdCBvdmVycmlk ZTogc291cmNlIDAsIGlycSAyDQppb2FwaWMwOiBSb3V0aW5nIElSUSAwIC0+ IGludHBpbiAyDQpNQURUOiBJbnRlcnJ1cHQgb3ZlcnJpZGU6IHNvdXJjZSA5 LCBpcnEgOQ0KaW9hcGljMDogaW50cGluIDkgdHJpZ2dlcjogbGV2ZWwNCmxh cGljOiBSb3V0aW5nIE5NSSAtPiBMSU5UMQ0KbGFwaWM6IExJTlQxIHRyaWdn ZXI6IGVkZ2UNCmxhcGljOiBMSU5UMSBwb2xhcml0eTogaGlnaA0KaW9hcGlj MCA8VmVyc2lvbiAyLjA+IGlycXMgMC0yMyBvbiBtb3RoZXJib2FyZA0KY3B1 MCBCU1A6DQogICAgIElEOiAweDAwMDAwMDAwICAgVkVSOiAweDAwMDYwMDE1 IExEUjogMHgwMDAwMDAwMCBERlI6IDB4ZmZmZmZmZmYNCiAgbGludDA6IDB4 MDAwMTA3MDAgbGludDE6IDB4MDAwMDA0MDAgVFBSOiAweDAwMDAwMDAwIFNW UjogMHgwMDAwMDFmZg0KICB0aW1lcjogMHgwMDAxMDBlZiB0aGVybTogMHgw MDAxMDAwMCBlcnI6IDB4MDAwMDAwZjAgcG1jOiAweDAwMDEwNDAwDQogICBj bWNpOiAweDAwMDAwMGYyDQppbzogPEkvTz4NCnJhbmRvbTogPGVudHJvcHkg c291cmNlLCBTb2Z0d2FyZSwgWWFycm93Pg0KbmZzbG9jazogcHNldWRvLWRl dmljZQ0Ka2JkOiBuZXcgYXJyYXkgc2l6ZSA0DQprYmQxIGF0IGtiZG11eDAN Cm1lbTogPG1lbW9yeT4NCm51bGw6IDxudWxsIGRldmljZSwgemVybyBkZXZp Y2U+DQphY3BpMDogPFNNQ0kgPiBvbiBtb3RoZXJib2FyZA0KUENJZTogTWVt b3J5IE1hcHBlZCBjb25maWd1cmF0aW9uIGJhc2UgQCAweGUwMDAwMDAwDQpp b2FwaWMwOiByb3V0aW5nIGludHBpbiA5IChJU0EgSVJRIDkpIHRvIGxhcGlj IDAgdmVjdG9yIDQ4DQphY3BpMDogW01QU0FGRV0NCmFjcGkwOiBbSVRIUkVB RF0NCkFDUEk6IEV4ZWN1dGVkIDEgYmxvY2tzIG9mIG1vZHVsZS1sZXZlbCBl eGVjdXRhYmxlIEFNTCBjb2RlDQphY3BpMDogUG93ZXIgQnV0dG9uIChmaXhl ZCkNCkFDUEkgdGltZXI6IDEvMSAxLzEgMS8xIDEvMSAxLzEgMS8xIDEvMSAx LzEgMS8xIDEvMSAtPiAxMA0KVGltZWNvdW50ZXIgIkFDUEktZmFzdCIgZnJl cXVlbmN5IDM1Nzk1NDUgSHogcXVhbGl0eSAxMDAwDQphY3BpX3RpbWVyMDog PDI0LWJpdCB0aW1lciBhdCAzLjU3OTU0NU1Iej4gcG9ydCAweDgwOC0weDgw YiBvbiBhY3BpMA0KY3B1MDogPEFDUEkgQ1BVPiBvbiBhY3BpMA0KQUNQSSBX YXJuaW5nOiBJbmNvcnJlY3QgY2hlY2tzdW0gaW4gdGFibGUgW09FTUJdIC0g MHg5MCwgc2hvdWxkIGJlIDB4OEYgKDIwMTAxMDEzL3RidXRpbHMtMzU0KQ0K QUNQSTogU1NEVCAweGJmN2IwMTgwIDAwRjIwICh2MDEgRHBnUG1tICBQMDAx SXN0IDAwMDAwMDExIElOVEwgMjAwNTExMTcpDQpBQ1BJOiBEeW5hbWljIE9F TSBUYWJsZSBMb2FkOg0KQUNQSTogU1NEVCAwIDAwRjIwICh2MDEgRHBnUG1t ICBQMDAxSXN0IDAwMDAwMDExIElOVEwgMjAwNTExMTcpDQpBQ1BJOiBTU0RU IDB4YmY3YjEwYTAgMDA0RDUgKHYwMSAgUG1SZWYgIFAwMDFDc3QgMDAwMDMw MDEgSU5UTCAyMDA1MTExNykNCkFDUEk6IER5bmFtaWMgT0VNIFRhYmxlIExv YWQ6DQpBQ1BJOiBTU0RUIDAgMDA0RDUgKHYwMSAgUG1SZWYgIFAwMDFDc3Qg MDAwMDMwMDEgSU5UTCAyMDA1MTExNykNCmNwdTE6IDxBQ1BJIENQVT4gb24g YWNwaTANCmNwdTI6IDxBQ1BJIENQVT4gb24gYWNwaTANCmNwdTM6IDxBQ1BJ IENQVT4gb24gYWNwaTANCnBjaV9saW5rMDogICAgICAgIEluZGV4ICBJUlEg IFJ0ZCAgUmVmICBJUlFzDQogIEluaXRpYWwgUHJvYmUgICAgICAgMCAgIDEw ICAgTiAgICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KICBWYWxpZGF0 aW9uICAgICAgICAgIDAgICAxMCAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEg MTIgMTQgMTUNCiAgQWZ0ZXIgRGlzYWJsZSAgICAgICAwICAyNTUgICBOICAg ICAwICAzIDQgNiA3IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazE6ICAgICAg ICBJbmRleCAgSVJRICBSdGQgIFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2Jl ICAgICAgIDAgICAxMSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQg MTUNCiAgVmFsaWRhdGlvbiAgICAgICAgICAwICAgMTEgICBOICAgICAwICAz IDQgNiA3IDEwIDExIDEyIDE0IDE1DQogIEFmdGVyIERpc2FibGUgICAgICAg MCAgMjU1ICAgTiAgICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KcGNp X2xpbmsyOiAgICAgICAgSW5kZXggIElSUSAgUnRkICBSZWYgIElSUXMNCiAg SW5pdGlhbCBQcm9iZSAgICAgICAwICAgIDcgICBOICAgICAwICAzIDQgNiA3 IDEwIDExIDEyIDE0IDE1DQogIFZhbGlkYXRpb24gICAgICAgICAgMCAgICA3 ICAgTiAgICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KICBBZnRlciBE aXNhYmxlICAgICAgIDAgIDI1NSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEg MTIgMTQgMTUNCnBjaV9saW5rMzogICAgICAgIEluZGV4ICBJUlEgIFJ0ZCAg UmVmICBJUlFzDQogIEluaXRpYWwgUHJvYmUgICAgICAgMCAgIDE0ICAgTiAg ICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KICBWYWxpZGF0aW9uICAg ICAgICAgIDAgICAxNCAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQg MTUNCiAgQWZ0ZXIgRGlzYWJsZSAgICAgICAwICAyNTUgICBOICAgICAwICAz IDQgNiA3IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazQ6ICAgICAgICBJbmRl eCAgSVJRICBSdGQgIFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2JlICAgICAg IDAgIDI1NSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQgMTUNCiAg VmFsaWRhdGlvbiAgICAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNiA3 IDEwIDExIDEyIDE0IDE1DQogIEFmdGVyIERpc2FibGUgICAgICAgMCAgMjU1 ICAgTiAgICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KcGNpX2xpbms1 OiAgICAgICAgSW5kZXggIElSUSAgUnRkICBSZWYgIElSUXMNCiAgSW5pdGlh bCBQcm9iZSAgICAgICAwICAgIDMgICBOICAgICAwICAzIDQgNiA3IDEwIDEx IDEyIDE0IDE1DQogIFZhbGlkYXRpb24gICAgICAgICAgMCAgICAzICAgTiAg ICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KICBBZnRlciBEaXNhYmxl ICAgICAgIDAgIDI1NSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQg MTUNCnBjaV9saW5rNjogICAgICAgIEluZGV4ICBJUlEgIFJ0ZCAgUmVmICBJ UlFzDQogIEluaXRpYWwgUHJvYmUgICAgICAgMCAgMjU1ICAgTiAgICAgMCAg MyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KICBWYWxpZGF0aW9uICAgICAgICAg IDAgIDI1NSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQgMTUNCiAg QWZ0ZXIgRGlzYWJsZSAgICAgICAwICAyNTUgICBOICAgICAwICAzIDQgNiA3 IDEwIDExIDEyIDE0IDE1DQpwY2lfbGluazc6ICAgICAgICBJbmRleCAgSVJR ICBSdGQgIFJlZiAgSVJRcw0KICBJbml0aWFsIFByb2JlICAgICAgIDAgICAx NSAgIE4gICAgIDAgIDMgNCA2IDcgMTAgMTEgMTIgMTQgMTUNCiAgVmFsaWRh dGlvbiAgICAgICAgICAwICAgMTUgICBOICAgICAwICAzIDQgNiA3IDEwIDEx IDEyIDE0IDE1DQogIEFmdGVyIERpc2FibGUgICAgICAgMCAgMjU1ICAgTiAg ICAgMCAgMyA0IDYgNyAxMCAxMSAxMiAxNCAxNQ0KcGNpYjA6IDxBQ1BJIEhv c3QtUENJIGJyaWRnZT4gcG9ydCAweGNmOC0weGNmZiBvbiBhY3BpMA0KcGNp MDogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjANCnBjaTA6IGRvbWFpbj0wLCBw aHlzaWNhbCBidXM9MA0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHhk MTMwLCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTAsIGZ1 bmM9MA0KCWNsYXNzPTA2LTAwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTAN CgljbWRyZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTAg KGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDAg KDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCmZvdW5kLT4JdmVuZG9yPTB4 ODA4NiwgZGV2PTB4ZDEzOCwgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBidXM9 MCwgc2xvdD0zLCBmdW5jPTANCgljbGFzcz0wNi0wNC0wMCwgaGRydHlwZT0w eDAxLCBtZmRldj0wDQoJY21kcmVnPTB4MDEwNCwgc3RhdHJlZz0weDAwMTAs IGNhY2hlbG5zej04IChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyks IG1pbmdudD0weDAyICg1MDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCglp bnRwaW49YSwgaXJxPTEwDQoJcG93ZXJzcGVjIDMgIHN1cHBvcnRzIEQwIEQz ICBjdXJyZW50IEQwDQoJTVNJIHN1cHBvcnRzIDIgbWVzc2FnZXMsIHZlY3Rv ciBtYXNrcw0KcGNpYjA6IG1hdGNoZWQgZW50cnkgZm9yIDAuMy5JTlRBDQpw Y2liMDogc2xvdCAzIElOVEEgaGFyZHdpcmVkIHRvIElSUSAxNg0KZm91bmQt Pgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHhkMTNhLCByZXZpZD0weDExDQoJZG9t YWluPTAsIGJ1cz0wLCBzbG90PTUsIGZ1bmM9MA0KCWNsYXNzPTA2LTA0LTAw LCBoZHJ0eXBlPTB4MDEsIG1mZGV2PTANCgljbWRyZWc9MHgwMTA3LCBzdGF0 cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTggKGR3b3JkcykNCglsYXR0aW1lcj0w eDAwICgwIG5zKSwgbWluZ250PTB4MDIgKDUwMCBucyksIG1heGxhdD0weDAw ICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9MTANCglwb3dlcnNwZWMgMyAgc3Vw cG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kgc3VwcG9ydHMgMiBtZXNz YWdlcywgdmVjdG9yIG1hc2tzDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3Ig MC41LklOVEENCnBjaWIwOiBzbG90IDUgSU5UQSBoYXJkd2lyZWQgdG8gSVJR IDE2DQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weGQxNTUsIHJldmlk PTB4MTENCglkb21haW49MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0wDQoJY2xh c3M9MDgtODAtMDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNtZHJlZz0w eDAwMDAsIHN0YXRyZWc9MHgwMDEwLCBjYWNoZWxuc3o9OCAoZHdvcmRzKQ0K CWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1h eGxhdD0weDAwICgwIG5zKQ0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9 MHhkMTU2LCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTgs IGZ1bmM9MQ0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2 PTENCgljbWRyZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6 PTggKGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4 MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCmZvdW5kLT4JdmVuZG9y PTB4ODA4NiwgZGV2PTB4ZDE1NywgcmV2aWQ9MHgxMQ0KCWRvbWFpbj0wLCBi dXM9MCwgc2xvdD04LCBmdW5jPTINCgljbGFzcz0wOC04MC0wMCwgaGRydHlw ZT0weDAwLCBtZmRldj0xDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAw MTAsIGNhY2hlbG5zej04IChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBu cyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpm b3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weGQxNTgsIHJldmlkPTB4MTEN Cglkb21haW49MCwgYnVzPTAsIHNsb3Q9OCwgZnVuYz0zDQoJY2xhc3M9MDgt ODAtMDAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MQ0KCWNtZHJlZz0weDAwMDAs IHN0YXRyZWc9MHgwMDAwLCBjYWNoZWxuc3o9OCAoZHdvcmRzKQ0KCWxhdHRp bWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0w eDAwICgwIG5zKQ0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHhkMTUw LCByZXZpZD0weDExDQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTE2LCBmdW5j PTANCgljbGFzcz0wOC04MC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0xDQoJ Y21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAwMDAsIGNhY2hlbG5zej04IChk d29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgw IG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpmb3VuZC0+CXZlbmRvcj0weDgw ODYsIGRldj0weGQxNTEsIHJldmlkPTB4MTENCglkb21haW49MCwgYnVzPTAs IHNsb3Q9MTYsIGZ1bmM9MQ0KCWNsYXNzPTA4LTgwLTAwLCBoZHJ0eXBlPTB4 MDAsIG1mZGV2PTENCgljbWRyZWc9MHgwMDAwLCBzdGF0cmVnPTB4MDAwMCwg Y2FjaGVsbnN6PTggKGR3b3JkcykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwg bWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCmZvdW5k LT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2IzYywgcmV2aWQ9MHgwNQ0KCWRv bWFpbj0wLCBidXM9MCwgc2xvdD0yNiwgZnVuYz0wDQoJY2xhc3M9MGMtMDMt MjAsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MA0KCWNtZHJlZz0weDAwMDYsIHN0 YXRyZWc9MHgwMjkwLCBjYWNoZWxuc3o9MCAoZHdvcmRzKQ0KCWxhdHRpbWVy PTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAw ICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9Mw0KCXBvd2Vyc3BlYyAyICBzdXBw b3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCW1hcFsxMF06IHR5cGUgTWVtb3J5 LCByYW5nZSAzMiwgYmFzZSAweGZiM2ZjMDAwLCBzaXplIDEwLCBlbmFibGVk DQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3IgMC4yNi5JTlRBDQpwY2liMDog c2xvdCAyNiBJTlRBIGhhcmR3aXJlZCB0byBJUlEgMjENCnVua25vd246IFJl c2VydmVkIDB4NDAwIGJ5dGVzIGZvciByaWQgMHgxMCB0eXBlIDMgYXQgMHhm YjNmYzAwMA0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHgzYjQyLCBy ZXZpZD0weDA1DQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTI4LCBmdW5jPTAN CgljbGFzcz0wNi0wNC0wMCwgaGRydHlwZT0weDAxLCBtZmRldj0xDQoJY21k cmVnPTB4MDEwNCwgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej04IChkd29y ZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAyICg1MDAg bnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCglpbnRwaW49YSwgaXJxPTExDQoJ cG93ZXJzcGVjIDIgIHN1cHBvcnRzIEQwIEQzICBjdXJyZW50IEQwDQoJTVNJ IHN1cHBvcnRzIDEgbWVzc2FnZQ0KcGNpYjA6IG1hdGNoZWQgZW50cnkgZm9y IDAuMjguSU5UQQ0KcGNpYjA6IHNsb3QgMjggSU5UQSBoYXJkd2lyZWQgdG8g SVJRIDE3DQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRldj0weDNiNGEsIHJl dmlkPTB4MDUNCglkb21haW49MCwgYnVzPTAsIHNsb3Q9MjgsIGZ1bmM9NA0K CWNsYXNzPTA2LTA0LTAwLCBoZHJ0eXBlPTB4MDEsIG1mZGV2PTENCgljbWRy ZWc9MHgwMTA3LCBzdGF0cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTggKGR3b3Jk cykNCglsYXR0aW1lcj0weDAwICgwIG5zKSwgbWluZ250PTB4MDIgKDUwMCBu cyksIG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1hLCBpcnE9MTENCglw b3dlcnNwZWMgMiAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kg c3VwcG9ydHMgMSBtZXNzYWdlDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3Ig MC4yOC5JTlRBDQpwY2liMDogc2xvdCAyOCBJTlRBIGhhcmR3aXJlZCB0byBJ UlEgMTcNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2I0YywgcmV2 aWQ9MHgwNQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0yOCwgZnVuYz01DQoJ Y2xhc3M9MDYtMDQtMDAsIGhkcnR5cGU9MHgwMSwgbWZkZXY9MQ0KCWNtZHJl Zz0weDAxMDcsIHN0YXRyZWc9MHgwMDEwLCBjYWNoZWxuc3o9OCAoZHdvcmRz KQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMiAoNTAwIG5z KSwgbWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWIsIGlycT0xMA0KCXBv d2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBz dXBwb3J0cyAxIG1lc3NhZ2UNCnBjaWIwOiBtYXRjaGVkIGVudHJ5IGZvciAw LjI4LklOVEINCnBjaWIwOiBzbG90IDI4IElOVEIgaGFyZHdpcmVkIHRvIElS USAxNg0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHgzYjM0LCByZXZp ZD0weDA1DQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90PTI5LCBmdW5jPTANCglj bGFzcz0wYy0wMy0yMCwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVn PTB4MDAwNiwgc3RhdHJlZz0weDAyOTAsIGNhY2hlbG5zej0wIChkd29yZHMp DQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwg bWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWEsIGlycT0xNQ0KCXBvd2Vy c3BlYyAyICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCW1hcFsxMF06 IHR5cGUgTWVtb3J5LCByYW5nZSAzMiwgYmFzZSAweGZiM2ZhMDAwLCBzaXpl IDEwLCBlbmFibGVkDQpwY2liMDogbWF0Y2hlZCBlbnRyeSBmb3IgMC4yOS5J TlRBDQpwY2liMDogc2xvdCAyOSBJTlRBIGhhcmR3aXJlZCB0byBJUlEgMjMN CnVua25vd246IFJlc2VydmVkIDB4NDAwIGJ5dGVzIGZvciByaWQgMHgxMCB0 eXBlIDMgYXQgMHhmYjNmYTAwMA0KZm91bmQtPgl2ZW5kb3I9MHg4MDg2LCBk ZXY9MHgyNDRlLCByZXZpZD0weGE1DQoJZG9tYWluPTAsIGJ1cz0wLCBzbG90 PTMwLCBmdW5jPTANCgljbGFzcz0wNi0wNC0wMSwgaGRydHlwZT0weDAxLCBt ZmRldj0wDQoJY21kcmVnPTB4MDEwNywgc3RhdHJlZz0weDAwMTAsIGNhY2hl bG5zej0wIChkd29yZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdu dD0weDFhICg2NTAwIG5zKSwgbWF4bGF0PTB4MDAgKDAgbnMpDQpmb3VuZC0+ CXZlbmRvcj0weDgwODYsIGRldj0weDNiMTQsIHJldmlkPTB4MDUNCglkb21h aW49MCwgYnVzPTAsIHNsb3Q9MzEsIGZ1bmM9MA0KCWNsYXNzPTA2LTAxLTAw LCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTENCgljbWRyZWc9MHgwMDA3LCBzdGF0 cmVnPTB4MDIxMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0w eDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAo MCBucykNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4M2IyMiwgcmV2 aWQ9MHgwNQ0KCWRvbWFpbj0wLCBidXM9MCwgc2xvdD0zMSwgZnVuYz0yDQoJ Y2xhc3M9MDEtMDYtMDEsIGhkcnR5cGU9MHgwMCwgbWZkZXY9MA0KCWNtZHJl Zz0weDAwMDcsIHN0YXRyZWc9MHgwMmIwLCBjYWNoZWxuc3o9MCAoZHdvcmRz KQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9MHgwMCAoMCBucyks IG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1iLCBpcnE9MTQNCglwb3dl cnNwZWMgMyAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQgRDANCglNU0kgc3Vw cG9ydHMgMSBtZXNzYWdlDQoJbWFwWzEwXTogdHlwZSBJL08gUG9ydCwgcmFu Z2UgMzIsIGJhc2UgMHhiNDAwLCBzaXplICAzLCBlbmFibGVkDQoJbWFwWzE0 XTogdHlwZSBJL08gUG9ydCwgcmFuZ2UgMzIsIGJhc2UgMHhiYzAwLCBzaXpl ICAyLCBlbmFibGVkDQoJbWFwWzE4XTogdHlwZSBJL08gUG9ydCwgcmFuZ2Ug MzIsIGJhc2UgMHhiODgwLCBzaXplICAzLCBlbmFibGVkDQoJbWFwWzFjXTog dHlwZSBJL08gUG9ydCwgcmFuZ2UgMzIsIGJhc2UgMHhiODAwLCBzaXplICAy LCBlbmFibGVkDQoJbWFwWzIwXTogdHlwZSBJL08gUG9ydCwgcmFuZ2UgMzIs IGJhc2UgMHhiNDgwLCBzaXplICA1LCBlbmFibGVkDQoJbWFwWzI0XTogdHlw ZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4ZmIzZjgwMDAsIHNpemUgMTEs IGVuYWJsZWQNCnBjaWIwOiBtYXRjaGVkIGVudHJ5IGZvciAwLjMxLklOVEIN CnBjaWIwOiBzbG90IDMxIElOVEIgaGFyZHdpcmVkIHRvIElSUSAxOQ0KZm91 bmQtPgl2ZW5kb3I9MHg4MDg2LCBkZXY9MHgzYjMwLCByZXZpZD0weDA1DQoJ ZG9tYWluPTAsIGJ1cz0wLCBzbG90PTMxLCBmdW5jPTMNCgljbGFzcz0wYy0w NS0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVnPTB4MDAwMywg c3RhdHJlZz0weDAyODAsIGNhY2hlbG5zej0wIChkd29yZHMpDQoJbGF0dGlt ZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5zKSwgbWF4bGF0PTB4 MDAgKDAgbnMpDQoJaW50cGluPWMsIGlycT03DQoJbWFwWzEwXTogdHlwZSBN ZW1vcnksIHJhbmdlIDY0LCBiYXNlIDB4ZmIzZjYwMDAsIHNpemUgIDgsIGVu YWJsZWQNCgltYXBbMjBdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAzMiwgYmFz ZSAweDQwMCwgc2l6ZSAgNSwgZW5hYmxlZA0KcGNpYjA6IG1hdGNoZWQgZW50 cnkgZm9yIDAuMzEuSU5UQw0KcGNpYjA6IHNsb3QgMzEgSU5UQyBoYXJkd2ly ZWQgdG8gSVJRIDE4DQpwY2liMTogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGly cSAxNiBhdCBkZXZpY2UgMy4wIG9uIHBjaTANCnBjaWIxOiAgIGRvbWFpbiAg ICAgICAgICAgIDANCnBjaWIxOiAgIHNlY29uZGFyeSBidXMgICAgIDENCnBj aWIxOiAgIHN1Ym9yZGluYXRlIGJ1cyAgIDENCnBjaWIxOiAgIEkvTyBkZWNv ZGUgICAgICAgIDB4MC0weDANCnBjaWIxOiAgIG5vIHByZWZldGNoZWQgZGVj b2RlDQpwY2kxOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMQ0KcGNpMTogZG9t YWluPTAsIHBoeXNpY2FsIGJ1cz0xDQpwY2liMjogPEFDUEkgUENJLVBDSSBi cmlkZ2U+IGlycSAxNiBhdCBkZXZpY2UgNS4wIG9uIHBjaTANCnBjaWIyOiAg IGRvbWFpbiAgICAgICAgICAgIDANCnBjaWIyOiAgIHNlY29uZGFyeSBidXMg ICAgIDINCnBjaWIyOiAgIHN1Ym9yZGluYXRlIGJ1cyAgIDINCnBjaWIyOiAg IEkvTyBkZWNvZGUgICAgICAgIDB4YzAwMC0weGNmZmYNCnBjaWIyOiAgIG1l bW9yeSBkZWNvZGUgICAgIDB4ZmI0MDAwMDAtMHhmYjRmZmZmZg0KcGNpYjI6 ICAgbm8gcHJlZmV0Y2hlZCBkZWNvZGUNCnBjaWIyOiBjb3VsZCBub3QgZ2V0 IFBDSSBpbnRlcnJ1cHQgcm91dGluZyB0YWJsZSBmb3IgXFxfU0JfLlBDSTAu UDBQNSAtIEFFX05PVF9GT1VORA0KcGNpMjogPEFDUEkgUENJIGJ1cz4gb24g cGNpYjINCnBjaTI6IGRvbWFpbj0wLCBwaHlzaWNhbCBidXM9Mg0KZm91bmQt Pgl2ZW5kb3I9MHgxMDAwLCBkZXY9MHgwMDcyLCByZXZpZD0weDAyDQoJZG9t YWluPTAsIGJ1cz0yLCBzbG90PTAsIGZ1bmM9MA0KCWNsYXNzPTAxLTA3LTAw LCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMTQ3LCBzdGF0 cmVnPTB4MDAxMCwgY2FjaGVsbnN6PTggKGR3b3JkcykNCglsYXR0aW1lcj0w eDAwICgwIG5zKSwgbWluZ250PTB4MDAgKDAgbnMpLCBtYXhsYXQ9MHgwMCAo MCBucykNCglpbnRwaW49YSwgaXJxPTEwDQoJcG93ZXJzcGVjIDMgIHN1cHBv cnRzIEQwIEQxIEQyIEQzICBjdXJyZW50IEQwDQoJTVNJIHN1cHBvcnRzIDEg bWVzc2FnZSwgNjQgYml0DQoJTVNJLVggc3VwcG9ydHMgMTUgbWVzc2FnZXMg aW4gbWFwIDB4MTQNCgltYXBbMTBdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAz MiwgYmFzZSAweGMwMDAsIHNpemUgIDgsIGVuYWJsZWQNCnBjaWIyOiByZXF1 ZXN0ZWQgSS9PIHJhbmdlIDB4YzAwMC0weGMwZmY6IGluIHJhbmdlDQoJbWFw WzE0XTogdHlwZSBNZW1vcnksIHJhbmdlIDY0LCBiYXNlIDB4ZmI0M2MwMDAs IHNpemUgMTQsIGVuYWJsZWQNCnBjaWIyOiByZXF1ZXN0ZWQgbWVtb3J5IHJh bmdlIDB4ZmI0M2MwMDAtMHhmYjQzZmZmZjogZ29vZA0KCW1hcFsxY106IHR5 cGUgTWVtb3J5LCByYW5nZSA2NCwgYmFzZSAweGZiNDQwMDAwLCBzaXplIDE4 LCBlbmFibGVkDQpwY2liMjogcmVxdWVzdGVkIG1lbW9yeSByYW5nZSAweGZi NDQwMDAwLTB4ZmI0N2ZmZmY6IGdvb2QNCnBjaWIwOiBtYXRjaGVkIGVudHJ5 IGZvciAwLjUuSU5UQQ0KcGNpYjA6IHNsb3QgNSBJTlRBIGhhcmR3aXJlZCB0 byBJUlEgMTYNCnBjaWIyOiBzbG90IDAgSU5UQSBpcyByb3V0ZWQgdG8gaXJx IDE2DQptcHMwOiA8TFNJIFNBUzIwMDg+IHBvcnQgMHhjMDAwLTB4YzBmZiBt ZW0gMHhmYjQzYzAwMC0weGZiNDNmZmZmLDB4ZmI0NDAwMDAtMHhmYjQ3ZmZm ZiBpcnEgMTYgYXQgZGV2aWNlIDAuMCBvbiBwY2kyDQptcHMwOiBSZXNlcnZl ZCAweDQwMDAgYnl0ZXMgZm9yIHJpZCAweDE0IHR5cGUgMyBhdCAweGZiNDNj MDAwDQptcHMwOiBGaXJtd2FyZTogMDQuMDAuMDAuMDANCm1wczA6IElPQ0Nh cGFiaWxpdGllczogMTg1YzxTY3NpVGFza0Z1bGwsRGlhZ1RyYWNlLFNuYXBC dWYsRUVEUCxUcmFuc1JldHJ5LElSPg0KbXBzMDogYXR0ZW1wdGluZyB0byBh bGxvY2F0ZSAxIE1TSS1YIHZlY3RvcnMgKDE1IHN1cHBvcnRlZCkNCm1zaTog cm91dGluZyBNU0ktWCBJUlEgMjU2IHRvIGxvY2FsIEFQSUMgMCB2ZWN0b3Ig NDkNCm1wczA6IHVzaW5nIElSUSAyNTYgZm9yIE1TSS1YDQptcHMwOiBbTVBT QUZFXQ0KbXBzMDogW0lUSFJFQURdDQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFs PiBhdCBkZXZpY2UgOC4wIChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8 YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgOC4xIChubyBkcml2ZXIgYXR0 YWNoZWQpDQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgOC4y IChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFs PiBhdCBkZXZpY2UgOC4zIChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8 YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgMTYuMCAobm8gZHJpdmVyIGF0 dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDE2 LjEgKG5vIGRyaXZlciBhdHRhY2hlZCkNCmVoY2kwOiA8SW50ZWwgUENIIFVT QiAyLjAgY29udHJvbGxlciBVU0ItQj4gbWVtIDB4ZmIzZmMwMDAtMHhmYjNm YzNmZiBpcnEgMjEgYXQgZGV2aWNlIDI2LjAgb24gcGNpMA0KaW9hcGljMDog cm91dGluZyBpbnRwaW4gMjEgKFBDSSBJUlEgMjEpIHRvIGxhcGljIDAgdmVj dG9yIDUwDQplaGNpMDogW01QU0FGRV0NCmVoY2kwOiBbSVRIUkVBRF0NCnVz YnVzMDogRUhDSSB2ZXJzaW9uIDEuMA0KdXNidXMwOiA8SW50ZWwgUENIIFVT QiAyLjAgY29udHJvbGxlciBVU0ItQj4gb24gZWhjaTANCnBjaWIzOiA8QUNQ SSBQQ0ktUENJIGJyaWRnZT4gaXJxIDE3IGF0IGRldmljZSAyOC4wIG9uIHBj aTANCnBjaWIzOiAgIGRvbWFpbiAgICAgICAgICAgIDANCnBjaWIzOiAgIHNl Y29uZGFyeSBidXMgICAgIDMNCnBjaWIzOiAgIHN1Ym9yZGluYXRlIGJ1cyAg IDMNCnBjaWIzOiAgIEkvTyBkZWNvZGUgICAgICAgIDB4MC0weDANCnBjaWIz OiAgIG5vIHByZWZldGNoZWQgZGVjb2RlDQpwY2kzOiA8QUNQSSBQQ0kgYnVz PiBvbiBwY2liMw0KcGNpMzogZG9tYWluPTAsIHBoeXNpY2FsIGJ1cz0zDQpw Y2liNDogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGlycSAxNyBhdCBkZXZpY2Ug MjguNCBvbiBwY2kwDQpwY2liNDogICBkb21haW4gICAgICAgICAgICAwDQpw Y2liNDogICBzZWNvbmRhcnkgYnVzICAgICA0DQpwY2liNDogICBzdWJvcmRp bmF0ZSBidXMgICA0DQpwY2liNDogICBJL08gZGVjb2RlICAgICAgICAweGQw MDAtMHhkZmZmDQpwY2liNDogICBtZW1vcnkgZGVjb2RlICAgICAweGZiNTAw MDAwLTB4ZmI1ZmZmZmYNCnBjaWI0OiAgIG5vIHByZWZldGNoZWQgZGVjb2Rl DQpwY2k0OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liNA0KcGNpNDogZG9tYWlu PTAsIHBoeXNpY2FsIGJ1cz00DQpmb3VuZC0+CXZlbmRvcj0weDgwODYsIGRl dj0weDEwZDMsIHJldmlkPTB4MDANCglkb21haW49MCwgYnVzPTQsIHNsb3Q9 MCwgZnVuYz0wDQoJY2xhc3M9MDItMDAtMDAsIGhkcnR5cGU9MHgwMCwgbWZk ZXY9MA0KCWNtZHJlZz0weDAwMDcsIHN0YXRyZWc9MHgwMDEwLCBjYWNoZWxu c3o9OCAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4MDAgKDAgbnMpLCBtaW5nbnQ9 MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCWludHBpbj1hLCBp cnE9MTANCglwb3dlcnNwZWMgMiAgc3VwcG9ydHMgRDAgRDMgIGN1cnJlbnQg RDANCglNU0kgc3VwcG9ydHMgMSBtZXNzYWdlLCA2NCBiaXQNCglNU0ktWCBz dXBwb3J0cyA1IG1lc3NhZ2VzIGluIG1hcCAweDFjDQoJbWFwWzEwXTogdHlw ZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4ZmI1ZTAwMDAsIHNpemUgMTcs IGVuYWJsZWQNCnBjaWI0OiByZXF1ZXN0ZWQgbWVtb3J5IHJhbmdlIDB4ZmI1 ZTAwMDAtMHhmYjVmZmZmZjogZ29vZA0KCW1hcFsxOF06IHR5cGUgSS9PIFBv cnQsIHJhbmdlIDMyLCBiYXNlIDB4ZGMwMCwgc2l6ZSAgNSwgZW5hYmxlZA0K cGNpYjQ6IHJlcXVlc3RlZCBJL08gcmFuZ2UgMHhkYzAwLTB4ZGMxZjogaW4g cmFuZ2UNCgltYXBbMWNdOiB0eXBlIE1lbW9yeSwgcmFuZ2UgMzIsIGJhc2Ug MHhmYjVkYzAwMCwgc2l6ZSAxNCwgZW5hYmxlZA0KcGNpYjQ6IHJlcXVlc3Rl ZCBtZW1vcnkgcmFuZ2UgMHhmYjVkYzAwMC0weGZiNWRmZmZmOiBnb29kDQpw Y2liNDogbWF0Y2hlZCBlbnRyeSBmb3IgNC4wLklOVEENCnBjaWI0OiBzbG90 IDAgSU5UQSBoYXJkd2lyZWQgdG8gSVJRIDE2DQplbTA6IDxJbnRlbChSKSBQ Uk8vMTAwMCBOZXR3b3JrIENvbm5lY3Rpb24gNy4xLjk+IHBvcnQgMHhkYzAw LTB4ZGMxZiBtZW0gMHhmYjVlMDAwMC0weGZiNWZmZmZmLDB4ZmI1ZGMwMDAt MHhmYjVkZmZmZiBpcnEgMTYgYXQgZGV2aWNlIDAuMCBvbiBwY2k0DQplbTA6 IFJlc2VydmVkIDB4MjAwMDAgYnl0ZXMgZm9yIHJpZCAweDEwIHR5cGUgMyBh dCAweGZiNWUwMDAwDQplbTA6IFJlc2VydmVkIDB4NDAwMCBieXRlcyBmb3Ig cmlkIDB4MWMgdHlwZSAzIGF0IDB4ZmI1ZGMwMDANCmVtMDogYXR0ZW1wdGlu ZyB0byBhbGxvY2F0ZSAzIE1TSS1YIHZlY3RvcnMgKDUgc3VwcG9ydGVkKQ0K bXNpOiByb3V0aW5nIE1TSS1YIElSUSAyNTcgdG8gbG9jYWwgQVBJQyAwIHZl Y3RvciA1MQ0KbXNpOiByb3V0aW5nIE1TSS1YIElSUSAyNTggdG8gbG9jYWwg QVBJQyAwIHZlY3RvciA1Mg0KbXNpOiByb3V0aW5nIE1TSS1YIElSUSAyNTkg dG8gbG9jYWwgQVBJQyAwIHZlY3RvciA1Mw0KZW0wOiB1c2luZyBJUlFzIDI1 Ny0yNTkgZm9yIE1TSS1YDQplbTA6IFVzaW5nIE1TSVggaW50ZXJydXB0cyB3 aXRoIDMgdmVjdG9ycw0KZW0wOiBbTVBTQUZFXQ0KZW0wOiBbSVRIUkVBRF0N CmVtMDogW01QU0FGRV0NCmVtMDogW0lUSFJFQURdDQplbTA6IFtNUFNBRkVd DQplbTA6IFtJVEhSRUFEXQ0KZW0wOiBicGYgYXR0YWNoZWQNCmVtMDogRXRo ZXJuZXQgYWRkcmVzczogMDA6MjU6OTA6MDA6NjQ6NjgNCnBjaWI1OiA8QUNQ SSBQQ0ktUENJIGJyaWRnZT4gaXJxIDE2IGF0IGRldmljZSAyOC41IG9uIHBj aTANCnBjaWI1OiAgIGRvbWFpbiAgICAgICAgICAgIDANCnBjaWI1OiAgIHNl Y29uZGFyeSBidXMgICAgIDUNCnBjaWI1OiAgIHN1Ym9yZGluYXRlIGJ1cyAg IDUNCnBjaWI1OiAgIEkvTyBkZWNvZGUgICAgICAgIDB4ZTAwMC0weGVmZmYN CnBjaWI1OiAgIG1lbW9yeSBkZWNvZGUgICAgIDB4ZmI2MDAwMDAtMHhmYjZm ZmZmZg0KcGNpYjU6ICAgbm8gcHJlZmV0Y2hlZCBkZWNvZGUNCnBjaTU6IDxB Q1BJIFBDSSBidXM+IG9uIHBjaWI1DQpwY2k1OiBkb21haW49MCwgcGh5c2lj YWwgYnVzPTUNCmZvdW5kLT4JdmVuZG9yPTB4ODA4NiwgZGV2PTB4MTBkMywg cmV2aWQ9MHgwMA0KCWRvbWFpbj0wLCBidXM9NSwgc2xvdD0wLCBmdW5jPTAN CgljbGFzcz0wMi0wMC0wMCwgaGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21k cmVnPTB4MDAwNywgc3RhdHJlZz0weDAwMTAsIGNhY2hlbG5zej04IChkd29y ZHMpDQoJbGF0dGltZXI9MHgwMCAoMCBucyksIG1pbmdudD0weDAwICgwIG5z KSwgbWF4bGF0PTB4MDAgKDAgbnMpDQoJaW50cGluPWEsIGlycT0xMQ0KCXBv d2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMyAgY3VycmVudCBEMA0KCU1TSSBz dXBwb3J0cyAxIG1lc3NhZ2UsIDY0IGJpdA0KCU1TSS1YIHN1cHBvcnRzIDUg bWVzc2FnZXMgaW4gbWFwIDB4MWMNCgltYXBbMTBdOiB0eXBlIE1lbW9yeSwg cmFuZ2UgMzIsIGJhc2UgMHhmYjZlMDAwMCwgc2l6ZSAxNywgZW5hYmxlZA0K cGNpYjU6IHJlcXVlc3RlZCBtZW1vcnkgcmFuZ2UgMHhmYjZlMDAwMC0weGZi NmZmZmZmOiBnb29kDQoJbWFwWzE4XTogdHlwZSBJL08gUG9ydCwgcmFuZ2Ug MzIsIGJhc2UgMHhlYzAwLCBzaXplICA1LCBlbmFibGVkDQpwY2liNTogcmVx dWVzdGVkIEkvTyByYW5nZSAweGVjMDAtMHhlYzFmOiBpbiByYW5nZQ0KCW1h cFsxY106IHR5cGUgTWVtb3J5LCByYW5nZSAzMiwgYmFzZSAweGZiNmRjMDAw LCBzaXplIDE0LCBlbmFibGVkDQpwY2liNTogcmVxdWVzdGVkIG1lbW9yeSBy YW5nZSAweGZiNmRjMDAwLTB4ZmI2ZGZmZmY6IGdvb2QNCnBjaWI1OiBtYXRj aGVkIGVudHJ5IGZvciA1LjAuSU5UQQ0KcGNpYjU6IHNsb3QgMCBJTlRBIGhh cmR3aXJlZCB0byBJUlEgMTcNCmVtMTogPEludGVsKFIpIFBSTy8xMDAwIE5l dHdvcmsgQ29ubmVjdGlvbiA3LjEuOT4gcG9ydCAweGVjMDAtMHhlYzFmIG1l bSAweGZiNmUwMDAwLTB4ZmI2ZmZmZmYsMHhmYjZkYzAwMC0weGZiNmRmZmZm IGlycSAxNyBhdCBkZXZpY2UgMC4wIG9uIHBjaTUNCmVtMTogUmVzZXJ2ZWQg MHgyMDAwMCBieXRlcyBmb3IgcmlkIDB4MTAgdHlwZSAzIGF0IDB4ZmI2ZTAw MDANCmVtMTogUmVzZXJ2ZWQgMHg0MDAwIGJ5dGVzIGZvciByaWQgMHgxYyB0 eXBlIDMgYXQgMHhmYjZkYzAwMA0KZW0xOiBhdHRlbXB0aW5nIHRvIGFsbG9j YXRlIDMgTVNJLVggdmVjdG9ycyAoNSBzdXBwb3J0ZWQpDQptc2k6IHJvdXRp bmcgTVNJLVggSVJRIDI2MCB0byBsb2NhbCBBUElDIDAgdmVjdG9yIDU0DQpt c2k6IHJvdXRpbmcgTVNJLVggSVJRIDI2MSB0byBsb2NhbCBBUElDIDAgdmVj dG9yIDU1DQptc2k6IHJvdXRpbmcgTVNJLVggSVJRIDI2MiB0byBsb2NhbCBB UElDIDAgdmVjdG9yIDU2DQplbTE6IHVzaW5nIElSUXMgMjYwLTI2MiBmb3Ig TVNJLVgNCmVtMTogVXNpbmcgTVNJWCBpbnRlcnJ1cHRzIHdpdGggMyB2ZWN0 b3JzDQplbTE6IFtNUFNBRkVdDQplbTE6IFtJVEhSRUFEXQ0KZW0xOiBbTVBT QUZFXQ0KZW0xOiBbSVRIUkVBRF0NCmVtMTogW01QU0FGRV0NCmVtMTogW0lU SFJFQURdDQplbTE6IGJwZiBhdHRhY2hlZA0KZW0xOiBFdGhlcm5ldCBhZGRy ZXNzOiAwMDoyNTo5MDowMDo2NDo2OQ0KZWhjaTE6IDxJbnRlbCBQQ0ggVVNC IDIuMCBjb250cm9sbGVyIFVTQi1BPiBtZW0gMHhmYjNmYTAwMC0weGZiM2Zh M2ZmIGlycSAyMyBhdCBkZXZpY2UgMjkuMCBvbiBwY2kwDQppb2FwaWMwOiBy b3V0aW5nIGludHBpbiAyMyAoUENJIElSUSAyMykgdG8gbGFwaWMgMCB2ZWN0 b3IgNTcNCmVoY2kxOiBbTVBTQUZFXQ0KZWhjaTE6IFtJVEhSRUFEXQ0KdXNi dXMxOiBFSENJIHZlcnNpb24gMS4wDQp1c2J1czE6IDxJbnRlbCBQQ0ggVVNC IDIuMCBjb250cm9sbGVyIFVTQi1BPiBvbiBlaGNpMQ0KcGNpYjY6IDxBQ1BJ IFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMzAuMCBvbiBwY2kwDQpwY2li NjogICBkb21haW4gICAgICAgICAgICAwDQpwY2liNjogICBzZWNvbmRhcnkg YnVzICAgICA2DQpwY2liNjogICBzdWJvcmRpbmF0ZSBidXMgICA2DQpwY2li NjogICBJL08gZGVjb2RlICAgICAgICAweGYwMDAtMHhmZmYNCnBjaWI2OiAg IG1lbW9yeSBkZWNvZGUgICAgIDB4ZmI3MDAwMDAtMHhmYmZmZmZmZg0KcGNp YjY6ICAgcHJlZmV0Y2hlZCBkZWNvZGUgMHhmYTAwMDAwMC0weGZhZmZmZmZm DQpwY2liNjogICBTdWJ0cmFjdGl2ZWx5IGRlY29kZWQgYnJpZGdlLg0KcGNp NjogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjYNCnBjaTY6IGRvbWFpbj0wLCBw aHlzaWNhbCBidXM9Ng0KZm91bmQtPgl2ZW5kb3I9MHgxMDJiLCBkZXY9MHgw NTMyLCByZXZpZD0weDBhDQoJZG9tYWluPTAsIGJ1cz02LCBzbG90PTMsIGZ1 bmM9MA0KCWNsYXNzPTAzLTAwLTAwLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTAN CgljbWRyZWc9MHgwMDA3LCBzdGF0cmVnPTB4MDI5MCwgY2FjaGVsbnN6PTgg KGR3b3JkcykNCglsYXR0aW1lcj0weDQwICgxOTIwIG5zKSwgbWluZ250PTB4 MTAgKDQwMDAgbnMpLCBtYXhsYXQ9MHgyMCAoODAwMCBucykNCglpbnRwaW49 YSwgaXJxPTE1DQoJcG93ZXJzcGVjIDEgIHN1cHBvcnRzIEQwIEQzICBjdXJy ZW50IEQwDQoJbWFwWzEwXTogdHlwZSBQcmVmZXRjaGFibGUgTWVtb3J5LCBy YW5nZSAzMiwgYmFzZSAweGZhMDAwMDAwLCBzaXplIDI0LCBlbmFibGVkDQpw Y2liNjogcmVxdWVzdGVkIG1lbW9yeSByYW5nZSAweGZhMDAwMDAwLTB4ZmFm ZmZmZmY6IGdvb2QNCgltYXBbMTRdOiB0eXBlIE1lbW9yeSwgcmFuZ2UgMzIs IGJhc2UgMHhmYjdmYzAwMCwgc2l6ZSAxNCwgZW5hYmxlZA0KcGNpYjY6IHJl cXVlc3RlZCBtZW1vcnkgcmFuZ2UgMHhmYjdmYzAwMC0weGZiN2ZmZmZmOiBn b29kDQoJbWFwWzE4XTogdHlwZSBNZW1vcnksIHJhbmdlIDMyLCBiYXNlIDB4 ZmI4MDAwMDAsIHNpemUgMjMsIGVuYWJsZWQNCnBjaWI2OiByZXF1ZXN0ZWQg bWVtb3J5IHJhbmdlIDB4ZmI4MDAwMDAtMHhmYmZmZmZmZjogZ29vZA0KcGNp YjY6IG1hdGNoZWQgZW50cnkgZm9yIDYuMy5JTlRBDQpwY2liNjogc2xvdCAz IElOVEEgaGFyZHdpcmVkIHRvIElSUSAyMw0KdmdhcGNpMDogPFZHQS1jb21w YXRpYmxlIGRpc3BsYXk+IG1lbSAweGZhMDAwMDAwLTB4ZmFmZmZmZmYsMHhm YjdmYzAwMC0weGZiN2ZmZmZmLDB4ZmI4MDAwMDAtMHhmYmZmZmZmZiBpcnEg MjMgYXQgZGV2aWNlIDMuMCBvbiBwY2k2DQppc2FiMDogPFBDSS1JU0EgYnJp ZGdlPiBhdCBkZXZpY2UgMzEuMCBvbiBwY2kwDQppc2EwOiA8SVNBIGJ1cz4g b24gaXNhYjANCmFoY2kwOiA8SW50ZWwgNSBTZXJpZXMvMzQwMCBTZXJpZXMg QUhDSSBTQVRBIGNvbnRyb2xsZXI+IHBvcnQgMHhiNDAwLTB4YjQwNywweGJj MDAtMHhiYzAzLDB4Yjg4MC0weGI4ODcsMHhiODAwLTB4YjgwMywweGI0ODAt MHhiNDlmIG1lbSAweGZiM2Y4MDAwLTB4ZmIzZjg3ZmYgaXJxIDE5IGF0IGRl dmljZSAzMS4yIG9uIHBjaTANCmFoY2kwOiBSZXNlcnZlZCAweDgwMCBieXRl cyBmb3IgcmlkIDB4MjQgdHlwZSAzIGF0IDB4ZmIzZjgwMDANCmFoY2kwOiBh dHRlbXB0aW5nIHRvIGFsbG9jYXRlIDEgTVNJIHZlY3RvcnMgKDEgc3VwcG9y dGVkKQ0KbXNpOiByb3V0aW5nIE1TSSBJUlEgMjYzIHRvIGxvY2FsIEFQSUMg MCB2ZWN0b3IgNTgNCmFoY2kwOiB1c2luZyBJUlEgMjYzIGZvciBNU0kNCmFo Y2kwOiBbTVBTQUZFXQ0KYWhjaTA6IFtJVEhSRUFEXQ0KYWhjaTA6IEFIQ0kg djEuMzAgd2l0aCA2IDNHYnBzIHBvcnRzLCBQb3J0IE11bHRpcGxpZXIgbm90 IHN1cHBvcnRlZA0KYWhjaTA6IENhcHM6IDY0Yml0IE5DUSBTTlRGIFNTIEFM UCBBTCBDTE8gM0dicHMgUE1EIFNTQyBQU0MgMzJjbWQgRU0gZVNBVEEgNnBv cnRzDQphaGNpMDogQ2FwczI6IEFQU1QNCmFoY2kwOiBFTSBDYXBzOiBBTEhE IFhNVCBTTUIgTEVEDQphaGNpY2gwOiA8QUhDSSBjaGFubmVsPiBhdCBjaGFu bmVsIDAgb24gYWhjaTANCmFoY2ljaDA6IFtNUFNBRkVdDQphaGNpY2gwOiBb SVRIUkVBRF0NCmFoY2ljaDA6IENhcHM6IEhQQ1ANCmFoY2ljaDE6IDxBSENJ IGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhaGNpMA0KYWhjaWNoMTogW01Q U0FGRV0NCmFoY2ljaDE6IFtJVEhSRUFEXQ0KYWhjaWNoMTogQ2FwczogSFBD UA0KYWhjaWNoMjogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAyIG9uIGFo Y2kwDQphaGNpY2gyOiBbTVBTQUZFXQ0KYWhjaWNoMjogW0lUSFJFQURdDQph aGNpY2gyOiBDYXBzOiBIUENQDQphaGNpY2gzOiA8QUhDSSBjaGFubmVsPiBh dCBjaGFubmVsIDMgb24gYWhjaTANCmFoY2ljaDM6IFtNUFNBRkVdDQphaGNp Y2gzOiBbSVRIUkVBRF0NCmFoY2ljaDM6IENhcHM6IEhQQ1ANCmFoY2ljaDQ6 IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgNCBvbiBhaGNpMA0KYWhjaWNo NDogW01QU0FGRV0NCmFoY2ljaDQ6IFtJVEhSRUFEXQ0KYWhjaWNoNDogQ2Fw czogSFBDUA0KYWhjaWNoNTogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCA1 IG9uIGFoY2kwDQphaGNpY2g1OiBbTVBTQUZFXQ0KYWhjaWNoNTogW0lUSFJF QURdDQphaGNpY2g1OiBDYXBzOiBIUENQDQpwY2kwOiA8c2VyaWFsIGJ1cywg U01CdXM+IGF0IGRldmljZSAzMS4zIChubyBkcml2ZXIgYXR0YWNoZWQpDQph Y3BpX2J1dHRvbjA6IDxQb3dlciBCdXR0b24+IG9uIGFjcGkwDQphdHJ0YzA6 IDxBVCByZWFsdGltZSBjbG9jaz4gcG9ydCAweDcwLTB4NzEgaXJxIDggb24g YWNwaTANCmF0cnRjMDogcmVnaXN0ZXJlZCBhcyBhIHRpbWUtb2YtZGF5IGNs b2NrIChyZXNvbHV0aW9uIDEwMDAwMDB1cykNCnVhcnQwOiA8MTY1NTAgb3Ig Y29tcGF0aWJsZT4gcG9ydCAweDNmOC0weDNmZiBpcnEgNCBmbGFncyAweDEw IG9uIGFjcGkwDQppb2FwaWMwOiByb3V0aW5nIGludHBpbiA0IChJU0EgSVJR IDQpIHRvIGxhcGljIDAgdmVjdG9yIDU5DQp1YXJ0MDogW0ZJTFRFUl0NCnVh cnQwOiBmYXN0IGludGVycnVwdA0KdWFydDA6IGNvbnNvbGUgKDExNTIwMCxu LDgsMSkNCnVhcnQyOiA8MTY1NTAgb3IgY29tcGF0aWJsZT4gcG9ydCAweDNl OC0weDNlZiBpcnEgNSBvbiBhY3BpMA0KaW9hcGljMDogcm91dGluZyBpbnRw aW4gNSAoSVNBIElSUSA1KSB0byBsYXBpYyAwIHZlY3RvciA2MA0KdWFydDI6 IFtGSUxURVJdDQp1YXJ0MjogZmFzdCBpbnRlcnJ1cHQNCmF0a2JkYzA6IDxL ZXlib2FyZCBjb250cm9sbGVyIChpODA0Mik+IHBvcnQgMHg2MCwweDY0IGly cSAxIG9uIGFjcGkwDQphdGtiZDA6IDxBVCBLZXlib2FyZD4gaXJxIDEgb24g YXRrYmRjMA0KYXRrYmQ6IHRoZSBjdXJyZW50IGtiZCBjb250cm9sbGVyIGNv bW1hbmQgYnl0ZSAwMDY1DQphdGtiZDoga2V5Ym9hcmQgSUQgMHg0MWFiICgy KQ0Ka2JkMCBhdCBhdGtiZDANCmtiZDA6IGF0a2JkMCwgQVQgMTAxLzEwMiAo MiksIGNvbmZpZzoweDAsIGZsYWdzOjB4M2QwMDAwDQppb2FwaWMwOiByb3V0 aW5nIGludHBpbiAxIChJU0EgSVJRIDEpIHRvIGxhcGljIDAgdmVjdG9yIDYx DQphdGtiZDA6IFtHSUFOVC1MT0NLRURdDQphdGtiZDA6IFtJVEhSRUFEXQ0K YWNwaV9ocGV0MDogPEhpZ2ggUHJlY2lzaW9uIEV2ZW50IFRpbWVyPiBpb21l bSAweGZlZDAwMDAwLTB4ZmVkMDAzZmYgb24gYWNwaTANCmFjcGlfaHBldDA6 IHZlbmQ6IDB4ODA4NiByZXY6IDB4MSBudW06IDggaHo6IDE0MzE4MTgwIG9w dHM6IGxlZ2FjeV9yb3V0ZSA2NC1iaXQNClRpbWVjb3VudGVyICJIUEVUIiBm cmVxdWVuY3kgMTQzMTgxODAgSHogcXVhbGl0eSA5MDANCmFjcGkwOiB3YWtl dXAgY29kZSB2YSAweGZmZmZmZjg0ODQwZTgwMDAgcGEgMHg0MDAwDQppc2Ff cHJvYmVfY2hpbGRyZW46IGRpc2FibGluZyBQblAgZGV2aWNlcw0KYXRrYmRj OiBhdGtiZGMwIGFscmVhZHkgZXhpc3RzOyBza2lwcGluZyBpdA0KYXRydGM6 IGF0cnRjMCBhbHJlYWR5IGV4aXN0czsgc2tpcHBpbmcgaXQNCnNjOiBzYzAg YWxyZWFkeSBleGlzdHM7IHNraXBwaW5nIGl0DQp1YXJ0OiB1YXJ0MCBhbHJl YWR5IGV4aXN0czsgc2tpcHBpbmcgaXQNCmlzYV9wcm9iZV9jaGlsZHJlbjog cHJvYmluZyBub24tUG5QIGRldmljZXMNCm9ybTA6IDxJU0EgT3B0aW9uIFJP TT4gYXQgaW9tZW0gMHhjMDAwMC0weGM3ZmZmIG9uIGlzYTANCnNjMDogPFN5 c3RlbSBjb25zb2xlPiBhdCBmbGFncyAweDEwMCBvbiBpc2EwDQpzYzA6IFZH QSA8MTYgdmlydHVhbCBjb25zb2xlcywgZmxhZ3M9MHgzMDA+DQpzYzA6IGZi MCwga2JkMSwgdGVybWluYWwgZW11bGF0b3I6IHNjdGVrZW4gKHRla2VuIHRl cm1pbmFsKQ0KdmdhMDogPEdlbmVyaWMgSVNBIFZHQT4gYXQgcG9ydCAweDNj MC0weDNkZiBpb21lbSAweGEwMDAwLTB4YmZmZmYgb24gaXNhMA0KZmRjMCBm YWlsZWQgdG8gcHJvYmUgYXQgcG9ydCAweDNmMCBpcnEgNiBkcnEgMiBvbiBp c2EwDQpwcGMwIGZhaWxlZCB0byBwcm9iZSBhdCBpcnEgNyBvbiBpc2EwDQp1 YXJ0MTogPG5zODI1MD4gZmFpbGVkIHRvIHByb2JlIGF0IHBvcnQgMHgyZjgt MHgyZmYgaXJxIDMgb24gaXNhMA0KaXNhX3Byb2JlX2NoaWxkcmVuOiBwcm9i aW5nIFBuUCBkZXZpY2VzDQpjb3JldGVtcDA6IDxDUFUgT24tRGllIFRoZXJt YWwgU2Vuc29ycz4gb24gY3B1MA0KY29yZXRlbXAwOiBTZXR0aW5nIFRqTWF4 PTk5DQplc3QwOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250 cm9sPiBvbiBjcHUwDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNywgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDIyNjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNiwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDIxMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNSwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDIwMDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNCwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDE4NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMywgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDE3MzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMiwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDE2MDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMSwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDE0NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMCwgMTgpDQplc3QwOiBDYW4ndCBjaGVjayBmcmVxIDEzMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QwOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICg5LCAxOCkNCmVzdDA6IENhbid0IGNoZWNrIGZyZXEgMTIwMCwgaXQgbWF5 IGJlIGludmFsaWQNCnA0dGNjMDogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBD b250cm9sPiBvbiBjcHUwDQpjb3JldGVtcDE6IDxDUFUgT24tRGllIFRoZXJt YWwgU2Vuc29ycz4gb24gY3B1MQ0KY29yZXRlbXAxOiBTZXR0aW5nIFRqTWF4 PTk5DQplc3QxOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250 cm9sPiBvbiBjcHUxDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNywgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDIyNjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNiwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDIxMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNSwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDIwMDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNCwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDE4NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMywgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDE3MzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMiwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDE2MDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMSwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDE0NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMCwgMTgpDQplc3QxOiBDYW4ndCBjaGVjayBmcmVxIDEzMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QxOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICg5LCAxOCkNCmVzdDE6IENhbid0IGNoZWNrIGZyZXEgMTIwMCwgaXQgbWF5 IGJlIGludmFsaWQNCnA0dGNjMTogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBD b250cm9sPiBvbiBjcHUxDQpjb3JldGVtcDI6IDxDUFUgT24tRGllIFRoZXJt YWwgU2Vuc29ycz4gb24gY3B1Mg0KY29yZXRlbXAyOiBTZXR0aW5nIFRqTWF4 PTk5DQplc3QyOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250 cm9sPiBvbiBjcHUyDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNywgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDIyNjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNiwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDIxMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNSwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDIwMDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNCwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDE4NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMywgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDE3MzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMiwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDE2MDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMSwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDE0NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMCwgMTgpDQplc3QyOiBDYW4ndCBjaGVjayBmcmVxIDEzMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QyOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICg5LCAxOCkNCmVzdDI6IENhbid0IGNoZWNrIGZyZXEgMTIwMCwgaXQgbWF5 IGJlIGludmFsaWQNCnA0dGNjMjogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBD b250cm9sPiBvbiBjcHUyDQpjb3JldGVtcDM6IDxDUFUgT24tRGllIFRoZXJt YWwgU2Vuc29ycz4gb24gY3B1Mw0KY29yZXRlbXAzOiBTZXR0aW5nIFRqTWF4 PTk5DQplc3QzOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250 cm9sPiBvbiBjcHUzDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNywgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDIyNjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNiwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDIxMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNSwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDIwMDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxNCwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDE4NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMywgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDE3MzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMiwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDE2MDAsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMSwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDE0NjcsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICgxMCwgMTgpDQplc3QzOiBDYW4ndCBjaGVjayBmcmVxIDEzMzMsIGl0IG1h eSBiZSBpbnZhbGlkDQplc3QzOiBJbnZhbGlkIGlkMTYgKHNldCwgY3VyKSA9 ICg5LCAxOCkNCmVzdDM6IENhbid0IGNoZWNrIGZyZXEgMTIwMCwgaXQgbWF5 IGJlIGludmFsaWQNCnA0dGNjMzogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBD b250cm9sPiBvbiBjcHUzDQpEZXZpY2UgY29uZmlndXJhdGlvbiBmaW5pc2hl ZC4NClpGUyBmaWxlc3lzdGVtIHZlcnNpb24gNQ0KWkZTIHN0b3JhZ2UgcG9v bCB2ZXJzaW9uIDI4DQpsYXBpYzogRGl2aXNvciAyLCBGcmVxdWVuY3kgNjY2 NjcxMzUgSHoNClRpbWVjb3VudGVyICJUU0MiIGZyZXF1ZW5jeSAyNDAwMDE3 MTU2IEh6IHF1YWxpdHkgLTEwMA0KVGltZWNvdW50ZXJzIHRpY2sgZXZlcnkg MS4wMDAgbXNlYw0KdmxhbjogaW5pdGlhbGl6ZWQsIHVzaW5nIGhhc2ggdGFi bGVzIHdpdGggY2hhaW5pbmcNCmxvMDogYnBmIGF0dGFjaGVkDQp1c2J1czA6 IDQ4ME1icHMgSGlnaCBTcGVlZCBVU0IgdjIuMA0KdXNidXMxOiA0ODBNYnBz IEhpZ2ggU3BlZWQgVVNCIHYyLjANCnVnZW4wLjE6IDxJbnRlbD4gYXQgdXNi dXMwDQp1aHViMDogPEludGVsIEVIQ0kgcm9vdCBIVUIsIGNsYXNzIDkvMCwg cmV2IDIuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2J1czANCnVnZW4xLjE6IDxJ bnRlbD4gYXQgdXNidXMxDQp1aHViMTogPEludGVsIEVIQ0kgcm9vdCBIVUIs IGNsYXNzIDkvMCwgcmV2IDIuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2J1czEN CmFoY2ljaDA6IEFIQ0kgcmVzZXQuLi4NCmFoY2ljaDA6IFNBVEEgY29ubmVj dCB0aW1lb3V0IHN0YXR1cz0wMDAwMDAwMA0KYWhjaWNoMDogQUhDSSByZXNl dDogZGV2aWNlIG5vdCBmb3VuZA0KYWhjaWNoMTogQUhDSSByZXNldC4uLg0K YWhjaWNoMTogU0FUQSBjb25uZWN0IHRpbWVvdXQgc3RhdHVzPTAwMDAwMDAw DQphaGNpY2gxOiBBSENJIHJlc2V0OiBkZXZpY2Ugbm90IGZvdW5kDQphaGNp Y2gyOiBBSENJIHJlc2V0Li4uDQphaGNpY2gyOiBTQVRBIGNvbm5lY3QgdGlt ZT0xMDB1cyBzdGF0dXM9MDAwMDAxMjMNCmFoY2ljaDI6IEFIQ0kgcmVzZXQ6 IGRldmljZSBmb3VuZA0KYWhjaWNoMzogQUhDSSByZXNldC4uLg0KYWhjaWNo MjogQUhDSSByZXNldDogZGV2aWNlIHJlYWR5IGFmdGVyIDEwMG1zDQooYXBy b2JlMDphaGNpY2gyOjA6MDowKTogU0lHTkFUVVJFOiAwMDAwDQphaGNpY2gz OiBTQVRBIGNvbm5lY3QgdGltZW91dCBzdGF0dXM9MDAwMDAwMDANCmFoY2lj aDM6IEFIQ0kgcmVzZXQ6IGRldmljZSBub3QgZm91bmQNCmFoY2ljaDQ6IEFI Q0kgcmVzZXQuLi4NCmFoY2ljaDQ6IFNBVEEgY29ubmVjdCB0aW1lb3V0IHN0 YXR1cz0wMDAwMDAwMA0KYWhjaWNoNDogQUhDSSByZXNldDogZGV2aWNlIG5v dCBmb3VuZA0KYWhjaWNoNTogQUhDSSByZXNldC4uLg0KYWhjaWNoNTogU0FU QSBjb25uZWN0IHRpbWVvdXQgc3RhdHVzPTAwMDAwMDAwDQphaGNpY2g1OiBB SENJIHJlc2V0OiBkZXZpY2Ugbm90IGZvdW5kDQp1aHViMDogMiBwb3J0cyB3 aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCnVodWIxOiAyIHBvcnRz IHdpdGggMiByZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KdWdlbjAuMjogPHZl bmRvciAweDgwODc+IGF0IHVzYnVzMA0KdWh1YjI6IDx2ZW5kb3IgMHg4MDg3 IHByb2R1Y3QgMHgwMDIwLCBjbGFzcyA5LzAsIHJldiAyLjAwLzAuMDAsIGFk ZHIgMj4gb24gdXNidXMwDQp1Z2VuMS4yOiA8dmVuZG9yIDB4ODA4Nz4gYXQg dXNidXMxDQp1aHViMzogPHZlbmRvciAweDgwODcgcHJvZHVjdCAweDAwMjAs IGNsYXNzIDkvMCwgcmV2IDIuMDAvMC4wMCwgYWRkciAyPiBvbiB1c2J1czEN CnVodWIyOiA2IHBvcnRzIHdpdGggNiByZW1vdmFibGUsIHNlbGYgcG93ZXJl ZA0KdWh1YjM6IDYgcG9ydHMgd2l0aCA2IHJlbW92YWJsZSwgc2VsZiBwb3dl cmVkDQp1Z2VuMC4zOiA8V2luYm9uZCBFbGVjdHJvbmljcyBDb3JwPiBhdCB1 c2J1czANCnVrYmQwOiA8V2luYm9uZCBFbGVjdHJvbmljcyBDb3JwIEhlcm1v biBVU0IgaGlkbW91c2UgRGV2aWNlLCBjbGFzcyAwLzAsIHJldiAxLjEwLzAu MDEsIGFkZHIgMz4gb24gdXNidXMwDQprYmQyIGF0IHVrYmQwDQprYmQyOiB1 a2JkMCwgZ2VuZXJpYyAoMCksIGNvbmZpZzoweDAsIGZsYWdzOjB4M2QwMDAw DQpwYXNzMCBhdCBhaGNpY2gyIGJ1cyAwIHNjYnVzMyB0YXJnZXQgMCBsdW4g MA0KcGFzczA6IDxTQU1TVU5HIEhEMjA0VUkgMUFRMTAwMDE+IEFUQS04IFNB VEEgMi54IGRldmljZQ0KcGFzczA6IFNlcmlhbCBOdW1iZXIgUzJIN0oxQ0Iy MjAzMDENCnBhc3MwOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMgKFNBVEEgMi54 LCBVRE1BNiwgUElPIDgxOTJieXRlcykNCmFkYTAgYXQgYWhjaWNoMiBidXMg MCBzY2J1czMgdGFyZ2V0IDAgbHVuIDANCmFkYTA6IDxTQU1TVU5HIEhEMjA0 VUkgMUFRMTAwMDE+IEFUQS04IFNBVEEgMi54IGRldmljZQ0KYWRhMDogU2Vy aWFsIE51bWJlciBTMkg3SjFDQjIyMDMwMQ0KYWRhMDogMzAwLjAwME1CL3Mg dHJhbnNmZXJzIChTQVRBIDIueCwgVURNQTYsIEdFT006IG5ldyBkaXNrIGFk YTANClBJTyA4MTkyYnl0ZXMpDQphZGEwOiAxOTA3NzI5TUIgKDM5MDcwMjkx NjggNTEyIGJ5dGUgc2VjdG9yczogMTZIIDYzUy9UIDE2MzgzQykNCkV4cGFu ZGVyIGZvdW5kIGF0IGVuY2xvc3VyZSAyDQptcHMwOiBGb3VuZCBkZXZpY2Ug PDE5MDI8U21wVGFyZyxEaXJlY3QsTHNpRGV2PixFZGdlIEV4cGFuZGVyPiA8 Ni4wR2Jwcz4gPDB4MDAwOT4gPDIvMD4NCm1wczA6IEZvdW5kIGRldmljZSA8 ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdicHM+IDwweDAwMGE+IDwy LzA+DQptcHMwOiBGb3VuZCBkZXZpY2UgPDgxPFNhdGFEZXY+LEVuZCBEZXZp Y2U+IDwzLjBHYnBzPiA8MHgwMDBiPiA8Mi8xPg0KbXBzMDogRm91bmQgZGV2 aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8My4wR2Jwcz4gPDB4MDAw Yz4gPDIvMj4NCm1wczA6IEZvdW5kIGRldmljZSA8ODE8U2F0YURldj4sRW5k IERldmljZT4gPDMuMEdicHM+IDwweDAwMGQ+IDwyLzM+DQptcHMwOiBGb3Vu ZCBkZXZpY2UgPDgxPFNhdGFEZXY+LEVuZCBEZXZpY2U+IDwzLjBHYnBzPiA8 MHgwMDBlPiA8Mi80Pg0KbXBzMDogRm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2 PixFbmQgRGV2aWNlPiA8My4wR2Jwcz4gPDB4MDAwZj4gPDIvNT4NCm1wczA6 IEZvdW5kIGRldmljZSA8ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdi cHM+IDwweDAwMTA+IDwyLzY+DQptcHMwOiBGb3VuZCBkZXZpY2UgPDgxPFNh dGFEZXY+LEVuZCBEZXZpY2U+IDwzLjBHYnBzPiA8MHgwMDExPiA8Mi83Pg0K bXBzMDogRm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8 My4wR2Jwcz4gPDB4MDAxMj4gPDIvOD4NCm1wczA6IEZvdW5kIGRldmljZSA8 ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdicHM+IDwweDAwMTM+IDwy Lzk+DQoocHJvYmUwOm1wczA6MDowOjApOiBEb3duIHJldmluZyBQcm90b2Nv bCBWZXJzaW9uIGZyb20gNCB0byAwPw0KKHByb2JlMTptcHMwOjA6MTowKTog RG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMD8NCihw cm9iZTI6bXBzMDowOjI6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNp b24gZnJvbSA0IHRvIDA/DQoocHJvYmUzOm1wczA6MDozOjApOiBEb3duIHJl dmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAwPw0KKHByb2JlNDpt cHMwOjA6NDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9t IDQgdG8gMD8NCihwcm9iZTU6bXBzMDowOjU6MCk6IERvd24gcmV2aW5nIFBy b3RvY29sIFZlcnNpb24gZnJvbSA0IHRvIDA/DQoocHJvYmU2Om1wczA6MDo2 OjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAw Pw0KKHByb2JlNzptcHMwOjA6NzowKTogRG93biByZXZpbmcgUHJvdG9jb2wg VmVyc2lvbiBmcm9tIDQgdG8gMD8NCihwcm9iZTg6bXBzMDowOjg6MCk6IERv d24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRvIDA/DQoocHJv YmU5Om1wczA6MDo5OjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJzaW9u IGZyb20gNCB0byAwPw0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IERvd24g cmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRvIDI/DQoocHJvYmUy MDA6bXBzMDowOjIwMDowKTogRXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9y DQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRG93biByZXZpbmcgUHJvdG9j b2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTIwMDptcHMwOjA6MjAw OjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTIwMDpt cHMwOjA6MjAwOjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJzaW9uIGZy b20gNCB0byAyPw0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IEVycm9yIDIy LCBVbnJldHJ5YWJsZSBlcnJvcg0KcGFzczEgYXQgbXBzMCBidXMgMCBzY2J1 czAgdGFyZ2V0IDUgbHVuIDANCnBhc3MxOiA8QVRBIFdEQyBXRDIwMDNGWVlT LTAgMUQwMT4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0K cGFzczE6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNjMwMzMNCnBh c3MxOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCnBhc3MxOiBDb21tYW5kIFF1 ZXVlaW5nIGVuYWJsZWQNCnBhc3MyIGF0IG1wczAgYnVzIDAgc2NidXMwIHRh cmdldCA3IGx1biAwDQpwYXNzMjogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFE MDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCnBhc3My OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAwNjExNDE1DQpwYXNzMjog MzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzMjogQ29tbWFuZCBRdWV1ZWlu ZyBlbmFibGVkDQpwYXNzMyBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJnZXQg MSBsdW4gMA0KcGFzczM6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBG aXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNzMzogU2Vy aWFsIE51bWJlciAgICAgIFdELVdNQVkwMDU3NzY2NQ0KcGFzczM6IDMwMC4w MDBNQi9zIHRyYW5zZmVycw0KcGFzczM6IENvbW1hbmQgUXVldWVpbmcgZW5h YmxlZA0KcGFzczQgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDMgbHVu IDANCnBhc3M0OiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4ZWQg RGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KcGFzczQ6IFNlcmlhbCBO dW1iZXIgICAgICBXRC1XTUFZMDA1NDg0NzgNCnBhc3M0OiAzMDAuMDAwTUIv cyB0cmFuc2ZlcnMNCnBhc3M0OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQN CnBhc3M1IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAwIGx1biAwDQpw YXNzNTogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZpeGVkIERpcmVj dCBBY2Nlc3MgU0NTSS01IGRldmljZSANCnBhc3M1OiBTZXJpYWwgTnVtYmVy ICAgICAgV0QtV01BWTAwNTYzNTkwDQpwYXNzNTogMzAwLjAwME1CL3MgdHJh bnNmZXJzDQpwYXNzNTogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkDQpwYXNz NiBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJnZXQgOSBsdW4gMA0KcGFzczY6 IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNj ZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNzNjogU2VyaWFsIE51bWJlciAgICAg IFdELVdNQVkwMDUyNTczOQ0KcGFzczY6IDMwMC4wMDBNQi9zIHRyYW5zZmVy cw0KcGFzczY6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KcGFzczcgYXQg bXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDYgbHVuIDANCnBhc3M3OiA8QVRB IFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4ZWQgRGlyZWN0IEFjY2VzcyBT Q1NJLTUgZGV2aWNlIA0KcGFzczc6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1X TUFZMDA2MDIzMTcNCnBhc3M3OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCnBh c3M3OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQNCnBhc3M4IGF0IG1wczAg YnVzIDAgc2NidXMwIHRhcmdldCA0IGx1biAwDQpwYXNzODogPEFUQSBXREMg V0QyMDAzRllZUy0wIDFEMDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01 IGRldmljZSANCnBhc3M4OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAw NjQ0NDAxDQpwYXNzODogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzODog Q29tbWFuZCBRdWV1ZWluZyBlbmFibGVkDQpwYXNzOSBhdCBtcHMwIGJ1cyAw IHNjYnVzMCB0YXJnZXQgOCBsdW4gMA0KcGFzczk6IDxBVEEgV0RDIFdEMjAw M0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZp Y2UgDQpwYXNzOTogU2VyaWFsIE51bWJlciAgICAgIFdELVdNQVkwMDQzNzk3 OA0KcGFzczk6IDMwMC4wMDBNQi9zIHRyYW5zZmVycw0KcGFzczk6IENvbW1h bmQgUXVldWVpbmcgZW5hYmxlZA0KcGFzczEwIGF0IG1wczAgYnVzIDAgc2Ni dXMwIHRhcmdldCAyIGx1biAwDQpwYXNzMTA6IDxBVEEgV0RDIFdEMjAwM0ZZ WVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2Ug DQpwYXNzMTA6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA2NTM2MzEN CnBhc3MxMDogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzMTA6IENvbW1h bmQgUXVldWVpbmcgZW5hYmxlZA0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6 IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRvIDI/DQoo cHJvYmUyMDA6bXBzMDowOjIwMDowKTogRXJyb3IgMjIsIFVucmV0cnlhYmxl IGVycm9yDQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRG93biByZXZpbmcg UHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTIwMDptcHMw OjA6MjAwOjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9i ZTIwMDptcHMwOjA6MjAwOjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJz aW9uIGZyb20gNCB0byAyPw0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IEVy cm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2JlMjAwOm1wczA6MDoy MDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRv IDI/DQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRXJyb3IgMjIsIFVucmV0 cnlhYmxlIGVycm9yDQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRG93biBy ZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTIw MDptcHMwOjA6MjAwOjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUgZXJyb3IN Cihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBEb3duIHJldmluZyBQcm90b2Nv bCBWZXJzaW9uIGZyb20gNCB0byAyPw0KKHByb2JlMjAwOm1wczA6MDoyMDA6 MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2JlMjAwOm1w czA6MDoyMDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJv bSA0IHRvIDI/DQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRXJyb3IgMjIs IFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTog RG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihw cm9iZTIwMDptcHMwOjA6MjAwOjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUg ZXJyb3INCkdFT006IG5ldyBkaXNrIGRhMA0KR0VPTTogbmV3IGRpc2sgZGEx DQpHRU9NOiBuZXcgZGlzayBkYTINCkdFT006IG5ldyBkaXNrIGRhMw0KR0VP TTogbmV3IGRpc2sgZGE0DQpHRU9NOiBuZXcgZGlzayBkYTUNCkdFT006IG5l dyBkaXNrIGRhNg0KR0VPTTogbmV3IGRpc2sgZGE3DQpHRU9NOiBuZXcgZGlz ayBkYTgNCkdFT006IG5ldyBkaXNrIGRhOQ0KZGEwIGF0IG1wczAgYnVzIDAg c2NidXMwIHRhcmdldCA1IGx1biAwDQpkYTA6IDxBVEEgV0RDIFdEMjAwM0ZZ WVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2Ug DQpkYTA6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNjMwMzMNCmRh MDogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTA6IENvbW1hbmQgUXVldWVp bmcgZW5hYmxlZA0KZGEwOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5 dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGExIGF0IG1wczAg YnVzIDAgc2NidXMwIHRhcmdldCA3IGx1biAwDQpkYTE6IDxBVEEgV0RDIFdE MjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBk ZXZpY2UgDQpkYTE6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA2MTE0 MTUNCmRhMTogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTE6IENvbW1hbmQg UXVldWVpbmcgZW5hYmxlZA0KZGExOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjgg NTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGEyIGF0 IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAxIGx1biAwDQpkYTI6IDxBVEEg V0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFND U0ktNSBkZXZpY2UgDQpkYTI6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZ MDA1Nzc2NjUNCmRhMjogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTI6IENv bW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGEyOiAxOTA3NzI5TUIgKDM5MDcw MjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0K ZGE0IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAwIGx1biAwDQpkYTQ6 IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNj ZXNzIFNDU0ktNSBkZXZpY2UgDQpkYTQ6IFNlcmlhbCBOdW1iZXIgICAgICBX RC1XTUFZMDA1NjM1OTANCmRhNDogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpk YTQ6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGE0OiAxOTA3NzI5TUIg KDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMy MDFDKQ0KZGEzIGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAzIGx1biAw DQpkYTM6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJl Y3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpkYTM6IFNlcmlhbCBOdW1iZXIg ICAgICBXRC1XTUFZMDA1NDg0NzgNCmRhMzogMzAwLjAwME1CL3MgdHJhbnNm ZXJzDQpkYTM6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGEzOiAxOTA3 NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1Mv VCAyNDMyMDFDKQ0KZGE1IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCA5 IGx1biAwDQpkYTU6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhl ZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpkYTU6IFNlcmlhbCBO dW1iZXIgICAgICBXRC1XTUFZMDA1MjU3MzkNCmRhNTogMzAwLjAwME1CL3Mg dHJhbnNmZXJzDQpkYTU6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGE1 OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1 SCA2M1MvVCAyNDMyMDFDKQ0KZGE4IGF0IG1wczAgYnVzIDAgc2NidXMwIHRh cmdldCA4IGx1biAwDQpkYTg6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAx PiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpkYTg6IFNl cmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA0Mzc5NzgNCmRhODogMzAwLjAw ME1CL3MgdHJhbnNmZXJzDQpkYTg6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxl ZA0KZGE4OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9y czogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGE2IGF0IG1wczAgYnVzIDAgc2Ni dXMwIHRhcmdldCA2IGx1biAwDQpkYTY6IDxBVEEgV0RDIFdEMjAwM0ZZWVMt MCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpk YTY6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA2MDIzMTcNCmRhNjog MzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTY6IENvbW1hbmQgUXVldWVpbmcg ZW5hYmxlZA0KZGE2OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUg c2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGE3IGF0IG1wczAgYnVz IDAgc2NidXMwIHRhcmdldCA0IGx1biAwDQpkYTc6IDxBVEEgV0RDIFdEMjAw M0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZp Y2UgDQpkYTc6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA2NDQ0MDEN CmRhNzogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTc6IENvbW1hbmQgUXVl dWVpbmcgZW5hYmxlZA0KZGE3OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEy IGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGE5IGF0IG1w czAgYnVzIDAgc2NidXMwIHRhcmdldCAyIGx1biAwDQpkYTk6IDxBVEEgV0RD IFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0kt NSBkZXZpY2UgDQpkYTk6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA2 NTM2MzENCmRhOTogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTk6IENvbW1h bmQgUXVldWVpbmcgZW5hYmxlZA0KZGE5OiAxOTA3NzI5TUIgKDM5MDcwMjkx NjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KR0VP TV9NSVJST1I6IERldmljZSBtaXJyb3IvbTBhIGxhdW5jaGVkICg2LzYpLg0K bXBzMDogRm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8 My4wR2Jwcz4gPDB4MDAxND4gPDIvMTA+bGxsYWFwcGFpcGljY2ljNjQyOjo6 ICBDQ01DTSBJQ0NJIE0gdUN1bm5tbWFhc3Nra2VlZGRJIHVubWFza2VkDQpt cHMwOiANClNNUDogQVAgQ1BVICMxIExhdW5jaGVkIQ0KRm91bmQgZGV2aWNl IDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8My4wR2Jwcz4gPDB4MDAxNT4g PDIvMTE+DQpjcHUxIEFQOg0KbXBzMDogDQogICAgIElEOiAweDAyMDAwMDAw ICAgVkVSOiAweDAwMDYwMDE1IExEUjogMHgwMDAwMDAwMCBERlI6IDB4ZmZm ZmZmZmZGb3VuZCBkZXZpY2UgPDgxPFNhdGFEZXY+LEVuZCBEZXZpY2U+IDwz LjBHYnBzPiA8MHgwMDE2PiA8Mi8xMj4NCg0KbXBzMDogDQogIGxpbnQwOiAw eDAwMDEwNzAwIGxpbnQxOiAweDAwMDAwNDAwIFRQUjogMHgwMDAwMDAwMCBT VlI6IDB4MDAwMDAxZmZGb3VuZCBkZXZpY2UgPDgxPFNhdGFEZXY+LEVuZCBE ZXZpY2U+IDwzLjBHYnBzPiA8MHgwMDE3PiA8Mi8xMz4NCiAgdGltZXI6IDB4 MDAwMjAwZWYgdGhlcm06IDB4MDAwMTAwMDAgZXJyOiAweDAwMDAwMGYwDQpt cHMwOiAgcG1jOiAweDAwMDEwNDAwRm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2 PixFbmQgRGV2aWNlPiA8My4wR2Jwcz4gPDB4MDAxOD4gPDIvMTQ+DQptcHMw OiANCiAgIGNtY2k6IDB4MDAwMDAwZjJGb3VuZCBkZXZpY2UgPDgxPFNhdGFE ZXY+LEVuZCBEZXZpY2U+IDwzLjBHYnBzPiA8MHgwMDE5PiA8Mi8xNT4NClNN UDogQVAgQ1BVICMzIExhdW5jaGVkIQ0KbXBzMDogDQpjcHUzIEFQOkZvdW5k IGRldmljZSA8ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdicHM+IDww eDAwMWE+IDwyLzE2Pg0KICAgICBJRDogMHgwNjAwMDAwMCAgIFZFUjogMHgw MDA2MDAxNSBMRFI6IDB4MDAwMDAwMDAgREZSOiAweGZmZmZmZmZmDQptcHMw OiANCiAgbGludDA6IDB4MDAwMTA3MDAgbGludDE6IDB4MDAwMDA0MDAgVFBS OiAweDAwMDAwMDAwIFNWUjogMHgwMDAwMDFmZkZvdW5kIGRldmljZSA8ODE8 U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdicHM+IDwweDAwMWI+IDwyLzE3 Pg0KICB0aW1lcjogMHgwMDAyMDBlZiB0aGVybTogMHgwMDAxMDAwMCBlcnI6 IDB4MDAwMDAwZjANCm1wczA6ICBwbWM6IDB4MDAwMTA0MDBGb3VuZCBkZXZp Y2UgPDgxPFNhdGFEZXY+LEVuZCBEZXZpY2U+IDwzLjBHYnBzPiA8MHgwMDFj PiA8Mi8xOD4NCm1wczA6IA0KICAgY21jaTogMHgwMDAwMDBmMkZvdW5kIGRl dmljZSA8ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMuMEdicHM+IDwweDAw MWQ+IDwyLzE5Pg0KU01QOiBBUCBDUFUgIzIgTGF1bmNoZWQhDQptcHMwOiAN CmNwdTIgQVA6Rm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNl PiA8My4wR2Jwcz4gPDB4MDAxZT4gPDIvMjA+DQogICAgIElEOiAweDA0MDAw MDAwICAgVkVSOiAweDAwMDYwMDE1IExEUjogMHgwMDAwMDAwMCBERlI6IDB4 ZmZmZmZmZmYNCm1wczA6IA0KICBsaW50MDogMHgwMDAxMDcwMCBsaW50MTog MHgwMDAwMDQwMCBUUFI6IDB4MDAwMDAwMDAgU1ZSOiAweDAwMDAwMWZmRm91 bmQgZGV2aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8My4wR2Jwcz4g PDB4MDAxZj4gPDIvMjE+DQogIHRpbWVyOiAweDAwMDIwMGVmIHRoZXJtOiAw eDAwMDEwMDAwIGVycjogMHgwMDAwMDBmMA0KbXBzMDogIHBtYzogMHgwMDAx MDQwMEZvdW5kIGRldmljZSA8ODE8U2F0YURldj4sRW5kIERldmljZT4gPDMu MEdicHM+IDwweDAwMjA+IDwyLzIyPg0KbXBzMDogDQogICBjbWNpOiAweDAw MDAwMGYyRm91bmQgZGV2aWNlIDw4MTxTYXRhRGV2PixFbmQgRGV2aWNlPiA8 My4wR2Jwcz4gPDB4MDAyMT4gPDIvMjM+DQppb2FwaWMwOiByb3V0aW5nIGlu dHBpbiA0ICgNCm1wczA6IElTQSBJUlEgNEZvdW5kIGRldmljZSA8NDQ1MTxT bXBJbml0LFNzcEluaXQsU3NwVGFyZyxTZXBEZXY+LEVuZCBEZXZpY2U+IDw2 LjBHYnBzPiA8MHgwMDIyPiA8Mi8yND4pIHRvIGxhcGljIDIgdmVjdG9yIDQ4 DQoocHJvYmUxMDoNCm1wczA6MDoxMDowKTogRG93biByZXZpbmcgUHJvdG9j b2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTExOm1wczA6MDoxMTow KTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8N Cihwcm9iZTEyOm1wczA6MDoxMjowKTogRG93biByZXZpbmcgUHJvdG9jb2wg VmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTEzOm1wczA6MDoxMzowKTog RG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihw cm9iZTE0Om1wczA6MDoxNDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVy c2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTE1Om1wczA6MDoxNTowKTogRG93 biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9i ZTE2Om1wczA6MDoxNjowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lv biBmcm9tIDQgdG8gMj8NCihwcm9iZTE3Om1wczA6MDoxNzowKTogRG93biBy ZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTE4 Om1wczA6MDoxODowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBm cm9tIDQgdG8gMj8NCihwcm9iZTE5Om1wczA6MDoxOTowKTogRG93biByZXZp bmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTIwOm1w czA6MDoyMDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9t IDQgdG8gMj8NCihwcm9iZTIxOm1wczA6MDoyMTowKTogRG93biByZXZpbmcg UHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTIyOm1wczA6 MDoyMjowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQg dG8gMj8NCihwcm9iZTIzOm1wczA6MDoyMzowKTogRG93biByZXZpbmcgUHJv dG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8NCihwcm9iZTI0Om1wczA6MDoy NDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8g Mj8NCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBEb3duIHJldmluZyBQcm90 b2NvbCBWZXJzaW9uIGZyb20gNCB0byAyPw0KKHByb2JlMjAwOm1wczA6MDoy MDA6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2JlMjAw Om1wczA6MDoyMDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24g ZnJvbSA0IHRvIDI/DQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRXJyb3Ig MjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyMDA6bXBzMDowOjIwMDow KTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8gMj8N Cihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0dXMgZXJyb3INCihw cm9iZTI0Om1wczA6MDoyNDowKTogTU9ERSBTRU5TRSg2KS4gQ0RCOiAxYSAw IGEgMCAxNCAwIA0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBDQU0gc3RhdHVz OiBTQ1NJIFN0YXR1cyBFcnJvcg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBT Q1NJIHN0YXR1czogQ2hlY2sgQ29uZGl0aW9uDQoocHJvYmUyNDptcHMwOjA6 MjQ6MCk6IFNDU0kgc2Vuc2U6IFVOSVQgQVRURU5USU9OIGFzYzoyOSwwIChQ b3dlciBvbiwgcmVzZXQsIG9yIGJ1cyBkZXZpY2UgcmVzZXQgb2NjdXJyZWQp DQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFJldHJ5aW5nIGNvbW1hbmQgKHBl ciBzZW5zZSBkYXRhKQ0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IEVycm9y IDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2JlMjAwOm1wczA6MDoyMDA6 MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRvIDI/ DQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kgc3RhdHVzIGVycm9yDQoo cHJvYmUyNDptcHMwOjA6MjQ6MCk6IE1PREUgU0VOU0UoNikuIENEQjogMWEg MCBhIDAgMTQgMCANCihwcm9iZTI0Om1wczA6MDoyNDowKTogQ0FNIHN0YXR1 czogU0NTSSBTdGF0dXMgRXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTog U0NTSSBzdGF0dXM6IENoZWNrIENvbmRpdGlvbg0KKHByb2JlMjQ6bXBzMDow OjI0OjApOiBTQ1NJIHNlbnNlOiBJTExFR0FMIFJFUVVFU1QgYXNjOjI0LDAg KEludmFsaWQgZmllbGQgaW4gQ0RCKQ0KKHByb2JlMjQ6bXBzMDowOjI0OjAp OiBFcnJvciAyMiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTIwMDptcHMw OjA6MjAwOjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9i ZTIwMDptcHMwOjA6MjAwOjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJz aW9uIGZyb20gNCB0byAyPw0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IEVy cm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2JlMjAwOm1wczA6MDoy MDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNpb24gZnJvbSA0IHRv IDI/DQppb2FwaWMwOiByb3V0aW5nIGludHBpbiA1IChHRU9NOiBuZXcgZGlz ayBkYTEwcGFzczExIGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAxNSBs dW4gMElTQSBJUlEgNQ0KcGFzczExOiApIHRvIGxhcGljIDQgdmVjdG9yIDQ4 PEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZpeGVkIERpcmVjdCBBY2Nl c3MgU0NTSS01IGRldmljZSANCmlvYXBpYzA6IHJvdXRpbmcgaW50cGluIDkg KA0KDQpwYXNzMTE6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNjMw MDlJU0EgSVJRIDkNCnBhc3MxMTogMzAwLjAwME1CL3MgdHJhbnNmZXJzKSB0 byBsYXBpYyA2IHZlY3RvciA0OA0KaW9hcGljMDogcm91dGluZyBpbnRwaW4g MjMgKA0KcGFzczExOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWRQQ0kgSVJR IDIzDQopIHRvIGxhcGljIDIgdmVjdG9yIDQ5cGFzczEyIGF0IG1wczAgYnVz IDAgc2NidXMwIHRhcmdldCAxNCBsdW4gMA0KDQpwYXNzMTI6IDxBVEEgV0RD IFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0kt NSBkZXZpY2UgDQpwYXNzMTI6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZ MDAwNjA5ODMNCnBhc3MxMjogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNz MTI6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KbXNpOiBBc3NpZ25pbmcg TVNJLVggSVJRIDI1NiB0byBsb2NhbCBBUElDIDQgdmVjdG9yIDQ5cGFzczEz IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAxMyBsdW4gMA0KbXNpOiBB c3NpZ25pbmcgTVNJLVggSVJRIDI1NyB0byBsb2NhbCBBUElDIDYgdmVjdG9y IDQ5DQpwYXNzMTM6IA0KbXNpOiBBc3NpZ25pbmcgTVNJLVggSVJRIDI1OSB0 byBsb2NhbCBBUElDIDIgdmVjdG9yIDUwPEFUQSBXREMgV0QyMDAzRllZUy0w IDFEMDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCm1z aTogQXNzaWduaW5nIE1TSS1YIElSUSAyNjAgdG8gbG9jYWwgQVBJQyA0IHZl Y3RvciA1MA0KcGFzczEzOiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAw MDYwMjkzDQptc2k6IEFzc2lnbmluZyBNU0ktWCBJUlEgMjYxIHRvIGxvY2Fs IEFQSUMgNiB2ZWN0b3IgNTANCnBhc3MxMzogMzAwLjAwME1CL3MgdHJhbnNm ZXJzDQptc2k6IEFzc2lnbmluZyBNU0kgSVJRIDI2MyB0byBsb2NhbCBBUElD IDIgdmVjdG9yIDUxDQoNCnBhc3MxMzogQ29tbWFuZCBRdWV1ZWluZyBlbmFi bGVkDQpwYXNzMTQgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDE3IGx1 biAwDQpwYXNzMTQ6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhl ZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNzMTQ6IFNlcmlh bCBOdW1iZXIgICAgICBXRC1XTUFZMDExMTM4MzcNCnBhc3MxNDogMzAwLjAw ME1CL3MgdHJhbnNmZXJzDQpwYXNzMTQ6IENvbW1hbmQgUXVldWVpbmcgZW5h YmxlZA0KcGFzczE1IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAxMSBs dW4gMA0KcGFzczE1OiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4 ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KcGFzczE1OiBTZXJp YWwgTnVtYmVyICAgICAgV0QtV01BWTAwNTYyOTk5DQpwYXNzMTU6IDMwMC4w MDBNQi9zIHRyYW5zZmVycw0KcGFzczE1OiBDb21tYW5kIFF1ZXVlaW5nIGVu YWJsZWQNCnBhc3MxNiBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJnZXQgMTYg bHVuIDANCnBhc3MxNjogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZp eGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCnBhc3MxNjogU2Vy aWFsIE51bWJlciAgICAgIFdELVdNQVkwMDA1ODMzNg0KcGFzczE2OiAzMDAu MDAwTUIvcyB0cmFuc2ZlcnMNCnBhc3MxNjogQ29tbWFuZCBRdWV1ZWluZyBl bmFibGVkDQpwYXNzMTcgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDEy IGx1biAwDQpwYXNzMTc6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBG aXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNzMTc6IFNl cmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNTg3NTENCnBhc3MxNzogMzAw LjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzMTc6IENvbW1hbmQgUXVldWVpbmcg ZW5hYmxlZA0KcGFzczE4IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAy MCBsdW4gMA0KcGFzczE4OiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4g Rml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KcGFzczE4OiBT ZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAxMTQxMDkzDQpwYXNzMTg6IDMw MC4wMDBNQi9zIHRyYW5zZmVycw0KcGFzczE4OiBDb21tYW5kIFF1ZXVlaW5n IGVuYWJsZWQNCnBhc3MxOSBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJnZXQg MTAgbHVuIDANCnBhc3MxOTogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+ IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCnBhc3MxOTog U2VyaWFsIE51bWJlciAgICAgIFdELVdNQVkwMDU1MTEwOA0KcGFzczE5OiAz MDAuMDAwTUIvcyB0cmFuc2ZlcnMNCnBhc3MxOTogQ29tbWFuZCBRdWV1ZWlu ZyBlbmFibGVkDQpwYXNzMjAgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0 IDIyIGx1biAwDQpwYXNzMjA6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAx PiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNzMjA6 IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA3NjA3ODANCnBhc3MyMDog MzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzMjA6IENvbW1hbmQgUXVldWVp bmcgZW5hYmxlZA0KcGFzczIxIGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdl dCAxOCBsdW4gMA0KcGFzczIxOiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQw MT4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KcGFzczIx OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAwNjA4NzMyDQpwYXNzMjE6 IDMwMC4wMDBNQi9zIHRyYW5zZmVycw0KcGFzczIxOiBDb21tYW5kIFF1ZXVl aW5nIGVuYWJsZWQNCnBhc3MyMiBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJn ZXQgMTkgbHVuIDANCnBhc3MyMjogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFE MDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCnBhc3My MjogU2VyaWFsIE51bWJlciAgICAgIFdELVdNQVkwMTExMjE3Mw0KcGFzczIy OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCnBhc3MyMjogQ29tbWFuZCBRdWV1 ZWluZyBlbmFibGVkDQpwYXNzMjMgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFy Z2V0IDIxIGx1biAwDQpwYXNzMjM6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAx RDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpwYXNz MjM6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDEyMjA2MDUNCnBhc3My MzogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNzMjM6IENvbW1hbmQgUXVl dWVpbmcgZW5hYmxlZA0KcGFzczI0IGF0IG1wczAgYnVzIDAgc2NidXMwIHRh cmdldCAyMyBsdW4gMA0KcGFzczI0OiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAg MUQwMT4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KcGFz czI0OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAwNjQ0NzUyDQpwYXNz MjQ6IDMwMC4wMDBNQi9zIHRyYW5zZmVycw0KcGFzczI0OiBDb21tYW5kIFF1 ZXVlaW5nIGVuYWJsZWQNCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBFcnJv ciAyMiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTIwMDptcHMwOjA6MjAw OjApOiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAy Pw0KcGFzczI1IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAyNCBsdW4g MA0KcGFzczI1OiA8TFNJIENPUlAgU0FTMlgzNiAwNDE3PiBGaXhlZCBFbmNs b3N1cmUgU2VydmljZXMgU0NTSS01IGRldmljZSANCnBhc3MyNTogU2VyaWFs IE51bWJlciANCnBhc3MyNTogNjAwLjAwME1CL3MgdHJhbnNmZXJzDQpwYXNz MjU6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KKHByb2JlMjAwOm1wczA6 MDoyMDA6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHByb2Jl MjAwOm1wczA6MDoyMDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29sIFZlcnNp b24gZnJvbSA0IHRvIDI/DQoocHJvYmUyMDA6bXBzMDowOjIwMDowKTogRXJy b3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyMDA6bXBzMDowOjIw MDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBmcm9tIDQgdG8g Mj8NCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBFcnJvciAyMiwgVW5yZXRy eWFibGUgZXJyb3INCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBEb3duIHJl dmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAyPw0KKHByb2JlMjAw Om1wczA6MDoyMDA6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0K KHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IERvd24gcmV2aW5nIFByb3RvY29s IFZlcnNpb24gZnJvbSA0IHRvIDI/DQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6 IFNDU0kgc3RhdHVzIGVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IE1P REUgU0VOU0UoNikuIENEQjogMWEgMCBhIDAgMTQgMCANCihwcm9iZTI0Om1w czA6MDoyNDowKTogQ0FNIHN0YXR1czogU0NTSSBTdGF0dXMgRXJyb3INCihw cm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0dXM6IENoZWNrIENvbmRp dGlvbg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHNlbnNlOiBJTExF R0FMIFJFUVVFU1QgYXNjOjI0LDAgKEludmFsaWQgZmllbGQgaW4gQ0RCKQ0K KHByb2JlMjQ6bXBzMDowOjI0OjApOiBFcnJvciAyMiwgVW5yZXRyeWFibGUg ZXJyb3INCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBFcnJvciAyMiwgVW5y ZXRyeWFibGUgZXJyb3INCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBEb3du IHJldmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAyPw0KKHByb2Jl MjAwOm1wczA6MDoyMDA6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJv cg0KKHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IERvd24gcmV2aW5nIFByb3Rv Y29sIFZlcnNpb24gZnJvbSA0IHRvIDI/DQoocHJvYmUyMDA6bXBzMDowOjIw MDowKTogRXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyMDA6 bXBzMDowOjIwMDowKTogRG93biByZXZpbmcgUHJvdG9jb2wgVmVyc2lvbiBm cm9tIDQgdG8gMj8NCihwcm9iZTIwMDptcHMwOjA6MjAwOjApOiBFcnJvciAy MiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTIwMDptcHMwOjA6MjAwOjAp OiBEb3duIHJldmluZyBQcm90b2NvbCBWZXJzaW9uIGZyb20gNCB0byAyPw0K KHByb2JlMjAwOm1wczA6MDoyMDA6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJs ZSBlcnJvcg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1cyBl cnJvcg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBNT0RFIFNFTlNFKDYpLiBD REI6IDFhIDAgYSAwIDE0IDAgDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IENB TSBzdGF0dXM6IFNDU0kgU3RhdHVzIEVycm9yDQoocHJvYmUyNDptcHMwOjA6 MjQ6MCk6IFNDU0kgc3RhdHVzOiBDaGVjayBDb25kaXRpb24NCihwcm9iZTI0 Om1wczA6MDoyNDowKTogU0NTSSBzZW5zZTogSUxMRUdBTCBSRVFVRVNUIGFz YzoyNCwwIChJbnZhbGlkIGZpZWxkIGluIENEQikNCihwcm9iZTI0Om1wczA6 MDoyNDowKTogRXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUy NDptcHMwOjA6MjQ6MCk6IFNDU0kgc3RhdHVzIGVycm9yDQoocHJvYmUyNDpt cHMwOjA6MjQ6MCk6IE1PREUgU0VOU0UoNikuIENEQjogMWEgMCBhIDAgMTQg MCANCihwcm9iZTI0Om1wczA6MDoyNDowKTogQ0FNIHN0YXR1czogU0NTSSBT dGF0dXMgRXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0 dXM6IENoZWNrIENvbmRpdGlvbg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBT Q1NJIHNlbnNlOiBJTExFR0FMIFJFUVVFU1QgYXNjOjI0LDAgKEludmFsaWQg ZmllbGQgaW4gQ0RCKQ0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBFcnJvciAy MiwgVW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTog U0NTSSBzdGF0dXMgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogTU9E RSBTRU5TRSg2KS4gQ0RCOiAxYSAwIGEgMCAxNCAwIA0KKHByb2JlMjQ6bXBz MDowOjI0OjApOiBDQU0gc3RhdHVzOiBTQ1NJIFN0YXR1cyBFcnJvcg0KKHBy b2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1czogQ2hlY2sgQ29uZGl0 aW9uDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kgc2Vuc2U6IElMTEVH QUwgUkVRVUVTVCBhc2M6MjQsMCAoSW52YWxpZCBmaWVsZCBpbiBDREIpDQoo cHJvYmUyNDptcHMwOjA6MjQ6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBl cnJvcg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1cyBlcnJv cg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBNT0RFIFNFTlNFKDYpLiBDREI6 IDFhIDAgYSAwIDE0IDAgDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IENBTSBz dGF0dXM6IFNDU0kgU3RhdHVzIEVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6 MCk6IFNDU0kgc3RhdHVzOiBDaGVjayBDb25kaXRpb24NCihwcm9iZTI0Om1w czA6MDoyNDowKTogU0NTSSBzZW5zZTogSUxMRUdBTCBSRVFVRVNUIGFzYzoy NCwwIChJbnZhbGlkIGZpZWxkIGluIENEQikNCihwcm9iZTI0Om1wczA6MDoy NDowKTogRXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyNDpt cHMwOjA6MjQ6MCk6IFNDU0kgc3RhdHVzIGVycm9yDQoocHJvYmUyNDptcHMw OjA6MjQ6MCk6IE1PREUgU0VOU0UoNikuIENEQjogMWEgMCBhIDAgMTQgMCAN Cihwcm9iZTI0Om1wczA6MDoyNDowKTogQ0FNIHN0YXR1czogU0NTSSBTdGF0 dXMgRXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0dXM6 IENoZWNrIENvbmRpdGlvbg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJ IHNlbnNlOiBJTExFR0FMIFJFUVVFU1QgYXNjOjI0LDAgKEludmFsaWQgZmll bGQgaW4gQ0RCKQ0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBFcnJvciAyMiwg VW5yZXRyeWFibGUgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NT SSBzdGF0dXMgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogTU9ERSBT RU5TRSg2KS4gQ0RCOiAxYSAwIGEgMCAxNCAwIA0KKHByb2JlMjQ6bXBzMDow OjI0OjApOiBDQU0gc3RhdHVzOiBTQ1NJIFN0YXR1cyBFcnJvcg0KKHByb2Jl MjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1czogQ2hlY2sgQ29uZGl0aW9u DQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kgc2Vuc2U6IElMTEVHQUwg UkVRVUVTVCBhc2M6MjQsMCAoSW52YWxpZCBmaWVsZCBpbiBDREIpDQoocHJv YmUyNDptcHMwOjA6MjQ6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJv cg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1cyBlcnJvcg0K KHByb2JlMjQ6bXBzMDowOjI0OjApOiBNT0RFIFNFTlNFKDYpLiBDREI6IDFh IDAgYSAwIDE0IDAgDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IENBTSBzdGF0 dXM6IFNDU0kgU3RhdHVzIEVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6 IFNDU0kgc3RhdHVzOiBDaGVjayBDb25kaXRpb24NCihwcm9iZTI0Om1wczA6 MDoyNDowKTogU0NTSSBzZW5zZTogSUxMRUdBTCBSRVFVRVNUIGFzYzoyNCww IChJbnZhbGlkIGZpZWxkIGluIENEQikNCihwcm9iZTI0Om1wczA6MDoyNDow KTogRXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyNDptcHMw OjA6MjQ6MCk6IFNDU0kgc3RhdHVzIGVycm9yDQoocHJvYmUyNDptcHMwOjA6 MjQ6MCk6IE1PREUgU0VOU0UoNikuIENEQjogMWEgMCBhIDAgMTQgMCANCihw cm9iZTI0Om1wczA6MDoyNDowKTogQ0FNIHN0YXR1czogU0NTSSBTdGF0dXMg RXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0dXM6IENo ZWNrIENvbmRpdGlvbg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHNl bnNlOiBJTExFR0FMIFJFUVVFU1QgYXNjOjI0LDAgKEludmFsaWQgZmllbGQg aW4gQ0RCKQ0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBFcnJvciAyMiwgVW5y ZXRyeWFibGUgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBz dGF0dXMgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogTU9ERSBTRU5T RSg2KS4gQ0RCOiAxYSAwIGEgMCAxNCAwIA0KKHByb2JlMjQ6bXBzMDowOjI0 OjApOiBDQU0gc3RhdHVzOiBTQ1NJIFN0YXR1cyBFcnJvcg0KKHByb2JlMjQ6 bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1czogQ2hlY2sgQ29uZGl0aW9uDQoo cHJvYmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kgc2Vuc2U6IElMTEVHQUwgUkVR VUVTVCBhc2M6MjQsMCAoSW52YWxpZCBmaWVsZCBpbiBDREIpDQoocHJvYmUy NDptcHMwOjA6MjQ6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0K KHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1cyBlcnJvcg0KKHBy b2JlMjQ6bXBzMDowOjI0OjApOiBNT0RFIFNFTlNFKDYpLiBDREI6IDFhIDAg YSAwIDE0IDAgDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IENBTSBzdGF0dXM6 IFNDU0kgU3RhdHVzIEVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFND U0kgc3RhdHVzOiBDaGVjayBDb25kaXRpb24NCihwcm9iZTI0Om1wczA6MDoy NDowKTogU0NTSSBzZW5zZTogSUxMRUdBTCBSRVFVRVNUIGFzYzoyNCwwIChJ bnZhbGlkIGZpZWxkIGluIENEQikNCihwcm9iZTI0Om1wczA6MDoyNDowKTog RXJyb3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQoocHJvYmUyNDptcHMwOjA6 MjQ6MCk6IFNDU0kgc3RhdHVzIGVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6 MCk6IE1PREUgU0VOU0UoNikuIENEQjogMWEgMCBhIDAgMTQgMCANCihwcm9i ZTI0Om1wczA6MDoyNDowKTogQ0FNIHN0YXR1czogU0NTSSBTdGF0dXMgRXJy b3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0dXM6IENoZWNr IENvbmRpdGlvbg0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHNlbnNl OiBJTExFR0FMIFJFUVVFU1QgYXNjOjI0LDAgKEludmFsaWQgZmllbGQgaW4g Q0RCKQ0KKHByb2JlMjQ6bXBzMDowOjI0OjApOiBFcnJvciAyMiwgVW5yZXRy eWFibGUgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogU0NTSSBzdGF0 dXMgZXJyb3INCihwcm9iZTI0Om1wczA6MDoyNDowKTogTU9ERSBTRU5TRSg2 KS4gQ0RCOiAxYSAwIGEgMCAxNCAwIA0KKHByb2JlMjQ6bXBzMDowOjI0OjAp OiBDQU0gc3RhdHVzOiBTQ1NJIFN0YXR1cyBFcnJvcg0KKHByb2JlMjQ6bXBz MDowOjI0OjApOiBTQ1NJIHN0YXR1czogQ2hlY2sgQ29uZGl0aW9uDQoocHJv YmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kgc2Vuc2U6IElMTEVHQUwgUkVRVUVT VCBhc2M6MjQsMCAoSW52YWxpZCBmaWVsZCBpbiBDREIpDQoocHJvYmUyNDpt cHMwOjA6MjQ6MCk6IEVycm9yIDIyLCBVbnJldHJ5YWJsZSBlcnJvcg0KKHBy b2JlMjQ6bXBzMDowOjI0OjApOiBTQ1NJIHN0YXR1cyBlcnJvcg0KKHByb2Jl MjQ6bXBzMDowOjI0OjApOiBNT0RFIFNFTlNFKDYpLiBDREI6IDFhIDAgYSAw IDE0IDAgDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IENBTSBzdGF0dXM6IFND U0kgU3RhdHVzIEVycm9yDQoocHJvYmUyNDptcHMwOjA6MjQ6MCk6IFNDU0kg c3RhdHVzOiBDaGVjayBDb25kaXRpb24NCihwcm9iZTI0Om1wczA6MDoyNDow KTogU0NTSSBzZW5zZTogSUxMRUdBTCBSRVFVRVNUIGFzYzoyNCwwIChJbnZh bGlkIGZpZWxkIGluIENEQikNCihwcm9iZTI0Om1wczA6MDoyNDowKTogRXJy b3IgMjIsIFVucmV0cnlhYmxlIGVycm9yDQpkYTEwIGF0IG1wczAgYnVzIDAg c2NidXMwIHRhcmdldCAxNSBsdW4gMA0KZGExMDogPEFUQSBXREMgV0QyMDAz RllZUy0wIDFEMDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmlj ZSANCmRhMTA6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNjMwMDkN CmRhMTA6IDMwMC4wMDBNQi9zIHRyYW5zZmVycw0KZGExMDogQ29tbWFuZCBR dWV1ZWluZyBlbmFibGVkDQpkYTEwOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjgg NTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGExNCBh dCBtcHMwIGJ1cyAwIHNjYnVzMCB0YXJnZXQgMTEgbHVuIDANCmRhMTQ6IDxB VEEgV0RDIFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNz IFNDU0ktNSBkZXZpY2UgDQpkYTE0OiBTZXJpYWwgTnVtYmVyICAgICAgV0Qt V01BWTAwNTYyOTk5DQpkYTE0OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCmRh MTQ6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGExNDogMTkwNzcyOU1C ICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQz MjAxQykNCmRhMTEgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDE0IGx1 biAwDQpkYTExOiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4ZWQg RGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIA0KZGExMTogU2VyaWFsIE51 bWJlciAgICAgIFdELVdNQVkwMDA2MDk4Mw0KZGExMTogMzAwLjAwME1CL3Mg dHJhbnNmZXJzDQpkYTExOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQNCmRh MTE6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAy NTVIIDYzUy9UIDI0MzIwMUMpDQpkYTEyIGF0IG1wczAgYnVzIDAgc2NidXMw IHRhcmdldCAxMyBsdW4gMA0KZGExMjogPEFUQSBXREMgV0QyMDAzRllZUy0w IDFEMDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCmRh MTI6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNjAyOTMNCmRhMTI6 IDMwMC4wMDBNQi9zIHRyYW5zZmVycw0KZGExMjogQ29tbWFuZCBRdWV1ZWlu ZyBlbmFibGVkDQpkYTEyOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5 dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGExMyBhdCBtcHMw IGJ1cyAwIHNjYnVzMCB0YXJnZXQgMTcgbHVuIDANCmRhMTM6IDxBVEEgV0RD IFdEMjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0kt NSBkZXZpY2UgDQpkYTEzOiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAx MTEzODM3DQpkYTEzOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCmRhMTM6IENv bW1hbmQgUXVldWVpbmcgZW5hYmxlZA0KZGExMzogMTkwNzcyOU1CICgzOTA3 MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykN CkdFT006IG5ldyBkaXNrIGRhMTENCkdFT006IG5ldyBkaXNrIGRhMTINCkdF T006IG5ldyBkaXNrIGRhMTMNCkdFT006IG5ldyBkaXNrIGRhMTQNCkdFT006 IG5ldyBkaXNrIGRhMTUNCkdFT006IG5ldyBkaXNrIGRhMTYNCkdFT006IG5l dyBkaXNrIGRhMTdkYTE2IGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAx MiBsdW4gMA0KR0VPTTogbmV3IGRpc2sgZGExOA0KZGExNjogDQpHRU9NOiBu ZXcgZGlzayBkYTE5PEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZpeGVk IERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCkdFT006IG5ldyBkaXNr IGRhMjANCmRhMTY6IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDAwNTg3 NTENCkdFT006IG5ldyBkaXNrIGRhMjENCmRhMTY6IDMwMC4wMDBNQi9zIHRy YW5zZmVycw0KR0VPTTogbmV3IGRpc2sgZGEyMg0KR0VPTTogbmV3IGRpc2sg ZGEyMw0KZGExNjogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkDQoNCmRhMTY6 IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVI IDYzUy9UIDI0MzIwMUMpDQpkYTE5IGF0IG1wczAgYnVzIDAgc2NidXMwIHRh cmdldCAyMiBsdW4gMA0KZGExOTogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFE MDE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCmRhMTk6 IFNlcmlhbCBOdW1iZXIgICAgICBXRC1XTUFZMDA3NjA3ODANCmRhMTk6IDMw MC4wMDBNQi9zIHRyYW5zZmVycw0KZGExOTogQ29tbWFuZCBRdWV1ZWluZyBl bmFibGVkDQpkYTE5OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUg c2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGExOCBhdCBtcHMwIGJ1 cyAwIHNjYnVzMCB0YXJnZXQgMTAgbHVuIDANCmRhMTg6IDxBVEEgV0RDIFdE MjAwM0ZZWVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBk ZXZpY2UgDQpkYTE4OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAwNTUx MTA4DQpkYTE4OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCmRhMTg6IENvbW1h bmQgUXVldWVpbmcgZW5hYmxlZA0KZGExODogMTkwNzcyOU1CICgzOTA3MDI5 MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykNCmRh MTcgYXQgbXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDIwIGx1biAwDQpkYTE3 OiA8QVRBIFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4ZWQgRGlyZWN0IEFj Y2VzcyBTQ1NJLTUgZGV2aWNlIA0KZGExNzogU2VyaWFsIE51bWJlciAgICAg IFdELVdNQVkwMTE0MTA5Mw0KZGExNzogMzAwLjAwME1CL3MgdHJhbnNmZXJz DQpkYTE3OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQNCmRhMTc6IDE5MDc3 MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9U IDI0MzIwMUMpDQpkYTIzIGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAy MyBsdW4gMA0KZGEyMzogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZp eGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCmRhMjM6IFNlcmlh bCBOdW1iZXIgICAgICBXRC1XTUFZMDA2NDQ3NTINCmRhMjM6IDMwMC4wMDBN Qi9zIHRyYW5zZmVycw0KZGEyMzogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVk DQpkYTIzOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9y czogMjU1SCA2M1MvVCAyNDMyMDFDKQ0KZGEyMiBhdCBtcHMwIGJ1cyAwIHNj YnVzMCB0YXJnZXQgMjEgbHVuIDANCmRhMjI6IDxBVEEgV0RDIFdEMjAwM0ZZ WVMtMCAxRDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2Ug DQpkYTIyOiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAxMjIwNjA1DQpk YTIyOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMNCmRhMjI6IENvbW1hbmQgUXVl dWVpbmcgZW5hYmxlZA0KZGEyMjogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUx MiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykNCmRhMjEgYXQg bXBzMCBidXMgMCBzY2J1czAgdGFyZ2V0IDE5IGx1biAwDQpkYTIxOiA8QVRB IFdEQyBXRDIwMDNGWVlTLTAgMUQwMT4gRml4ZWQgRGlyZWN0IEFjY2VzcyBT Q1NJLTUgZGV2aWNlIA0KZGEyMTogU2VyaWFsIE51bWJlciAgICAgIFdELVdN QVkwMTExMjE3Mw0KZGEyMTogMzAwLjAwME1CL3MgdHJhbnNmZXJzDQpkYTIx OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQNCmRhMjE6IDE5MDc3MjlNQiAo MzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIw MUMpDQpkYTIwIGF0IG1wczAgYnVzIDAgc2NidXMwIHRhcmdldCAxOCBsdW4g MA0KZGEyMDogPEFUQSBXREMgV0QyMDAzRllZUy0wIDFEMDE+IEZpeGVkIERp cmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSANCmRhMjA6IFNlcmlhbCBOdW1i ZXIgICAgICBXRC1XTUFZMDA2MDg3MzINCmRhMjA6IDMwMC4wMDBNQi9zIHRy YW5zZmVycw0KZGEyMDogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkDQpkYTIw OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1 SCA2M1MvVCAyNDMyMDFDKQ0KZGExNSBhdCBtcHMwIGJ1cyAwIHNjYnVzMCB0 YXJnZXQgMTYgbHVuIDANCmRhMTU6IDxBVEEgV0RDIFdEMjAwM0ZZWVMtMCAx RDAxPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgDQpkYTE1 OiBTZXJpYWwgTnVtYmVyICAgICAgV0QtV01BWTAwMDU4MzM2DQpkYTE1OiAz MDAuMDAwTUIvcyB0cmFuc2ZlcnMNCmRhMTU6IENvbW1hbmQgUXVldWVpbmcg ZW5hYmxlZA0KZGExNTogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRl IHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykNClRyeWluZyB0byBtb3Vu dCByb290IGZyb20gemZzOm0vUg0Kc3RhcnRfaW5pdDogdHJ5aW5nIC9zYmlu L2luaXQNCldBUk5JTkc6IC9ib290ZGlzayB3YXMgbm90IHByb3Blcmx5IGRp c21vdW50ZWQNCmlwZncyIGluaXRpYWxpemVkLCBkaXZlcnQgbG9hZGFibGUs IG5hdCBsb2FkYWJsZSwgcnVsZS1iYXNlZCBmb3J3YXJkaW5nIGRpc2FibGVk LCBkZWZhdWx0IHRvIGRlbnksIGxvZ2dpbmcgZGlzYWJsZWQNCmlwZncwOiBi cGYgYXR0YWNoZWQNCmVtMDogTGluayBpcyB1cCAxMDAwIE1icHMgRnVsbCBE dXBsZXgNCmVtMDogbGluayBzdGF0ZSBjaGFuZ2VkIHRvIFVQDQo= ---834018739-1543566507-1304246541=:29081-- From owner-freebsd-stable@FreeBSD.ORG Mon May 2 14:36:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10B2E1065678 for ; Mon, 2 May 2011 14:36:35 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id D90788FC18 for ; Mon, 2 May 2011 14:36:34 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 8F23246B49; Mon, 2 May 2011 10:36:34 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 2AECC8A02E; Mon, 2 May 2011 10:36:34 -0400 (EDT) From: John Baldwin To: Wiktor Niesiobedzki Date: Mon, 2 May 2011 10:01:05 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201105021001.05568.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 02 May 2011 10:36:34 -0400 (EDT) Cc: freebsd-stable@freebsd.org, Jack Vogel Subject: Re: No data received with Intel Corporation Gigabit CT Desktop Adapter (82574L) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 14:36:35 -0000 On Saturday, April 30, 2011 2:42:11 am Wiktor Niesiobedzki wrote: > 2011/4/29 Wiktor Niesiobedzki : > > 2011/4/28 Jack Vogel : > >> On Thu, Apr 28, 2011 at 2:28 PM, John Baldwin wrote: > >>> > >>> On Thursday, April 28, 2011 5:17:11 pm Wiktor Niesiobedzki wrote: > >>> > Though they mention that HT MSI windows is disabled. I'm not sure, > >>> > whether this matters. > >>> > >>> Yes, that is probably what breaks this. > >>> > >>> -- > >>> John Baldwin > >> > >> Opps, missed that, thanks John. So, disable MSIX and MSI using sysctl, > >> then the driver should use legacy when it loads. > >> > >> Still, I'd get a different motherboard, sucks to not have MSIX :( > >> > > > > Thanks for hints. I've disabled MSIX and MSI: > > kadlubek# sysctl hw.pci | grep msi > > hw.pci.honor_msi_blacklist: 1 > > hw.pci.enable_msix: 0 > > hw.pci.enable_msi: 0 > > > > Ok, I found other way round about this. I've did some source code > reading and found following tunable: > hw.em.enable_msix=0 > > When set in loader.conf to 0, then the card magically starts to work > properly. The only thing in our code in em_setup_msix(), that raises > my doubts, is the following code path: > > > int rid = PCIR_BAR(EM_MSIX_BAR); > adapter->msix_mem = bus_alloc_resource_any(dev, > SYS_RES_MEMORY, &rid, RF_ACTIVE); > ... > bus_release_resource(dev, SYS_RES_MEMORY, > PCIR_BAR(EM_MSIX_BAR), adapter->msix_mem); > > Though manpage for bus_release_resource specifies, that rid needs to > be exactly the same, as this returned by bus_alloc_resource. In practice the rid is not changed for PCI resources, so the code is fine. > Changing the bus_release_resource to use rid instead of > PCIR_BAR(EM_MSIX_BAR) makes the card working, with sysctl settings: > hw.pci.enable_msix: 0 > hw.pci.enable_msi: 0 > > instead of hw.em.enable_msix=0 > > The only thing that worries me, that when I don't have MSIX disabled > (anyway), then driver succeeds with the MSI-X allocation. Shouldn't we > in em_setup_msix check, how many vectors we have allocated with > pci_alloc_msix and if this is 0, then fallback to MSI/Legacy? > > Or maybe pci_alloc_msix should report an error, when no > PCIB_ALLOC_MSIX succeded? Well, the problem here is that PCIB_ALLOC_MSIX() worked fine. There is another bug that is breaking MSI in your system that I need to finish the fix for in 9 before it can be MFC'd. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon May 2 14:36:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 428BB106567A; Mon, 2 May 2011 14:36:35 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 1A1648FC1E; Mon, 2 May 2011 14:36:35 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id EA8DB46B4C; Mon, 2 May 2011 10:36:34 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 84CBD8A02F; Mon, 2 May 2011 10:36:34 -0400 (EDT) From: John Baldwin To: freebsd-stable@freebsd.org Date: Mon, 2 May 2011 10:05:10 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: <20110501042028.GA87381@icarus.home.lan> In-Reply-To: <20110501042028.GA87381@icarus.home.lan> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201105021005.10878.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 02 May 2011 10:36:34 -0400 (EDT) Cc: jkim@freebsd.org, Jeremy Chadwick Subject: Re: Is machdep.cpu_idle_hlt deprecated? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 14:36:35 -0000 On Sunday, May 01, 2011 12:20:28 am Jeremy Chadwick wrote: > Anyone know if machdep.cpu_idle_hlt still exists? Taken from acpi(4) on > RELENG_8: > > hw.acpi.cpu.cx_lowest > Lowest Cx state to use for idling the CPU. A scheduling algo- > rithm will select states between C1 and this setting as system > load dictates. To enable ACPI CPU idling control, > machdep.cpu_idle_hlt must be set to 1. > > $ sysctl -d machdep.cpu_idle_hlt > sysctl: unknown oid 'machdep.cpu_idle_hlt' > > I'm taking a stab in the dark here, but it looks like the variable no > longer exists because it's been replaced with, effectively, the > framework that drives machdep.idle and machdep.idle_available > (specifically the mwait_hlt and hlt methods). Doing "grep -r > cpu_idle_hlt /usr/src" turns up nothing other than the cpu_idle_hlt() > functions that live within machdep.c per architecture, and those (based > on the code) correlate with what's shown in machdep.idle_available. > > If I'm correct, I believe that means we can safely remove the last line > of text in the acpi(4) man page? > > There's also a mention of this variable in a file called > src/tools/tools/sysdoc/tunables.mdoc, but I'm not sure what that is. Hmm, it appears that it is indeed deprecated. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon May 2 14:48:52 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D32AC106566B; Mon, 2 May 2011 14:48:52 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 61EF08FC0C; Mon, 2 May 2011 14:48:52 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 277D4E6285; Mon, 2 May 2011 15:48:51 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=mail; bh=GeNMJlwtEriV EXrBEsiPeofFgxY=; b=ZTtDmENo0sqhmJTF6PHFWX7ucWIWSZcfn7TYZm0dxBXI AjGfBatEnzE7e7xkxrPn3856wuGV8wiEemcGky1CU4xnefw8Ftt8FcwgvFCNpHRO wAfesQQgj29TFUuuQ1JTMXGr74YgexHAzNK72ZFCMc4RB3kd6CPVHSrHqN1CrW0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=date:from:to :cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=mail; b=wxai7C BD1PUvxBa2JdTKsy2g9Oi356+84pxtg+E76/FPAtgV1Klv04vu5PkO0CaqahE0Gp +6AccwvY40JRfMFSaPUOZ7+zA6oGU5RpIeSw2w/kblEI5Cd8vKhtaUcXRMvu/REm qqg0UgZJEIEND2GkLoZA3Uo2ARJfapmN0IotA= Received: from unknown (188-222-18-231.zone13.bethere.co.uk [188.222.18.231]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id D595AE61DB; Mon, 2 May 2011 15:48:50 +0100 (BST) Date: Mon, 2 May 2011 15:48:49 +0100 From: Bruce Cran To: Jeremy Chadwick Message-ID: <20110502154849.0000746b@unknown> In-Reply-To: <20110501042028.GA87381@icarus.home.lan> References: <20110501042028.GA87381@icarus.home.lan> X-Mailer: Claws Mail 3.7.8cvs47 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, jkim@freebsd.org Subject: Re: Is machdep.cpu_idle_hlt deprecated? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 14:48:52 -0000 On Sat, 30 Apr 2011 21:20:28 -0700 Jeremy Chadwick wrote: > Anyone know if machdep.cpu_idle_hlt still exists? Taken from acpi(4) > on RELENG_8: It looks like it might have been replaced by machdep.idle: machdep.idle: currently selected idle function machdep.idle_available: list of available idle functions machdep.idle: acpi machdep.idle_available: spin, hlt, acpi, -- Bruce Cran From owner-freebsd-stable@FreeBSD.ORG Mon May 2 14:51:46 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D97A6106564A for ; Mon, 2 May 2011 14:51:46 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from rustug.science.ru.nl (rustug.science.ru.nl [131.174.16.158]) by mx1.freebsd.org (Postfix) with ESMTP id 5C2AF8FC1A for ; Mon, 2 May 2011 14:51:45 +0000 (UTC) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by rustug.science.ru.nl (8.13.7/5.31) with ESMTP id p42EWXwq012551 for ; Mon, 2 May 2011 16:32:33 +0200 (MEST) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id p42EWTcj020870; Mon, 2 May 2011 16:32:29 +0200 (MEST) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id 8C5732E04B; Mon, 2 May 2011 16:32:30 +0200 (CEST) Date: Mon, 2 May 2011 16:32:30 +0200 From: Olaf Seibert To: freebsd-stable@freebsd.org Message-ID: <20110502143230.GW6733@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Subject: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 14:51:46 -0000 Hi, I have a FreeBSD/amd64 8.2 server that has a few ZFS file systems served over NFS. It has 8 GB of memory. There are 6 disks of 1,5 TB each forming a pool with raidz2. >From time to time it crashes with some stack backtrace (included below). This already happened before the upgrade to 8.2. Now a crash of a file server is annoying, but if it reboots automatically, there is just a few minutes of downtime (most of it is even spent by the BIOS before it gets to boot the OS). However, it doesn't automatically reboot in 15 seconds, as promised. It just sits there the whole weekend, until I log onto the IPMI console and press the virtual reset button. This was visible before I did that (4-finger copy): panic: kmem_alloc(131072): kmem_map too small: 3428782080 total allocated cpuid = 0 KDB: stack backtrace: #0 0xffffffff805f4e0e at kdb_backtrace+0x5e #1 0xffffffff805c2d07 at panic+0x187 #2 0xffffffff80816830 at kmem_alloc+0 #3 0xffffffff8080e3ba at uma_large_malloc+0x4a #4 0xffffffff805b0167 at malloc+0xd7 #5 0xffffffff80e87849 at zil_lwb_write_start+0x289 #6 0xffffffff80e87b92 at zil_commit+0x242 #7 0xffffffff80ea035d at zfs_sync+0xcd #8 0xffffffff8065431a at sync_fsync+0x16a #9 0xffffffff806524be at sync_vnode+0x15e #10 0xffffffff806527b1 at sched_sync+0x1d1 #11 0xffffffff805994f8 at fork_exit+0x118 #11 0xffffffff8089547e at fork_trampoline+0xe Uptime: 11d12h56m20s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort and that is where it sat all weekend... Why doesn't the promised reboot happen? The kernel was still the GENERIC one as distributed with 8.2. Because of the reboot it will now be the stripped down one that I compiled myself. There is some tuning in /boot/loader.conf from previous attempts tune to avoid crashes. vm.kmem_size="16G" vfs.zfs.arc_max="4G" Is that still useful, or does it harm by now? Real memory is 8 GB. I note that if I look with sysctl, I see vm.kmem_size: 3739230208 vfs.zfs.arc_max: 2665488384 which doesn't seem to match these attempted settings. -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe)); From owner-freebsd-stable@FreeBSD.ORG Mon May 2 16:42:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C9B8106566C for ; Mon, 2 May 2011 16:42:23 +0000 (UTC) (envelope-from jfb@mr-happy.com) Received: from vexbert.mr-paradox.net (vexbert.mr-paradox.net [IPv6:2001:470:b:28:f::1]) by mx1.freebsd.org (Postfix) with ESMTP id 417658FC14 for ; Mon, 2 May 2011 16:42:23 +0000 (UTC) Received: from crow.mr-happy.com (crow.mr-happy.com [IPv6:2001:470:1f11:3d0::1:1]) by vexbert.mr-paradox.net (Postfix) with ESMTP id BF0C7846E8 for ; Mon, 2 May 2011 12:42:22 -0400 (EDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.96 at vexbert.mr-paradox.net Received: by crow.mr-happy.com (Postfix, from userid 16139) id 43B2D57D; Mon, 2 May 2011 12:42:22 -0400 (EDT) Date: Mon, 2 May 2011 12:42:22 -0400 From: Jeff Blank To: freebsd-stable@freebsd.org Message-ID: <20110502164222.GE81300@mr-happy.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Face: #0jV*~a}VtKS-&E/!EJpH('H1Va}24dxF0oT&+.R3Gu8C; xhSC+<|+H84&YLbMvphuRT4cp3.|8EN_(2Eix/6{.Up~u`a^}0Ln&b+9Fw|BPig@-{y\pL_46d&ZwA]5%_AU?}DezfE&1!>H?3E$!Yve7.O<+..Jnb4:'6Ey_]FtFzU9=*l$1p/@gA,Ze>^5<]+r(XJ+m7`/vMDc$'wy|`e Subject: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 16:42:23 -0000 Hi, I recently upgraded from 8.0-STABLE to 8.2-STABLE (Apr. 29 checkout) and upgraded my zpool (includes root FS) from v13 to v15. This is a dual-boot laptop, so I'm using MBR/boot0 and not GPT. Here's what happens when I boot: F1 Win F2 ? F3 FreeBSD F6 PXE Boot: F3 ZFS: unsupported ZFS version 15 (should be 13) No ZFS pools located, can't boot I've googled around, but I can't find anything relevant for MBR/boot0 configurations, just GPT. I've ensured that the loaders and boot0/boot1/boot2 are all new, and I rebuilt/reinstalled them in a fixit environment just to be sure. I also ran 'boot0cfg -B' (with an appropriate -b), but nothing has changed. How can I get my pool booting again? thanks, Jeff From owner-freebsd-stable@FreeBSD.ORG Mon May 2 16:49:31 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E74F1065672; Mon, 2 May 2011 16:49:31 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id A78F18FC14; Mon, 2 May 2011 16:49:30 +0000 (UTC) Received: by vws18 with SMTP id 18so5789696vws.13 for ; Mon, 02 May 2011 09:49:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=TxHeE304kzJAdvU5O+n1fTOTfuqXx6coUcVwMUdxCUo=; b=uc/fNvlvmwqpRXcZSyK/SeR0256dro0hCp83CIZKzbcKIE5RR6KFfCMPrkAZi3M2iR 6H4kHJxwRksRJpmP0ahu52QE58Acd4yXssMbyvZjNwSpZLNVM89ia/Xe6Q1lMCjgoNFk pBMAJqSFwh5nGELSVRF/QxYspgErwDq+aT1dQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=MaD3j2oHjMkrhDL1QQoBcs0L/8zRRGi2WRWIOaz0f892//HpdcFwkP3JJcaUT1Tk5U 7waHI42KlUkmdLmDR1c4k6na8Oi02zrddJ3IzvmmiRjBEXL+qXqZWRWtW47soc+z2uZk uvQXTs+f3pndv0Jp0ug3cMG6Fzx8Iv1VMTcJs= MIME-Version: 1.0 Received: by 10.52.180.234 with SMTP id dr10mr3175488vdc.124.1304354969545; Mon, 02 May 2011 09:49:29 -0700 (PDT) Received: by 10.52.184.169 with HTTP; Mon, 2 May 2011 09:49:29 -0700 (PDT) In-Reply-To: <201105021001.05568.jhb@freebsd.org> References: <201105021001.05568.jhb@freebsd.org> Date: Mon, 2 May 2011 09:49:29 -0700 Message-ID: From: Jack Vogel To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, Wiktor Niesiobedzki Subject: Re: No data received with Intel Corporation Gigabit CT Desktop Adapter (82574L) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 16:49:31 -0000 Thanks John. Was gonna say... the code has been as it is forever, with everyone else in the world working fine, figured something odd was going on. Jack On Mon, May 2, 2011 at 7:01 AM, John Baldwin wrote: > On Saturday, April 30, 2011 2:42:11 am Wiktor Niesiobedzki wrote: > > 2011/4/29 Wiktor Niesiobedzki : > > > 2011/4/28 Jack Vogel : > > >> On Thu, Apr 28, 2011 at 2:28 PM, John Baldwin > wrote: > > >>> > > >>> On Thursday, April 28, 2011 5:17:11 pm Wiktor Niesiobedzki wrote: > > >>> > Though they mention that HT MSI windows is disabled. I'm not sure, > > >>> > whether this matters. > > >>> > > >>> Yes, that is probably what breaks this. > > >>> > > >>> -- > > >>> John Baldwin > > >> > > >> Opps, missed that, thanks John. So, disable MSIX and MSI using > sysctl, > > >> then the driver should use legacy when it loads. > > >> > > >> Still, I'd get a different motherboard, sucks to not have MSIX :( > > >> > > > > > > Thanks for hints. I've disabled MSIX and MSI: > > > kadlubek# sysctl hw.pci | grep msi > > > hw.pci.honor_msi_blacklist: 1 > > > hw.pci.enable_msix: 0 > > > hw.pci.enable_msi: 0 > > > > > > > Ok, I found other way round about this. I've did some source code > > reading and found following tunable: > > hw.em.enable_msix=0 > > > > When set in loader.conf to 0, then the card magically starts to work > > properly. The only thing in our code in em_setup_msix(), that raises > > my doubts, is the following code path: > > > > > > int rid = PCIR_BAR(EM_MSIX_BAR); > > adapter->msix_mem = bus_alloc_resource_any(dev, > > SYS_RES_MEMORY, &rid, RF_ACTIVE); > > ... > > bus_release_resource(dev, SYS_RES_MEMORY, > > PCIR_BAR(EM_MSIX_BAR), adapter->msix_mem); > > > > Though manpage for bus_release_resource specifies, that rid needs to > > be exactly the same, as this returned by bus_alloc_resource. > > In practice the rid is not changed for PCI resources, so the code is fine. > > > Changing the bus_release_resource to use rid instead of > > PCIR_BAR(EM_MSIX_BAR) makes the card working, with sysctl settings: > > hw.pci.enable_msix: 0 > > hw.pci.enable_msi: 0 > > > > instead of hw.em.enable_msix=0 > > > > The only thing that worries me, that when I don't have MSIX disabled > > (anyway), then driver succeeds with the MSI-X allocation. Shouldn't we > > in em_setup_msix check, how many vectors we have allocated with > > pci_alloc_msix and if this is 0, then fallback to MSI/Legacy? > > > > Or maybe pci_alloc_msix should report an error, when no > > PCIB_ALLOC_MSIX succeded? > > Well, the problem here is that PCIB_ALLOC_MSIX() worked fine. There is > another bug that is breaking MSI in your system that I need to finish the > fix for in 9 before it can be MFC'd. > > -- > John Baldwin > From owner-freebsd-stable@FreeBSD.ORG Mon May 2 17:57:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from [127.0.0.1] (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by hub.freebsd.org (Postfix) with ESMTP id 6B0F3106566C; Mon, 2 May 2011 17:57:04 +0000 (UTC) (envelope-from jkim@FreeBSD.org) From: Jung-uk Kim To: Bruce Cran Date: Mon, 2 May 2011 13:56:47 -0400 User-Agent: KMail/1.6.2 References: <20110501042028.GA87381@icarus.home.lan> <20110502154849.0000746b@unknown> In-Reply-To: <20110502154849.0000746b@unknown> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201105021356.49585.jkim@FreeBSD.org> Cc: jeff@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: Is machdep.cpu_idle_hlt deprecated? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 17:57:05 -0000 On Monday 02 May 2011 10:48 am, Bruce Cran wrote: > On Sat, 30 Apr 2011 21:20:28 -0700 > > Jeremy Chadwick wrote: > > Anyone know if machdep.cpu_idle_hlt still exists? Taken from > > acpi(4) on RELENG_8: > > It looks like it might have been replaced by machdep.idle: > > machdep.idle: currently selected idle function > machdep.idle_available: list of available idle functions > > machdep.idle: acpi > machdep.idle_available: spin, hlt, acpi, It seems "machdep.cpu_idle_hlt" was deprecated long ago with this commit by jeff (CC'ed): http://svnweb.freebsd.org/base?view=revision&revision=178471 Jung-uk Kim From owner-freebsd-stable@FreeBSD.ORG Mon May 2 23:19:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D35E106566C for ; Mon, 2 May 2011 23:19:56 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 443BC8FC1B for ; Mon, 2 May 2011 23:19:55 +0000 (UTC) Received: by iwn33 with SMTP id 33so7079701iwn.13 for ; Mon, 02 May 2011 16:19:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=NTUdbTemRMP+tzPmCtIPOgcsedYq6nIV8W1TUgjR4HM=; b=VvODzPWnThbjb3xkyAOndUrbIIuhrLGCeoGqsrrugDvFnFpEB69ArNqU4Ab+RpkRt5 CxjRpD+gleQnE7/mlm2SGmVx8z9hotk7JtLcpYyCKvw2c7TURSAFG13rjupznBAOReJT q7KSl/I4dag2vbP0jtcCxUlDCsHlKYXv+7lQw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=gKwi7BIU7VAAIbhQZqPDE4wh0L+9LhhCp3wia9JU0iWv6W9UKO0G9qbSRsITAzWpSV K2hqmakm8GPt0lWX+b0TOpFSHLttoNZoN8EqCCMaRQFjRMIxVKLclH+byQnc4D+NLs2S HWbVddEttGTpDQSqC9IKOCBR9MM8V4bnC8qUk= Received: by 10.42.52.133 with SMTP id j5mr582882icg.348.1304378395465; Mon, 02 May 2011 16:19:55 -0700 (PDT) Received: from DataIX.net ([99.190.84.116]) by mx.google.com with ESMTPS id u17sm2507263ibm.28.2011.05.02.16.19.53 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 02 May 2011 16:19:54 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p42NJobP078470 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 May 2011 19:19:51 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p42NJm38078469; Mon, 2 May 2011 19:19:48 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Mon, 2 May 2011 19:19:48 -0400 From: Jason Hellenthal To: Jeff Blank Message-ID: <20110502231947.GA53414@DataIX.net> References: <20110502164222.GE81300@mr-happy.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: <20110502164222.GE81300@mr-happy.com> X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-stable@freebsd.org Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 23:19:56 -0000 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Jeff, On Mon, May 02, 2011 at 12:42:22PM -0400, Jeff Blank wrote: >Hi, > >I recently upgraded from 8.0-STABLE to 8.2-STABLE (Apr. 29 checkout) >and upgraded my zpool (includes root FS) from v13 to v15. This is a >dual-boot laptop, so I'm using MBR/boot0 and not GPT. Here's what >happens when I boot: > >F1 Win >F2 ? >F3 FreeBSD > >F6 PXE >Boot: F3 >ZFS: unsupported ZFS version 15 (should be 13) >No ZFS pools located, can't boot > This has been discussed thoroughly in the past. You need to update your=20 bootcode. You can use gpart(8) with the bootcode option to do this or your= =20 standard preferred way. You will also have to do this same step when v28 makes it into the code. Good Luck --=20 Regards, (jhell) Jason Hellenthal --uAKRQypu60I7Lcqm Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNvzwTAAoJEJBXh4mJ2FR+W+UH/2YWbv5nJDsR4U+BoeKxL6o/ v/0NEU0aH0fzV8A6gobKtsqFsfkcX5ckeXw48TZZusaAmqEApNDDpV4YwJaQc4uU cFcl3ayW3ZOaM6Wf/ObPTCsHEXywybHWh/6UTTMUY+K8v5pM69WOjvTD5tvNHDy5 N7aO4+UUZ9PMWMwsTOY2wDmVGvBvaXB9ZY3XxtQgYxC2zj5dZod8/+fbNb9qWCCV EqWhqN/gPiJ8WK4gKUbcSyvTI1qcV6bn6wxGnIXwSNmPAFqIKA1sBawg6imdrfpk K5dEqobv+JHjUvhC/lTyP/U908KKtaxvDIa7qeszX0YQeb+NXgdej+zey+4tw1c= =rVgB -----END PGP SIGNATURE----- --uAKRQypu60I7Lcqm-- From owner-freebsd-stable@FreeBSD.ORG Tue May 3 01:58:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65C5D1065670 for ; Tue, 3 May 2011 01:58:59 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id C95F28FC13 for ; Tue, 3 May 2011 01:58:58 +0000 (UTC) Received: from omta21.westchester.pa.mail.comcast.net ([76.96.62.72]) by qmta09.westchester.pa.mail.comcast.net with comcast id epse1g0011ZXKqc59pyy8Z; Tue, 03 May 2011 01:58:58 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta21.westchester.pa.mail.comcast.net with comcast id epyw1g00d1t3BNj3hpyxTl; Tue, 03 May 2011 01:58:58 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id E58B49B418; Mon, 2 May 2011 18:58:54 -0700 (PDT) Date: Mon, 2 May 2011 18:58:54 -0700 From: Jeremy Chadwick To: freebsd-pf@freebsd.org Message-ID: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 01:58:59 -0000 (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies for cross-posting, but the issue is severe enough that I wanted to make it known on -stable) The below issue I'm describing is from a machine running 8.2-PRERELEASE (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. Please read the story in full, as I have taken the time to describe everything I did, plus log output, as well as induce a panic via "call doadump" from ddb so I have a capture of the system at the time. I also have a theory as to what caused the problem, but how to trigger it is unknown; it may be a rare race condition. This morning I woke up to find a report from one of our users that he could not connect to a specific TCP port (not SSH) on one of our servers. I also found that I couldn't SSH into the same box. Serial console was working fine, and the serial console log showed no sign of any problems. I started to debug the issue of me not being able to SSH into the machine and within a few minutes became immediately concerned: pfctl indicated we had reached the maximum number permitted state table entries (10,000). ============================================================ # pfctl -s info Status: Enabled for 76 days 06:49:10 Debug: Urgent Interface Stats for em0 IPv4 IPv6 Bytes In 8969748840 0 Bytes Out 8296135477 0 Packets In Passed 128211763 0 Blocked 621379 0 Packets Out Passed 138483868 0 Blocked 2579 0 State Table Total Rate current entries 10000 searches 267316807 40.6/s inserts 4440553 0.7/s removals 4430553 0.7/s Counters match 5067474 0.8/s bad-offset 0 0.0/s fragment 324 0.0/s short 0 0.0/s normalize 32 0.0/s memory 336946 0.1/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 1611 0.0/s state-mismatch 509 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -s memory states hard limit 10000 src-nodes hard limit 10000 frags hard limit 5000 tables hard limit 1000 table-entries hard limit 100000 ============================================================ The above is mainly for em0 (our WAN interface); our LAN interface (em1) was not impacted because we use "set skip on em1". And it's a good thing too: we have lots of LAN-based services that this machine provides that would have been impacted. We also use "set skip on lo0". I immediately went to look at our monitoring graphs, which monitor pf state (specifically state table entries), polled via bsnmpd(8). This data is even more frightening: http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png Literally something was spiraling out of control, starting at approx. 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. 19:45 PDT the same day, but I wasn't aware of it until said user brought an issue to my attention. You can see from the network I/O graphs (taken from SNMP polling our switch, NOT from the host/box itself) that there was no DoS attack or anything like that occurring -- this was something within FreeBSD itself. More evidence of that will become apparent. http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png The first thing I did was "/etc/rc.d/pf reload". This command hung. Any attempt to send Ctrl-C/SIGINT did nothing. I was able to Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did not truly die (despite csh stating "Terminated"). The only way to kill it was to kill -9. Attempts to shut down any daemons which utilised the network -- including things that only used em1 -- would not shut down. This included things like postfix, mysqld, and some inet-based services. I was forced to kill -9 them. Things like bsnmpd, however, did shut down. Equally as uncomfortable, "shutdown -r now" did not reboot the system. That is to say, wall(1)'s announcement was shown, but the actual stopping of services did not begin. The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I did "/etc/rc.d/pf start", which also worked. However, what I saw next surely indicated a bug in the pf layer somewhere -- "pfctl -s states" and "pfctl -s info" disagreed on the state count: ============================================================ # pfctl -s info Status: Enabled for 0 days 00:00:16 Debug: Urgent Interface Stats for em0 IPv4 IPv6 Bytes In 3459 0 Bytes Out 0 0 Packets In Passed 0 0 Blocked 29 0 Packets Out Passed 0 0 Blocked 0 0 State Table Total Rate current entries 10000 searches 29 1.8/s inserts 0 0.0/s removals 0 0.0/s Counters match 29 1.8/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 18 1.1/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -s state | wc -l 0 ============================================================ The "pf uptime" shown above, by the way, matches system uptime. I then attempted "pfctl -F state", but nothing changed (looked the same as above). Since I could not reboot the box, I was forced to drop to ddb via serial console. I did some commands like "ps" and the like, and then "call doadump" to induce a kernel panic, and then "reboot" (which worked). Once the machine came back up, savecore(8) ran, wrote the data out, and everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and I do not feel comfortable sharing its content publicly, but will be happy to hand it to developer(s) who are interested. Relevant tidbits I can discern: ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] ------------------------------------------------------------------------ ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES pfsrctrpl: 152, 10000, 0, 0, 0, 0 pfrulepl: 912, 0, 40, 88, 806, 0 pfstatepl: 392, 10000, 10000, 0, 4440553, 341638 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 0, 0, 0, 0 pfrktable: 1296, 1002, 4, 20, 112, 0 pfrkentry: 216, 100008, 603, 891, 15384, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 303, 1620, 0 pffrag: 80, 0, 0, 135, 807, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 0, 0, 0, 0 pfospfen: 112, 0, 696, 30, 18096, 0 pfosfp: 40, 0, 407, 97, 10582, 0 ------------------------------------------------------------------------ You can see evidence of processes not exiting/doing what they should do here: ------------------------------------------------------------------------ fstat USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root shutdown 91155 root / 2 drwxr-xr-x 512 r root shutdown 91155 wd / 2 drwxr-xr-x 512 r root shutdown 91155 text / 47195 -r-sr-x--- 15912 r root shutdown 91155 0 /dev 38 crw------- ttyu0 rw root shutdown 91155 1 /dev 38 crw------- ttyu0 rw root shutdown 91155 2 /dev 38 crw------- ttyu0 rw root sh 91129 root / 2 drwxr-xr-x 512 r root sh 91129 wd / 2 drwxr-xr-x 512 r root sh 91129 text / 44 -r-xr-xr-x 134848 r root sh 91129 0 /dev 38 crw------- ttyu0 rw root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 0 rw root sh 91129 2 /dev 20 crw-rw-rw- null w root shutdown 91115 root / 2 drwxr-xr-x 512 r root shutdown 91115 wd /storage 5 drwx------ 37 r root shutdown 91115 text / 47195 -r-sr-x--- 15912 r root shutdown 91115 0 /dev 38 crw------- ttyu0 rw root shutdown 91115 1 /dev 38 crw------- ttyu0 rw root shutdown 91115 2 /dev 38 crw------- ttyu0 rw root shutdown 91115 3* local dgram ffffff008ff92960 root sh 90818 root / 2 drwxr-xr-x 512 r root sh 90818 wd / 70659 drwxr-xr-x 512 r root sh 90818 text / 44 -r-xr-xr-x 134848 r root sh 90818 0 /dev 38 crw------- ttyu0 rw root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 0 rw root sh 90818 2 /dev 20 crw-rw-rw- null w root csh 90802 root / 2 drwxr-xr-x 512 r root csh 90802 wd / 2 drwxr-xr-x 512 r root csh 90802 text / 51 -r-xr-xr-x 358752 r root csh 90802 15 /dev 38 crw------- ttyu0 rw root csh 90802 16 /dev 38 crw------- ttyu0 rw root csh 90802 17 /dev 38 crw------- ttyu0 rw root csh 90802 18 /dev 38 crw------- ttyu0 rw root csh 90802 19 /dev 38 crw------- ttyu0 rw ------------------------------------------------------------------------ No indication of mbuf exhaustion, putting further focus on the pf stack: ------------------------------------------------------------------------ netstat -m 2054/1786/3840 mbufs in use (current/cache/total) 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) 0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) 4609K/4554K/9164K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ------------------------------------------------------------------------ Here's one piece of core.0.txt which makes no sense to me -- the "rate" column. I have a very hard time believing that was the interrupt rate of all the relevant devices at the time (way too high). Maybe this data becomes wrong only during a coredump? The total column I could believe. ------------------------------------------------------------------------ vmstat -i interrupt total rate irq4: uart0 54768 912 irq6: fdc0 1 0 irq17: uhci1+ 172 2 irq23: uhci3 ehci1+ 2367 39 cpu0: timer 13183882632 219731377 irq256: em0 260491055 4341517 irq257: em1 127555036 2125917 irq258: ahci0 225923164 3765386 cpu2: timer 13183881837 219731363 cpu1: timer 13002196469 216703274 cpu3: timer 13183881783 219731363 Total 53167869284 886131154 ------------------------------------------------------------------------ Here's what a normal "vmstat -i" shows from the command-line: # vmstat -i interrupt total rate irq4: uart0 518 0 irq6: fdc0 1 0 irq23: uhci3 ehci1+ 145 0 cpu0: timer 19041199 1999 irq256: em0 614280 64 irq257: em1 168529 17 irq258: ahci0 355536 37 cpu2: timer 19040462 1999 cpu1: timer 19040458 1999 cpu3: timer 19040454 1999 Total 77301582 8119 We graph many aspects of this box, including CPU load, memory/swap usage, etc. and none show any sign that the interrupt rate on all of those devices was even remotely out of control. (I would expect to see CPU through the roof given the above data) I have since rebuilt/reinstalled world/kernel on the machine with the latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT 2011), hoping whatever this was may have been fixed. As for what I think may have triggered it, but I have no hard evidence of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf reload". The pf.conf change was a single line: Old: scrub on em0 all New: scrub in on em0 all Why it took the problem approximately 3 days to start is unknown. It's the only change we've made to the system (truly/honestly), and it was a change to pf.conf. If anyone has advice (or has seen the above problem), or is interested in debugging it -- as I said, I have a vmcore -- I'm happy to assist in any way I can. I would hate for someone else to get bit by this, and really am hoping its something that has been fixed between February and now. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 03:47:39 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75C49106566B for ; Tue, 3 May 2011 03:47:39 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 3459A8FC0A for ; Tue, 3 May 2011 03:47:38 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id p433lcHu052724; Mon, 2 May 2011 21:47:38 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id p433lbfl052723; Mon, 2 May 2011 21:47:37 -0600 (MDT) (envelope-from ken) Date: Mon, 2 May 2011 21:47:37 -0600 From: "Kenneth D. Merry" To: Dmitry Morozovsky Message-ID: <20110503034737.GA52416@nargothrond.kdm.org> References: <20110430211927.GA67374@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: freebsd-stable@FreeBSD.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 03:47:39 -0000 On Sun, May 01, 2011 at 14:42:21 +0400, Dmitry Morozovsky wrote: > On Sat, 30 Apr 2011, Kenneth D. Merry wrote: > > KDM> On Fri, Apr 29, 2011 at 11:51:21 +0400, Dmitry Morozovsky wrote: > KDM> > Dear Ken, > KDM> > > KDM> > I have SuperMicro Server with mps driver you managed, with 24 SATA disks under > KDM> > SAS x36 expander with large ZFS > KDM> > > KDM> > Sometimes, under random disk load such as daily find, it lost all its devices: > KDM> > > KDM> > [-- MARK -- Fri Apr 29 03:00:00 2011] > KDM> > mps0: IOC Fault 0x40005900, Resetting^M > KDM> > (pass20:mps0:0:22:0): SCSI command timeout on device handle 0x0020 SMID 442^M > KDM> > mps0: IOC Fault 0x40001500, Resetting^M > KDM> > (da19:mps0:0:21:0): SCSI command timeout on device handle 0x001f SMID 172^M > KDM> > (da19:mps0:0:21:0): SCSI command timeout on device handle 0x001f SMID 511^M > KDM> > (da20:mps0:0:20:0): SCSI command timeout on device handle 0x001e SMID 240^M > KDM> > > KDM> > .. > KDM> > > KDM> > (da4:mps0:0:0:0): SCSI command timeout on device handle 0x000a SMID 844^M > KDM> > (da22:mps0:0:23:0): SCSI command timeout on device handle 0x0021 SMID 713^M > KDM> > (da18:mps0:0:22:0): SCSI command timeout on device handle 0x0020 SMID 603^M > KDM> > > KDM> > and hangs there forever (in zio state). > KDM> > > KDM> > I've prepared debugging kernel with DDB and would be glad to help catch the > KDM> > situation. > KDM> > KDM> Hmm... > KDM> > KDM> Can you send full dmesg output? > > Attached Thanks. It looks like you have a SAS2008, with the 4.0 firmware. I think it would be worthwhile to upgrade to the 9.0 firmware. I know for sure there are issues with the 2.0 firmware, and I know the 9.0 firmware works fairly well. I don't know whether the 4.0 firmware has any severe issues, but it would be good to eliminate firmware bugs before we chase driver issues. > KDM> What I'm most interested in is whether > KDM> there is more kernel output before the IOC Fault that might shed some light > KDM> on what is going on. > > Nope. I use boot_verbose, but none of mps-related debug options yet Okay. If there's nothing before the IOC fault message, then we really don't have any clues to what caused the fault... The rest is just fallout from the IOC fault. > KDM> > KDM> Also, what brand (LSI, Maxim, etc.) and speed (3Gb, 6Gb) is the expander on > KDM> the backplane? > > LSI 6G: Okay. > KDM> What model LSI controller do you have? How many lanes are connected > KDM> between the controller and the backplane? > > 2x4 IIR. BTW, how can investigate real SASA topology? So 8 lanes total? That's what I wanted to know. The primary thing I'm getting at is to see how much lane contention we may have. With 24 SATA disks, you can only talk to 8 at a time with 8 lanes connected from the controller to the backplane. I've run into issues with a lot of contention with SATA drives, but that was with a 3Gb Maxim expander. In theory things should work better with an LSI expander. (You would think that they test scenarios like yours.) > KDM> What model disks do you have in the system? (dmesg will show that > KDM> obviously.) > > 24 x WD RE4 2T Ok. My SATA testing has been primarily with WD 2TB drives as well. > KDM> Hopefully we can find some clues to point to the problem. > > /me too ;) > > Thank you very much! > > BTW, I have serial console, DDB kernel, so while this machine is in > production, but not too heavy, and I can spend some time in kernel debugger if > needed. Well, I think the first thing to do is upgrade the firmware and see if that fixes it. If not, we'll start instrumenting things and see how much information we can get about the cause of the fault. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-stable@FreeBSD.ORG Tue May 3 05:06:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 811A4106564A; Tue, 3 May 2011 05:06:41 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 282878FC08; Tue, 3 May 2011 05:06:40 +0000 (UTC) Received: by iyj12 with SMTP id 12so7359715iyj.13 for ; Mon, 02 May 2011 22:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=neU4BmS9TB85aasLW7PY2eL/gr8SmoXox+uWqdd219o=; b=UFMfL3ZwWcOm4C4KIVQMmCYL9E24Mtg9xm+1mroVu3yHONU6Gh376zyAa7NkT8Av3I VQ2cosaeD/tBMgwZYFdzKQUc70dPf+4Wi5MkwHFr8avLJ9hco9L+qwMyy3kd22ZCEnI1 ndl9tfQHAuU9wvQQwPYnO+F7wH+BFOZ2PeeVU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=KoeOo1hQYNSzv1kFi6NZFI/gtphI38eqsxvpfxXprI03NSYf8norXJmq2SoY6U8Spw hbneoKdZpilfCQ6GidEFKHZFY27RGwN5HiLccoUToxHQFlMjoz/+9ODRep0ZgXNfu+ji /cyEKbJ7d94ZdLhZOx4JIXGXSRLdSwJ8SnQWI= Received: by 10.42.115.138 with SMTP id k10mr1900703icq.81.1304399200506; Mon, 02 May 2011 22:06:40 -0700 (PDT) Received: from DataIX.net ([99.190.84.116]) by mx.google.com with ESMTPS id vr5sm2395488icb.12.2011.05.02.22.06.37 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 02 May 2011 22:06:38 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p4356YJY018800 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 01:06:35 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p4356YPf018799; Tue, 3 May 2011 01:06:34 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Tue, 3 May 2011 01:06:34 -0400 From: Jason Hellenthal To: Jeremy Chadwick Message-ID: <20110503050634.GB53414@DataIX.net> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DBIVS5p969aUjpLe" Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:06:41 -0000 --DBIVS5p969aUjpLe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Jeremy, On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: >(Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies >for cross-posting, but the issue is severe enough that I wanted to make >it known on -stable) > >The below issue I'm describing is from a machine running 8.2-PRERELEASE >(RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > >Please read the story in full, as I have taken the time to describe >everything I did, plus log output, as well as induce a panic via "call >doadump" from ddb so I have a capture of the system at the time. I also >have a theory as to what caused the problem, but how to trigger it is >unknown; it may be a rare race condition. > > >This morning I woke up to find a report from one of our users that he >could not connect to a specific TCP port (not SSH) on one of our >servers. I also found that I couldn't SSH into the same box. Serial >console was working fine, and the serial console log showed no sign of >any problems. > >I started to debug the issue of me not being able to SSH into the >machine and within a few minutes became immediately concerned: pfctl >indicated we had reached the maximum number permitted state table >entries (10,000). > >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ># pfctl -s info >Status: Enabled for 76 days 06:49:10 Debug: Urgent > >Interface Stats for em0 IPv4 IPv6 > Bytes In 8969748840 0 > Bytes Out 8296135477 0 > Packets In > Passed 128211763 0 > Blocked 621379 0 > Packets Out > Passed 138483868 0 > Blocked 2579 0 > >State Table Total Rate > current entries 10000 > searches 267316807 40.6/s > inserts 4440553 0.7/s > removals 4430553 0.7/s >Counters > match 5067474 0.8/s > bad-offset 0 0.0/s > fragment 324 0.0/s > short 0 0.0/s > normalize 32 0.0/s > memory 336946 0.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 1611 0.0/s > state-mismatch 509 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > ># pfctl -s memory >states hard limit 10000 >src-nodes hard limit 10000 >frags hard limit 5000 >tables hard limit 1000 >table-entries hard limit 100000 >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >The above is mainly for em0 (our WAN interface); our LAN interface (em1) >was not impacted because we use "set skip on em1". And it's a good >thing too: we have lots of LAN-based services that this machine provides >that would have been impacted. We also use "set skip on lo0". > >I immediately went to look at our monitoring graphs, which monitor pf >state (specifically state table entries), polled via bsnmpd(8). This >data is even more frightening: > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png >http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > >Literally something was spiraling out of control, starting at approx. >2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. >19:45 PDT the same day, but I wasn't aware of it until said user brought >an issue to my attention. > >You can see from the network I/O graphs (taken from SNMP polling our >switch, NOT from the host/box itself) that there was no DoS attack or >anything like that occurring -- this was something within FreeBSD >itself. More evidence of that will become apparent. > >http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png >http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > >The first thing I did was "/etc/rc.d/pf reload". This command hung. >Any attempt to send Ctrl-C/SIGINT did nothing. I was able to >Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did >not truly die (despite csh stating "Terminated"). The only way to kill >it was to kill -9. > >Attempts to shut down any daemons which utilised the network -- >including things that only used em1 -- would not shut down. This >included things like postfix, mysqld, and some inet-based services. I >was forced to kill -9 them. Things like bsnmpd, however, did shut down. > >Equally as uncomfortable, "shutdown -r now" did not reboot the system. >That is to say, wall(1)'s announcement was shown, but the actual >stopping of services did not begin. > >The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I >did "/etc/rc.d/pf start", which also worked. However, what I saw next >surely indicated a bug in the pf layer somewhere -- "pfctl -s states" >and "pfctl -s info" disagreed on the state count: > >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ># pfctl -s info >Status: Enabled for 0 days 00:00:16 Debug: Urgent > >Interface Stats for em0 IPv4 IPv6 > Bytes In 3459 0 > Bytes Out 0 0 > Packets In > Passed 0 0 > Blocked 29 0 > Packets Out > Passed 0 0 > Blocked 0 0 > >State Table Total Rate > current entries 10000 > searches 29 1.8/s > inserts 0 0.0/s > removals 0 0.0/s >Counters > match 29 1.8/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 0 0.0/s > memory 18 1.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 0 0.0/s > state-mismatch 0 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > ># pfctl -s state | wc -l > 0 >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >The "pf uptime" shown above, by the way, matches system uptime. > >I then attempted "pfctl -F state", but nothing changed (looked the same >as above). > >Since I could not reboot the box, I was forced to drop to ddb via serial >console. I did some commands like "ps" and the like, and then "call >doadump" to induce a kernel panic, and then "reboot" (which worked). > >Once the machine came back up, savecore(8) ran, wrote the data out, and >everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and >I do not feel comfortable sharing its content publicly, but will be >happy to hand it to developer(s) who are interested. Relevant tidbits I >can discern: > >------------------------------------------------------------------------ >ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00= [pfpurge] >------------------------------------------------------------------------ > >------------------------------------------------------------------------ >vmstat -z > >ITEM SIZE LIMIT USED FREE REQUESTS FAI= LURES >pfsrctrpl: 152, 10000, 0, 0, 0, = 0 >pfrulepl: 912, 0, 40, 88, 806, = 0 >pfstatepl: 392, 10000, 10000, 0, 4440553, 3= 41638 >pfaltqpl: 240, 0, 0, 0, 0, = 0 >pfpooladdrpl: 88, 0, 0, 0, 0, = 0 >pfrktable: 1296, 1002, 4, 20, 112, = 0 >pfrkentry: 216, 100008, 603, 891, 15384, = 0 >pfrkentry2: 216, 0, 0, 0, 0, = 0 >pffrent: 32, 5050, 0, 303, 1620, = 0 >pffrag: 80, 0, 0, 135, 807, = 0 >pffrcache: 80, 10035, 0, 0, 0, = 0 >pffrcent: 24, 50022, 0, 0, 0, = 0 >pfstatescrub: 40, 0, 0, 0, 0, = 0 >pfiaddrpl: 120, 0, 0, 0, 0, = 0 >pfospfen: 112, 0, 696, 30, 18096, = 0 >pfosfp: 40, 0, 407, 97, 10582, = 0 >------------------------------------------------------------------------ > >You can see evidence of processes not exiting/doing what they should do >here: > >------------------------------------------------------------------------ >fstat > >USER CMD PID FD MOUNT INUM MODE SZ|DV R/W >root shutdown 91155 root / 2 drwxr-xr-x 512 r >root shutdown 91155 wd / 2 drwxr-xr-x 512 r >root shutdown 91155 text / 47195 -r-sr-x--- 15912 r >root shutdown 91155 0 /dev 38 crw------- ttyu0 rw >root shutdown 91155 1 /dev 38 crw------- ttyu0 rw >root shutdown 91155 2 /dev 38 crw------- ttyu0 rw >root sh 91129 root / 2 drwxr-xr-x 512 r >root sh 91129 wd / 2 drwxr-xr-x 512 r >root sh 91129 text / 44 -r-xr-xr-x 134848 r >root sh 91129 0 /dev 38 crw------- ttyu0 rw >root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888= 0 rw >root sh 91129 2 /dev 20 crw-rw-rw- null w >root shutdown 91115 root / 2 drwxr-xr-x 512 r >root shutdown 91115 wd /storage 5 drwx------ 37 r >root shutdown 91115 text / 47195 -r-sr-x--- 15912 r >root shutdown 91115 0 /dev 38 crw------- ttyu0 rw >root shutdown 91115 1 /dev 38 crw------- ttyu0 rw >root shutdown 91115 2 /dev 38 crw------- ttyu0 rw >root shutdown 91115 3* local dgram ffffff008ff92960 >root sh 90818 root / 2 drwxr-xr-x 512 r >root sh 90818 wd / 70659 drwxr-xr-x 512 r >root sh 90818 text / 44 -r-xr-xr-x 134848 r >root sh 90818 0 /dev 38 crw------- ttyu0 rw >root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60= 0 rw >root sh 90818 2 /dev 20 crw-rw-rw- null w >root csh 90802 root / 2 drwxr-xr-x 512 r >root csh 90802 wd / 2 drwxr-xr-x 512 r >root csh 90802 text / 51 -r-xr-xr-x 358752 r >root csh 90802 15 /dev 38 crw------- ttyu0 rw >root csh 90802 16 /dev 38 crw------- ttyu0 rw >root csh 90802 17 /dev 38 crw------- ttyu0 rw >root csh 90802 18 /dev 38 crw------- ttyu0 rw >root csh 90802 19 /dev 38 crw------- ttyu0 rw >------------------------------------------------------------------------ > >No indication of mbuf exhaustion, putting further focus on the pf stack: > >------------------------------------------------------------------------ >netstat -m > >2054/1786/3840 mbufs in use (current/cache/total) >2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) >2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) >0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/= max) >0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) >0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) >4609K/4554K/9164K bytes allocated to network (current/cache/total) >0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >0/0/0 requests for jumbo clusters denied (4k/9k/16k) >0 requests for sfbufs denied >0 requests for sfbufs delayed >0 requests for I/O initiated by sendfile >0 calls to protocol drain routines >------------------------------------------------------------------------ > >Here's one piece of core.0.txt which makes no sense to me -- the "rate" >column. I have a very hard time believing that was the interrupt rate >of all the relevant devices at the time (way too high). Maybe this data >becomes wrong only during a coredump? The total column I could believe. > >------------------------------------------------------------------------ >vmstat -i > >interrupt total rate >irq4: uart0 54768 912 >irq6: fdc0 1 0 >irq17: uhci1+ 172 2 >irq23: uhci3 ehci1+ 2367 39 >cpu0: timer 13183882632 219731377 >irq256: em0 260491055 4341517 >irq257: em1 127555036 2125917 >irq258: ahci0 225923164 3765386 >cpu2: timer 13183881837 219731363 >cpu1: timer 13002196469 216703274 >cpu3: timer 13183881783 219731363 >Total 53167869284 886131154 >------------------------------------------------------------------------ > >Here's what a normal "vmstat -i" shows from the command-line: > ># vmstat -i >interrupt total rate >irq4: uart0 518 0 >irq6: fdc0 1 0 >irq23: uhci3 ehci1+ 145 0 >cpu0: timer 19041199 1999 >irq256: em0 614280 64 >irq257: em1 168529 17 >irq258: ahci0 355536 37 >cpu2: timer 19040462 1999 >cpu1: timer 19040458 1999 >cpu3: timer 19040454 1999 >Total 77301582 8119 > >We graph many aspects of this box, including CPU load, memory/swap >usage, etc. and none show any sign that the interrupt rate on all of >those devices was even remotely out of control. (I would expect to see >CPU through the roof given the above data) > >I have since rebuilt/reinstalled world/kernel on the machine with the >latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT >2011), hoping whatever this was may have been fixed. > >As for what I think may have triggered it, but I have no hard evidence >of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf >reload". The pf.conf change was a single line: > >Old: scrub on em0 all >New: scrub in on em0 all > >Why it took the problem approximately 3 days to start is unknown. It's >the only change we've made to the system (truly/honestly), and it was a >change to pf.conf. > >If anyone has advice (or has seen the above problem), or is interested >in debugging it -- as I said, I have a vmcore -- I'm happy to assist in >any way I can. I would hate for someone else to get bit by this, and >really am hoping its something that has been fixed between February and >now. > That's quite the deduction there. I've noticed recently that you were also= =20 experimenting with the new NFS server recompiling kernel etc etc. Seeing=20 as weird things can happen with DNS, NFS and mountpoint's, is this the same= =20 machine that you were doing that on ? If so can you check to see how many requests for NFS operations were done= =20 to/from that box ? as well the names that would be being resolved and if=20 that machine can resolve them ? Also I would believe your using tables in your pf.conf, if so do any of=20 those tables contain a FQDN that cannot be resolved from that machine ? I think you probably see what I am getting at here as it could be some=20 sort of concurrent recursive DNS failure that can only be seen from the=20 machine caused by possibly the new NFS backend or a change in one of the=20 tables that pf would use. --=20 Regards, (jhell) Jason Hellenthal --DBIVS5p969aUjpLe Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNv41ZAAoJEJBXh4mJ2FR+kv0H/jLwWK5wBbRHO0t4PEeUTITN pKljwFvrnTSTDC1WIeYAe6AEL5zERfHaYsl2zGG2XiYCiVoSOaVRRYP/4lkztG3h X3K68z0q8WgTksr9L1iIR/oj2YRLGzvTI9ChV2lQh+lORF2mLW+doYlipPa0sAGq PJrruYlQk5uaY/9rEQn+/5QxvFM1A73yBVIoQKI4UAEdlv8tGlFAcyrqkOGnJ8ju ji6VOQfOndbwI6B+0rezkvtFGlSjPQTCVsl8+IAkA2rgcUPkGgOm9LxGFHsVzbc+ Kkj9zRKCSDc92qVKyw7+Asaucnm6lnhF/IHDYgvVPIGcG/eLi/PvVnbh003svmg= =xLPk -----END PGP SIGNATURE----- --DBIVS5p969aUjpLe-- From owner-freebsd-stable@FreeBSD.ORG Tue May 3 05:45:33 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 185BA106566C for ; Tue, 3 May 2011 05:45:33 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id F1C858FC12 for ; Tue, 3 May 2011 05:45:31 +0000 (UTC) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta08.emeryville.ca.mail.comcast.net with comcast id etje1g0060FhH24A8tlXCs; Tue, 03 May 2011 05:45:31 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta08.emeryville.ca.mail.comcast.net with comcast id etlW1g00D1t3BNj8UtlWNj; Tue, 03 May 2011 05:45:31 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id EAD8F9B418; Mon, 2 May 2011 22:45:29 -0700 (PDT) Date: Mon, 2 May 2011 22:45:29 -0700 From: Jeremy Chadwick To: Jason Hellenthal Message-ID: <20110503054529.GA35688@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503050634.GB53414@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503050634.GB53414@DataIX.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:45:33 -0000 On Tue, May 03, 2011 at 01:06:34AM -0400, Jason Hellenthal wrote: > On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > >(Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > >for cross-posting, but the issue is severe enough that I wanted to make > >it known on -stable) > > > >The below issue I'm describing is from a machine running 8.2-PRERELEASE > >(RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > >Please read the story in full, as I have taken the time to describe > >everything I did, plus log output, as well as induce a panic via "call > >doadump" from ddb so I have a capture of the system at the time. I also > >have a theory as to what caused the problem, but how to trigger it is > >unknown; it may be a rare race condition. > > > > > >This morning I woke up to find a report from one of our users that he > >could not connect to a specific TCP port (not SSH) on one of our > >servers. I also found that I couldn't SSH into the same box. Serial > >console was working fine, and the serial console log showed no sign of > >any problems. > > > >I started to debug the issue of me not being able to SSH into the > >machine and within a few minutes became immediately concerned: pfctl > >indicated we had reached the maximum number permitted state table > >entries (10,000). > > > >============================================================ > ># pfctl -s info > >Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > >Interface Stats for em0 IPv4 IPv6 > > Bytes In 8969748840 0 > > Bytes Out 8296135477 0 > > Packets In > > Passed 128211763 0 > > Blocked 621379 0 > > Packets Out > > Passed 138483868 0 > > Blocked 2579 0 > > > >State Table Total Rate > > current entries 10000 > > searches 267316807 40.6/s > > inserts 4440553 0.7/s > > removals 4430553 0.7/s > >Counters > > match 5067474 0.8/s > > bad-offset 0 0.0/s > > fragment 324 0.0/s > > short 0 0.0/s > > normalize 32 0.0/s > > memory 336946 0.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 1611 0.0/s > > state-mismatch 509 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > ># pfctl -s memory > >states hard limit 10000 > >src-nodes hard limit 10000 > >frags hard limit 5000 > >tables hard limit 1000 > >table-entries hard limit 100000 > >============================================================ > > > >The above is mainly for em0 (our WAN interface); our LAN interface (em1) > >was not impacted because we use "set skip on em1". And it's a good > >thing too: we have lots of LAN-based services that this machine provides > >that would have been impacted. We also use "set skip on lo0". > > > >I immediately went to look at our monitoring graphs, which monitor pf > >state (specifically state table entries), polled via bsnmpd(8). This > >data is even more frightening: > > > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > >Literally something was spiraling out of control, starting at approx. > >2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > >19:45 PDT the same day, but I wasn't aware of it until said user brought > >an issue to my attention. > > > >You can see from the network I/O graphs (taken from SNMP polling our > >switch, NOT from the host/box itself) that there was no DoS attack or > >anything like that occurring -- this was something within FreeBSD > >itself. More evidence of that will become apparent. > > > >http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > >http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > >The first thing I did was "/etc/rc.d/pf reload". This command hung. > >Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > >Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > >not truly die (despite csh stating "Terminated"). The only way to kill > >it was to kill -9. > > > >Attempts to shut down any daemons which utilised the network -- > >including things that only used em1 -- would not shut down. This > >included things like postfix, mysqld, and some inet-based services. I > >was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > > >Equally as uncomfortable, "shutdown -r now" did not reboot the system. > >That is to say, wall(1)'s announcement was shown, but the actual > >stopping of services did not begin. > > > >The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > >did "/etc/rc.d/pf start", which also worked. However, what I saw next > >surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > >and "pfctl -s info" disagreed on the state count: > > > >============================================================ > ># pfctl -s info > >Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > >Interface Stats for em0 IPv4 IPv6 > > Bytes In 3459 0 > > Bytes Out 0 0 > > Packets In > > Passed 0 0 > > Blocked 29 0 > > Packets Out > > Passed 0 0 > > Blocked 0 0 > > > >State Table Total Rate > > current entries 10000 > > searches 29 1.8/s > > inserts 0 0.0/s > > removals 0 0.0/s > >Counters > > match 29 1.8/s > > bad-offset 0 0.0/s > > fragment 0 0.0/s > > short 0 0.0/s > > normalize 0 0.0/s > > memory 18 1.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 0 0.0/s > > state-mismatch 0 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > ># pfctl -s state | wc -l > > 0 > >============================================================ > > > >The "pf uptime" shown above, by the way, matches system uptime. > > > >I then attempted "pfctl -F state", but nothing changed (looked the same > >as above). > > > >Since I could not reboot the box, I was forced to drop to ddb via serial > >console. I did some commands like "ps" and the like, and then "call > >doadump" to induce a kernel panic, and then "reboot" (which worked). > > > >Once the machine came back up, savecore(8) ran, wrote the data out, and > >everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > >I do not feel comfortable sharing its content publicly, but will be > >happy to hand it to developer(s) who are interested. Relevant tidbits I > >can discern: > > > >------------------------------------------------------------------------ > >ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] > >------------------------------------------------------------------------ > > > >------------------------------------------------------------------------ > >vmstat -z > > > >ITEM SIZE LIMIT USED FREE REQUESTS FAILURES > >pfsrctrpl: 152, 10000, 0, 0, 0, 0 > >pfrulepl: 912, 0, 40, 88, 806, 0 > >pfstatepl: 392, 10000, 10000, 0, 4440553, 341638 > >pfaltqpl: 240, 0, 0, 0, 0, 0 > >pfpooladdrpl: 88, 0, 0, 0, 0, 0 > >pfrktable: 1296, 1002, 4, 20, 112, 0 > >pfrkentry: 216, 100008, 603, 891, 15384, 0 > >pfrkentry2: 216, 0, 0, 0, 0, 0 > >pffrent: 32, 5050, 0, 303, 1620, 0 > >pffrag: 80, 0, 0, 135, 807, 0 > >pffrcache: 80, 10035, 0, 0, 0, 0 > >pffrcent: 24, 50022, 0, 0, 0, 0 > >pfstatescrub: 40, 0, 0, 0, 0, 0 > >pfiaddrpl: 120, 0, 0, 0, 0, 0 > >pfospfen: 112, 0, 696, 30, 18096, 0 > >pfosfp: 40, 0, 407, 97, 10582, 0 > >------------------------------------------------------------------------ > > > >You can see evidence of processes not exiting/doing what they should do > >here: > > > >------------------------------------------------------------------------ > >fstat > > > >USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > >root shutdown 91155 root / 2 drwxr-xr-x 512 r > >root shutdown 91155 wd / 2 drwxr-xr-x 512 r > >root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > >root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > >root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > >root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > >root sh 91129 root / 2 drwxr-xr-x 512 r > >root sh 91129 wd / 2 drwxr-xr-x 512 r > >root sh 91129 text / 44 -r-xr-xr-x 134848 r > >root sh 91129 0 /dev 38 crw------- ttyu0 rw > >root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 0 rw > >root sh 91129 2 /dev 20 crw-rw-rw- null w > >root shutdown 91115 root / 2 drwxr-xr-x 512 r > >root shutdown 91115 wd /storage 5 drwx------ 37 r > >root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > >root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 3* local dgram ffffff008ff92960 > >root sh 90818 root / 2 drwxr-xr-x 512 r > >root sh 90818 wd / 70659 drwxr-xr-x 512 r > >root sh 90818 text / 44 -r-xr-xr-x 134848 r > >root sh 90818 0 /dev 38 crw------- ttyu0 rw > >root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 0 rw > >root sh 90818 2 /dev 20 crw-rw-rw- null w > >root csh 90802 root / 2 drwxr-xr-x 512 r > >root csh 90802 wd / 2 drwxr-xr-x 512 r > >root csh 90802 text / 51 -r-xr-xr-x 358752 r > >root csh 90802 15 /dev 38 crw------- ttyu0 rw > >root csh 90802 16 /dev 38 crw------- ttyu0 rw > >root csh 90802 17 /dev 38 crw------- ttyu0 rw > >root csh 90802 18 /dev 38 crw------- ttyu0 rw > >root csh 90802 19 /dev 38 crw------- ttyu0 rw > >------------------------------------------------------------------------ > > > >No indication of mbuf exhaustion, putting further focus on the pf stack: > > > >------------------------------------------------------------------------ > >netstat -m > > > >2054/1786/3840 mbufs in use (current/cache/total) > >2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > >2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > >0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > >0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > >0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > >4609K/4554K/9164K bytes allocated to network (current/cache/total) > >0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > >0/0/0 requests for jumbo clusters denied (4k/9k/16k) > >0 requests for sfbufs denied > >0 requests for sfbufs delayed > >0 requests for I/O initiated by sendfile > >0 calls to protocol drain routines > >------------------------------------------------------------------------ > > > >Here's one piece of core.0.txt which makes no sense to me -- the "rate" > >column. I have a very hard time believing that was the interrupt rate > >of all the relevant devices at the time (way too high). Maybe this data > >becomes wrong only during a coredump? The total column I could believe. > > > >------------------------------------------------------------------------ > >vmstat -i > > > >interrupt total rate > >irq4: uart0 54768 912 > >irq6: fdc0 1 0 > >irq17: uhci1+ 172 2 > >irq23: uhci3 ehci1+ 2367 39 > >cpu0: timer 13183882632 219731377 > >irq256: em0 260491055 4341517 > >irq257: em1 127555036 2125917 > >irq258: ahci0 225923164 3765386 > >cpu2: timer 13183881837 219731363 > >cpu1: timer 13002196469 216703274 > >cpu3: timer 13183881783 219731363 > >Total 53167869284 886131154 > >------------------------------------------------------------------------ > > > >Here's what a normal "vmstat -i" shows from the command-line: > > > ># vmstat -i > >interrupt total rate > >irq4: uart0 518 0 > >irq6: fdc0 1 0 > >irq23: uhci3 ehci1+ 145 0 > >cpu0: timer 19041199 1999 > >irq256: em0 614280 64 > >irq257: em1 168529 17 > >irq258: ahci0 355536 37 > >cpu2: timer 19040462 1999 > >cpu1: timer 19040458 1999 > >cpu3: timer 19040454 1999 > >Total 77301582 8119 > > > >We graph many aspects of this box, including CPU load, memory/swap > >usage, etc. and none show any sign that the interrupt rate on all of > >those devices was even remotely out of control. (I would expect to see > >CPU through the roof given the above data) > > > >I have since rebuilt/reinstalled world/kernel on the machine with the > >latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > >2011), hoping whatever this was may have been fixed. > > > >As for what I think may have triggered it, but I have no hard evidence > >of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > >reload". The pf.conf change was a single line: > > > >Old: scrub on em0 all > >New: scrub in on em0 all > > > >Why it took the problem approximately 3 days to start is unknown. It's > >the only change we've made to the system (truly/honestly), and it was a > >change to pf.conf. > > > >If anyone has advice (or has seen the above problem), or is interested > >in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > >any way I can. I would hate for someone else to get bit by this, and > >really am hoping its something that has been fixed between February and > >now. > > > > That's quite the deduction there. I've noticed recently that you were also > experimenting with the new NFS server recompiling kernel etc etc. Seeing > as weird things can happen with DNS, NFS and mountpoint's, is this the same > machine that you were doing that on ? Completely different machine. The machine which exhibited the problem does mount a single NFS filesystem from the other box you're referring to, but it does not use NFSv4. Most importantly: the NFS mounts are also done across the em1 interface exclusively, which as mentioned falls under "set skip". I realise my analysis results in a very ballsy claim, but the graphs are quite damning, as is pfctl output and machine behaviour. I wish I knew why the machine couldn't be rebooted cleanly / why processes were "sort of" wedging. That seems to indicate something weird going on in the kernel or the pf stack; not sure which. > If so can you check to see how many requests for NFS operations were done > to/from that box ? as well the names that would be being resolved and if > that machine can resolve them ? Each machine on our network runs named. Each named instance slaves a copy of three zones from the masters: reverse in-addr.arpa for the LAN, reverse in-addr.arpa for the WAN, and the forward zone associated with the domain name (parodius.com). So, literally every machine on the network avoids having to go out and ask the Internet for lookups under our domain, or our IP blocks (public or private). Our resolv.conf is tuned appropriately for this too ("search parodius.com private.network" and a nameserver of 127.0.0.1). The slaving does happen over the public interface for a lot of reasons I'd rather not get into here. Regarding NFS statistics -- no NFS operations were occurring during the 24 hour window where pf's state count went through the roof, except for at approximately 04:00 PDT when our automated backups ran. The issue did not start then as indicated by my analysis. Regardless, below are the NFS statistics which were provided by "call doadump" and are in core.0.txt: ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 172 0 33 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 3 0 3 Mknod Fsstat Fsinfo PathConf Commit 0 56 1 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 268 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 95 172 26 33 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses 0 0 3 3 3 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ These numbers should act as confirmation that the problem is almost certainly not NFS-related; these numbers are very small. For fun, here's the "netstat -s" output of the core. ------------------------------------------------------------------------ netstat -s tcp: 70963757 packets sent 56716136 data packets (93106798714 bytes) 20493 data packets (14888440 bytes) retransmitted 3258 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 7640376 ack-only packets (433685 delayed) 0 URG only packets 0 window probe packets 3629755 window update packets 2956596 control packets 109589225 packets received 81346171 acks (for 93113064503 bytes) 2970456 duplicate acks 0 acks for unsent data 56067087 packets (8649172006 bytes) received in-sequence 5950 completely duplicate packets (5254577 bytes) 26 old duplicate packets 152 packets with some dup. data (35886 bytes duped) 7412 out-of-order packets (10192893 bytes) 0 packets (0 bytes) of data after window 0 window probes 10001480 window update packets 16 packets received after close 1561 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 6 discarded due to memory problems 2159 connection requests 2954080 connection accepts 0 bad connection attempts 0 listen queue overflows 491 ignored RSTs in the windows 2955571 connections established (including accepts) 3066783 connections closed (including 1401 drops) 2952708 connections updated cached RTT on close 2952890 connections updated cached RTT variance on close 2928379 connections updated cached ssthresh on close 220 embryonic connections dropped 81149655 segments updated rtt (of 58223063 attempts) 17684 retransmit timeouts 169 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 76 keepalive timeouts 76 keepalive probes sent 0 connections dropped by keepalive 222190 correct ACK header predictions 6584196 correct data packet header predictions 2955062 syncache entries added 603 retransmitted 188 dupsyn 0 dropped 2954080 completed 0 bucket overflow 0 cache overflow 869 reset 113 stale 0 aborted 0 badack 0 unreach 0 zone failures 2955062 cookies sent 0 cookies received 126 SACK recovery episodes 78 segment rexmits in SACK recovery episodes 78254 byte rexmits in SACK recovery episodes 2962 SACK options (SACK blocks) received 8870 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 6218432 datagrams received 0 with incomplete header 0 with bad data length field 29 with bad checksum 15493 with no checksum 4896 dropped due to no socket 0 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 6213507 delivered 6405573 datagrams output 0 times multicast source filter matched ip: 236156523 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 235463206 packets for this host 66789 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 81178132 packets sent from this host 126684324 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 4896 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 3801761 destination unreachable: 4896 0 messages with bad code fields 0 messages less than the minimum length 21 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: echo reply: 7552810 destination unreachable: 58965 source quench: 10 routing redirect: 75 echo: 3801761 time exceeded: 108309026 3801761 message responses generated 0 invalid return addresses 0 no return routes igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent arp: 5537 ARP requests sent 26179 ARP replies sent 1794911 ARP requests received 5531 ARP replies received 1800442 ARP packets received 18 total packets dropped due to no ARP entry 9834 ARP entrys timed out 0 Duplicate IPs seen ------------------------------------------------------------------------ > Also I would believe your using tables in your pf.conf, if so do any of > those tables contain a FQDN that cannot be resolved from that machine ? > > I think you probably see what I am getting at here as it could be some > sort of concurrent recursive DNS failure that can only be seen from the > machine caused by possibly the new NFS backend or a change in one of the > tables that pf would use. I absolutely see where you're going with this, but as I understand it, DNS resolution on pf.conf rules happens during pf load-time (though I have no idea how TTL would be honoured). However/regardless: There are no FQDNs or hostnames used in the pf.conf on the machine which exhibited the above problem; everything uses hard-coded IP addresses. The same goes for the contents of the (extremely few; only 4) tables referenced in pf.conf. The only non-numeric data used for resolution of any kind are a couple service names (e.g. "ssh" instead of 22). The pf.conf on the machine which witnessed the issue is extremely small and very simple. It's also the 2nd-to-lowest machine on our network when it comes to amount of network traffic received/sent. How can I get a dump of the contents of the pf state table -- specifically referring to whatever it thinks still contained 10,000 state entries despite "pfctl -s state" showing 0? There is obviously a mismatch there, so I'm wondering how the 10,000 counter could stay at 10,000 (especially after doing a full stop/start of the pf stack) when "pfctl -s state" shows 0 (ditto with "pfctl -F state"). I really, *really* want to debug this. It's to the point where I'm happily willing to pay for a kernel developer's time, hourly, to figure it out (if at all possible). It might be easier to figure it out in real-time rather than post-mortem, but post-mortem is what I've got right now. I need to get off my butt and get an actual dev/testbed box in our production co-lo where I can bang on this stuff for days and see if I can reproduce it. All I can conclude on my own is that "something bad" can happen when reloading pf rules + changing "scrub" from both directions to just incoming. I don't even know if the scrub change is what triggered it, maybe just the reload did. Maybe the problem can only happen when a packet of a certain type is inbound or outbound and in a certain internal state. Again, it's speculation, but there's some pretty strong evidence that something very bad happened, almost like an infinite loop that eventually exhausted all stateful memory/entries. A DoS across em0 is what it sounds like, but the network graphs on the switch confirm that isn't the case. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 05:52:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACEE01065670 for ; Tue, 3 May 2011 05:52:56 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 466998FC15 for ; Tue, 3 May 2011 05:52:55 +0000 (UTC) Received: by qyk27 with SMTP id 27so3879114qyk.13 for ; Mon, 02 May 2011 22:52:55 -0700 (PDT) Received: by 10.229.114.77 with SMTP id d13mr6803064qcq.219.1304400170184; Mon, 02 May 2011 22:22:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Mon, 2 May 2011 22:22:10 -0700 (PDT) In-Reply-To: <20110503015854.GA31444@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> From: Vlad Galu Date: Tue, 3 May 2011 07:22:10 +0200 Message-ID: To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:52:56 -0000 On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick wrote: > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > for cross-posting, but the issue is severe enough that I wanted to make > it known on -stable) > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > Please read the story in full, as I have taken the time to describe > everything I did, plus log output, as well as induce a panic via "call > doadump" from ddb so I have a capture of the system at the time. I also > have a theory as to what caused the problem, but how to trigger it is > unknown; it may be a rare race condition. > > > This morning I woke up to find a report from one of our users that he > could not connect to a specific TCP port (not SSH) on one of our > servers. I also found that I couldn't SSH into the same box. Serial > console was working fine, and the serial console log showed no sign of > any problems. > > I started to debug the issue of me not being able to SSH into the > machine and within a few minutes became immediately concerned: pfctl > indicated we had reached the maximum number permitted state table > entries (10,000). > > ============================================================ > # pfctl -s info > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > Interface Stats for em0 IPv4 IPv6 > Bytes In 8969748840 0 > Bytes Out 8296135477 0 > Packets In > Passed 128211763 0 > Blocked 621379 0 > Packets Out > Passed 138483868 0 > Blocked 2579 0 > > State Table Total Rate > current entries 10000 > searches 267316807 40.6/s > inserts 4440553 0.7/s > removals 4430553 0.7/s > Counters > match 5067474 0.8/s > bad-offset 0 0.0/s > fragment 324 0.0/s > short 0 0.0/s > normalize 32 0.0/s > memory 336946 0.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 1611 0.0/s > state-mismatch 509 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s memory > states hard limit 10000 > src-nodes hard limit 10000 > frags hard limit 5000 > tables hard limit 1000 > table-entries hard limit 100000 > ============================================================ > > The above is mainly for em0 (our WAN interface); our LAN interface (em1) > was not impacted because we use "set skip on em1". And it's a good > thing too: we have lots of LAN-based services that this machine provides > that would have been impacted. We also use "set skip on lo0". > > I immediately went to look at our monitoring graphs, which monitor pf > state (specifically state table entries), polled via bsnmpd(8). This > data is even more frightening: > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > Literally something was spiraling out of control, starting at approx. > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > 19:45 PDT the same day, but I wasn't aware of it until said user brought > an issue to my attention. > > You can see from the network I/O graphs (taken from SNMP polling our > switch, NOT from the host/box itself) that there was no DoS attack or > anything like that occurring -- this was something within FreeBSD > itself. More evidence of that will become apparent. > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > not truly die (despite csh stating "Terminated"). The only way to kill > it was to kill -9. > > Attempts to shut down any daemons which utilised the network -- > including things that only used em1 -- would not shut down. This > included things like postfix, mysqld, and some inet-based services. I > was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > That is to say, wall(1)'s announcement was shown, but the actual > stopping of services did not begin. > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > did "/etc/rc.d/pf start", which also worked. However, what I saw next > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > and "pfctl -s info" disagreed on the state count: > > ============================================================ > # pfctl -s info > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > Interface Stats for em0 IPv4 IPv6 > Bytes In 3459 0 > Bytes Out 0 0 > Packets In > Passed 0 0 > Blocked 29 0 > Packets Out > Passed 0 0 > Blocked 0 0 > > State Table Total Rate > current entries 10000 > searches 29 1.8/s > inserts 0 0.0/s > removals 0 0.0/s > Counters > match 29 1.8/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 0 0.0/s > memory 18 1.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 0 0.0/s > state-mismatch 0 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s state | wc -l > 0 > ============================================================ > > The "pf uptime" shown above, by the way, matches system uptime. > > I then attempted "pfctl -F state", but nothing changed (looked the same > as above). > > Since I could not reboot the box, I was forced to drop to ddb via serial > console. I did some commands like "ps" and the like, and then "call > doadump" to induce a kernel panic, and then "reboot" (which worked). > > Once the machine came back up, savecore(8) ran, wrote the data out, and > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > I do not feel comfortable sharing its content publicly, but will be > happy to hand it to developer(s) who are interested. Relevant tidbits I > can discern: > > ------------------------------------------------------------------------ > ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 > [pfpurge] > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------ > vmstat -z > > ITEM SIZE LIMIT USED FREE REQUESTS > FAILURES > pfsrctrpl: 152, 10000, 0, 0, 0, > 0 > pfrulepl: 912, 0, 40, 88, 806, > 0 > pfstatepl: 392, 10000, 10000, 0, 4440553, > 341638 > pfaltqpl: 240, 0, 0, 0, 0, > 0 > pfpooladdrpl: 88, 0, 0, 0, 0, > 0 > pfrktable: 1296, 1002, 4, 20, 112, > 0 > pfrkentry: 216, 100008, 603, 891, 15384, > 0 > pfrkentry2: 216, 0, 0, 0, 0, > 0 > pffrent: 32, 5050, 0, 303, 1620, > 0 > pffrag: 80, 0, 0, 135, 807, > 0 > pffrcache: 80, 10035, 0, 0, 0, > 0 > pffrcent: 24, 50022, 0, 0, 0, > 0 > pfstatescrub: 40, 0, 0, 0, 0, > 0 > pfiaddrpl: 120, 0, 0, 0, 0, > 0 > pfospfen: 112, 0, 696, 30, 18096, > 0 > pfosfp: 40, 0, 407, 97, 10582, > 0 > ------------------------------------------------------------------------ > > You can see evidence of processes not exiting/doing what they should do > here: > > ------------------------------------------------------------------------ > fstat > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > root shutdown 91155 root / 2 drwxr-xr-x 512 r > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > root sh 91129 root / 2 drwxr-xr-x 512 r > root sh 91129 wd / 2 drwxr-xr-x 512 r > root sh 91129 text / 44 -r-xr-xr-x 134848 r > root sh 91129 0 /dev 38 crw------- ttyu0 rw > root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 > 0 rw > root sh 91129 2 /dev 20 crw-rw-rw- null w > root shutdown 91115 root / 2 drwxr-xr-x 512 r > root shutdown 91115 wd /storage 5 drwx------ 37 r > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > root shutdown 91115 3* local dgram ffffff008ff92960 > root sh 90818 root / 2 drwxr-xr-x 512 r > root sh 90818 wd / 70659 drwxr-xr-x 512 r > root sh 90818 text / 44 -r-xr-xr-x 134848 r > root sh 90818 0 /dev 38 crw------- ttyu0 rw > root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 > 0 rw > root sh 90818 2 /dev 20 crw-rw-rw- null w > root csh 90802 root / 2 drwxr-xr-x 512 r > root csh 90802 wd / 2 drwxr-xr-x 512 r > root csh 90802 text / 51 -r-xr-xr-x 358752 r > root csh 90802 15 /dev 38 crw------- ttyu0 rw > root csh 90802 16 /dev 38 crw------- ttyu0 rw > root csh 90802 17 /dev 38 crw------- ttyu0 rw > root csh 90802 18 /dev 38 crw------- ttyu0 rw > root csh 90802 19 /dev 38 crw------- ttyu0 rw > ------------------------------------------------------------------------ > > No indication of mbuf exhaustion, putting further focus on the pf stack: > > ------------------------------------------------------------------------ > netstat -m > > 2054/1786/3840 mbufs in use (current/cache/total) > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/320/320/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > ------------------------------------------------------------------------ > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ > > Here's what a normal "vmstat -i" shows from the command-line: > > # vmstat -i > interrupt total rate > irq4: uart0 518 0 > irq6: fdc0 1 0 > irq23: uhci3 ehci1+ 145 0 > cpu0: timer 19041199 1999 > irq256: em0 614280 64 > irq257: em1 168529 17 > irq258: ahci0 355536 37 > cpu2: timer 19040462 1999 > cpu1: timer 19040458 1999 > cpu3: timer 19040454 1999 > Total 77301582 8119 > > We graph many aspects of this box, including CPU load, memory/swap > usage, etc. and none show any sign that the interrupt rate on all of > those devices was even remotely out of control. (I would expect to see > CPU through the roof given the above data) > > I have since rebuilt/reinstalled world/kernel on the machine with the > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > 2011), hoping whatever this was may have been fixed. > > As for what I think may have triggered it, but I have no hard evidence > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > reload". The pf.conf change was a single line: > > Old: scrub on em0 all > New: scrub in on em0 all > > Why it took the problem approximately 3 days to start is unknown. It's > the only change we've made to the system (truly/honestly), and it was a > change to pf.conf. > > If anyone has advice (or has seen the above problem), or is interested > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > any way I can. I would hate for someone else to get bit by this, and > really am hoping its something that has been fixed between February and > now. > > I'm seeing this as well. You could change your scrub rules so that you specifically avoid TCP reassembly (that creates states). -- Good, fast & cheap. Pick any two. From owner-freebsd-stable@FreeBSD.ORG Tue May 3 06:01:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEE8D1065673 for ; Tue, 3 May 2011 06:01:09 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id D5C728FC14 for ; Tue, 3 May 2011 06:01:09 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta11.emeryville.ca.mail.comcast.net with comcast id etzD1g0011smiN4ABu19BB; Tue, 03 May 2011 06:01:09 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id eu161g0071t3BNj8gu18Qm; Tue, 03 May 2011 06:01:08 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6662D9B418; Mon, 2 May 2011 23:01:06 -0700 (PDT) Date: Mon, 2 May 2011 23:01:06 -0700 From: Jeremy Chadwick To: Vlad Galu Message-ID: <20110503060106.GA36331@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:01:10 -0000 On Tue, May 03, 2011 at 07:22:10AM +0200, Vlad Galu wrote: > On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick wrote: > > > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > > for cross-posting, but the issue is severe enough that I wanted to make > > it known on -stable) > > > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > > Please read the story in full, as I have taken the time to describe > > everything I did, plus log output, as well as induce a panic via "call > > doadump" from ddb so I have a capture of the system at the time. I also > > have a theory as to what caused the problem, but how to trigger it is > > unknown; it may be a rare race condition. > > > > > > This morning I woke up to find a report from one of our users that he > > could not connect to a specific TCP port (not SSH) on one of our > > servers. I also found that I couldn't SSH into the same box. Serial > > console was working fine, and the serial console log showed no sign of > > any problems. > > > > I started to debug the issue of me not being able to SSH into the > > machine and within a few minutes became immediately concerned: pfctl > > indicated we had reached the maximum number permitted state table > > entries (10,000). > > > > ============================================================ > > # pfctl -s info > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > > Interface Stats for em0 IPv4 IPv6 > > Bytes In 8969748840 0 > > Bytes Out 8296135477 0 > > Packets In > > Passed 128211763 0 > > Blocked 621379 0 > > Packets Out > > Passed 138483868 0 > > Blocked 2579 0 > > > > State Table Total Rate > > current entries 10000 > > searches 267316807 40.6/s > > inserts 4440553 0.7/s > > removals 4430553 0.7/s > > Counters > > match 5067474 0.8/s > > bad-offset 0 0.0/s > > fragment 324 0.0/s > > short 0 0.0/s > > normalize 32 0.0/s > > memory 336946 0.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 1611 0.0/s > > state-mismatch 509 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > > # pfctl -s memory > > states hard limit 10000 > > src-nodes hard limit 10000 > > frags hard limit 5000 > > tables hard limit 1000 > > table-entries hard limit 100000 > > ============================================================ > > > > The above is mainly for em0 (our WAN interface); our LAN interface (em1) > > was not impacted because we use "set skip on em1". And it's a good > > thing too: we have lots of LAN-based services that this machine provides > > that would have been impacted. We also use "set skip on lo0". > > > > I immediately went to look at our monitoring graphs, which monitor pf > > state (specifically state table entries), polled via bsnmpd(8). This > > data is even more frightening: > > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > > Literally something was spiraling out of control, starting at approx. > > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > > 19:45 PDT the same day, but I wasn't aware of it until said user brought > > an issue to my attention. > > > > You can see from the network I/O graphs (taken from SNMP polling our > > switch, NOT from the host/box itself) that there was no DoS attack or > > anything like that occurring -- this was something within FreeBSD > > itself. More evidence of that will become apparent. > > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > > not truly die (despite csh stating "Terminated"). The only way to kill > > it was to kill -9. > > > > Attempts to shut down any daemons which utilised the network -- > > including things that only used em1 -- would not shut down. This > > included things like postfix, mysqld, and some inet-based services. I > > was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > > That is to say, wall(1)'s announcement was shown, but the actual > > stopping of services did not begin. > > > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > > did "/etc/rc.d/pf start", which also worked. However, what I saw next > > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > > and "pfctl -s info" disagreed on the state count: > > > > ============================================================ > > # pfctl -s info > > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > > Interface Stats for em0 IPv4 IPv6 > > Bytes In 3459 0 > > Bytes Out 0 0 > > Packets In > > Passed 0 0 > > Blocked 29 0 > > Packets Out > > Passed 0 0 > > Blocked 0 0 > > > > State Table Total Rate > > current entries 10000 > > searches 29 1.8/s > > inserts 0 0.0/s > > removals 0 0.0/s > > Counters > > match 29 1.8/s > > bad-offset 0 0.0/s > > fragment 0 0.0/s > > short 0 0.0/s > > normalize 0 0.0/s > > memory 18 1.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 0 0.0/s > > state-mismatch 0 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > > # pfctl -s state | wc -l > > 0 > > ============================================================ > > > > The "pf uptime" shown above, by the way, matches system uptime. > > > > I then attempted "pfctl -F state", but nothing changed (looked the same > > as above). > > > > Since I could not reboot the box, I was forced to drop to ddb via serial > > console. I did some commands like "ps" and the like, and then "call > > doadump" to induce a kernel panic, and then "reboot" (which worked). > > > > Once the machine came back up, savecore(8) ran, wrote the data out, and > > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > > I do not feel comfortable sharing its content publicly, but will be > > happy to hand it to developer(s) who are interested. Relevant tidbits I > > can discern: > > > > ------------------------------------------------------------------------ > > ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 > > [pfpurge] > > ------------------------------------------------------------------------ > > > > ------------------------------------------------------------------------ > > vmstat -z > > > > ITEM SIZE LIMIT USED FREE REQUESTS > > FAILURES > > pfsrctrpl: 152, 10000, 0, 0, 0, > > 0 > > pfrulepl: 912, 0, 40, 88, 806, > > 0 > > pfstatepl: 392, 10000, 10000, 0, 4440553, > > 341638 > > pfaltqpl: 240, 0, 0, 0, 0, > > 0 > > pfpooladdrpl: 88, 0, 0, 0, 0, > > 0 > > pfrktable: 1296, 1002, 4, 20, 112, > > 0 > > pfrkentry: 216, 100008, 603, 891, 15384, > > 0 > > pfrkentry2: 216, 0, 0, 0, 0, > > 0 > > pffrent: 32, 5050, 0, 303, 1620, > > 0 > > pffrag: 80, 0, 0, 135, 807, > > 0 > > pffrcache: 80, 10035, 0, 0, 0, > > 0 > > pffrcent: 24, 50022, 0, 0, 0, > > 0 > > pfstatescrub: 40, 0, 0, 0, 0, > > 0 > > pfiaddrpl: 120, 0, 0, 0, 0, > > 0 > > pfospfen: 112, 0, 696, 30, 18096, > > 0 > > pfosfp: 40, 0, 407, 97, 10582, > > 0 > > ------------------------------------------------------------------------ > > > > You can see evidence of processes not exiting/doing what they should do > > here: > > > > ------------------------------------------------------------------------ > > fstat > > > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > > root shutdown 91155 root / 2 drwxr-xr-x 512 r > > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > > root sh 91129 root / 2 drwxr-xr-x 512 r > > root sh 91129 wd / 2 drwxr-xr-x 512 r > > root sh 91129 text / 44 -r-xr-xr-x 134848 r > > root sh 91129 0 /dev 38 crw------- ttyu0 rw > > root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 > > 0 rw > > root sh 91129 2 /dev 20 crw-rw-rw- null w > > root shutdown 91115 root / 2 drwxr-xr-x 512 r > > root shutdown 91115 wd /storage 5 drwx------ 37 r > > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 3* local dgram ffffff008ff92960 > > root sh 90818 root / 2 drwxr-xr-x 512 r > > root sh 90818 wd / 70659 drwxr-xr-x 512 r > > root sh 90818 text / 44 -r-xr-xr-x 134848 r > > root sh 90818 0 /dev 38 crw------- ttyu0 rw > > root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 > > 0 rw > > root sh 90818 2 /dev 20 crw-rw-rw- null w > > root csh 90802 root / 2 drwxr-xr-x 512 r > > root csh 90802 wd / 2 drwxr-xr-x 512 r > > root csh 90802 text / 51 -r-xr-xr-x 358752 r > > root csh 90802 15 /dev 38 crw------- ttyu0 rw > > root csh 90802 16 /dev 38 crw------- ttyu0 rw > > root csh 90802 17 /dev 38 crw------- ttyu0 rw > > root csh 90802 18 /dev 38 crw------- ttyu0 rw > > root csh 90802 19 /dev 38 crw------- ttyu0 rw > > ------------------------------------------------------------------------ > > > > No indication of mbuf exhaustion, putting further focus on the pf stack: > > > > ------------------------------------------------------------------------ > > netstat -m > > > > 2054/1786/3840 mbufs in use (current/cache/total) > > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > > 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > > 0/320/320/12800 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > ------------------------------------------------------------------------ > > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > column. I have a very hard time believing that was the interrupt rate > > of all the relevant devices at the time (way too high). Maybe this data > > becomes wrong only during a coredump? The total column I could believe. > > > > ------------------------------------------------------------------------ > > vmstat -i > > > > interrupt total rate > > irq4: uart0 54768 912 > > irq6: fdc0 1 0 > > irq17: uhci1+ 172 2 > > irq23: uhci3 ehci1+ 2367 39 > > cpu0: timer 13183882632 219731377 > > irq256: em0 260491055 4341517 > > irq257: em1 127555036 2125917 > > irq258: ahci0 225923164 3765386 > > cpu2: timer 13183881837 219731363 > > cpu1: timer 13002196469 216703274 > > cpu3: timer 13183881783 219731363 > > Total 53167869284 886131154 > > ------------------------------------------------------------------------ > > > > Here's what a normal "vmstat -i" shows from the command-line: > > > > # vmstat -i > > interrupt total rate > > irq4: uart0 518 0 > > irq6: fdc0 1 0 > > irq23: uhci3 ehci1+ 145 0 > > cpu0: timer 19041199 1999 > > irq256: em0 614280 64 > > irq257: em1 168529 17 > > irq258: ahci0 355536 37 > > cpu2: timer 19040462 1999 > > cpu1: timer 19040458 1999 > > cpu3: timer 19040454 1999 > > Total 77301582 8119 > > > > We graph many aspects of this box, including CPU load, memory/swap > > usage, etc. and none show any sign that the interrupt rate on all of > > those devices was even remotely out of control. (I would expect to see > > CPU through the roof given the above data) > > > > I have since rebuilt/reinstalled world/kernel on the machine with the > > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > > 2011), hoping whatever this was may have been fixed. > > > > As for what I think may have triggered it, but I have no hard evidence > > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > > reload". The pf.conf change was a single line: > > > > Old: scrub on em0 all > > New: scrub in on em0 all > > > > Why it took the problem approximately 3 days to start is unknown. It's > > the only change we've made to the system (truly/honestly), and it was a > > change to pf.conf. > > > > If anyone has advice (or has seen the above problem), or is interested > > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > > any way I can. I would hate for someone else to get bit by this, and > > really am hoping its something that has been fixed between February and > > now. > > > > > I'm seeing this as well. You could change your scrub rules so that you > specifically avoid TCP reassembly (that creates states). Thank you very much. This helps -- and I'm also glad someone else has seen this behaviour. That confirms it's not specific to my equipment, which is good. Regarding scrubbing and TCP reassembly (option "reassemble tcp" according to pf.conf): I wasn't under the impression this option was enabled by default. This got me wondering what the defaults actually are (pf.conf(5) is somewhat vague in this regard, but it does state that "fragment reassemble" is enabled by default). The rule I use: scrub in on em0 all Appears to get evaluated into this (per "pfctl -s rules -v"): scrub in on em0 all fragment reassemble Did you mean to tell me to disable the "fragment reassemble" option (which is different from "reassemble tcp")? If so, how do I do that? It looks like I could use a "no scrub" rule, except I can't find any examples of this on the web or in the docs. Or is it just better to remove use of scrub entirely until whatever this is gets fixed? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 06:10:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85BF5106566B; Tue, 3 May 2011 06:10:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id F24458FC15; Tue, 3 May 2011 06:10:01 +0000 (UTC) Received: by qwc9 with SMTP id 9so3744649qwc.13 for ; Mon, 02 May 2011 23:10:01 -0700 (PDT) Received: by 10.229.43.209 with SMTP id x17mr6770395qce.257.1304403001141; Mon, 02 May 2011 23:10:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Mon, 2 May 2011 23:09:21 -0700 (PDT) In-Reply-To: <20110503060106.GA36331@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503060106.GA36331@icarus.home.lan> From: Vlad Galu Date: Tue, 3 May 2011 08:09:21 +0200 Message-ID: To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:10:02 -0000 On Tue, May 3, 2011 at 8:01 AM, Jeremy Chadwick wrote: > On Tue, May 03, 2011 at 07:22:10AM +0200, Vlad Galu wrote: > > On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick < > freebsd@jdc.parodius.com>wrote: > > > > > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And > apologies > > > for cross-posting, but the issue is severe enough that I wanted to make > > > it known on -stable) > > > > > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > > > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > > > > Please read the story in full, as I have taken the time to describe > > > everything I did, plus log output, as well as induce a panic via "call > > > doadump" from ddb so I have a capture of the system at the time. I > also > > > have a theory as to what caused the problem, but how to trigger it is > > > unknown; it may be a rare race condition. > > > > > > > > > This morning I woke up to find a report from one of our users that he > > > could not connect to a specific TCP port (not SSH) on one of our > > > servers. I also found that I couldn't SSH into the same box. Serial > > > console was working fine, and the serial console log showed no sign of > > > any problems. > > > > > > I started to debug the issue of me not being able to SSH into the > > > machine and within a few minutes became immediately concerned: pfctl > > > indicated we had reached the maximum number permitted state table > > > entries (10,000). > > > > > > ============================================================ > > > # pfctl -s info > > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > > > > Interface Stats for em0 IPv4 IPv6 > > > Bytes In 8969748840 0 > > > Bytes Out 8296135477 0 > > > Packets In > > > Passed 128211763 0 > > > Blocked 621379 0 > > > Packets Out > > > Passed 138483868 0 > > > Blocked 2579 0 > > > > > > State Table Total Rate > > > current entries 10000 > > > searches 267316807 40.6/s > > > inserts 4440553 0.7/s > > > removals 4430553 0.7/s > > > Counters > > > match 5067474 0.8/s > > > bad-offset 0 0.0/s > > > fragment 324 0.0/s > > > short 0 0.0/s > > > normalize 32 0.0/s > > > memory 336946 0.1/s > > > bad-timestamp 0 0.0/s > > > congestion 0 0.0/s > > > ip-option 0 0.0/s > > > proto-cksum 1611 0.0/s > > > state-mismatch 509 0.0/s > > > state-insert 0 0.0/s > > > state-limit 0 0.0/s > > > src-limit 0 0.0/s > > > synproxy 0 0.0/s > > > > > > # pfctl -s memory > > > states hard limit 10000 > > > src-nodes hard limit 10000 > > > frags hard limit 5000 > > > tables hard limit 1000 > > > table-entries hard limit 100000 > > > ============================================================ > > > > > > The above is mainly for em0 (our WAN interface); our LAN interface > (em1) > > > was not impacted because we use "set skip on em1". And it's a good > > > thing too: we have lots of LAN-based services that this machine > provides > > > that would have been impacted. We also use "set skip on lo0". > > > > > > I immediately went to look at our monitoring graphs, which monitor pf > > > state (specifically state table entries), polled via bsnmpd(8). This > > > data is even more frightening: > > > > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > > > > Literally something was spiraling out of control, starting at approx. > > > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > > > 19:45 PDT the same day, but I wasn't aware of it until said user > brought > > > an issue to my attention. > > > > > > You can see from the network I/O graphs (taken from SNMP polling our > > > switch, NOT from the host/box itself) that there was no DoS attack or > > > anything like that occurring -- this was something within FreeBSD > > > itself. More evidence of that will become apparent. > > > > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > > > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > > > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > > > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > > > not truly die (despite csh stating "Terminated"). The only way to kill > > > it was to kill -9. > > > > > > Attempts to shut down any daemons which utilised the network -- > > > including things that only used em1 -- would not shut down. This > > > included things like postfix, mysqld, and some inet-based services. I > > > was forced to kill -9 them. Things like bsnmpd, however, did shut > down. > > > > > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > > > That is to say, wall(1)'s announcement was shown, but the actual > > > stopping of services did not begin. > > > > > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > > > did "/etc/rc.d/pf start", which also worked. However, what I saw next > > > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > > > and "pfctl -s info" disagreed on the state count: > > > > > > ============================================================ > > > # pfctl -s info > > > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > > > > Interface Stats for em0 IPv4 IPv6 > > > Bytes In 3459 0 > > > Bytes Out 0 0 > > > Packets In > > > Passed 0 0 > > > Blocked 29 0 > > > Packets Out > > > Passed 0 0 > > > Blocked 0 0 > > > > > > State Table Total Rate > > > current entries 10000 > > > searches 29 1.8/s > > > inserts 0 0.0/s > > > removals 0 0.0/s > > > Counters > > > match 29 1.8/s > > > bad-offset 0 0.0/s > > > fragment 0 0.0/s > > > short 0 0.0/s > > > normalize 0 0.0/s > > > memory 18 1.1/s > > > bad-timestamp 0 0.0/s > > > congestion 0 0.0/s > > > ip-option 0 0.0/s > > > proto-cksum 0 0.0/s > > > state-mismatch 0 0.0/s > > > state-insert 0 0.0/s > > > state-limit 0 0.0/s > > > src-limit 0 0.0/s > > > synproxy 0 0.0/s > > > > > > # pfctl -s state | wc -l > > > 0 > > > ============================================================ > > > > > > The "pf uptime" shown above, by the way, matches system uptime. > > > > > > I then attempted "pfctl -F state", but nothing changed (looked the same > > > as above). > > > > > > Since I could not reboot the box, I was forced to drop to ddb via > serial > > > console. I did some commands like "ps" and the like, and then "call > > > doadump" to induce a kernel panic, and then "reboot" (which worked). > > > > > > Once the machine came back up, savecore(8) ran, wrote the data out, and > > > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > > > I do not feel comfortable sharing its content publicly, but will be > > > happy to hand it to developer(s) who are interested. Relevant tidbits > I > > > can discern: > > > > > > > ------------------------------------------------------------------------ > > > ps -axl > > > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME > COMMAND > > > 0 422 0 0 -16 0 0 0 pftm DL ?? > 1362773081:04.00 > > > [pfpurge] > > > > ------------------------------------------------------------------------ > > > > > > > ------------------------------------------------------------------------ > > > vmstat -z > > > > > > ITEM SIZE LIMIT USED FREE REQUESTS > > > FAILURES > > > pfsrctrpl: 152, 10000, 0, 0, 0, > > > 0 > > > pfrulepl: 912, 0, 40, 88, 806, > > > 0 > > > pfstatepl: 392, 10000, 10000, 0, 4440553, > > > 341638 > > > pfaltqpl: 240, 0, 0, 0, 0, > > > 0 > > > pfpooladdrpl: 88, 0, 0, 0, 0, > > > 0 > > > pfrktable: 1296, 1002, 4, 20, 112, > > > 0 > > > pfrkentry: 216, 100008, 603, 891, 15384, > > > 0 > > > pfrkentry2: 216, 0, 0, 0, 0, > > > 0 > > > pffrent: 32, 5050, 0, 303, 1620, > > > 0 > > > pffrag: 80, 0, 0, 135, 807, > > > 0 > > > pffrcache: 80, 10035, 0, 0, 0, > > > 0 > > > pffrcent: 24, 50022, 0, 0, 0, > > > 0 > > > pfstatescrub: 40, 0, 0, 0, 0, > > > 0 > > > pfiaddrpl: 120, 0, 0, 0, 0, > > > 0 > > > pfospfen: 112, 0, 696, 30, 18096, > > > 0 > > > pfosfp: 40, 0, 407, 97, 10582, > > > 0 > > > > ------------------------------------------------------------------------ > > > > > > You can see evidence of processes not exiting/doing what they should do > > > here: > > > > > > > ------------------------------------------------------------------------ > > > fstat > > > > > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > > > root shutdown 91155 root / 2 drwxr-xr-x 512 r > > > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > > > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > > > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > > > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > > > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > > > root sh 91129 root / 2 drwxr-xr-x 512 r > > > root sh 91129 wd / 2 drwxr-xr-x 512 r > > > root sh 91129 text / 44 -r-xr-xr-x 134848 r > > > root sh 91129 0 /dev 38 crw------- ttyu0 rw > > > root sh 91129 1* pipe ffffff01e78fc9e0 <-> > ffffff01e78fc888 > > > 0 rw > > > root sh 91129 2 /dev 20 crw-rw-rw- null w > > > root shutdown 91115 root / 2 drwxr-xr-x 512 r > > > root shutdown 91115 wd /storage 5 drwx------ 37 r > > > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > > > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 3* local dgram ffffff008ff92960 > > > root sh 90818 root / 2 drwxr-xr-x 512 r > > > root sh 90818 wd / 70659 drwxr-xr-x 512 r > > > root sh 90818 text / 44 -r-xr-xr-x 134848 r > > > root sh 90818 0 /dev 38 crw------- ttyu0 rw > > > root sh 90818 1* pipe ffffff0043f1ecb8 <-> > ffffff0043f1eb60 > > > 0 rw > > > root sh 90818 2 /dev 20 crw-rw-rw- null w > > > root csh 90802 root / 2 drwxr-xr-x 512 r > > > root csh 90802 wd / 2 drwxr-xr-x 512 r > > > root csh 90802 text / 51 -r-xr-xr-x 358752 r > > > root csh 90802 15 /dev 38 crw------- ttyu0 rw > > > root csh 90802 16 /dev 38 crw------- ttyu0 rw > > > root csh 90802 17 /dev 38 crw------- ttyu0 rw > > > root csh 90802 18 /dev 38 crw------- ttyu0 rw > > > root csh 90802 19 /dev 38 crw------- ttyu0 rw > > > > ------------------------------------------------------------------------ > > > > > > No indication of mbuf exhaustion, putting further focus on the pf > stack: > > > > > > > ------------------------------------------------------------------------ > > > netstat -m > > > > > > 2054/1786/3840 mbufs in use (current/cache/total) > > > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > > > 2048/896 mbuf+clusters out of packet secondary zone in use > (current/cache) > > > 0/320/320/12800 4k (page size) jumbo clusters in use > > > (current/cache/total/max) > > > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > > > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > > > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > > 0 requests for sfbufs denied > > > 0 requests for sfbufs delayed > > > 0 requests for I/O initiated by sendfile > > > 0 calls to protocol drain routines > > > > ------------------------------------------------------------------------ > > > > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > > column. I have a very hard time believing that was the interrupt rate > > > of all the relevant devices at the time (way too high). Maybe this > data > > > becomes wrong only during a coredump? The total column I could > believe. > > > > > > > ------------------------------------------------------------------------ > > > vmstat -i > > > > > > interrupt total rate > > > irq4: uart0 54768 912 > > > irq6: fdc0 1 0 > > > irq17: uhci1+ 172 2 > > > irq23: uhci3 ehci1+ 2367 39 > > > cpu0: timer 13183882632 219731377 > > > irq256: em0 260491055 4341517 > > > irq257: em1 127555036 2125917 > > > irq258: ahci0 225923164 3765386 > > > cpu2: timer 13183881837 219731363 > > > cpu1: timer 13002196469 216703274 > > > cpu3: timer 13183881783 219731363 > > > Total 53167869284 886131154 > > > > ------------------------------------------------------------------------ > > > > > > Here's what a normal "vmstat -i" shows from the command-line: > > > > > > # vmstat -i > > > interrupt total rate > > > irq4: uart0 518 0 > > > irq6: fdc0 1 0 > > > irq23: uhci3 ehci1+ 145 0 > > > cpu0: timer 19041199 1999 > > > irq256: em0 614280 64 > > > irq257: em1 168529 17 > > > irq258: ahci0 355536 37 > > > cpu2: timer 19040462 1999 > > > cpu1: timer 19040458 1999 > > > cpu3: timer 19040454 1999 > > > Total 77301582 8119 > > > > > > We graph many aspects of this box, including CPU load, memory/swap > > > usage, etc. and none show any sign that the interrupt rate on all of > > > those devices was even remotely out of control. (I would expect to see > > > CPU through the roof given the above data) > > > > > > I have since rebuilt/reinstalled world/kernel on the machine with the > > > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > > > 2011), hoping whatever this was may have been fixed. > > > > > > As for what I think may have triggered it, but I have no hard evidence > > > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > > > reload". The pf.conf change was a single line: > > > > > > Old: scrub on em0 all > > > New: scrub in on em0 all > > > > > > Why it took the problem approximately 3 days to start is unknown. It's > > > the only change we've made to the system (truly/honestly), and it was a > > > change to pf.conf. > > > > > > If anyone has advice (or has seen the above problem), or is interested > > > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > > > any way I can. I would hate for someone else to get bit by this, and > > > really am hoping its something that has been fixed between February and > > > now. > > > > > > > > I'm seeing this as well. You could change your scrub rules so that you > > specifically avoid TCP reassembly (that creates states). > > Thank you very much. This helps -- and I'm also glad someone else has > seen this behaviour. That confirms it's not specific to my equipment, > which is good. > > Regarding scrubbing and TCP reassembly (option "reassemble tcp" > according to pf.conf): I wasn't under the impression this option was > enabled by default. This got me wondering what the defaults actually > are (pf.conf(5) is somewhat vague in this regard, but it does state > that "fragment reassemble" is enabled by default). The rule I use: > > scrub in on em0 all > > Appears to get evaluated into this (per "pfctl -s rules -v"): > > scrub in on em0 all fragment reassemble > Ah, ok, my bad then. I used to have "reassemble tcp" enabled and disabled it this morning. However, if your symptoms are there without it, that's bad news. I disabled tcp reassembly to make sure there are no other states than the ones I allow via explicit keep/modulate/synproxy state rules. Disabling scrubbing altogether seems like a good next step. > Did you mean to tell me to disable the "fragment reassemble" option > (which is different from "reassemble tcp")? If so, how do I do that? > It looks like I could use a "no scrub" rule, except I can't find any > examples of this on the web or in the docs. Or is it just better to > remove use of scrub entirely until whatever this is gets fixed? > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > -- Good, fast & cheap. Pick any two. From owner-freebsd-stable@FreeBSD.ORG Tue May 3 06:26:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C84A21065670 for ; Tue, 3 May 2011 06:26:23 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4E9938FC1C for ; Tue, 3 May 2011 06:26:22 +0000 (UTC) Received: by bwz12 with SMTP id 12so7507233bwz.13 for ; Mon, 02 May 2011 23:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=xURSY1cdM0Aa4wUs6PJbV1yOQ6Yyj8/IUNdX4VQ7nXo=; b=HFufKhk608wAgcjRUrZUeBFUpmdDF1bc7hg7bh9s70qFb+Gyl0mHjUICyqHy0K34lB gPytF9dnvke108Z3qDdi1idpMIzwwLixxdxRVTalzMD29npGkWZ2UNm44FyHzeoFYRRi 6OumzvDeUiEhvNET5UTMm7jlpB0GukiV+Gou4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=YtBYdFINg16vVhJvYNtGWIwzqzXeVh/xIiuXJzTKH5Y9mRrk3e+Nlv10/NQ8o1Ttrg t67sKfExLeuvc4IYbYkGWjk+jhCjXoe8OqaVLC7zpsMnRfNKsD6yinMvT30VYUNm8HbO 2sspa24TjVcZIcBiRhab18lDMw9tsPrwUhQNs= MIME-Version: 1.0 Received: by 10.204.7.213 with SMTP id e21mr1316571bke.209.1304403981553; Mon, 02 May 2011 23:26:21 -0700 (PDT) Received: by 10.204.78.134 with HTTP; Mon, 2 May 2011 23:26:21 -0700 (PDT) In-Reply-To: <20110502164222.GE81300@mr-happy.com> References: <20110502164222.GE81300@mr-happy.com> Date: Tue, 3 May 2011 01:26:21 -0500 Message-ID: From: Scot Hetzel To: Jeff Blank Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:26:23 -0000 On Mon, May 2, 2011 at 11:42 AM, Jeff Blank wrote: > Hi, > > I recently upgraded from 8.0-STABLE to 8.2-STABLE (Apr. 29 checkout) > and upgraded my zpool (includes root FS) from v13 to v15. =A0This is a > dual-boot laptop, so I'm using MBR/boot0 and not GPT. =A0Here's what > happens when I boot: > > F1 =A0Win > F2 =A0? > F3 =A0FreeBSD > > F6 PXE > Boot: =A0F3 > ZFS: unsupported ZFS version 15 (should be 13) > No ZFS pools located, can't boot > > I've googled around, but I can't find anything relevant for MBR/boot0 > configurations, just GPT. =A0I've ensured that the loaders and > boot0/boot1/boot2 are all new, and I rebuilt/reinstalled them in a > fixit environment just to be sure. =A0I also ran 'boot0cfg -B' (with an > appropriate -b), but nothing has changed. =A0How can I get my pool > booting again? > You need to re-install the zfsboot code similar to step 10 (Install ZFS boot) in http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition Scot From owner-freebsd-stable@FreeBSD.ORG Tue May 3 06:29:33 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DA58106566C; Tue, 3 May 2011 06:29:33 +0000 (UTC) (envelope-from snabb@epipe.com) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0AB538FC19; Tue, 3 May 2011 06:29:32 +0000 (UTC) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by tiktik.epipe.com (8.14.4/8.14.4) with ESMTP id p436TT0K022213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 06:29:30 GMT (envelope-from snabb@epipe.com) X-DKIM: Sendmail DKIM Filter v2.8.3 tiktik.epipe.com p436TT0K022213 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=epipe.com; s=default; t=1304404170; x=1305008970; bh=uz5U/CRUnb5x4xoCMzRhNwlsTr1SkMKxm+/BIfct7rc=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=aQM90aRv08ZtDD1p7KCqj7HcAUj05Wtta4CVhMs7bp8S3hIQRtGgIFH4mUz+uoBpy bHpK7w1kKyY5iugl/WyTg/8WDw6WkF8bA8S5+D0I2OSXWxG/OiYQFL9IzT18kHfQgT drmK1HQUxCitPlV0o/MNDBFTOeBcxp6kSfuRroVU= Date: Tue, 3 May 2011 06:29:29 +0000 (UTC) From: Janne Snabb To: Vlad Galu In-Reply-To: Message-ID: References: <20110503015854.GA31444@icarus.home.lan> <20110503060106.GA36331@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (tiktik.epipe.com [IPv6:2001:1828:0:3::2]); Tue, 03 May 2011 06:29:30 +0000 (UTC) Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:29:33 -0000 On Tue, 3 May 2011, Vlad Galu wrote: > Disabling scrubbing altogether seems like a good next step. I used to get all kinds of strange problems when I tried scrubbing on FreeBSD 8.1. Especially with IPv6 traffic. After I disabled scrubbing altogether I have had zero problems. The IP & TCP stacks behind this particular pf are good ones anyway, so scrubbing was useless anyway. My belief is that scrubbing is just broken, but I do not have any hard facts about it. I did not bother wasting my time trying to debug it after I noticed that the pf code has not been updated from the upstream for quite a while. The first thing would be to get on the same level with the upstream in case the problem is fixed there. However, I do not want to touch OpenBSD code for personal reasons. -- Janne Snabb / EPIPE Communications snabb@epipe.com - http://epipe.com/ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 07:29:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BD2F106566C for ; Tue, 3 May 2011 07:29:26 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 69EEA8FC08 for ; Tue, 3 May 2011 07:29:25 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p4370gxU015502 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 09:00:42 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p4370gYQ032745; Tue, 3 May 2011 09:00:42 +0200 (MEST) Date: Tue, 3 May 2011 09:00:42 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503070042.GA9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 07:29:26 -0000 I read those graphs differently: the problem doesn't arise slowly, but rather seems to start suddenly at 13:00. Right after 13:00, traffic on em0 drops, i.e. the firewall seems to stop forwarding packets completely. Yet, at the same time, the states start to increase, almost linearly at about one state every two seconds, until the limit of 10,000 is reached. Reaching the limit seems to be only a side-effect of a problem that started at 13:00. > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ I find this suspect as well, but I don't have an explanation yet. Are you using anything non-GENERIC related to timers, like change HZ or enable polling? Are you sure the problem didn't start right at 13:00, and cause complete packet loss for the entire period, and that it grew gradually worse instead? Daniel From owner-freebsd-stable@FreeBSD.ORG Tue May 3 07:32:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A62B1065670 for ; Tue, 3 May 2011 07:32:12 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.emeryville.ca.mail.comcast.net (qmta02.emeryville.ca.mail.comcast.net [76.96.30.24]) by mx1.freebsd.org (Postfix) with ESMTP id 000C28FC1A for ; Tue, 3 May 2011 07:32:11 +0000 (UTC) Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27]) by qmta02.emeryville.ca.mail.comcast.net with comcast id evXq1g0010b6N64A2vYBSs; Tue, 03 May 2011 07:32:11 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.emeryville.ca.mail.comcast.net with comcast id evY91g0041t3BNj8PvY9Nh; Tue, 03 May 2011 07:32:11 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7088E9B418; Tue, 3 May 2011 00:32:09 -0700 (PDT) Date: Tue, 3 May 2011 00:32:09 -0700 From: Jeremy Chadwick To: Daniel Hartmeier Message-ID: <20110503073209.GA37676@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503070042.GA9657@insomnia.benzedrine.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503070042.GA9657@insomnia.benzedrine.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 07:32:12 -0000 On Tue, May 03, 2011 at 09:00:42AM +0200, Daniel Hartmeier wrote: > I read those graphs differently: the problem doesn't arise slowly, > but rather seems to start suddenly at 13:00. > > Right after 13:00, traffic on em0 drops, i.e. the firewall seems > to stop forwarding packets completely. > > Yet, at the same time, the states start to increase, almost linearly > at about one state every two seconds, until the limit of 10,000 is > reached. Reaching the limit seems to be only a side-effect of a > problem that started at 13:00. > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > column. I have a very hard time believing that was the interrupt rate > > of all the relevant devices at the time (way too high). Maybe this data > > becomes wrong only during a coredump? The total column I could believe. > > > > ------------------------------------------------------------------------ > > vmstat -i > > > > interrupt total rate > > irq4: uart0 54768 912 > > irq6: fdc0 1 0 > > irq17: uhci1+ 172 2 > > irq23: uhci3 ehci1+ 2367 39 > > cpu0: timer 13183882632 219731377 > > irq256: em0 260491055 4341517 > > irq257: em1 127555036 2125917 > > irq258: ahci0 225923164 3765386 > > cpu2: timer 13183881837 219731363 > > cpu1: timer 13002196469 216703274 > > cpu3: timer 13183881783 219731363 > > Total 53167869284 886131154 > > ------------------------------------------------------------------------ > > I find this suspect as well, but I don't have an explanation yet. > > Are you using anything non-GENERIC related to timers, like change > HZ or enable polling? HZ is standard (1000 is the default I believe), and I do not use polling. > Are you sure the problem didn't start right at 13:00, and cause complete > packet loss for the entire period, and that it grew gradually worse > instead? It's hard to discern from the graphs, but I can tell you exactly what I saw TCP-wise since I did have some already-existing/established TCP connections to the box (e.g. connections which already had ESTABLISHED states according to pfctl -s state) when it began exhibiting issues. Any packets which already had existing state entries in pf's state table continued to work, and bidirectionally. New inbound connections to the box via em0 would result in no response/timeout (and as indicated per pfctl, such packets were being dropped due to the state limit being reached). Outbound connections from the box via em0 to the outside world also resulted in no response/timeout. I will show you evidence of the latter. The first indication of a problem in syslog is the following message from named -- this is the first in my entire life I've ever seen this message, but seems to indicate some kind of internal watchdog was fired within named itself. The log I'm looking at, by the way, is /var/log/all.log -- yes, I do turn that on (for reasons exactly like this). This box is a secondary nameserver (public), so keep that in mind too. Anyway: May 1 12:50:14 isis named[728]: *** POKED TIMER *** Seconds later, I see unexpected RCODE messages, lame server messages, etc.. -- all which indicate packets to some degree are working ("the usual" badly-configured nameservers on the Internet). A few minutes later: May 1 12:53:15 isis named[728]: *** POKED TIMER *** May 1 12:53:54 isis named[728]: *** POKED TIMER *** With more of the usual unexpected RCODE/SERVFAIL messages after that. The next message: May 1 13:28:55 isis named[728]: *** POKED TIMER *** May 1 13:29:13 isis named[728]: *** POKED TIMER *** May 1 13:30:11 isis last message repeated 3 times Then more RCODE/SERVFAIL and something called "FORMERR" but that could be normal as well. Remember, all from named. This "cycle" of behaviour continued, with the number of POKED TIMER messages gradually increasing more and more as time went on. By 16:07 on May 1st, these messages were arriving usually in "bursts" of 5 or 6. Things finally "exploded", from named's perspective, here (with slaved zones X'd out): May 1 19:23:21 isis named[728]: *** POKED TIMER *** May 1 19:28:59 isis named[728]: zone XXXXXXXX/IN: refresh: failure trying master x.x.x.x#53 (source x.x.x.x#0): operation canceled May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 213.179.160.66#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 193.0.12.4#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 193.194.64.242#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 192.134.0.49#53 And many other slaved zones reporting the exact same error. The hostnames which couldn't be resolved have no relevancy (honestly there is no consistency in any of them, and given what our service does, the randomness is expected. (Read: I just picked linear log lines above; don't focus on dns2.djaweb.dz like it's somehow responsible for this)) To me, this indicates a serious IP stack, firewall, or network exhaustion situation. The "host unreachable" messages became more and more regular, along with the occasional POKED TIMER. So what I'm saying is that the "most chatty" of all the daemons on the machine during this time was named, which does not surprise me given the machine's role. I finally logged in on May 2nd at 13:41 and tried to put an end to the madness. Other non-syslog-based daemons on the machine doing logging show no signs of issues, I/O errors (network or otherwise), etc.. Again, this doesn't surprise me, because nobody with existing connections to the machine was complaining (and hadn't). Only when I was told by a user that *new* connections were failing did I begin to investigate. So, it seems likely that network I/O mostly stopped because the IP stack stopped handling new I/O due to whatever was going on with pf. While at the same time, pf's state counter started gradually incrementing for reasons unknown -- an indicator that something bad was happening, almost certainly within pf itself, or somewhere within the kernel. I'm inclined to believe pf, because existing/ESTABLISHED stateful entries continued to get processed okay. Therefore, to me, the graphs are indicators that the problem started at around 12:50 PDT (meaning impact began at 12:50 PDT), and something very bad continued to happen within the pf stack, until I managed to intervene. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 08:46:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E509106566B for ; Tue, 3 May 2011 08:46:39 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from fallbackmx09.syd.optusnet.com.au (fallbackmx09.syd.optusnet.com.au [211.29.132.242]) by mx1.freebsd.org (Postfix) with ESMTP id AB6CB8FC25 for ; Tue, 3 May 2011 08:46:38 +0000 (UTC) Received: from mail14.syd.optusnet.com.au (mail14.syd.optusnet.com.au [211.29.132.195]) by fallbackmx09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p437M83S002458 for ; Tue, 3 May 2011 17:22:08 +1000 Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail14.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p437Lsdj016726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 17:21:55 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.4/8.14.4) with ESMTP id p437Lrxu079007; Tue, 3 May 2011 17:21:53 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.4/8.14.4/Submit) id p437LqcU079006; Tue, 3 May 2011 17:21:52 +1000 (EST) (envelope-from peter) Date: Tue, 3 May 2011 17:21:52 +1000 From: Peter Jeremy To: Olaf Seibert Message-ID: <20110503072152.GA78915@server.vk2pj.dyndns.org> References: <20110502143230.GW6733@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OgqxwSJOaUobr8KG" Content-Disposition: inline In-Reply-To: <20110502143230.GW6733@twoquid.cs.ru.nl> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 08:46:39 -0000 --OgqxwSJOaUobr8KG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-May-02 16:32:30 +0200, Olaf Seibert wrote: >However, it doesn't automatically reboot in 15 seconds, as promised. >It just sits there the whole weekend, until I log onto the IPMI console >and press the virtual reset button. Your reference to IMPI indicates this is not a consumer PC. Can you please provide some details of the hardware. Are you running ipmitools or similar? Does "shutdown -r" or "reboot" work normally? >panic: kmem_alloc(131072): kmem_map too small: 3428782080 total allocated I suggest you have a read of the thread beginning http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010862.html (note that mailman has split it into at least 3 threads). --=20 Peter Jeremy --OgqxwSJOaUobr8KG Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iEYEARECAAYFAk2/rRAACgkQ/opHv/APuIeROgCfbFLnO0IrMr4CxC0TpmJArXan LwMAoLgCsLX84CHl9Uahlj4/+6WR18VZ =uOHj -----END PGP SIGNATURE----- --OgqxwSJOaUobr8KG-- From owner-freebsd-stable@FreeBSD.ORG Tue May 3 08:48:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17146106566B; Tue, 3 May 2011 08:48:02 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 731378FC12; Tue, 3 May 2011 08:48:01 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p438m1oY009095 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 10:48:01 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p438m0eU029544; Tue, 3 May 2011 10:48:00 +0200 (MEST) Date: Tue, 3 May 2011 10:48:00 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503084800.GB9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 08:48:02 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > Status: Enabled for 76 days 06:49:10 Debug: Urgent > The "pf uptime" shown above, by the way, matches system uptime. > ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] This looks weird, too. 1362773081 minutes would be >2500 years. Usually, you should see [idle] with almost uptime in minutes, and [pfpurge] with much less, like in # uptime 10:22AM up 87 days, 19:36, 1 user, load averages: 0.00, 0.03, 0.05 # echo "((87*24)+19)*60+36" | bc 126456 # ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 7 0 0 44 0 0 8 pftm DL ?? 0:13.16 [pfpurge] 0 11 0 0 171 0 0 8 - RL ?? 124311:23.04 [idle] How is time handled on your machine? ntpdate on boot and then ntpd? Any manual time changes since the last boot? Daniel From owner-freebsd-stable@FreeBSD.ORG Tue May 3 09:16:22 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26DD0106567B for ; Tue, 3 May 2011 09:16:22 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.westchester.pa.mail.comcast.net (qmta01.westchester.pa.mail.comcast.net [76.96.62.16]) by mx1.freebsd.org (Postfix) with ESMTP id C23F38FC1D for ; Tue, 3 May 2011 09:16:21 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta01.westchester.pa.mail.comcast.net with comcast id exFF1g0010bG4ec51xGNng; Tue, 03 May 2011 09:16:22 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.westchester.pa.mail.comcast.net with comcast id exGL1g00A1t3BNj3PxGMoV; Tue, 03 May 2011 09:16:21 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 486319B418; Tue, 3 May 2011 02:16:19 -0700 (PDT) Date: Tue, 3 May 2011 02:16:19 -0700 From: Jeremy Chadwick To: Daniel Hartmeier Message-ID: <20110503091619.GA39329@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503084800.GB9657@insomnia.benzedrine.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:16:22 -0000 On Tue, May 03, 2011 at 10:48:00AM +0200, Daniel Hartmeier wrote: > On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > The "pf uptime" shown above, by the way, matches system uptime. > > > ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] > > This looks weird, too. 1362773081 minutes would be >2500 years. > > Usually, you should see [idle] with almost uptime in minutes, and > [pfpurge] with much less, like in > > # uptime > 10:22AM up 87 days, 19:36, 1 user, load averages: 0.00, 0.03, 0.05 > # echo "((87*24)+19)*60+36" | bc > 126456 > > # ps -axl > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 7 0 0 44 0 0 8 pftm DL ?? 0:13.16 [pfpurge] > 0 11 0 0 171 0 0 8 - RL ?? 124311:23.04 [idle] Agreed -- and that's exactly how things look on the same box right now: $ ps -axl | egrep 'UID|pfpurge|idle' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 11 0 0 171 0 0 64 - RL ?? 2375:15.91 [idle] 0 422 0 0 -16 0 0 16 pftm DL ?? 0:00.28 [pfpurge] The ps -axl output I provided earlier came from /var/crash/core.0.txt. So it's interesting that ps -axl as well as vmstat -i both showed something off-the-wall. I wonder if this can happen when within ddb? Unsure. I do have the core from "call doadump", so I should be able to go back and re-examine it with kgdb. I just wish I knew what to poke around looking for in there. Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt usage, etc. otherwise I'd be graphing that. The more monitoring the better; at least then I could say "wow, interrupts really did shoot through the roof -- the box went crazy!" and RMA the thing. :-) > How is time handled on your machine? ntpdate on boot and then ntpd? Yep, you got it: ntpdate_enable="yes" ntpdate_config="/conf/ME/ntp.conf" ntpd_enable="yes" ntpd_config="/conf/ME/ntp.conf" I don't use ntpd_sync_on_start because I've never had reason to. I always set the system/BIOS clock to UTC time when building a system. I use ntpd's complaint about excessive offset as an indicator that something bad happened. /conf/ME/ntp.conf on this machine syncs from another on the private network (em1) only, and that machine syncs from a series of geographically-diverse stratum 2 servers and one stratum 1 server. I've never seen high delays, offsets, or jitter using "ntpq -c peers" on any box we have. Actual timecounters (not time itself) are handled by ACPI-safe or ACPI-fast (varies per boot; I've talked to jhb@ about this before and it's normal). powerd is in use on all our systems, and on this box use of processor sleep states (lowest state = C2; physical CPU only supports C0-C2 and I wouldn't go any lower than that anyway :-) ). Appropriate /boot/loader.conf entries that pertain to it: # Enable use of P-state CPU frequency throttling. # http://wiki.freebsd.org/TuningPowerConsumption hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" There are numerous other systems exactly like this one (literally same model of hardware, RAM amount, CPU model, BIOS version and settings, and system configuration, including pf) that have much higher load and fire many more interrupts (particularly the NFS server!) that haven't exhibited any problems. This box had an uptime of 72 days, and prior to that around 100 (before being taken down for world/kernel upgrades). All machines have ECC RAM too, and MCA/MCE is in use. You don't know how bad I'd love to blame this on a hardware issue (it's always possible in some way or another), but the way this manifest itself was extremely specific. The problem could be super rare and something triggered it that hasn't been seen before by developers. So far there's only 1 other user who has seen this behaviour but his was attributed to use of "reassemble tcp" which I wasn't using; so the true problem could still be out there. I feel better knowing I'm not the only one who's seen this oddity. Since his post, I've removed all scrub rules from all of our machines as a precaution. If it ever happens again we'll have one more thing to safely rule out. We have other machines (different hardware, running RELENG_7 i386) which have had 1+ year uptimes also using pf, so the possibility of just some "crazy fluke" is plausible to me. > Any manual time changes since the last boot? None unless adjkerntz did something during the PST->PDT switchover, but that would manifest itself as a +1 hour offset difference. Since the machine rebooted the system synced its time without issue and well within acceptable delta (1.075993 sec). I did not power-cycle the box during any of this; pure soft reboots. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 09:21:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15359106566C for ; Tue, 3 May 2011 09:21:15 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id ED1D88FC1C for ; Tue, 3 May 2011 09:21:14 +0000 (UTC) Received: from omta07.emeryville.ca.mail.comcast.net ([76.96.30.59]) by qmta11.emeryville.ca.mail.comcast.net with comcast id exMB1g0021GXsucABxMElS; Tue, 03 May 2011 09:21:14 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta07.emeryville.ca.mail.comcast.net with comcast id exMD1g0061t3BNj8UxMDqa; Tue, 03 May 2011 09:21:14 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 0FE709B418; Tue, 3 May 2011 02:21:13 -0700 (PDT) Date: Tue, 3 May 2011 02:21:13 -0700 From: Jeremy Chadwick To: Olaf Seibert Message-ID: <20110503092113.GA39704@icarus.home.lan> References: <20110502143230.GW6733@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110502143230.GW6733@twoquid.cs.ru.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:21:15 -0000 On Mon, May 02, 2011 at 04:32:30PM +0200, Olaf Seibert wrote: > I have a FreeBSD/amd64 8.2 server that has a few ZFS file systems served > over NFS. It has 8 GB of memory. There are 6 disks of 1,5 TB each > forming a pool with raidz2. > > >From time to time it crashes with some stack backtrace (included below). > This already happened before the upgrade to 8.2. > > Now a crash of a file server is annoying, but if it reboots > automatically, there is just a few minutes of downtime (most of it is > even spent by the BIOS before it gets to boot the OS). > > However, it doesn't automatically reboot in 15 seconds, as promised. > It just sits there the whole weekend, until I log onto the IPMI console > and press the virtual reset button. There are two things you might try fiddling with. These are sysctls so you can try them on the fly: hw.acpi.disable_on_reboot hw.acpi.handle_reboot On our systems we set hw.acpi.handle_reboot=1 to speed up the reboot process. I remember hearing long ago how some people had issues getting their machines to reboot (sometimes 100% of the time, other times occasionally); using ACPI to reboot the machine fixed their issues. > This was visible before I did that (4-finger copy): > > panic: kmem_alloc(131072): kmem_map too small: 3428782080 total allocated > cpuid = 0 Check out the thread Peter Jeremy provided. This is a near-sure indicator of ZFS ARC exhaustion, and you seem to know of that. What's very interesting to me is this part of your mail: > There is some tuning in /boot/loader.conf from previous attempts tune to > avoid crashes. > > vm.kmem_size="16G" > vfs.zfs.arc_max="4G" > > Is that still useful, or does it harm by now? Real memory is 8 GB. > I note that if I look with sysctl, I see > > vm.kmem_size: 3739230208 > vfs.zfs.arc_max: 2665488384 > > which doesn't seem to match these attempted settings. Is this box running i386 or amd64? If amd64, I can't explain why your /boot/loader.conf settings aren't taking -- they should be for sure. Maybe provide us a full dmesg and XXX out things you consider sensitive. If i386, I'm not too surprised that some automatic defaults get chosen instead of what you ask. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 09:22:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 639CE106567A; Tue, 3 May 2011 09:22:59 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 8EE768FC12; Tue, 3 May 2011 09:22:57 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p439MvwH015703 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 11:22:57 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p439Mvm4009848; Tue, 3 May 2011 11:22:57 +0200 (MEST) Date: Tue, 3 May 2011 11:22:57 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503092257.GC9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:22:59 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ > > Here's what a normal "vmstat -i" shows from the command-line: > > # vmstat -i > interrupt total rate > irq4: uart0 518 0 > irq6: fdc0 1 0 > irq23: uhci3 ehci1+ 145 0 > cpu0: timer 19041199 1999 > irq256: em0 614280 64 > irq257: em1 168529 17 > irq258: ahci0 355536 37 > cpu2: timer 19040462 1999 > cpu1: timer 19040458 1999 > cpu3: timer 19040454 1999 > Total 77301582 8119 The cpu0-3 timer totals seem consistent in the first output: 13183881783/1999/60/60/24 matches 76 days of uptime. The high rate in the first output comes from vmstat.c dointr()'s division of the total by the uptime: struct timespec sp; clock_gettime(CLOCK_MONOTONIC, &sp); uptime = sp.tv_sec; for (i = 0; i < nintr; i++) { printf("%-*s %20lu %10lu\n", istrnamlen, intrname, *intrcnt, *intrcnt / uptime); } >From this we can deduce that the value of uptime must have been 13183881783/219731363 = 60 (seconds). Since the uptime was 76 days (and not just 60 seconds), the CLOCK_MONOTONIC clock must have reset, wrapped, or been overwritten. I don't know how that's possible, but if this means that the kernel variable time_second was possibly going back, that could very well have messed up pf's state purging. Daniel From owner-freebsd-stable@FreeBSD.ORG Tue May 3 09:32:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F11D106564A; Tue, 3 May 2011 09:32:02 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id A75BA8FC18; Tue, 3 May 2011 09:32:01 +0000 (UTC) Received: from vhoffman.lon.namesco.net (lon.namesco.net [195.7.254.102]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id p439VvXc016393 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 3 May 2011 10:31:58 +0100 (BST) (envelope-from vince@unsane.co.uk) Message-ID: <4DBFCB8D.10105@unsane.co.uk> Date: Tue, 03 May 2011 10:31:57 +0100 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Jeremy Chadwick References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> In-Reply-To: <20110503091619.GA39329@icarus.home.lan> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniel Hartmeier , freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:32:02 -0000 On 03/05/2011 10:16, Jeremy Chadwick wrote: > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > usage, etc. otherwise I'd be graphing that. The more monitoring the > better; at least then I could say "wow, interrupts really did shoot > through the roof -- the box went crazy!" and RMA the thing. :-) > you could use net-mgmt/bsnmp-regex although I dont know what the overhead for that is like. Vince From owner-freebsd-stable@FreeBSD.ORG Tue May 3 09:49:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2D97106564A for ; Tue, 3 May 2011 09:49:34 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by mx1.freebsd.org (Postfix) with ESMTP id 830D08FC1F for ; Tue, 3 May 2011 09:49:34 +0000 (UTC) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id p439nSA1020828; Tue, 3 May 2011 11:49:28 +0200 (MEST) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id B49B12E05D; Tue, 3 May 2011 11:49:29 +0200 (CEST) Date: Tue, 3 May 2011 11:49:29 +0200 From: Olaf Seibert To: Peter Jeremy Message-ID: <20110503094929.GX6733@twoquid.cs.ru.nl> References: <20110502143230.GW6733@twoquid.cs.ru.nl> <20110503072152.GA78915@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503072152.GA78915@server.vk2pj.dyndns.org> User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Cc: freebsd-stable@freebsd.org, Olaf Seibert Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:49:35 -0000 On Tue 03 May 2011 at 17:21:52 +1000, Peter Jeremy wrote: > On 2011-May-02 16:32:30 +0200, Olaf Seibert wrote: > >However, it doesn't automatically reboot in 15 seconds, as promised. > >It just sits there the whole weekend, until I log onto the IPMI console > >and press the virtual reset button. > > Your reference to IMPI indicates this is not a consumer PC. Can you > please provide some details of the hardware. It is a Supermicro H8DME-2 motherboard with 2 dual Opteron S-F 2000 series CPUs (according to the spec sheet I have here). The IPMI console (front-end processor as one would call it in the mainframe years ;-) is an AOC-SiM1U+ with KVM over a dedicated LAN port. I usually access it via its built-in webserver. I have appendend dmesg.boot at the end. > Are you running ipmitools or similar? Not so far. > Does "shutdown -r" or "reboot" work normally? Yes, when I last used it while upgrading from 8.1 to 8.2 "shutdown -r" worked fine, and on previous upgrades it worked too. I can possibly imagine that the IPMI console would press a key just at this inconvenient moment (so that the fault is entirely outside FreeBSD's domain), but since it doesn't seem to do this at other moments, it seems unlikely. Would pressing a key like "shift" stop a reboot? > >panic: kmem_alloc(131072): kmem_map too small: 3428782080 total allocated > > I suggest you have a read of the thread beginning > http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010862.html > (note that mailman has split it into at least 3 threads). Thanks for the link. There seem to be some contradictory advices there though: tune vm.kmem_size to twice the physical RAM size, or the same size, or even 1,5 times. Apparently it is supposed to default to 1 x RAM size, but for some reason on this machine it doesn't: $ sysctl hw.realmem hw.physmem hw.usermem vm.kmem_size hw.realmem: 9126805504 hw.physmem: 8580272128 hw.usermem: 3317899264 vm.kmem_size: 3739230208 $ sysctl vm.kmem_size_scale vm.kmem_size_scale: 1 despite even the tune to 2 x RAM size in /boot/loader.conf. I can imagine that since vfs.zfs.arc_max="4G" is larger than vm.kmem_size, this might present a problem. On the other hand the currently set value has apparently also been adjusted down: $ sysctl vfs.zfs.arc_max vfs.zfs.arc_max: 2665488384 This resembles the findings of Jeremy Chadwick in http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010880.html . I think, based on that, that I will simply take out these setting altogether, and after the next reboot we'll see how that affects matters. > -- > Peter Jeremy Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE #3: Tue Apr 19 13:02:11 CEST 2011 root@fourquid.cs.ru.nl:/usr/obj/usr/src/sys/FOURQUID amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Dual-Core AMD Opteron(tm) Processor 2212 (2010.32-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x40f13 Family = f Model = 41 Stepping = 3 Features=0x178bfbff Features2=0x2001 AMD Features=0xea500800 AMD Features2=0x1f real memory = 8589934592 (8192 MB) avail memory = 8267616256 (7884 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 2 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xfe9bf000-0xfe9bffff irq 22 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ehci0: mem 0xfe9bec00-0xfe9becff irq 23 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 4.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xd480-0xd487,0xd400-0xd403,0xd080-0xd087,0xd000-0xd003,0xcc00-0xcc0f mem 0xfe9bd000-0xfe9bdfff irq 21 at device 5.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0xc880-0xc887,0xc800-0xc803,0xc480-0xc487,0xc400-0xc403,0xc080-0xc08f mem 0xfe9bc000-0xfe9bcfff irq 22 at device 5.1 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] atapci3: port 0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb887,0xb800-0xb803,0xb480-0xb48f mem 0xfe9bb000-0xfe9bbfff irq 23 at device 5.2 on pci0 atapci3: [ITHREAD] ata6: on atapci3 ata6: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] pcib1: at device 6.0 on pci0 pci1: on pcib1 vgapci0: port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xfeaf0000-0xfeafffff irq 16 at device 5.0 on pci1 nfe0: port 0xb400-0xb407 mem 0xfe9ba000-0xfe9bafff,0xfe9be800-0xfe9be8ff,0xfe9be400-0xfe9be40f irq 20 at device 8.0 on pci0 miibus0: on nfe0 e1000phy0: PHY 2 on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow nfe0: Ethernet address: 00:30:48:cd:b9:9e nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe1: port 0xb080-0xb087 mem 0xfe9b9000-0xfe9b9fff,0xfe9be000-0xfe9be0ff,0xfe9b8c00-0xfe9b8c0f irq 20 at device 9.0 on pci0 miibus1: on nfe1 e1000phy1: PHY 3 on miibus1 e1000phy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow nfe1: Ethernet address: 00:30:48:cd:b9:9f nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] nfe1: [FILTER] pcib2: at device 10.0 on pci0 pci2: on pcib2 pcib3: at device 0.0 on pci2 pci3: on pcib3 pcib4: at device 0.1 on pci2 pci4: on pcib4 pcib5: at device 13.0 on pci0 pci5: on pcib5 pcib6: at device 14.0 on pci0 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 arcmsr0: mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 17 at device 14.0 on pci7 ARECA RAID ADAPTER0: Driver Version 1.20.00.19 2010-11-11 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 arcmsr0: [ITHREAD] pcib8: at device 0.2 on pci6 pci8: on pcib8 pcib9: at device 15.0 on pci0 pci9: on pcib9 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] atrtc0: port 0x70-0x71 irq 8 on acpi0 orm0: at iomem 0xc0000-0xcafff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 A kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range powernow0: on cpu0 powernow1: on cpu1 powernow2: on cpu2 powernow3: on cpu3 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 acd0: DVDROM at ata0-slave UDMA33 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ad4: 238475MB at ata2-master UDMA100 SATA 3Gb/s ad6: 238475MB at ata3-master UDMA100 SATA 3Gb/s GEOM_MIRROR: Device mirror/gm0 launched (2/2). uhub0: 10 ports with 10 removable, self powered da0 at arcmsr0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: Command Queueing enabled da0: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) da1 at arcmsr0 bus 0 scbus0 target 0 lun 1 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da1: Command Queueing enabled da1: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) da2 at arcmsr0 bus 0 scbus0 target 0 lun 2 da2: Fixed Direct Access SCSI-5 device da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da2: Command Queueing enabled da2: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) da3 at arcmsr0 bus 0 scbus0 target 0 lun 3 da3: Fixed Direct Access SCSI-5 device da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da3: Command Queueing enabled da3: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) da4 at arcmsr0 bus 0 scbus0 target 0 lun 4 da4: Fixed Direct Access SCSI-5 device da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da4: Command Queueing enabled da4: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) da5 at arcmsr0 bus 0 scbus0 target 0 lun 5 da5: Fixed Direct Access SCSI-5 device da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da5: Command Queueing enabled da5: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C) pass6 at arcmsr0 bus 0 scbus0 target 16 lun 0 pass6: Fixed Processor SCSI-0 device SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! GEOM: da0: partition 2 does not start on a track boundary. GEOM: da0: partition 2 does not end on a track boundary. GEOM: da0: partition 1 does not end on a track boundary. GEOM: da1: partition 2 does not start on a track boundary. GEOM: da1: partition 2 does not end on a track boundary. GEOM: da1: partition 1 does not end on a track boundary. Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered ugen1.2: at usbus1 ums0: on usbus1 ums0: 3 buttons and [Z] coordinates ID=0 ukbd0: on usbus1 kbd2 at ukbd0 Trying to mount root from ufs:/dev/mirror/gm0s1a WARNING: / was not properly dismounted ZFS filesystem version 4 ZFS storage pool version 15 WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted (the messages about "GEOM: da0: partition..." are remainders from some previously installed operating system that apparently were not overwritten. The da? disks store the zpool. -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe)); From owner-freebsd-stable@FreeBSD.ORG Tue May 3 10:08:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71B711065674 for ; Tue, 3 May 2011 10:08:59 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by mx1.freebsd.org (Postfix) with ESMTP id ECAB18FC1F for ; Tue, 3 May 2011 10:08:58 +0000 (UTC) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id p43A8swG021518; Tue, 3 May 2011 12:08:54 +0200 (MEST) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id F02132E05D; Tue, 3 May 2011 12:08:54 +0200 (CEST) Date: Tue, 3 May 2011 12:08:54 +0200 From: Olaf Seibert To: Jeremy Chadwick Message-ID: <20110503100854.GY6733@twoquid.cs.ru.nl> References: <20110502143230.GW6733@twoquid.cs.ru.nl> <20110503092113.GA39704@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503092113.GA39704@icarus.home.lan> User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Cc: freebsd-stable@freebsd.org, Olaf Seibert Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:08:59 -0000 On Tue 03 May 2011 at 02:21:13 -0700, Jeremy Chadwick wrote: > There are two things you might try fiddling with. These are sysctls so > you can try them on the fly: > > hw.acpi.disable_on_reboot > hw.acpi.handle_reboot Thanks. For now I've set the second to 1 and we'll see if that affects matters. > Check out the thread Peter Jeremy provided. This is a near-sure > indicator of ZFS ARC exhaustion, and you seem to know of that. What's > very interesting to me is this part of your mail: ... > > Is this box running i386 or amd64? If amd64, I can't explain why your It's amd64. I double-checked just one, you never know what stupid mistakes one might make :-) > /boot/loader.conf settings aren't taking -- they should be for sure. > Maybe provide us a full dmesg and XXX out things you consider > sensitive. If i386, I'm not too surprised that some automatic defaults > get chosen instead of what you ask. Based on one of your mails where setting vm.kmem_size to twice the real RAM size had adverse effects, I've taken the setting out to see if that improves matters. I'll have to wait until the next crash (or opportunity to reboot without too much disturbance) to see the effect. I put dmesg.boot in my other reply. Thanks, > | Jeremy Chadwick jdc@parodius.com | -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe)); From owner-freebsd-stable@FreeBSD.ORG Tue May 3 10:13:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FF081065672; Tue, 3 May 2011 10:13:36 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 444588FC13; Tue, 3 May 2011 10:13:35 +0000 (UTC) Received: by qyk27 with SMTP id 27so3979990qyk.13 for ; Tue, 03 May 2011 03:13:35 -0700 (PDT) Received: by 10.229.43.209 with SMTP id x17mr6945302qce.257.1304417615239; Tue, 03 May 2011 03:13:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Tue, 3 May 2011 03:12:55 -0700 (PDT) In-Reply-To: <4DBFCB8D.10105@unsane.co.uk> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> From: Vlad Galu Date: Tue, 3 May 2011 12:12:55 +0200 Message-ID: To: Vincent Hoffman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Daniel Hartmeier , freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:13:36 -0000 On Tue, May 3, 2011 at 11:31 AM, Vincent Hoffman wrote: > On 03/05/2011 10:16, Jeremy Chadwick wrote: > > > > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > > usage, etc. otherwise I'd be graphing that. The more monitoring the > > better; at least then I could say "wow, interrupts really did shoot > > through the roof -- the box went crazy!" and RMA the thing. :-) > > > you could use net-mgmt/bsnmp-regex although I dont know what the > overhead for that is like. > I use munin for graphing, as it allows easy scripting without using SNMP. My case is a bit different from Jeremy's. Every once in a while there is a sudden traffic spike which impacts pf performance as well. However, the graphed figures are nowhere near what I'd consider alarming levels (this box has withstood more in the past). I was able to coincidentally log in after such a spike and noticed the pfpurge thread eating up about 30% of the CPU while using the normal optimization policy. In my case, it could be related to another issue I'm seeing on this box - mbuma allocation failures. Here are my graphs: http://dl.dropbox.com/u/14650083/PF/bge_bits_1-week.png http://dl.dropbox.com/u/14650083/PF/bge_packets_1-week.png http://dl.dropbox.com/u/14650083/PF/bge_stats_1-week.png http://dl.dropbox.com/u/14650083/PF/load-week.png http://dl.dropbox.com/u/14650083/PF/mbuf_errors-week.png http://dl.dropbox.com/u/14650083/PF/mbuf_usage-week.png http://dl.dropbox.com/u/14650083/PF/pf_inserts-week.png http://dl.dropbox.com/u/14650083/PF/pf_matches-week.png http://dl.dropbox.com/u/14650083/PF/pf_removals-week.png http://dl.dropbox.com/u/14650083/PF/pf_searches-week.png http://dl.dropbox.com/u/14650083/PF/pf_src_limit-week.png http://dl.dropbox.com/u/14650083/PF/pf_states-week.png http://dl.dropbox.com/u/14650083/PF/pf_synproxy-week.png I'll wait for the next time the symptom occurs to switch to a stateless configuration. -- Good, fast & cheap. Pick any two. From owner-freebsd-stable@FreeBSD.ORG Tue May 3 10:18:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B172B1065677; Tue, 3 May 2011 10:18:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5322C8FC0A; Tue, 3 May 2011 10:18:01 +0000 (UTC) Received: by qyk27 with SMTP id 27so3982165qyk.13 for ; Tue, 03 May 2011 03:18:01 -0700 (PDT) Received: by 10.229.29.20 with SMTP id o20mr7031652qcc.181.1304417881210; Tue, 03 May 2011 03:18:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Tue, 3 May 2011 03:17:21 -0700 (PDT) In-Reply-To: References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> From: Vlad Galu Date: Tue, 3 May 2011 12:17:21 +0200 Message-ID: To: Vincent Hoffman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Daniel Hartmeier , freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:18:02 -0000 On Tue, May 3, 2011 at 12:12 PM, Vlad Galu wrote: > > > On Tue, May 3, 2011 at 11:31 AM, Vincent Hoffman wrote: > >> On 03/05/2011 10:16, Jeremy Chadwick wrote: >> >> >> > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt >> > usage, etc. otherwise I'd be graphing that. The more monitoring the >> > better; at least then I could say "wow, interrupts really did shoot >> > through the roof -- the box went crazy!" and RMA the thing. :-) >> > >> you could use net-mgmt/bsnmp-regex although I dont know what the >> overhead for that is like. >> > > I use munin for graphing, as it allows easy scripting without using SNMP. > > My case is a bit different from Jeremy's. Every once in a while there is a > sudden traffic spike which impacts pf performance as well. However, the > graphed figures are nowhere near what I'd consider alarming levels (this box > has withstood more in the past). I was able to coincidentally log in after > such a spike and noticed the pfpurge thread eating up about 30% of the CPU > while using the normal optimization policy. In my case, it could be related > to another issue I'm seeing on this box - mbuma allocation failures. Here > are my graphs: > > http://dl.dropbox.com/u/14650083/PF/bge_bits_1-week.png > http://dl.dropbox.com/u/14650083/PF/bge_packets_1-week.png > http://dl.dropbox.com/u/14650083/PF/bge_stats_1-week.png > http://dl.dropbox.com/u/14650083/PF/load-week.png > http://dl.dropbox.com/u/14650083/PF/mbuf_errors-week.png > http://dl.dropbox.com/u/14650083/PF/mbuf_usage-week.png > http://dl.dropbox.com/u/14650083/PF/pf_inserts-week.png > http://dl.dropbox.com/u/14650083/PF/pf_matches-week.png > http://dl.dropbox.com/u/14650083/PF/pf_removals-week.png > http://dl.dropbox.com/u/14650083/PF/pf_searches-week.png > http://dl.dropbox.com/u/14650083/PF/pf_src_limit-week.png > http://dl.dropbox.com/u/14650083/PF/pf_states-week.png > http://dl.dropbox.com/u/14650083/PF/pf_synproxy-week.png > > I'll wait for the next time the symptom occurs to switch to a stateless > configuration. > > I forgot to mention this is a UP box using TSC for timekeeping and running ntpd. -- /boot/loader.conf -- hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" debug.acpi.disabled="timer" -- /boot/loader.conf -- -- sysctl output -- kern.timecounter.choice: TSC(800) i8254(0) dummy(-1000000) kern.timecounter.hardware: TSC -- sysctl output -- -- Good, fast & cheap. Pick any two. From owner-freebsd-stable@FreeBSD.ORG Tue May 3 10:39:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 881D21065673; Tue, 3 May 2011 10:39:53 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id E10438FC14; Tue, 3 May 2011 10:39:52 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p43AdqG0018522 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 12:39:52 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p43AdohQ030494; Tue, 3 May 2011 12:39:50 +0200 (MEST) Date: Tue, 3 May 2011 12:39:50 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503103950.GD9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:39:53 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > did "/etc/rc.d/pf start", which also worked. However, what I saw next > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > and "pfctl -s info" disagreed on the state count: This can be explained. Note that "/etc/rc.d/pf start" does first flush all states by calling pfctl -F all. This calls pf_unlink_state() for every state in the kernel, which marks each state with PFTM_UNLINKED, but doesn't free it yet. Such states do not show up in pfctl -s state output, but are still counted in pfctl -s info output. Normally, they are freed the next time the pfpurge thread runs (once per second). It looks like the pfpurge thread was either a) sleeping indefinitely, not returning once a second from tsleep(pf_purge_thread, PWAIT, "pftm", 1 * hz); or b) constantly failing to acquire a lock with if (!sx_try_upgrade(&pf_consistency_lock)) return (0); Maybe a) is possible when CLOCK_MONOTONIC is decreasing? And the "POKED TIMER" messages you get from BIND, too? Kind regards, Daniel From owner-freebsd-stable@FreeBSD.ORG Tue May 3 12:20:55 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62785106566C for ; Tue, 3 May 2011 12:20:55 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id 44EBC8FC0A for ; Tue, 3 May 2011 12:20:54 +0000 (UTC) Received: from omta16.emeryville.ca.mail.comcast.net ([76.96.30.72]) by qmta11.emeryville.ca.mail.comcast.net with comcast id f06f1g0011ZMdJ4AB0LuBF; Tue, 03 May 2011 12:20:54 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta16.emeryville.ca.mail.comcast.net with comcast id f0Ls1g00E1t3BNj8c0LspK; Tue, 03 May 2011 12:20:53 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1EA57102C31; Tue, 3 May 2011 05:20:52 -0700 (PDT) Date: Tue, 3 May 2011 05:20:52 -0700 From: Jeremy Chadwick To: Olaf Seibert Message-ID: <20110503122052.GA13811@icarus.home.lan> References: <20110502143230.GW6733@twoquid.cs.ru.nl> <20110503092113.GA39704@icarus.home.lan> <20110503100854.GY6733@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503100854.GY6733@twoquid.cs.ru.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:20:55 -0000 On Tue, May 03, 2011 at 12:08:54PM +0200, Olaf Seibert wrote: > On Tue 03 May 2011 at 02:21:13 -0700, Jeremy Chadwick wrote: > > There are two things you might try fiddling with. These are sysctls so > > you can try them on the fly: > > > > hw.acpi.disable_on_reboot > > hw.acpi.handle_reboot > > Thanks. For now I've set the second to 1 and we'll see if that affects > matters. > > > Check out the thread Peter Jeremy provided. This is a near-sure > > indicator of ZFS ARC exhaustion, and you seem to know of that. What's > > very interesting to me is this part of your mail: > ... > > > > Is this box running i386 or amd64? If amd64, I can't explain why your > > It's amd64. I double-checked just one, you never know what stupid > mistakes one might make :-) > > > /boot/loader.conf settings aren't taking -- they should be for sure. > > Maybe provide us a full dmesg and XXX out things you consider > > sensitive. If i386, I'm not too surprised that some automatic defaults > > get chosen instead of what you ask. > > Based on one of your mails where setting vm.kmem_size to twice the real > RAM size had adverse effects, I've taken the setting out to see if that > improves matters. I'll have to wait until the next crash (or opportunity > to reboot without too much disturbance) to see the effect. The ill-effects are a result of an underlying change that I had forgotten about but others remembered -- vm.kmem_size_scale used to be set to something like "2" by default, but it was changed to "1" prior to 8.2-RELEASE. So basically here's the current situation and how all of our 8.2-STABLE machines are tuned for ARC: we only set one single tunable for ARC "management": vfs.zfs.arc_max. We don't touch vm.kmem_size. Here's what we have literally in our /boot/loader.conf: # Limit ZFS ARC maximum. # NOTE #1: In 8.2-RELEASE and onward, vm.kmem_size_scale defaults to 1, # which means vm.kmem_size should match the amount of RAM installed # in the system. If using an earlier FreeBSD release, be sure to set # vm.kmem_size manually to the amount of RAM you have. # NOTE #2: Do not set vm.kmem_size to 2x that of physical RAM, otherwise # vfs.zfs.arc_max effectively becomes halved. # http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010875.html vfs.zfs.arc_max="6144M" The value specified here (6144MBytes) is for a machine with 8GB of RAM. Keep in mind that there is evidence that kmap/kmem exhaustion can still happen even if you tune the ARC like this. Apparently memory fragmentation plays a role, and there's some overhead as well, so calculating a 100% stable value is a little difficult. I can point you to that (very recent, as in last month) thread if you'd like. To be on the safe side, pick something that's small at first, then work your way up. You'll need probably 1+ weeks of heavy ZFS I/O between tests (e.g. don't change the tunable, reboot, then 4 hours later declare the new (larger) value as stable). So for example on an 8GB RAM machine, I might recommend starting with vfs.zfs.arc_max="4096M" and let that run for a while. If you find your "Wired" value in top(1) remains fairly constant after a week or so of heavy I/O, consider bumping up the value a bit more (say 4608M). Sorry to make this long-winded; bad habit of mine that I've never managed to break. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 12:30:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B53FE1065670 for ; Tue, 3 May 2011 12:30:19 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by mx1.freebsd.org (Postfix) with ESMTP id 3EE658FC1F for ; Tue, 3 May 2011 12:30:18 +0000 (UTC) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id p43CUE37025288; Tue, 3 May 2011 14:30:14 +0200 (MEST) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id 4AF9E2E05D; Tue, 3 May 2011 14:30:15 +0200 (CEST) Date: Tue, 3 May 2011 14:30:15 +0200 From: Olaf Seibert To: Jeremy Chadwick Message-ID: <20110503123015.GZ6733@twoquid.cs.ru.nl> References: <20110502143230.GW6733@twoquid.cs.ru.nl> <20110503092113.GA39704@icarus.home.lan> <20110503100854.GY6733@twoquid.cs.ru.nl> <20110503122052.GA13811@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503122052.GA13811@icarus.home.lan> User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Cc: freebsd-stable@freebsd.org, Olaf Seibert Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:30:19 -0000 On Tue 03 May 2011 at 05:20:52 -0700, Jeremy Chadwick wrote: > To be on the safe side, pick something that's small at first, then work > your way up. You'll need probably 1+ weeks of heavy ZFS I/O between > tests (e.g. don't change the tunable, reboot, then 4 hours later declare > the new (larger) value as stable). Ah, that's important: so far it seemed to me that a *too small* value (for all various tunables) would cause problems, but now you're saying that *too large* is the problem (at least for vfs.zfs.arc_max)! This machine has mixed loads; from time to time somebody starts a big job with lots of I/O, and in between it is much more modestly loaded. > So for example on an 8GB RAM machine, I might recommend starting with > vfs.zfs.arc_max="4096M" and let that run for a while. If you find your > "Wired" value in top(1) remains fairly constant after a week or so of > heavy I/O, consider bumping up the value a bit more (say 4608M). I'll do just that. > Sorry to make this long-winded; bad habit of mine that I've never > managed to break. Oh no problem, it turns out to be eye-opening! > | Jeremy Chadwick jdc@parodius.com | -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe)); From owner-freebsd-stable@FreeBSD.ORG Tue May 3 12:42:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46B071065675 for ; Tue, 3 May 2011 12:42:25 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id 272158FC12 for ; Tue, 3 May 2011 12:42:24 +0000 (UTC) Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90]) by qmta11.emeryville.ca.mail.comcast.net with comcast id f05g1g0051wfjNsAB0iQCY; Tue, 03 May 2011 12:42:24 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta23.emeryville.ca.mail.comcast.net with comcast id f0iM1g00H1t3BNj8j0iMp6; Tue, 03 May 2011 12:42:24 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4B220102C31; Tue, 3 May 2011 05:42:21 -0700 (PDT) Date: Tue, 3 May 2011 05:42:21 -0700 From: Jeremy Chadwick To: Vincent Hoffman Message-ID: <20110503124221.GB13811@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4DBFCB8D.10105@unsane.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Daniel Hartmeier , freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:42:25 -0000 On Tue, May 03, 2011 at 10:31:57AM +0100, Vincent Hoffman wrote: > On 03/05/2011 10:16, Jeremy Chadwick wrote: > > > > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > > usage, etc. otherwise I'd be graphing that. The more monitoring the > > better; at least then I could say "wow, interrupts really did shoot > > through the roof -- the box went crazy!" and RMA the thing. :-) > > > you could use net-mgmt/bsnmp-regex although I dont know what the > overhead for that is like. Thanks for the tip. I've investigated that plugin before, and its implementation model seems like a very hackish way to accomplish something that should ultimately be done inside of bsnmpd(8) itself via native C. It's good for parsing a single log file via tail -F (not "tail -f" like the man page indicates), but it doesn't scale well. bsnmpd(8) just needs to be enhanced and fixed, and I know there's efforts underway by syrinx@ to do exactly that. I have chatted with her about some existing problems with bsnmpd(8) and its SNMP parser, and have chatted with philip@ about a pf-related bug with bsnmp(8) (but I can't remember what the details of that one is; I have a file with the info around here somewhere...) There was also a recent commit to net-mgmt/net-snmp that pertains to *properly* monitoring swap, which makes me wonder if net-mgmt/bsnmp-ucd (which a lot of people, myself included, rely on) also does the wrong thing. http://www.freebsd.org/cgi/query-pr.cgi?pr=153179 http://www.freebsd.org/cgi/cvsweb.cgi/ports/net-mgmt/net-snmp/files/patch-memory_freebsd.c Things like this make me question my graphs and my monitoring data pretty much every time I look at them. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 12:53:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3DED1065670 for ; Tue, 3 May 2011 12:53:19 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 856078FC1A for ; Tue, 3 May 2011 12:53:19 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta01.emeryville.ca.mail.comcast.net with comcast id ezma1g0010QkzPwA10tKdG; Tue, 03 May 2011 12:53:19 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta02.emeryville.ca.mail.comcast.net with comcast id f0tG1g00V1t3BNj8N0tGUQ; Tue, 03 May 2011 12:53:18 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7A06F102C31; Tue, 3 May 2011 05:53:16 -0700 (PDT) Date: Tue, 3 May 2011 05:53:16 -0700 From: Jeremy Chadwick To: Olaf Seibert Message-ID: <20110503125316.GA17085@icarus.home.lan> References: <20110502143230.GW6733@twoquid.cs.ru.nl> <20110503092113.GA39704@icarus.home.lan> <20110503100854.GY6733@twoquid.cs.ru.nl> <20110503122052.GA13811@icarus.home.lan> <20110503123015.GZ6733@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503123015.GZ6733@twoquid.cs.ru.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Automatic reboot doesn't reboot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:53:19 -0000 On Tue, May 03, 2011 at 02:30:15PM +0200, Olaf Seibert wrote: > On Tue 03 May 2011 at 05:20:52 -0700, Jeremy Chadwick wrote: > > To be on the safe side, pick something that's small at first, then work > > your way up. You'll need probably 1+ weeks of heavy ZFS I/O between > > tests (e.g. don't change the tunable, reboot, then 4 hours later declare > > the new (larger) value as stable). > > Ah, that's important: so far it seemed to me that a *too small* value > (for all various tunables) would cause problems, but now you're saying > that *too large* is the problem (at least for vfs.zfs.arc_max)! Too small = not-so-great performance (less data in the ARC means more reads from the disks. Disks are slower than RAM :-) ). Too large = increased risk of kmem exhaustion panic. > This machine has mixed loads; from time to time somebody starts a big > job with lots of I/O, and in between it is much more modestly loaded. I would recommend starting small (maybe 1/3rd of your physical RAM?) and increase from there. You can try the opposite technique too -- start large (e.g. 3/4ths of RAM) and wait for a panic. I'm of the opinion that I'd rather have a stable system with less memory used for ARC than a system which could panic and have more memory for ARC. Sadly there's no 100% reliable way to calculate what's "ideal". For example I might use a smaller value than 6144M on a machine where mysqld is tuned to utilise lots of RAM. There's a balancing act that goes on that takes some time to figure out. For example, on our FreeBSD ZFS-backed NFS filer on our network, I ran with a 3/4th amount for quite some time (we're talking 4-5 months). Then suddenly one day I noticed the client machines were complaining about NFS timeouts, etc... Got on the filer, lo and behold kmem exhaustion. I decreased arc_max by about 1024M and it's been fine since. There's a lot of evolution that's occurred in the FreeBSD ZFS kernel code over the years too. Originally arc_max was a "high-water mark" of some sort, but code was changed to make it a hard limit as much as it could be. Then some edge cases were found where it could still exceed the maximum, so those were fixed, etc... Tracking all the changes is very difficult (I became very frustrated/irate at having to do so, wishing that there was more of a "state of ZFS" announcement sent out every so often so users/admins would know what's changed and adjust things appropriately), requiring an admin to follow commits. That's just the nature of the beast. > > So for example on an 8GB RAM machine, I might recommend starting with > > vfs.zfs.arc_max="4096M" and let that run for a while. If you find your > > "Wired" value in top(1) remains fairly constant after a week or so of > > heavy I/O, consider bumping up the value a bit more (say 4608M). > > I'll do just that. Let us know how things turn out. Follow-ups that indicate things are working are just as important as initial mails stating things aren't, especially if you're someone searching the Web to try and find an answer to what this kmem thing is all about..... :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue May 3 13:52:28 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6469106564A; Tue, 3 May 2011 13:52:28 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 621588FC13; Tue, 3 May 2011 13:52:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p43DqQHf080494; Tue, 3 May 2011 17:52:26 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Tue, 3 May 2011 17:52:26 +0400 (MSD) From: Dmitry Morozovsky To: "Kenneth D. Merry" In-Reply-To: <20110503034737.GA52416@nargothrond.kdm.org> Message-ID: References: <20110430211927.GA67374@nargothrond.kdm.org> <20110503034737.GA52416@nargothrond.kdm.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (woozle.rinet.ru [0.0.0.0]); Tue, 03 May 2011 17:52:26 +0400 (MSD) Cc: freebsd-stable@FreeBSD.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 13:52:29 -0000 On Mon, 2 May 2011, Kenneth D. Merry wrote: KDM> It looks like you have a SAS2008, with the 4.0 firmware. I think it would KDM> be worthwhile to upgrade to the 9.0 firmware. I know for sure there are KDM> issues with the 2.0 firmware, and I know the 9.0 firmware works fairly KDM> well. I don't know whether the 4.0 firmware has any severe issues, but it KDM> would be good to eliminate firmware bugs before we chase driver issues. [snip] KDM> Well, I think the first thing to do is upgrade the firmware and see if that KDM> fixes it. Well, I tried, and unfortunately I can not say that I'm happy after the upgrade. :( Particularly, adapter now takes *VERY* long time (>10 minutes) to initialize, and report as "ERROR" in BIOS utility (while seeing all 24 disks; however, it reports 8 x36 expanders instead of one). I can't boot the system off this array yet; will experiment further :( -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 14:11:17 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26755106566C for ; Tue, 3 May 2011 14:11:17 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (unknown [IPv6:2001:44b8:7c07:5581:266:e1ff:fe0c:8f16]) by mx1.freebsd.org (Postfix) with ESMTP id BB0178FC19 for ; Tue, 3 May 2011 14:11:14 +0000 (UTC) Received: from [10.0.2.77] (ppp208-198.lns1.adl2.internode.on.net [203.122.208.198] (may be forged)) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id p43EBACJ059744 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Tue, 3 May 2011 23:41:10 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: "Daniel O'Connor" In-Reply-To: <20110429010829.GA36744@icarus.home.lan> Date: Tue, 3 May 2011 23:41:09 +0930 Content-Transfer-Encoding: quoted-printable Message-Id: <6B48340C-9A34-41BC-A332-4D0D703E660F@gsoft.com.au> References: <537A8F4F-A302-40F9-92DF-403388D99B4B@gsoft.com.au> <20110428195601.GA31807@icarus.home.lan> <20110429010829.GA36744@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1084) X-Spam-Score: -0.272 () BAYES_00,RDNS_NONE X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 Cc: freebsd-stable List Subject: Re: ZFS vs OSX Time Machine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 14:11:17 -0000 On 29/04/2011, at 10:38, Jeremy Chadwick wrote: >> The OSX box is connected via an Airport Express (11n). >=20 > Can you connect something to it via Ethernet and attempt an FTP = transfer > (both PUT (store on server) and GET (retrieve from server)) from a > client on the wired network? Make sure whatever you're PUT'ing and > GET'ing are using the ZFS filesystem. Don't forget "binary" mode too. I tried dd'ing /dev/zero over SMB and got 40MB/sec (although I'm not = using AIO yet..) FTP'ing a 300 MB file averages 60-70MB/sec (the speed of my laptop HD) ttcp between the hosts hits wire speed (100MB/sec) >> OK. I don't think TM can use CIFS, I will try ISCSI as someone else = suggested, perhaps it will help. >=20 > Be aware there are all sorts of caveats/complexities with iSCSI on > FreeBSD. There are past threads on -stable and -fs talking about them > in great detail. I personally wouldn't go this route. >=20 > Why can't OS X use CIFS? It has the ability to mount a SMB = filesystem, > right? Is there some reason you can't mount that, then tell TM to = write > its backups to /mountedcifs? It looks like I had a dodgy disk which was being tickled by the time = machine backup (eg dodgy sector where the backup was located) so I have = been chasing a ghost :) However, thanks to everyone for your helpful suggestions! I still haven't tried iSCSI, given I can't do a bare metal restore from = it it doesn't seem worth it (also I don't have the time..) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Tue May 3 14:56:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B85CA106564A; Tue, 3 May 2011 14:56:53 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 435838FC18; Tue, 3 May 2011 14:56:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p43EupPj081538; Tue, 3 May 2011 18:56:51 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Tue, 3 May 2011 18:56:51 +0400 (MSD) From: Dmitry Morozovsky To: "Kenneth D. Merry" In-Reply-To: Message-ID: References: <20110430211927.GA67374@nargothrond.kdm.org> <20110503034737.GA52416@nargothrond.kdm.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (woozle.rinet.ru [0.0.0.0]); Tue, 03 May 2011 18:56:51 +0400 (MSD) Cc: freebsd-stable@freebsd.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 14:56:53 -0000 On Tue, 3 May 2011, Dmitry Morozovsky wrote: DM> KDM> It looks like you have a SAS2008, with the 4.0 firmware. I think it would DM> KDM> be worthwhile to upgrade to the 9.0 firmware. I know for sure there are DM> KDM> issues with the 2.0 firmware, and I know the 9.0 firmware works fairly DM> KDM> well. I don't know whether the 4.0 firmware has any severe issues, but it DM> KDM> would be good to eliminate firmware bugs before we chase driver issues. DM> DM> [snip] DM> DM> KDM> Well, I think the first thing to do is upgrade the firmware and see if that DM> KDM> fixes it. DM> DM> Well, I tried, and unfortunately I can not say that I'm happy after the DM> upgrade. :( DM> DM> Particularly, adapter now takes *VERY* long time (>10 minutes) to initialize, DM> and report as "ERROR" in BIOS utility (while seeing all 24 disks; however, it DM> reports 8 x36 expanders instead of one). DM> DM> I can't boot the system off this array yet; will experiment further :( booted from USB stick, I have constantly repeating (ses3:mps0:0:25:0): lost device (ses3:mps0:0:25:0): removing device entry ses3 at mps0 bus 0 scbus0 target 25 lun 0 ses3: Fixed Enclosure Services SCSI-5 device ses3: 600.000MB/s transfers ses3: Command Queueing enabled ses3: SCSI-3 SES Device for different sesN, which are detected many times: at scbus0 target 0 lun 0 (da0,pass0) at scbus0 target 1 lun 0 (da1,pass1) at scbus0 target 2 lun 0 (da2,pass2) at scbus0 target 3 lun 0 (pass11,da4) at scbus0 target 4 lun 0 (pass12,da5) at scbus0 target 5 lun 0 (pass9,da3) at scbus0 target 24 lun 0 (pass5,ses3) at scbus0 target 25 lun 0 (pass19,ses5) at scbus0 target 26 lun 0 (pass10,ses4) at scbus0 target 27 lun 0 (pass14,ses7) at scbus0 target 33 lun 0 (pass13,ses6) at scbus0 target 39 lun 0 (pass3,ses0) at scbus0 target 45 lun 0 (pass4,ses1) at scbus0 target 51 lun 0 (pass8,ses2) at scbus0 target 55 lun 0 (pass15,da7) at scbus0 target 63 lun 0 (pass16,da8) at scbus0 target 71 lun 0 (pass17,da9) at scbus0 target 79 lun 0 (pass18,da10) at scbus0 target 87 lun 0 (pass6,da11) at scbus0 target 95 lun 0 (pass20,da12) at scbus0 target 103 lun 0 (pass21,da13) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 16:15:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98AA01065707 for ; Tue, 3 May 2011 16:15:08 +0000 (UTC) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Received: from pis.elm.toba-cmt.ac.jp (pis.elm.toba-cmt.ac.jp [202.26.248.196]) by mx1.freebsd.org (Postfix) with ESMTP id 130EF8FC18 for ; Tue, 3 May 2011 16:15:06 +0000 (UTC) Received: from kiri.pis.pis.elm.toba-cmt.ac.jp (localhost [127.0.0.1]) by pis.elm.toba-cmt.ac.jp (8.14.3/8.14.2) with ESMTP id p43Fh92T041708 for ; Wed, 4 May 2011 00:43:09 +0900 (JST) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Message-Id: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> Date: Wed, 04 May 2011 00:43:09 +0900 From: KIRIYAMA Kazuhiko To: freebsd-stable@freebsd.org In-Reply-To: References: User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 MULE XEmacs/21.4 (patch 21) (Educational Television) (i386--freebsd) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Mailman-Approved-At: Tue, 03 May 2011 16:27:15 +0000 Subject: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 16:15:08 -0000 Hi all, Recently I upgraded to 8.2-STABLE and reconfigured natd + jailed box, but all packets could not over nat box. I've researched and found /etc/rc.firewall does not recieve argument of firewall_type. So ipfw does not divert and natd could not be performed. The reason is /etc/rc.d/ipfw incorrect. I think an patch below should be applyed to /etc/rc.d/ipfw. Is there any problem to do this? --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 +++ /etc/rc.d/ipfw 2011-05-03 22:08:14.000000000 +0900 @@ -35,15 +35,11 @@ ipfw_start() { - local _firewall_type - - _firewall_type=$1 - # set the firewall rules script if none was specified [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall if [ -r "${firewall_script}" ]; then - /bin/sh "${firewall_script}" "${_firewall_type}" + /bin/sh "${firewall_script}" "${firewall_type}" echo 'Firewall rules loaded.' elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then echo 'Warning: kernel has firewall functionality, but' \ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 17:28:29 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 881C91065678; Tue, 3 May 2011 17:28:29 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 190BB8FC12; Tue, 3 May 2011 17:28:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p43HSRvf083134; Tue, 3 May 2011 21:28:27 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Tue, 3 May 2011 21:28:27 +0400 (MSD) From: Dmitry Morozovsky To: "Kenneth D. Merry" In-Reply-To: Message-ID: References: <20110430211927.GA67374@nargothrond.kdm.org> <20110503034737.GA52416@nargothrond.kdm.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (woozle.rinet.ru [0.0.0.0]); Tue, 03 May 2011 21:28:27 +0400 (MSD) Cc: freebsd-stable@freebsd.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 17:28:29 -0000 On Tue, 3 May 2011, Dmitry Morozovsky wrote: DM> DM> Well, I tried, and unfortunately I can not say that I'm happy after the DM> DM> upgrade. :( DM> DM> DM> DM> Particularly, adapter now takes *VERY* long time (>10 minutes) to initialize, DM> DM> and report as "ERROR" in BIOS utility (while seeing all 24 disks; however, it DM> DM> reports 8 x36 expanders instead of one). DM> DM> DM> DM> I can't boot the system off this array yet; will experiment further :( DM> DM> booted from USB stick, I have constantly repeating DM> DM> (ses3:mps0:0:25:0): lost device DM> (ses3:mps0:0:25:0): removing device entry DM> ses3 at mps0 bus 0 scbus0 target 25 lun 0 DM> ses3: Fixed Enclosure Services SCSI-5 device DM> ses3: 600.000MB/s transfers DM> ses3: Command Queueing enabled DM> ses3: SCSI-3 SES Device DM> DM> for different sesN, which are detected many times: DM> DM> at scbus0 target 0 lun 0 (da0,pass0) DM> at scbus0 target 1 lun 0 (da1,pass1) DM> at scbus0 target 2 lun 0 (da2,pass2) DM> at scbus0 target 3 lun 0 (pass11,da4) DM> at scbus0 target 4 lun 0 (pass12,da5) DM> at scbus0 target 5 lun 0 (pass9,da3) DM> at scbus0 target 24 lun 0 (pass5,ses3) DM> at scbus0 target 25 lun 0 (pass19,ses5) DM> at scbus0 target 26 lun 0 (pass10,ses4) DM> at scbus0 target 27 lun 0 (pass14,ses7) DM> at scbus0 target 33 lun 0 (pass13,ses6) DM> at scbus0 target 39 lun 0 (pass3,ses0) DM> at scbus0 target 45 lun 0 (pass4,ses1) DM> at scbus0 target 51 lun 0 (pass8,ses2) DM> at scbus0 target 55 lun 0 (pass15,da7) DM> at scbus0 target 63 lun 0 (pass16,da8) DM> at scbus0 target 71 lun 0 (pass17,da9) DM> at scbus0 target 79 lun 0 (pass18,da10) DM> at scbus0 target 87 lun 0 (pass6,da11) DM> at scbus0 target 95 lun 0 (pass20,da12) DM> at scbus0 target 103 lun 0 (pass21,da13) Well, using http://kb.lsi.com/KnowledgebaseArticle16414.aspx I downgraded to version 8-fixed, and at least topology errors disappear. Just booted successfully (errm, it was a few nervous hours, to be honest :) Now I have in verbose kernel messages mps0: port 0xc000-0xc0ff mem 0xfb43c000-0xfb43ffff,0xfb440000-0xfb47ffff irq 16 at device 0.0 on pci2 mps0: Reserved 0x4000 bytes for rid 0x14 type 3 at 0xfb43c000 mps0: Firmware: 08.00.00.00 mps0: IOCCapabilities: 185c mps0: attempting to allocate 1 MSI-X vectors (15 supported) msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 mps0: using IRQ 256 for MSI-X mps0: [MPSAFE] mps0: [ITHREAD] Will see whether it helps. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 17:35:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27677106564A for ; Tue, 3 May 2011 17:35:10 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id D98378FC20 for ; Tue, 3 May 2011 17:35:09 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id p43HZ85M075788; Tue, 3 May 2011 11:35:08 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id p43HZ868075787; Tue, 3 May 2011 11:35:08 -0600 (MDT) (envelope-from ken) Date: Tue, 3 May 2011 11:35:08 -0600 From: "Kenneth D. Merry" To: Dmitry Morozovsky Message-ID: <20110503173508.GA75740@nargothrond.kdm.org> References: <20110430211927.GA67374@nargothrond.kdm.org> <20110503034737.GA52416@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: freebsd-stable@freebsd.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 17:35:10 -0000 On Tue, May 03, 2011 at 21:28:27 +0400, Dmitry Morozovsky wrote: > > On Tue, 3 May 2011, Dmitry Morozovsky wrote: > > DM> DM> Well, I tried, and unfortunately I can not say that I'm happy after the > DM> DM> upgrade. :( > DM> DM> > DM> DM> Particularly, adapter now takes *VERY* long time (>10 minutes) to initialize, > DM> DM> and report as "ERROR" in BIOS utility (while seeing all 24 disks; however, it > DM> DM> reports 8 x36 expanders instead of one). > DM> DM> > DM> DM> I can't boot the system off this array yet; will experiment further :( > DM> > DM> booted from USB stick, I have constantly repeating > DM> > DM> (ses3:mps0:0:25:0): lost device > DM> (ses3:mps0:0:25:0): removing device entry > DM> ses3 at mps0 bus 0 scbus0 target 25 lun 0 > DM> ses3: Fixed Enclosure Services SCSI-5 device > DM> ses3: 600.000MB/s transfers > DM> ses3: Command Queueing enabled > DM> ses3: SCSI-3 SES Device > DM> > DM> for different sesN, which are detected many times: > DM> > DM> at scbus0 target 0 lun 0 (da0,pass0) > DM> at scbus0 target 1 lun 0 (da1,pass1) > DM> at scbus0 target 2 lun 0 (da2,pass2) > DM> at scbus0 target 3 lun 0 (pass11,da4) > DM> at scbus0 target 4 lun 0 (pass12,da5) > DM> at scbus0 target 5 lun 0 (pass9,da3) > DM> at scbus0 target 24 lun 0 (pass5,ses3) > DM> at scbus0 target 25 lun 0 (pass19,ses5) > DM> at scbus0 target 26 lun 0 (pass10,ses4) > DM> at scbus0 target 27 lun 0 (pass14,ses7) > DM> at scbus0 target 33 lun 0 (pass13,ses6) > DM> at scbus0 target 39 lun 0 (pass3,ses0) > DM> at scbus0 target 45 lun 0 (pass4,ses1) > DM> at scbus0 target 51 lun 0 (pass8,ses2) > DM> at scbus0 target 55 lun 0 (pass15,da7) > DM> at scbus0 target 63 lun 0 (pass16,da8) > DM> at scbus0 target 71 lun 0 (pass17,da9) > DM> at scbus0 target 79 lun 0 (pass18,da10) > DM> at scbus0 target 87 lun 0 (pass6,da11) > DM> at scbus0 target 95 lun 0 (pass20,da12) > DM> at scbus0 target 103 lun 0 (pass21,da13) > > Well, using > http://kb.lsi.com/KnowledgebaseArticle16414.aspx > I downgraded to version 8-fixed, and at least topology errors disappear. > > Just booted successfully (errm, it was a few nervous hours, to be honest :) > > Now I have in verbose kernel messages > > mps0: port 0xc000-0xc0ff mem > 0xfb43c000-0xfb43ffff,0xfb440000-0xfb47ffff irq 16 at device 0.0 on pci2 > mps0: Reserved 0x4000 bytes for rid 0x14 type 3 at 0xfb43c000 > mps0: Firmware: 08.00.00.00 > mps0: IOCCapabilities: 185c > mps0: attempting to allocate 1 MSI-X vectors (15 supported) > msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 > mps0: using IRQ 256 for MSI-X > mps0: [MPSAFE] > mps0: [ITHREAD] Sorry you ran into all of those problems! Needless to say I haven't seen that with the 9.0 firmware in my environment, but then again I've got a different setup. > Will see whether it helps. Yes. I know the 8.0 firmware also works well. The only issue I ran into there was the topology issues that I'm guessing they fixed in that build. If the firmware doesn't fix it, we'll go down the path of trying to see why the IOC fault is happening. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-stable@FreeBSD.ORG Tue May 3 17:45:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7A33106564A; Tue, 3 May 2011 17:45:54 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 3310A8FC0C; Tue, 3 May 2011 17:45:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p43HjqrH083816; Tue, 3 May 2011 21:45:52 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Tue, 3 May 2011 21:45:52 +0400 (MSD) From: Dmitry Morozovsky To: "Kenneth D. Merry" In-Reply-To: <20110503173508.GA75740@nargothrond.kdm.org> Message-ID: References: <20110430211927.GA67374@nargothrond.kdm.org> <20110503034737.GA52416@nargothrond.kdm.org> <20110503173508.GA75740@nargothrond.kdm.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (woozle.rinet.ru [0.0.0.0]); Tue, 03 May 2011 21:45:52 +0400 (MSD) Cc: freebsd-stable@freebsd.org Subject: Re: mps driver instability under stable/8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 17:45:54 -0000 On Tue, 3 May 2011, Kenneth D. Merry wrote: KDM> Sorry you ran into all of those problems! Needless to say I haven't seen KDM> that with the 9.0 firmware in my environment, but then again I've got a KDM> different setup. I just postd comment on LSI kb forum, will see how they'd comment it. KDM> If the firmware doesn't fix it, we'll go down the path of trying to see why KDM> the IOC fault is happening. I'm staying tuned, while conserver is writing logs ;-) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 17:47:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEADD106564A for ; Tue, 3 May 2011 17:47:05 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 5A2C88FC22 for ; Tue, 3 May 2011 17:47:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id p43Hl2BA022312; Wed, 4 May 2011 03:47:03 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Wed, 4 May 2011 03:47:02 +1000 (EST) From: Ian Smith To: KIRIYAMA Kazuhiko In-Reply-To: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> Message-ID: <20110504030404.O85801@sola.nimnet.asn.au> References: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-stable@freebsd.org Subject: Re: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 17:47:05 -0000 On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > Hi all, > Recently I upgraded to 8.2-STABLE and reconfigured natd + jailed box, but > all packets could not over nat box. I've researched and found > /etc/rc.firewall does not recieve argument of firewall_type. So ipfw does > not divert and natd could not be performed. The reason is /etc/rc.d/ipfw > incorrect. I think an patch below should be applyed to /etc/rc.d/ipfw. Is > there any problem to do this? Yes. Assuming using the default firewall_script="/etc/rc.firewall", then as it says early in /etc/rc.firewall, you just needed to: # Define the firewall type in /etc/rc.conf. Valid values are: [..] Sure, /etc/rc.firewall can set firewall_type to a parameter if you pass it one, but otherwise uses whatever $firewall_type is set to when you start ipfw. I guess the code below allows you to use syntax like: # /etc/rc.d/ipfw start client to override the $firewall_type set in /etc/rc.conf, but it's not the common usage, nor is it how ipfw is started normally by rc. So just set firewall_type in rc.conf and you should be fine .. unless you meant that you're trying to run ipfw & natd INSIDE a jail? cheers, Ian > --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 > +++ /etc/rc.d/ipfw 2011-05-03 22:08:14.000000000 +0900 > @@ -35,15 +35,11 @@ > > ipfw_start() > { > - local _firewall_type > - > - _firewall_type=$1 > - > # set the firewall rules script if none was specified > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall > > if [ -r "${firewall_script}" ]; then > - /bin/sh "${firewall_script}" "${_firewall_type}" > + /bin/sh "${firewall_script}" "${firewall_type}" > echo 'Firewall rules loaded.' > elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then > echo 'Warning: kernel has firewall functionality, but' \ From owner-freebsd-stable@FreeBSD.ORG Tue May 3 18:09:38 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0693E106567A for ; Tue, 3 May 2011 18:09:38 +0000 (UTC) (envelope-from jafa82@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id B3EF08FC13 for ; Tue, 3 May 2011 18:09:36 +0000 (UTC) Received: by gyg13 with SMTP id 13so164194gyg.13 for ; Tue, 03 May 2011 11:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:subject:message-id:in-reply-to :references:x-mailer:mime-version:content-type :content-transfer-encoding; bh=nXAWmP0IiPVxdvejg+LxTwB4AQB8wpkU3EF2FC4vBwc=; b=i/YHEk961DKsbTL3IL1OaORMX1hIRGFKeyR2ubIKVM4BylAMLOv2CluENho/UVL7QV G8ycWw8wJciolaNh8tSPDCkwOKjACZ/uX07iKJOZzROuEC0LVtH8SoVZEdmnyh+jLZQ5 pKZ54NyrQ0MJaWPcXqAhPwtaGI7c1UgE5IZ4Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding; b=AJzfOYQ2JpTaKUbQlyRtrh7TlkjddoVHvzHpDbhU8QmXcI5FTbs8VgopHsFwP+vk/X XbBHLvO41RlT7J6ELnp/lU/hSfbDnN9Uxcow6w61brBv3DDOXZHbQ+9mQHTX/tQaGkzV FusYLH9f1yHWiuqkkfq96Q18R3JIkrc5bhgHc= Received: by 10.236.183.165 with SMTP id q25mr71585yhm.470.1304444310709; Tue, 03 May 2011 10:38:30 -0700 (PDT) Received: from unknown ([184.163.87.109]) by mx.google.com with ESMTPS id 13sm134161yhl.73.2011.05.03.10.38.29 (version=SSLv3 cipher=OTHER); Tue, 03 May 2011 10:38:30 -0700 (PDT) Date: Tue, 3 May 2011 13:34:40 -0400 From: Eric Damien To: freebsd-stable@freebsd.org Message-ID: <20110503133440.000031b9@unknown> In-Reply-To: References: <20110502164222.GE81300@mr-happy.com> X-Mailer: Claws Mail 3.7.4cvs1 (GTK+ 2.16.0; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 18:09:38 -0000 Hi Scot, the link you provided is for a FreeBSD MBR Slice.=20 How about the GPT? Because I have the exact same problem, and after following 2.7 (modified for no mirror) on=20 http://wiki.freebsd.org/RootOnZFS/InstallingFreeBSD I did Fixit# sysctl kern.geom.debugflags=3D0x10 Fixit# gpart bootcode -b /zroot/boot/pmbr -p /zroot/boot/gptzfsboot -i 1 a= d0 but got the following error: gpart: /dev/ad0p1: operation not permitted On Tue, 3 May 2011 01:26:21 -0500 Scot Hetzel wrote: > On Mon, May 2, 2011 at 11:42 AM, Jeff Blank > wrote: > > Hi, > > > > I recently upgraded from 8.0-STABLE to 8.2-STABLE (Apr. 29 checkout) > > and upgraded my zpool (includes root FS) from v13 to v15. =A0This is a > > dual-boot laptop, so I'm using MBR/boot0 and not GPT. =A0Here's what > > happens when I boot: > > > > F1 =A0Win > > F2 =A0? > > F3 =A0FreeBSD > > > > F6 PXE > > Boot: =A0F3 > > ZFS: unsupported ZFS version 15 (should be 13) > > No ZFS pools located, can't boot > > > > I've googled around, but I can't find anything relevant for > > MBR/boot0 configurations, just GPT. =A0I've ensured that the loaders > > and boot0/boot1/boot2 are all new, and I rebuilt/reinstalled them > > in a fixit environment just to be sure. =A0I also ran 'boot0cfg > > -B' (with an appropriate -b), but nothing has changed. =A0How can I > > get my pool booting again? > > >=20 > You need to re-install the zfsboot code similar to step 10 (Install > ZFS boot) in >=20 > http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition >=20 > Scot > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Tue May 3 18:36:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A08511065716 for ; Tue, 3 May 2011 18:36:02 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 24C1B8FC1C for ; Tue, 3 May 2011 18:36:01 +0000 (UTC) Received: by bwz12 with SMTP id 12so510863bwz.13 for ; Tue, 03 May 2011 11:36:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=teAiEODnZnh1OA3RnmeE4ElHzkDzQR86puBWd5omZdk=; b=ZAtl7i5bAV/EUOf10yz2ucnG/uKpXBTFixdRVYkHGU1QLoF5Byc2PmHJ/doORQCEpM 5cDv6zqZMSxGG7W+fvLGgm/VDovOM4zNXNf+V3JvtyeF5b0linRvrqm7bX9JZFd1AnFp /9nCJzjG6+cjQiRhx3XgTIVebtS4ordjJxGYQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=UpsqDn07ySLPUXQMvSyQdmGnLlyXZclSgIQz00EMQFDzfZ//PYpMKPUNqRdVbOocJO ZFMBbWib9LfK5AJBSZoKl6J915iHb7JF8OhyWDf3U6hDQT92SXtC1MEVC9A0o3I/eh2d g8Ai7a1MaYt6eRR7TuIvXuqVQclLzxkP20MWk= MIME-Version: 1.0 Received: by 10.204.20.142 with SMTP id f14mr141976bkb.155.1304447760538; Tue, 03 May 2011 11:36:00 -0700 (PDT) Received: by 10.204.78.134 with HTTP; Tue, 3 May 2011 11:36:00 -0700 (PDT) In-Reply-To: <20110503133440.000031b9@unknown> References: <20110502164222.GE81300@mr-happy.com> <20110503133440.000031b9@unknown> Date: Tue, 3 May 2011 13:36:00 -0500 Message-ID: From: Scot Hetzel To: Eric Damien Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 18:36:02 -0000 On Tue, May 3, 2011 at 12:34 PM, Eric Damien wrote: > Hi Scot, > > the link you provided is for a FreeBSD MBR Slice. > How about the GPT? Because I have the exact same problem, > and after following =A02.7 (modified for no mirror) on > =A0 =A0 =A0 =A0http://wiki.freebsd.org/RootOnZFS/InstallingFreeBSD > > I did > =A0Fixit# sysctl kern.geom.debugflags=3D0x10 > =A0Fixit# gpart bootcode -b /zroot/boot/pmbr -p /zroot/boot/gptzfsboot -i= 1 ad0 > > but got the following error: > =A0 =A0 =A0 =A0gpart: /dev/ad0p1: operation not permitted > > That should have worked. Is partition 1 (ad0p1) your freebsd-boot partition? Scot From owner-freebsd-stable@FreeBSD.ORG Wed May 4 01:40:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 175DF1065794 for ; Wed, 4 May 2011 01:40:15 +0000 (UTC) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Received: from pis.elm.toba-cmt.ac.jp (pis.elm.toba-cmt.ac.jp [202.26.248.196]) by mx1.freebsd.org (Postfix) with ESMTP id CBCFB8FC20 for ; Wed, 4 May 2011 01:40:14 +0000 (UTC) Received: from kiri.pis.pis.elm.toba-cmt.ac.jp (localhost [127.0.0.1]) by pis.elm.toba-cmt.ac.jp (8.14.3/8.14.2) with ESMTP id p441eClM054591; Wed, 4 May 2011 10:40:13 +0900 (JST) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Message-Id: <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> Date: Wed, 04 May 2011 10:40:12 +0900 From: KIRIYAMA Kazuhiko To: Ian Smith In-Reply-To: <20110504030404.O85801@sola.nimnet.asn.au> References: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> <20110504030404.O85801@sola.nimnet.asn.au> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 MULE XEmacs/21.4 (patch 21) (Educational Television) (i386--freebsd) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: KIRIYAMA Kazuhiko , freebsd-stable@freebsd.org Subject: Re: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 01:40:15 -0000 At Wed, 4 May 2011 03:47:02 +1000 (EST), Ian Smith wrote: > > On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > > Hi all, > > Recently I upgraded to 8.2-STABLE and reconfigured natd + jailed box, but > > all packets could not over nat box. I've researched and found > > /etc/rc.firewall does not recieve argument of firewall_type. So ipfw does > > not divert and natd could not be performed. The reason is /etc/rc.d/ipfw > > incorrect. I think an patch below should be applyed to /etc/rc.d/ipfw. Is > > there any problem to do this? > > Yes. Assuming using the default firewall_script="/etc/rc.firewall", > then as it says early in /etc/rc.firewall, you just needed to: > > # Define the firewall type in /etc/rc.conf. Valid values are: > [..] > > Sure, /etc/rc.firewall can set firewall_type to a parameter if you pass > it one, but otherwise uses whatever $firewall_type is set to when you > start ipfw. I guess the code below allows you to use syntax like: > > # /etc/rc.d/ipfw start client I missed it intended to use in commandline but usually /etc/rc.d/* script uses at startup rc. If /etc/rc.d/ipfw must be 2 arguments,firewall_type always undefined at startup nevertheless it specified in /etc/rc.conf. It is the very serious problem isn't it? > to override the $firewall_type set in /etc/rc.conf, but it's not the > common usage, nor is it how ipfw is started normally by rc. > > So just set firewall_type in rc.conf and you should be fine .. unless > you meant that you're trying to run ipfw & natd INSIDE a jail? The network being configure is as follows: xxxx.xxxx.xxxx.xxxx/27 -------------------------+---------------------------------------- |53 +----------------------+---------------------------------------+ | bge0 jailed natd box | | t2.st.foo (ipfw `OPEN') | | +--------+--------+--------+--------+--------+--------+ |firewall| ns | ldap |diskless| mail | web | ftp | | bge1 | bge1 | bge1 | bge1 | bge1 | bge1 | bge1 | +----+---+----+---+----+---+----+---+----+---+----+---+----+---+ 254| 1| 2| 3| 4| 5| 6| -------+--------+--------+--------+--------+--------+--------+---- 192.168.2.0/24 > cheers, Ian > > > --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 > > +++ /etc/rc.d/ipfw 2011-05-03 22:08:14.000000000 +0900 > > @@ -35,15 +35,11 @@ > > > > ipfw_start() > > { > > - local _firewall_type > > - > > - _firewall_type=$1 > > - > > # set the firewall rules script if none was specified > > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall > > > > if [ -r "${firewall_script}" ]; then > > - /bin/sh "${firewall_script}" "${_firewall_type}" > > + /bin/sh "${firewall_script}" "${firewall_type}" > > echo 'Firewall rules loaded.' > > elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then > > echo 'Warning: kernel has firewall functionality, but' \ For the case of commandline usage, above patch should be modified as follows: --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 +++ /etc/rc.d/ipfw 2011-05-04 09:31:09.000000000 +0900 @@ -37,7 +37,11 @@ { local _firewall_type - _firewall_type=$1 + if [ -n "${1}" ]; then + _firewall_type=$1 + elif [ -n "${firewall_type}" ] + _firewall_type=${firewall_type} + fi # set the firewall rules script if none was specified [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall From owner-freebsd-stable@FreeBSD.ORG Wed May 4 01:49:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CFFB1065670 for ; Wed, 4 May 2011 01:49:49 +0000 (UTC) (envelope-from nokiacashsplashpromo2011@nokia.com) Received: from nm27-vm0.bullet.mail.sp2.yahoo.com (nm27-vm0.bullet.mail.sp2.yahoo.com [98.139.91.232]) by mx1.freebsd.org (Postfix) with SMTP id 502738FC15 for ; Wed, 4 May 2011 01:49:49 +0000 (UTC) Received: from [98.139.91.64] by nm27.bullet.mail.sp2.yahoo.com with NNFMP; 04 May 2011 01:36:31 -0000 Received: from [98.139.91.44] by tm4.bullet.mail.sp2.yahoo.com with NNFMP; 04 May 2011 01:36:31 -0000 Received: from [127.0.0.1] by omp1044.mail.sp2.yahoo.com with NNFMP; 04 May 2011 01:36:31 -0000 X-Yahoo-Newman-Id: 44210.44409.bm@omp1044.mail.sp2.yahoo.com Received: (qmail 83125 invoked from network); 4 May 2011 01:36:30 -0000 Received: from Udala-PC (nokiacashsplashpromo2011@80.251.4.10 with login) by smtp111.biz.mail.sp1.yahoo.com with SMTP; 03 May 2011 18:36:28 -0700 PDT X-Yahoo-SMTP: 8L_BkGeswBCqDjl0FMKSiY4vqAp4lbHL X-YMail-OSG: qR9q8psVM1lWv9X2EcfO6rgGAtSRsY54Ey_hXn.90SX7r3x Umi7OgKual99gswFY1ImPcglWRaYQ1Y282aufgFdwFEE_TG8VJyQLw.1L1d7 JeQVNroZ4M2vcTQ798oMiKrqFkCoRfjSNl7Pg0jN0Y.UHF70QvbICYHtzpkw 1.qzADQnYiXYXnIvwotZieEUi.QceOXtZRJLQIN3tNacc4mQgHnj5fAbKxWa o3tK46eQgFToB9TfjB8fkztZI_1ATkWwVKDtvn7WYl4nHW2GKgrOgclV2wH. Qqtp_Moot.QoCa3JJgXbFGZHOG7aHQrBhKYG6809QySi7u5jMOGN4zSXV1XZ tyQ-- X-Yahoo-Newman-Property: ymail-3 Message-ID: <01cead5d-40667-0fcd1085917708@udala-pc> From: "Richard A Simonson" To: freebsd-stable@freebsd.org Date: Wed, 4 May 2011 02:36:23 +0100 Content-Transfer-Encoding: 8bit X-Priority: 3 X-Mailer: Power Sending Sockets v5.1 MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Very Important X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Richard A Simonson List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 01:49:49 -0000 [logo.gif] _________________________________________________________________ Dear freebsd-stable@freebsd.org, You or someone entered your information on our web Ad for the Nokia 2011 Cash Splash Promo and we are using this medium to officially notify you that the result of the promo was recently released as shown below: S/N Winners List Amount Won Ticket Number Status 1 mohammed.matinshah@shell.com 700,000.00 GBP *** Unclaimed 2 freebsd-stable@freebsd.org 400,000.00 GBP BNK04022818T Unclaimed 3 elliana.fed@bluewin.ch 200,000.00 GBP *** Unclaimed We are happy to inform you that you are our second prize winner and you won 400,000.00 GBP (Four Hundred thousand Great British Pounds) at this promo. Your ticket number is BNK04022818T. Please take note of your ticket number as it will be needed for verification. To receive your prize money as soon as possible, simply fax us the following information for verification: 1. Your Full names 2. Residential/Office address 3. Phone number 4. Fax number 5.Email address 6. Occupation 7. Ticket number Your prize money will be released to you as soon as you have faxed the above information to our office for verification. Our fax number is +44-700-593-0907. The required information should be sent via fax only. Bear it in mind that you won big in this promo because you are one of our valuable customers. However, if you wish to decline the receipt of your prize money, you are to notify us on time so that the opportunity will be given to another Nokia phone user. WARNING: Because of numerous fraudulent schemes going around the internet, confidentiality should be accorded at all times. You are expected to keep the news of your winning and your ticket number to yourself until you have received your prize money. This is to avoid false claims and abuse of the program. Nokia Corporation will never ask you to pay any fee before you receive your prize money. You can call +44-7024014788 OR +44-7024019130 if you wish to speak to someone from our office. Congratulations!!! Richard A Simonson Chief Financial Officer Nokia Corporation - UK Tel: +44-7024014788 Tel: +44-7024019130 Web: http://www.nokia.com Copyright � 2011 Nokia Corporation. All rights reserved From owner-freebsd-stable@FreeBSD.ORG Wed May 4 02:07:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 688A5106564A for ; Wed, 4 May 2011 02:07:18 +0000 (UTC) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Received: from pis.elm.toba-cmt.ac.jp (pis.elm.toba-cmt.ac.jp [202.26.248.196]) by mx1.freebsd.org (Postfix) with ESMTP id 16D2B8FC12 for ; Wed, 4 May 2011 02:07:17 +0000 (UTC) Received: from kiri.pis.pis.elm.toba-cmt.ac.jp (localhost [127.0.0.1]) by pis.elm.toba-cmt.ac.jp (8.14.3/8.14.2) with ESMTP id p4427Hs1055072 for ; Wed, 4 May 2011 11:07:17 +0900 (JST) (envelope-from kiri@pis.elm.toba-cmt.ac.jp) Message-Id: <201105040207.p4427Hs1055072@pis.elm.toba-cmt.ac.jp> Date: Wed, 04 May 2011 11:07:17 +0900 From: KIRIYAMA Kazuhiko To: freebsd-stable@freebsd.org In-Reply-To: <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> References: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> <20110504030404.O85801@sola.nimnet.asn.au> <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 MULE XEmacs/21.4 (patch 21) (Educational Television) (i386--freebsd) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: Re: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 02:07:18 -0000 At Wed, 04 May 2011 10:40:12 +0900, My wrote: > > At Wed, 4 May 2011 03:47:02 +1000 (EST), > Ian Smith wrote: > > > > > +++ /etc/rc.d/ipfw 2011-05-03 22:08:14.000000000 +0900 > > > @@ -35,15 +35,11 @@ > > > > > > ipfw_start() > > > { > > > - local _firewall_type > > > - > > > - _firewall_type=$1 > > > - > > > # set the firewall rules script if none was specified > > > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall > > > > > > if [ -r "${firewall_script}" ]; then > > > - /bin/sh "${firewall_script}" "${_firewall_type}" > > > + /bin/sh "${firewall_script}" "${firewall_type}" > > > echo 'Firewall rules loaded.' > > > elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then > > > echo 'Warning: kernel has firewall functionality, but' \ > > For the case of commandline usage, above patch should be modified as > follows: > > --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 > +++ /etc/rc.d/ipfw 2011-05-04 09:31:09.000000000 +0900 > @@ -37,7 +37,11 @@ > { > local _firewall_type > > - _firewall_type=$1 > + if [ -n "${1}" ]; then > + _firewall_type=$1 > + elif [ -n "${firewall_type}" ] > + _firewall_type=${firewall_type} > + fi > > # set the firewall rules script if none was specified > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall Above patch has typo. Collect one is as follows: --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 +++ /etc/rc.d/ipfw 2011-05-04 09:53:40.000000000 +0900 @@ -37,7 +37,11 @@ { local _firewall_type - _firewall_type=$1 + if [ -n "${1}" ]; then + _firewall_type=$1 + elif [ -n "${firewall_type}" ]; then + _firewall_type=${firewall_type} + fi # set the firewall rules script if none was specified [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall From owner-freebsd-stable@FreeBSD.ORG Wed May 4 04:37:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4C8B106566C for ; Wed, 4 May 2011 04:37:23 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 0F3718FC0C for ; Wed, 4 May 2011 04:37:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id p444bHOY057175; Wed, 4 May 2011 14:37:18 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Wed, 4 May 2011 14:37:17 +1000 (EST) From: Ian Smith To: KIRIYAMA Kazuhiko In-Reply-To: <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> Message-ID: <20110504131936.B85801@sola.nimnet.asn.au> References: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> <20110504030404.O85801@sola.nimnet.asn.au> <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-stable@freebsd.org Subject: Re: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 04:37:23 -0000 On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > At Wed, 4 May 2011 03:47:02 +1000 (EST), > Ian Smith wrote: > > > > On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > > > Hi all, > > > Recently I upgraded to 8.2-STABLE and reconfigured natd + jailed box, but > > > all packets could not over nat box. I've researched and found > > > /etc/rc.firewall does not recieve argument of firewall_type. So ipfw does > > > not divert and natd could not be performed. The reason is /etc/rc.d/ipfw > > > incorrect. I think an patch below should be applyed to /etc/rc.d/ipfw. Is > > > there any problem to do this? > > > > Yes. Assuming using the default firewall_script="/etc/rc.firewall", > > then as it says early in /etc/rc.firewall, you just needed to: > > > > # Define the firewall type in /etc/rc.conf. Valid values are: > > [..] > > > > Sure, /etc/rc.firewall can set firewall_type to a parameter if you pass > > it one, but otherwise uses whatever $firewall_type is set to when you > > start ipfw. I guess the code below allows you to use syntax like: > > > > # /etc/rc.d/ipfw start client > I missed it intended to use in commandline but usually /etc/rc.d/* script > uses at startup rc. If /etc/rc.d/ipfw must be 2 arguments,firewall_type > always undefined at startup nevertheless it specified in /etc/rc.conf. It > is the very serious problem isn't it? /etc/rc.d/ipfw normally only takes one argument, {,quiet}start|stop|etc. The use of $1 in ipfw_start() surprised me actually, I'm only assuming its above intended use, but it's clearly an extra argument passed by rc, not the first argument to /etc/rc.d/ipfw itself (ie start|stop etc). Sorry to repeat, but normally firewall_type should be set in rc.conf - which works properly; no patching of /etc/rc.d/ipfw is needed. > > to override the $firewall_type set in /etc/rc.conf, but it's not the > > common usage, nor is it how ipfw is started normally by rc. > > > > So just set firewall_type in rc.conf and you should be fine .. unless > > you meant that you're trying to run ipfw & natd INSIDE a jail? > > The network being configure is as follows: > xxxx.xxxx.xxxx.xxxx/27 > -------------------------+---------------------------------------- > |53 > +----------------------+---------------------------------------+ > | bge0 jailed natd box | > | t2.st.foo (ipfw `OPEN') | > | +--------+--------+--------+--------+--------+--------+ > |firewall| ns | ldap |diskless| mail | web | ftp | > | bge1 | bge1 | bge1 | bge1 | bge1 | bge1 | bge1 | > +----+---+----+---+----+---+----+---+----+---+----+---+----+---+ > 254| 1| 2| 3| 4| 5| 6| > -------+--------+--------+--------+--------+--------+--------+---- > 192.168.2.0/24 I'm not entirely sure how to interpret your diagram, but as far as I am aware you can run neither ipfw nor natd within a jail; both scripts have 'KEYWORD: nojail' so they won't be run on jail startup. There's been mention of work underway with VIMAGE toward a full stack inside jail(s), but for now you can run ipfw (and natd) only on the host system. > > > --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 > > > +++ /etc/rc.d/ipfw 2011-05-03 22:08:14.000000000 +0900 > > > @@ -35,15 +35,11 @@ > > > > > > ipfw_start() > > > { > > > - local _firewall_type > > > - > > > - _firewall_type=$1 > > > - > > > # set the firewall rules script if none was specified > > > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall > > > > > > if [ -r "${firewall_script}" ]; then > > > - /bin/sh "${firewall_script}" "${_firewall_type}" > > > + /bin/sh "${firewall_script}" "${firewall_type}" > > > echo 'Firewall rules loaded.' > > > elif [ "`ipfw list 65535`" = "65535 deny ip from any to any" ]; then > > > echo 'Warning: kernel has firewall functionality, but' \ > > For the case of commandline usage, above patch should be modified as > follows: > > --- /etc/rc.d/ipfw.org 2011-05-03 18:19:28.000000000 +0900 > +++ /etc/rc.d/ipfw 2011-05-04 09:31:09.000000000 +0900 > @@ -37,7 +37,11 @@ > { > local _firewall_type > > - _firewall_type=$1 > + if [ -n "${1}" ]; then > + _firewall_type=$1 > + elif [ -n "${firewall_type}" ] > + _firewall_type=${firewall_type} > + fi > > # set the firewall rules script if none was specified > [ -z "${firewall_script}" ] && firewall_script=/etc/rc.firewall It's still unnecessary to mess with this. See /etc/rc.firewall for its use of $firewall_type which may be set to open|closed|client|simple etc, or can be the full path to another script. rc.firewall MAY also take a single parameter to override the firewall_type set in rc.conf, but again that's an override, not the regular usage. cheers, Ian From owner-freebsd-stable@FreeBSD.ORG Wed May 4 06:46:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8EAF1065670; Wed, 4 May 2011 06:46:27 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id B82528FC15; Wed, 4 May 2011 06:46:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id p446kNx4063838; Wed, 4 May 2011 16:46:23 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Wed, 4 May 2011 16:46:22 +1000 (EST) From: Ian Smith To: KIRIYAMA Kazuhiko In-Reply-To: <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> Message-ID: <20110504160556.Q85801@sola.nimnet.asn.au> References: <201105031543.p43Fh92T041708@pis.elm.toba-cmt.ac.jp> <20110504030404.O85801@sola.nimnet.asn.au> <201105040140.p441eClM054591@pis.elm.toba-cmt.ac.jp> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-stable@freebsd.org Subject: Re: /etc/rc.d/ipfw can't deal with firewall_type? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 06:46:27 -0000 On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > At Wed, 4 May 2011 03:47:02 +1000 (EST), > Ian Smith wrote: > > > > On Wed, 4 May 2011, KIRIYAMA Kazuhiko wrote: > > > Hi all, > > > Recently I upgraded to 8.2-STABLE and reconfigured natd + jailed box, but > > > all packets could not over nat box. I've researched and found > > > /etc/rc.firewall does not recieve argument of firewall_type. So ipfw does > > > not divert and natd could not be performed. The reason is /etc/rc.d/ipfw > > > incorrect. I think an patch below should be applyed to /etc/rc.d/ipfw. Is > > > there any problem to do this? > > > > Yes. Assuming using the default firewall_script="/etc/rc.firewall", > > then as it says early in /etc/rc.firewall, you just needed to: > > > > # Define the firewall type in /etc/rc.conf. Valid values are: > > [..] It's just occured to me that - assuming you are NOT trying to start ipfw or natd inside a jail, which won't work - you may well be running into another problem related to some PRs/patches hrs@ (cc'd) is reviewing re startup order and loading of modules for ipfw and natd. You mentioned running an 'OPEN' firewall which (like any other type) will fail to load divert rule/s unless ipdivert.ko is already loaded or built into kernel. This can be solved meanwhile by either a) adding to /boot/loader.conf: ipdivert_load="YES" or b) by applying the following patch to /etc/rc.d/ipfw (on 7.x or 8.x) cheers, Ian --- rc.d_ipfw.1.24 Sat Jan 8 18:13:46 2011 +++ ipfw Sat Jan 8 21:00:18 2011 @@ -27,9 +27,9 @@ fi if checkyesno firewall_nat_enable; then - if ! checkyesno natd_enable; then - required_modules="$required_modules ipfw_nat" - fi + required_modules="$required_modules ipfw_nat" + elif checkyesno natd_enable; then + required_modules="$required_modules ipdivert" fi } @@ -105,6 +105,7 @@ } load_rc_config $name -firewall_coscripts="/etc/rc.d/natd ${firewall_coscripts}" +checkyesno natd_enable && ! checkyesno firewall_nat_enable && \ + firewall_coscripts="/etc/rc.d/natd ${firewall_coscripts}" run_rc_command $* From owner-freebsd-stable@FreeBSD.ORG Wed May 4 11:58:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9057A106564A for ; Wed, 4 May 2011 11:58:18 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 43DEC8FC15 for ; Wed, 4 May 2011 11:58:18 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1QHaie-0004iC-U5 for freebsd-stable@freebsd.org; Wed, 04 May 2011 13:58:16 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 04 May 2011 13:58:16 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 04 May 2011 13:58:16 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Wed, 04 May 2011 13:57:53 +0200 Lines: 26 Message-ID: References: <20110501042028.GA87381@icarus.home.lan> <20110502154849.0000746b@unknown> <201105021356.49585.jkim@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 In-Reply-To: <201105021356.49585.jkim@FreeBSD.org> X-Enigmail-Version: 1.1.2 Subject: Re: Is machdep.cpu_idle_hlt deprecated? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 11:58:18 -0000 On 02/05/2011 19:56, Jung-uk Kim wrote: > On Monday 02 May 2011 10:48 am, Bruce Cran wrote: >> On Sat, 30 Apr 2011 21:20:28 -0700 >> >> Jeremy Chadwick wrote: >>> Anyone know if machdep.cpu_idle_hlt still exists? Taken from >>> acpi(4) on RELENG_8: >> >> It looks like it might have been replaced by machdep.idle: >> >> machdep.idle: currently selected idle function >> machdep.idle_available: list of available idle functions >> >> machdep.idle: acpi >> machdep.idle_available: spin, hlt, acpi, > > It seems "machdep.cpu_idle_hlt" was deprecated long ago with this > commit by jeff (CC'ed): > > http://svnweb.freebsd.org/base?view=revision&revision=178471 How likely is this to affect real-world performance on e.g. a busy web server? As the comments say, the "fastest" are spin and mwait and the slowest is acpi, which is also the default. I have no reference point for what "fast" and "slow" mean in this context :) From owner-freebsd-stable@FreeBSD.ORG Wed May 4 12:01:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B29501065670 for ; Wed, 4 May 2011 12:01:24 +0000 (UTC) (envelope-from fbsdstable@beasties.demon.nl) Received: from smtp-vbr15.xs4all.nl (smtp-vbr15.xs4all.nl [194.109.24.35]) by mx1.freebsd.org (Postfix) with ESMTP id 4CBAE8FC1B for ; Wed, 4 May 2011 12:01:23 +0000 (UTC) Received: from merom.hotsoft.nl (beasties.demon.nl [82.161.3.114]) by smtp-vbr15.xs4all.nl (8.13.8/8.13.8) with ESMTP id p44Bn60f043737 for ; Wed, 4 May 2011 13:49:06 +0200 (CEST) (envelope-from fbsdstable@beasties.demon.nl) Message-ID: <4DC13D32.1070503@beasties.demon.nl> Date: Wed, 04 May 2011 13:49:06 +0200 From: Hans Ottevanger User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by XS4ALL Virus Scanner Subject: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 12:01:24 -0000 Hi, I upgraded my Soekris 4801 boxes from 8.1 to 8.2-STABLE (r221326) a few days ago and now I get the following error in the daily mail: Backing up package db directory: tar: : Cannot stat: No such file or directory tar: Error exit delayed from previous errors. These messages originate from /etc/periodic/daily/220.backup-pkgdb, apparently a recent addition. The culprit is probably on line 21: make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR "/usr/share/mk/bsd.port.mk", line 11: Could not find /usr/ports/Mk/bsd.port.mk make: fatal errors encountered -- cannot continue If there is no /usr/ports present on the system, the script will fail. Of course my systems are not unique in this respect: many people install pre-built packages instead of building ports themselves, especially on minimal hardware configurations. Would it be a good idea to check for the presence of /usr/ports (or /usr/ports/Mk/bsd.ports.mk) first before calling make, and then try $PKG_DBDIR and /var/db/pkg, in that order, if this is not the case? As far as I know this issue also affects 9.0-CURRENT. Kind regards, Hans Ottevanger From owner-freebsd-stable@FreeBSD.ORG Wed May 4 12:30:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 140B2106566C for ; Wed, 4 May 2011 12:30:13 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id EDC218FC0C for ; Wed, 4 May 2011 12:30:12 +0000 (UTC) Received: from omta24.emeryville.ca.mail.comcast.net ([76.96.30.92]) by qmta15.emeryville.ca.mail.comcast.net with comcast id fQ7W1g0021zF43QAFQWC5Z; Wed, 04 May 2011 12:30:12 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.emeryville.ca.mail.comcast.net with comcast id fQWA1g00k1t3BNj8kQWAAE; Wed, 04 May 2011 12:30:12 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4A7FF102C36; Wed, 4 May 2011 05:30:10 -0700 (PDT) Date: Wed, 4 May 2011 05:30:10 -0700 From: Jeremy Chadwick To: Hans Ottevanger Message-ID: <20110504123010.GA89218@icarus.home.lan> References: <4DC13D32.1070503@beasties.demon.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4DC13D32.1070503@beasties.demon.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dougb@freebsd.org, freebsd-stable@freebsd.org, florent.thoumie@gmail.com Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 12:30:13 -0000 On Wed, May 04, 2011 at 01:49:06PM +0200, Hans Ottevanger wrote: > I upgraded my Soekris 4801 boxes from 8.1 to 8.2-STABLE (r221326) a > few days ago and now I get the following error in the daily mail: > > Backing up package db directory: > tar: : Cannot stat: No such file or directory > tar: Error exit delayed from previous errors. > > These messages originate from /etc/periodic/daily/220.backup-pkgdb, > apparently a recent addition. CC'ing committer and author. > The culprit is probably on line 21: > > make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR > "/usr/share/mk/bsd.port.mk", line 11: Could not find > /usr/ports/Mk/bsd.port.mk > make: fatal errors encountered -- cannot continue > > If there is no /usr/ports present on the system, the script will > fail. Of course my systems are not unique in this respect: many > people install pre-built packages instead of building ports > themselves, especially on minimal hardware configurations. Agreed. > Would it be a good idea to check for the presence of /usr/ports (or > /usr/ports/Mk/bsd.ports.mk) first before calling make, and then try > $PKG_DBDIR and /var/db/pkg, in that order, if this is not the case? Again, agreed. Others may have other recommendations/solutions that are equally palatable. Overall, the script needs some more test conditionals for added ""error"" checking. I should also take a moment to point out a PR I just created tonight for RELENG_7, since this periodic script did get backported to that branch. PR pertains to tar(1) on RELENG_7 complaining about leading slashes on stderr, while RELENG_8 and newer do not due to differences in each respective branches' util.c: http://www.freebsd.org/cgi/query-pr.cgi?pr=156810 -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed May 4 12:49:06 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB9CB106564A for ; Wed, 4 May 2011 12:49:06 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 86CDB8FC0A for ; Wed, 4 May 2011 12:49:06 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:c9d:ce4b:69a5:29c1]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 1AE834AC2D for ; Wed, 4 May 2011 16:49:05 +0400 (MSD) Date: Wed, 4 May 2011 16:49:01 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <455293202.20110504164901@serebryakov.spb.ru> To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Subject: How to understand, what userland program does in kernel? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 12:49:07 -0000 Hello, Freebsd-stable. I have userland program (transmission BT client), which spent 100% of one core of E4500 CPU when it has many peers. It is surprises me, as channel is only 35Mbit, and my "Linux" friends can upload much more on comparable hardware. But what surprises me even more, that 50% of this time it spends as System time. Is here any way to understand, what transmission does in kernel for so much time? It seems, that userland profiling doesn't help me, am I right? --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-stable@FreeBSD.ORG Wed May 4 13:18:16 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E765106566C; Wed, 4 May 2011 13:18:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 35A308FC0C; Wed, 4 May 2011 13:18:14 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA22690; Wed, 04 May 2011 16:18:12 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4DC15213.2090602@FreeBSD.org> Date: Wed, 04 May 2011 16:18:11 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.15) Gecko/20110309 Lightning/1.0b2 Thunderbird/3.1.9 MIME-Version: 1.0 To: lev@FreeBSD.org References: <455293202.20110504164901@serebryakov.spb.ru> In-Reply-To: <455293202.20110504164901@serebryakov.spb.ru> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: How to understand, what userland program does in kernel? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 13:18:16 -0000 on 04/05/2011 15:49 Lev Serebryakov said the following: > Hello, Freebsd-stable. > > I have userland program (transmission BT client), which spent 100% > of one core of E4500 CPU when it has many peers. It is surprises me, > as channel is only 35Mbit, and my "Linux" friends can upload much more > on comparable hardware. > > But what surprises me even more, that 50% of this time it spends as > System time. > > Is here any way to understand, what transmission does in kernel for > so much time? It seems, that userland profiling doesn't help me, am I > right? ktrace can easily show you what system calls are made by your program and how much time they take. Maybe that would be sufficient for you. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Wed May 4 13:18:52 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F1721065754 for ; Wed, 4 May 2011 13:18:52 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 3E7CD8FC1E for ; Wed, 4 May 2011 13:18:52 +0000 (UTC) Received: from outgoing.leidinger.net (p5B155BB6.dip.t-dialin.net [91.21.91.182]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 2DEE0844010; Wed, 4 May 2011 15:18:38 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [IPv6:fd73:10c7:2053:1::2:102]) by outgoing.leidinger.net (Postfix) with ESMTP id 3B92D11DD; Wed, 4 May 2011 15:18:35 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id p44DIZ3l054047; Wed, 4 May 2011 15:18:35 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Wed, 04 May 2011 15:18:35 +0200 Message-ID: <20110504151835.10858df55klyghf4@webmail.leidinger.net> Date: Wed, 04 May 2011 15:18:35 +0200 From: Alexander Leidinger To: lev@FreeBSD.org References: <455293202.20110504164901@serebryakov.spb.ru> In-Reply-To: <455293202.20110504164901@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.6) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 2DEE0844010.AFE34 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=0, required 6, autolearn=disabled) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1305119919.56075@3fBHYaJBMf6KqRqSZUMJBA X-EBL-Spam-Status: No Cc: freebsd-stable@FreeBSD.org Subject: Re: How to understand, what userland program does in kernel? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 13:18:52 -0000 Quoting Lev Serebryakov (from Wed, 4 May 2011 16:49:01 +0400): > Hello, Freebsd-stable. > > I have userland program (transmission BT client), which spent 100% > of one core of E4500 CPU when it has many peers. It is surprises me, > as channel is only 35Mbit, and my "Linux" friends can upload much more > on comparable hardware. > > But what surprises me even more, that 50% of this time it spends as > System time. > > Is here any way to understand, what transmission does in kernel for > so much time? It seems, that userland profiling doesn't help me, am I > right? ktrace and dtrace are your friends. ktrace for a simple "it makes those syscalls/ioctls/..." type of information gathering, and dtrace for in-deep investigation. Bye, Alexander. -- The difference between a good haircut and a bad one is seven days. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-stable@FreeBSD.ORG Wed May 4 13:26:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C180106566B; Wed, 4 May 2011 13:26:41 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id E40008FC18; Wed, 4 May 2011 13:26:40 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:c9d:ce4b:69a5:29c1]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 1882A4AC1C; Wed, 4 May 2011 17:26:40 +0400 (MSD) Date: Wed, 4 May 2011 17:26:36 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <369893537.20110504172636@serebryakov.spb.ru> To: Lev Serebryakov In-Reply-To: <455293202.20110504164901@serebryakov.spb.ru> References: <455293202.20110504164901@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: How to understand, what userland program does in kernel? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 13:26:41 -0000 Hello, Lev. You wrote 4 =EC=E0=FF 2011 =E3., 16:49:01: > I have userland program (transmission BT client), which spent 100% > of one core of E4500 CPU when it has many peers. It is surprises me, > as channel is only 35Mbit, and my "Linux" friends can upload much more > on comparable hardware. > But what surprises me even more, that 50% of this time it spends as > System time. When it spents, says, 75% of one core, SYSTEM is only 1-2%, not 75%/2 =3D~ 37%, what is interesting... --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-stable@FreeBSD.ORG Wed May 4 19:26:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 7DE58106564A for ; Wed, 4 May 2011 19:26:03 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 65-241-43-5.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id C98D914FCED; Wed, 4 May 2011 19:26:02 +0000 (UTC) Message-ID: <4DC1A849.70807@FreeBSD.org> Date: Wed, 04 May 2011 12:26:01 -0700 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Hans Ottevanger References: <4DC13D32.1070503@beasties.demon.nl> In-Reply-To: <4DC13D32.1070503@beasties.demon.nl> X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 19:26:03 -0000 On 05/04/2011 04:49, Hans Ottevanger wrote: > Hi, > > I upgraded my Soekris 4801 boxes from 8.1 to 8.2-STABLE (r221326) a few > days ago and now I get the following error in the daily mail: > > Backing up package db directory: > tar: : Cannot stat: No such file or directory > tar: Error exit delayed from previous errors. > > These messages originate from /etc/periodic/daily/220.backup-pkgdb, > apparently a recent addition. > > The culprit is probably on line 21: > > make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR > "/usr/share/mk/bsd.port.mk", line 11: Could not find > /usr/ports/Mk/bsd.port.mk > make: fatal errors encountered -- cannot continue Thanks, I'll take a look. -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Wed May 4 19:28:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEC801065670 for ; Wed, 4 May 2011 19:28:35 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from smtp03.lnh.mail.rcn.net (smtp03.lnh.mail.rcn.net [207.172.157.103]) by mx1.freebsd.org (Postfix) with ESMTP id 9BD148FC08 for ; Wed, 4 May 2011 19:28:35 +0000 (UTC) Received: from mr16.lnh.mail.rcn.net ([207.172.157.36]) by smtp02.lnh.mail.rcn.net with ESMTP; 04 May 2011 14:59:37 -0400 Received: from smtp01.lnh.mail.rcn.net (smtp01.lnh.mail.rcn.net [207.172.4.11]) by mr16.lnh.mail.rcn.net (MOS 4.2.3-GA) with ESMTP id BBF68263; Wed, 4 May 2011 14:59:36 -0400 Received-SPF: None identity=pra; client-ip=209.6.61.133; receiver=smtp01.lnh.mail.rcn.net; envelope-from="mi+thun@aldan.algebra.com"; x-sender="mi+thun@aldan.algebra.com"; x-conformance=sidf_compatible Received-SPF: None identity=mailfrom; client-ip=209.6.61.133; receiver=smtp01.lnh.mail.rcn.net; envelope-from="mi+thun@aldan.algebra.com"; x-sender="mi+thun@aldan.algebra.com"; x-conformance=sidf_compatible Received-SPF: None identity=helo; client-ip=209.6.61.133; receiver=smtp01.lnh.mail.rcn.net; envelope-from="mi+thun@aldan.algebra.com"; x-sender="postmaster@utka.zajac"; x-conformance=sidf_compatible X-Auth-ID: anat Received: from unknown (HELO utka.zajac) ([209.6.61.133]) by smtp01.lnh.mail.rcn.net with ESMTP; 04 May 2011 14:59:37 -0400 Message-ID: <4DC1A1CC.4080901@aldan.algebra.com> Date: Wed, 04 May 2011 14:58:20 -0400 From: "Mikhail T." User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.13) Gecko/20101229 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: syslogd spinning the CPU, not logging... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 19:28:36 -0000 On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this problem the first time (a week or two ago), but just saw it again. Something triggers syslogd into spinning all available CPU -- while not logging anything. Attempts to log anything -- such as by invoking logger(1) -- simply hang. sendmail hangs too -- now new e-mail is coming. It is not pretty. ktrace-ing the process yields empty file -- it is not doing system-calls, while spinning. Attaching the debugger several times shows the same stack: #0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 #1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 #2 0x00000008007d4825 in free () from /lib/libc.so.7 #3 0x00000008007c5aab in catopen () from /lib/libc.so.7 #4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7 #5 0x00000008007c556d in strerror () from /lib/libc.so.7 #6 0x0000000000403c10 in ?? () ... Among the logging-destinations in my syslog.conf there is a program, that exits sometimes -- but never "too fast". Normally, syslogd simply restarts it without an issue... Any ideas? Thanks! -mi From owner-freebsd-stable@FreeBSD.ORG Wed May 4 21:42:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A8BB1065670 for ; Wed, 4 May 2011 21:42:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 4FE108FC08 for ; Wed, 4 May 2011 21:42:37 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta13.emeryville.ca.mail.comcast.net with comcast id fZge1g0041afHeLADZidic; Wed, 04 May 2011 21:42:37 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id fZia1g00C1t3BNj8dZiadn; Wed, 04 May 2011 21:42:35 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 0234D102C36; Wed, 4 May 2011 14:42:34 -0700 (PDT) Date: Wed, 4 May 2011 14:42:33 -0700 From: Jeremy Chadwick To: "Mikhail T." Message-ID: <20110504214233.GA97428@icarus.home.lan> References: <4DC1A1CC.4080901@aldan.algebra.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4DC1A1CC.4080901@aldan.algebra.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: syslogd spinning the CPU, not logging... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 21:42:37 -0000 On Wed, May 04, 2011 at 02:58:20PM -0400, Mikhail T. wrote: > On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this > problem the first time (a week or two ago), but just saw it again. > Something triggers syslogd into spinning all available CPU -- while > not logging anything. > > Attempts to log anything -- such as by invoking logger(1) -- simply > hang. sendmail hangs too -- now new e-mail is coming. It is not > pretty. > > ktrace-ing the process yields empty file -- it is not doing > system-calls, while spinning. > > Attaching the debugger several times shows the same stack: > > #0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #2 0x00000008007d4825 in free () from /lib/libc.so.7 > #3 0x00000008007c5aab in catopen () from /lib/libc.so.7 > #4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7 > #5 0x00000008007c556d in strerror () from /lib/libc.so.7 > #6 0x0000000000403c10 in ?? () > ... > > Among the logging-destinations in my syslog.conf there is a program, > that exits sometimes -- but never "too fast". Normally, syslogd > simply restarts it without an issue... > > Any ideas? Thanks! If ktrace doesn't show any syscalls (did you use "ktrace -i -t+"?), then I'm not sure how the process is actually spinning (100% CPU). "procstat -kk" on the PID of syslogd might be more useful in this case. The output is mainly useful for kernel folks. You may end up having to drop to ddb to examine what's going on within the pid. I'm not sure if attaching gdb to a running process on FreeBSD works with things like threads, etc.. Meaning I'm not sure if the above stack trace is indicative of a problem or not. No offence to anyone intended, but I try to avoid gdb on FreeBSD wherever possible (it's so old that I don't know how useful it is for extreme debugging). I've never seen syslogd spiral out of control on any of our systems, but all logging destinations in syslog.conf on our systems are files on local disks. If you can find a way to reliably reproduce the issue that would be useful, otherwise this sounds like one of those things that's going to be a problem to track down. :-( -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed May 4 23:26:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AAED106564A; Wed, 4 May 2011 23:26:00 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 21A718FC12; Wed, 4 May 2011 23:25:59 +0000 (UTC) Received: by iyj12 with SMTP id 12so1943013iyj.13 for ; Wed, 04 May 2011 16:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=XXhzHaM0Fzp+/PPWuWNU/yUAFiuRKj4GQdWQ6+WDo+0=; b=o9jKYceutTdFcu75ODr9oUUBfAbUlG/481k6fbVW7yhig8jvoUQ9jIu/wWGXMKVyfj j9dXB1rNbMCcbG3p5EljPpLFOedXbNTidLw6nKGESVBebflDmVJkHS9lRI28Fkf9KC9Q flT5c9dxwgwPMRAlY8H7Bl8G01jyK+EvGpJKI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=W5qK8A3/S6TjeDlUMrrVIRzH2MARRKyfFy2qjnd6BGJDzJDoHS5SHmVM40z9DaBBWz HXKIRJwgUjm/rJzQApIEoxDkW5hBDC0L0dECmTAPNO/uikHh7SCXY4NMY5dWkWV8j89m gt0CEN2YdGvl0+CF4DsxbnXOVpmi1y2Vr0BMI= Received: by 10.43.60.200 with SMTP id wt8mr24971icb.358.1304551559432; Wed, 04 May 2011 16:25:59 -0700 (PDT) Received: from DataIX.net ([99.190.84.116]) by mx.google.com with ESMTPS id 14sm641106ibc.25.2011.05.04.16.25.57 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 04 May 2011 16:25:58 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p44NPr2W031030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 4 May 2011 19:25:54 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p44NPqeP031029; Wed, 4 May 2011 19:25:52 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Wed, 4 May 2011 19:25:52 -0400 From: Jason Hellenthal To: Doug Barton Message-ID: <20110504232552.GB6943@DataIX.net> References: <4DC13D32.1070503@beasties.demon.nl> <4DC1A849.70807@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GRPZ8SYKNexpdSJ7" Content-Disposition: inline In-Reply-To: <4DC1A849.70807@FreeBSD.org> X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-stable@freebsd.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 23:26:00 -0000 --GRPZ8SYKNexpdSJ7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Doug, On Wed, May 04, 2011 at 12:26:01PM -0700, Doug Barton wrote: >On 05/04/2011 04:49, Hans Ottevanger wrote: >> Hi, >> >> I upgraded my Soekris 4801 boxes from 8.1 to 8.2-STABLE (r221326) a few >> days ago and now I get the following error in the daily mail: >> >> Backing up package db directory: >> tar: : Cannot stat: No such file or directory >> tar: Error exit delayed from previous errors. >> >> These messages originate from /etc/periodic/daily/220.backup-pkgdb, >> apparently a recent addition. >> >> The culprit is probably on line 21: >> >> make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR >> "/usr/share/mk/bsd.port.mk", line 11: Could not find >> /usr/ports/Mk/bsd.port.mk >> make: fatal errors encountered -- cannot continue > >Thanks, I'll take a look. > Move PKG_DBDIR out of ports(7) and/or duplicate it to=20 /usr/share/mk/bsd.port.mk. Seems this could be used for far off more things than when ports(7) is=20 unpacked to its usual location. --=20 Regards, (jhell) Jason Hellenthal --GRPZ8SYKNexpdSJ7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNweB/AAoJEJBXh4mJ2FR+F1cIAJKv1bdIseh4hRSe/+GY2aap qE7dXFncq9bTnZwg5T7xu1xgUKgCh3A3frJr5E2Y02/ZWAPwa1zczGZFXIqDVwFH HVJU8yLR6R+FZn66Q4WKgjm5B6EcUHle3SfXyEeOp3pEmXGrhhnx24l3yOLcd46J 9ul8sTq9LMX4WIDz2k4YokDALSBD8Rj03EKuURhCF7ou/d2iajT4JNhddXtVg4BW XWRXiJ9G9hUDAupxPWcJlteBQHaKskMC31BT4E68DtNQCQOz9gg6KGH3h3OVHo4V brUfFf0ynmqx8sHfR6yudHiIQo7q+yVEQXVmdDvAZHGzGf8VpRROdKLIW2JXbvA= =09tS -----END PGP SIGNATURE----- --GRPZ8SYKNexpdSJ7-- From owner-freebsd-stable@FreeBSD.ORG Thu May 5 02:37:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 59ECE106564A for ; Thu, 5 May 2011 02:37:51 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 65-241-43-5.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 6717B1519D6; Thu, 5 May 2011 02:37:50 +0000 (UTC) Message-ID: <4DC20D7C.9040203@FreeBSD.org> Date: Wed, 04 May 2011 19:37:48 -0700 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Jason Hellenthal References: <4DC13D32.1070503@beasties.demon.nl> <4DC1A849.70807@FreeBSD.org> <20110504232552.GB6943@DataIX.net> In-Reply-To: <20110504232552.GB6943@DataIX.net> X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 02:37:51 -0000 On 05/04/2011 16:25, Jason Hellenthal wrote: > Move PKG_DBDIR out of ports(7) and/or duplicate it to > /usr/share/mk/bsd.port.mk. A) That's a non-starter B) Doesn't actually solve the problem at hand -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu May 5 02:44:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id ED23F106564A for ; Thu, 5 May 2011 02:44:09 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 65-241-43-5.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 2FED815F3BA; Thu, 5 May 2011 02:43:55 +0000 (UTC) Message-ID: <4DC20EEA.5000703@FreeBSD.org> Date: Wed, 04 May 2011 19:43:54 -0700 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Hans Ottevanger References: <4DC13D32.1070503@beasties.demon.nl> In-Reply-To: <4DC13D32.1070503@beasties.demon.nl> X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 02:44:10 -0000 On 05/04/2011 04:49, Hans Ottevanger wrote: > make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR > "/usr/share/mk/bsd.port.mk", line 11: Could not find > /usr/ports/Mk/bsd.port.mk > make: fatal errors encountered -- cannot continue I fixed this in HEAD by setting the default if pulling it from make fails. I will MFC ASAP. -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu May 5 03:42:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BAB51065673; Thu, 5 May 2011 03:42:44 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 204A38FC17; Thu, 5 May 2011 03:42:43 +0000 (UTC) Received: by iwn33 with SMTP id 33so2094061iwn.13 for ; Wed, 04 May 2011 20:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=jXG1lvgUYAwb/fCpZEDWzrKk2rCyhAmuLHiCGEOuX/M=; b=f8V9J5XT1J2yTbOAoV+xMauChgghD8hq1OaOtkwCXSAq7FRqVn+mFiOaDtscgXlV6K 54Up+IgSLqIIzugX3UfrZk4Cgjoud/BlCI8WaPLkzr1KxqkhkZCH1jlXrLbTm2MEuAfy KjKo8B0yKwkkooUux1bu61qy4Yj/BpuZdVP1I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=ERTKtXQywO726olTd9ty/kWeW6d/7LBB8JOzbrA/yfjh+wPvxaMBBcK3QnSM/fZc4U a32OBf3SJMRBQKtAbY6jmEnWMFm8QNIdrhNkSEqzMZelaGhayOXV6QPjYfPM2/HR/kMV XrCX7sAVfup6wIaBtZpY7m5HcLnqtFLU/5IMo= Received: by 10.42.200.14 with SMTP id eu14mr240041icb.481.1304566963347; Wed, 04 May 2011 20:42:43 -0700 (PDT) Received: from DataIX.net (adsl-99-190-84-116.dsl.klmzmi.sbcglobal.net [99.190.84.116]) by mx.google.com with ESMTPS id y10sm726607iba.46.2011.05.04.20.42.41 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 04 May 2011 20:42:42 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p453gbDw056864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 4 May 2011 23:42:38 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p453gaJt056863; Wed, 4 May 2011 23:42:36 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Wed, 4 May 2011 23:42:36 -0400 From: Jason Hellenthal To: Doug Barton Message-ID: <20110505034235.GC6943@DataIX.net> References: <4DC13D32.1070503@beasties.demon.nl> <4DC1A849.70807@FreeBSD.org> <20110504232552.GB6943@DataIX.net> <4DC20D7C.9040203@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="eRtJSFbw+EEWtPj3" Content-Disposition: inline In-Reply-To: <4DC20D7C.9040203@FreeBSD.org> X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-stable@freebsd.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 03:42:44 -0000 --eRtJSFbw+EEWtPj3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Doug, On Wed, May 04, 2011 at 07:37:48PM -0700, Doug Barton wrote: >On 05/04/2011 16:25, Jason Hellenthal wrote: >> Move PKG_DBDIR out of ports(7) and/or duplicate it to >> /usr/share/mk/bsd.port.mk. > >A) That's a non-starter >B) Doesn't actually solve the problem at hand Considering the PKG_DBDIR doesn't require ports being installed I fail to= =20 see why duplicating it to a location that's system-wide would be a problem.= =20 If that is a problem then maybe a global environment file should be put=20 into place that will export that variable directly to user-land for=20 consumption. FTP_PASSIVE_MODE is already set via login.conf. Would there be any harm in= =20 exporting one more for PKG_DBDIR instead of having that in the Make files= =20 at all ? I think something a little more solid should become of this variable than= =20 whats in place now. --=20 Regards, (jhell) Jason Hellenthal --eRtJSFbw+EEWtPj3 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNwhyrAAoJEJBXh4mJ2FR+Ot8IAJ+ii7O4ZI2EggAtXFpO7OWt ciDadHMoJbCJWGdp8li9ckfmQwOAOWPjyMM/5+PTC14g63wvSjSDYXcw7oPAEE1f nLG7p8FhsjepjGer/xFK+rwuKmFBj/aSX1DcznrGFgBlzqdvOQvHLMUbuqoKBTVM W46XfUC5wm7REnQxfaGpvR5aFFncibEzmCHmhZnNySjtetX0MIBHnJi+Vv80HdBq VrU8jZxps+MlUS9E0EyQpBpfTFStxm9U9VDOsgWwdw75FnsZUY/0lq4NuuNHYYIT IcKF76jpr0mMXEELHj61squdaNdT6u72dOsPeMytkeZb+BjLhhQeSzhxd6Osw1Y= =ydO0 -----END PGP SIGNATURE----- --eRtJSFbw+EEWtPj3-- From owner-freebsd-stable@FreeBSD.ORG Thu May 5 06:57:04 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD01E106566C for ; Thu, 5 May 2011 06:57:04 +0000 (UTC) (envelope-from fbsdstable@beasties.demon.nl) Received: from smtp-vbr14.xs4all.nl (smtp-vbr14.xs4all.nl [194.109.24.34]) by mx1.freebsd.org (Postfix) with ESMTP id 40BEE8FC1D for ; Thu, 5 May 2011 06:57:03 +0000 (UTC) Received: from merom.hotsoft.nl (beasties.demon.nl [82.161.3.114]) by smtp-vbr14.xs4all.nl (8.13.8/8.13.8) with ESMTP id p456uWlB052494; Thu, 5 May 2011 08:56:32 +0200 (CEST) (envelope-from fbsdstable@beasties.demon.nl) Message-ID: <4DC24A20.5050702@beasties.demon.nl> Date: Thu, 05 May 2011 08:56:32 +0200 From: Hans Ottevanger User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Doug Barton References: <4DC13D32.1070503@beasties.demon.nl> <4DC20EEA.5000703@FreeBSD.org> In-Reply-To: <4DC20EEA.5000703@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by XS4ALL Virus Scanner Cc: freebsd-stable@FreeBSD.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 06:57:04 -0000 On 05/05/11 04:43, Doug Barton wrote: > On 05/04/2011 04:49, Hans Ottevanger wrote: >> make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR >> "/usr/share/mk/bsd.port.mk", line 11: Could not find >> /usr/ports/Mk/bsd.port.mk >> make: fatal errors encountered -- cannot continue > > I fixed this in HEAD by setting the default if pulling it from make > fails. I will MFC ASAP. > > Of course this will solve my "problem" 8-) But if you use something like pkg_dbdir=${PKG_DBDIR-/var/db/pkg} you will also cover the (infrequent) case where people redefine PKG_DBDIR while running pkg_add et al (and actually remember to set PKG_DBDIR in /etc/crontab !). Kind regards, Hans Ottevanger From owner-freebsd-stable@FreeBSD.ORG Thu May 5 07:09:43 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id D97361065670 for ; Thu, 5 May 2011 07:09:43 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 65-241-43-5.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id D5BC3157684; Thu, 5 May 2011 07:09:07 +0000 (UTC) Message-ID: <4DC24D12.1040306@FreeBSD.org> Date: Thu, 05 May 2011 00:09:06 -0700 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Hans Ottevanger References: <4DC13D32.1070503@beasties.demon.nl> <4DC20EEA.5000703@FreeBSD.org> <4DC24A20.5050702@beasties.demon.nl> In-Reply-To: <4DC24A20.5050702@beasties.demon.nl> X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 07:09:43 -0000 On 05/04/2011 23:56, Hans Ottevanger wrote: > On 05/05/11 04:43, Doug Barton wrote: >> On 05/04/2011 04:49, Hans Ottevanger wrote: >>> make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR >>> "/usr/share/mk/bsd.port.mk", line 11: Could not find >>> /usr/ports/Mk/bsd.port.mk >>> make: fatal errors encountered -- cannot continue >> >> I fixed this in HEAD by setting the default if pulling it from make >> fails. I will MFC ASAP. >> >> > > Of course this will solve my "problem" 8-) > > But if you use something like > > pkg_dbdir=${PKG_DBDIR-/var/db/pkg} > > you will also cover the (infrequent) case where people redefine > PKG_DBDIR while running pkg_add et al (and actually remember to set > PKG_DBDIR in /etc/crontab !). To be honest, I don't care. If you are doing this kind of thing you deserve what you get. If you really want to move your PKG_DBDIR and can't be bothered to define it correctly, use a symlink. Or, if that's a problem, disable the backup. This "problem" is not going to affect the overwhelming number of freebsd users, and I don't think going through a lot of gymnastics to support people who do silly things is a good idea for either side. From owner-freebsd-stable@FreeBSD.ORG Thu May 5 07:57:48 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1BEA1065672; Thu, 5 May 2011 07:57:48 +0000 (UTC) (envelope-from fbsdstable@beasties.demon.nl) Received: from smtp-vbr13.xs4all.nl (smtp-vbr13.xs4all.nl [194.109.24.33]) by mx1.freebsd.org (Postfix) with ESMTP id 78CE08FC18; Thu, 5 May 2011 07:57:47 +0000 (UTC) Received: from merom.hotsoft.nl (beasties.demon.nl [82.161.3.114]) by smtp-vbr13.xs4all.nl (8.13.8/8.13.8) with ESMTP id p457vGtL072908; Thu, 5 May 2011 09:57:16 +0200 (CEST) (envelope-from fbsdstable@beasties.demon.nl) Message-ID: <4DC2585C.6090207@beasties.demon.nl> Date: Thu, 05 May 2011 09:57:16 +0200 From: Hans Ottevanger User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110429 Thunderbird/3.1.10 MIME-Version: 1.0 To: Doug Barton References: <4DC13D32.1070503@beasties.demon.nl> <4DC20EEA.5000703@FreeBSD.org> <4DC24A20.5050702@beasties.demon.nl> <4DC24D12.1040306@FreeBSD.org> In-Reply-To: <4DC24D12.1040306@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by XS4ALL Virus Scanner Cc: freebsd-stable@FreeBSD.org Subject: Re: Daily backups of pkgdb failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 07:57:49 -0000 On 05/05/11 09:09, Doug Barton wrote: > On 05/04/2011 23:56, Hans Ottevanger wrote: >> On 05/05/11 04:43, Doug Barton wrote: >>> On 05/04/2011 04:49, Hans Ottevanger wrote: >>>> make -f/usr/share/mk/bsd.port.mk -V PKG_DBDIR >>>> "/usr/share/mk/bsd.port.mk", line 11: Could not find >>>> /usr/ports/Mk/bsd.port.mk >>>> make: fatal errors encountered -- cannot continue >>> >>> I fixed this in HEAD by setting the default if pulling it from make >>> fails. I will MFC ASAP. >>> >>> >> >> Of course this will solve my "problem" 8-) >> >> But if you use something like >> >> pkg_dbdir=${PKG_DBDIR-/var/db/pkg} >> >> you will also cover the (infrequent) case where people redefine >> PKG_DBDIR while running pkg_add et al (and actually remember to set >> PKG_DBDIR in /etc/crontab !). > > To be honest, I don't care. If you are doing this kind of thing you > deserve what you get. > To be clear. it is OK for me, I use defaults wherever possible. > If you really want to move your PKG_DBDIR and can't be bothered to > define it correctly, use a symlink. Or, if that's a problem, disable the > backup. This "problem" is not going to affect the overwhelming number of > freebsd users, and I don't think going through a lot of gymnastics to > support people who do silly things is a good idea for either side. > My idea was that if you go through the trouble to use make to get PKG_DBDIR from the ports (as possibly redefined in /etc/make.conf) you might as well try something similar while using packages. But again: every deliberate choice is OK for me. Regards, Hans From owner-freebsd-stable@FreeBSD.ORG Thu May 5 09:36:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EA6C106566C for ; Thu, 5 May 2011 09:36:01 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from fish.ish.com.au (eth5921.nsw.adsl.internode.on.net [59.167.240.32]) by mx1.freebsd.org (Postfix) with ESMTP id 5D7598FC1B for ; Thu, 5 May 2011 09:36:01 +0000 (UTC) Received: from ip-136.ish.com.au ([203.29.62.136]:50829) by fish.ish.com.au with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.69) (envelope-from ) id 1QHunn-00059O-1i; Thu, 05 May 2011 19:24:55 +1000 X-CTCH-RefID: str=0001.0A150203.4DC26CE7.0131:SCFSTAT13512334,ss=1,fgs=0 Message-ID: <4DC26CE7.3070300@ish.com.au> Date: Thu, 05 May 2011 19:24:55 +1000 From: Aristedes Maniatis User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b13pre) Gecko/20110314 Thunderbird/3.3a3 MIME-Version: 1.0 To: Jeff Blank References: <20110502164222.GE81300@mr-happy.com> In-Reply-To: <20110502164222.GE81300@mr-happy.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 09:36:01 -0000 On 3/05/11 2:42 AM, Jeff Blank wrote: > Hi, > > I recently upgraded from 8.0-STABLE to 8.2-STABLE (Apr. 29 checkout) > and upgraded my zpool (includes root FS) from v13 to v15. This is a > dual-boot laptop, so I'm using MBR/boot0 and not GPT. Here's what > happens when I boot: > > F1 Win > F2 ? > F3 FreeBSD > > F6 PXE > Boot: F3 > ZFS: unsupported ZFS version 15 (should be 13) > No ZFS pools located, can't boot > > I've googled around, but I can't find anything relevant for MBR/boot0 > configurations, just GPT. I've ensured that the loaders and > boot0/boot1/boot2 are all new, and I rebuilt/reinstalled them in a > fixit environment just to be sure. I also ran 'boot0cfg -B' (with an > appropriate -b), but nothing has changed. How can I get my pool > booting again? Not only do you have to get the boot loaders installed properly [1] but also there is a breakage in the FreeBSD 8.2-RELEASE code [2]. The MBR bootloader is broken in 8.2 and will not work with ZFS under at least some circumstances (2 of our boxes had the problem). The problem has been reported on the freebsd-fs list and I notice a fix has gone into svn for the 8-STABLE branch. You need to get a bootloader from 8-CURRENT or convert your partitions over to GPT if you hit that particular bug. But you aren't up to hitting that bug yet... you haven't installed the newer bootloader at the point you are up to. Ari [1] [2] http://www.freebsd.org/cgi/query-pr.cgi?pr=153552 -- --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A From owner-freebsd-stable@FreeBSD.ORG Thu May 5 09:51:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A29121065674 for ; Thu, 5 May 2011 09:51:01 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from fish.ish.com.au (eth5921.nsw.adsl.internode.on.net [59.167.240.32]) by mx1.freebsd.org (Postfix) with ESMTP id 615688FC17 for ; Thu, 5 May 2011 09:51:00 +0000 (UTC) Received: from ip-136.ish.com.au ([203.29.62.136]:51046) by fish.ish.com.au with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.69) (envelope-from ) id 1QHvCs-0003Tn-2L; Thu, 05 May 2011 19:50:51 +1000 X-CTCH-RefID: str=0001.0A150201.4DC272FA.01AA:SCFSTAT13512334,ss=1,fgs=0 Message-ID: <4DC272FA.2000201@ish.com.au> Date: Thu, 05 May 2011 19:50:50 +1000 From: Aristedes Maniatis User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b13pre) Gecko/20110314 Thunderbird/3.3a3 MIME-Version: 1.0 To: Jeff Blank References: <20110502164222.GE81300@mr-happy.com> <4DC26CE7.3070300@ish.com.au> In-Reply-To: <4DC26CE7.3070300@ish.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable Subject: Re: zpool upgrade, can't boot X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 09:51:01 -0000 On 5/05/11 7:24 PM, Aristedes Maniatis wrote: > > Not only do you have to get the boot loaders installed properly [1] but also there is a breakage in the FreeBSD 8.2-RELEASE code [2]. The MBR bootloader is broken in 8.2 and will not work with ZFS under at least some circumstances (2 of our boxes had the problem). The problem has been reported on the freebsd-fs list and I notice a fix has gone into svn for the 8-STABLE branch. > > You need to get a bootloader from 8-CURRENT or convert your partitions over to GPT if you hit that particular bug. But you aren't up to hitting that bug yet... you haven't installed the newer bootloader at the point you are up to. > > Ari > > > [1] > [2] http://www.freebsd.org/cgi/query-pr.cgi?pr=153552 Sorry, [1] should be http://www.ish.com.au/node/1434 That has some useful pointers for installing the zfsboot onto MBR disks, but I'm sure you can find similar information in other places (just not in the FreeBSD handbooks unforuntately) Ari -- --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A From owner-freebsd-stable@FreeBSD.ORG Thu May 5 13:35:52 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D935D106566C for ; Thu, 5 May 2011 13:35:52 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from megatron.madpilot.net (megatron.madpilot.net [88.149.173.206]) by mx1.freebsd.org (Postfix) with ESMTP id 342A48FC12 for ; Thu, 5 May 2011 13:35:50 +0000 (UTC) Received: from megatron.madpilot.net (localhost [127.0.0.1]) by megatron.madpilot.net (Postfix) with ESMTP id 5F19013D7 for ; Thu, 5 May 2011 15:17:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= user-agent:content-transfer-encoding:content-disposition :content-type:content-type:mime-version:message-id:subject :subject:from:from:date:date:received:received; s=mail; t= 1304601456; x=1306415856; bh=EplKjbW5hMmVkuBEqEjqlG26T/8dyyHciAf VsLoJe8c=; b=nNcaMY9epQcCkRpFNmI6xXd4XJZrtl1C3PShs3qnzkz2jDkPaDY ywR4vU8vrNWYL97Rg4rb/+wxQgMqaIV/aAIPJFZZ8JytUMvYfv/QiAUoJto63ffE qZQAMyp7FnDX8tls1IpgeuhHtamfYIyBX7g6s8nV6Dk9is4QWgJMB0p0= X-Virus-Scanned: amavisd-new at madpilot.net Received: from megatron.madpilot.net ([127.0.0.1]) by megatron.madpilot.net (megatron.madpilot.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5qgsuTY7qFyD for ; Thu, 5 May 2011 15:17:36 +0200 (CEST) Received: by megatron.madpilot.net (Postfix, from userid 1000) id 9115F13C6; Thu, 5 May 2011 15:17:36 +0200 (CEST) Date: Thu, 5 May 2011 15:17:36 +0200 From: Guido Falsi To: freebsd-stable@freebsd.org Message-ID: <20110505131736.GE29667@megatron.madpilot.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="Fba/0zbH8Xs+Fj9o" Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Operating-System: FreeBSD 8.2-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Xorg lockup in drmwtq state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 13:35:52 -0000 --Fba/0zbH8Xs+Fj9o Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi! In the last week I have started to see the situation described in the subject several times. I'd really appreciate some help about this issue, since I don't know where to look at to better diagnose it. The screen locks up, but the mouse cursor still moves. I can log in via ssh and from top I see this line for Xorg: 1563 root 1 44 0 388M 326M drmwtq 1 5:05 0.00% Xorg the last line from Xorg.0.log reads: [mi] EQ overflowing. The server is probably stuck in an infinite loop. I updated the system on Apr 28 just after cvsupping the latest 8-stable. I also upgraded to Firefox 4 on that day. I mention Firefox 4 because these lockups are always happening while using FF4. It also looks like thay are happening when I use FF4 while I have at least one VirtualBox machine running. I know this sounds crazy, but this is what I'm seeing. I'm trying to do some more testing but I can't reproduce this with just FF4 or just VirtualBox running. This machine has intel graphics: vgapci0: port 0x1210-0x1217 mem 0xf0100000-0xf017ffff,0xe0000000-0xefffffff,0xf0000000-0xf00fffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: aperture size is 256M, detected 6140k stolen memory drm0: on vgapci0 Up to March 10 this was not happening.(I have not used this machine in the days between March 10 and Apr 28, I'm sorry I can't really narrow it down any better) Putting blame on VirtualBox I disabled 3D acceleration in all virtual machines, but this did not solve nor mitigate the problem. Thank in advance for any help! I'm attaching dmesg, xorg.conf and Xorg.0.log. -- Guido Falsi --Fba/0zbH8Xs+Fj9o Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg.boot" Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #1: Thu Apr 28 14:19:08 CEST 2011 root@hostname:/usr/obj/usr/src/sys/HOST amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz (2327.51-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3fd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 8594128896 (8196 MB) avail memory = 8165249024 (7786 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 1 ioapic0 irqs 0-23 on motherboard netsmb_dev: loaded kbd1 at kbdmux0 cryptosoft0: on motherboard aesni0: No AESNI support. acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf808-0xf80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0x1210-0x1217 mem 0xf0100000-0xf017ffff,0xe0000000-0xefffffff,0xf0000000-0xf00fffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: aperture size is 256M, detected 6140k stolen memory drm0: on vgapci0 info: [drm] MSI enabled 1 message(s) info: [drm] AGP at 0xe0000000 256MB info: [drm] Initialized i915 1.6.0 20080730 pci0: at device 3.0 (no driver attached) pci0: at device 3.2 (no driver attached) pci0: at device 3.3 (no driver attached) em0: port 0x1100-0x111f mem 0xf0180000-0xf019ffff,0xf01a5000-0xf01a5fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1e:0b:ab:e6:a7 uhci0: port 0x1120-0x113f irq 20 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0x1140-0x115f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x1160-0x117f irq 22 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xf01a6800-0xf01a6bff irq 22 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 hdac0: mem 0xf01a0000-0xf01a3fff irq 21 at device 27.0 on pci0 hdac0: HDA Driver Revision: 20100226_0142 hdac0: [ITHREAD] pcib1: at device 28.0 on pci0 pci32: on pcib1 pcib2: irq 21 at device 28.1 on pci0 pci48: on pcib2 uhci3: port 0x1180-0x119f irq 20 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 uhci4: port 0x11a0-0x11bf irq 21 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: on uhci4 ehci1: mem 0xf01a6c00-0xf01a6fff irq 20 at device 29.7 on pci0 ehci1: [ITHREAD] usbus6: EHCI version 1.0 usbus6: on ehci1 pcib3: at device 30.0 on pci0 pci7: on pcib3 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x1230-0x1237,0x1248-0x124b,0x1238-0x123f,0x124c-0x124f,0x11c0-0x11df mem 0xf01a6000-0xf01a67ff irq 18 at device 31.2 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: at channel 4 on ahci0 ahcich2: [ITHREAD] ahcich3: at channel 5 on ahci0 ahcich3: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: port 0x378-0x37f,0x778-0x77d irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/13 bytes threshold ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] acpi_hpet1: iomem 0xfed00000-0xfed003ff on acpi0 device_attach: acpi_hpet1 attach returned 12 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 ZFS filesystem version 4 ZFS storage pool version 15 Timecounters tick every 1.000 msec vboxdrv: fAsync=0 offMin=0x10a offMax=0x4b4 hdac0: HDA Codec #0: Analog Devices AD1884 pcm0: at cad 0 nid 1 on hdac0 pcm1: at cad 0 nid 1 on hdac0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-7 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 ada1: ATA-7 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada1: Command Queueing enabled ada1: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) cd0 at ahcich1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #1 Launched! uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Root mount waiting for: usbus6 usbus3 uhub6: 4 ports with 4 removable, self powered uhub3: 6 ports with 6 removable, self powered Root mount waiting for: usbus3 ugen3.2: at usbus3 umass0: on usbus3 da0 at umass-sim0 bus 0 scbus4 target 0 lun 0 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: 7663MB (15695871 512 byte sectors: 255H 63S/T 977C) Root mount waiting for: usbus3 Root mount waiting for: usbus3 Trying to mount root from zfs:tank ugen2.2: at usbus2 ums0: on usbus2 ums0: 16 buttons and [XYZT] coordinates ID=0 uhid0: on usbus2 ugen1.2: at usbus1 ubt0: on usbus1 vboxnet0: Ethernet address: 0a:00:27:00:00:00 fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8 drm0: [ITHREAD] <<111188>>.T e Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 0 0 0 0 0 done All buffers synced. Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #1: Thu Apr 28 14:19:08 CEST 2011 root@hostname:/usr/obj/usr/src/sys/HOST amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz (2327.52-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3fd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 8594128896 (8196 MB) avail memory = 8165249024 (7786 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 1 ioapic0 irqs 0-23 on motherboard netsmb_dev: loaded kbd1 at kbdmux0 cryptosoft0: on motherboard aesni0: No AESNI support. acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf808-0xf80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0x1210-0x1217 mem 0xf0100000-0xf017ffff,0xe0000000-0xefffffff,0xf0000000-0xf00fffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: aperture size is 256M, detected 6140k stolen memory drm0: on vgapci0 info: [drm] MSI enabled 1 message(s) info: [drm] AGP at 0xe0000000 256MB info: [drm] Initialized i915 1.6.0 20080730 pci0: at device 3.0 (no driver attached) pci0: at device 3.2 (no driver attached) pci0: at device 3.3 (no driver attached) em0: port 0x1100-0x111f mem 0xf0180000-0xf019ffff,0xf01a5000-0xf01a5fff irq 19 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1e:0b:ab:e6:a7 uhci0: port 0x1120-0x113f irq 20 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0x1140-0x115f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x1160-0x117f irq 22 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xf01a6800-0xf01a6bff irq 22 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 hdac0: mem 0xf01a0000-0xf01a3fff irq 21 at device 27.0 on pci0 hdac0: HDA Driver Revision: 20100226_0142 hdac0: [ITHREAD] pcib1: at device 28.0 on pci0 pci32: on pcib1 pcib2: irq 21 at device 28.1 on pci0 pci48: on pcib2 uhci3: port 0x1180-0x119f irq 20 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 uhci4: port 0x11a0-0x11bf irq 21 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: on uhci4 ehci1: mem 0xf01a6c00-0xf01a6fff irq 20 at device 29.7 on pci0 ehci1: [ITHREAD] usbus6: EHCI version 1.0 usbus6: on ehci1 pcib3: at device 30.0 on pci0 pci7: on pcib3 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x1230-0x1237,0x1248-0x124b,0x1238-0x123f,0x124c-0x124f,0x11c0-0x11df mem 0xf01a6000-0xf01a67ff irq 18 at device 31.2 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: at channel 4 on ahci0 ahcich2: [ITHREAD] ahcich3: at channel 5 on ahci0 ahcich3: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: port 0x378-0x37f,0x778-0x77d irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/13 bytes threshold ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] acpi_hpet1: iomem 0xfed00000-0xfed003ff on acpi0 device_attach: acpi_hpet1 attach returned 12 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 ZFS filesystem version 4 ZFS storage pool version 15 Timecounters tick every 1.000 msec vboxdrv: fAsync=0 offMin=0x103 offMax=0x413 hdac0: HDA Codec #0: Analog Devices AD1884 pcm0: at cad 0 nid 1 on hdac0 pcm1: at cad 0 nid 1 on hdac0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-7 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 ada1: ATA-7 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada1: Command Queueing enabled ada1: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) cd0 at ahcich1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #1 Launched! Root mount waiting for: usbus6 usbus5 usbus4 usbus3 usbus2 usbus1 usbus0 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Root mount waiting for: usbus6 usbus3 uhub6: 4 ports with 4 removable, self powered uhub3: 6 ports with 6 removable, self powered Root mount waiting for: usbus3 ugen3.2: at usbus3 umass0: on usbus3 Root mount waiting for: usbus3 da0 at umass-sim0 bus 0 scbus4 target 0 lun 0 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: 7663MB (15695871 512 byte sectors: 255H 63S/T 977C) Root mount waiting for: usbus3 Trying to mount root from zfs:tank ugen2.2: at usbus2 ums0: on usbus2 ums0: 16 buttons and [XYZT] coordinates ID=0 uhid0: on usbus2 ugen1.2: at usbus1 ubt0: on usbus1 vboxnet0: Ethernet address: 0a:00:27:00:00:00 --Fba/0zbH8Xs+Fj9o Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="xorg.conf" Section "Module" Load "dbe" # Double buffer extension Load "extmod" Load "type1" Load "freetype" # Load "xtt" Load "glx" # Load "glx2" Load "dri" Disable "dri2" EndSection #Section "Extensions" # Option "Composite" "enable" #EndSection Section "Files" FontPath "/usr/local/lib/X11/fonts/misc/" FontPath "/usr/local/lib/X11/fonts/TTF/" FontPath "/usr/local/lib/X11/fonts/OTF" FontPath "/usr/local/lib/X11/fonts/Type1/" FontPath "/usr/local/lib/X11/fonts/100dpi/" FontPath "/usr/local/lib/X11/fonts/75dpi/" FontPath "/usr/local/lib/X11/fonts/local/" FontPath "/usr/local/lib/X11/fonts/dejavu/" FontPath "/usr/local/lib/X11/fonts/bitstream-vera/" FontPath "/usr/local/lib/X11/fonts/Liberation/" EndSection Section "ServerFlags" Option "DontZap" "false" EndSection Section "InputDevice" Identifier "Keyboard1" Driver "kbd" Option "AutoRepeat" "500 30" Option "XkbRules" "xorg" Option "XkbModel" "pc104" Option "XkbLayout" "us" Option "XkbOptions" "terminate:ctrl_alt_bksp" EndSection Section "InputDevice" Identifier "Mouse1" Driver "mouse" Option "Protocol" "Auto" Option "Device" "/dev/sysmouse" Option "ZAxisMapping" "4 5 6 7" EndSection Section "Monitor" Identifier "LCD1" Option "DPMS" EndSection Section "Device" Identifier "Standard VGA" VendorName "Unknown" BoardName "Unknown" Driver "vga" EndSection Section "Device" Identifier "internal VGA" Driver "intel" EndSection Section "Screen" Identifier "Screen 1" Device "internal VGA" Monitor "LCD1" DefaultDepth 24 Subsection "Display" Depth 8 ViewPort 0 0 EndSubsection Subsection "Display" Depth 16 ViewPort 0 0 EndSubsection Subsection "Display" Depth 24 ViewPort 0 0 EndSubsection Subsection "Display" Depth 32 ViewPort 0 0 EndSubsection EndSection Section "ServerLayout" Identifier "Simple Layout" Screen "Screen 1" InputDevice "Mouse1" "CorePointer" InputDevice "Keyboard1" "CoreKeyboard" EndSection Section "DRI" Mode 0666 EndSection --Fba/0zbH8Xs+Fj9o Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: attachment; filename="Xorg.0.log" Content-Transfer-Encoding: 8bit _XSERVTransSocketOpenCOTSServer: Unable to open socket for inet6 _XSERVTransOpen: transport open failed for inet6/hostname:0 _XSERVTransMakeAllCOTSServerListeners: failed to open listener for inet6 X.Org X Server 1.7.7 Release Date: 2010-05-04 X Protocol Version 11, Revision 0 Build Operating System: FreeBSD 8.2-STABLE amd64 Current Operating System: FreeBSD hostname 8.2-STABLE FreeBSD 8.2-STABLE #1: Thu Apr 28 14:19:08 CEST 2011 root@hostname:/usr/obj/usr/src/sys/HOST amd64 Build Date: 05 May 2011 09:52:08AM Current version of pixman: 0.21.4 Before reporting problems, check http://wiki.x.org to make sure that you have the latest version. Markers: (--) probed, (**) from config file, (==) default setting, (++) from command line, (!!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (??) unknown. (==) Log file: "/var/log/Xorg.0.log", Time: Thu May 5 13:11:19 2011 (==) Using config file: "/etc/X11/xorg.conf" (==) ServerLayout "Simple Layout" (**) |-->Screen "Screen 1" (0) (**) | |-->Monitor "LCD1" (**) | |-->Device "internal VGA" (**) |-->Input Device "Mouse1" (**) |-->Input Device "Keyboard1" (**) Option "DontZap" "false" (==) Automatically adding devices (==) Automatically enabling devices (**) FontPath set to: /usr/local/lib/X11/fonts/misc/, /usr/local/lib/X11/fonts/TTF/, /usr/local/lib/X11/fonts/OTF, /usr/local/lib/X11/fonts/Type1/, /usr/local/lib/X11/fonts/100dpi/, /usr/local/lib/X11/fonts/75dpi/, /usr/local/lib/X11/fonts/local/, /usr/local/lib/X11/fonts/dejavu/, /usr/local/lib/X11/fonts/bitstream-vera/, /usr/local/lib/X11/fonts/Liberation/, /usr/local/lib/X11/fonts/misc/, /usr/local/lib/X11/fonts/TTF/, /usr/local/lib/X11/fonts/OTF, /usr/local/lib/X11/fonts/Type1/, /usr/local/lib/X11/fonts/100dpi/, /usr/local/lib/X11/fonts/75dpi/ (==) ModulePath set to "/usr/local/lib/xorg/modules" (WW) AllowEmptyInput is on, devices using drivers 'kbd', 'mouse' or 'vmmouse' will be disabled. (WW) Disabling Mouse1 (WW) Disabling Keyboard1 (II) Loader magic: 0x689d80 (II) Module ABI versions: X.Org ANSI C Emulation: 0.4 X.Org Video Driver: 6.0 X.Org XInput driver : 7.0 X.Org Server Extension : 2.0 (--) Using syscons driver with X support (version 2.0) (--) using VT number 9 (--) PCI:*(0:0:2:0) 8086:29b2:103c:2818 Intel Corporation 82Q35 Express Integrated Graphics Controller rev 2, Mem @ 0xf0100000/524288, 0xe0000000/268435456, 0xf0000000/1048576, I/O @ 0x00001210/8, BIOS @ 0x????????/65536 (WW) "dri2" will not be loaded unless you've specified it to be loaded elsewhere. (II) "extmod" will be loaded. This was enabled by default and also specified in the config file. (II) "dbe" will be loaded. This was enabled by default and also specified in the config file. (II) "glx" will be loaded. This was enabled by default and also specified in the config file. (II) "record" will be loaded by default. (II) "dri" will be loaded. This was enabled by default and also specified in the config file. (II) "dri2" will be loaded even though the default is to disable it. (II) LoadModule: "dbe" (II) Loading /usr/local/lib/xorg/modules/extensions/libdbe.so (II) Module dbe: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.0.0 Module class: X.Org Server Extension ABI class: X.Org Server Extension, version 2.0 (II) Loading extension DOUBLE-BUFFER (II) LoadModule: "extmod" (II) Loading /usr/local/lib/xorg/modules/extensions/libextmod.so (II) Module extmod: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.0.0 Module class: X.Org Server Extension ABI class: X.Org Server Extension, version 2.0 (II) Loading extension MIT-SCREEN-SAVER (II) Loading extension XFree86-VidModeExtension (II) Loading extension XFree86-DGA (II) Loading extension DPMS (II) Loading extension XVideo (II) Loading extension XVideo-MotionCompensation (II) Loading extension X-Resource (II) LoadModule: "glx" (II) Loading /usr/local/lib/xorg/modules/extensions/libglx.so (II) Module glx: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.0.0 ABI class: X.Org Server Extension, version 2.0 (==) AIGLX disabled (II) Loading extension GLX (II) LoadModule: "dri" (II) Loading /usr/local/lib/xorg/modules/extensions/libdri.so (II) Module dri: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.0.0 ABI class: X.Org Server Extension, version 2.0 (II) Loading extension XFree86-DRI (II) LoadModule: "record" (II) Loading /usr/local/lib/xorg/modules/extensions/librecord.so (II) Module record: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.13.0 Module class: X.Org Server Extension ABI class: X.Org Server Extension, version 2.0 (II) Loading extension RECORD (II) LoadModule: "intel" (II) Loading /usr/local/lib/xorg/modules/drivers/intel_drv.so (II) Module intel: vendor="X.Org Foundation" compiled for 1.7.7, module version = 2.7.1 Module class: X.Org Video Driver ABI class: X.Org Video Driver, version 6.0 (II) intel: Driver for Intel Integrated Graphics Chipsets: i810, i810-dc100, i810e, i815, i830M, 845G, 852GM/855GM, 865G, 915G, E7221 (i915), 915GM, 945G, 945GM, 945GME, IGD_GM, IGD_G, 965G, G35, 965Q, 946GZ, 965GM, 965GME/GLE, G33, Q35, Q33, Mobile Intel® GM45 Express Chipset, Intel Integrated Graphics Device, G45/G43, Q45/Q43, G41 (II) Primary Device is: PCI 00@00:02:0 (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support (II) Loading sub module "vgahw" (II) LoadModule: "vgahw" (II) Loading /usr/local/lib/xorg/modules/libvgahw.so (II) Module vgahw: vendor="X.Org Foundation" compiled for 1.7.7, module version = 0.1.0 ABI class: X.Org Video Driver, version 6.0 (II) Loading sub module "ramdac" (II) LoadModule: "ramdac" (II) Module "ramdac" already built-in (**) intel(0): Depth 24, (--) framebuffer bpp 32 (==) intel(0): RGB weight 888 (==) intel(0): Default visual is TrueColor (II) intel(0): Integrated Graphics Chipset: Intel(R) Q35 (--) intel(0): Chipset: "Q35" (--) intel(0): Linear framebuffer at 0xE0000000 (--) intel(0): IO registers at addr 0xF0100000 (==) intel(0): Using EXA for acceleration (II) intel(0): 2 display pipes available. (II) Loading sub module "ddc" (II) LoadModule: "ddc" (II) Module "ddc" already built-in (II) Loading sub module "i2c" (II) LoadModule: "i2c" (II) Module "i2c" already built-in (II) intel(0): Output VGA using monitor section LCD1 (==) intel(0): Write-combining range (0xa0000,0x10000) was already clear (II) intel(0): Resizable framebuffer: not available (1 3) (II) intel(0): I2C bus "CRTDDC_A" initialized. (II) intel(0): I2C device "CRTDDC_A:ddc2" registered at address 0xA0. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" registered at address 0x6E. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" removed. (II) intel(0): I2C device "CRTDDC_A:ddc2" removed. (II) intel(0): I2C bus "CRTDDC_A" removed. (II) intel(0): EDID vendor "HWP", prod id 9858 (II) intel(0): Using EDID range info for horizontal sync (II) intel(0): Using EDID range info for vertical refresh (II) intel(0): Printing DDC gathered Modelines: (II) intel(0): Modeline "1280x1024"x0.0 108.00 1280 1328 1440 1688 1024 1025 1028 1066 +hsync +vsync (64.0 kHz) (II) intel(0): Modeline "800x600"x0.0 40.00 800 840 968 1056 600 601 605 628 +hsync +vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 656 720 840 480 481 484 500 -hsync -vsync (37.5 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 664 704 832 480 489 492 520 -hsync -vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 25.18 640 656 752 800 480 490 492 525 -hsync -vsync (31.5 kHz) (II) intel(0): Modeline "720x400"x0.0 28.32 720 738 846 900 400 412 414 449 -hsync +vsync (31.5 kHz) (II) intel(0): Modeline "1280x1024"x0.0 135.00 1280 1296 1440 1688 1024 1025 1028 1066 +hsync +vsync (80.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 78.75 1024 1040 1136 1312 768 769 772 800 +hsync +vsync (60.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 75.00 1024 1048 1184 1328 768 771 777 806 -hsync -vsync (56.5 kHz) (II) intel(0): Modeline "1024x768"x0.0 65.00 1024 1048 1184 1344 768 771 777 806 -hsync -vsync (48.4 kHz) (II) intel(0): Modeline "832x624"x0.0 57.28 832 864 928 1152 624 625 628 667 -hsync -vsync (49.7 kHz) (II) intel(0): Modeline "800x600"x0.0 49.50 800 816 896 1056 600 601 604 625 +hsync +vsync (46.9 kHz) (II) intel(0): Modeline "800x600"x0.0 50.00 800 856 976 1040 600 637 643 666 +hsync +vsync (48.1 kHz) (II) intel(0): Modeline "1152x864"x0.0 108.00 1152 1216 1344 1600 864 865 868 900 +hsync +vsync (67.5 kHz) (II) intel(0): EDID vendor "HWP", prod id 9858 (II) intel(0): Output VGA connected (II) intel(0): Using exact sizes for initial modes (II) intel(0): Output VGA using initial mode 1280x1024 (II) intel(0): Using default gamma of (1.0, 1.0, 1.0) unless otherwise stated. (==) intel(0): Write-combining range (0xa0000,0x10000) was already clear (II) intel(0): detected 2048 kB GTT. (II) intel(0): detected 6140 kB stolen memory. (==) intel(0): video overlay key set to 0x101fe (**) intel(0): Display dimensions: (380, 300) mm (**) intel(0): DPI set to (85, 108) (II) Loading sub module "fb" (II) LoadModule: "fb" (II) Loading /usr/local/lib/xorg/modules/libfb.so (II) Module fb: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.0.0 ABI class: X.Org ANSI C Emulation, version 0.4 (II) Loading sub module "exa" (II) LoadModule: "exa" (II) Loading /usr/local/lib/xorg/modules/libexa.so (II) Module exa: vendor="X.Org Foundation" compiled for 1.7.7, module version = 2.5.0 ABI class: X.Org Video Driver, version 6.0 (II) intel(0): Comparing regs from server start up to After PreInit (==) Depth 24 pixmap format is 32 bpp (II) intel(0): Kernel reported 1006592 total, 0 used (II) intel(0): I830CheckAvailableMemory: 4026368 kB available (WW) intel(0): DRI2 requires UXA drmOpenDevice: node name is /dev/dri/card0 drmOpenDevice: open result is 10, (OK) drmOpenDevice: node name is /dev/dri/card0 drmOpenDevice: open result is 10, (OK) drmOpenByBusid: Searching for BusID pci:0000:00:02.0 drmOpenDevice: node name is /dev/dri/card0 drmOpenDevice: open result is 10, (OK) drmOpenByBusid: drmOpenMinor returns 10 drmOpenByBusid: drmGetBusid reports pci:0000:00:02.0 (II) [drm] DRM interface version 1.2 (II) [drm] DRM open master succeeded. (II) intel(0): [drm] Using the DRM lock SAREA also for drawables. (II) intel(0): [drm] framebuffer mapped by ddx driver (II) intel(0): [drm] added 1 reserved context for kernel (II) intel(0): X context handle = 0x1 (II) intel(0): [drm] installed DRM signal handler (**) intel(0): Framebuffer compression disabled (**) intel(0): Tiling enabled (==) intel(0): VideoRam: 262144 KB (II) intel(0): Attempting memory allocation with tiled buffers. (II) intel(0): Tiled allocation successful. (II) intel(0): [drm] Registers = 0xf0100000 (II) intel(0): [drm] ring buffer = 0xe0000000 (II) intel(0): [drm] mapped front buffer at 0xe1000000, handle = 0xe1000000 (II) intel(0): [drm] mapped back buffer at 0xe4000000, handle = 0xe4000000 (II) intel(0): [drm] mapped depth buffer at 0xe5000000, handle = 0xe5000000 (II) intel(0): [drm] mapped classic textures at 0xe6000000, handle = 0xe6000000 (II) intel(0): [drm] Initialized kernel agp heap manager, 33554432 (II) intel(0): [dri] visual configs initialized (II) intel(0): vgaHWGetIOBase: hwp->IOBase is 0x03d0, hwp->PIOOffset is 0x0000 (==) intel(0): Write-combining range (0xa0000,0x10000) was already clear (II) EXA(0): Offscreen pixmap area of 31457280 bytes (II) EXA(0): Driver registered support for the following operations: (II) Solid (II) Copy (II) Composite (RENDER acceleration) (==) intel(0): Backing store disabled (==) intel(0): Silken mouse enabled (II) intel(0): Initializing HW Cursor (II) intel(0): [DRI] installation complete (WW) intel(0): drmDropMaster failed: Unknown error: -1 (II) intel(0): xf86BindGARTMemory: bind key 1 at 0x005ff000 (pgoffset 1535) (II) intel(0): xf86BindGARTMemory: bind key 2 at 0x0082a000 (pgoffset 2090) (II) intel(0): xf86BindGARTMemory: bind key 5 at 0x0082b000 (pgoffset 2091) (II) intel(0): xf86BindGARTMemory: bind key 3 at 0x01000000 (pgoffset 4096) (II) intel(0): xf86BindGARTMemory: bind key 4 at 0x02000000 (pgoffset 8192) (II) intel(0): xf86BindGARTMemory: bind key 6 at 0x04000000 (pgoffset 16384) (II) intel(0): xf86BindGARTMemory: bind key 7 at 0x05000000 (pgoffset 20480) (II) intel(0): xf86BindGARTMemory: bind key 8 at 0x06000000 (pgoffset 24576) (II) intel(0): Fixed memory allocation layout: (II) intel(0): 0x00000000-0x0001ffff: ring buffer (128 kB) (II) intel(0): 0x00020000-0x00029fff: HW cursors (40 kB) (II) intel(0): 0x0002a000-0x00829fff: fake bufmgr (8192 kB) (II) intel(0): 0x005ff000: end of stolen memory (II) intel(0): 0x0082a000-0x0082afff: overlay registers (4 kB) (II) intel(0): 0x0082b000-0x0082bfff: HW status (4 kB) (II) intel(0): 0x01000000-0x01ffffff: front buffer (16384 kB) X tiled (II) intel(0): 0x02000000-0x03dfffff: exa offscreen (30720 kB) (II) intel(0): 0x04000000-0x04ffffff: back buffer (16384 kB) X tiled (II) intel(0): 0x05000000-0x05ffffff: depth buffer (16384 kB) X tiled (II) intel(0): 0x06000000-0x07ffffff: classic textures (32768 kB) (II) intel(0): 0x10000000: end of aperture (II) intel(0): Output configuration: (II) intel(0): Pipe A is on (II) intel(0): Display plane A is now enabled and connected to pipe A. (II) intel(0): Pipe B is off (II) intel(0): Display plane B is now disabled and connected to pipe B. (II) intel(0): Output VGA is connected to pipe A (II) intel(0): [drm] dma control initialized, using IRQ 256 (II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message. (**) intel(0): DPMS enabled (==) intel(0): Intel XvMC decoder disabled (II) intel(0): Set up textured video (II) intel(0): Set up overlay video (II) intel(0): direct rendering: XF86DRI Enabled (--) RandR disabled (II) Initializing built-in extension Generic Event Extension (II) Initializing built-in extension SHAPE (II) Initializing built-in extension MIT-SHM (II) Initializing built-in extension XInputExtension (II) Initializing built-in extension XTEST (II) Initializing built-in extension BIG-REQUESTS (II) Initializing built-in extension SYNC (II) Initializing built-in extension XKEYBOARD (II) Initializing built-in extension XC-MISC (II) Initializing built-in extension XINERAMA (II) Initializing built-in extension XFIXES (II) Initializing built-in extension RENDER (II) Initializing built-in extension RANDR (II) Initializing built-in extension COMPOSITE (II) Initializing built-in extension DAMAGE (II) AIGLX: Loaded and initialized /usr/local/lib/dri/swrast_dri.so (II) GLX: Initialized DRISWRAST GL provider for screen 0 (II) intel(0): Setting screen physical size to 338 x 270 (EE) config/hal: couldn't initialise context: unknown error (null) (II) config/hal: Adding input device USB Gaming Mouse (II) LoadModule: "mouse" (II) Loading /usr/local/lib/xorg/modules/input/mouse_drv.so (II) Module mouse: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.6.0 Module class: X.Org XInput Driver ABI class: X.Org XInput driver, version 7.0 (**) USB Gaming Mouse: Device: "/dev/sysmouse" (==) USB Gaming Mouse: Protocol: "Auto" (**) USB Gaming Mouse: always reports core events (**) Option "Device" "/dev/sysmouse" (==) USB Gaming Mouse: Emulate3Buttons, Emulate3Timeout: 50 (**) USB Gaming Mouse: ZAxisMapping: buttons 4 and 5 (**) USB Gaming Mouse: Buttons: 9 (**) USB Gaming Mouse: Sensitivity: 1 (II) XINPUT: Adding extended input device "USB Gaming Mouse" (type: MOUSE) (**) USB Gaming Mouse: (accel) keeping acceleration scheme 1 (**) USB Gaming Mouse: (accel) acceleration profile 0 (II) USB Gaming Mouse: SetupAuto: hw.iftype is 4, hw.model is 0 (II) USB Gaming Mouse: SetupAuto: protocol is SysMouse (II) config/hal: Adding input device AT Keyboard (II) LoadModule: "kbd" (II) Loading /usr/local/lib/xorg/modules/input/kbd_drv.so (II) Module kbd: vendor="X.Org Foundation" compiled for 1.7.7, module version = 1.5.0 Module class: X.Org XInput Driver ABI class: X.Org XInput driver, version 7.0 (**) AT Keyboard: always reports core events (**) Option "Protocol" "standard" (**) AT Keyboard: Protocol: standard (**) Option "XkbRules" "base" (**) AT Keyboard: XkbRules: "base" (**) Option "XkbModel" "pc105" (**) AT Keyboard: XkbModel: "pc105" (**) Option "XkbLayout" "us" (**) AT Keyboard: XkbLayout: "us" (**) Option "XkbOptions" "terminate:ctrl_alt_bksp" (**) AT Keyboard: XkbOptions: "terminate:ctrl_alt_bksp" (**) Option "CustomKeycodes" "off" (**) AT Keyboard: CustomKeycodes disabled (II) XINPUT: Adding extended input device "AT Keyboard" (type: KEYBOARD) (II) intel(0): I2C bus "CRTDDC_A" initialized. (II) intel(0): I2C device "CRTDDC_A:ddc2" registered at address 0xA0. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" registered at address 0x6E. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" removed. (II) intel(0): I2C device "CRTDDC_A:ddc2" removed. (II) intel(0): I2C bus "CRTDDC_A" removed. (II) intel(0): EDID vendor "HWP", prod id 9858 (II) intel(0): Using hsync ranges from config file (II) intel(0): Using vrefresh ranges from config file (II) intel(0): Printing DDC gathered Modelines: (II) intel(0): Modeline "1280x1024"x0.0 108.00 1280 1328 1440 1688 1024 1025 1028 1066 +hsync +vsync (64.0 kHz) (II) intel(0): Modeline "800x600"x0.0 40.00 800 840 968 1056 600 601 605 628 +hsync +vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 656 720 840 480 481 484 500 -hsync -vsync (37.5 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 664 704 832 480 489 492 520 -hsync -vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 25.18 640 656 752 800 480 490 492 525 -hsync -vsync (31.5 kHz) (II) intel(0): Modeline "720x400"x0.0 28.32 720 738 846 900 400 412 414 449 -hsync +vsync (31.5 kHz) (II) intel(0): Modeline "1280x1024"x0.0 135.00 1280 1296 1440 1688 1024 1025 1028 1066 +hsync +vsync (80.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 78.75 1024 1040 1136 1312 768 769 772 800 +hsync +vsync (60.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 75.00 1024 1048 1184 1328 768 771 777 806 -hsync -vsync (56.5 kHz) (II) intel(0): Modeline "1024x768"x0.0 65.00 1024 1048 1184 1344 768 771 777 806 -hsync -vsync (48.4 kHz) (II) intel(0): Modeline "832x624"x0.0 57.28 832 864 928 1152 624 625 628 667 -hsync -vsync (49.7 kHz) (II) intel(0): Modeline "800x600"x0.0 49.50 800 816 896 1056 600 601 604 625 +hsync +vsync (46.9 kHz) (II) intel(0): Modeline "800x600"x0.0 50.00 800 856 976 1040 600 637 643 666 +hsync +vsync (48.1 kHz) (II) intel(0): Modeline "1152x864"x0.0 108.00 1152 1216 1344 1600 864 865 868 900 +hsync +vsync (67.5 kHz) (II) intel(0): EDID vendor "HWP", prod id 9858 (II) intel(0): I2C bus "CRTDDC_A" initialized. (II) intel(0): I2C device "CRTDDC_A:ddc2" registered at address 0xA0. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" registered at address 0x6E. (II) intel(0): I2C device "CRTDDC_A:DDC control interface" removed. (II) intel(0): I2C device "CRTDDC_A:ddc2" removed. (II) intel(0): I2C bus "CRTDDC_A" removed. (II) intel(0): EDID vendor "HWP", prod id 9858 (II) intel(0): Using hsync ranges from config file (II) intel(0): Using vrefresh ranges from config file (II) intel(0): Printing DDC gathered Modelines: (II) intel(0): Modeline "1280x1024"x0.0 108.00 1280 1328 1440 1688 1024 1025 1028 1066 +hsync +vsync (64.0 kHz) (II) intel(0): Modeline "800x600"x0.0 40.00 800 840 968 1056 600 601 605 628 +hsync +vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 656 720 840 480 481 484 500 -hsync -vsync (37.5 kHz) (II) intel(0): Modeline "640x480"x0.0 31.50 640 664 704 832 480 489 492 520 -hsync -vsync (37.9 kHz) (II) intel(0): Modeline "640x480"x0.0 25.18 640 656 752 800 480 490 492 525 -hsync -vsync (31.5 kHz) (II) intel(0): Modeline "720x400"x0.0 28.32 720 738 846 900 400 412 414 449 -hsync +vsync (31.5 kHz) (II) intel(0): Modeline "1280x1024"x0.0 135.00 1280 1296 1440 1688 1024 1025 1028 1066 +hsync +vsync (80.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 78.75 1024 1040 1136 1312 768 769 772 800 +hsync +vsync (60.0 kHz) (II) intel(0): Modeline "1024x768"x0.0 75.00 1024 1048 1184 1328 768 771 777 806 -hsync -vsync (56.5 kHz) (II) intel(0): Modeline "1024x768"x0.0 65.00 1024 1048 1184 1344 768 771 777 806 -hsync -vsync (48.4 kHz) (II) intel(0): Modeline "832x624"x0.0 57.28 832 864 928 1152 624 625 628 667 -hsync -vsync (49.7 kHz) (II) intel(0): Modeline "800x600"x0.0 49.50 800 816 896 1056 600 601 604 625 +hsync +vsync (46.9 kHz) (II) intel(0): Modeline "800x600"x0.0 50.00 800 856 976 1040 600 637 643 666 +hsync +vsync (48.1 kHz) (II) intel(0): Modeline "1152x864"x0.0 108.00 1152 1216 1344 1600 864 865 868 900 +hsync +vsync (67.5 kHz) (II) intel(0): EDID vendor "HWP", prod id 9858 (II) 3rd Button detected: disabling emulate3Button [mi] EQ overflowing. The server is probably stuck in an infinite loop. --Fba/0zbH8Xs+Fj9o-- From owner-freebsd-stable@FreeBSD.ORG Thu May 5 18:39:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C241106566B for ; Thu, 5 May 2011 18:39:21 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id BC6238FC15 for ; Thu, 5 May 2011 18:39:20 +0000 (UTC) Received: by vxc34 with SMTP id 34so3474593vxc.13 for ; Thu, 05 May 2011 11:39:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=hyv9FhSYC8bTMfywdAsVFexK8BF7VupTYfFXiekoEz8=; b=K9VlvuKd17MtqEv4GTO6xyFmh42Sz/FapDUimQw0ny1wARvtSGaMDIh1pNDBH2SFCG 4hPQ+HHT9aRDZ7BcXsKpHhCTlG0gQl4VUYI+uEZr23u4ceuO32my7csj2nyB9VtqkDUS jvZ4xlr5+e/V/1FR2cOo1Ahs6zvPSzYaOC/O4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Pj4jezARQk9cfR5b1j6T4wW1wgIjwGfpNMVmgSAotbAbIiw7D5IicDmxKsFU1XrTiM 11tyMm+Y9/7TBQXIPOWqDZFAQSvKa23Ac56JOW5N9Vfy70blRz3EExz8YXn8i/n4ddKx SCAYR4coeiJFexMrnpjcyv5PYF5KtScUhd3Yg= MIME-Version: 1.0 Received: by 10.52.98.137 with SMTP id ei9mr3501575vdb.64.1304619099833; Thu, 05 May 2011 11:11:39 -0700 (PDT) Received: by 10.52.157.104 with HTTP; Thu, 5 May 2011 11:11:39 -0700 (PDT) Date: Thu, 5 May 2011 14:11:39 -0400 Message-ID: From: Zaphod Beeblebrox To: FreeBSD Stable Content-Type: text/plain; charset=ISO-8859-1 Subject: Intel "em" driver sleeps with non-sleepable lock. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 18:39:21 -0000 The motherboard in question is made by Intel and contains a Xeon 3440 (4 core x 2 HT per core). 16 Gig of RAM is installed and we are installing the 64 bit FreeBSD 8.2 using the PC-BSD installer (to install zfs root faster). The motherboard has 4 "igb" ethernet and one "em" ethernet. The "em" ethernet is shared with an internal "RMM3" remote management card and/or the onboard ILOM. This error happens when rebooting after installation and is repeatable with at least FreeBSD 8.1 and FreeBSD 8.2. The last boot message is "Starting devd" ... so I assume that the active link on em0 might be making devd start dhclient. After this last boot message, the screen reads: Sleeping thread (tid 100195, pid 619) owns a non-sleepable lock panic: sleeping thread cpuid = 2 KDB: stack backtrace: #0 0xffffffff805f4e03 at kdb_backtrace+0x5e #1 0xffffffff805c2d07 at panic+0x187 #2 0xffffffff80601a5d at propagate_priority+0x1cd #3 0xffffffff8060278a at turnstile_wait+0x1aa #4 0xffffffff805b34c0 at _mtx_lock_sleep+0xb0 #5 0xffffffff8032fd97 at em_init_locked+0xce7 #6 0xffffffff80331b8e at em_ioctl+0x5fe #7 0xffffffff80671114 at ifioctl+0x9e4 #8 0xffffffff806043c2 at kern_ioctl+0x102 #9 0xffffffff806045fd at ioctl+0xfd #10 0xffffffffff80600dd5 at syscallenter+0x1e5 #11 0xffffffffff808aca5b at syscall+0x4b #12 0xffffffffff80895292 at Xfast_syscall+0xe2 Now. I assume that booting without link on em0 (inconvenient) or booting without "em" in the kernel will fix things. I'll be checking this out shortly. I can provide (for a limited time) full console access and/or I can test code if someone sees a patch for this. From owner-freebsd-stable@FreeBSD.ORG Thu May 5 20:44:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE013106566C for ; Thu, 5 May 2011 20:44:21 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 94E7E8FC12 for ; Thu, 5 May 2011 20:44:21 +0000 (UTC) Received: by qyk35 with SMTP id 35so4444775qyk.13 for ; Thu, 05 May 2011 13:44:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=qsSHLt74gNp1Bs1xK5L9pnzKW+/Qw1AibH6XRnGKIlE=; b=BYWU144bUL4S1aDnkJIfVZ109oGaZWItXxBAo3wNNnibsa41V3QMxhdhqRol3iIU1k M3/0HF1DMxSsgmHWVlSnUNpKo2OIywHNI+tn1ZsG0UmWsQ6bxWJbsd15tCo1/bdrYQVP zVhWd5KYqoMhv/4czLq1Fc13kBEpzIKlGOnXk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Dld+qObN+XKLS7PIxVTzpqIlSx9HOzvf4IsPSboKG1yhB7ZGcR47J/1BXMiYnlnKKW cWW2Tqu1SOGzcOM7v9sFTsadFHKOtioz4CdcCVHT2cTlUfL3jYL6SNQX7A3NeBtcD2Cx TxzFhP7Ef0MMz7DEiBYp94JSe9uNY6bYJa5kI= MIME-Version: 1.0 Received: by 10.52.76.193 with SMTP id m1mr407922vdw.204.1304628260313; Thu, 05 May 2011 13:44:20 -0700 (PDT) Received: by 10.52.184.169 with HTTP; Thu, 5 May 2011 13:44:20 -0700 (PDT) In-Reply-To: References: Date: Thu, 5 May 2011 13:44:20 -0700 Message-ID: From: Jack Vogel To: Zaphod Beeblebrox Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Stable Subject: Re: Intel "em" driver sleeps with non-sleepable lock. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 20:44:21 -0000 So, this happens EVERY time after an install of 8.2 ?? Give me details about the hardware please. Jack On Thu, May 5, 2011 at 11:11 AM, Zaphod Beeblebrox wrote: > The motherboard in question is made by Intel and contains a Xeon 3440 > (4 core x 2 HT per core). 16 Gig of RAM is installed and we are > installing the 64 bit FreeBSD 8.2 using the PC-BSD installer (to > install zfs root faster). The motherboard has 4 "igb" ethernet and > one "em" ethernet. The "em" ethernet is shared with an internal > "RMM3" remote management card and/or the onboard ILOM. This error > happens when rebooting after installation and is repeatable with at > least FreeBSD 8.1 and FreeBSD 8.2. > > The last boot message is "Starting devd" ... so I assume that the > active link on em0 might be making devd start dhclient. After this > last boot message, the screen reads: > > Sleeping thread (tid 100195, pid 619) owns a non-sleepable lock > panic: sleeping thread > cpuid = 2 > KDB: stack backtrace: > #0 0xffffffff805f4e03 at kdb_backtrace+0x5e > #1 0xffffffff805c2d07 at panic+0x187 > #2 0xffffffff80601a5d at propagate_priority+0x1cd > #3 0xffffffff8060278a at turnstile_wait+0x1aa > #4 0xffffffff805b34c0 at _mtx_lock_sleep+0xb0 > #5 0xffffffff8032fd97 at em_init_locked+0xce7 > #6 0xffffffff80331b8e at em_ioctl+0x5fe > #7 0xffffffff80671114 at ifioctl+0x9e4 > #8 0xffffffff806043c2 at kern_ioctl+0x102 > #9 0xffffffff806045fd at ioctl+0xfd > #10 0xffffffffff80600dd5 at syscallenter+0x1e5 > #11 0xffffffffff808aca5b at syscall+0x4b > #12 0xffffffffff80895292 at Xfast_syscall+0xe2 > > Now. I assume that booting without link on em0 (inconvenient) or > booting without "em" in the kernel will fix things. I'll be checking > this out shortly. > > I can provide (for a limited time) full console access and/or I can > test code if someone sees a patch for this. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Fri May 6 12:42:06 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6848D106566B for ; Fri, 6 May 2011 12:42:06 +0000 (UTC) (envelope-from Holger.Kipp@alogis.com) Received: from alogis.com (firewall.solit-ag.de [212.184.102.1]) by mx1.freebsd.org (Postfix) with ESMTP id 685178FC17 for ; Fri, 6 May 2011 12:42:04 +0000 (UTC) Received: from msx3.exchange.alogis.com (exchange.alogis.com [10.1.1.6]) by alogis.com (8.13.4/8.13.1) with ESMTP id p46Cg3XY073971 for ; Fri, 6 May 2011 14:42:03 +0200 (CEST) (envelope-from Holger.Kipp@alogis.com) Received: from MSX3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61]) by msx3.exchange.alogis.com ([fe80::c8ed:428a:a157:b61%13]) with mapi id 14.01.0255.000; Fri, 6 May 2011 14:43:40 +0200 From: Holger Kipp To: "stable@freebsd.org" Thread-Topic: resilvering takes ages on 8.2 (amd64 from 18.04.2011) Thread-Index: AcwL6zkZX7VYhWsgQ2qQHvTpcQf1ZA== Date: Fri, 6 May 2011 12:43:39 +0000 Message-ID: <814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81@msx3.exchange.alogis.com> Accept-Language: en-GB, de-DE, en-US Content-Language: en-GB X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [212.184.101.130] Content-Type: multipart/related; boundary="_004_814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81msx3exchangealo_"; type="multipart/alternative" MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: resilvering takes ages on 8.2 (amd64 from 18.04.2011) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 12:42:06 -0000 --_004_814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81msx3exchangealo_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Resilvering a disk in raidz2 ZFS is taking ages. Any ideas? I had replaced = a different disk this morning (da7) and it took only about 1 hour alltogeth= er. Any ideas? Or did I do something very wrong (tm)? Best regards, Holger -------------------- 8.2-STABLE FreeBSD 8.2-STABLE #12: Mon Apr 18 12:48:56 CEST 2011 # zpool status pool: tank state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h21m, 1.10% done, 32h19m to go config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 da0/old OFFLINE 0 0 0 da0 ONLINE 0 0 0 158M resilvered da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da7 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 errors: No known data errors ----------------- 1 users Load 0.00 0.00 0.00 May 6 14:37 Mem:KB REAL VIRTUAL VN PAGER SWAP PAG= ER Tot Share Tot Share Free in out in o= ut Act 119320 18980 1362144 25756 7648824 count All 344920 23296 1075218k 57012 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 4765 total 76 9171 11 130 766 952 zfod atkbd= 0 1 ozfod ata0 = irq14 0.7%Sys 0.3%Intr 0.0%User 0.0%Nice 99.0%Idle %ozfod 1 uhci0= 16 | | | | | | | | | | | daefr uhci1= 17 prcfr 1 twe0 = irq24 7 dtbuf 1106 totfr 1999 cpu0:= time Namei Name-cache Dir-cache 206492 desvn react 763 isp0 = 256 Calls hits % hits % 3598 numvn pdwak 2 em0 i= rq257 12 11 92 1382 frevn pdpgs 1999 cpu1:= time intrn Disks da0 da1 da2 da3 da4 da5 da6 328004 wire KB/t 0.67 4.92 4.68 4.46 4.53 4.66 4.91 84728 act tps 151 138 150 151 146 149 140 39788 inact MB/s 0.10 0.66 0.69 0.66 0.65 0.68 0.67 1520 cache %busy 101 41 44 42 41 40 39 7647296 free 60464 buf -- Holger Kipp Diplom-Mathematiker Senior Consultant [alogis] Tel. : +49 30 436 58 114 Fax : +49 30 436 58 214 Mobil : +49 178 36 58 114 E-Mail : holger.kipp@alogis.com alogis AG Alt-Moabit 90 B D- 10559 Berlin Web: www.alogis.com alogis AG Sitz/Registergericht: Berlin/AG Charlottenburg, HRB 71484 Vorstand: Arne Friedrichs, Joern Samuelson Aufsichtsratsvorsitzender: Reinhard Mielke --_004_814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81msx3exchangealo_-- From owner-freebsd-stable@FreeBSD.ORG Fri May 6 15:23:51 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96C05106564A for ; Fri, 6 May 2011 15:23:51 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4EB8C8FC0C for ; Fri, 6 May 2011 15:23:50 +0000 (UTC) Received: by qyk27 with SMTP id 27so3088903qyk.13 for ; Fri, 06 May 2011 08:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=DlgUrysrD+UZArKlxNR6PG8oBV5wLaa++FR/eSEX0cw=; b=xaUbvA1Za2p/W7kFxMIyUhEBvUe35vE1ofxSTqNkFR7TyU927irDW90fD5E4HdX3fG navuvEhL9dIv36C1WQ8V9jyXfmv25jZ7nutfzs+HvaFPBTrp4IXJOivCI5xI74S7ZPnp kl+eE+GEHpWfIOO0MC+WHxCU2Zddj2TM6gGkY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=P20NypMCTAG/Yn3NEGfRrsDJllBphqQefEQxEPGK0DUBBF8LtFcfwxS3lmLRmE8lFU ClYG3bAJf3m4780/ZTuQAxuaFdKn8+Uzo3pyTMUSvPz+7TJnXTuos0d7RYhMY7ZVYhlV t/qWviAKGRKtpptxAwGGa3B+ww5VsfpjhuhFw= MIME-Version: 1.0 Received: by 10.229.102.165 with SMTP id g37mr2588648qco.120.1304693808792; Fri, 06 May 2011 07:56:48 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.229.95.140 with HTTP; Fri, 6 May 2011 07:56:48 -0700 (PDT) In-Reply-To: <814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81@msx3.exchange.alogis.com> References: <814C9E9472FDCC40AAC3FC95A2D67E3B0BD7BB81@msx3.exchange.alogis.com> Date: Fri, 6 May 2011 07:56:48 -0700 X-Google-Sender-Auth: m1c1516t-S2qZr03p9rU5bfynLo Message-ID: From: Artem Belevich To: Holger Kipp Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "stable@freebsd.org" Subject: Re: resilvering takes ages on 8.2 (amd64 from 18.04.2011) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 15:23:51 -0000 On Fri, May 6, 2011 at 5:43 AM, Holger Kipp wrote: > Resilvering a disk in raidz2 ZFS is taking ages. Any ideas? I had replace= d a different disk this morning (da7) and it took only about 1 hour alltoge= ther. Any ideas? Or did I do something very wrong (tm)? Don't believe everything you see. On my pool I often see scrub time estimates of few hundred hours even though it always takes about 6 hours. Once ZFS is done thrashing disks while it scrubs metadata and starts doing bulk data transfer, estimates would eventually converge to a sensible value. > Disks =A0 da0 =A0 da1 =A0 da2 =A0 da3 =A0 da4 =A0 da5 =A0 da6 =A0 =A03280= 04 wire > KB/t =A0 0.67 =A04.92 =A04.68 =A04.46 =A04.53 =A04.66 =A04.91 =A0 =A0 847= 28 act > tps =A0 =A0 151 =A0 138 =A0 150 =A0 151 =A0 146 =A0 149 =A0 140 =A0 =A0 3= 9788 inact > MB/s =A0 0.10 =A00.66 =A00.69 =A00.66 =A00.65 =A00.68 =A00.67 =A0 =A0 =A0= 1520 cache > %busy =A0 101 =A0 =A041 =A0 =A044 =A0 =A042 =A0 =A041 =A0 =A040 =A0 =A039= =A0 7647296 free Yup. It does look like your pool is in "thrashing" stage -- lots of seeking= . --Artem > > Best regards, > Holger > > -------------------- > > =A08.2-STABLE FreeBSD 8.2-STABLE #12: Mon Apr 18 12:48:56 CEST 2011 > > # zpool status > =A0pool: tank > =A0state: DEGRADED > status: One or more devices is currently being resilvered. =A0The pool wi= ll > =A0 =A0 =A0 =A0continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > =A0scrub: resilver in progress for 0h21m, 1.10% done, 32h19m to go > config: > > =A0 =A0 =A0 =A0NAME =A0 =A0 =A0 =A0 =A0 STATE =A0 =A0 READ WRITE CKSUM > =A0 =A0 =A0 =A0tank =A0 =A0 =A0 =A0 =A0 DEGRADED =A0 =A0 0 =A0 =A0 0 =A0 = =A0 0 > =A0 =A0 =A0 =A0 =A0raidz2 =A0 =A0 =A0 DEGRADED =A0 =A0 0 =A0 =A0 0 =A0 = =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0replacing =A0DEGRADED =A0 =A0 0 =A0 =A0 0 =A0 =A0 = 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0da0/old =A0OFFLINE =A0 =A0 =A00 =A0 =A0 0 =A0 = =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0da0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 =A0158M resilvered > =A0 =A0 =A0 =A0 =A0 =A0da1 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da2 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da7 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da3 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da4 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da5 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0da6 =A0 =A0 =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > > errors: No known data errors > > > > ----------------- > =A0 =A01 users =A0 =A0Load =A00.00 =A00.00 =A00.00 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0May =A06 14:37 > > Mem:KB =A0 =A0REAL =A0 =A0 =A0 =A0 =A0 =A0VIRTUAL =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 VN PAGER =A0 SWAP PAGER > =A0 =A0 =A0 =A0Tot =A0 Share =A0 =A0 =A0Tot =A0 =A0Share =A0 =A0Free =A0 = =A0 =A0 =A0 =A0 in =A0 out =A0 =A0 in =A0 out > Act =A0119320 =A0 18980 =A01362144 =A0 =A025756 7648824 =A0count > All =A0344920 =A0 23296 1075218k =A0 =A057012 =A0 =A0 =A0 =A0 =A0pages > Proc: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Interrupts > =A0r =A0 p =A0 d =A0 s =A0 w =A0 Csw =A0Trp =A0Sys =A0Int =A0Sof =A0Flt = =A0 =A0 =A0 =A0cow =A0 =A04765 total > =A0 =A0 =A0 =A0 =A0 =A0 76 =A0 =A0 =A09171 =A0 11 =A0130 =A0766 =A0952 = =A0 =A0 =A0 =A0 =A0 =A0 zfod =A0 =A0 =A0 =A0atkbd0 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ozfod =A0 =A0 =A0 ata0 irq14 > =A00.7%Sys =A0 0.3%Intr =A00.0%User =A00.0%Nice 99.0%Idle =A0 =A0 =A0 =A0= %ozfod =A0 =A0 1 uhci0 16 > | =A0 =A0| =A0 =A0| =A0 =A0| =A0 =A0| =A0 =A0| =A0 =A0| =A0 =A0| =A0 =A0|= =A0 =A0| =A0 =A0| =A0 =A0 =A0 daefr =A0 =A0 =A0 uhci1 17 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0prcfr =A0 =A0 1 twe0 irq24 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 7 dtbuf =A0 =A0 1106 totfr =A01999 cpu0: time > Namei =A0 =A0 Name-cache =A0 Dir-cache =A0 =A0206492 desvn =A0 =A0 =A0 = =A0 =A0react =A0 763 isp0 256 > =A0 Calls =A0 =A0hits =A0 % =A0 =A0hits =A0 % =A0 =A0 =A03598 numvn =A0 = =A0 =A0 =A0 =A0pdwak =A0 =A0 2 em0 irq257 > =A0 =A0 =A012 =A0 =A0 =A011 =A092 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01382= frevn =A0 =A0 =A0 =A0 =A0pdpgs =A01999 cpu1: time > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0intrn > Disks =A0 da0 =A0 da1 =A0 da2 =A0 da3 =A0 da4 =A0 da5 =A0 da6 =A0 =A03280= 04 wire > KB/t =A0 0.67 =A04.92 =A04.68 =A04.46 =A04.53 =A04.66 =A04.91 =A0 =A0 847= 28 act > tps =A0 =A0 151 =A0 138 =A0 150 =A0 151 =A0 146 =A0 149 =A0 140 =A0 =A0 3= 9788 inact > MB/s =A0 0.10 =A00.66 =A00.69 =A00.66 =A00.65 =A00.68 =A00.67 =A0 =A0 =A0= 1520 cache > %busy =A0 101 =A0 =A041 =A0 =A044 =A0 =A042 =A0 =A041 =A0 =A040 =A0 =A039= =A0 7647296 free > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A060464 buf > > > -- > > > Holger Kipp > Diplom-Mathematiker > Senior Consultant > > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[alogis] > > Tel. =A0 =A0: =A0 =A0 =A0 +49 30 436 58 114 > Fax =A0 =A0 : =A0 =A0 =A0 +49 30 436 58 214 > Mobil =A0 : =A0 =A0 =A0 +49 178 36 58 114 > > E-Mail =A0: =A0 =A0 =A0 holger.kipp@alogis.com > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0alogis AG > Alt-Moabit 90 B > D- 10559 Berlin > > Web: www.alogis.com > > > > alogis AG > Sitz/Registergericht: Berlin/AG Charlottenburg, HRB 71484 > Vorstand: Arne Friedrichs, Joern Samuelson > Aufsichtsratsvorsitzender: Reinhard Mielke > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Fri May 6 19:05:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 467DD1065672 for ; Fri, 6 May 2011 19:05:07 +0000 (UTC) (envelope-from benzene@arcor.de) Received: from fh-dortmund.de (fhdo-2.dvz.FH-Dortmund.DE [193.25.16.6]) by mx1.freebsd.org (Postfix) with ESMTP id DFEAE8FC08 for ; Fri, 6 May 2011 19:05:06 +0000 (UTC) Received: from gatekeeper.informatik.fh-dortmund.de (gatekeeper.informatik.FH-Dortmund.DE [193.25.22.84]) by fh-dortmund.de (Switch-3.3.0/Switch-3.3.0) with ESMTP id p46IcAW7004949 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 6 May 2011 20:38:21 +0200 (MEST) Received: from custos.hw1.fb4.fh (n43.informatik.FH-Dortmund.DE [193.25.22.43]) by gatekeeper.informatik.fh-dortmund.de (8.12.10/8.12.10) with ESMTP id p46Ibp5x001260 for ; Fri, 6 May 2011 20:38:11 +0200 From: Michael Hoffmann To: freebsd-stable@freebsd.org Date: Fri, 6 May 2011 20:37:50 +0200 User-Agent: KMail/1.9.10 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201105062037.50937.benzene@arcor.de> Subject: Double installation of liblzma.so.5 breaks libarchive.so.5 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 19:05:07 -0000 Hi, May be some of you will be affected by this like me. Today I upgraded from RELEASE_8_1 to RELEASE_8_2. The box was upgraded over the years starting from 6.2, as far as I know. But now this happened: > # pkg_create -b xz-5.0.1 > /libexec/ld-elf.so.1: /usr/local/lib/liblzma.so.5: version XZ_5.0 required by /usr/lib/libarchive.so.5 not defined > > # ldd /usr/lib/libarchive.so.5 > /usr/lib/libarchive.so.5: > libz.so.5 => /lib/libz.so.5 (0x281d5000) > libmd.so.5 => /lib/libmd.so.5 (0x281e7000) > libbz2.so.4 => /usr/lib/libbz2.so.4 (0x28300000) > liblzma.so.5 => /usr/local/lib/liblzma.so.5 (0x28311000) <== !!! > libcrypto.so.6 => /lib/libcrypto.so.6 (0x28331000) > libc.so.7 => /lib/libc.so.7 (0x2808f000) > libthr.so.3 => /lib/libthr.so.3 (0x28483000) > > # find / -name "liblzma.so.5" > /usr/lib/liblzma.so.5 > /usr/local/lib/liblzma.so.5 > > pkg_info -W /usr/local/lib/liblzma.so.5 > /usr/local/lib/liblzma.so.5 was installed by package xz-5.0.1 I think trouble partially comes from preserving an old LD_LIBRARY_PATH entry in csh.cshrc: > # mergemaster > [...] > *** Displaying differences between ./etc/csh.cshrc and installed version: > --- /etc/csh.cshrc 2011-05-05 20:20:26.000000000 +0200 > +++ ./etc/csh.cshrc 2011-05-06 18:23:46.000000000 +0200 > @@ -1,4 +1,3 @@ > -# $FreeBSD: src/etc/csh.cshrc,v 1.3.56.1.4.1 2010/06/14 02:09:06 kensmith Exp $ > +# $FreeBSD: src/etc/csh.cshrc,v 1.3.56.1.6.1 2010/12/21 17:09:25 kensmith Exp $ # > # System-wide .cshrc file for csh(1). > -setenv LD_LIBRARY_PATH /usr/local/lib:/usr/local/lib/pth > > Use 'd' to delete the temporary ./etc/csh.cshrc > Use 'i' to install the temporary ./etc/csh.cshrc > Use 'm' to merge the temporary and installed versions > Use 'v' to view the diff results again > > Default is to leave the temporary file to deal with by hand > > How should I deal with this? [Leave it for later] m > > # $FreeBSD: src/etc/csh.cshrc,v 1.3.5 | # $FreeBSD: src/etc/csh.cshrc,v 1.3.5 > %2 > setenv LD_LIBRARY_PATH /usr/local/lib < > %1 > > Use 'i' to install merged file > Use 'r' to re-do the merge > Use 'v' to view the merged file > Default is to leave the temporary file to deal with by hand > > *** How should I deal with the merged file? [Leave it for later] i After deinstalling xz-5.0.1 everything works fine again. But probably cleaning the environment from LD_LIBRARY_PATH would have done, too. (I thought it to be wise to remove it from csh.cshrc. After that I used libchk and pkg_libchk, and everything seems fine.) I cannot reconstruct why 'pkgdb -Fu' did not tried to prevent me from using 'archivers/xz' furthermore, allthough it now seems to be marked as IGNORE. I also don't know how this LD_LIBRARY thing ever popped up in csh.cshrc, but it seems to persist there from at least 7.2. Newer systems don't seem to have such an entry. Kind regards, Michael Hoffmann From owner-freebsd-stable@FreeBSD.ORG Fri May 6 20:48:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39748106566B for ; Fri, 6 May 2011 20:48:03 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 108898FC0A for ; Fri, 6 May 2011 20:48:03 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id B26F246B23; Fri, 6 May 2011 16:48:02 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3854A8A027; Fri, 6 May 2011 16:48:02 -0400 (EDT) From: John Baldwin To: freebsd-stable@freebsd.org Date: Fri, 6 May 2011 16:48:01 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201105061648.01646.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 06 May 2011 16:48:02 -0400 (EDT) Cc: Zaphod Beeblebrox Subject: Re: Intel "em" driver sleeps with non-sleepable lock. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 20:48:03 -0000 On Thursday, May 05, 2011 2:11:39 pm Zaphod Beeblebrox wrote: > The motherboard in question is made by Intel and contains a Xeon 3440 > (4 core x 2 HT per core). 16 Gig of RAM is installed and we are > installing the 64 bit FreeBSD 8.2 using the PC-BSD installer (to > install zfs root faster). The motherboard has 4 "igb" ethernet and > one "em" ethernet. The "em" ethernet is shared with an internal > "RMM3" remote management card and/or the onboard ILOM. This error > happens when rebooting after installation and is repeatable with at > least FreeBSD 8.1 and FreeBSD 8.2. > > The last boot message is "Starting devd" ... so I assume that the > active link on em0 might be making devd start dhclient. After this > last boot message, the screen reads: > > Sleeping thread (tid 100195, pid 619) owns a non-sleepable lock > panic: sleeping thread > cpuid = 2 > KDB: stack backtrace: > #0 0xffffffff805f4e03 at kdb_backtrace+0x5e > #1 0xffffffff805c2d07 at panic+0x187 > #2 0xffffffff80601a5d at propagate_priority+0x1cd > #3 0xffffffff8060278a at turnstile_wait+0x1aa > #4 0xffffffff805b34c0 at _mtx_lock_sleep+0xb0 > #5 0xffffffff8032fd97 at em_init_locked+0xce7 > #6 0xffffffff80331b8e at em_ioctl+0x5fe > #7 0xffffffff80671114 at ifioctl+0x9e4 > #8 0xffffffff806043c2 at kern_ioctl+0x102 > #9 0xffffffff806045fd at ioctl+0xfd > #10 0xffffffffff80600dd5 at syscallenter+0x1e5 > #11 0xffffffffff808aca5b at syscall+0x4b > #12 0xffffffffff80895292 at Xfast_syscall+0xe2 You need to trace the thread that owns the lock (in this case 619). I thought the kernel automatically did that actually, but maybe it only does that if certain debugging options are enabled. If you can break into KDB and do 'tr 100195' that would be ideal. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Sat May 7 08:56:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18CCF106566C for ; Sat, 7 May 2011 08:56:00 +0000 (UTC) (envelope-from matthias.andree@gmx.de) Received: from mailout-de.gmx.net (mailout-de.gmx.net [213.165.64.23]) by mx1.freebsd.org (Postfix) with SMTP id 6387A8FC08 for ; Sat, 7 May 2011 08:55:59 +0000 (UTC) Received: (qmail invoked by alias); 07 May 2011 08:55:57 -0000 Received: from f055104069.adsl.alicedsl.de (EHLO apollo.emma.line.org) [78.55.104.69] by mail.gmx.net (mp025) with SMTP; 07 May 2011 10:55:57 +0200 X-Authenticated: #428038 X-Provags-ID: V01U2FsdGVkX18tCZ5xYOUtSdhfSL5A1OwLV1sllheC3pUaxjJirK drwGM+eBV+3PjU Received: from [127.0.0.1] (localhost.localdomain [127.0.0.1]) by apollo.emma.line.org (Postfix) with ESMTP id 2471423DBA9 for ; Sat, 7 May 2011 10:55:53 +0200 (CEST) Message-ID: <4DC50918.3030100@gmx.de> Date: Sat, 07 May 2011 10:55:52 +0200 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Mnenhy/0.8.3 Thunderbird/3.1.10 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <201105062037.50937.benzene@arcor.de> In-Reply-To: <201105062037.50937.benzene@arcor.de> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 Subject: Re: Double installation of liblzma.so.5 breaks libarchive.so.5 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 May 2011 08:56:00 -0000 Am 06.05.2011 20:37, schrieb Michael Hoffmann: > I cannot reconstruct why 'pkgdb -Fu' did not tried to prevent me from > using 'archivers/xz' furthermore, allthough it now seems to be marked as > IGNORE. I also don't know how this LD_LIBRARY thing ever popped up in > csh.cshrc, but it seems to persist there from at least 7.2. Newer systems > don't seem to have such an entry. Michael, It's apparently not designed to do that. Does "portmaster --check-depends" complain? It's a shell script from ports-mgmt/portmaster, just install and run it, it's lightweight (as opposed to portupgrade). Was your library path in csh created or suggested by pth-related ports? (Perhaps there could be something in the ports framework that warns if port duplicates base system functionality and suggests to deinstall a port that is no longer needed -- however, that would have to work also in cases where a newer ports is supposed to complement base system services. There were such situations for ssh, for instance.)