From owner-freebsd-stable@FreeBSD.ORG Wed Nov 19 22:15:08 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4CD5E582; Wed, 19 Nov 2014 22:15:08 +0000 (UTC) Received: from mail-la0-x22f.google.com (mail-la0-x22f.google.com [IPv6:2a00:1450:4010:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9ED8DDF6; Wed, 19 Nov 2014 22:15:07 +0000 (UTC) Received: by mail-la0-f47.google.com with SMTP id hz20so1366725lab.34 for ; Wed, 19 Nov 2014 14:15:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Ivbgx+ZC0LED4idJVKXuOvJtDLukpnN/yy/aKyWyHus=; b=BH2fWoHolYI+rYJRDLijQE5F5WdTzcjjino8Fc2YGjTOOqI7PIbM5YG6FRv9W9Uvcw /vI2cYVTlpeQmFmILYxyBTujojlwDB84v7DmqGiX/5O7Lh+5k6fqYSHxZXrNUDmR/JrZ X8qyHZpMtY2ancIzTid+bpK1igRiXaNo1ZoSQoGzDNc2XTxRCRxkm6i98cQcIlG9M8/8 dw+JgiZ9uoAOi35RtBpjOs4HRPzAV5pwNnQ4tkAARZ2EeynSvy0tuWaIbChx8HNqMLkA LeCrqY7i+CLRek4i550j8ylsGt7mfAFy5Ez/0/vp4jyyDB0DgAR9o/weRAORhcP9yj1y fFNg== MIME-Version: 1.0 X-Received: by 10.112.130.132 with SMTP id oe4mr4474294lbb.82.1416435305611; Wed, 19 Nov 2014 14:15:05 -0800 (PST) Received: by 10.25.170.66 with HTTP; Wed, 19 Nov 2014 14:15:05 -0800 (PST) In-Reply-To: <3870083.h75yLjv6As@ralph.baldwin.cx> References: <3870083.h75yLjv6As@ralph.baldwin.cx> Date: Wed, 19 Nov 2014 23:15:05 +0100 Message-ID: Subject: Re: SuperMicro IPMI/SOL and ipmitool troubles From: Andreas Nilsson To: John Baldwin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Daniel O'Connor , FreeBSD Stable Mailing List , Dmitry Morozovsky X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2014 22:15:08 -0000 On Wed, Nov 19, 2014 at 8:05 PM, John Baldwin wrote: > On Wednesday, November 19, 2014 05:02:49 PM Andreas Nilsson wrote: > > On Wed, Nov 19, 2014 at 3:28 PM, Andreas Nilsson > wrote: > > > On Fri, Nov 14, 2014 at 7:30 PM, Dmitry Morozovsky > wrote: > > >> Daniel, > > >> > > >> nice to see you here too ;) > > >> > > >> On Fri, 14 Nov 2014, Daniel O'Connor wrote: > > >> > On 12 Nov 2014, at 19:43, Andreas Nilsson > wrote: > > >> > > unclear is the word for it :) And thanks for looking into this. > > >> > > >> ipmi/ilo is > > >> > > >> > > important on a server os. > > >> > > >> > > I found a reference to it in a ML post: > > >> > http://lists.freebsd.org/pipermail/freebsd-stable/2013-February/072464.ht > > >> ml > > >> > > >> > I started that thread :) > > >> > I did get it working on the hardware I was using (Supermicro X9SCL-F > > >> > > >> and X8SIL-F) > > >> > > >> > I used the following BIOS settings > > >> > > > >> > ? Remote Access - Enabled > > >> > ? Serial Port Number - COM3 > > >> > ? Serial Port Mode - 115200, 8, n, 1 > > >> > ? Flow Control - Hardware > > >> > ? Redirection After BIOS POST - Always > > >> > ? Terminal Type - VT100 > > >> > ? VT-UTF8 Combo Key Support - Disabled > > >> > ? Sredir Memory Display Delay - No Delay > > >> > > > >> > And the following in loader.conf > > >> > # Give preference to VGA console > > >> > console="vidconsole,comconsole" > > >> > # Uncomment below and comment above to give serial console > preference > > >> > #console="comconsole,vidconsole" > > >> > comconsole_speed="115200" > > >> > boot_multicons="YES" > > >> > hint.uart.0.flags="0x0" > > >> > hint.uart.2.at="isa" > > >> > hint.uart.2.port="0x3E8" > > >> > hint.uart.2.flags="0x30" > > >> > > > >> > And this in /etc/ttys > > >> > # IPMI console > > >> > # Note: The Java console viewer doesn't seem to be very smart as it > > >> > > >> doesn't > > >> > > >> > # properly support VT100 > > >> > cuau2 "/usr/libexec/getty 3wire.115200" vt100 on secure > > >> > > > >> > I could then access it using ipmitool like so > > >> > ipmitool -H remoteip -U ADMIN -I lanplus sol activate > > >> > [login] > > >> > export TERM=xterm > > >> > > > >> > Note that I wanted vidconsole by default because mostly the systems > > >> > > >> were used by people local to them, however we could break into the > loader > > >> and type 'set console=comconsole,vidconsole? and then get everything > over > > >> the serial console for remote trouble shooting. > > >> > > >> > You may also wish to check the IPMI configuration via the web > interface > > >> > > >> - by default it will failover to port 0 and it has terrible default > > >> passwords. I changed the passwords and forced it to use the dedicated > > >> IPMI > > >> port even if nothing was connected to it. > > >> > > >> Well, I'm almost done with most of our SM server, even concentrated > > >> console on > > >> our console server with such a simple config: > > >> > > >> ---- 8< ---- > > >> # ipmi/sol console template > > >> default ipmi { > > >> > > >> master localhost; > > >> type exec; > > >> exec /usr/local/bin/ipmitool -f /usr/local/etc/ipmi-pass -U > > >> > > >> root -I lanplus -H %.int sol activate; > > >> > > >> execsubst %=cs; > > >> #idletimeout 6h; > > >> > > >> break 0 { string "~B"; } > > >> > > >> } > > >> > > >> console gwn1 { include ipmi; } > > >> console gwn2 { include ipmi; } > > >> console gwn3 { include ipmi; } > > >> console gwn4 { include ipmi; } > > >> console gwn5 { include ipmi; } > > >> console gwn6 { include ipmi; } > > >> console gwn7 { include ipmi; } > > >> console gwn8 { include ipmi; } > > >> > > >> console gwc2 { include ipmi; } > > >> ---- 8< ---- > > >> > > >> This has console logging (including possible panics) as a surplus > > >> > > >> -- > > >> Sincerely, > > >> D.Marck [DM5020, MCK-RIPE, > DM3-RIPN] > > >> [ FreeBSD committer: marck@FreeBSD.org > ] > > >> > ------------------------------------------------------------------------ > > >> *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru > *** > > >> > ------------------------------------------------------------------------ > > > > > > Hello again, > > > > > > Searching on hw.uart.console, I found: > > > > http://lists.freebsd.org/pipermail/svn-src-head/2013-February/044641.html > > > , a very enlightening thread. > > > > > > Basically: "ohh, you want to use something other than COM1 and tried to > > > get away with just changing hint.uart stuff, which has worked for a > while, > > > ha, no way..." No heads up, nothing. > > > > > > Sorry to say jhb@ but is not a rare case. It is if not the default, a > > > very common setup on every HP server with iLO, and it holds for most > all > > > OOB style serial emulation I have ever had the (dis)pleasure of working > > > with. > > This was done _specifically_ so you could use non-COM1 for both loader and > kernel with one thing to change. That is, you don't use hint.uart.X.flags > after this. I have used this with many SuperMicro servers that use COM2 > and > COM3 because I wanted the entire path (boot loader and kernel) to work, not > the kernel only. Having only the kernel means I can't break into the > loader > prompt to boot a different kernel, single user, etc. > > To clarify, are you using _different_ serial ports for the loader vs the > kernel? That is the use case I considered to be rare. Every single server > I have ever worked with (though not iLO, mostly Dell and SuperMicro) uses > the > same COM port for serial redirection rather for SOL or via actual cables. > I've yet to use a system that, for example, used COM1 for the loader and > COM2 > for the kernel. You are saying that every HP server uses COM1 for the > loader > and COM2 for the kernel? > > -- > John Baldwin > Hello, For Supermicro: The box was in a rack when I started this, so I don't really know how the bios is setup, but as far as I can gather, getting "BTX loader yadda yadda" seems to always happen, on any COM-port. The actual output from loading modules and so on only happens with specific settings. Here ( as in an answer to Dmitry I accidentally sent in private ) things seems to be on different COMs: Setting: comconsole_port=0x2F8 (com2) show loader and goes dark on handoff to kernel comconsole_port=0x3E8 (com3) show no loader and stays dark Neither of them gives me console even after successful boot. The only way to get serial is setting hw.uart.console="br:9600" explicitly. For HP I asked a colleague, and we couldn't exactly remember ( it was quite a while since these boxes were deployed ), but our recollection was that serial setup was left as-is, which seems to be that "vsp" from iLO gives you a tty on 0x2F8. Setting that in comconsole_port in loader.conf works. Both In an effort of standardising stuff, we put hw-dependent stuff in loader.conf, ie hint.uart redirection to have the same entry in the ttys file. Having serial over ipmi/ilo go dark during remote upgrades is scary at best. Seeing that this change has such a potential, how come it is not mentioned in the release notes? Best regards Andreas