From owner-freebsd-stable@FreeBSD.ORG Thu Nov 20 18:09:09 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C208E6A; Thu, 20 Nov 2014 18:09:09 +0000 (UTC) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CF37E6; Thu, 20 Nov 2014 18:09:08 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id mc6so2880767lab.38 for ; Thu, 20 Nov 2014 10:09:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6mX6TZCTIW3r7loeE4NyCnXZo1IwfkWbIAtwOGiL6Gc=; b=VRS8JCFfJmrihJxKfjYV+zKkyjOvx3wHheHNWQRDjJZOVTodpduioa0MDlo4ljPFHS JeyHwRxjBid5U6EbQBu2pa+e4dEjiP9P/Edwrok1o3BqxAVJFfHtbS3zhHrMtdvSv0Wd MpmdSvSFN9Xg01UVfZhaGbSWjeVWVP6flv/2P6OoAMhWKgQUzZA+L7thT/rE2dBbn6YY pN7B12G0VMPCgo3iAVcQBmk/CT6ShfKXAJPLF1iKA1eLY7ALt+5KUQIHN/V2GgZTya4b BJUYYmXFkBNTgTINq0+1/7+BmDX6ItXcAfGbeZ7TKr2fxRTygrHxRPwW/NSYNcac4Fh8 RXew== MIME-Version: 1.0 X-Received: by 10.112.199.100 with SMTP id jj4mr12908842lbc.71.1416506946319; Thu, 20 Nov 2014 10:09:06 -0800 (PST) Received: by 10.25.170.66 with HTTP; Thu, 20 Nov 2014 10:09:06 -0800 (PST) In-Reply-To: <201411201145.01162.jhb@freebsd.org> References: <3870083.h75yLjv6As@ralph.baldwin.cx> <201411201145.01162.jhb@freebsd.org> Date: Thu, 20 Nov 2014 19:09:06 +0100 Message-ID: Subject: Re: SuperMicro IPMI/SOL and ipmitool troubles From: Andreas Nilsson To: John Baldwin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Daniel O'Connor , FreeBSD Stable Mailing List , Dmitry Morozovsky X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2014 18:09:09 -0000 On Thu, Nov 20, 2014 at 5:45 PM, John Baldwin wrote: > On Wednesday, November 19, 2014 5:15:05 pm Andreas Nilsson wrote: > > On Wed, Nov 19, 2014 at 8:05 PM, John Baldwin wrote: > > > > > On Wednesday, November 19, 2014 05:02:49 PM Andreas Nilsson wrote: > > > > On Wed, Nov 19, 2014 at 3:28 PM, Andreas Nilsson > > > > wrote: > > > > > On Fri, Nov 14, 2014 at 7:30 PM, Dmitry Morozovsky > > > > wrote: > > > > >> Daniel, > > > > >> > > > > >> nice to see you here too ;) > > > > >> > > > > >> On Fri, 14 Nov 2014, Daniel O'Connor wrote: > > > > >> > On 12 Nov 2014, at 19:43, Andreas Nilsson > > > wrote: > > > > >> > > unclear is the word for it :) And thanks for looking into > this. > > > > >> > > > > >> ipmi/ilo is > > > > >> > > > > >> > > important on a server os. > > > > >> > > > > >> > > I found a reference to it in a ML post: > > > > >> > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2013-February/072464.ht > > > > >> ml > > > > >> > > > > >> > I started that thread :) > > > > >> > I did get it working on the hardware I was using (Supermicro > X9SCL- > F > > > > >> > > > > >> and X8SIL-F) > > > > >> > > > > >> > I used the following BIOS settings > > > > >> > > > > > >> > ? Remote Access - Enabled > > > > >> > ? Serial Port Number - COM3 > > > > >> > ? Serial Port Mode - 115200, 8, n, 1 > > > > >> > ? Flow Control - Hardware > > > > >> > ? Redirection After BIOS POST - Always > > > > >> > ? Terminal Type - VT100 > > > > >> > ? VT-UTF8 Combo Key Support - Disabled > > > > >> > ? Sredir Memory Display Delay - No Delay > > > > >> > > > > > >> > And the following in loader.conf > > > > >> > # Give preference to VGA console > > > > >> > console="vidconsole,comconsole" > > > > >> > # Uncomment below and comment above to give serial console > > > preference > > > > >> > #console="comconsole,vidconsole" > > > > >> > comconsole_speed="115200" > > > > >> > boot_multicons="YES" > > > > >> > hint.uart.0.flags="0x0" > > > > >> > hint.uart.2.at="isa" > > > > >> > hint.uart.2.port="0x3E8" > > > > >> > hint.uart.2.flags="0x30" > > > > >> > > > > > >> > And this in /etc/ttys > > > > >> > # IPMI console > > > > >> > # Note: The Java console viewer doesn't seem to be very smart > as it > > > > >> > > > > >> doesn't > > > > >> > > > > >> > # properly support VT100 > > > > >> > cuau2 "/usr/libexec/getty 3wire.115200" vt100 on > secure > > > > >> > > > > > >> > I could then access it using ipmitool like so > > > > >> > ipmitool -H remoteip -U ADMIN -I lanplus sol activate > > > > >> > [login] > > > > >> > export TERM=xterm > > > > >> > > > > > >> > Note that I wanted vidconsole by default because mostly the > systems > > > > >> > > > > >> were used by people local to them, however we could break into the > > > loader > > > > >> and type 'set console=comconsole,vidconsole? and then get > everything > > > over > > > > >> the serial console for remote trouble shooting. > > > > >> > > > > >> > You may also wish to check the IPMI configuration via the web > > > interface > > > > >> > > > > >> - by default it will failover to port 0 and it has terrible > default > > > > >> passwords. I changed the passwords and forced it to use the > dedicated > > > > >> IPMI > > > > >> port even if nothing was connected to it. > > > > >> > > > > >> Well, I'm almost done with most of our SM server, even > concentrated > > > > >> console on > > > > >> our console server with such a simple config: > > > > >> > > > > >> ---- 8< ---- > > > > >> # ipmi/sol console template > > > > >> default ipmi { > > > > >> > > > > >> master localhost; > > > > >> type exec; > > > > >> exec /usr/local/bin/ipmitool -f > /usr/local/etc/ipmi-pass - > U > > > > >> > > > > >> root -I lanplus -H %.int sol activate; > > > > >> > > > > >> execsubst %=cs; > > > > >> #idletimeout 6h; > > > > >> > > > > >> break 0 { string "~B"; } > > > > >> > > > > >> } > > > > >> > > > > >> console gwn1 { include ipmi; } > > > > >> console gwn2 { include ipmi; } > > > > >> console gwn3 { include ipmi; } > > > > >> console gwn4 { include ipmi; } > > > > >> console gwn5 { include ipmi; } > > > > >> console gwn6 { include ipmi; } > > > > >> console gwn7 { include ipmi; } > > > > >> console gwn8 { include ipmi; } > > > > >> > > > > >> console gwc2 { include ipmi; } > > > > >> ---- 8< ---- > > > > >> > > > > >> This has console logging (including possible panics) as a surplus > > > > >> > > > > >> -- > > > > >> Sincerely, > > > > >> D.Marck [DM5020, MCK-RIPE, > > > DM3-RIPN] > > > > >> [ FreeBSD committer: > marck@FreeBSD.org > > > ] > > > > >> > > > > ------------------------------------------------------------------------ > > > > >> *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- > marck@rinet.ru > > > *** > > > > >> > > > > ------------------------------------------------------------------------ > > > > > > > > > > Hello again, > > > > > > > > > > Searching on hw.uart.console, I found: > > > > > > > > > http://lists.freebsd.org/pipermail/svn-src-head/2013-February/044641.html > > > > > , a very enlightening thread. > > > > > > > > > > Basically: "ohh, you want to use something other than COM1 and > tried > to > > > > > get away with just changing hint.uart stuff, which has worked for a > > > while, > > > > > ha, no way..." No heads up, nothing. > > > > > > > > > > Sorry to say jhb@ but is not a rare case. It is if not the > default, a > > > > > very common setup on every HP server with iLO, and it holds for > most > > > all > > > > > OOB style serial emulation I have ever had the (dis)pleasure of > working > > > > > with. > > > > > > This was done _specifically_ so you could use non-COM1 for both loader > and > > > kernel with one thing to change. That is, you don't use > hint.uart.X.flags > > > after this. I have used this with many SuperMicro servers that use > COM2 > > > and > > > COM3 because I wanted the entire path (boot loader and kernel) to work, > not > > > the kernel only. Having only the kernel means I can't break into the > > > loader > > > prompt to boot a different kernel, single user, etc. > > > > > > To clarify, are you using _different_ serial ports for the loader vs > the > > > kernel? That is the use case I considered to be rare. Every single > server > > > I have ever worked with (though not iLO, mostly Dell and SuperMicro) > uses > > > the > > > same COM port for serial redirection rather for SOL or via actual > cables. > > > I've yet to use a system that, for example, used COM1 for the loader > and > > > COM2 > > > for the kernel. You are saying that every HP server uses COM1 for the > > > loader > > > and COM2 for the kernel? > > > > > > -- > > > John Baldwin > > > > > > > Hello, > > > > For Supermicro: > > The box was in a rack when I started this, so I don't really know how the > > bios is setup, but as far as I can gather, getting > > "BTX loader yadda yadda" seems to always happen, on any COM-port. > > The actual output from loading modules and so on only happens with > specific > > settings. Here ( as in an answer to Dmitry I accidentally sent in > private ) > > things seems to be on different COMs: Setting: comconsole_port=0x2F8 > (com2) > > show loader and goes dark on handoff to kernel > > comconsole_port=0x3E8 (com3) show no loader and stays dark > > > > Neither of them gives me console even after successful boot. The only way > > to get serial is setting hw.uart.console="br:9600" explicitly. > > On the SuperMicro systems I have played with (X8DTU and the X9DRW > equivalent), > comconsole_port=0x2f8 worked for both BIOS, loader, and kernel, but for > BIOS > to work I did have to enable Console Redirection in the BIOS. For 2U > machines > (X8DAH at least, I think also for X9 equivalent), SOL was on COM3 instead > of > COM2, but using comconsole_port alone worked for BIOS, loader, and kernel. > > I also set the speed to 115200 (conspeed) which IIRC might have been the > default setting in the BIOS when Console Redirection was enabled via SOL. > You can check that remotely for SOL via ipmitool by seeing what baud rate > it is set to. I think if IPMI expects 115200 and you use 9600, that might > explain black output. > > -- > John Baldwin > Hello, I spent the day with new ( as in directly out of the box ) supermicro x9sc(a/i) machine, which as you say has SOL on com3. I don't have logs available now, but short summary was that only way to get it work was to set the hints.uart and then hw.uart.console="br:9600" ( or another speed ) or some bogus value. I'll get back to you tomorrow with the specifics ( the files contains kenv|grep uart , dmesg | grep uart , tail -4 /boot/loader.conf and results with loader (shown|not shown), kernel output (shown|not shown) and login prompt (shown|not shown). Best regards Andreas