From owner-freebsd-arm@freebsd.org Thu Jan 7 22:28:42 2016 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E2CDA673ED for ; Thu, 7 Jan 2016 22:28:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52E581A99 for ; Thu, 7 Jan 2016 22:28:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ig0-x236.google.com with SMTP id z14so45060669igp.1 for ; Thu, 07 Jan 2016 14:28:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=nOn7T1eos4wqU6p31YC9qeIqtbAx4mT5XW0iDJ2L8i0=; b=CkhEW5aaRhJ670p9+2p9OKCe10sOhGfXCvOGdJrsPLK8gXrq+Xk71XcLj8O3+gpn8k nTUWTPRzqq7LLkZKMc1uJvLBvI99cx6ASFueUizF/IPnYF+Ykb+BuhVCsiW3oYg8OLGG 2Ng/F1CI92BZPz7Kfr8Zf5fLWOFLlIvNdSk/oaN1RBj4h7ZjfmgVf13iWKL0oJ196FLD 97P+EuxnR17uzjuSmS5SGGyoBn3W2pNvOJnApQ/XUa3Jph94vKfMinTbD2sfKrXDPCjK HCSuv+y5YdQMkWTDqtGHyDoVwTcH5R2A15N89W7gVBKLuwSNsySM3TlRoHZGvx66w6hg nyzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=nOn7T1eos4wqU6p31YC9qeIqtbAx4mT5XW0iDJ2L8i0=; b=Gmn7YnT3D/5Qwin1xo6aL8mP7dHILOvBs0y4kbvtsu9w9ais/wpHqw+UomfL3g0R48 EAtuzEYAugd+Ajkqjh77JUaEvokCYtS73AHnUckI87J5pfBFErfY4VRp/qnRUbdFuhQg 5xaFawHHVLvYRXZNiIiO5EMEVfNoenYETr9u55nT5YoF7lJUoQxhzHBI4nODhTgYQA2Q SO1ME7FeT+pKubrrUBe37eg8VY252u+Ymzy6TpGZIl1HvIXu4d9B5eKjGRhRclw89xXd nDBipQIXFNtwg3iDEtI9QdDpks+TlL3sAdFIHzJzB49hLsFC3gpF1e7FJLiGg9DVh0WU F0Lg== X-Gm-Message-State: ALoCoQlDIIEIrdtLAW8XLeMrMNe6o0jXvZ12yq6M9QfHb9ZBl8WsI3+hJksYQhHPtdn+XHaBUF1DkRcPK72ryct3Xd+zwVw5eg== MIME-Version: 1.0 X-Received: by 10.50.156.35 with SMTP id wb3mr18662887igb.55.1452205721722; Thu, 07 Jan 2016 14:28:41 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.79.96.193 with HTTP; Thu, 7 Jan 2016 14:28:41 -0800 (PST) X-Originating-IP: [2601:280:4900:3700:fc79:e1ee:2b69:3672] In-Reply-To: References: <1452183170.1215.4.camel@freebsd.org> <1452196099.1215.12.camel@freebsd.org> <568EC4D8.7010106@selasky.org> <8B728C93-9C90-4821-A607-5D157F028812@dsl-only.net> <568ED810.8010309@selasky.org> <568ED92C.9070602@selasky.org> Date: Thu, 7 Jan 2016 15:28:41 -0700 X-Google-Sender-Auth: 7652ge9pduG6h7sFlKImEW5wm4A Message-ID: Subject: Re: FYI: various 11.0-CURRENT -r293227 (and older) hangs on arm (rpi2): a description of sorts From: Warner Losh To: Mark Millard Cc: Hans Petter Selasky , Ian Lepore , freebsd-arm Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jan 2016 22:28:42 -0000 4 page requests shouldn't hang the whole system. That should be more like hundreds or thousands depending on the tuning you've done. Warner On Thu, Jan 7, 2016 at 3:16 PM, Mark Millard wrote: > I'm top posting this change of information about the hang status seen via > gstat: > > After a long time the gstat -cod is showing a non-zero value in one place: > > L(q) for md0 is showing 4 now. > > (I've no clue when it changed. I do not expect that I missed the 4 before.) > > md0 is for the file-system based page file. That file is on the SSD, not > the sdcard. > > > === > Mark Millard > markmi at dsl-only.net > > On 2016-Jan-7, at 2:04 PM, Mark Millard wrote: > > > > > On 2016-Jan-7, at 1:31 PM, Hans Petter Selasky > wrote: > >> > >> On 01/07/16 22:26, Hans Petter Selasky wrote: > >>> On 01/07/16 21:20, Mark Millard wrote: > >>>> > >>>> On 2016-Jan-7, at 12:04 PM, Hans Petter Selasky > >>>> wrote: > >>>>> > >>>>> On 01/07/16 20:48, Ian Lepore wrote: > >>>>>> If the filesystems and swap space are on a usb drive, then maybe > it's > >>>>>> the usb subsystem that's hanging. The wait states you showed for > those > >>>>>> processes are consistant with what I've seen when all buffers get > >>>>>> backed up in a queue on one non-responsive or slow device. It may > be > >>>>>> that there's a way to get the system deadlocked when it's low on > >>>>>> buffers and there is memory pressure causing the swap to be used (I > >>>>>> generally run arms systems without any swap configured). > >>>>>> > >>>>>> Running gstat in another window while this is going on may give you > >>>>>> some insight into the situation. Beyond that I don't know what to > look > >>>>>> at, especially since you generally can't launch any new tools once > the > >>>>>> system gets into this kind of state. > >>>>>> > >>>>>> -- Ian > >>>>> > >>>>> Hi, > >>>>> > >>>>> All USB transfers towards disk devices have timeouts, so if something > >>>>> is hanging at USB level, you'll get a printout eventually. > >>>> > >>>> What sort of timescale after deadlock/live-lock is observed to > >>>> apparently have started does one have to wait in order to conclude > >>>> that the timeouts would have happened and so they do not apply to the > >>>> deadlock/live-lock? > >>>> > >>>>> The USB kernel processes needed for doing I/O transfers are not > >>>>> pinned to RAM. Can it happen if a USB process is swapped to disk, > >>>>> that the system cannot wakeup a swapped out process to get more swap? > >>>>> > >>>>> --HPS > >>>> > >>> > >>> Hi, > >>> > >>>> Wow. Could I use ddb to somehow check on the "USB kernel processes" > >>>> swap status when the overall context is deadlocked/live-locked? > >>> > >>> Are you able to run something like: > >>> > >>> ps auxwwH | grep usb > >>> > >>>> If yes, how? Otherwise something in top or some such display that I'd > >>> left running over the serial console would have to present useful > >>> information on the subject. Is there anything that would? > >>> > >> > >> Are you able to SSH into the box or ping it? > >> > >> --HPS > > > > Once the live-lock condition is reached no new processes can be created > as far as I can tell: the attempt will hang any process that attempts the > creation. > > > > I'd need "ps auxwwH" to be internally repeating to even get that much: > I'd have to start it before the live-lock happened and it would have to be > still running when the hang occurs, no on-going process creations involved. > > > > I'm not so sure that two communicating processes (ps and grep over a > pipe) would work but I can not get to even one new process so far. > > > > ssh sessions also hang, input and output stop for them fairly generally. > (Sometimes the context is such that ^t still works but shows no progress in > what it reports.) No new ssh connections are possible: "Operation timed > out". > > > > ping does respond normally: it is more of a live-lock status then a true > deadlock one overall. > > > > The serial console still outputs what it was already running if that > process does nothing that locks up. Changing what it is doing generally > locks it up too. > > > > Doing something like unplugging a usb keyboard or mouse or plugging one > in does show the expected messages via the console: it is more of a > live-lock status then a true deadlock one overall. > > > > I can get to ddb after the hang. But I do not know what I'd do with it > to find any useful information. > > > > > > As noted in another message: I used gstat instead of top on the serial > console: > > > >> gstat shows everything zero during a hang, even L(q) column. (Length of > queue?) > >> > >> I used: > >> > >> gstat -cod > >> > >> and had it running over the serial console port during the attempted > portmaster activity. > > > > > === > Mark Millard > markmi at dsl-only.net > > > > > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" >