From owner-freebsd-net@freebsd.org Tue Aug 8 07:04:43 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA8BDDC6BF4 for ; Tue, 8 Aug 2017 07:04:43 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x241.google.com (mail-wr0-x241.google.com [IPv6:2a00:1450:400c:c0c::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9285D72C2A for ; Tue, 8 Aug 2017 07:04:43 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x241.google.com with SMTP id c24so1819883wra.2 for ; Tue, 08 Aug 2017 00:04:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=3rBWCzvg3ziXZB77GtgvLomEdcEMu8lJp9SLdrrzh9A=; b=L2cUJr9ArzLt9EvUp2x/pkFOTe2BNnFHnwfgWUaNl1O5iELyT2namYgSsEF341U18e OqO0ao5meXo33wFXUpsWivRL9MEzwzBjrFSCM5WEkGTBBAn7Sm4OvUhSdL7NoVLkTyUi 8O23ilFuFjmGxPsSb+C5nz1blwWguKoKpjo3p+xpjvF3Em0NIIJDjkiNK+UkyKa7scSO PKcMVx/ce/Ba4SsytOoIC2pb59GyIdeCzBEqJO6ZuNuMVWsVGGPzAYtK+XbbhipaHrNF 7G9EzItZj8B29AJWQkUVWFgQtfC/PUC/WXvyK9gYNjJhGfOxmw2kJ7JD2xax/MCtNOEd rUEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=3rBWCzvg3ziXZB77GtgvLomEdcEMu8lJp9SLdrrzh9A=; b=iTIWH2Kl+4ZE9dURnmcMPlXyTYPXjQulHlx70UV4iiH36jAOWzvTRLaWjvFKDpWKwU GJgoUKl5Dh3EsNsVQBADsWwQt2M5+HpwnmdcEITSsS6aN0FoTRRPwZhPQt5xDhfNxw8P ZNiKe2lXH9RKEdhxEpXN/IN4vvuppCeAhhTXs4Sdoq9QH4zrS0JMx57tdORfMp8ZEq6Y bSuNdaVM167HmBJB/tD+uhn1N+QX8uhgxqbmHI+277BKWMPtUNDoFkzYMwjKdIeipqNk /Y0NMMPwN8UkrNpA+ExLq67E/QEDhrY1QlogAAXqWyNZxmfTH48Fki/QDND7Vzf8pdNs TTYw== X-Gm-Message-State: AHYfb5gJOAlyAyXDqdSWnrxhHu4VS6ENlTISQKtqwwGCz0pEpENcWwt0 quOyRaOlDg7COHvwvLg= X-Received: by 10.223.177.143 with SMTP id q15mr1917786wra.200.1502175881732; Tue, 08 Aug 2017 00:04:41 -0700 (PDT) Received: from ben.home (LFbn-1-6951-179.w90-116.abo.wanadoo.fr. [90.116.132.179]) by smtp.gmail.com with ESMTPSA id 23sm651285wrz.8.2017.08.08.00.04.40 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 08 Aug 2017 00:04:41 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mlx4en, timer irq @100%... From: Ben RUBSON In-Reply-To: Date: Tue, 8 Aug 2017 09:04:39 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <4C91C6E5-0725-42E7-9813-1F3ACF3DDD6E@gmail.com> References: <72b9de84-5572-3737-b274-34d2c5cdf634@selasky.org> <91DCB96E-4C08-44C5-94E7-E7C686DEFE5F@gmail.com> <4DF74CB8-23D2-4CCF-B699-5B86DAEA65E5@gmail.com> <40602CEA-D417-4E5B-8C68-916958D49A0B@gmail.com> <9c306f10-7c05-d28d-e551-a930603aaafa@selasky.org> <896dd782-cb2c-0259-65d1-b00daae452de@FreeBSD.org> <0DB9F6FF-8BC9-48F5-B359-AC1905B9EB06@gmail.com> <7f14c95d-1ef8-bf82-c469-e6566c3aba66@selasky.org> <76A5EE7E-1D2E-46B4-86F1-F219C3DCE6EA@gmail.com> To: FreeBSD Net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Aug 2017 07:04:44 -0000 > On 08 Aug 2017, at 08:51, Hans Petter Selasky wrote: >=20 > On 08/08/17 01:52, Ben RUBSON wrote: >>> On 07 Aug 2017, at 19:57, Hans Petter Selasky = wrote: >>>=20 >>> On 08/07/17 19:19, Ben RUBSON wrote: >>>>> On 07 Aug 2017, at 18:19, Matt Joras wrote: >>>>>=20 >>>>> On 08/07/2017 09:11, Hans Petter Selasky wrote: >>>>>> Hi, >>>>>>=20 >>>>>> Try to enter "kgdb" and run: >>>>>>=20 >>>>>> thread apply all bt >>>>>>=20 >>>>>> Look for the callout function in question. >>>>>>=20 >>>>>> --HPS >>>>>>=20 >>>>> If you don't have a way to attach kgdb handy you could also break = into >>>>> ddb(4) and run "alltrace". Though gdb would be more useful for an >>>>> ongoing session if we need more than the backtrace since you could >>>>> switch to that thread and investigate it directly. >>>>>=20 >>>> Hi Hans & Matt, >>>> Thank you for your answers, glad to hear from you :) >>>> So here is the full kgdb(thread apply all bt) command log : >>>> https://benrubson.github.io/kgdb.log >>>> We found the faulty thread : >>>> # procstat -ak | grep "swi4.*tcp" >>>> 12 100029 intr swi4: clock (0) tcp_tw_2msl_scan = pfslowtimo softclock_call_cc softclock intr_event_execute_handlers = ithread_loop fork_exit fork_trampoline >>>> # kgdb >>>> (...) >>>> Thread 747 (Thread 100029): >>>> #0 sched_switch (td=3D0xfffff8000f337500, = newtd=3D0xfffff8010e144000, flags=3D) at = /usr/src/sys/kern/sched_ule.c:1973 >>>> #1 0xfffffe1000f92d80 in ?? () >>>> #2 0xfffffe0f8f74b6e0 in ?? () >>>> #3 0xffffffff810bd274 in handleevents (now=3D, fake=3DError accessing memory address 0xffffffffffffffcc: Bad = address. >>>> ) at /usr/src/sys/kern/kern_clocksource.c:223 >>>> Previous frame inner to this frame (corrupt stack?) >>>> (...) >>>> Of course let me know if you need further info. >>>=20 >>> Can you try to dump "td": >>>=20 >>> set print pretty on >>> thread 747 >>> frame 0 >>> print *td >>>=20 >>> It might give some more clues. >> Here it is : >> https://benrubson.github.io/td.log >> Thx ! >=20 > Can you show output from: >=20 > vmstat -z >=20 > Can you from kgdb do: >=20 > print V_twq_2msl >=20 > And follow the next link field and see where it goes? Here is vmstat -z : https://benrubson.github.io/vmstatz.log "print V_twq_2msl" returns the following : No symbol "V_twq_2msl" in current context. Even if I rerun this before (as I exited kgdb) : set print pretty on thread 747 frame 0 Ben=