From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 17 10:34:42 2014 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6451FDCB; Thu, 17 Jul 2014 10:34:42 +0000 (UTC) Received: from zibbi.meraka.csir.co.za (zibbi.meraka.csir.co.za [146.64.24.58]) by mx1.freebsd.org (Postfix) with ESMTP id 55FF4276F; Thu, 17 Jul 2014 10:34:40 +0000 (UTC) Received: by zibbi.meraka.csir.co.za (Postfix, from userid 3973) id A8C0CB843; Thu, 17 Jul 2014 12:34:30 +0200 (SAST) Date: Thu, 17 Jul 2014 12:34:30 +0200 From: John Hay To: Adrian Chadd Subject: Re: CAMBRIA and more than one atheros card Message-ID: <20140717103430.GA88108@zibbi.meraka.csir.co.za> References: <20140714184209.GA21922@zibbi.meraka.csir.co.za> <20140714190946.GA49930@zibbi.meraka.csir.co.za> <20140714191416.GB49930@zibbi.meraka.csir.co.za> <20140714194932.GA51947@zibbi.meraka.csir.co.za> <20140714200015.GA54311@zibbi.meraka.csir.co.za> <20140714203707.GB54311@zibbi.meraka.csir.co.za> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140714203707.GB54311@zibbi.meraka.csir.co.za> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-wireless@freebsd.org" , "freebsd-embedded@freebsd.org" , Ian Lepore X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 10:34:42 -0000 On Mon, Jul 14, 2014 at 10:37:07PM +0200, John Hay wrote: > On Mon, Jul 14, 2014 at 01:02:03PM -0700, Adrian Chadd wrote: > > > > Ah, you're not even using it in 11n mode. Ok. > > > > Well, that's 72 mbufs allocated, and up to 200 more being allocated > > during transmit. Then there's the handful of pages being used for DMA > > descriptors. So yeah, over time that may become a problem. > > > > Try doubling the number of bounce buffers available and see what happens. > > The first boot looks good, all 3 aths came up. Bounce buffers just > after login: > > tst-11-arm:~ # sysctl hw.busdma > hw.busdma.total_bpages: 128 > hw.busdma.zone0.total_bpages: 128 > hw.busdma.zone0.free_bpages: 128 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.active_bpages: 0 > hw.busdma.zone0.total_bounced: 0 > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.lowaddr: 0x4000fff > hw.busdma.zone0.alignment: 4096 ... > Great! At last it seems like we are making progress. I'll leave it > till the morning and then do a few reboots and then some traffic. > > One question, should the active bounce pages keep growing or stabilize > somewhere? How do I know if I have to few or if there is a leak? > > Thanks Adrian for your help. I still ran out sometimes during a reboot, so I upped it to 512 and added a sysctl to show the max that active_bpages has been. Sometimes just after a reboot active_bpages is low and other times higher, I have seen 140. A few times during a reboot I see timeouts, which only a reboot seem to recover from. Can it be a left over config from the previous boot that does not get cleaned out? Or maybe traffic being received while the config is happening? Here is an example during the boot: #################### ... ath0: irq 28 at device 1.0 on pci0 alloc_bounce_pages: numpages 63 [ath] enabling AN_TOP2_FIXUP alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 63 alloc_bounce_pages: numpages 8 ath0: AR9220 mac 128.2 RF5133 phy 13.0 ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 ath1: irq 27 at device 2.0 on pci0 [ath] enabling AN_TOP2_FIXUP ath1: AR9220 mac 128.2 RF5133 phy 13.0 ath1: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 ath2: irq 26 at device 3.0 on pci0 ath2: AR5413 mac 10.5 RF5413 phy 6.1 ath2: 2GHz radio: 0x0000; 5GHz radio: 0x0063 ixpclk0: on ixp0 ixpiic0: on ixp0 ... /sbin/ifconfig.bin wlan0 create wlandev ath0 wlanmode adhoc country ZA wlan0: Ethernet address: 00:21:a4:35:70:42 /sbin/ifconfig.bin wlan0 inet6 ifdisabled /sbin/ifconfig.bin wlan0 -bgscan channel 140 ssid mesh bssid 02:8c:ca:fe:ca:00 /sbin/ifconfig.bin wlan0 inet6 -ifdisabled /sbin/ifconfig.bin wlan0 inet6 auto_linklocal /sbin/ifconfig.bin wlan0 inet6 -ifdisabled dev.ath.0.debug: 0 -> 8 /sbin/ifconfig.bin wlan0 up ath0: ath_init: if_flags 0x8803 ath0: ath_stop_locked: invalid 0 if_flags 0x8803 dev.ath.0.debug: 8 -> 0 ath0: hardware error; resetting ath0: 0x00000000 0x00000020 0x00000000, 0x00000000 0x00000000 0x00000000 ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast /sbin/ifconfig.bin wlan1 create wlandev ath1 wlanmode adhoc country ZA wlan1: Ethernet address: 00:21:a4:35:6c:96 /sbin/ifconfig.bin wlan1 inet6 ifdisabled /sbin/ifconfig.bin wlan1 -bgscan channel 136 ssid mesh bssid 02:88:ca:fe:ca:00 /sbin/ifconfig.bin wlan1 inet6 -ifdisabled /sbin/ifconfig.bin wlan1 inet6 auto_linklocal /sbin/ifconfig.bin wlan1 inet6 -ifdisabled dev.ath.1.debug: 0 -> 8 /sbin/ifconfig.bin wlan1 up ath1: ath_init: if_flags 0x8803 ath1: ath_stop_locked: invalid 0 if_flags 0x8803 R[ 0] (DS.V:0xcd027000 DS.P:0x1227000) L:01227060 D:00089000 * 00000000 00000800 0b802c2c 000000b2 0000175f 00000100 2f808080 00000000 00000000 00000000 00000003 R[ 0] (DS.V:0xcd027060 DS.P:0x1227060) L:012270c0 D:0008a000 00000000 00000800 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dev.ath.1.debug: 8 -> 0 /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast ifa_add_loopback_route: insertion failed: 17 ifconfig.bin: ioctl (SIOCAIFADDR): File exists /sbin/ifconfig.bin wlan2 create wlandev ath2 wlanmode adhoc country ZA wlan2: Ethernet address: 00:21:a4:32:38:c2 /sbin/ifconfig.bin wlan2 inet6 ifdisabled ath0: device timeout ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 /sbin/ifconfig.bin wlan2 -bgscan channel 13 ssid hotspot bssid 02:0d:ca:fe:ca:00 /sbin/ifconfig.bin wlan2 inet6 -ifdisabled /sbin/ifconfig.bin wlan2 inet6 auto_linklocal /sbin/ifconfig.bin wlan2 inet6 -ifdisabled dev.ath.2.debug: 0 -> 8 /sbin/ifconfig.bin wlan2 up ath2: ath_init: if_flags 0x8803 ath2: ath_stop_locked: invalid 0 if_flags 0x8803 dev.ath.2.debug: 8 -> 0 /sbin/ifconfig.bin wlan2 inet 10.56.244.1/24 alias /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 eui64 alias /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 alias anycast ... ath0: device timeout ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 ... tst-11-arm:~ # uptime 9:18AM up 2 mins, 1 user, load averages: 1.63, 0.70, 0.28 tst-11-arm:~ # sysctl hw.busdma hw.busdma.total_bpages: 512 hw.busdma.zone0.total_bpages: 512 hw.busdma.zone0.free_bpages: 375 hw.busdma.zone0.reserved_bpages: 0 hw.busdma.zone0.active_bpages: 137 hw.busdma.zone0.max_active_bpages: 137 hw.busdma.zone0.total_bounced: 1739 hw.busdma.zone0.total_deferred: 0 hw.busdma.zone0.lowaddr: 0x4000fff hw.busdma.zone0.alignment: 4096 tst-11-arm:~ # ath0: device timeout ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 tst-11-arm:~ # uptime 9:20AM up 4 mins, 1 user, load averages: 0.68, 0.68, 0.34 tst-11-arm:~ # sysctl hw.busdma hw.busdma.total_bpages: 512 hw.busdma.zone0.total_bpages: 512 hw.busdma.zone0.free_bpages: 326 hw.busdma.zone0.reserved_bpages: 0 hw.busdma.zone0.active_bpages: 186 hw.busdma.zone0.max_active_bpages: 190 hw.busdma.zone0.total_bounced: 6592 hw.busdma.zone0.total_deferred: 0 hw.busdma.zone0.lowaddr: 0x4000fff hw.busdma.zone0.alignment: 4096 tst-11-arm:~ # ath0: device timeout ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 ath0: device timeout ar5416StopDmaReceive: dma failed to stop in 10ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 #################### The 2 other interfaces did not seem to have a problem and after a reboot ath0/wlan0 was working fine again. John -- John Hay -- jhay@meraka.csir.co.za / jhay@meraka.org.za From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 17 19:17:20 2014 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5F2BC4F; Thu, 17 Jul 2014 19:17:20 +0000 (UTC) Received: from mail-qa0-x234.google.com (mail-qa0-x234.google.com [IPv6:2607:f8b0:400d:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DD2F274E; Thu, 17 Jul 2014 19:17:20 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id j15so2183644qaq.25 for ; Thu, 17 Jul 2014 12:17:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=P80TVwi0E2jblqpRt6/SX8aLJsmQT2ljOVF5qrl1i4U=; b=nKyK28bdIcp243pVAMPzcZKlGgZWJnTyiodFy1p2jggeR2zrIh2iQjlPCXIJxLGZZQ 0L9QYYJdB1uB8KOtbzd8rSmozdvEMBRjjcUVK1+7xbArzByG93XF1JVJD8qC5oR64nYP w2AjNdiFYgbM30zWqLycBaRf0Fpfj7QWcg9h5UajXtd8T42GzQweoehCE0JbPt/0Htv4 bXDdHaYsFrjqcyW7TxCgSboJAqEfwhxjQI/qVoU93d/lQAsumMJnqRrVn6fo8vTzCXqx Dm9WKL8PmOTLdtyvtP8t4Jqeu21kpGSO4UC9ucH5LnzGwssE0ize+RtNj6KVNdtUEsGW typQ== MIME-Version: 1.0 X-Received: by 10.140.93.161 with SMTP id d30mr30379155qge.53.1405624639542; Thu, 17 Jul 2014 12:17:19 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.202.193 with HTTP; Thu, 17 Jul 2014 12:17:19 -0700 (PDT) In-Reply-To: <20140717103430.GA88108@zibbi.meraka.csir.co.za> References: <20140714184209.GA21922@zibbi.meraka.csir.co.za> <20140714190946.GA49930@zibbi.meraka.csir.co.za> <20140714191416.GB49930@zibbi.meraka.csir.co.za> <20140714194932.GA51947@zibbi.meraka.csir.co.za> <20140714200015.GA54311@zibbi.meraka.csir.co.za> <20140714203707.GB54311@zibbi.meraka.csir.co.za> <20140717103430.GA88108@zibbi.meraka.csir.co.za> Date: Thu, 17 Jul 2014 12:17:19 -0700 X-Google-Sender-Auth: k-ZRM9hqtyr1G8dyihnY6DYNIyA Message-ID: Subject: Re: CAMBRIA and more than one atheros card From: Adrian Chadd To: John Hay Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-wireless@freebsd.org" , "freebsd-embedded@freebsd.org" , Ian Lepore X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 19:17:21 -0000 I've seen some weird cases in the past where there's just no interrupts making out after boot. I don't know why. Chances are that's what you're seeing - the frame is going out but there's no interrupts. Check vmstat -ia ; see if ath0 is generating interrupts. -a On 17 July 2014 03:34, John Hay wrote: > On Mon, Jul 14, 2014 at 10:37:07PM +0200, John Hay wrote: >> On Mon, Jul 14, 2014 at 01:02:03PM -0700, Adrian Chadd wrote: >> > >> > Ah, you're not even using it in 11n mode. Ok. >> > >> > Well, that's 72 mbufs allocated, and up to 200 more being allocated >> > during transmit. Then there's the handful of pages being used for DMA >> > descriptors. So yeah, over time that may become a problem. >> > >> > Try doubling the number of bounce buffers available and see what happens. >> >> The first boot looks good, all 3 aths came up. Bounce buffers just >> after login: >> >> tst-11-arm:~ # sysctl hw.busdma >> hw.busdma.total_bpages: 128 >> hw.busdma.zone0.total_bpages: 128 >> hw.busdma.zone0.free_bpages: 128 >> hw.busdma.zone0.reserved_bpages: 0 >> hw.busdma.zone0.active_bpages: 0 >> hw.busdma.zone0.total_bounced: 0 >> hw.busdma.zone0.total_deferred: 0 >> hw.busdma.zone0.lowaddr: 0x4000fff >> hw.busdma.zone0.alignment: 4096 > ... >> Great! At last it seems like we are making progress. I'll leave it >> till the morning and then do a few reboots and then some traffic. >> >> One question, should the active bounce pages keep growing or stabilize >> somewhere? How do I know if I have to few or if there is a leak? >> >> Thanks Adrian for your help. > > I still ran out sometimes during a reboot, so I upped it to 512 and added > a sysctl to show the max that active_bpages has been. Sometimes just > after a reboot active_bpages is low and other times higher, I have seen > 140. A few times during a reboot I see timeouts, which only a reboot > seem to recover from. Can it be a left over config from the previous boot > that does not get cleaned out? Or maybe traffic being received while the > config is happening? > > Here is an example during the boot: > > #################### > ... > ath0: irq 28 at device 1.0 on pci0 > alloc_bounce_pages: numpages 63 > [ath] enabling AN_TOP2_FIXUP > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 63 > alloc_bounce_pages: numpages 8 > ath0: AR9220 mac 128.2 RF5133 phy 13.0 > ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > ath1: irq 27 at device 2.0 on pci0 > [ath] enabling AN_TOP2_FIXUP > ath1: AR9220 mac 128.2 RF5133 phy 13.0 > ath1: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > ath2: irq 26 at device 3.0 on pci0 > ath2: AR5413 mac 10.5 RF5413 phy 6.1 > ath2: 2GHz radio: 0x0000; 5GHz radio: 0x0063 > ixpclk0: on ixp0 > ixpiic0: on ixp0 > ... > /sbin/ifconfig.bin wlan0 create wlandev ath0 wlanmode adhoc country ZA > wlan0: Ethernet address: 00:21:a4:35:70:42 > /sbin/ifconfig.bin wlan0 inet6 ifdisabled > /sbin/ifconfig.bin wlan0 -bgscan channel 140 ssid mesh bssid 02:8c:ca:fe:ca:00 > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > /sbin/ifconfig.bin wlan0 inet6 auto_linklocal > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > dev.ath.0.debug: 0 -> 8 > /sbin/ifconfig.bin wlan0 up > ath0: ath_init: if_flags 0x8803 > ath0: ath_stop_locked: invalid 0 if_flags 0x8803 > dev.ath.0.debug: 8 -> 0 > ath0: hardware error; resetting > ath0: 0x00000000 0x00000020 0x00000000, 0x00000000 0x00000000 0x00000000 > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > /sbin/ifconfig.bin wlan1 create wlandev ath1 wlanmode adhoc country ZA > wlan1: Ethernet address: 00:21:a4:35:6c:96 > /sbin/ifconfig.bin wlan1 inet6 ifdisabled > /sbin/ifconfig.bin wlan1 -bgscan channel 136 ssid mesh bssid 02:88:ca:fe:ca:00 > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > /sbin/ifconfig.bin wlan1 inet6 auto_linklocal > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > dev.ath.1.debug: 0 -> 8 > /sbin/ifconfig.bin wlan1 up > ath1: ath_init: if_flags 0x8803 > ath1: ath_stop_locked: invalid 0 if_flags 0x8803 > R[ 0] (DS.V:0xcd027000 DS.P:0x1227000) L:01227060 D:00089000 * > 00000000 00000800 0b802c2c 000000b2 > 0000175f 00000100 2f808080 00000000 00000000 00000000 00000003 > R[ 0] (DS.V:0xcd027060 DS.P:0x1227060) L:012270c0 D:0008a000 > 00000000 00000800 00000000 00000000 > 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > dev.ath.1.debug: 8 -> 0 > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > ifa_add_loopback_route: insertion failed: 17 > ifconfig.bin: ioctl (SIOCAIFADDR): File exists > /sbin/ifconfig.bin wlan2 create wlandev ath2 wlanmode adhoc country ZA > wlan2: Ethernet address: 00:21:a4:32:38:c2 > /sbin/ifconfig.bin wlan2 inet6 ifdisabled > ath0: device timeout > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > /sbin/ifconfig.bin wlan2 -bgscan channel 13 ssid hotspot bssid 02:0d:ca:fe:ca:00 > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > /sbin/ifconfig.bin wlan2 inet6 auto_linklocal > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > dev.ath.2.debug: 0 -> 8 > /sbin/ifconfig.bin wlan2 up > ath2: ath_init: if_flags 0x8803 > ath2: ath_stop_locked: invalid 0 if_flags 0x8803 > dev.ath.2.debug: 8 -> 0 > /sbin/ifconfig.bin wlan2 inet 10.56.244.1/24 alias > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 eui64 alias > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 alias anycast > ... > ath0: device timeout > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > ... > tst-11-arm:~ # uptime > 9:18AM up 2 mins, 1 user, load averages: 1.63, 0.70, 0.28 > tst-11-arm:~ # sysctl hw.busdma > hw.busdma.total_bpages: 512 > hw.busdma.zone0.total_bpages: 512 > hw.busdma.zone0.free_bpages: 375 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.active_bpages: 137 > hw.busdma.zone0.max_active_bpages: 137 > hw.busdma.zone0.total_bounced: 1739 > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.lowaddr: 0x4000fff > hw.busdma.zone0.alignment: 4096 > tst-11-arm:~ # ath0: device timeout > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > > tst-11-arm:~ # uptime > 9:20AM up 4 mins, 1 user, load averages: 0.68, 0.68, 0.34 > tst-11-arm:~ # sysctl hw.busdma > hw.busdma.total_bpages: 512 > hw.busdma.zone0.total_bpages: 512 > hw.busdma.zone0.free_bpages: 326 > hw.busdma.zone0.reserved_bpages: 0 > hw.busdma.zone0.active_bpages: 186 > hw.busdma.zone0.max_active_bpages: 190 > hw.busdma.zone0.total_bounced: 6592 > hw.busdma.zone0.total_deferred: 0 > hw.busdma.zone0.lowaddr: 0x4000fff > hw.busdma.zone0.alignment: 4096 > tst-11-arm:~ # ath0: device timeout > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > ath0: device timeout > ar5416StopDmaReceive: dma failed to stop in 10ms > AR_CR=0x00000024 > AR_DIAG_SW=0x42000020 > #################### > > The 2 other interfaces did not seem to have a problem and after a reboot > ath0/wlan0 was working fine again. > > John > -- > John Hay -- jhay@meraka.csir.co.za / jhay@meraka.org.za From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 17 19:24:37 2014 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73FFCE8C; Thu, 17 Jul 2014 19:24:37 +0000 (UTC) Received: from zibbi.meraka.csir.co.za (zibbi.meraka.csir.co.za [IPv6:2001:4200:7000:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 7DAC62849; Thu, 17 Jul 2014 19:24:36 +0000 (UTC) Received: by zibbi.meraka.csir.co.za (Postfix, from userid 3973) id 7F721B843; Thu, 17 Jul 2014 21:24:33 +0200 (SAST) Date: Thu, 17 Jul 2014 21:24:33 +0200 From: John Hay To: Adrian Chadd Subject: Re: CAMBRIA and more than one atheros card Message-ID: <20140717192433.GA35445@zibbi.meraka.csir.co.za> References: <20140714190946.GA49930@zibbi.meraka.csir.co.za> <20140714191416.GB49930@zibbi.meraka.csir.co.za> <20140714194932.GA51947@zibbi.meraka.csir.co.za> <20140714200015.GA54311@zibbi.meraka.csir.co.za> <20140714203707.GB54311@zibbi.meraka.csir.co.za> <20140717103430.GA88108@zibbi.meraka.csir.co.za> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-wireless@freebsd.org" , "freebsd-embedded@freebsd.org" , Ian Lepore X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 19:24:37 -0000 On Thu, Jul 17, 2014 at 12:17:19PM -0700, Adrian Chadd wrote: > I've seen some weird cases in the past where there's just no > interrupts making out after boot. I don't know why. > > Chances are that's what you're seeing - the frame is going out but > there's no interrupts. > > Check vmstat -ia ; see if ath0 is generating interrupts. I don't think vmstat -ia work properly. On a boot where ath works, this is what I see: tst-11-arm:~ # vmstat -ia interrupt total rate irq0: npe1 0 0 irq2: npe0 0 0 irq3: ixpqmgr0 0 0 irq4: ixpqmgr0 0 0 irq5: ixpclk0 0 0 Total 3520443 172 Ath0-2 do not pitch up and npe0 stays 0 even though packets have been sent and received on it. John > > > > -a > > > On 17 July 2014 03:34, John Hay wrote: > > On Mon, Jul 14, 2014 at 10:37:07PM +0200, John Hay wrote: > >> On Mon, Jul 14, 2014 at 01:02:03PM -0700, Adrian Chadd wrote: > >> > > >> > Ah, you're not even using it in 11n mode. Ok. > >> > > >> > Well, that's 72 mbufs allocated, and up to 200 more being allocated > >> > during transmit. Then there's the handful of pages being used for DMA > >> > descriptors. So yeah, over time that may become a problem. > >> > > >> > Try doubling the number of bounce buffers available and see what happens. > >> > >> The first boot looks good, all 3 aths came up. Bounce buffers just > >> after login: > >> > >> tst-11-arm:~ # sysctl hw.busdma > >> hw.busdma.total_bpages: 128 > >> hw.busdma.zone0.total_bpages: 128 > >> hw.busdma.zone0.free_bpages: 128 > >> hw.busdma.zone0.reserved_bpages: 0 > >> hw.busdma.zone0.active_bpages: 0 > >> hw.busdma.zone0.total_bounced: 0 > >> hw.busdma.zone0.total_deferred: 0 > >> hw.busdma.zone0.lowaddr: 0x4000fff > >> hw.busdma.zone0.alignment: 4096 > > ... > >> Great! At last it seems like we are making progress. I'll leave it > >> till the morning and then do a few reboots and then some traffic. > >> > >> One question, should the active bounce pages keep growing or stabilize > >> somewhere? How do I know if I have to few or if there is a leak? > >> > >> Thanks Adrian for your help. > > > > I still ran out sometimes during a reboot, so I upped it to 512 and added > > a sysctl to show the max that active_bpages has been. Sometimes just > > after a reboot active_bpages is low and other times higher, I have seen > > 140. A few times during a reboot I see timeouts, which only a reboot > > seem to recover from. Can it be a left over config from the previous boot > > that does not get cleaned out? Or maybe traffic being received while the > > config is happening? > > > > Here is an example during the boot: > > > > #################### > > ... > > ath0: irq 28 at device 1.0 on pci0 > > alloc_bounce_pages: numpages 63 > > [ath] enabling AN_TOP2_FIXUP > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 63 > > alloc_bounce_pages: numpages 8 > > ath0: AR9220 mac 128.2 RF5133 phy 13.0 > > ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > > ath1: irq 27 at device 2.0 on pci0 > > [ath] enabling AN_TOP2_FIXUP > > ath1: AR9220 mac 128.2 RF5133 phy 13.0 > > ath1: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > > ath2: irq 26 at device 3.0 on pci0 > > ath2: AR5413 mac 10.5 RF5413 phy 6.1 > > ath2: 2GHz radio: 0x0000; 5GHz radio: 0x0063 > > ixpclk0: on ixp0 > > ixpiic0: on ixp0 > > ... > > /sbin/ifconfig.bin wlan0 create wlandev ath0 wlanmode adhoc country ZA > > wlan0: Ethernet address: 00:21:a4:35:70:42 > > /sbin/ifconfig.bin wlan0 inet6 ifdisabled > > /sbin/ifconfig.bin wlan0 -bgscan channel 140 ssid mesh bssid 02:8c:ca:fe:ca:00 > > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > > /sbin/ifconfig.bin wlan0 inet6 auto_linklocal > > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > > dev.ath.0.debug: 0 -> 8 > > /sbin/ifconfig.bin wlan0 up > > ath0: ath_init: if_flags 0x8803 > > ath0: ath_stop_locked: invalid 0 if_flags 0x8803 > > dev.ath.0.debug: 8 -> 0 > > ath0: hardware error; resetting > > ath0: 0x00000000 0x00000020 0x00000000, 0x00000000 0x00000000 0x00000000 > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > > /sbin/ifconfig.bin wlan1 create wlandev ath1 wlanmode adhoc country ZA > > wlan1: Ethernet address: 00:21:a4:35:6c:96 > > /sbin/ifconfig.bin wlan1 inet6 ifdisabled > > /sbin/ifconfig.bin wlan1 -bgscan channel 136 ssid mesh bssid 02:88:ca:fe:ca:00 > > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > > /sbin/ifconfig.bin wlan1 inet6 auto_linklocal > > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > > dev.ath.1.debug: 0 -> 8 > > /sbin/ifconfig.bin wlan1 up > > ath1: ath_init: if_flags 0x8803 > > ath1: ath_stop_locked: invalid 0 if_flags 0x8803 > > R[ 0] (DS.V:0xcd027000 DS.P:0x1227000) L:01227060 D:00089000 * > > 00000000 00000800 0b802c2c 000000b2 > > 0000175f 00000100 2f808080 00000000 00000000 00000000 00000003 > > R[ 0] (DS.V:0xcd027060 DS.P:0x1227060) L:012270c0 D:0008a000 > > 00000000 00000800 00000000 00000000 > > 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > > dev.ath.1.debug: 8 -> 0 > > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > > ifa_add_loopback_route: insertion failed: 17 > > ifconfig.bin: ioctl (SIOCAIFADDR): File exists > > /sbin/ifconfig.bin wlan2 create wlandev ath2 wlanmode adhoc country ZA > > wlan2: Ethernet address: 00:21:a4:32:38:c2 > > /sbin/ifconfig.bin wlan2 inet6 ifdisabled > > ath0: device timeout > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > /sbin/ifconfig.bin wlan2 -bgscan channel 13 ssid hotspot bssid 02:0d:ca:fe:ca:00 > > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > > /sbin/ifconfig.bin wlan2 inet6 auto_linklocal > > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > > dev.ath.2.debug: 0 -> 8 > > /sbin/ifconfig.bin wlan2 up > > ath2: ath_init: if_flags 0x8803 > > ath2: ath_stop_locked: invalid 0 if_flags 0x8803 > > dev.ath.2.debug: 8 -> 0 > > /sbin/ifconfig.bin wlan2 inet 10.56.244.1/24 alias > > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 eui64 alias > > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 alias anycast > > ... > > ath0: device timeout > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > ... > > tst-11-arm:~ # uptime > > 9:18AM up 2 mins, 1 user, load averages: 1.63, 0.70, 0.28 > > tst-11-arm:~ # sysctl hw.busdma > > hw.busdma.total_bpages: 512 > > hw.busdma.zone0.total_bpages: 512 > > hw.busdma.zone0.free_bpages: 375 > > hw.busdma.zone0.reserved_bpages: 0 > > hw.busdma.zone0.active_bpages: 137 > > hw.busdma.zone0.max_active_bpages: 137 > > hw.busdma.zone0.total_bounced: 1739 > > hw.busdma.zone0.total_deferred: 0 > > hw.busdma.zone0.lowaddr: 0x4000fff > > hw.busdma.zone0.alignment: 4096 > > tst-11-arm:~ # ath0: device timeout > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > > > tst-11-arm:~ # uptime > > 9:20AM up 4 mins, 1 user, load averages: 0.68, 0.68, 0.34 > > tst-11-arm:~ # sysctl hw.busdma > > hw.busdma.total_bpages: 512 > > hw.busdma.zone0.total_bpages: 512 > > hw.busdma.zone0.free_bpages: 326 > > hw.busdma.zone0.reserved_bpages: 0 > > hw.busdma.zone0.active_bpages: 186 > > hw.busdma.zone0.max_active_bpages: 190 > > hw.busdma.zone0.total_bounced: 6592 > > hw.busdma.zone0.total_deferred: 0 > > hw.busdma.zone0.lowaddr: 0x4000fff > > hw.busdma.zone0.alignment: 4096 > > tst-11-arm:~ # ath0: device timeout > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > ath0: device timeout > > ar5416StopDmaReceive: dma failed to stop in 10ms > > AR_CR=0x00000024 > > AR_DIAG_SW=0x42000020 > > #################### > > > > The 2 other interfaces did not seem to have a problem and after a reboot > > ath0/wlan0 was working fine again. > > > > John > > -- > > John Hay -- jhay@meraka.csir.co.za / jhay@meraka.org.za From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 17 19:30:54 2014 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7EA9C2E8; Thu, 17 Jul 2014 19:30:54 +0000 (UTC) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 167CE2903; Thu, 17 Jul 2014 19:30:54 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id f51so2446646qge.11 for ; Thu, 17 Jul 2014 12:30:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=4twzPreickY8kuyo3Kn0IbYgcBWI2Sm8sPq0BLqUXIQ=; b=hnG0JiqIQOpq3iCw+sCYvZxdCb3aq40qUfPWCChC3cvc33MexTH9Xk6nIF+GwFE0Cs T69Ig4lEaTcgUUobzY8SxgGh+LWjPAaun6VnRhui+S7EnkQE6Y4cbJ2b5ZgpAtSSr5z2 if8uLpnFXL86k+4+crTFz8Ob+bGPgjN7tAScjFA7pFXcbMwbavuHAikzdfLBTNpvD1GJ WDsBu1RYusOhoJ/X58gSoQ8vrrxKPH0jktNX5hDE3zCeOjio7pKkYZ7NVfio2odKt5FC Xj1Yz+ezskJnnsacwcrsVEDN2+V8RuhHxObI3SxFqDMzxfRgQIufSdpegZGRaBlsNshv rFQw== MIME-Version: 1.0 X-Received: by 10.229.174.70 with SMTP id s6mr6999708qcz.29.1405625453172; Thu, 17 Jul 2014 12:30:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.202.193 with HTTP; Thu, 17 Jul 2014 12:30:53 -0700 (PDT) In-Reply-To: <20140717192433.GA35445@zibbi.meraka.csir.co.za> References: <20140714190946.GA49930@zibbi.meraka.csir.co.za> <20140714191416.GB49930@zibbi.meraka.csir.co.za> <20140714194932.GA51947@zibbi.meraka.csir.co.za> <20140714200015.GA54311@zibbi.meraka.csir.co.za> <20140714203707.GB54311@zibbi.meraka.csir.co.za> <20140717103430.GA88108@zibbi.meraka.csir.co.za> <20140717192433.GA35445@zibbi.meraka.csir.co.za> Date: Thu, 17 Jul 2014 12:30:53 -0700 X-Google-Sender-Auth: Yki6lzc7FAHF020S9K5DmvHHfLw Message-ID: Subject: Re: CAMBRIA and more than one atheros card From: Adrian Chadd To: John Hay Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-wireless@freebsd.org" , "freebsd-embedded@freebsd.org" , Ian Lepore X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 19:30:54 -0000 okay, I think that needs to be fixed first. I can't debug it otherwise. :) -a On 17 July 2014 12:24, John Hay wrote: > On Thu, Jul 17, 2014 at 12:17:19PM -0700, Adrian Chadd wrote: >> I've seen some weird cases in the past where there's just no >> interrupts making out after boot. I don't know why. >> >> Chances are that's what you're seeing - the frame is going out but >> there's no interrupts. >> >> Check vmstat -ia ; see if ath0 is generating interrupts. > > I don't think vmstat -ia work properly. On a boot where ath works, this > is what I see: > > tst-11-arm:~ # vmstat -ia > interrupt total rate > irq0: npe1 0 0 > irq2: npe0 0 0 > irq3: ixpqmgr0 0 0 > irq4: ixpqmgr0 0 0 > irq5: ixpclk0 0 0 > Total 3520443 172 > > Ath0-2 do not pitch up and npe0 stays 0 even though packets have been > sent and received on it. > > John > >> >> >> >> -a >> >> >> On 17 July 2014 03:34, John Hay wrote: >> > On Mon, Jul 14, 2014 at 10:37:07PM +0200, John Hay wrote: >> >> On Mon, Jul 14, 2014 at 01:02:03PM -0700, Adrian Chadd wrote: >> >> > >> >> > Ah, you're not even using it in 11n mode. Ok. >> >> > >> >> > Well, that's 72 mbufs allocated, and up to 200 more being allocated >> >> > during transmit. Then there's the handful of pages being used for DMA >> >> > descriptors. So yeah, over time that may become a problem. >> >> > >> >> > Try doubling the number of bounce buffers available and see what happens. >> >> >> >> The first boot looks good, all 3 aths came up. Bounce buffers just >> >> after login: >> >> >> >> tst-11-arm:~ # sysctl hw.busdma >> >> hw.busdma.total_bpages: 128 >> >> hw.busdma.zone0.total_bpages: 128 >> >> hw.busdma.zone0.free_bpages: 128 >> >> hw.busdma.zone0.reserved_bpages: 0 >> >> hw.busdma.zone0.active_bpages: 0 >> >> hw.busdma.zone0.total_bounced: 0 >> >> hw.busdma.zone0.total_deferred: 0 >> >> hw.busdma.zone0.lowaddr: 0x4000fff >> >> hw.busdma.zone0.alignment: 4096 >> > ... >> >> Great! At last it seems like we are making progress. I'll leave it >> >> till the morning and then do a few reboots and then some traffic. >> >> >> >> One question, should the active bounce pages keep growing or stabilize >> >> somewhere? How do I know if I have to few or if there is a leak? >> >> >> >> Thanks Adrian for your help. >> > >> > I still ran out sometimes during a reboot, so I upped it to 512 and added >> > a sysctl to show the max that active_bpages has been. Sometimes just >> > after a reboot active_bpages is low and other times higher, I have seen >> > 140. A few times during a reboot I see timeouts, which only a reboot >> > seem to recover from. Can it be a left over config from the previous boot >> > that does not get cleaned out? Or maybe traffic being received while the >> > config is happening? >> > >> > Here is an example during the boot: >> > >> > #################### >> > ... >> > ath0: irq 28 at device 1.0 on pci0 >> > alloc_bounce_pages: numpages 63 >> > [ath] enabling AN_TOP2_FIXUP >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 63 >> > alloc_bounce_pages: numpages 8 >> > ath0: AR9220 mac 128.2 RF5133 phy 13.0 >> > ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 >> > ath1: irq 27 at device 2.0 on pci0 >> > [ath] enabling AN_TOP2_FIXUP >> > ath1: AR9220 mac 128.2 RF5133 phy 13.0 >> > ath1: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 >> > ath2: irq 26 at device 3.0 on pci0 >> > ath2: AR5413 mac 10.5 RF5413 phy 6.1 >> > ath2: 2GHz radio: 0x0000; 5GHz radio: 0x0063 >> > ixpclk0: on ixp0 >> > ixpiic0: on ixp0 >> > ... >> > /sbin/ifconfig.bin wlan0 create wlandev ath0 wlanmode adhoc country ZA >> > wlan0: Ethernet address: 00:21:a4:35:70:42 >> > /sbin/ifconfig.bin wlan0 inet6 ifdisabled >> > /sbin/ifconfig.bin wlan0 -bgscan channel 140 ssid mesh bssid 02:8c:ca:fe:ca:00 >> > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled >> > /sbin/ifconfig.bin wlan0 inet6 auto_linklocal >> > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled >> > dev.ath.0.debug: 0 -> 8 >> > /sbin/ifconfig.bin wlan0 up >> > ath0: ath_init: if_flags 0x8803 >> > ath0: ath_stop_locked: invalid 0 if_flags 0x8803 >> > dev.ath.0.debug: 8 -> 0 >> > ath0: hardware error; resetting >> > ath0: 0x00000000 0x00000020 0x00000000, 0x00000000 0x00000000 0x00000000 >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias >> > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast >> > /sbin/ifconfig.bin wlan1 create wlandev ath1 wlanmode adhoc country ZA >> > wlan1: Ethernet address: 00:21:a4:35:6c:96 >> > /sbin/ifconfig.bin wlan1 inet6 ifdisabled >> > /sbin/ifconfig.bin wlan1 -bgscan channel 136 ssid mesh bssid 02:88:ca:fe:ca:00 >> > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled >> > /sbin/ifconfig.bin wlan1 inet6 auto_linklocal >> > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled >> > dev.ath.1.debug: 0 -> 8 >> > /sbin/ifconfig.bin wlan1 up >> > ath1: ath_init: if_flags 0x8803 >> > ath1: ath_stop_locked: invalid 0 if_flags 0x8803 >> > R[ 0] (DS.V:0xcd027000 DS.P:0x1227000) L:01227060 D:00089000 * >> > 00000000 00000800 0b802c2c 000000b2 >> > 0000175f 00000100 2f808080 00000000 00000000 00000000 00000003 >> > R[ 0] (DS.V:0xcd027060 DS.P:0x1227060) L:012270c0 D:0008a000 >> > 00000000 00000800 00000000 00000000 >> > 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >> > dev.ath.1.debug: 8 -> 0 >> > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias >> > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast >> > ifa_add_loopback_route: insertion failed: 17 >> > ifconfig.bin: ioctl (SIOCAIFADDR): File exists >> > /sbin/ifconfig.bin wlan2 create wlandev ath2 wlanmode adhoc country ZA >> > wlan2: Ethernet address: 00:21:a4:32:38:c2 >> > /sbin/ifconfig.bin wlan2 inet6 ifdisabled >> > ath0: device timeout >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > /sbin/ifconfig.bin wlan2 -bgscan channel 13 ssid hotspot bssid 02:0d:ca:fe:ca:00 >> > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled >> > /sbin/ifconfig.bin wlan2 inet6 auto_linklocal >> > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled >> > dev.ath.2.debug: 0 -> 8 >> > /sbin/ifconfig.bin wlan2 up >> > ath2: ath_init: if_flags 0x8803 >> > ath2: ath_stop_locked: invalid 0 if_flags 0x8803 >> > dev.ath.2.debug: 8 -> 0 >> > /sbin/ifconfig.bin wlan2 inet 10.56.244.1/24 alias >> > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 eui64 alias >> > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 alias anycast >> > ... >> > ath0: device timeout >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > ... >> > tst-11-arm:~ # uptime >> > 9:18AM up 2 mins, 1 user, load averages: 1.63, 0.70, 0.28 >> > tst-11-arm:~ # sysctl hw.busdma >> > hw.busdma.total_bpages: 512 >> > hw.busdma.zone0.total_bpages: 512 >> > hw.busdma.zone0.free_bpages: 375 >> > hw.busdma.zone0.reserved_bpages: 0 >> > hw.busdma.zone0.active_bpages: 137 >> > hw.busdma.zone0.max_active_bpages: 137 >> > hw.busdma.zone0.total_bounced: 1739 >> > hw.busdma.zone0.total_deferred: 0 >> > hw.busdma.zone0.lowaddr: 0x4000fff >> > hw.busdma.zone0.alignment: 4096 >> > tst-11-arm:~ # ath0: device timeout >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > >> > tst-11-arm:~ # uptime >> > 9:20AM up 4 mins, 1 user, load averages: 0.68, 0.68, 0.34 >> > tst-11-arm:~ # sysctl hw.busdma >> > hw.busdma.total_bpages: 512 >> > hw.busdma.zone0.total_bpages: 512 >> > hw.busdma.zone0.free_bpages: 326 >> > hw.busdma.zone0.reserved_bpages: 0 >> > hw.busdma.zone0.active_bpages: 186 >> > hw.busdma.zone0.max_active_bpages: 190 >> > hw.busdma.zone0.total_bounced: 6592 >> > hw.busdma.zone0.total_deferred: 0 >> > hw.busdma.zone0.lowaddr: 0x4000fff >> > hw.busdma.zone0.alignment: 4096 >> > tst-11-arm:~ # ath0: device timeout >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > ath0: device timeout >> > ar5416StopDmaReceive: dma failed to stop in 10ms >> > AR_CR=0x00000024 >> > AR_DIAG_SW=0x42000020 >> > #################### >> > >> > The 2 other interfaces did not seem to have a problem and after a reboot >> > ath0/wlan0 was working fine again. >> > >> > John >> > -- >> > John Hay -- jhay@meraka.csir.co.za / jhay@meraka.org.za From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 17 19:44:16 2014 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6687499; Thu, 17 Jul 2014 19:44:16 +0000 (UTC) Received: from zibbi.meraka.csir.co.za (zibbi.meraka.csir.co.za [146.64.24.58]) by mx1.freebsd.org (Postfix) with ESMTP id A494E29E6; Thu, 17 Jul 2014 19:44:15 +0000 (UTC) Received: by zibbi.meraka.csir.co.za (Postfix, from userid 3973) id 958B6B843; Thu, 17 Jul 2014 21:44:12 +0200 (SAST) Date: Thu, 17 Jul 2014 21:44:12 +0200 From: John Hay To: Adrian Chadd Subject: Re: CAMBRIA and more than one atheros card Message-ID: <20140717194412.GA37369@zibbi.meraka.csir.co.za> References: <20140714194932.GA51947@zibbi.meraka.csir.co.za> <20140714200015.GA54311@zibbi.meraka.csir.co.za> <20140714203707.GB54311@zibbi.meraka.csir.co.za> <20140717103430.GA88108@zibbi.meraka.csir.co.za> <20140717192433.GA35445@zibbi.meraka.csir.co.za> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-wireless@freebsd.org" , "freebsd-embedded@freebsd.org" , Ian Lepore X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 19:44:16 -0000 On Thu, Jul 17, 2014 at 12:30:53PM -0700, Adrian Chadd wrote: > okay, I think that needs to be fixed first. I can't debug it otherwise. :) vmstat -ia output looks the same on an Avila board, so it isn't Cambria specific. John > > > > -a > > > On 17 July 2014 12:24, John Hay wrote: > > On Thu, Jul 17, 2014 at 12:17:19PM -0700, Adrian Chadd wrote: > >> I've seen some weird cases in the past where there's just no > >> interrupts making out after boot. I don't know why. > >> > >> Chances are that's what you're seeing - the frame is going out but > >> there's no interrupts. > >> > >> Check vmstat -ia ; see if ath0 is generating interrupts. > > > > I don't think vmstat -ia work properly. On a boot where ath works, this > > is what I see: > > > > tst-11-arm:~ # vmstat -ia > > interrupt total rate > > irq0: npe1 0 0 > > irq2: npe0 0 0 > > irq3: ixpqmgr0 0 0 > > irq4: ixpqmgr0 0 0 > > irq5: ixpclk0 0 0 > > Total 3520443 172 > > > > Ath0-2 do not pitch up and npe0 stays 0 even though packets have been > > sent and received on it. > > > > John > > > >> > >> > >> > >> -a > >> > >> > >> On 17 July 2014 03:34, John Hay wrote: > >> > On Mon, Jul 14, 2014 at 10:37:07PM +0200, John Hay wrote: > >> >> On Mon, Jul 14, 2014 at 01:02:03PM -0700, Adrian Chadd wrote: > >> >> > > >> >> > Ah, you're not even using it in 11n mode. Ok. > >> >> > > >> >> > Well, that's 72 mbufs allocated, and up to 200 more being allocated > >> >> > during transmit. Then there's the handful of pages being used for DMA > >> >> > descriptors. So yeah, over time that may become a problem. > >> >> > > >> >> > Try doubling the number of bounce buffers available and see what happens. > >> >> > >> >> The first boot looks good, all 3 aths came up. Bounce buffers just > >> >> after login: > >> >> > >> >> tst-11-arm:~ # sysctl hw.busdma > >> >> hw.busdma.total_bpages: 128 > >> >> hw.busdma.zone0.total_bpages: 128 > >> >> hw.busdma.zone0.free_bpages: 128 > >> >> hw.busdma.zone0.reserved_bpages: 0 > >> >> hw.busdma.zone0.active_bpages: 0 > >> >> hw.busdma.zone0.total_bounced: 0 > >> >> hw.busdma.zone0.total_deferred: 0 > >> >> hw.busdma.zone0.lowaddr: 0x4000fff > >> >> hw.busdma.zone0.alignment: 4096 > >> > ... > >> >> Great! At last it seems like we are making progress. I'll leave it > >> >> till the morning and then do a few reboots and then some traffic. > >> >> > >> >> One question, should the active bounce pages keep growing or stabilize > >> >> somewhere? How do I know if I have to few or if there is a leak? > >> >> > >> >> Thanks Adrian for your help. > >> > > >> > I still ran out sometimes during a reboot, so I upped it to 512 and added > >> > a sysctl to show the max that active_bpages has been. Sometimes just > >> > after a reboot active_bpages is low and other times higher, I have seen > >> > 140. A few times during a reboot I see timeouts, which only a reboot > >> > seem to recover from. Can it be a left over config from the previous boot > >> > that does not get cleaned out? Or maybe traffic being received while the > >> > config is happening? > >> > > >> > Here is an example during the boot: > >> > > >> > #################### > >> > ... > >> > ath0: irq 28 at device 1.0 on pci0 > >> > alloc_bounce_pages: numpages 63 > >> > [ath] enabling AN_TOP2_FIXUP > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 63 > >> > alloc_bounce_pages: numpages 8 > >> > ath0: AR9220 mac 128.2 RF5133 phy 13.0 > >> > ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > >> > ath1: irq 27 at device 2.0 on pci0 > >> > [ath] enabling AN_TOP2_FIXUP > >> > ath1: AR9220 mac 128.2 RF5133 phy 13.0 > >> > ath1: 2GHz radio: 0x0000; 5GHz radio: 0x00c0 > >> > ath2: irq 26 at device 3.0 on pci0 > >> > ath2: AR5413 mac 10.5 RF5413 phy 6.1 > >> > ath2: 2GHz radio: 0x0000; 5GHz radio: 0x0063 > >> > ixpclk0: on ixp0 > >> > ixpiic0: on ixp0 > >> > ... > >> > /sbin/ifconfig.bin wlan0 create wlandev ath0 wlanmode adhoc country ZA > >> > wlan0: Ethernet address: 00:21:a4:35:70:42 > >> > /sbin/ifconfig.bin wlan0 inet6 ifdisabled > >> > /sbin/ifconfig.bin wlan0 -bgscan channel 140 ssid mesh bssid 02:8c:ca:fe:ca:00 > >> > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > >> > /sbin/ifconfig.bin wlan0 inet6 auto_linklocal > >> > /sbin/ifconfig.bin wlan0 inet6 -ifdisabled > >> > dev.ath.0.debug: 0 -> 8 > >> > /sbin/ifconfig.bin wlan0 up > >> > ath0: ath_init: if_flags 0x8803 > >> > ath0: ath_stop_locked: invalid 0 if_flags 0x8803 > >> > dev.ath.0.debug: 8 -> 0 > >> > ath0: hardware error; resetting > >> > ath0: 0x00000000 0x00000020 0x00000000, 0x00000000 0x00000000 0x00000000 > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > >> > /sbin/ifconfig.bin wlan0 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > >> > /sbin/ifconfig.bin wlan1 create wlandev ath1 wlanmode adhoc country ZA > >> > wlan1: Ethernet address: 00:21:a4:35:6c:96 > >> > /sbin/ifconfig.bin wlan1 inet6 ifdisabled > >> > /sbin/ifconfig.bin wlan1 -bgscan channel 136 ssid mesh bssid 02:88:ca:fe:ca:00 > >> > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > >> > /sbin/ifconfig.bin wlan1 inet6 auto_linklocal > >> > /sbin/ifconfig.bin wlan1 inet6 -ifdisabled > >> > dev.ath.1.debug: 0 -> 8 > >> > /sbin/ifconfig.bin wlan1 up > >> > ath1: ath_init: if_flags 0x8803 > >> > ath1: ath_stop_locked: invalid 0 if_flags 0x8803 > >> > R[ 0] (DS.V:0xcd027000 DS.P:0x1227000) L:01227060 D:00089000 * > >> > 00000000 00000800 0b802c2c 000000b2 > >> > 0000175f 00000100 2f808080 00000000 00000000 00000000 00000003 > >> > R[ 0] (DS.V:0xcd027060 DS.P:0x1227060) L:012270c0 D:0008a000 > >> > 00000000 00000800 00000000 00000000 > >> > 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > >> > dev.ath.1.debug: 8 -> 0 > >> > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 eui64 alias > >> > /sbin/ifconfig.bin wlan1 inet6 fd35:ac5b:b116:10:: prefixlen 64 alias anycast > >> > ifa_add_loopback_route: insertion failed: 17 > >> > ifconfig.bin: ioctl (SIOCAIFADDR): File exists > >> > /sbin/ifconfig.bin wlan2 create wlandev ath2 wlanmode adhoc country ZA > >> > wlan2: Ethernet address: 00:21:a4:32:38:c2 > >> > /sbin/ifconfig.bin wlan2 inet6 ifdisabled > >> > ath0: device timeout > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > /sbin/ifconfig.bin wlan2 -bgscan channel 13 ssid hotspot bssid 02:0d:ca:fe:ca:00 > >> > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > >> > /sbin/ifconfig.bin wlan2 inet6 auto_linklocal > >> > /sbin/ifconfig.bin wlan2 inet6 -ifdisabled > >> > dev.ath.2.debug: 0 -> 8 > >> > /sbin/ifconfig.bin wlan2 up > >> > ath2: ath_init: if_flags 0x8803 > >> > ath2: ath_stop_locked: invalid 0 if_flags 0x8803 > >> > dev.ath.2.debug: 8 -> 0 > >> > /sbin/ifconfig.bin wlan2 inet 10.56.244.1/24 alias > >> > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 eui64 alias > >> > /sbin/ifconfig.bin wlan2 inet6 fd35:ac5b:b116:38f4:: prefixlen 64 alias anycast > >> > ... > >> > ath0: device timeout > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > ... > >> > tst-11-arm:~ # uptime > >> > 9:18AM up 2 mins, 1 user, load averages: 1.63, 0.70, 0.28 > >> > tst-11-arm:~ # sysctl hw.busdma > >> > hw.busdma.total_bpages: 512 > >> > hw.busdma.zone0.total_bpages: 512 > >> > hw.busdma.zone0.free_bpages: 375 > >> > hw.busdma.zone0.reserved_bpages: 0 > >> > hw.busdma.zone0.active_bpages: 137 > >> > hw.busdma.zone0.max_active_bpages: 137 > >> > hw.busdma.zone0.total_bounced: 1739 > >> > hw.busdma.zone0.total_deferred: 0 > >> > hw.busdma.zone0.lowaddr: 0x4000fff > >> > hw.busdma.zone0.alignment: 4096 > >> > tst-11-arm:~ # ath0: device timeout > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > > >> > tst-11-arm:~ # uptime > >> > 9:20AM up 4 mins, 1 user, load averages: 0.68, 0.68, 0.34 > >> > tst-11-arm:~ # sysctl hw.busdma > >> > hw.busdma.total_bpages: 512 > >> > hw.busdma.zone0.total_bpages: 512 > >> > hw.busdma.zone0.free_bpages: 326 > >> > hw.busdma.zone0.reserved_bpages: 0 > >> > hw.busdma.zone0.active_bpages: 186 > >> > hw.busdma.zone0.max_active_bpages: 190 > >> > hw.busdma.zone0.total_bounced: 6592 > >> > hw.busdma.zone0.total_deferred: 0 > >> > hw.busdma.zone0.lowaddr: 0x4000fff > >> > hw.busdma.zone0.alignment: 4096 > >> > tst-11-arm:~ # ath0: device timeout > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > ath0: device timeout > >> > ar5416StopDmaReceive: dma failed to stop in 10ms > >> > AR_CR=0x00000024 > >> > AR_DIAG_SW=0x42000020 > >> > #################### > >> > > >> > The 2 other interfaces did not seem to have a problem and after a reboot > >> > ath0/wlan0 was working fine again. > >> > > >> > John > >> > -- > >> > John Hay -- jhay@meraka.csir.co.za / jhay@meraka.org.za