Date: Sat, 23 Jan 2010 17:22:09 -0800 From: Sam Leffler <sam@errno.com> To: Russell Yount <russell.yount@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: atheros broadcast/multicast corruption with multiple hostap's Message-ID: <4B5BA0C1.8010901@errno.com> In-Reply-To: <c62ff5ca1001181929j3de9818ct785bfbb18883c55e@mail.gmail.com> References: <c62ff5ca0912302316o59c01ec5wd9efd008afd59c7f@mail.gmail.com> <4B521FC2.4050402@errno.com> <c62ff5ca1001171010v5ed0458dg7f066e4ef9a15de4@mail.gmail.com> <4B535AAE.3060308@errno.com> <c62ff5ca1001181929j3de9818ct785bfbb18883c55e@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Russell Yount wrote:
> On Sun, Jan 17, 2010 at 1:45 PM, Sam Leffler <sam@errno.com
> <mailto:sam@errno.com>> wrote:
>
> Russell Yount wrote:
>
>
>
> On Sat, Jan 16, 2010 at 3:21 PM, Sam Leffler <sam@errno.com
> <mailto:sam@errno.com> <mailto:sam@errno.com
> <mailto:sam@errno.com>>> wrote:
>
> Russell Yount wrote:
>
> It seems AP to client broadcasts/multicasts traffic is
> broken when using WPA2/802.11i with multiple hostapds in 8.0.
>
> Only the SSID associated with the last hostapd to be
> started has
> AP to client broadcasts/multicasts being delivered correctly.
>
> The AP and client are 8.0 freebsd systems althought I see
> same
> problems with windows XP as a client.
>
> The AP has 4 hostapds configured to use TLS with client
> certificates for
> authentication. (hostapd recompiled with
> HOSTAPD_CFLAGS=-DEAP_SERVER)
> The AP and client radio are shown as ath0: AR5212 mac 5.9
> RF5112
> phy 4.3
> in dmesg.
>
> Client authenticate using client certificates associate
> correctly
> to all 4 SSIDs. Unicast traffic flows correctly between
> clients
> and AP
> for all for 4 SSIDs. Client to AP broadcast/multicast
> traffic works
> on of 4 SSIDs. AP to client broadcast/multicast traffic
> only works
> on 1 of the SSIDs. I have documented this using ARP
> broadcasts,
> but normal IP broadcasts also observed to corrupted.
>
> When an ARP request is send through the AP to an
> associated client
> it seems to be trashed on any of the SSID except the one
> associated
> with the last hostapd to be started. Here is the output of
> client side
> tcpdump showing the problems.
>
> In the first client side tcpdump with the hostapd associated
> with the SSID
> being associaed with the last hostapd started and the traffic
> flowing
> normally.
>
> In the second client side tcpdump with the hostapd associated
> with the SSID
> being not the last hostapd started the ARP request is resent
> multiple times
> and appears corrupted.
>
> I would really like to find a fix for this.
> Any help would be greatly appreciated.
>
>
> This sounds like the crypto encap of the frame is clobbering the
> mbuf contents. You can verify this by setting up multiple
> vaps w/o
> WPA. If this is the problem look for the mbuf copy logic for
> mcast
> frames and make sure a deep copy is done.
>
> Sam
>
> The four VAPs broadcast traffic works find without WPA if I
> do not start hostapds on them
> I have been trying to discovery why broadcast traffic only
> works correctly on the VAP associated with the last hostapd to
> be started. I have move with VAP has the working broadcast
> traffic by restarting the hostapd
> associated with it.
> It would seem something in the WPA/802.1x layer initialization
> remembers which hostapd was started last and that affected the
> crypto encap.
> I keep looking but do not see any place in the code that could
> account for this.
> It seems the corrupt crypto encap also happens on broadcast
> between stations.
> Please correct me if I am wrong:
> but when using hostapd normally traffic is bridged withing the card.
> So if a station sends to the VAP a broadcast it is actaully
> sending a non- broadcast frame to the AP
> and the AP sends the frame to all the other stations.
>
>
> I told you waht the likely problem is. Look in the net80211 layer
> in the kernel for the problem.
>
> Sam
>
>
>
>
> I tried to find problems in mbuf corruption
> in ieee80211_output.c by placing
>
> m = m_unshare(m,M_NOWAIT);
> if (m == NULL) {
> IEEE80211_DPRINTF(vap, IEEE80211_MSG_OUTPUT,
> "%s: cannot get writable mbuf\n", __func__);
> return NULL;
> }
>
> at begining ieee80211_mbuf_adjust() and at
> beginning of ieee80211_encap() with no change
> in the broadcast traffic behaviour.
>
> I tried then to in ieee80211_crypto.c substituting
>
> flags |= IEEE80211_KEY_SWCRYPT;
>
> for the encryption capabilities test code
>
> if ((ic->ic_cryptocaps & (1<<cipher)) == 0) {
> IEEE80211_DPRINTF(vap, IEEE80211_MSG_CRYPTO,
> "%s: no h/w support for cipher %s, falling back to s/w\n",
> __func__, cip->ic_name);
> flags |= IEEE80211_KEY_SWCRYPT;
> }
>
> to force all the encryption to be done in software.
>
> This fixed the broadcast traffic problem but without
> hardware support its very slow and really loads machine.
> Enabling in the debug code to ath and net80211
>
> and enabled ATH_DEBUG_KEYCACHE in if_ath.c and
> IEEE80211_MSG_CRYPTO in net80211 code.
>
> It seems that all the VAPS sets the broadcast key for
> mac ff:ff:ff:ff:ff:ff in the ath device so I assume
> they conflict and the last one setting the key is the
> working one; that would explain why the last hostapd
> started is the only one with working broadcast code
> to clients.
>
> In if_ath.c the code
>
> if ((k->wk_flags & IEEE80211_KEY_GROUP) && sc->sc_mcastkey) {
> /*
> * Group keys on hardware that supports multicast frame
> * key search use a mac that is the sender's address with
> * the high bit set instead of the app-specified address.
> */
> IEEE80211_ADDR_COPY(gmac, bss->ni_macaddr);
> gmac[0] |= 0x80;
> mac = gmac;
> } else
> mac = k->wk_macaddr;
>
> seems to indicate that for multiple VAPs the ath chips
> needs to be able to distinguish between broadcast keys
> by using a permutation of VAPs bssid.
>
> But in if_athvar.h the code does not seem complete
> and sc->sc_mcastkey is also set false.
>
> #ifdef notyet
> #define ath_hal_hasmcastkeysearch(_ah) \
> (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, NULL) ==
> HAL_OK)
> #define ath_hal_getmcastkeysearch(_ah) \
> (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, NULL) ==
> HAL_OK)
> #else
> #define ath_hal_getmcastkeysearch(_ah) 0
> #endif
>
> I am using cards with an AR5212 which does seem to have
> multiple bssid support working so I hope they should also
> support mcastkeysearch capability. Maybe only some
> firmware revisions have this?
>
> Do you know what the status of the HAL code is for
> supporting this? Why it is commented out in if_athvar.h?
Good work analyzing things; been a long time since I looked at this. I
vaguely recall disabling mcastkey searching because of problems with
WEP. I'm surprised WPA is broken as that was a standard test case.
You can try enabling the notyet code and see if the right thing happens.
I don't see any indication of the mac rev for your part but I expect it
supports this as it was only very early parts that had issues.
If enabling the mcastkey search mechanism doesn't fix this I might've
broken things with changes to explicitly mark group keys when hostapd
plumbs their contents. I recall doing this for mwl which doesn't have
an indexed key table like ath (it uses the mac address of the local bss
and/or associated station to find the data structure where crypto keys
are stored).
I haven't looked at this stuff in a long time and can't setup a system
to test but if you keep pushing on this I'll try to help w/ advise.
Sam
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B5BA0C1.8010901>