From nobody Thu Feb 9 21:15:21 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCV561yvyz3nkSq; Thu, 9 Feb 2023 21:15:22 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCV560YGdz45FX; Thu, 9 Feb 2023 21:15:22 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675977322; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vSVAitlNFE71kAVvw8g1JCgf4JNVKKDmPq97llzUgY8=; b=MGrH4zC+TDFYcbDHHUA/+6ZEftaDKqGedLcH06gaQln4dmlsJV4MJksfg75/rJyl0NPFy9 HnMMsiNVfzyq+p4IzXdNOZA6aTyEL8wERKBAKnXxOq+SsOZFQQO1k0FMIfa/QBsiQqi98H ANWbebCNHwLhGpEH0IxtVqMEELspuBL1emnTjA5xue07ADkIUjZbMjrvVEKRGonvl7+Ega 0a/6yCVUess3Ud2q+XbG9IyK10/bpvBWWvrSPwnEAcrp8WN4Ycp2tEErRs+iYa2bt4eRSq OnR/rfFlEsOX2BqjxsNTNrjCYdl9PPbrwPfz4n7UYxUnttyAI7InlW7mDlE42g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675977322; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=vSVAitlNFE71kAVvw8g1JCgf4JNVKKDmPq97llzUgY8=; b=uwr91Ly28wFhgOSP2hQvvCEmME6mL1+R1lsDxY+64SqbGFh9IcN4nLoxwxUcPu4h85YTOj F7psXM4oQs9vdaaN+/GM8KweSWzkd3emCzoqDww8T/gMDsPjFcdgP6W0YX+7VUpHmaUXsV WGL3LQDtoyK1mMaDeqTdnQhHhkym0wo98nvDotAXiYeXAmU96Ct9QXoDu6R/l73OrLndE5 qaEaP+UNo9VCV+ABEtWpVr7t/v9rKqriQVcFouBFU0h5u3hEnhSIG9NEApMsUHCOnBhJ3j FJX4u2N8+ElJ3AVWH7zYNmF2NiQD2W80U/6Ry00PdUvxMzMAlF9HPmvH0vWcRg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675977322; a=rsa-sha256; cv=none; b=eIM7d7x05t7YE5Yif3xxaFPMmfKTbvEJUAN74zGMBU8azez3Acj4Ps6gHqu47gNr+Fsq/s hkaiYCL+oNHefKwmPFplq4Z3SJd2CTu4ZnMDxs5nsGPz7UcphqoK12mTGLbPZRh29NFqBg 5mkEWyHnLd69wJsUfnGRxt5cGFQfipBvcEonNOWHrvxLiZ1tB48JFAkgM6KuwrbYcjBtM6 aqR3qVOZBkEBs/yH1/eKLTxsYUIPJU1V43dRESab4j09ERhJ0rWfoLd+NAp1IXB6mpV/xO YSqgPvSKZBVigtnrTm1iItyW4Uq634F6oG26bqoXi4ba4zPhFMriUkRSgKi4+w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PCV556TvKzsYG; Thu, 9 Feb 2023 21:15:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 319LFLR0043036; Thu, 9 Feb 2023 21:15:21 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 319LFL7U043035; Thu, 9 Feb 2023 21:15:21 GMT (envelope-from git) Date: Thu, 9 Feb 2023 21:15:21 GMT Message-Id: <202302092115.319LFL7U043035@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 4130ea611fe0 - main - inpcb: Split in_pcblookup_hash_locked() and clean up a bit List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 4130ea611fe078035c4d85fcc223a7acd23de933 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=4130ea611fe078035c4d85fcc223a7acd23de933 commit 4130ea611fe078035c4d85fcc223a7acd23de933 Author: Mark Johnston AuthorDate: 2023-02-09 20:59:27 +0000 Commit: Mark Johnston CommitDate: 2023-02-09 21:15:03 +0000 inpcb: Split in_pcblookup_hash_locked() and clean up a bit Split the in_pcblookup_hash_locked() function into several independent subroutine calls, each of which does some kind of hash table lookup. This refactoring makes it easier to introduce variants of the lookup algorithm that behave differently depending on whether they are synchronized by SMR or the PCB database hash lock. While here, do some related cleanup: - Remove an unused ifnet parameter from internal functions. Keep it in external functions so that it can be used in the future to derive a v6 scopeid. - Reorder the parameters to in_pcblookup_lbgroup() to be consistent with the other lookup functions. - Remove an always-true check from in_pcblookup_lbgroup(): we can assume that we're performing a wildcard match. No functional change intended. Reviewed by: glebius Differential Revision: https://reviews.freebsd.org/D38364 --- sys/netinet/in_pcb.c | 228 ++++++++++++++++++++++++++----------------------- sys/netinet6/in6_pcb.c | 205 ++++++++++++++++++++++---------------------- sys/netinet6/in6_pcb.h | 2 +- 3 files changed, 226 insertions(+), 209 deletions(-) diff --git a/sys/netinet/in_pcb.c b/sys/netinet/in_pcb.c index 728178f1011c..3369f16e15c7 100644 --- a/sys/netinet/in_pcb.c +++ b/sys/netinet/in_pcb.c @@ -139,8 +139,7 @@ VNET_DEFINE(int, ipport_randomized) = 1; static struct inpcb *in_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport_arg, struct in_addr laddr, u_int lport_arg, - int lookupflags, struct ifnet *ifp, - uint8_t numa_domain); + int lookupflags, uint8_t numa_domain); #define RANGECHK(var, min, max) \ if ((var) < (min)) { (var) = (min); } \ @@ -785,14 +784,14 @@ in_pcb_lport_dest(struct inpcb *inp, struct sockaddr *lsa, u_short *lportp, if (lsa->sa_family == AF_INET) { tmpinp = in_pcblookup_hash_locked(pcbinfo, faddr, fport, laddr, lport, lookupflags, - NULL, M_NODOM); + M_NODOM); } #endif #ifdef INET6 if (lsa->sa_family == AF_INET6) { tmpinp = in6_pcblookup_hash_locked(pcbinfo, faddr6, fport, laddr6, lport, lookupflags, - NULL, M_NODOM); + M_NODOM); } #endif } else { @@ -1416,7 +1415,7 @@ in_pcbconnect_setup(struct inpcb *inp, struct sockaddr_in *sin, if (lport != 0) { if (in_pcblookup_hash_locked(inp->inp_pcbinfo, faddr, - fport, laddr, lport, 0, NULL, M_NODOM) != NULL) + fport, laddr, lport, 0, M_NODOM) != NULL) return (EADDRINUSE); } else { struct sockaddr_in lsin, fsin; @@ -2141,8 +2140,8 @@ in_pcblookup_lb_numa_match(const struct inpcblbgroup *grp, int domain) static struct inpcb * in_pcblookup_lbgroup(const struct inpcbinfo *pcbinfo, - const struct in_addr *laddr, uint16_t lport, const struct in_addr *faddr, - uint16_t fport, int lookupflags, int domain) + const struct in_addr *faddr, uint16_t fport, const struct in_addr *laddr, + uint16_t lport, int domain) { const struct inpcblbgrouphead *hdr; struct inpcblbgroup *grp; @@ -2185,8 +2184,7 @@ in_pcblookup_lbgroup(const struct inpcbinfo *pcbinfo, in_pcblookup_lb_numa_match(grp, domain)) { local_exact = grp; } - } else if (grp->il_laddr.s_addr == INADDR_ANY && - (lookupflags & INPLOOKUP_WILDCARD) != 0) { + } else if (grp->il_laddr.s_addr == INADDR_ANY) { if (injail) { if (jail_wild == NULL || in_pcblookup_lb_numa_match(grp, domain)) @@ -2213,32 +2211,16 @@ out: grp->il_inpcnt]); } -/* - * Lookup PCB in hash list, using pcbinfo tables. This variation assumes - * that the caller has either locked the hash list, which usually happens - * for bind(2) operations, or is in SMR section, which happens when sorting - * out incoming packets. - */ static struct inpcb * -in_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in_addr faddr, - u_int fport_arg, struct in_addr laddr, u_int lport_arg, int lookupflags, - struct ifnet *ifp, uint8_t numa_domain) +in_pcblookup_hash_exact(struct inpcbinfo *pcbinfo, struct in_addr faddr, + u_short fport, struct in_addr laddr, u_short lport) { struct inpcbhead *head; - struct inpcb *inp; - u_short fport = fport_arg, lport = lport_arg; + struct inpcb *inp, *match; - KASSERT((lookupflags & ~(INPLOOKUP_WILDCARD)) == 0, - ("%s: invalid lookup flags %d", __func__, lookupflags)); - KASSERT(faddr.s_addr != INADDR_ANY, - ("%s: invalid foreign address", __func__)); - KASSERT(laddr.s_addr != INADDR_ANY, - ("%s: invalid local address", __func__)); INP_HASH_LOCK_ASSERT(pcbinfo); - /* - * First look for an exact match. - */ + match = NULL; head = &pcbinfo->ipi_hashbase[INP_PCBHASH(&faddr, lport, fport, pcbinfo->ipi_hashmask)]; CK_LIST_FOREACH(inp, head, inp_hash) { @@ -2253,99 +2235,128 @@ in_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in_addr faddr, inp->inp_lport == lport) return (inp); } + return (match); +} - /* - * Then look for a wildcard match, if requested. - */ - if ((lookupflags & INPLOOKUP_WILDCARD) != 0) { - struct inpcb *local_wild = NULL, *local_exact = NULL; +static struct inpcb * +in_pcblookup_hash_wild_locked(struct inpcbinfo *pcbinfo, struct in_addr faddr, + u_short fport, struct in_addr laddr, u_short lport) +{ + struct inpcbhead *head; + struct inpcb *inp, *local_wild, *local_exact, *jail_wild; #ifdef INET6 - struct inpcb *local_wild_mapped = NULL; + struct inpcb *local_wild_mapped; #endif - struct inpcb *jail_wild = NULL; - int injail; - /* - * First see if an LB group matches the request before scanning - * all sockets on this port. - */ - inp = in_pcblookup_lbgroup(pcbinfo, &laddr, lport, &faddr, - fport, lookupflags, numa_domain); - if (inp != NULL) - return (inp); + INP_HASH_LOCK_ASSERT(pcbinfo); - /* - * Order of socket selection - we always prefer jails. - * 1. jailed, non-wild. - * 2. jailed, wild. - * 3. non-jailed, non-wild. - * 4. non-jailed, wild. - */ + /* + * Order of socket selection - we always prefer jails. + * 1. jailed, non-wild. + * 2. jailed, wild. + * 3. non-jailed, non-wild. + * 4. non-jailed, wild. + */ + head = &pcbinfo->ipi_hashbase[INP_PCBHASH_WILD(lport, + pcbinfo->ipi_hashmask)]; + local_wild = local_exact = jail_wild = NULL; +#ifdef INET6 + local_wild_mapped = NULL; +#endif + CK_LIST_FOREACH(inp, head, inp_hash) { + bool injail; - head = &pcbinfo->ipi_hashbase[INP_PCBHASH_WILD(lport, - pcbinfo->ipi_hashmask)]; - CK_LIST_FOREACH(inp, head, inp_hash) { #ifdef INET6 - /* XXX inp locking */ - if ((inp->inp_vflag & INP_IPV4) == 0) - continue; + /* XXX inp locking */ + if ((inp->inp_vflag & INP_IPV4) == 0) + continue; #endif - if (inp->inp_faddr.s_addr != INADDR_ANY || - inp->inp_lport != lport) - continue; + if (inp->inp_faddr.s_addr != INADDR_ANY || + inp->inp_lport != lport) + continue; - injail = prison_flag(inp->inp_cred, PR_IP4); - if (injail) { - if (prison_check_ip4_locked( - inp->inp_cred->cr_prison, &laddr) != 0) - continue; - } else { - if (local_exact != NULL) - continue; - } + injail = prison_flag(inp->inp_cred, PR_IP4) != 0; + if (injail) { + if (prison_check_ip4_locked(inp->inp_cred->cr_prison, + &laddr) != 0) + continue; + } else { + if (local_exact != NULL) + continue; + } - if (inp->inp_laddr.s_addr == laddr.s_addr) { - if (injail) - return (inp); - else - local_exact = inp; - } else if (inp->inp_laddr.s_addr == INADDR_ANY) { + if (inp->inp_laddr.s_addr == laddr.s_addr) { + if (injail) + return (inp); + local_exact = inp; + } else if (inp->inp_laddr.s_addr == INADDR_ANY) { #ifdef INET6 - /* XXX inp locking, NULL check */ - if (inp->inp_vflag & INP_IPV6PROTO) - local_wild_mapped = inp; - else + /* XXX inp locking, NULL check */ + if (inp->inp_vflag & INP_IPV6PROTO) + local_wild_mapped = inp; + else #endif - if (injail) - jail_wild = inp; - else - local_wild = inp; - } - } /* LIST_FOREACH */ - if (jail_wild != NULL) - return (jail_wild); - if (local_exact != NULL) - return (local_exact); - if (local_wild != NULL) - return (local_wild); + if (injail) + jail_wild = inp; + else + local_wild = inp; + } + } + if (jail_wild != NULL) + return (jail_wild); + if (local_exact != NULL) + return (local_exact); + if (local_wild != NULL) + return (local_wild); #ifdef INET6 - if (local_wild_mapped != NULL) - return (local_wild_mapped); + if (local_wild_mapped != NULL) + return (local_wild_mapped); #endif - } /* if ((lookupflags & INPLOOKUP_WILDCARD) != 0) */ - return (NULL); } /* - * Lookup PCB in hash list, using pcbinfo tables. This variation locks the - * hash list lock, and will return the inpcb locked (i.e., requires - * INPLOOKUP_LOCKPCB). + * Lookup PCB in hash list, using pcbinfo tables. This variation assumes + * that the caller has either locked the hash list, which usually happens + * for bind(2) operations, or is in SMR section, which happens when sorting + * out incoming packets. */ static struct inpcb * -in_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in_addr faddr, +in_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in_addr faddr, + u_int fport_arg, struct in_addr laddr, u_int lport_arg, int lookupflags, + uint8_t numa_domain) +{ + struct inpcb *inp; + const u_short fport = fport_arg, lport = lport_arg; + + KASSERT((lookupflags & ~INPLOOKUP_WILDCARD) == 0, + ("%s: invalid lookup flags %d", __func__, lookupflags)); + KASSERT(faddr.s_addr != INADDR_ANY, + ("%s: invalid foreign address", __func__)); + KASSERT(laddr.s_addr != INADDR_ANY, + ("%s: invalid local address", __func__)); + INP_HASH_LOCK_ASSERT(pcbinfo); + + inp = in_pcblookup_hash_exact(pcbinfo, faddr, fport, laddr, lport); + if (inp != NULL) + return (inp); + + if ((lookupflags & INPLOOKUP_WILDCARD) != 0) { + inp = in_pcblookup_lbgroup(pcbinfo, &faddr, fport, &laddr, + lport, numa_domain); + if (inp == NULL) { + inp = in_pcblookup_hash_wild_locked(pcbinfo, faddr, + fport, laddr, lport); + } + } + + return (inp); +} + +static struct inpcb * +in_pcblookup_hash_smr(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport, struct in_addr laddr, u_int lport, int lookupflags, - struct ifnet *ifp, uint8_t numa_domain) + uint8_t numa_domain) { struct inpcb *inp; @@ -2356,7 +2367,7 @@ in_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in_addr faddr, smr_enter(pcbinfo->ipi_smr); inp = in_pcblookup_hash_locked(pcbinfo, faddr, fport, laddr, lport, - lookupflags & INPLOOKUP_WILDCARD, ifp, numa_domain); + lookupflags & INPLOOKUP_WILDCARD, numa_domain); if (inp != NULL) { if (__predict_false(inp_smr_lock(inp, (lookupflags & INPLOOKUP_LOCKMASK)) == false)) @@ -2373,19 +2384,20 @@ in_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in_addr faddr, */ struct inpcb * in_pcblookup(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport, - struct in_addr laddr, u_int lport, int lookupflags, struct ifnet *ifp) + struct in_addr laddr, u_int lport, int lookupflags, + struct ifnet *ifp __unused) { - return (in_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport, - lookupflags, ifp, M_NODOM)); + return (in_pcblookup_hash_smr(pcbinfo, faddr, fport, laddr, lport, + lookupflags, M_NODOM)); } struct inpcb * in_pcblookup_mbuf(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport, struct in_addr laddr, u_int lport, int lookupflags, - struct ifnet *ifp, struct mbuf *m) + struct ifnet *ifp __unused, struct mbuf *m) { - return (in_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport, - lookupflags, ifp, m->m_pkthdr.numa_domain)); + return (in_pcblookup_hash_smr(pcbinfo, faddr, fport, laddr, lport, + lookupflags, m->m_pkthdr.numa_domain)); } #endif /* INET */ diff --git a/sys/netinet6/in6_pcb.c b/sys/netinet6/in6_pcb.c index 80540f2695e7..2b8e48090f28 100644 --- a/sys/netinet6/in6_pcb.c +++ b/sys/netinet6/in6_pcb.c @@ -449,12 +449,10 @@ in6_pcbconnect(struct inpcb *inp, struct sockaddr_in6 *sin6, struct ucred *cred, return (error); if (in6_pcblookup_hash_locked(pcbinfo, &sin6->sin6_addr, - sin6->sin6_port, - IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr) - ? &laddr6.sin6_addr : &inp->in6p_laddr, - inp->inp_lport, 0, NULL, M_NODOM) != NULL) { + sin6->sin6_port, IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr) ? + &laddr6.sin6_addr : &inp->in6p_laddr, inp->inp_lport, 0, + M_NODOM) != NULL) return (EADDRINUSE); - } if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr)) { if (inp->inp_lport == 0) { /* @@ -885,8 +883,8 @@ in6_pcblookup_lb_numa_match(const struct inpcblbgroup *grp, int domain) static struct inpcb * in6_pcblookup_lbgroup(const struct inpcbinfo *pcbinfo, - const struct in6_addr *laddr, uint16_t lport, const struct in6_addr *faddr, - uint16_t fport, int lookupflags, uint8_t domain) + const struct in6_addr *faddr, uint16_t fport, const struct in6_addr *laddr, + uint16_t lport, uint8_t domain) { const struct inpcblbgrouphead *hdr; struct inpcblbgroup *grp; @@ -929,8 +927,7 @@ in6_pcblookup_lbgroup(const struct inpcbinfo *pcbinfo, in6_pcblookup_lb_numa_match(grp, domain)) { local_exact = grp; } - } else if (IN6_IS_ADDR_UNSPECIFIED(&grp->il6_laddr) && - (lookupflags & INPLOOKUP_WILDCARD) != 0) { + } else if (IN6_IS_ADDR_UNSPECIFIED(&grp->il6_laddr)) { if (injail) { if (jail_wild == NULL || in6_pcblookup_lb_numa_match(grp, domain)) @@ -957,30 +954,19 @@ out: grp->il_inpcnt]); } -/* - * Lookup PCB in hash list. Used in in_pcb.c as well as here. - */ -struct inpcb * -in6_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, - u_int fport_arg, struct in6_addr *laddr, u_int lport_arg, - int lookupflags, struct ifnet *ifp, uint8_t numa_domain) +static struct inpcb * +in6_pcblookup_hash_exact(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, + u_short fport, struct in6_addr *laddr, u_short lport) { struct inpcbhead *head; - struct inpcb *inp; - u_short fport = fport_arg, lport = lport_arg; - - KASSERT((lookupflags & ~(INPLOOKUP_WILDCARD)) == 0, - ("%s: invalid lookup flags %d", __func__, lookupflags)); - KASSERT(!IN6_IS_ADDR_UNSPECIFIED(faddr), - ("%s: invalid foreign address", __func__)); - KASSERT(!IN6_IS_ADDR_UNSPECIFIED(laddr), - ("%s: invalid local address", __func__)); + struct inpcb *inp, *match; INP_HASH_LOCK_ASSERT(pcbinfo); /* * First look for an exact match. */ + match = NULL; head = &pcbinfo->ipi_hashbase[INP6_PCBHASH(faddr, lport, fport, pcbinfo->ipi_hashmask)]; CK_LIST_FOREACH(inp, head, inp_hash) { @@ -993,89 +979,107 @@ in6_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, inp->inp_lport == lport) return (inp); } + return (match); +} + +static struct inpcb * +in6_pcblookup_hash_wild_locked(struct inpcbinfo *pcbinfo, + struct in6_addr *faddr, u_short fport, struct in6_addr *laddr, + u_short lport) +{ + struct inpcbhead *head; + struct inpcb *inp, *jail_wild, *local_exact, *local_wild; /* - * Then look for a wildcard match, if requested. + * Order of socket selection - we always prefer jails. + * 1. jailed, non-wild. + * 2. jailed, wild. + * 3. non-jailed, non-wild. + * 4. non-jailed, wild. */ - if ((lookupflags & INPLOOKUP_WILDCARD) != 0) { - struct inpcb *local_wild = NULL, *local_exact = NULL; - struct inpcb *jail_wild = NULL; - int injail; + head = &pcbinfo->ipi_hashbase[INP_PCBHASH_WILD(lport, + pcbinfo->ipi_hashmask)]; + local_wild = local_exact = jail_wild = NULL; + CK_LIST_FOREACH(inp, head, inp_hash) { + bool injail; - /* - * First see if an LB group matches the request before scanning - * all sockets on this port. - */ - inp = in6_pcblookup_lbgroup(pcbinfo, laddr, lport, faddr, - fport, lookupflags, numa_domain); - if (inp != NULL) - return (inp); + /* XXX inp locking */ + if ((inp->inp_vflag & INP_IPV6) == 0) + continue; - /* - * Order of socket selection - we always prefer jails. - * 1. jailed, non-wild. - * 2. jailed, wild. - * 3. non-jailed, non-wild. - * 4. non-jailed, wild. - */ - head = &pcbinfo->ipi_hashbase[INP_PCBHASH_WILD(lport, - pcbinfo->ipi_hashmask)]; - CK_LIST_FOREACH(inp, head, inp_hash) { - /* XXX inp locking */ - if ((inp->inp_vflag & INP_IPV6) == 0) - continue; + if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr) || + inp->inp_lport != lport) { + continue; + } - if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr) || - inp->inp_lport != lport) { + injail = prison_flag(inp->inp_cred, PR_IP6) != 0; + if (injail) { + if (prison_check_ip6_locked( + inp->inp_cred->cr_prison, laddr) != 0) continue; - } + } else { + if (local_exact != NULL) + continue; + } - injail = prison_flag(inp->inp_cred, PR_IP6); - if (injail) { - if (prison_check_ip6_locked( - inp->inp_cred->cr_prison, laddr) != 0) - continue; - } else { - if (local_exact != NULL) - continue; - } + if (IN6_ARE_ADDR_EQUAL(&inp->in6p_laddr, laddr)) { + if (injail) + return (inp); + else + local_exact = inp; + } else if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr)) { + if (injail) + jail_wild = inp; + else + local_wild = inp; + } + } - if (IN6_ARE_ADDR_EQUAL(&inp->in6p_laddr, laddr)) { - if (injail) - return (inp); - else - local_exact = inp; - } else if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr)) { - if (injail) - jail_wild = inp; - else - local_wild = inp; - } - } /* LIST_FOREACH */ + if (jail_wild != NULL) + return (jail_wild); + if (local_exact != NULL) + return (local_exact); + if (local_wild != NULL) + return (local_wild); + return (NULL); +} - if (jail_wild != NULL) - return (jail_wild); - if (local_exact != NULL) - return (local_exact); - if (local_wild != NULL) - return (local_wild); - } /* if ((lookupflags & INPLOOKUP_WILDCARD) != 0) */ +struct inpcb * +in6_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, + u_int fport_arg, struct in6_addr *laddr, u_int lport_arg, + int lookupflags, uint8_t numa_domain) +{ + struct inpcb *inp; + u_short fport = fport_arg, lport = lport_arg; - /* - * Not found. - */ - return (NULL); + KASSERT((lookupflags & ~INPLOOKUP_WILDCARD) == 0, + ("%s: invalid lookup flags %d", __func__, lookupflags)); + KASSERT(!IN6_IS_ADDR_UNSPECIFIED(faddr), + ("%s: invalid foreign address", __func__)); + KASSERT(!IN6_IS_ADDR_UNSPECIFIED(laddr), + ("%s: invalid local address", __func__)); + + INP_HASH_LOCK_ASSERT(pcbinfo); + + inp = in6_pcblookup_hash_exact(pcbinfo, faddr, fport, laddr, lport); + if (inp != NULL) + return (inp); + + if ((lookupflags & INPLOOKUP_WILDCARD) != 0) { + inp = in6_pcblookup_lbgroup(pcbinfo, faddr, fport, laddr, + lport, numa_domain); + if (inp == NULL) { + inp = in6_pcblookup_hash_wild_locked(pcbinfo, faddr, + fport, laddr, lport); + } + } + return (inp); } -/* - * Lookup PCB in hash list, using pcbinfo tables. This variation locks the - * hash list lock, and will return the inpcb locked (i.e., requires - * INPLOOKUP_LOCKPCB). - */ static struct inpcb * -in6_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, +in6_pcblookup_hash_smr(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, u_int fport, struct in6_addr *laddr, u_int lport, int lookupflags, - struct ifnet *ifp, uint8_t numa_domain) + uint8_t numa_domain) { struct inpcb *inp; @@ -1086,7 +1090,7 @@ in6_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, smr_enter(pcbinfo->ipi_smr); inp = in6_pcblookup_hash_locked(pcbinfo, faddr, fport, laddr, lport, - lookupflags & INPLOOKUP_WILDCARD, ifp, numa_domain); + lookupflags & INPLOOKUP_WILDCARD, numa_domain); if (inp != NULL) { if (__predict_false(inp_smr_lock(inp, (lookupflags & INPLOOKUP_LOCKMASK)) == false)) @@ -1103,19 +1107,20 @@ in6_pcblookup_hash(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, */ struct inpcb * in6_pcblookup(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, u_int fport, - struct in6_addr *laddr, u_int lport, int lookupflags, struct ifnet *ifp) + struct in6_addr *laddr, u_int lport, int lookupflags, + struct ifnet *ifp __unused) { - return (in6_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport, - lookupflags, ifp, M_NODOM)); + return (in6_pcblookup_hash_smr(pcbinfo, faddr, fport, laddr, lport, + lookupflags, M_NODOM)); } struct inpcb * in6_pcblookup_mbuf(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, u_int fport, struct in6_addr *laddr, u_int lport, int lookupflags, - struct ifnet *ifp, struct mbuf *m) + struct ifnet *ifp __unused, struct mbuf *m) { - return (in6_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport, - lookupflags, ifp, m->m_pkthdr.numa_domain)); + return (in6_pcblookup_hash_smr(pcbinfo, faddr, fport, laddr, lport, + lookupflags, m->m_pkthdr.numa_domain)); } void diff --git a/sys/netinet6/in6_pcb.h b/sys/netinet6/in6_pcb.h index 0d3de62ddd54..800d26e8b3d6 100644 --- a/sys/netinet6/in6_pcb.h +++ b/sys/netinet6/in6_pcb.h @@ -84,7 +84,7 @@ struct inpcb * struct inpcb * in6_pcblookup_hash_locked(struct inpcbinfo *pcbinfo, struct in6_addr *faddr, u_int fport_arg, struct in6_addr *laddr, - u_int lport_arg, int lookupflags, struct ifnet *ifp, uint8_t); + u_int lport_arg, int lookupflags, uint8_t); struct inpcb * in6_pcblookup(struct inpcbinfo *, struct in6_addr *, u_int, struct in6_addr *, u_int, int,