Date: Mon, 24 Mar 2025 15:33:36 +0900 From: Kristof Provost <kp@FreeBSD.org> To: Gleb Smirnoff <glebius@freebsd.org> Cc: igoro@freebsd.org, src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 0849f1634a70 - main - tests/netinet: add test for IP_MULTICAST_IF Message-ID: <EF2BE5F9-C9D0-46C6-AE19-C97478A7CE44@FreeBSD.org> In-Reply-To: <Z-D3yP4O8sKBvlbM@cell.glebi.us> References: <202503222340.52MNekCX071219@gitrepo.freebsd.org> <7440BE85-66E8-4D1A-AC15-8B944F5C2951@FreeBSD.org> <Z-D3yP4O8sKBvlbM@cell.glebi.us>
next in thread | previous in thread | raw e-mail | index | archive | help
On 24 Mar 2025, at 15:12, Gleb Smirnoff wrote: > On Mon, Mar 24, 2025 at 11:45:55AM +0900, Kristof Provost wrote: > K> > tests/netinet: add test for IP_MULTICAST_IF > K> > > K> It bugs me a little that we have to jump through hoops to get a C pr= ogram > K> running in a specific jail/network setup. > K> It=E2=80=99s somewhat common for that to happen, and I=E2=80=99m sur= e there=E2=80=99d be more cases > K> if it were easier to do (e.g. the getaddrinfo tests you=E2=80=99ve b= een working on > K> could be even more useful with a custom DNS server, so in their own = vnet > K> jail). > > It bugs me, too. :) Igor has already pushed changes that allow you to s= pecify > in the Makefile: > > TEST_METADATA.foo+=3D "execenv=3D"jail" execenv_jail_params=3D"vnet" > The primary intent behind that was to allow for parallel test execution. A lot of tests, and a lot of this is my fault, use the same network range= s and jail names, so if we run them at the same time they=E2=80=99d confl= ict and break in unexpected ways. Running them inside a vnet jail avoids this issue. I was actually looking at adding this to your multicast tests, but you do= n=E2=80=99t assign addresses on the host=E2=80=99s epair, so there=E2=80=99= s not much point. > Then this gets into Kyuafile after build. This is a step forward, but n= ot > enough. First problem is that the created jail is empty and doesn't hav= e any > interfaces. Second problem is that the specified environment affects al= l tests > inside one test file. > > Ideally we'd like to have a declarative language to describe jail envir= onment > for a test. This is basically what vnet.subr does for shell tests and = what > atf_python.sys.net.vnet does for python tests, but brought to kyua leve= l and of > course agnostic to the language of a test. Igor is thinking in this di= rection > and hopefully will eventually deliver something. > Interesting idea. I fear it would be difficult to get something that=E2=80= =99s expressive enough to cover all the cases we care about, while still = being simple enough to be easy to use. On the other hand, perhaps we don=E2=80=99t need something that covers 10= 0% of cases. We=E2=80=99re probably never going to be able to build the s= orts of complex setups (for values of complex that are =E2=80=98three jai= ls with routes, a simple daemon in one or more and some pf rules) I often= build for pf tests, but perhaps we can cover 80% of cases with a simple = =E2=80=9CBuild a jail with one interface with a known address and a defau= lt route=E2=80=9D. Perhaps a =E2=80=98execenv_jail_init=3Dfoo.sh=E2=80=99 calling a separate= script to do the setup? > K> The downside is that we=E2=80=99re compiling the C test code on ever= y run, but I > K> expect test programs to be tiny, so that=E2=80=99s not too much of a= cost. > K> > K> Does this seem useful to you too? > > That would fail on decreased installation, that has tests, but doesn't = have > compiler. This is not just a made up problem, this is what many would = do in > CI, cause you want usually as quick as possible response time from CI, = and if > your team is not hacking the compiler, you won't add compiler to the CI= build. > That=E2=80=99s a good point, and not something I think we can reasonably = do from atf-sh. Not unless we do a pre-processing step during =E2=80=98ma= ke install=E2=80=99 (on top of the =E2=80=98add the shebang line=E2=80=99= we already do) to extract C code, build it and install it somewhere the = test can find it. I think I don=E2=80=99t dislike the current approach enough to try that. Best regards, Kristof
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EF2BE5F9-C9D0-46C6-AE19-C97478A7CE44>