Date: Sat, 16 Dec 2023 15:19:22 +0100 From: Florian Walpen <dev@submerge.ch> To: Joseph Mingrone <jrm@freebsdfoundation.org>, freebsd-multimedia@freebsd.org Subject: Re: RFC - Work on FreeBSD's Audio Stack Message-ID: <6705256.qJWK8QVVMX@x230> In-Reply-To: <86y1dxpjzy.fsf@phe.ftfl.ca> References: <86ttomxg11.fsf@phe.ftfl.ca> <86y1dxpjzy.fsf@phe.ftfl.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday, December 14, 2023 3:59:13 AM CET Joseph Mingrone wrote: > Re-sending the text because the original contained a few errors, the worst > being broken URLs from a PDF to text conversion. >=20 > Hello, >=20 > The FreeBSD Foundation will be contracting out work on FreeBSD's audio > stack. If you have any comments about the work, please share them before > next Wednesday, December 20. >=20 > Joe Any effort in this department is very welcome indeed. I seem to be in the=20 target group of this project - I'm the maintainer of audio/ardour and audio/ jack, and I wrote both the current and the upcoming (low latency) OSS backe= nd=20 for Jack. Currently I am working on an improved driver for snd_hdspe(4). >=20 > * Project Description >=20 > The end goal of the project is to provide FreeBSD audio developers with > useful tools and frameworks to make sound development on FreeBSD easier, > without having to resort to custom solutions for everything. >=20 > On the user side, FreeBSD will ship with a more stable audio stack and a > larger collection of userland programs, making for a more coherent > ecosystem in general. OSS generally does not come with many native tools > except mixer(8), and users have to resort to using tools from multiple > different audio systems (ALSA, PulseAudio, sndio) to compensate. > Additionally, I think the introduction of new development frameworks will > encourage more native audio development, which is going to be especially > beneficial for people - me included - who want to use FreeBSD for music > production. Let's be realistic here. The OSS API is a dead end in the middle to long te= rm.=20 It's neither widely used nor particularly well suited for modern hardware. = And=20 it's conceptually broken on many levels, including parameter negotiation,=20 consistency of parameters, buffer and latency management, error handling. =46ixes and usability improvements are a good thing. But we have to think a= bout=20 what to replace the OSS API with and how much time we still want to invest= =20 into the OSS ecosystem. Besides that, for most users the OSS API is just a sink for their primary=20 sound server like PulseAudio, sndio, pipewire. These will not go away. I'd= =20 focus on complementary features and coexistence here. Music production is a very different beast altogether. It'll always need so= me=20 more knowledge and custom solutions. >=20 > snd uaudio(4) fixes >=20 > The project will also address bugs in the USB audio driver, snd uaudio(4), > which I have been able to reproduce using my Focusrite Scarlett USB sound > card, with the most prominent and consistent one being noise produced > during the first 12 seconds of playback and when moving along a > track/video. If the user tries to move forward multiple times in a short > time window, the audio device most of the time becomes unusable (i.e no > audio) and it has to be replugged. Though this issue is largely bypassed = if > audio is routed to the USB device through virtual oss, this is still a bug > that needs to be addressed. =46rom the description here this sounds more like an issue with the player = or=20 the vchan feeders, given that virtual_oss doesn't produce noise. Did you fi= le=20 a bug report? >=20 > Related bug reports include: >=20 > - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D257082 > - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194527 I'm not convinced that these are related. There's probably better examples = for=20 snd_uaudio problems, if you skim the bug reports. >=20 > Initially I am going to test whether the open() and read() routines cause > this issue, as DTrace suggests that this happens around the same time > open(2) or the first read(2) is called. As mentioned in the previous > paragraph, virtual oss partially fixes this issue, so I would like to > investigate and understand why, and maybe find the root cause that way. > Another source of information will be Linux=E2=80=99s Scarlett and USB au= dio > drivers which, as far as I know, do not have this issue. Most problems I've seen are about bad vchan constellations. A good=20 troubleshooting checklist would be of great help for bug reports, something= =20 along the lines of # sysctl hw.snd.verbose=3D2 # cat /dev/sndstat # ... and some more user friendly output from /dev/sndstat etc. My Scarlett18i20 works fine, BTW. Maybe a different hardware generation=20 though. >=20 > Other USB audio bugs include 1) those mentioned in the snd uaudio(4) man > page, 2) snd uaudio(4) not passing enough information (e.g device name, > associated device in /dev, and more) to the OSS API, 3) no explicit list = of > supported sound cards. >=20 > MIDI device cloning >=20 > Unlike DSP devices, sound(4) does not allow cloning of MIDI devices, mean= ing > applications cannot open the same MIDI device multiple times (e.g when > recording more than one track with the same device). This can be verified > by tracing the dsp clone() routine found in sys/dev/sound/pcm/dsp.c, which > is triggered during a dev clone EVENTHANDLER(9) event. For example, given= a > /dev/dsp8.0 device, trying to cat(1) /dev/dsp8.1 would trigger dsp clone() > and create it on the fly (see FILES section in sound(4) man page). Howeve= r, > doing that with a MIDI device would trigger the handler, but result in an > error, as MIDI devices do not support cloning. To fix this, the MIDI code > will be patched to use the snd clone framework used for cloning DSP > devices. Is there really demand for this? Multiplexing MIDI is usually done in the=20 sound server or DAW. And ALSA MIDI ports can be cloned, AFAIK, so audio=20 software ported from Linux will be fine without it. >=20 > Other kernel improvements >=20 > Other improvements to the kernel code include 1) better-syncing with the > 4Front OSSv4 API, 2) addressing open bug reports, 3) making optimizations > where possible. Again, OSSv4 compatibility won't help much, it's basically abandoned by aud= io=20 software developers. Apart from open bug reports there's also missing keven= t=20 support (instead of poll() being the only option), or the buffer / blocksiz= e=20 based latency setting which is conceptually broken. I can give more details= on=20 these topics if there's interest. >=20 > oss(3) >=20 > Following the motivation behind mixer(3), which was to make OSS mixer > development more pleasant and easier, I will implement an oss(3) library, > responsible for handling the audio device=E2=80=99s state. oss(3) will se= rve as a > unified interface for OSS audio and MIDI developers and allow for quicker > application development, like Digital Audio Workstations, instrument > effects, audio servers, and general utilities. >=20 > Widely-used existing codebases that can benefit from oss(3) are virtual o= ss > and Mozilla=E2=80=99s cubeb oss audio framework, which are also great sou= rces of > inspiration for features included in oss(3). What's the scope of this library? Main PCM devices, virtual PCM devices,=20 hardware devices? Settings and state only, or also playback and recording=20 operation? Beware that there's a great variety of operation strategies for different u= se=20 cases. With a simple poll() based read() / write() at one end of the spectr= um,=20 and something like my new Jack OSS backend on the other end. It's timer bas= ed,=20 preferably mmap()ed, uses its own buffer management and strives to keep=20 latency reproducible within +/- 1ms. The backend code is here: https://github.com/0EVSG/sosso Due to different requirements, it will be difficult to simplify much of the= =20 OSS device setup while still catering to all operation strategies. >=20 > audio(8) >=20 > FreeBSD lacks an easy-to-use audio device handling utility similar to > mixer(8), but for the device-side of OSS, like OpenBSD=E2=80=99s audioctl= (8). Such > a utility will make use of the new oss(3) library, and offer a cleaner and > user-friendlier interface, concentrating all useful - but scattered - > information currently provided by /dev/sndstat, hw.snd, dev.pcm and > hw.usb.uaudio, the OSS API into the output of a single application. Many of the sysctls mentioned here are not dynamic and only apply on playba= ck=20 / recording restart or device attach. That could limit this tool's usefulne= ss. =46or reference, this part of my Jack on FreeBSD notes should give you an=20 overview of knobs that are beneficial for music production: https://github.com/0EVSG/freebsd_jack_notes#system-settings >=20 > Hot-swapping >=20 > Users of plain OSS, that is, without virtual oss, will have noticed that > hot-swapping audio devices (i.e changing the default unit, hw.snd.default > unit) mid-track does not work and sound keeps coming out of the previous > device until the track is restarted. This is because applications open(2) > the device at the start of playback and close(2) it when the track has > stopped. virtual oss(8) can create a virtual audio device responsible for > routing audio to the appropriate physical audio device. The device(s) sou= nd > is routed to can be changed at run-time using virtual oss cmd(8). >=20 > This method has the advantage that sound can be re-routed to different > devices while applications keep the same /dev/dsp open; virtual oss(8) wi= ll > make any necessary conversions on the fly and redirect audio to the new > device without requiring a restart of the track/application. >=20 > This functionality will be embedded in either mixer(8) or the new audio(8) > program. What you outline here is a complete sound server, but it doesn't tell how y= ou=20 want to implement that. Extend the vchan infrastructure, autostart virtual_= oss=20 on all PCM devices, or something else? In many use cases this just duplicates the sound server that sits on top of= =20 the dsp device. At worst it compromises quality and latency because of=20 conversions and routing buffers. As a provocative question, wouldn't the=20 average user be served better if we just implemented full pipewire support,= =20 going forward? >=20 > Testing >=20 > Testing will be done on both VMs and actual hardware, with each patch > build-tested locally and on the FreeBSD cluster. The project will require > testing some of the kernel patches with multiple different audio devices. > Additionally, there will be new tests written for the FreeBSD test suite, > for both the kernel and userland parts. >=20 > The audio driver will be tested by writing a test program to go through m= ost > of the IOCTLs provided to by the driver, to both confirm that the > information returned is correct, and also to make sure that users cannot > pass values that would break the driver. Exact cases will be considered > further down the project. You mean IOCTLs provided by the dsp devices? On a dummy driver? Because the= =20 hardware drivers are usually well separated in kernel modules, which means= =20 they can be tested separately. Sorry to sound a bit negative here, but some of the objectives look rather= =20 vague to me and I think another iteration of problem analysis and scope=20 definition would increase chances of success. Anyway, I'm glad to help if you have questions or want to discuss something. Regards, =46lorian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6705256.qJWK8QVVMX>