Date: Wed, 13 Dec 2023 22:59:13 -0400 From: Joseph Mingrone <jrm@freebsdfoundation.org> To: freebsd-multimedia@freebsd.org Subject: Re: RFC - Work on FreeBSD's Audio Stack Message-ID: <86y1dxpjzy.fsf@phe.ftfl.ca> In-Reply-To: <86ttomxg11.fsf@phe.ftfl.ca> (Joseph Mingrone's message of "Wed, 13 Dec 2023 11:45:30 -0400") References: <86ttomxg11.fsf@phe.ftfl.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Re-sending the text because the original contained a few errors, the worst = being broken URLs from a PDF to text conversion. Hello, The FreeBSD Foundation will be contracting out work on FreeBSD's audio stac= k. If you have any comments about the work, please share them before next = Wednesday, December 20. Joe =3D=3D=3D * Executive Summary The FreeBSD audio stack is one of those fields that does not attract the sa= me attention and development as others do, since it has been left largely unmaintained, and, although high in quality,= there is still room for improvement - from lack of audio development frameworks, to missing userland utilities and kernel d= river-related bugs. This project is meant to touch on all those areas, and as such, will be more of a general improvemen= t project, than an implementation of a specific feature. * Project Description The end goal of the project is to provide FreeBSD audio developers with use= ful tools and frameworks to make sound development on FreeBSD easier, without having to resort to custom solutions= for everything. On the user side, FreeBSD will ship with a more stable audio stack and a la= rger collection of userland programs, making for a more coherent ecosystem in general. OSS generally does not come with = many native tools except mixer(8), and users have to resort to using tools from multiple different audio systems (ALSA, = PulseAudio, sndio) to compensate. Additionally, I think the introduction of new development frameworks will encourage more = native audio development, which is going to be especially beneficial for people - me included - who want to use FreeBSD= for music production. * Deliverables Note: By nature of the project, it is possible that the exact details of so= me deliverables may change. The deliverable list mentioned in the proposal is likely to grow if time constraints allow, so c= onsider it a non-exhaustive list. snd hda(4) pin-patching Regarding the stability of the audio stack, the project will address the pi= n-patching issue present in the snd hda(4) (Intel High Definition Audio) driver. Essentially, some laptop models have non-sta= ndardized mappings for the headphone and speaker jack pins, which results in absence of audio from the headphones, u= ntil a patch is manually applied in snd hda(4) to correctly map the headphone and speaker pins for that model so that the = same audio stream is output from both the speakers and headphones when they are plugged in. The initial strategy to address the issue will be to see if it is possible = to do the patching automatically by figuring out what the Speaker pin=E2=80=99s as number is (see dev.hdac.<n>.pindump=3D1) = and "forcibly" assign the same value to the headphone jack pin as well. The following solutions follow the same logic: - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D273809 - https://i-bsd.com/freebsd-wireless-8265/ OpenBSD=E2=80=99s azalia(4) (their version of the HDA driver) will also ser= ve as a point of reference, even though it too contains a kind of manual patching mechanism. snd uaudio(4) fixes The project will also address bugs in the USB audio driver, snd uaudio(4), = which I have been able to reproduce using my Focusrite Scarlett USB sound card, with the most prominent and consistent o= ne being noise produced during the first 12 seconds of playback and when moving along a track/video. If the user tries = to move forward multiple times in a short time window, the audio device most of the time becomes unusable (i.e no audio) a= nd it has to be replugged. Though this issue is largely bypassed if audio is routed to the USB device through virtual os= s, this is still a bug that needs to be addressed. Related bug reports include: - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D257082 - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194527 Initially I am going to test whether the open() and read() routines cause t= his issue, as DTrace suggests that this happens around the same time open(2) or the first read(2) is called. As mentioned i= n the previous paragraph, virtual oss partially fixes this issue, so I would like to investigate and understand why, and ma= ybe find the root cause that way. Another source of information will be Linux=E2=80=99s Scarlett and USB audio driver= s which, as far as I know, do not have this issue. Other USB audio bugs include 1) those mentioned in the snd uaudio(4) man pa= ge, 2) snd uaudio(4) not passing enough information (e.g device name, associated device in /dev, and more) to the O= SS API, 3) no explicit list of supported sound cards. MIDI device cloning Unlike DSP devices, sound(4) does not allow cloning of MIDI devices, meanin= g applications cannot open the same MIDI device multiple times (e.g when recording more than one track with the same= device). This can be verified by tracing the dsp clone() routine found in sys/dev/sound/pcm/dsp.c, which is triggere= d during a dev clone EVENTHANDLER(9) event. For example, given a /dev/dsp8.0 device, trying to cat(1) /dev/dsp8.= 1 would trigger dsp clone() and create it on the fly (see FILES section in sound(4) man page). However, doing that with = a MIDI device would trigger the handler, but result in an error, as MIDI devices do not support cloning. To fix this= , the MIDI code will be patched to use the snd clone framework used for cloning DSP devices. Other kernel improvements Other improvements to the kernel code include 1) better-syncing with the 4F= ront OSSv4 API, 2) addressing open bug reports, 3) making optimizations where possible. oss(3) Following the motivation behind mixer(3), which was to make OSS mixer devel= opment more pleasant and easier, I will implement an oss(3) library, responsible for handling the audio device=E2= =80=99s state. oss(3) will serve as a unified interface for OSS audio and MIDI developers and allow for quicker application development, li= ke Digital Audio Workstations, instrument effects, audio servers, and general utilities. Widely-used existing codebases that can benefit from oss(3) are virtual oss= and Mozilla=E2=80=99s cubeb oss audio framework, which are also great sources of inspiration for features included in oss(3). audio(8) FreeBSD lacks an easy-to-use audio device handling utility similar to mixer= (8), but for the device-side of OSS, like OpenBSD=E2=80=99s audioctl(8). Such a utility will make use of the new oss(= 3) library, and offer a cleaner and user-friendlier interface, concentrating all useful - but scattered - information currently= provided by /dev/sndstat, hw.snd, dev.pcm and hw.usb.uaudio, the OSS API into the output of a single application. Hot-swapping Users of plain OSS, that is, without virtual oss, will have noticed that ho= t-swapping audio devices (i.e changing the default unit, hw.snd.default unit) mid-track does not work and sound keeps coming o= ut of the previous device until the track is restarted. This is because applications open(2) the device at the start = of playback and close(2) it when the track has stopped. virtual oss(8) can create a virtual audio device responsible for r= outing audio to the appropriate physical audio device. The device(s) sound is routed to can be changed at run-time using v= irtual oss cmd(8). This method has the advantage that sound can be re-routed to different devi= ces while applications keep the same /dev/dsp open; virtual oss(8) will make any necessary conversions on the fly and red= irect audio to the new device without requiring a restart of the track/application. This functionality will be embedded in either mixer(8) or the new audio(8) = program. Bluetooth device management utility Although not strictly audio-related, I plan to write a (possibly using bsdd= ialog) bluetooth device management utility. Setting up bluetooth devices on FreeBSD is still rather complicated and con= fusing, and involves setting up multiple different programs to even pair with a bluetooth device. This tool will det= ect nearby bluetooth devices, prompt the user to choose which one(s) they want to (un-)pair with, and handle configuratio= n automatically. Other userland improvements Smaller improvements include 1) revisiting some parts of mixer(3) and mixer= (8) with backwards compatibility in mind, 2) writing a MIDI testing utility similar to sndio=E2=80=99s midicat(1) and= ALSA=E2=80=99s aseqdump(1). * Development Process Development will be done on a local Git branch of main, which will also be = available on GitHub. Patches will be submitted for review on FreeBSD=E2=80=99s Phabricator and eventually committed to ups= tream by me once reviewed and accepted. Testing Testing will be done on both VMs and actual hardware, with each patch build= -tested locally and on the FreeBSD cluster. The project will require testing some of the kernel patches with multiple d= ifferent audio devices. Additionally, there will be new tests written for the FreeBSD test suite, for both the kernel and us= erland parts. The audio driver will be tested by writing a test program to go through mos= t of the IOCTLs provided to by the driver, to both confirm that the information returned is correct, and also to make = sure that users cannot pass values that would break the driver. Exact cases will be considered further down the project. For the userland parts, I will write scripts similar to the old mixer(8)=E2= =80=99s test script. Documentation The documentation part of the project includes updating the Wiki, Handbook = and Foundation pages to reflect the most recent changes and additions. New Handbook sections will be added to showca= se the use of oss(3) and mixer(3). with additional examples added to /usr/share/examples/sound. Man pages with miss= ing, incomplete or outdated information will be addressed.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86y1dxpjzy.fsf>