Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 13 Dec 2023 11:45:30 -0400
From:      Joseph Mingrone <jrm@freebsdfoundation.org>
To:        freebsd-multimedia@freebsd.org
Subject:   RFC - Work on FreeBSD's Audio Stack
Message-ID:  <86ttomxg11.fsf@phe.ftfl.ca>

next in thread | raw e-mail | index | archive | help
Hello,

The FreeBSD Foundation will be contracting out work on FreeBSD's audio stac=
k.  If you have any comments about the work, please share them before next =
Wednesday, December 20?

Joe

=3D=3D=3D

* Executive Summary

The FreeBSD audio stack is one of those fields that does not attract the sa=
me attention and development as others do,
since it has been left largely unmaintained, and, although high in quality,=
 there is still room for improvement - from lack
of audio development frameworks, to missing userland utilities and kernel d=
river-related bugs. This project is meant to
touch on all those areas, and as such, will be more of a general improvemen=
t project, than an implementation of a specific
feature.

* Project Description

The end goal of the project is to provide FreeBSD audio developers with use=
ful tools and frameworks to make sound
development on FreeBSD easier, without having to resort to custom solutions=
 for everything.

On the user side, FreeBSD will ship with a more stable audio stack and a la=
rger collection of userland programs, making
for a more coherent ecosystem in general. OSS generally does not come with =
many native tools except mixer(8), and users
have to resort to using tools from multiple different audio systems (ALSA, =
PulseAudio, sndio) to compensate. Additionally,
I think the introduction of new development frameworks will encourage more =
native audio development, which is going to
be especially beneficial for people - me included - who want to use FreeBSD=
 for music production.

* Deliverables

Note: By nature of the project, it is possible that the exact details of so=
me deliverables may change. The deliverable list
mentioned in the proposal is likely to grow if time constraints allow, so c=
onsider it a non-exhaustive list.

snd hda(4) pin-patching

Regarding the stability of the audio stack, the project will address the pi=
n-patching issue present in the snd hda(4) (Intel
High Definition Audio) driver. Essentially, some laptop models have non-sta=
ndardized mappings for the headphone and
speaker jack pins, which results in absence of audio from the headphones, u=
ntil a patch is manually applied in snd hda(4)
to correctly map the headphone and speaker pins for that model so that the =
same audio stream is output from both the
speakers and headphones when they are plugged in.

The initial strategy to address the issue will be to see if it is possible =
to do the patching automatically by figuring out
what the Speaker pin=E2=80=99s as number is (see dev.hdac.=C2=A1n=C2=BF.pin=
dump=3D1) and =E2=80=9Cforcibly=E2=80=9D assign the same value to the headp=
hone
jack pin as well. The following solutions follow the same logic:

   =E2=80=A2 https://bugs.freebsd.org/bugzilla/show bug.cgi?id=3D273809
   =E2=80=A2 https://i-bsd.com/freebsd-wireless-8265/

OpenBSD=E2=80=99s azalia(4) (their version of the HDA driver) will also ser=
ve as a point of reference, even though it too contains
a kind of manual patching mechanism.

snd uaudio(4) fixes

The project will also address bugs in the USB audio driver, snd uaudio(4), =
which I have been able to reproduce using my
Focusrite Scarlett USB sound card, with the most prominent and consistent o=
ne being noise produced during the first 12
seconds of playback and when moving along a track/video. If the user tries =
to move forward multiple times in a short time
window, the audio device most of the time becomes unusable (i.e no audio) a=
nd it has to be replugged. Though this issue
is largely bypassed if audio is routed to the USB device through virtual os=
s, this is still a bug that needs to be addressed.

Related bug reports include:

   =E2=80=A2 https://bugs.freebsd.org/bugzilla/show bug.cgi?id=3D257082
   =E2=80=A2 https://bugs.freebsd.org/bugzilla/show bug.cgi?id=3D194527

Initially I am going to test whether the open() and read() routines cause t=
his issue, as DTrace suggests that this happens
around the same time open(2) or the first read(2) is called. As mentioned i=
n the previous paragraph, virtual oss partially
fixes this issue, so I would like to investigate and understand why, and ma=
ybe find the root cause that way. Another
source of information will be Linux=E2=80=99s Scarlett and USB audio driver=
s which, as far as I know, do not have this issue.

Other USB audio bugs include 1) those mentioned in the snd uaudio(4) man pa=
ge, 2) snd uaudio(4) not passing enough
information (e.g device name, associated device in /dev, and more) to the O=
SS API, 3) no explicit list of supported sound
cards.

MIDI device cloning

Unlike DSP devices, sound(4) does not allow cloning of MIDI devices, meanin=
g applications cannot open the same MIDI
device multiple times (e.g when recording more than one track with the same=
 device). This can be verified by tracing
the dsp clone() routine found in sys/dev/sound/pcm/dsp.c, which is triggere=
d during a dev clone EVENTHANDLER(9)
event. For example, given a /dev/dsp8.0 device, trying to cat(1) /dev/dsp8.=
1 would trigger dsp clone() and create it on
the fly (see FILES section in sound(4) man page). However, doing that with =
a MIDI device would trigger the handler,
but result in an error, as MIDI devices do not support cloning. To fix this=
, the MIDI code will be patched to use the
snd clone framework used for cloning DSP devices.

Other kernel improvements

Other improvements to the kernel code include 1) better-syncing with the 4F=
ront OSSv4 API, 2) addressing open bug
reports, 3) making optimizations where possible.

oss(3)

Following the motivation behind mixer(3), which was to make OSS mixer devel=
opment more pleasant and easier, I will
implement an oss(3) library, responsible for handling the audio device=E2=
=80=99s state. oss(3) will serve as a unified interface for OSS
audio and MIDI developers and allow for quicker application development, li=
ke Digital Audio Workstations, instrument
effects, audio servers, and general utilities.

Widely-used existing codebases that can benefit from oss(3) are virtual oss=
 and Mozilla=E2=80=99s cubeb oss audio framework,
which are also great sources of inspiration for features included in oss(3).

audio(8)

FreeBSD lacks an easy-to-use audio device handling utility similar to mixer=
(8), but for the device-side of OSS, like
OpenBSD=E2=80=99s audioctl(8). Such a utility will make use of the new oss(=
3) library, and offer a cleaner and user-friendlier
interface, concentrating all useful - but scattered - information currently=
 provided by /dev/sndstat, hw.snd, dev.pcm and
hw.usb.uaudio, the OSS API into the output of a single application.

Hot-swapping

Users of plain OSS, that is, without virtual oss, will have noticed that ho=
t-swapping audio devices (i.e changing the default
unit, hw.snd.default unit) mid-track does not work and sound keeps coming o=
ut of the previous device until the track
is restarted. This is because applications open(2) the device at the start =
of playback and close(2) it when the track has
stopped. virtual oss(8) can create a virtual audio device responsible for r=
outing audio to the appropriate physical audio
device. The device(s) sound is routed to can be changed at run-time using v=
irtual oss cmd(8).

This method has the advantage that sound can be re-routed to different devi=
ces while applications keep the same /dev/dsp
open; virtual oss(8) will make any necessary conversions on the fly and red=
irect audio to the new device without requiring
a restart of the track/application.

This functionality will be embedded in either mixer(8) or the new audio(8) =
program.

Bluetooth device management utility

Although not strictly audio-related, I plan to write a (possibly using bsdd=
ialog) bluetooth device management utility.
Setting up bluetooth devices on FreeBSD is still rather complicated and con=
fusing, and involves setting up multiple

different programs to even pair with a bluetooth device. This tool will det=
ect nearby bluetooth devices, prompt the user
to choose which one(s) they want to (un-)pair with, and handle configuratio=
n automatically.

Other userland improvements

Smaller improvements include 1) revisiting some parts of mixer(3) and mixer=
(8) with backwards compatibility in mind,
2) writing a MIDI testing utility similar to sndio=E2=80=99s midicat(1) and=
 ALSA=E2=80=99s aseqdump(1).

* Development Process

Development will be done on a local Git branch of main, which will also be =
available on GitHub. Patches will be submitted
for review on FreeBSD=E2=80=99s Phabricator and eventually committed to ups=
tream by me once reviewed and accepted.

Testing

Testing will be done on both VMs and actual hardware, with each patch build=
-tested locally and on the FreeBSD cluster.
The project will require testing some of the kernel patches with multiple d=
ifferent audio devices. Additionally, there will
be new tests written for the FreeBSD test suite, for both the kernel and us=
erland parts.

The audio driver will be tested by writing a test program to go through mos=
t of the IOCTLs provided to by the driver,
to both confirm that the information returned is correct, and also to make =
sure that users cannot pass values that would
break the driver. Exact cases will be considered further down the project.

For the userland parts, I will write scripts similar to the old mixer(8)=E2=
=80=99s test script.

Documentation

The documentation part of the project includes updating the Wiki, Handbook =
and Foundation pages to reflect the most
recent changes and additions. New Handbook sections will be added to showca=
se the use of oss(3) and mixer(3). with
additional examples added to /usr/share/examples/sound. Man pages with miss=
ing, incomplete or outdated information
will be addressed.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86ttomxg11.fsf>