Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scsynth: macOS: clip values on hw out busses #5110

Merged
merged 2 commits into from
Nov 28, 2020

Conversation

elgiano
Copy link
Contributor

@elgiano elgiano commented Jul 30, 2020

Purpose and Motivation

Related to #4894.
Proposes a fix for CoreAudio handling of loud volumes. Since CoreAudio only applies a relative attenuation, setting a low system volume doesn't prevent output values > +- 1 from sounding absolutely loud, possibly leading to hear damage.
This PR attempts to fix it by:

  • introducing a new ServerOption + cli switch: -s safetyClipThreshold
  • clipping values between +-safetyClipThreshold when CoreAudioDriver writes out
  • safetyClipThreshold defaults to 2.0 to allow some headroom and save some ears

Although clipping affects inter-app routing (as it happens through the same out busses), it doesn't affect recording.

Note: s.options.safetyClipThreshold = inf is valid (thx c++!) and makes clipping ineffective

Needs discussion: cross-platform

This is initially macos only. Any consideration about making it general? I wouldn't like to have it on linux for instance, where it is not needed, but maybe it would be good to have the same behavior for all platforms?

Types of changes

  • Documentation
  • New feature
  • Breaking change
    ( doesn't change any interface, but introduces hard clipping on macos scsynth by default and adds a cli switch )

To-do list

  • Code is tested
  • All tests are passing
  • Updated documentation
  • This PR is ready for review

@dyfer
Copy link
Member

dyfer commented Jul 30, 2020

Thanks @elgiano !

I wouldn't like to have it on linux for instance, where it is not needed, but maybe it would be good to have the same behavior for all platforms?

IMO it should be macOS only, as other backends seem to clip by themselves (AFAIU).

I'd also like to see a way to disable clipping completely, as in omitting sc_clip2 to preserve current behavior. We could use -s 0 or -s -1 for that, although maybe that's a little counterintuitive.

I'd also prefer to have less headroom - full scale signal with high RMS will still be much louder than other sounds and I don't think we need 6dB (2.0) headroom, but maybe 3 or 2dB. But if others want to stick with 6dB I'm fine with that.

@elgiano
Copy link
Contributor Author

elgiano commented Jul 30, 2020

I don't know how to disable clip completely. I mean, I could put an "if", but that would be one more "if" that wasn't there before anyway. Isn't -s inf a fair enough bypass switch? Comparing to inf will never clip

@dyfer
Copy link
Member

dyfer commented Jul 30, 2020

I don't know how to disable clip completely. I mean, I could put an "if", but that would be one more "if" that wasn't there before anyway. Isn't -s inf a fair enough bypass switch? Comparing to inf will never clip

Yeah, I can't say that I know how to do it either. One ugly, but likely well-performing way would be to put if around the whole for (int s = 0, b = 0; b < numOutputBuses && s < numOutputStreams; s++) { loop (and make a copy of it without clip) but that would be a lot of code duplication.

@elgiano
Copy link
Contributor Author

elgiano commented Jul 30, 2020

Idea: one way could be to create a subclass of SC_CoreAudioDriver, say SC_CoreAudioClippingDriver, override only the Run method, and select which class to use in here:

SC_AudioDriver* SC_NewAudioDriver(struct World* inWorld) { return new SC_CoreAudioDriver(inWorld); }

@dyfer
Copy link
Member

dyfer commented Jul 30, 2020

Idea: one way could be to create a subclass of SC_CoreAudioDriver, say SC_CoreAudioClippingDriver, override only the Run method, and select which class to use in here:

SC_AudioDriver* SC_NewAudioDriver(struct World* inWorld) { return new SC_CoreAudioDriver(inWorld); }

Yeah, that might be a good way to go about it...

@elgiano
Copy link
Contributor Author

elgiano commented Jul 30, 2020

I've written a POC of the subclass implementation.
The cost of it is to make SC_CoreAudioDriver::Run a virtual function, so that it can be overwritten by SC_CoreAudioClippingDriver.

I'm not sure if it's worth it, I have a feeling we should avoid having a virtual Run function. I'm not an expert at this level of precision, so any help would be fundamental. Virtuality help avoiding code duplication... but is the vtable bad for Run's performance? Do anyone know any other way around this?

@jamshark70
Copy link
Contributor

IMO it should be macOS only, as other backends seem to clip by themselves (AFAIU).

I don't recall if we tested other OSes rigorously for this. SC+JACK in Linux won't overload because JACK has no volume control -- but I bet you could replicate the problem by running SC --> another app that scales down volume --> hardware.

If I have time, I might also try it with the experimental PulseAudio backend: PA does have a volume control.

Did anyone check Windows?

The test would be: Turn system volume down, and try signals with amp > 1. If it gets louder out of the speakers at +10, +20 etc dB, then we knew the system isn't clipping internally.

I'd also prefer to have less headroom - full scale signal with high RMS will still be much louder than other sounds and I don't think we need 6dB (2.0) headroom, but maybe 3 or 2dB.

I proposed the small headroom -- I wouldn't mind reducing it to 2 dB.

@elgiano
Copy link
Contributor Author

elgiano commented Jul 31, 2020

I don't recall if we tested other OSes rigorously for this. SC+JACK in Linux won't overload because JACK has no volume control -- but I bet you could replicate the problem by running SC --> another app that scales down volume --> hardware.

On Linux it's managed by alsa, right ?(one level deeper than jack). I don't know how it's done internally, but I do know that a blown-up filter sounds quiet when system volume is quiet.

The test would be: Turn system volume down, and try signals with amp > 1. If it gets louder out of the speakers at +10, +20 etc dB, then we knew the system isn't clipping internally.

Great! I usually tested this with something like what was reported to be harmful on the forum:

{ LPF.ar(WhiteNoise.ar, -200) }.play // this is super loud, it spikes around 1e14

@jamshark70
Copy link
Contributor

jamshark70 commented Jul 31, 2020

Using the PulseAudio backend from #5081, I can reproduce the same problem that macOS users have reported.

The TL;DR conclusion, then, is that it isn't safe to implement this only for macOS -- we should do it for all platforms.

The issue, as I understand it, is:

  • Floating-point signal that is typically +/- 1, but with the possibility of far exceeding that range.
  • --> Volume control set to some low level, say -20 or -40 dB (reducing the sound pressure level by a factor of 10 or 100 respectively).
  • --> Hardware. (We definitely expect clipping to happen in the hardware -- soundcards usually work with integer sample formats, where it's literally impossible to represent out of range sample values.)

It's not at all far-fetched to imagine a user generating full-scale signals in SC, and using the system volume control to bring that down to a listenable level. User expectation at this point is, "I set my volume control to -40 dB so I shouldn't get anything louder than that in my earphones."

If SC accidentally generates a +700 dB signal (which I've done before), then the volume control makes that +660 dB and the hardware clips at full-scale -- but the user turned down the volume because full-scale is too loud = ear damage.

The defining feature of the issue, then, is that system volume control can pass through a signal louder than the set level.

To test, I ran this on scsynth built for PulseAudio:

(
a = {
	var trig = Impulse.kr(2);
	var amp = Demand.kr(trig, 0, Dseq([Dseries(-18, 6, 8)], inf));
	
	SinOsc.ar(440, 0, Lag.kr(amp, 0.1).dbamp).dup
}.play;
)

-18, -12, -6, 0 -- no clipping
+6, +12, +18, +24 -- clipping

Then I went to the PulseAudio volume panel.

When the scsynth volume level is set in pavucontrol to 0 dB, as expected, I hear four clean tones and four distorted tones.

If I turn pavucontrol-scsynth volume down to -20 dB, then I hear seven clean tones and one distorted tone -- and the output level goes above -20 dB.

If the volume control is clipping as we want (to protect hearing), then I would hear four clean and four distorted tones, just quieter.

That's definitely not the result I got.

Therefore, the same problem can occur in Linux.

@jamshark70
Copy link
Contributor

Moreover, if the intention is ear safety, the end-stage clipper needs to be based on the server's volume setting, not full scale.

Same scenario: user's speakers/headphones are loud. Only now they discover, "if I write s.volume = -40 in my startup file, then SC is at a good level and I don't have to think about it"... until getting a super loud signal, which SC's volume control reduces slightly and SC's end stage clipping brings to... full scale again, bang and "why didn't that volume control protect my hearing?"

I.e. you can imagine this case purely within SC, irrespective of OS.

That makes implementation trickier... or carefully document the limitations of SC's internal volume control (which is tricky to message in a way that doesn't look dumb).

@dyfer
Copy link
Member

dyfer commented Jul 31, 2020 via email

@elgiano
Copy link
Contributor Author

elgiano commented Jul 31, 2020

So, maybe we shouldn't deal with this at the backed level, and focus on solving the issues with a language-side implementation. After having tried the server path, I'm much more willing to abandon it.

Server pros:

  • backends that don't need clipping are unaffected
  • backends that need clipping are reliably clipped
  • doesn't affect recording

Server cons:

  • we need to implement a clipping alternative for all present and future backends that show this behavior
  • different behaviors for different backends
  • a clipping alternative is already some more backend-specific code, then if we want it to be completely switchable-off, it either becomes a lot more duplicated code, or seems to require a virtual Run function

Client pros:

  • same code, same behavior for all platforms and backends
  • it would be easier to implement other features, such as removing bad values, like SafetyNet does
  • easy to switch off completely if one wants

Client cons:

  • one could potentially put some nodes after volumeAmpControl. Examples:Synth.tail(0, \default), or Synth.after(s, \default) before volumeAmpControl is created. Maybe that's not so easy to do after all?
  • Recorder will put its node at tail(0), so after volumeAmpControl. This means that clipping will affect recording. But maybe there could be a "pre-fader" option for recorder, or anyway the behavior of "after default group" utilities should be figured out (Recorder, ServerMeter, Volume, any others?)

@dyfer
Copy link
Member

dyfer commented Jul 31, 2020

I'd add one more Server-side pro / Client-side con: server-side implementation will also provide basic safety for other clients.

@elgiano elgiano force-pushed the topic/macos-clip branch 3 times, most recently from 0dc0c9f to 8250de8 Compare August 24, 2020 12:06
@elgiano
Copy link
Contributor Author

elgiano commented Aug 24, 2020

I pushed a template-based implementation:

  • clipping can be transparently disabled by setting s.options.safetyClipThreshold = inf
  • implementation is through templates: the correct (clipping or not-clipping) function is selected at DriverStart time. No extra checks are performed after that moment
  • code duplication is completely avoided

The cost is that now we have two audioIOProc funcs and two SC_CoreAudioDriver:Run funcs (from templates, no code duplication), which I don't think is a cost at all, or very minimal. Feedback would be welcome and very important :)

This doesn't deal with s.volume (which I would leave to the sclang-side stage limiter implementation) or other platforms than macOS. Since so far we don't have that many backends, would it be reasonable to double-check all of them and add this feature only to the ones that need it, and then evaluate every future backend addition (e.g. pulse)?

Thanks to @brianlheim for the precious tips on how to use templates for this!

@@ -49,6 +49,7 @@ struct WorldOptions {

bool mRealTime = true;
bool mMemoryLocking = false;
float mSafetyClipThreshold = 2.0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note -- look for convo w @Spacechild1 about ABI/API changes

Copy link
Contributor

@Spacechild1 Spacechild1 Nov 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @brianlheim for catching that! WorldOptions is part of the public libscsynth API and changing a public struct causes API/ABI breakage.

Note that the libscsynth API/ABI has already been broken in past. For example, 695aea5 changed function signatures, and 22c9dea turned WorldOptions from a old-style C++ struct to a C++11 struct. The first commit causes a linker error if header and binary don't have the same version. The second commit has no consequences expect possible compilation errors for pre-C++11 projects.

However, changing the layout of a public struct can cause runtime crashes, e.g. when the host links against a new libsynth binary, but is compiled against an older header. This means it is now required that the SC_WorldOptions.h header and the libscsynth binary have the same SC version (which means you can't update the libscsynth binary without recompiling the host app against the same SuperCollider SDK version - and vice versa!) This is just something to be aware of.

I see two solutions:
a) make it clear that header and binary must have the same version
b) add a simple versioning mechanism similar to the SC plugin API, so that World_New fails on a version mismatch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @Spacechild1 !!

i don't know if many people use this part of the API at all. just thinking back, i have never seen or heard any concerns over the shared-libscsynth build breaking -- except for the Arch Linux distribution, but David Runge said he's not aware of anyone making use of it either. i've also never seen complaints about crashes or other breakage when those API changes did happen.

i wouldn't want to spend time on adding a feature that nobody will make use of, so for now i think we could just document in the header that the API may change across minor versions, so it should be compiled against headers from the same minor version used to link. that's at least a small stability guarantee. what do you think?

Copy link
Contributor

@Spacechild1 Spacechild1 Nov 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't know if many people use this part of the API at all

I'm sure there are a few people who use libscsynth, but they probably all use a binary + header with the same version. So in practice, ABI breakage is not such a big deal.

i've also never seen complaints about crashes or other breakage when those API changes did happen.

The difference is that previous breakages would lead to a compiler resp. linker error, which are easy to spot/fix. Changing the struct can cause runtime crashes, which are harder to debug.

so for now i think we could just document in the header that the API may change across minor versions, so it should be compiled against headers from the same minor version used to link. that's at least a small stability guarantee. what do you think?

That would be option a), and I think that's probably sufficient for now :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elgiano can you please add this note?

Copy link
Contributor Author

@elgiano elgiano Nov 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like this?

// PLEASE NOTE:
// This API might change across minor versions.
// Always make sure, when using libscsynth, that the binary and this header come from the same minor version.

@mossheim mossheim mentioned this pull request Nov 17, 2020
@mossheim mossheim added this to the 3.12.0 milestone Nov 18, 2020
@elgiano
Copy link
Contributor Author

elgiano commented Nov 23, 2020

Thanks @dyfer and @brianlheim. I put a short summary here since navigating reviews is starting to become difficult for me:

  • I think I addressed everything: the only point left where I need comments is the warning in SC_WorldOptions.h. I put it in a review comment, now I can't find it anymore, so I'll copy it here. What do you think of:
// PLEASE NOTE:
// This API might change across minor versions.
// Always make sure, when using libscsynth, that the binary and this header come from the same minor version.
  • amp vs dB: I kept it in amp and changed the default threshold to 1.26 (ca. 2 dB)
  • SUPERNOVA: I added another switch to classlib's ServerOptions, to not pass the "-s" flag when using supernova, otherwise supernova would refuse to start. We will have to implement clipping for supernova as well, I hope it can be done quickly after we are clear with the scsynth implementation.

@elgiano elgiano force-pushed the topic/macos-clip branch 2 times, most recently from 724a9a0 to 9689923 Compare November 23, 2020 14:52
@mossheim
Copy link
Contributor

I put a short summary here since navigating reviews is starting to become difficult for me:

no worries, it is quite disordered. just one thing, it might help to collapse review comments with "mark as resolved" once you resolve each one, then it's clearer what still remains to be done.

I think I addressed everything: the only point left where I need comments is the warning in SC_WorldOptions.h. I put it in a review comment, now I can't find it anymore, so I'll copy it here. What do you think of:

i think it's good. you should definitely also note that "this API" applies specifically to things in server/include, which i guess you could call the "libscsynth API". so that people don't get confused and think this also applies to the plugin API which has an actual versioning system. in fact, i would place this comment directly above the first line of code in all 3 files in server/include. also change "using libscsynth" to "using libscsynth as a shared library". libscsynth can also be built as a static lib (that's the default).

SUPERNOVA: I added another switch to classlib's ServerOptions, to not pass the "-s" flag when using supernova, otherwise supernova would refuse to start. We will have to implement clipping for supernova as well, I hope it can be done quickly after we are clear with the scsynth implementation.

oh yes right i forgot about that! will you make the supernova implementation in another PR? it may be better for this change because there are already a lot of PR comments to sift through. i guess it will be done to the portaudio backend since there is no supernova coreaudio backend yet.

@elgiano
Copy link
Contributor Author

elgiano commented Nov 23, 2020

no worries, it is quite disordered. just one thing, it might help to collapse review comments with "mark as resolved" once you resolve each one, then it's clearer what still remains to be done.

This I usually leave to the one who made the first comment, it feels a bit rude for me to decide when it is resolved :)

i think it's good. you should definitely also note that "this API" applies specifically to things in server/include, which i guess you could call the "libscsynth API". so that people don't get confused and think this also applies to the plugin API which has an actual versioning system. in fact, i would place this comment directly above the first line of code in all 3 files in server/include. also change "using libscsynth" to "using libscsynth as a shared library". libscsynth can also be built as a static lib (that's the default).

I've updated the message and added it to all three files in include/server

oh yes right i forgot about that! will you make the supernova implementation in another PR? it may be better for this change because there are already a lot of PR comments to sift through. i guess it will be done to the portaudio backend since there is no supernova coreaudio backend yet.

I'll open a new PR as soon as we are done with this one

@dyfer
Copy link
Member

dyfer commented Nov 23, 2020

just a note for supernova: we use portaudio on macOS so this is the backend for which this needs to be implemented. It would be good to test whether windows APIs have the same problem (namely MME as this is the default) and then decide if we apply it only for mac or mac and windows.

EDIT: I clearly missed Brian's comment earlier about this, sorry for the noise!

@mossheim
Copy link
Contributor

This I usually leave to the one who made the first comment, it feels a bit rude for me to decide when it is resolved :)

i'm used to the opposite workflow but i don't mind doing it that way, thanks for explaining!

mossheim
mossheim previously approved these changes Nov 24, 2020
Copy link
Contributor

@mossheim mossheim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@mossheim mossheim dismissed their stale review November 24, 2020 01:15

ah, looks like the linting job is failing, can you please format this @elgiano ? https://travis-ci.org/github/supercollider/supercollider/jobs/745436435

@elgiano
Copy link
Contributor Author

elgiano commented Nov 24, 2020

Sorry, I didn't expect to have linting problems with comments, but ofc it makes sense! it's clang-formatted now :)

@elgiano
Copy link
Contributor Author

elgiano commented Nov 24, 2020

@timblechmann would it be possible for you to get in touch about implementing this for supernova? I'd really benefit from a quick chat about it!

EDIT: you can see my current draft here: elgiano@aba9f87

Copy link
Contributor

@mossheim mossheim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! gonna let someone else approve/merge as this is a slightly complex feature.

@mossheim
Copy link
Contributor

fwiw @elgiano your branch looks good to me aside from the capitalization missing here elgiano@aba9f87#diff-95e496ab2ce1c36e8c20a690148fa7def521da35f7d1fac6e3cd2917241609e3R290

Copy link
Member

@dyfer dyfer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to merge it as is, but if you feel like making these changes I think it would be even better. Thanks!

HelpSource/Classes/ServerOptions.schelp Outdated Show resolved Hide resolved
@elgiano
Copy link
Contributor Author

elgiano commented Nov 28, 2020

@brianlheim

fwiw @elgiano your branch looks good to me aside from the capitalization missing here elgiano/supercollider@aba9f87#diff-95e496ab2ce1c36e8c20a690148fa7def521da35f7d1fac6e3cd2917241609e3R290

Thanks for looking at this Brian, and catching that typo! I haven't tested it yet (haven't built it once actually) because at this moment my access to a mac computer is somehow limited. But I will get to do it soon!

Copy link
Member

@dyfer dyfer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good, thank you!

IMO can be merged once CI passes (or maybe earlier? since last change did not touch cpp/sc code)

@mossheim mossheim merged commit f441a41 into supercollider:develop Nov 28, 2020
@elgiano
Copy link
Contributor Author

elgiano commented Nov 28, 2020

Thanks everyone, I'm very happy this got merged 👍

@dyfer
Copy link
Member

dyfer commented Apr 2, 2021

@elgiano are you still planning to make the supernova PR for this functionality? IMO we should have sn implementation on our radar for 3.12

@elgiano
Copy link
Contributor Author

elgiano commented Apr 3, 2021

I had a branch going on for supernova, couldn't really get it to work though, and then I ended up not having a Mac anymore.... however I might get one from my job soon and then I would look again into it!
If anyone wants to look at my branch, here it is: https://github.com/elgiano/supercollider/tree/topic/macos-supernova-clip

@joshpar
Copy link
Member

joshpar commented Apr 18, 2021

@elgiano - how is your progress on this looking? More than happy to take it over if you aren't able to keep on it.

@elgiano
Copy link
Contributor Author

elgiano commented Apr 25, 2021

@joshpar
Thanks! I would be super happy if you could have a look at it and take it over! You can see all the changes here:
develop...elgiano:topic/macos-supernova-clip

I still don't have a working Mac right now, so I can't do anything about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants