-
Notifications
You must be signed in to change notification settings - Fork 0
/
Documentation010_recording_and_inputs.txt
107 lines (75 loc) · 8.61 KB
/
Documentation010_recording_and_inputs.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
::v3.0::1o.2o22::2019.4.0::
================================================================================
Audio input:
================================================================================
To stream audio from any available system recording input - just attach an AudioStreamInput component to an empty game object and from custom script access audio buffer data of AudioSource which it automatically creates.
See how to interact with it in the AudioStreamInputDemo/AudioStreamInput2DDemo scene.
Latency is rather high for Unity input streams since it has to go via full Unity audio buffer processing (including optional spatialization). For significantly lower latency you can use AudioStreamInput2D component - with downside that it is 2D only.
ResonanceInput is fully spatialized and has near realtime latency (it doesn't use Unity spatialization at all)
All input components can stream/record from almost any connected input with a recording interface and they support broader class of hardware compared to Unity Microphone class.
See also 'Audio input interfaces' below.
As for latency currently this is best option I could come up with, for even lower latency native plugin is needed ( such as https://github.com/keijiro/Lasp ), with but the same 2D limitation since it uses just OnAudioFilterRead
[ OnAudioFilterRead has limitation of not being able to support 3D sound ]. For LASP interop with Unity AudioSource see this gist: https://gist.github.com/r618/d74f07b6049fce20f1dc0dfa546bda89 ( LASP have to be patched though currently since it can't be run not from main thread, and frequency is not exposed - those are just minor changes).
AudioStreamInput2D latency is very usable though and e.g. on iOS and recent phones it has almost immediate response to audio input in the scene.
AudioStreamInput* components will use DSP buffer setting for output #0, if specified, since FMOD's system for output #0 is also used as recording system.
See 'Devices config' in Documentation.txt
For latency you can also refer to answers to this post: http://www.fmod.org/questions/question/delaylatency-when-playing-sounds/
Since the way the resampling of input signal by just setting the pitch on the AudioSource was handled led to big drifting over longer time if input and output sample rates were significantly different,
in 1.7.7 I added option to not use Unity's built AudioSource resampling and channel conversion this way.
Instead a custom resampler is used, with possibility to specify custom mix matrix to provide mapping between input and output channels. Default mix matrix is computed by FMOD if not customised by user,
on iOS default mix matrix is computed manually (might not be best fit for all scenarios).
See demo scene for reference, and call SetCustomMixMatrix on component before starting the recording, if needed.
The custom resampler is based on simple interpolation of original audio/samplerate for target/output,
and is probably lower quality than Unity's builtin one - but is stable when running long/er.
! For iOS/mobiles please see specific mobile recording notes.
Documentation060_mobiles
With ResonanceInput it's possible to capture any input device and play it back in 3D - see in 3D spatialisation.
====
! Please be aware that each component which manipulates and manages its own AudioSource makes the Unity AudioSource component on GameObject unusable by user/Unity/other components at runtime
So in general it's advisable to place each AudioStream* component which uses AudioSource on its own exclusive GameObject
Audio input interfaces:
--------------------------------------------------------------------------------
On desktops/standalones it is possible to capture and listen to the system output
- for Windows, see this forum post: https://forum.unity.com/threads/audiostream-an-audio-streaming-solution-for-all-and-everywhere.412029/page-3#post-3120495
- on macOS it's possible to configure new recording device using e.g. Soundflower [https://github.com/mattingalls/Soundflower], newer stuff from RogueAmoeba, or BlackHole virtual audio driver/s [https://github.com/ExistentialAudio/BlackHole]
*Windows specific*:
- any device/s which can be opened for recording, not designated as a micophone/audio input by system, is postfixed with "[loopback]" in their name by FMOD
- for example on Windows a VirtualCable port/channel can be opened and streamed into Unity this way -
Note: that device access like this is only possible on desktops, mobile OSs typically don't expose this functionality of main audio output.
Note: don't use *loopback* interfaces for signals which should be audible/heard - this will immediately cause feedback loop -
HW compatibility:
- FMOD currently has limit 'MAX_CHANNEL_WIDTH' which at the times of writing is =32= for number of channels of a given interface - so it doesn't support more than 32 inputs/outputs on a soundcard/interface
- multichannel input/microphones are supported as opposed to Unity's Microphone class
Unity support for multichannel inputs might got better with latest (2023) updates though
Android recording permissions:
--------------------------------------------------------------------------------
there is a Microphone class referenced in Android specific code of AudioStreamInputBase.cs, which causes Unity to automatically include recording permission in the manifest
If you don't want/need this permission and recording in your Android build, please delete the whole 'Scripts/AudioStreamInput' folder and demos in 'Demo/AudioStreamInput' folder.
Android OpenSL ES support:
--------------------------------------------------------------------------------
FMOD requires 'OpenSL ES' support on Android in order to enable recording - this is currently enabled for all cases/devices, even when recording is not used in the app
- contact me if you need just playback in the app w/o recording enabled, alternatively you can just remove the requirement in FMOD_System.cs: comment out/remove 'outputType = FMOD.OUTPUTTYPE.OPENSL;'
'resampleInput' setting:
--------------------------------------------------------------------------------
- AudioStreamInput* 's setting for better interoperability with other plugins/assets which need original sample rate of the input signal preserved (such as AVProMediaRecorder at that time) when they want to do their own resampling/encoding.
Default is ON and the input signal is resampled to current output sample rate so it can be e.g. heard normally on speakers. When OFF no resampling is done and input signal can be manually forwarded as needed - for this custom scripting is needed if it is e.g. not being captured from attached AudioSource.
'hotplugging':
--------------------------------------------------------------------------------
Hotplugging - dynamic updates of attached input devices - can be enabled by including separate 'AudioStreamDevicesChangedNotify' component anywhere in the scene
Devices are checked for changes in Update
If a change is detected, an user Unity event is invoked where the new list can be queried/updated via 'AvailableInputs' again.
'AudioStreamDevicesChangedNotify' is included in few demo scenes together with event handler.
================================================================================
'AudioStream InputDevice' mixer effect:
================================================================================
To record directly into an AudioMixer group add 'AudioStream InputDevice' effect and set InputDeviceID to 0 (default system input) or above. The effect will start recording immediately and pass the recording along in the mixer
Set InputDeviceID to '-1' to stop recording
The recording will be running depending on the setting of the mixer:
- if 'Auto Mixer Suspend' is ON (default) the recording will be running (recording from the input) as long as the signal is above the Threshold Volume (default -80 dB) of the mixer
note: this is regardless of playmode, so if the recording was started during play time after exiting play mode in Editor, it will stil be running if the volume is above the threshold
- if 'Auto Mixer Suspend' is OFF the recording will start and be running regardless of e.g. play mode
Moreover, the demo scene for the effect - InputDeviceUnityMixerDemo - has additional AudioSouce in the scene, which outputs to (a single) mixer group, is played automatically
and has its volume set to 0. This is in order to activate the mixer processing by Unity, since 'Auto Mixer Suspend' is by default ON and the gorup wouldn't be even processed otherwise
Note: * macOS *
Please note the recording effect currently doesn't work on macOS -- this should be hopefully fixed later
- in any case all sources for the plugin are included in AudioPlugin.zip archive