diff --git a/README.md b/README.md index 53f918f..b8433d9 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,16 @@ Presentation at PyCode Conference 2019 in Gdansk. Video recording. Coming, maybe in November. +## SenseCamp2019: Classification of Environmental Sound using IoT sensors + + +Slides + + +Presentation at SenseCamp 2019 hosted by FORCE Technology Senselab. +Slides: [web](https://jonnor.github.io/machinehearing/pycode2019/slides.html), +[.PDF](./sensecamp2019/slides.pdf) + ## NMBU lecture on Audio Classification diff --git a/sensecamp2019/Makefile b/sensecamp2019/Makefile new file mode 100644 index 0000000..cee7a60 --- /dev/null +++ b/sensecamp2019/Makefile @@ -0,0 +1,6 @@ + +slides: + pandoc -t revealjs -s presentation.md -o slides.html --slide-level=2 --mathml -V theme=white + +slides.pdf: + pandoc -t beamer -s presentation.md -o slides.pdf --slide-level=2 --mathml diff --git a/sensecamp2019/README.md b/sensecamp2019/README.md new file mode 100644 index 0000000..84652a0 --- /dev/null +++ b/sensecamp2019/README.md @@ -0,0 +1,216 @@ + +# Context + +https://forcetechnology.com/en/events/2019/sensecamp-2019 +"Opportunities with Machine Learning in audio” + + +09.45 - 10.25 Deep audio - data, representations and interactivity +Lars Kai Hansen, Professor, DTU Compute - Technical University of Denmark +10.25 - 11.05 On applying AI/ML in Audiology/Hearing aids +Jens Brehm Bagger Nielsen, Architect, Machine Learning & Data Science, Widex +11.05 - 11.45 Data-driven services in hearing health care +Niels H. Pontoppidan, Research Area Manager, Augmented Hearing Science - Eriksholm Research Center + + +## Format + +30 minutes, 10 minutes QA + +# TODO + +- Add some Research Projects at the end + +Pretty + +- Add Soundsensing logo to frontpage +- Add Soundsensing logo to ending page +- Add Soundensing logo at bottom of each page + + +# Goals + +From our POV + +1. Attract partners for Soundsensing +Research institutes. Public or private. +Joint technology development? +2. Attract pilot projects for Soundsensing +(3. Attract contacts for consulting on ML+audio+embedded ) + +From audience POV + +> you as audio proffesionals, understand: +> +> possibilities of on-sensor ML +> +> how Soundsensing applies this to Noise Monitoring + +> basics of machine learning for audio + + + +## Partnerships + +Research + +What do we want to get out of a partnership? +How can someone be of benefit to us? + +- Provide funding from their existing R&D project budgets +- Provide resources (students etc) to work on our challenges +- Help secure funding in joint project + + + +## Calls to action + +1-2 Data Science students in Spring 2020. + +Looking for pilot projects for Autumn 2020 (or maybe spring). + +Interested in machine learning (for audio) on embedded devices? +Come talk to me! +Send email. + + +## Title +Classification of environmental sound using IoT sensors + + +## Audience + +Audio practitioners. Many technical, some management. + +- Familiar with Sound. +Audio aquisition, Sampling rate, Frequency spectrum, Spectrograms +- Not familiar with Machine Learning +Supervised learning. Convolutional Neural Networks. +- Not familiar with Internet of THings + +## Scope + +Style. +Less training/tutorial/howto compared to EuroPython/PyCode +More Research&Development oriented. +More Soundsensing focused. + + + +# Outline + +Introduction + +- About me +- About Soundsensing +- Noise Monitoring +- Thesis + +- Environmental Sound Classification +- Wireless sensor network contraints. IoT +- On-edge classification +- Future sneakpeak: Neural accelerators for HW + + +- Existing ESC work +- SB-CNN model +- Typical Audio classification pipeline +- Performance vs compute landscape + + +- How to get this to fit on a small device? +Limiting input size +Depthwise Convolutions + +Tricks + +- Unknown class +- Merging to more high-level classes +- Mapping over longer times + +## Out of scope + +On-edge challenges + +## Q + +Availability of + +- Low-power microcontroller. ARM Cortex M4F +- FPGA. +- ASIC. + +ST Orlando + +Cortex-M4 microcontroller (MCU) and 128 KB of memory +6.2 mm x 5.5 mm die +200 Mhz +41 mWatt +2.9 TOPS/W +AlexNet at 10 FPS. + + +Microphone becomes the bottleneck. + +Vesper VM1010 +Wake on Sound +18 uWatt + +PUI Audio PMM-3738-VM1010 +Wake on Sound +9 μW of power + +https://www.digikey.com/en/product-highlight/p/pui-audio/wake-on-sound-piezoelectric-mems-microphone + + +https://blog.st.com/orlando-neural-network-iot/ + + +What is the TOPS/watt for current Cortex M4F? +How does it compare with proposed milli-watt scale accelerators + + +Lattice sensAI stack +FPGA +1 mW-1W + +https://www.latticesemi.com/Blog/2019/05/17/18/25/sensAI + +Human presence detection. 5 FPS 64x64x3. 7 mW +VGG8. 8 layer CNN. + +Lattice ICE40 UltraPlus CNN accelerator IP +http://www.latticesemi.com/Products/DesignSoftwareAndIP/IntellectualProperty/IPCore/IPCores04/compactcnn + +TensorFlow Lite for microcontrollers +https://www.tensorflow.org/lite/microcontrollers + +STM32Cube.AI +STM32 X-CUBE-AI +https://www.st.com/en/embedded-software/x-cube-ai.html + + +emlearn + + + + + +## Takeaways +Or talking points... + +- ML on audio close to human-level performance on some tasks +(when not compute constrainted) + +- On-edge inference is desirable to keep data traffic down. +Enable battery power / energy harvesting - cheaper installation costs - denser networks. +Lower data traffic - cheaper wireless costs. + +- ML-accelerators for low-power sensor units are expected in 2020 + +- Soundsensing has developed a low-power sensor unit for Noise Monitoring. +- We are running pilot projects now. + +- Strong cross pollination from bigger ML domains. +Image and Natural Language Processing pushes Audio forward +CNNs. Sequence modelling (RNNs). diff --git a/sensecamp2019/img/CMSIS-NN-functions.png b/sensecamp2019/img/CMSIS-NN-functions.png new file mode 100644 index 0000000..82fcaf1 Binary files /dev/null and b/sensecamp2019/img/CMSIS-NN-functions.png differ diff --git a/sensecamp2019/img/SONYC-CPS.png b/sensecamp2019/img/SONYC-CPS.png new file mode 100644 index 0000000..ae981f1 Binary files /dev/null and b/sensecamp2019/img/SONYC-CPS.png differ diff --git a/sensecamp2019/img/ST-Orlando-SoC.png b/sensecamp2019/img/ST-Orlando-SoC.png new file mode 100644 index 0000000..cbc5b78 Binary files /dev/null and b/sensecamp2019/img/ST-Orlando-SoC.png differ diff --git a/sensecamp2019/img/STM32F103VGT6-LD.jpg b/sensecamp2019/img/STM32F103VGT6-LD.jpg new file mode 100644 index 0000000..2e4a0dd Binary files /dev/null and b/sensecamp2019/img/STM32F103VGT6-LD.jpg differ diff --git a/sensecamp2019/img/activation-functions.png b/sensecamp2019/img/activation-functions.png new file mode 100644 index 0000000..5bbed47 Binary files /dev/null and b/sensecamp2019/img/activation-functions.png differ diff --git a/sensecamp2019/img/activation-functions.svg b/sensecamp2019/img/activation-functions.svg new file mode 100644 index 0000000..753632d --- /dev/null +++ b/sensecamp2019/img/activation-functions.svg @@ -0,0 +1,206 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + Sigmoid + Tanh + ReLU + Leaky ReLU + + diff --git a/sensecamp2019/img/analysis-windows.png b/sensecamp2019/img/analysis-windows.png new file mode 100644 index 0000000..1f65943 Binary files /dev/null and b/sensecamp2019/img/analysis-windows.png differ diff --git a/sensecamp2019/img/analysis-windows.svg b/sensecamp2019/img/analysis-windows.svg new file mode 100644 index 0000000..d411d26 --- /dev/null +++ b/sensecamp2019/img/analysis-windows.svg @@ -0,0 +1,343 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + Audio stream + + Analysis window + + + diff --git a/sensecamp2019/img/artificial-neuron.png b/sensecamp2019/img/artificial-neuron.png new file mode 100644 index 0000000..9f6ca2e Binary files /dev/null and b/sensecamp2019/img/artificial-neuron.png differ diff --git a/sensecamp2019/img/artificial-neuron.svg b/sensecamp2019/img/artificial-neuron.svg new file mode 100644 index 0000000..f320412 --- /dev/null +++ b/sensecamp2019/img/artificial-neuron.svg @@ -0,0 +1,328 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + Inputs + Weights + Output + Bias + + + + ... + + diff --git a/sensecamp2019/img/artificial-neuron.svg.2019_05_10_17_19_22.0.svg b/sensecamp2019/img/artificial-neuron.svg.2019_05_10_17_19_22.0.svg new file mode 100644 index 0000000..855332e --- /dev/null +++ b/sensecamp2019/img/artificial-neuron.svg.2019_05_10_17_19_22.0.svg @@ -0,0 +1,331 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + Inputs + Weights + Output + Bias + + + + ... + + diff --git a/sensecamp2019/img/audio-aquisition.png b/sensecamp2019/img/audio-aquisition.png new file mode 100644 index 0000000..1257342 Binary files /dev/null and b/sensecamp2019/img/audio-aquisition.png differ diff --git a/sensecamp2019/img/audio-aquisition.svg b/sensecamp2019/img/audio-aquisition.svg new file mode 100644 index 0000000..6c2f340 --- /dev/null +++ b/sensecamp2019/img/audio-aquisition.svg @@ -0,0 +1,828 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + Sound + Microphone + Analog-to-DigitalConverter + Digital Waveform + + + ADC + + + + + Mic + + Voltage + Numbers + Air Pressure + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/sensecamp2019/img/classification-pipeline.png b/sensecamp2019/img/classification-pipeline.png new file mode 100644 index 0000000..9c38e8b Binary files /dev/null and b/sensecamp2019/img/classification-pipeline.png differ diff --git a/sensecamp2019/img/classification-pipeline.svg b/sensecamp2019/img/classification-pipeline.svg new file mode 100644 index 0000000..af0e095 --- /dev/null +++ b/sensecamp2019/img/classification-pipeline.svg @@ -0,0 +1,788 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + Voting + + + + + + + + + + 0.1110.888........0.222 + Children playing + + + + + + + + + 0.1110.888 ...0.222 + + + + + + 1 x n_classes + FeatureExtractor + Classifier + + 60 melsx31 frames + + + + Prediction for clip + + Predictions forclassification window + + Audio + + Mel-spectrogram + 0.72 seconds + + diff --git a/sensecamp2019/img/confusion_test.png b/sensecamp2019/img/confusion_test.png new file mode 100644 index 0000000..71dc819 Binary files /dev/null and b/sensecamp2019/img/confusion_test.png differ diff --git a/sensecamp2019/img/conv-block-effnet.png b/sensecamp2019/img/conv-block-effnet.png new file mode 100644 index 0000000..018dbaa Binary files /dev/null and b/sensecamp2019/img/conv-block-effnet.png differ diff --git a/sensecamp2019/img/conv-block-mobilenet.png b/sensecamp2019/img/conv-block-mobilenet.png new file mode 100644 index 0000000..fb61ee8 Binary files /dev/null and b/sensecamp2019/img/conv-block-mobilenet.png differ diff --git a/sensecamp2019/img/conv-block-shufflenet.png b/sensecamp2019/img/conv-block-shufflenet.png new file mode 100644 index 0000000..5c24f9f Binary files /dev/null and b/sensecamp2019/img/conv-block-shufflenet.png differ diff --git a/sensecamp2019/img/conv-blocks-imagenets.png b/sensecamp2019/img/conv-blocks-imagenets.png new file mode 100644 index 0000000..11f0248 Binary files /dev/null and b/sensecamp2019/img/conv-blocks-imagenets.png differ diff --git a/sensecamp2019/img/conv-blocks-imagenets.svg b/sensecamp2019/img/conv-blocks-imagenets.svg new file mode 100644 index 0000000..9d11af4 --- /dev/null +++ b/sensecamp2019/img/conv-blocks-imagenets.svg @@ -0,0 +1,3370 @@ + + + + + + + + + + image/svg+xml + + + + + + + + + + EffNet + MobileNet + ShuffleNet + + + + + + + dw: Depthwise Convolutionmp: Max Poolinggc: Group Convolution + + + diff --git a/sensecamp2019/img/conv-depthwise-separable.png b/sensecamp2019/img/conv-depthwise-separable.png new file mode 100644 index 0000000..046bdce Binary files /dev/null and b/sensecamp2019/img/conv-depthwise-separable.png differ diff --git a/sensecamp2019/img/conv-grouped-1x1-g3.png b/sensecamp2019/img/conv-grouped-1x1-g3.png new file mode 100644 index 0000000..aca41c8 Binary files /dev/null and b/sensecamp2019/img/conv-grouped-1x1-g3.png differ diff --git a/sensecamp2019/img/conv-grouped-3x3-g3.png b/sensecamp2019/img/conv-grouped-3x3-g3.png new file mode 100644 index 0000000..76cbace Binary files /dev/null and b/sensecamp2019/img/conv-grouped-3x3-g3.png differ diff --git a/sensecamp2019/img/conv-standard.png b/sensecamp2019/img/conv-standard.png new file mode 100644 index 0000000..15ffb65 Binary files /dev/null and b/sensecamp2019/img/conv-standard.png differ diff --git a/sensecamp2019/img/convolution-2d.png b/sensecamp2019/img/convolution-2d.png new file mode 100644 index 0000000..48e85e3 Binary files /dev/null and b/sensecamp2019/img/convolution-2d.png differ diff --git a/sensecamp2019/img/convolution-2d.svg b/sensecamp2019/img/convolution-2d.svg new file mode 100644 index 0000000..4315912 --- /dev/null +++ b/sensecamp2019/img/convolution-2d.svg @@ -0,0 +1,1692 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + 1 + 1 + 1 + 0 + 0 + 0 + -1 + -1 + -1 + + + + + + + + + + + + 7 + 2 + 5 + 1 + 3 + 6 + 5 + 8 + 4 + 4 + 4 + 1 + 6 + 3 + 7 + 2 + + + 7 + 2 + 4 + 1 + 7 + 3 + 4 + 6 + 7 + Filter + Input + Completeoutput + + + + + + + + + 1 + 1 + 1 + 0 + 0 + 0 + -1 + -1 + -1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 7 + 2 + 5 + 1 + 3 + 6 + 5 + 8 + 4 + 4 + 4 + 1 + 6 + 3 + 7 + 2 + + + 7 + 2 + 4 + 1 + 7 + 3 + 4 + 6 + 7 + Filter swept over Input + + + 1 + 1 + 1 + 0 + 0 + 0 + -1 + -1 + -1 + + x + x + x + x + x + x + x + x + x + + + + + + + + + + + + + + + + + + + 1x7 + 1x2 + 1x5 0x7 + 0x2 + 0x5-1x3 + -1x6 + -1x5 = + + Filter output at one location + + -1 + 2 + + diff --git a/sensecamp2019/img/cortex-m4.jpeg b/sensecamp2019/img/cortex-m4.jpeg new file mode 100644 index 0000000..61d01e2 Binary files /dev/null and b/sensecamp2019/img/cortex-m4.jpeg differ diff --git a/sensecamp2019/img/cortexM4.png b/sensecamp2019/img/cortexM4.png new file mode 100644 index 0000000..377c0a5 Binary files /dev/null and b/sensecamp2019/img/cortexM4.png differ diff --git a/sensecamp2019/img/cortexM4.webp b/sensecamp2019/img/cortexM4.webp new file mode 100644 index 0000000..a115e9d Binary files /dev/null and b/sensecamp2019/img/cortexM4.webp differ diff --git a/sensecamp2019/img/cover.png b/sensecamp2019/img/cover.png new file mode 100644 index 0000000..f9a25c5 Binary files /dev/null and b/sensecamp2019/img/cover.png differ diff --git a/sensecamp2019/img/cover.svg b/sensecamp2019/img/cover.svg new file mode 100644 index 0000000..a249d02 --- /dev/null +++ b/sensecamp2019/img/cover.svg @@ -0,0 +1,587 @@ + + + + + + + + + + image/svg+xml + + + + + + + + Jon NordbyMaster of Science in Data Science26 June, 2018 + Environmental Sound Classificationon Microcontrollersusing Convolutional Neural Networks + + diff --git a/sensecamp2019/img/cpu-efficiency.png b/sensecamp2019/img/cpu-efficiency.png new file mode 100644 index 0000000..3526933 Binary files /dev/null and b/sensecamp2019/img/cpu-efficiency.png differ diff --git a/sensecamp2019/img/crossvalidation.png b/sensecamp2019/img/crossvalidation.png new file mode 100644 index 0000000..9a64ce9 Binary files /dev/null and b/sensecamp2019/img/crossvalidation.png differ diff --git a/sensecamp2019/img/crossvalidation.svg b/sensecamp2019/img/crossvalidation.svg new file mode 100644 index 0000000..3c6c01e --- /dev/null +++ b/sensecamp2019/img/crossvalidation.svg @@ -0,0 +1,954 @@ + + + + + + + + + + image/svg+xml + + + + + + + + + + + 1 + + + + 2 + + + + 3 + + + + 4 + + + + 5 + + + + + + + + Test + + + + + + + + + + + 1 + + + + 2 + + + + 3 + + + + 4 + + + + 5 + + + + 1 + + + + 2 + + + + 3 + + + + 5 + + + + 4 + + + + 1 + + + + 2 + + + + 4 + + + + 5 + + + + 3 + + + + 1 + + + + 3 + + + + 4 + + + + 5 + + + + 2 + + + + 2 + + + + 3 + + + + 4 + + + + 5 + + + + 1 + + Validation + Training data + + 5-fold split + Train/test split + Training with cross-validation + Full dataset + Final model evaluation + + + + + + + diff --git a/sensecamp2019/img/demo-tightcrop.jpg b/sensecamp2019/img/demo-tightcrop.jpg new file mode 100644 index 0000000..dea74d4 Binary files /dev/null and b/sensecamp2019/img/demo-tightcrop.jpg differ diff --git a/sensecamp2019/img/depthwise-separable-convolution.png b/sensecamp2019/img/depthwise-separable-convolution.png new file mode 100644 index 0000000..37a696b Binary files /dev/null and b/sensecamp2019/img/depthwise-separable-convolution.png differ diff --git a/sensecamp2019/img/depthwise-separable-convolution.svg b/sensecamp2019/img/depthwise-separable-convolution.svg new file mode 100644 index 0000000..c9a5455 --- /dev/null +++ b/sensecamp2019/img/depthwise-separable-convolution.svg @@ -0,0 +1,2100 @@ + + + + + + + + + + image/svg+xml + + + + + + + + + Standard Convolution + Depthwise Separable Convolution + + diff --git a/sensecamp2019/img/envnet.png b/sensecamp2019/img/envnet.png new file mode 100644 index 0000000..6b5e7b0 Binary files /dev/null and b/sensecamp2019/img/envnet.png differ diff --git a/sensecamp2019/img/fail-dropout.png b/sensecamp2019/img/fail-dropout.png new file mode 100644 index 0000000..cb634ff Binary files /dev/null and b/sensecamp2019/img/fail-dropout.png differ diff --git a/sensecamp2019/img/fail-truncation.png b/sensecamp2019/img/fail-truncation.png new file mode 100644 index 0000000..29af236 Binary files /dev/null and b/sensecamp2019/img/fail-truncation.png differ diff --git a/sensecamp2019/img/framing.png b/sensecamp2019/img/framing.png new file mode 100644 index 0000000..b627cb2 Binary files /dev/null and b/sensecamp2019/img/framing.png differ diff --git a/sensecamp2019/img/frontpage.png b/sensecamp2019/img/frontpage.png new file mode 100644 index 0000000..e773f3d Binary files /dev/null and b/sensecamp2019/img/frontpage.png differ diff --git a/sensecamp2019/img/grouped_confusion_test_foreground.png b/sensecamp2019/img/grouped_confusion_test_foreground.png new file mode 100644 index 0000000..53a3a3d Binary files /dev/null and b/sensecamp2019/img/grouped_confusion_test_foreground.png differ diff --git a/sensecamp2019/img/iCE40UltraPlus.png b/sensecamp2019/img/iCE40UltraPlus.png new file mode 100644 index 0000000..b138faa Binary files /dev/null and b/sensecamp2019/img/iCE40UltraPlus.png differ diff --git a/sensecamp2019/img/input-size.svg b/sensecamp2019/img/input-size.svg new file mode 100644 index 0000000..d09cd63 --- /dev/null +++ b/sensecamp2019/img/input-size.svg @@ -0,0 +1,322 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + 44.1kHz, 2 seconds, 128x128 + 16kHz, 0.75 seconds, 32x32 + + diff --git a/sensecamp2019/img/lenet5.png b/sensecamp2019/img/lenet5.png new file mode 100644 index 0000000..265d825 Binary files /dev/null and b/sensecamp2019/img/lenet5.png differ diff --git a/sensecamp2019/img/maxpooling.png b/sensecamp2019/img/maxpooling.png new file mode 100644 index 0000000..d7cc2e0 Binary files /dev/null and b/sensecamp2019/img/maxpooling.png differ diff --git a/sensecamp2019/img/maxpooling.svg b/sensecamp2019/img/maxpooling.svg new file mode 100644 index 0000000..cdaed86 --- /dev/null +++ b/sensecamp2019/img/maxpooling.svg @@ -0,0 +1,498 @@ + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + 7 + 2 + 5 + 1 + + + 3 + 6 + 5 + 8 + + + 4 + 4 + 4 + 1 + + + 6 + 3 + 7 + 2 + + + + + + + + 7 + 8 + 6 + 7 + + + + + + Maxpool2x2 filter2x2 stride + + diff --git a/sensecamp2019/img/models-list.png b/sensecamp2019/img/models-list.png new file mode 100644 index 0000000..74c7f52 Binary files /dev/null and b/sensecamp2019/img/models-list.png differ diff --git a/sensecamp2019/img/models.svg b/sensecamp2019/img/models.svg new file mode 100644 index 0000000..ed849c3 --- /dev/null +++ b/sensecamp2019/img/models.svg @@ -0,0 +1,813 @@ + + + + + + + + + + image/svg+xml + + + + + + + + + Conv2d + + + + CONV + + + + CONV + + + + Flatten + + 24 + 36 + 54 + + + Dense + 64 + + + + Dense + 10 + + + + Conv2d + + + + CONV + + + + CONV + + + + Flatten + + 24 + 48 + 48 + + + Dense + 64 + + + 3,2 + + MaxPool2d + 3,2 + MaxPool2d + Baseline + ReLu + BatchNorm + + + 10 + Dense + + Dropout + ReLu + BatchNorm + BatchNorm + ReLu + Dropout + ReLu + Stride + 2,2 + 2,2 + 2,2 + ReLu + BatchNorm + ReLu + BatchNorm + BatchNorm + ReLu + Softmax + Softmax + Dropout + ReLu + Dropout + F + F + D + D + + + diff --git a/sensecamp2019/img/models_accuracy.png b/sensecamp2019/img/models_accuracy.png new file mode 100644 index 0000000..05cf651 Binary files /dev/null and b/sensecamp2019/img/models_accuracy.png differ diff --git a/sensecamp2019/img/models_efficiency.png b/sensecamp2019/img/models_efficiency.png new file mode 100644 index 0000000..e942c04 Binary files /dev/null and b/sensecamp2019/img/models_efficiency.png differ diff --git a/sensecamp2019/img/multilayer-perceptron.png b/sensecamp2019/img/multilayer-perceptron.png new file mode 100644 index 0000000..8879e0e Binary files /dev/null and b/sensecamp2019/img/multilayer-perceptron.png differ diff --git a/sensecamp2019/img/multilayer-perceptron.svg b/sensecamp2019/img/multilayer-perceptron.svg new file mode 100644 index 0000000..4e2ce30 --- /dev/null +++ b/sensecamp2019/img/multilayer-perceptron.svg @@ -0,0 +1,789 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ... + ... + ... + Input layer + Hidden layers + Output layer + + diff --git a/sensecamp2019/img/nmbu_logo_eng_rgb.jpg b/sensecamp2019/img/nmbu_logo_eng_rgb.jpg new file mode 100644 index 0000000..6d6a8d7 Binary files /dev/null and b/sensecamp2019/img/nmbu_logo_eng_rgb.jpg differ diff --git a/sensecamp2019/img/nmbu_logo_eng_rgb_trans.png b/sensecamp2019/img/nmbu_logo_eng_rgb_trans.png new file mode 100644 index 0000000..d1c38b4 Binary files /dev/null and b/sensecamp2019/img/nmbu_logo_eng_rgb_trans.png differ diff --git a/sensecamp2019/img/noise-monitoring.jpg b/sensecamp2019/img/noise-monitoring.jpg new file mode 100644 index 0000000..37cb80d Binary files /dev/null and b/sensecamp2019/img/noise-monitoring.jpg differ diff --git a/sensecamp2019/img/noiseseverity.png b/sensecamp2019/img/noiseseverity.png new file mode 100644 index 0000000..5c94219 Binary files /dev/null and b/sensecamp2019/img/noiseseverity.png differ diff --git a/sensecamp2019/img/noiseseverity.svg b/sensecamp2019/img/noiseseverity.svg new file mode 100644 index 0000000..e3309f9 --- /dev/null +++ b/sensecamp2019/img/noiseseverity.svg @@ -0,0 +1,355 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + Mortality + Disease + Risk factors + Stress indicators + Feeling of discomfort + (insomnia, cardiovascular) + (blood pressure, cholesterol,blood clotting, glucose) + (autonomous response, stress hormones) + (disturbance, annoyance, sleep disturbance) + Number of people affected + Severity + + diff --git a/sensecamp2019/img/piczak-cnn.png b/sensecamp2019/img/piczak-cnn.png new file mode 100644 index 0000000..a3f6983 Binary files /dev/null and b/sensecamp2019/img/piczak-cnn.png differ diff --git a/sensecamp2019/img/results.csv b/sensecamp2019/img/results.csv new file mode 100644 index 0000000..98cab63 --- /dev/null +++ b/sensecamp2019/img/results.csv @@ -0,0 +1,12 @@ +experiment,result_path,maccs_frame,flash_usage,ram_usage_max,ram_usage_min,test_acc_mean,foreground_test_acc_mean,background_test_acc_mean,model,conv_block,n_stages,conv_size,downsample_size,filters,modelcheck,nickname,classifications_per_second +0,./data/results/20190501-0223/0.confusion.npz,10185806.0,415100.0,36290.0,36290.0,0.7311827956989249,0.8427230046948355,0.4956629491945476,sbcnn,conv,3,5x5,3x2,24,skip,Baseline,2.7777777777777777 +1,./data/results/20190501-0223/1.confusion.npz,2980798.0,381150.0,56720.0,56720.0,0.7185716182131953,0.8210093896713615,0.5022717885171417,strided,conv,3,5x5,2x2,22,,Stride-5x5,2.7777777777777777 +10,./data/results/20190501-0223/10.confusion.npz,468649.0,128750.0,48750.0,48750.0,0.6713128899508827,0.7546948356807511,0.49524989673688563,strided,effnet,3,5x5,2x2,22,,Stride-Effnet-5x5,2.7777777777777777 +2,./data/results/20190501-0223/2.confusion.npz,477236.0,184640.0,56250.0,56250.0,0.7254745785211735,0.8133802816901408,0.5398595621643948,strided,depthwise_separable,3,5x5,2x2,24,,Stride-DS-5x5,2.7777777777777777 +3,./data/results/20190501-0223/3.confusion.npz,318497.0,97650.0,56250.0,56250.0,0.7011814682065578,0.7938184663536776,0.5055762081784386,strided,depthwise_separable,4,3x3,2x2,24,,Stride-DS-3x3,2.7777777777777777 +4,./data/results/20190501-0223/4.confusion.npz,445688.0,81940.0,48750.0,48750.0,0.685517058276915,0.7767996870109547,0.49277158199091287,strided,bottleneck_ds,3,5x5,2x2,22,,Stride-BN-DS-5x5,2.7777777777777777 +5,./data/results/20190501-0223/5.confusion.npz,477236.0,184640.0,56250.0,56250.0,0.7145891411124385,0.812793427230047,0.5072284180090871,strided,depthwise_separable,3,5x5,2x2,24,,DS-5x5-24,2.7777777777777777 +6,./data/results/20190501-0223/6.confusion.npz,380749.0,152810.0,46880.0,46880.0,0.7285278109650869,0.8194444444444443,0.5365551425030979,strided,depthwise_separable,3,5x5,2x2,20,,DS-5x5-20,2.7777777777777777 +7,./data/results/20190501-0223/7.confusion.npz,291318.0,121590.0,37500.0,37500.0,0.7155183857692818,0.8092723004694835,0.5175547294506402,strided,depthwise_separable,3,5x5,2x2,16,,DS-5x5-16,2.7777777777777777 +8,./data/results/20190501-0223/8.confusion.npz,208943.0,90970.0,28130.0,28130.0,0.6998539758396389,0.7924491392801252,0.5043370508054523,strided,depthwise_separable,3,5x5,2x2,12,,DS-5x5-12,2.7777777777777777 +9,./data/results/20190501-0223/9.confusion.npz,1567280.0,98410.0,56350.0,56350.0,0.7265365724147085,0.8395931142410016,0.4878149524989674,sbcnn,depthwise_separable,3,5x5,3x2,24,,Baseline-DS,2.7777777777777777 diff --git a/sensecamp2019/img/results.png b/sensecamp2019/img/results.png new file mode 100644 index 0000000..71df9da Binary files /dev/null and b/sensecamp2019/img/results.png differ diff --git a/sensecamp2019/img/sensornetworks.png b/sensecamp2019/img/sensornetworks.png new file mode 100644 index 0000000..992fc12 Binary files /dev/null and b/sensecamp2019/img/sensornetworks.png differ diff --git a/sensecamp2019/img/sensornetworks.svg b/sensecamp2019/img/sensornetworks.svg new file mode 100644 index 0000000..ae671f0 --- /dev/null +++ b/sensecamp2019/img/sensornetworks.svg @@ -0,0 +1,6254 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + B) Spectrogram + C) Audio Embedding + D) Sensor classification + + A) Audio transmission + + + [ 0.4, 0.9, 0.1, 0.2, 0.3,, .. .. .. .. ... 0.7, 0.5, 0.9, 0.2] + + 128 + + + + + + + + + + + + + + + Spectrogram Extraction + Classification + Feature Learning + + + + + + + + + [ 0.11, 0.05, 0.88, 0.22, 0.12, 0.09, 0.30, 0.07, 0.04] + 'car horn' + + + + + + + diff --git a/sensecamp2019/img/sensortile-annotated.jpg b/sensecamp2019/img/sensortile-annotated.jpg new file mode 100644 index 0000000..191f620 Binary files /dev/null and b/sensecamp2019/img/sensortile-annotated.jpg differ diff --git a/sensecamp2019/img/sensortile-annotated.svg b/sensecamp2019/img/sensortile-annotated.svg new file mode 100644 index 0000000..736e041 --- /dev/null +++ b/sensecamp2019/img/sensortile-annotated.svg @@ -0,0 +1,2149 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + Microcontroller + + Bluetooth + + Accelerometer, Gyro, Compass + + + Microphone + + + + + + + + + diff --git a/sensecamp2019/img/sensortile-devkit.jpg b/sensecamp2019/img/sensortile-devkit.jpg new file mode 100644 index 0000000..bb3d3ea Binary files /dev/null and b/sensecamp2019/img/sensortile-devkit.jpg differ diff --git a/sensecamp2019/img/soundsensing-logo.png b/sensecamp2019/img/soundsensing-logo.png new file mode 100644 index 0000000..8c41f81 Binary files /dev/null and b/sensecamp2019/img/soundsensing-logo.png differ diff --git a/sensecamp2019/img/soundsensing-logo.xcf b/sensecamp2019/img/soundsensing-logo.xcf new file mode 100644 index 0000000..f5d3ada Binary files /dev/null and b/sensecamp2019/img/soundsensing-logo.xcf differ diff --git a/sensecamp2019/img/soundsensing-sensor-metro.jpg b/sensecamp2019/img/soundsensing-sensor-metro.jpg new file mode 100644 index 0000000..56a1c04 Binary files /dev/null and b/sensecamp2019/img/soundsensing-sensor-metro.jpg differ diff --git a/sensecamp2019/img/soundsensing-withlogo.png b/sensecamp2019/img/soundsensing-withlogo.png new file mode 100644 index 0000000..0050358 Binary files /dev/null and b/sensecamp2019/img/soundsensing-withlogo.png differ diff --git a/sensecamp2019/img/soundsensing-withlogo.svg b/sensecamp2019/img/soundsensing-withlogo.svg new file mode 100644 index 0000000..386e606 --- /dev/null +++ b/sensecamp2019/img/soundsensing-withlogo.svg @@ -0,0 +1,52247 @@ + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/sensecamp2019/img/spatially-separable-convolution.png b/sensecamp2019/img/spatially-separable-convolution.png new file mode 100644 index 0000000..22fb1e5 Binary files /dev/null and b/sensecamp2019/img/spatially-separable-convolution.png differ diff --git a/sensecamp2019/img/spatially-separable-convolution.svg b/sensecamp2019/img/spatially-separable-convolution.svg new file mode 100644 index 0000000..a3f267c --- /dev/null +++ b/sensecamp2019/img/spatially-separable-convolution.svg @@ -0,0 +1,611 @@ + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + Input + Output + + + Input + + + + Intermediate + + + + Output + + 3x3convolution + + Spatially Separable Convolution + + + 3x1convolution + + + + + 1x3convolution + + + Standard Convolution + + diff --git a/sensecamp2019/img/spectrograms.svg b/sensecamp2019/img/spectrograms.svg new file mode 100644 index 0000000..fce3261 --- /dev/null +++ b/sensecamp2019/img/spectrograms.svg @@ -0,0 +1,13106 @@ + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + diff --git a/sensecamp2019/img/stm32cubeai.png b/sensecamp2019/img/stm32cubeai.png new file mode 100644 index 0000000..2675d88 Binary files /dev/null and b/sensecamp2019/img/stm32cubeai.png differ diff --git a/sensecamp2019/img/stoykart.png b/sensecamp2019/img/stoykart.png new file mode 100644 index 0000000..c3311a9 Binary files /dev/null and b/sensecamp2019/img/stoykart.png differ diff --git a/sensecamp2019/img/strided-convolution.png b/sensecamp2019/img/strided-convolution.png new file mode 100644 index 0000000..fe478ac Binary files /dev/null and b/sensecamp2019/img/strided-convolution.png differ diff --git a/sensecamp2019/img/strided-convolution.svg b/sensecamp2019/img/strided-convolution.svg new file mode 100644 index 0000000..9111228 --- /dev/null +++ b/sensecamp2019/img/strided-convolution.svg @@ -0,0 +1,868 @@ + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + 1 + 1 + 1 + 0 + 0 + 0 + -1 + -1 + -1 + + + + + + + + + + + + + + + + + + 7 + 2 + 5 + 1 + 3 + 6 + 5 + 8 + 4 + 4 + 4 + 1 + 6 + 3 + 7 + 2 + + + 7 + 2 + 5 + 1 + 7 + 3 + 4 + 6 + 7 + + 3x3 filter + 5x5 input + + + + stride: 2 + + + = + 2x2 output + + + + + + + + + + 1 + 1 + 1 + 0 + 0 + 0 + -1 + -1 + -1 + + + diff --git a/sensecamp2019/img/thesis.png b/sensecamp2019/img/thesis.png new file mode 100644 index 0000000..7299af0 Binary files /dev/null and b/sensecamp2019/img/thesis.png differ diff --git a/sensecamp2019/img/training-inference.png b/sensecamp2019/img/training-inference.png new file mode 100644 index 0000000..db822d5 Binary files /dev/null and b/sensecamp2019/img/training-inference.png differ diff --git a/sensecamp2019/img/training-inference.svg b/sensecamp2019/img/training-inference.svg new file mode 100644 index 0000000..af5f59a --- /dev/null +++ b/sensecamp2019/img/training-inference.svg @@ -0,0 +1,594 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + Hyperparameters + Training data + Labels + + Adjust model + + Calculate error + + + + + + Trainingsystem + + + Predictivemodel + + + Classify + + Preprocess + Data + Predictions + Parameters + + + + + + + + Data + + diff --git a/sensecamp2019/img/training-settings.png b/sensecamp2019/img/training-settings.png new file mode 100644 index 0000000..fbe24e4 Binary files /dev/null and b/sensecamp2019/img/training-settings.png differ diff --git a/sensecamp2019/img/unknown-class-tradeoffs.png b/sensecamp2019/img/unknown-class-tradeoffs.png new file mode 100644 index 0000000..d525746 Binary files /dev/null and b/sensecamp2019/img/unknown-class-tradeoffs.png differ diff --git a/sensecamp2019/img/unknown-class.png b/sensecamp2019/img/unknown-class.png new file mode 100644 index 0000000..3df8b79 Binary files /dev/null and b/sensecamp2019/img/unknown-class.png differ diff --git a/sensecamp2019/img/urbansound8k-examples.png b/sensecamp2019/img/urbansound8k-examples.png new file mode 100644 index 0000000..5ddb53c Binary files /dev/null and b/sensecamp2019/img/urbansound8k-examples.png differ diff --git a/sensecamp2019/img/urbansound8k-existing-models-logmel.png b/sensecamp2019/img/urbansound8k-existing-models-logmel.png new file mode 100644 index 0000000..6552a2d Binary files /dev/null and b/sensecamp2019/img/urbansound8k-existing-models-logmel.png differ diff --git a/sensecamp2019/img/urbansound8k-existing-models-logmel.tex b/sensecamp2019/img/urbansound8k-existing-models-logmel.tex new file mode 100644 index 0000000..caa3c73 --- /dev/null +++ b/sensecamp2019/img/urbansound8k-existing-models-logmel.tex @@ -0,0 +1,12 @@ +\begin{tabular}{lrrr} +\toprule +{} & Accuracy (\%) & MACC / second & Model parameters \\ +name & & & \\ +\midrule +Dmix-CNN-mel & 82.6 & 298M & 1180k \\ +D-CNN & 81.9 & 458M & 33000k \\ +SB-CNN & 79.0 & 25M & 432k \\ +LD-CNN & 79.0 & 10M & 580k \\ +PiczakCNN & 75.0 & 88M & 25534k \\ +\bottomrule +\end{tabular} diff --git a/sensecamp2019/img/what-we-do.png b/sensecamp2019/img/what-we-do.png new file mode 100644 index 0000000..c96d285 Binary files /dev/null and b/sensecamp2019/img/what-we-do.png differ diff --git a/sensecamp2019/img/xcubeai.png b/sensecamp2019/img/xcubeai.png new file mode 100644 index 0000000..0db1da0 Binary files /dev/null and b/sensecamp2019/img/xcubeai.png differ diff --git a/sensecamp2019/notes.md b/sensecamp2019/notes.md new file mode 100644 index 0000000..344b57e --- /dev/null +++ b/sensecamp2019/notes.md @@ -0,0 +1,170 @@ + + +## Talk 1 + +DTU. Danish Technical University + +Demo. DANSpeech system + + +Safe AI. +Trustworthy. +Explainable AI. + +! New book. Explainable AI, Interpreting Explaining and Visualizing Deep Learning + +DUT. Introduction to ML. 1000 engineering students per year. + +Close collaboration with Hearing Systems. +ML used quite a bit. +Early patenting. + +DABAI. Open source ML workflow. +Danish language. + + +Data and Representation + +- Division of labor. +- Neural network. Simple units. +Reused in different contexts. +- Learning. Adaptivity, Plasticity +- Attention + +What is it we attend to? +Race to short term memory + +Looking for things that "move indepdendetly" + + +Paper. +Lewincki, 2002. Efficient Coding of natural sounds. +Adapted ICA in primarly audidotury. +Mel MFCC + +How to model this independence? + +Comparing generative/unsupervised with supervised learning + +Train independently. Are they similar? + +? how do they estimate the similarity. Mutual information? + +Ling Feng. + +! can predict height of person based on their speech + +ICLR2016. +Convergent Learning. +Do different neural networks learn the same representation? +Authors suggest Yes! + +! Controversial paper + +Neural networks are permutation invariant. + + +Podcast indexing based on audio +- AudiobUrst + +Using wikipedia as knowledge based of ML system + +Danspeech + +http://github.com/danspeech/danspeech +"An open-source python package for Danish speech recognition" + + + +## Talk 2 + +WS Audiology + +Started PhD in 2015 +Everyone talks about "Big Data" + +Dreams about future in 2017 + +Rule-based system -> Data driven +Opinion-based decision -> Fact-based +Increasing system/use complexity -> Simple, intutive + +Prerequisites + +- Data Collection +- Data Warehousing +- High performance Compute +- Machine Learning + + +### SoundSense Learn + +Mobile app. +Allows to play around with personalized aud +Primarily for hearing aid tuning +Compares 2-and-2 settings + +Using Baysian optimization for Active Learning + +Jensen et. al 2019 + + +### Data Infrastructure + +From Lake to Warehouse + +Go from lots of unstructured data. Independent. +To lower amount of strucutred data. Inter-linked. Contextualized. + + + +## Talk 3 + +Eriksholm Research Center + +``` +Audio Enhancment ++ Preferences += new services for hearing in healthcare +``` + +Hearing aids + +Source separation + +! want to do it on hearing aid hardware +! with 8-10 millisecond latency + +Single microphone +For more than 2 people need directionality + +Competition? +Hearables. Will get more crowded + +Training? +Audio-Visual correspondence + +EVOTIUM Consurtiom + + +### Privacy perserving + +Temporal +1 sample per minute + +SPL 3 bands +Noise floor. + +Environment Classification. +Quiet, Noise, Speech, Noisy Speech + +EOTOION Hearing Aid data + +51 million datapoints +1000 users + +? is some of this dataset available + +5 M parameters + + diff --git a/sensecamp2019/presentation.md b/sensecamp2019/presentation.md new file mode 100644 index 0000000..4d1f8cf --- /dev/null +++ b/sensecamp2019/presentation.md @@ -0,0 +1,673 @@ + +--- +title: Classification of Environmental Sound using IoT sensors +author: Jon Nordby +date: November 19, 2019 +css: style.css +width: 1920 +height: 1080 +margin: 0 +pagetitle: 'Sensecamp2019: Classification of Environmental Sound using IoT sensors' +--- + + +# Introduction + +## Jon Nordby + +Internet of Things specialist + +- B.Eng in **Electronics** +- 9 years as **Software** developer. **Embedded** + **Web** +- M. Sc in **Data** Science + +Now: + +- CTO at Soundsensing +- Machine Learning Consultant + + +## Soundsensing + +![](./img/soundsensing-withlogo.png){width=100%} + + +::: notes +Provide **Noise Monitoring** and Audio **Condition Monitoring** solutions +that are used in Real-Estate, Industry, and Smart Cities. + +Perform Machine Learning for sound classification **on sensor**. +::: + + +## Dashboard + +![Pilot projects with customers Now - 2020](img/what-we-do.png) + + +## Thesis + +> Environmental Sound Classification +> on Microcontrollers +> using Convolutional Neural Networks + +![Report & Code: https://github.com/jonnor/ESC-CNN-microcontroller](./img/thesis.png){width=30%} + + +## Wireless Sensor Networks + +- Want: Wide and dense coverage +- Need: Sensors need to be low-cost +- **Opportunity**: Wireless reduces costs +- **Challenge**: Power consumption + +::: notes + +* No network cabling, no power cabling +* No site infrastructure needed +* Less invasive +* Fewer approvals needed +* Temporary installs feasible +* Mobile sensors possible + +Electrician is 750 NOK/hour + +Image: https://www.nti-audio.com/en/applications/noise-measurement/unattended-monitoring +::: + + +## Sensor Network Architectures + +![](img/sensornetworks.png){width=70%} + + +# Audio Machine Learning on low-power sensors + +## What do you mean by low-power? + +Want: 1 year lifetime for palm-sized battery + +Need: `<1mW` system power + +## General purpose microcontroller + + +![](img/cortexM4.png){width=40%} + +STM32L4 @ 80 MHz. Approx **10 mW**. + +- TensorFlow Lite for Microcontrollers (Google) +- ST X-CUBE-AI (ST Microelectronics) + + +## FPGA + +![Lattice ICE40 UltraPlus with Lattice sensAI](img/iCE40UltraPlus.png){width=50%} + +Human presence detection. VGG8 on 64x64 RGB image, 5 FPS: 7 mW. + +Audio ML approx **1 mW** + +## Neural Network co-processors + +![Project Orlando (ST Microelectronics), expected 2020](img/ST-Orlando-SoC.png){width=25%} + +2.9 TOPS/W. AlexNet, 1000 classes, 10 FPS. 41 mWatt + +Audio models probably **< 1 mWatt**. + +::: notes + +https://www.latticesemi.com/Blog/2019/05/17/18/25/sensAI + +::: + + +# On-edge Classification of Noise + +## Environmental Sound Classification + +> Given an audio signal of environmental sounds, +> +> determine which class it belongs to + +* Widely researched. 1000 hits on Google Scholar +* Datasets. Urbansound8k (10 classes), ESC-50, AudioSet (632 classes) +* 2017: Human-level performance on ESC-50 + +::: notes + +https://github.com/karoldvl/ESC-50 + +::: + + + + + +::: notes + +STM32L476 + +ARM Cortex M4F +Hardware floating-point unit (FPU) +DSP SIMD instructions +80 MHz CPU clock +1024 kB of program memory (Flash) +128 kB of RAM. + +25 mWatt max + +::: + +## Urbansound8k + +![](img/urbansound8k-examples.png){width=100%} + +::: notes + +Classes from an urban sound taxonomy, +based on noise complains in New York city + +Most sounds around 4 seconds. Some classes around 1 second + +Foreground/background + +::: + + +## Existing work + +- Convolutional Neural Networks dominate +- Techniques come from image classification +- Mel-spectrogram input standard +- End2end models: getting close in accuracy +- "Edge ML" focused on mobile-phone class HW +- "Tiny ML" (sensors) just starting + +::: notes + +* Efficient Keyword-Spotting +* Efficient (image) CNNs +* Efficient ESC-CNN + +ESC-CNN + +* 23 papers reviewed in detail +* 10 referenced in thesis +* Only 4 consider computational efficiency + +::: + +## Model requirements + +With 50% of STM32L476 capacity: + +* 64 kB RAM +* 512 kB FLASH memory +* 4.5 M MACC/second + +::: notes + +* RAM: 1000x 64 MB +* PROGMEM: 1000x 512 MB +* CPU: 1000x 5 GFLOPS +* GPU: 1000'000X 5 TFLOPS + +::: + +## Existing models + +![Green: Feasible region](img/urbansound8k-existing-models-logmel.png){width=100%} + +eGRU: running on ARM Cortex-M0 microcontroller, accuracy 61% with **non-standard** evaluation + +::: notes + +Assuming no overlap. Most models use very high overlap, 100X higher compute + +::: + +## Pipeline + +![](img/classification-pipeline.png){max-height=100%} + + +## Models + + + +![](img/models.svg){width=70%} + + +::: notes + +Baseline from SB-CNN + +Few modifications + +* Uses smaller input feature representation +* Reduced downsample factor to accommodate + +CONV = entry point for trying different convolution operators + +::: + + + + +# Strategies for shrinking Convolutional Neural Network + + +## Reduce input dimensionality + +![](img/input-size.svg){width=70%} + +- Lower frequency range +- Lower frequency resolution +- Lower time duration in window +- Lower time resolution + +::: notes + +Directly limits time and RAM use first few layers. + +Follow-on effects. +A simpler input representation is (hopefully) easier to learn +allowing for a simpler model + +TODO: make a picture illustrating this + +::: + +## Reduce overlap + +![](img/framing.png){width=80%} + +Models in literature use 95% overlap or more. 20x penalty in inference time! + +Often low performance benefit. Use 0% (1x) or 50% (2x). + + + +## Depthwise-separable Convolution + + +![](img/depthwise-separable-convolution.png){width=90%} + +MobileNet, "Hello Edge", AclNet. 3x3 kernel,64 filters: 7.5x speedup + +::: notes + +* Much fewer operations +* Less expressive - but regularization effect can be beneficial + +::: + +## Spatially-separable Convolution + +![](img/spatially-separable-convolution.png){width=90%} + +EffNet, LD-CNN. 5x5 kernel: 2.5x speedup + + +## Downsampling using max-pooling + +![](img/maxpooling.png){width=100%} + +Wasteful? Computing convolutions, then throwing away 3/4 of results! + +## Downsampling using strided convolution + +![](img/strided-convolution.png){width=100%} + +Striding means fewer computations and "learned" downsampling + +## Model comparison + +![](img/models_accuracy.png){width=100%} + +::: notes + +- Baseline relative to SB-CNN and LD-CNN is down from 79% to 73% +Expected because poorer input representation. +Much lower overlap + +::: + + +## Performance vs compute + +![](img/models_efficiency.png){width=100%} + +::: + +- Performance of Strided-DS-24 similar to Baseline despite 12x the CPU use +- Suprising? Stride alone worse than Strided-DS-24 +- Bottleneck and EffNet performed poorly +- Practical speedup not linear with MACC + +::: + + + + + +## Quantization + +Inference can often use 8 bit integers instead of 32 bit floats + +- 1/4 the size for weights (FLASH) and activations (RAM) +- 8bit **SIMD** on ARM Cortex M4F: 1/4 the inference time +- Supported in X-CUBE-AI 4.x (July 2019) + + + +::: notes + +EnvNet-v2 got 78.3% on Urbansound8k with 16 kHz +::: + + + +## Conclusions + +- Able to perform Environmental Sound Classification at `~ 10mW` power, +- Using *general purpose microcontroller*, ARM Cortex M4F +- Best performance: 70.9% mean accuracy, under 20% CPU load +- Highest reported Urbansound8k on microcontroller (over eGRU 62%) +- Best architecture: Depthwise-Separable convolutions with striding +- Quantization enables 4x bigger models (and higher perf) +- With dedicated Neural Network Hardware + + +# Further Research + + +## Waveform input to model + +- Preprocessing. Mel-spectrogram: **60** milliseconds +- CNN. Stride-DS-24: **81** milliseconds +- With quantization, spectrogram conversion is the bottleneck! +- Convolutions can be used to learn a Time-Frequency transformation. + +Can this be faster than the standard FFT? And still perform well? + + +::: notes + +- Especially interesting with CNN hardware acceleration. + +::: + + +## On-sensor inference challenges + +- Reducing power consumption. Adaptive sampling +- Efficient training data collection in WSN. Active Learning? +- Real-life performance evaluations. Out-of-domain samples + +::: notes + +TODO: Add few more projects here. From research document + +::: + + + +# Wrapping up + +## Summary + +- Noise pollution is a growing problem +- Wireless Sensor Networks can used to quantify +- Noise Classification can provide more information +- Want high density of sensors. Need to be low cost +- On-sensor classification desirable for power/cost and privacy + + +## More resources + +Machine Hearing. ML on Audio + +- [github.com/jonnor/machinehearing](https://github.com/jonnor/machinehearing) + +Machine Learning for Embedded / IoT + +- [github.com/jonnor/embeddedml](https://github.com/jonnor/embeddedml) + +Thesis Report & Code + +- [github.com/jonnor/ESC-CNN-microcontroller](https://github.com/jonnor/ESC-CNN-microcontroller) + + +## Questions + +

?

+ +Email: + +## Come talk to me! + +- Noise Monitoring sensors. Pilot projects for 2020? +- Environmental Sound, Wireless Sensor Networks for Audio. Research partnering? +- "On-edge" / Embedded Device ML. Happy to advise! + +Email: + + + + +# Thesis results + + +## Model comparison + +![](img/models_accuracy.png){width=100%} + +::: notes + +- Baseline relative to SB-CNN and LD-CNN is down from 79% to 73% +Expected because poorer input representation. +Much lower overlap + +::: + + + +## List of results + +![](img/results.png){width=100%} + + +## Confusion + +![](img/confusion_test.png){width=70%} + +## Grouped classification + +![](img/grouped_confusion_test_foreground.png){width=60%} + +Foreground-only + +## Unknown class + +![](img/unknown-class.png){width=100%} + +::: notes + +Idea: If confidence of model is low, consider it as "unknown" + +* Left: Histogram of correct/incorrect predictions +* Right: Precision/recall curves +* Precision improves at expense of recall +* 90%+ precision possible at 40% recall + +Usefulness: + +* Avoids making decisions on poor grounds +* "Unknown" samples good candidates for labeling->dataset. Active Learning +* Low recall not a problem? Data is abundant, 15 samples a 4 seconds per minute per sensor + +::: + + +# Experimental Details + + +## All models + +![](img/models-list.png) + +::: notes + +* Baseline is outside requirements +* Rest fits the theoretical constraints +* Sometimes had to reduce number of base filters to 22 to fit in RAM + +::: + + +# Methods + +Standard procedure for Urbansound8k + +- Classification problem +- 4 second sound clips +- 10 classes +- 10-fold cross-validation, predefined +- Metric: Accuracy + +## Training settings + +![](img/training-settings.png) + +## Training + +- NVidia RTX2060 GPU 6 GB +- 10 models x 10 folds = 100 training jobs +- 100 epochs +- 3 jobs in parallel +- 36 hours total + +::: notes + +- ! GPU utilization only 15% +- CPU utilization was near 100% +- Larger models to utilize GPU better? +- Parallel processing limited by RAM of biggest models +- GPU-based augmentation might be faster + +::: + +## Evaluation + +For each fold of each model + +1. Select best model based on validation accuracy +2. Calculate accuracy on test set + +For each model + +- Measure CPU time on device + + +# Your model will trick you + +And the bugs can be hard to spot + +## FAIL: Integer truncation + +![](img/fail-truncation.png){width=100%} + +## FAIL. Dropout location + +![](img/fail-dropout.png){width=100%} + + +# Background + + +## Mel-spectrogram + +![](img/spectrograms.svg) + +## Noise Pollution + +Reduces health due to stress and loss of sleep + +In Norway + +* 1.9 million affected by road noise (2014, SSB) +* 10'000 healty years lost per year (Folkehelseinstituttet) + +In Europe + +* 13 million suffering from sleep disturbance (EEA) +* 900'000 DALY lost (WHO) + + +::: notes + +1.9 million +https://www.ssb.no/natur-og-miljo/artikler-og-publikasjoner/flere-nordmenn-utsatt-for-stoy + +1999: 1.2 million + +10 245 tapte friske leveår i Norge hvert år +https://www.miljostatus.no/tema/stoy/stoy-og-helse/ + + +https://www.eea.europa.eu/themes/human/noise/noise-2 + +Burden of Disease WHO +http://www.euro.who.int/__data/assets/pdf_file/0008/136466/e94888.pdf + +::: + + +## Noise Mapping + +Simulation only, no direct measurements + +![](img/stoykart.png) + +::: notes + +- Known sources +- Yearly average value +- Updated every 5 years +- Low data quality. Ex: communal roads + +Image: https://www.regjeringen.no/no/tema/plan-bygg-og-eiendom/plan--og-bygningsloven/plan/kunnskapsgrunnlaget-i-planlegging/statistikk-i-plan/id2396747/ + +::: + + diff --git a/sensecamp2019/reveal.js/CONTRIBUTING.md b/sensecamp2019/reveal.js/CONTRIBUTING.md new file mode 100644 index 0000000..c2091e8 --- /dev/null +++ b/sensecamp2019/reveal.js/CONTRIBUTING.md @@ -0,0 +1,23 @@ +## Contributing + +Please keep the [issue tracker](http://github.com/hakimel/reveal.js/issues) limited to **bug reports**, **feature requests** and **pull requests**. + + +### Personal Support +If you have personal support or setup questions the best place to ask those are [StackOverflow](http://stackoverflow.com/questions/tagged/reveal.js). + + +### Bug Reports +When reporting a bug make sure to include information about which browser and operating system you are on as well as the necessary steps to reproduce the issue. If possible please include a link to a sample presentation where the bug can be tested. + + +### Pull Requests +- Should follow the coding style of the file you work in, most importantly: + - Tabs to indent + - Single-quoted strings +- Should be made towards the **dev branch** +- Should be submitted from a feature/topic branch (not your master) + + +### Plugins +Please do not submit plugins as pull requests. They should be maintained in their own separate repository. More information here: https://github.com/hakimel/reveal.js/wiki/Plugin-Guidelines diff --git a/sensecamp2019/reveal.js/LICENSE b/sensecamp2019/reveal.js/LICENSE new file mode 100644 index 0000000..697d156 --- /dev/null +++ b/sensecamp2019/reveal.js/LICENSE @@ -0,0 +1,19 @@ +Copyright (C) 2019 Hakim El Hattab, http://hakim.se, and reveal.js contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. \ No newline at end of file diff --git a/sensecamp2019/reveal.js/README.md b/sensecamp2019/reveal.js/README.md new file mode 100644 index 0000000..33956e9 --- /dev/null +++ b/sensecamp2019/reveal.js/README.md @@ -0,0 +1,1438 @@ +# reveal.js [![Build Status](https://travis-ci.org/hakimel/reveal.js.svg?branch=master)](https://travis-ci.org/hakimel/reveal.js) Slides + +A framework for easily creating beautiful presentations using HTML. [Check out the live demo](http://revealjs.com/). + +reveal.js comes with a broad range of features including [nested slides](https://github.com/hakimel/reveal.js#markup), [Markdown contents](https://github.com/hakimel/reveal.js#markdown), [PDF export](https://github.com/hakimel/reveal.js#pdf-export), [speaker notes](https://github.com/hakimel/reveal.js#speaker-notes) and a [JavaScript API](https://github.com/hakimel/reveal.js#api). There's also a fully featured visual editor and platform for sharing reveal.js presentations at [slides.com](https://slides.com?ref=github). + + +## Table of contents + +- [Online Editor](#online-editor) +- [Installation](#installation) + - [Basic setup](#basic-setup) + - [Full setup](#full-setup) + - [Folder Structure](#folder-structure) +- [Instructions](#instructions) + - [Markup](#markup) + - [Markdown](#markdown) + - [Element Attributes](#element-attributes) + - [Slide Attributes](#slide-attributes) +- [Configuration](#configuration) +- [Presentation Size](#presentation-size) +- [Dependencies](#dependencies) +- [Ready Event](#ready-event) +- [Auto-sliding](#auto-sliding) +- [Keyboard Bindings](#keyboard-bindings) +- [Vertical Slide Navigation](#vertical-slide-navigation) +- [Touch Navigation](#touch-navigation) +- [Lazy Loading](#lazy-loading) +- [API](#api) + - [Slide Changed Event](#slide-changed-event) + - [Presentation State](#presentation-state) + - [Slide States](#slide-states) + - [Slide Backgrounds](#slide-backgrounds) + - [Parallax Background](#parallax-background) + - [Slide Transitions](#slide-transitions) + - [Internal links](#internal-links) + - [Fragments](#fragments) + - [Fragment events](#fragment-events) + - [Code syntax highlighting](#code-syntax-highlighting) + - [Slide number](#slide-number) + - [Overview mode](#overview-mode) + - [Fullscreen mode](#fullscreen-mode) + - [Embedded media](#embedded-media) + - [Stretching elements](#stretching-elements) + - [Resize Event](#resize-event) + - [postMessage API](#postmessage-api) +- [PDF Export](#pdf-export) +- [Theming](#theming) +- [Speaker Notes](#speaker-notes) + - [Share and Print Speaker Notes](#share-and-print-speaker-notes) + - [Server Side Speaker Notes](#server-side-speaker-notes) +- [Plugins](#plugins) +- [Multiplexing](#multiplexing) + - [Master presentation](#master-presentation) + - [Client presentation](#client-presentation) + - [Socket.io server](#socketio-server) +- [MathJax](#mathjax) +- [License](#license) + +#### More reading + +- [Changelog](https://github.com/hakimel/reveal.js/releases): Up-to-date version history. +- [Examples](https://github.com/hakimel/reveal.js/wiki/Example-Presentations): Presentations created with reveal.js, add your own! +- [Browser Support](https://github.com/hakimel/reveal.js/wiki/Browser-Support): Explanation of browser support and fallbacks. +- [Plugins](https://github.com/hakimel/reveal.js/wiki/Plugins,-Tools-and-Hardware): A list of plugins that can be used to extend reveal.js. + + +## Online Editor + +Presentations are written using HTML or Markdown but there's also an online editor for those of you who prefer a graphical interface. Give it a try at [https://slides.com](https://slides.com?ref=github). + + +## Installation + +The **basic setup** is for authoring presentations only. The **full setup** gives you access to all reveal.js features and plugins such as speaker notes as well as the development tasks needed to make changes to the source. + +### Basic setup + +The core of reveal.js is very easy to install. You'll simply need to download a copy of this repository and open the index.html file directly in your browser. + +1. Download the latest version of reveal.js from +2. Unzip and replace the example contents in index.html with your own +3. Open index.html in a browser to view it + +### Full setup + +Some reveal.js features, like external Markdown and speaker notes, require that presentations run from a local web server. The following instructions will set up such a server as well as all of the development tasks needed to make edits to the reveal.js source code. + +1. Install [Node.js](http://nodejs.org/) (4.0.0 or later) + +1. Clone the reveal.js repository + ```sh + $ git clone https://github.com/hakimel/reveal.js.git + ``` + +1. Navigate to the reveal.js folder + ```sh + $ cd reveal.js + ``` + +1. Install dependencies + ```sh + $ npm install + ``` + +1. Serve the presentation and monitor source files for changes + ```sh + $ npm start + ``` + +1. Open to view your presentation + + You can change the port by using `npm start -- --port=8001`. + +### Folder Structure + +- **css/** Core styles without which the project does not function +- **js/** Like above but for JavaScript +- **plugin/** Components that have been developed as extensions to reveal.js +- **lib/** All other third party assets (JavaScript, CSS, fonts) + + +## Instructions + +### Markup + +Here's a barebones example of a fully working reveal.js presentation: +```html + + + + + + +
+
+
Slide 1
+
Slide 2
+
+
+ + + + +``` + +The presentation markup hierarchy needs to be `.reveal > .slides > section` where the `section` represents one slide and can be repeated indefinitely. If you place multiple `section` elements inside of another `section` they will be shown as vertical slides. The first of the vertical slides is the "root" of the others (at the top), and will be included in the horizontal sequence. For example: + +```html +
+
+
Single Horizontal Slide
+
+
Vertical Slide 1
+
Vertical Slide 2
+
+
+
+``` + +### Markdown + +It's possible to write your slides using Markdown. To enable Markdown, add the `data-markdown` attribute to your `
` elements and wrap the contents in a ` +
+``` + +#### External Markdown + +You can write your content as a separate file and have reveal.js load it at runtime. Note the separator arguments which determine how slides are delimited in the external file: the `data-separator` attribute defines a regular expression for horizontal slides (defaults to `^\r?\n---\r?\n$`, a newline-bounded horizontal rule) and `data-separator-vertical` defines vertical slides (disabled by default). The `data-separator-notes` attribute is a regular expression for specifying the beginning of the current slide's speaker notes (defaults to `notes?:`, so it will match both "note:" and "notes:"). The `data-charset` attribute is optional and specifies which charset to use when loading the external file. + +When used locally, this feature requires that reveal.js [runs from a local web server](#full-setup). The following example customises all available options: + +```html +
+ +
+``` + +#### Element Attributes + +Special syntax (through HTML comments) is available for adding attributes to Markdown elements. This is useful for fragments, amongst other things. + +```html +
+ +
+``` + +#### Slide Attributes + +Special syntax (through HTML comments) is available for adding attributes to the slide `
` elements generated by your Markdown. + +```html +
+ +
+``` + +#### Configuring *marked* + +We use [marked](https://github.com/chjj/marked) to parse Markdown. To customise marked's rendering, you can pass in options when [configuring Reveal](#configuration): + +```javascript +Reveal.initialize({ + // Options which are passed into marked + // See https://marked.js.org/#/USING_ADVANCED.md#options + markdown: { + smartypants: true + } +}); +``` + +### Configuration + +At the end of your page you need to initialize reveal by running the following code. Note that all configuration values are optional and will default to the values specified below. + +```javascript +Reveal.initialize({ + + // Display presentation control arrows + controls: true, + + // Help the user learn the controls by providing hints, for example by + // bouncing the down arrow when they first encounter a vertical slide + controlsTutorial: true, + + // Determines where controls appear, "edges" or "bottom-right" + controlsLayout: 'bottom-right', + + // Visibility rule for backwards navigation arrows; "faded", "hidden" + // or "visible" + controlsBackArrows: 'faded', + + // Display a presentation progress bar + progress: true, + + // Display the page number of the current slide + slideNumber: false, + + // Add the current slide number to the URL hash so that reloading the + // page/copying the URL will return you to the same slide + hash: false, + + // Push each slide change to the browser history. Implies `hash: true` + history: false, + + // Enable keyboard shortcuts for navigation + keyboard: true, + + // Enable the slide overview mode + overview: true, + + // Vertical centering of slides + center: true, + + // Enables touch navigation on devices with touch input + touch: true, + + // Loop the presentation + loop: false, + + // Change the presentation direction to be RTL + rtl: false, + + // See https://github.com/hakimel/reveal.js/#navigation-mode + navigationMode: 'default', + + // Randomizes the order of slides each time the presentation loads + shuffle: false, + + // Turns fragments on and off globally + fragments: true, + + // Flags whether to include the current fragment in the URL, + // so that reloading brings you to the same fragment position + fragmentInURL: false, + + // Flags if the presentation is running in an embedded mode, + // i.e. contained within a limited portion of the screen + embedded: false, + + // Flags if we should show a help overlay when the questionmark + // key is pressed + help: true, + + // Flags if speaker notes should be visible to all viewers + showNotes: false, + + // Global override for autoplaying embedded media (video/audio/iframe) + // - null: Media will only autoplay if data-autoplay is present + // - true: All media will autoplay, regardless of individual setting + // - false: No media will autoplay, regardless of individual setting + autoPlayMedia: null, + + // Global override for preloading lazy-loaded iframes + // - null: Iframes with data-src AND data-preload will be loaded when within + // the viewDistance, iframes with only data-src will be loaded when visible + // - true: All iframes with data-src will be loaded when within the viewDistance + // - false: All iframes with data-src will be loaded only when visible + preloadIframes: null, + + // Number of milliseconds between automatically proceeding to the + // next slide, disabled when set to 0, this value can be overwritten + // by using a data-autoslide attribute on your slides + autoSlide: 0, + + // Stop auto-sliding after user input + autoSlideStoppable: true, + + // Use this method for navigation when auto-sliding + autoSlideMethod: Reveal.navigateNext, + + // Specify the average time in seconds that you think you will spend + // presenting each slide. This is used to show a pacing timer in the + // speaker view + defaultTiming: 120, + + // Enable slide navigation via mouse wheel + mouseWheel: false, + + // Hide cursor if inactive + hideInactiveCursor: true, + + // Time before the cursor is hidden (in ms) + hideCursorTime: 5000, + + // Hides the address bar on mobile devices + hideAddressBar: true, + + // Opens links in an iframe preview overlay + // Add `data-preview-link` and `data-preview-link="false"` to customise each link + // individually + previewLinks: false, + + // Transition style + transition: 'slide', // none/fade/slide/convex/concave/zoom + + // Transition speed + transitionSpeed: 'default', // default/fast/slow + + // Transition style for full page slide backgrounds + backgroundTransition: 'fade', // none/fade/slide/convex/concave/zoom + + // Number of slides away from the current that are visible + viewDistance: 3, + + // Parallax background image + parallaxBackgroundImage: '', // e.g. "'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg'" + + // Parallax background size + parallaxBackgroundSize: '', // CSS syntax, e.g. "2100px 900px" + + // Number of pixels to move the parallax background per slide + // - Calculated automatically unless specified + // - Set to 0 to disable movement along an axis + parallaxBackgroundHorizontal: null, + parallaxBackgroundVertical: null, + + // The display mode that will be used to show slides + display: 'block' + +}); +``` + +The configuration can be updated after initialization using the `configure` method: + +```javascript +// Turn autoSlide off +Reveal.configure({ autoSlide: 0 }); + +// Start auto-sliding every 5s +Reveal.configure({ autoSlide: 5000 }); +``` + +### Presentation Size + +All presentations have a normal size, that is, the resolution at which they are authored. The framework will automatically scale presentations uniformly based on this size to ensure that everything fits on any given display or viewport. + +See below for a list of configuration options related to sizing, including default values: + +```javascript +Reveal.initialize({ + + // ... + + // The "normal" size of the presentation, aspect ratio will be preserved + // when the presentation is scaled to fit different resolutions. Can be + // specified using percentage units. + width: 960, + height: 700, + + // Factor of the display size that should remain empty around the content + margin: 0.1, + + // Bounds for smallest/largest possible scale to apply to content + minScale: 0.2, + maxScale: 1.5 + +}); +``` + +If you wish to disable this behavior and do your own scaling (e.g. using media queries), try these settings: + +```javascript +Reveal.initialize({ + + // ... + + width: "100%", + height: "100%", + margin: 0, + minScale: 1, + maxScale: 1 +}); +``` + +### Dependencies + +Reveal.js doesn't _rely_ on any third party scripts to work but a few optional libraries are included by default. These libraries are loaded as dependencies in the order they appear, for example: + +```javascript +Reveal.initialize({ + dependencies: [ + // Interpret Markdown in
elements + { src: 'plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, + { src: 'plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, + + // Syntax highlight for elements + { src: 'plugin/highlight/highlight.js', async: true }, + + // Zoom in and out with Alt+click + { src: 'plugin/zoom-js/zoom.js', async: true }, + + // Speaker notes + { src: 'plugin/notes/notes.js', async: true }, + + // MathJax + { src: 'plugin/math/math.js', async: true } + ] +}); +``` + +You can add your own extensions using the same syntax. The following properties are available for each dependency object: +- **src**: Path to the script to load +- **async**: [optional] Flags if the script should load after reveal.js has started, defaults to false +- **callback**: [optional] Function to execute when the script has loaded +- **condition**: [optional] Function which must return true for the script to be loaded + +### Ready Event + +A `ready` event is fired when reveal.js has loaded all non-async dependencies and is ready to start navigating. To check if reveal.js is already 'ready' you can call `Reveal.isReady()`. + +```javascript +Reveal.addEventListener( 'ready', function( event ) { + // event.currentSlide, event.indexh, event.indexv +} ); +``` + +Note that we also add a `.ready` class to the `.reveal` element so that you can hook into this with CSS. + +### Auto-sliding + +Presentations can be configured to progress through slides automatically, without any user input. To enable this you will need to tell the framework how many milliseconds it should wait between slides: + +```javascript +// Slide every five seconds +Reveal.configure({ + autoSlide: 5000 +}); +``` + +When this is turned on a control element will appear that enables users to pause and resume auto-sliding. Alternatively, sliding can be paused or resumed by pressing »A« on the keyboard. Sliding is paused automatically as soon as the user starts navigating. You can disable these controls by specifying `autoSlideStoppable: false` in your reveal.js config. + +You can also override the slide duration for individual slides and fragments by using the `data-autoslide` attribute: + +```html +
+

After 2 seconds the first fragment will be shown.

+

After 10 seconds the next fragment will be shown.

+

Now, the fragment is displayed for 2 seconds before the next slide is shown.

+
+``` + +To override the method used for navigation when auto-sliding, you can specify the `autoSlideMethod` setting. To only navigate along the top layer and ignore vertical slides, set this to `Reveal.navigateRight`. + +Whenever the auto-slide mode is resumed or paused the `autoslideresumed` and `autoslidepaused` events are fired. + +### Keyboard Bindings + +If you're unhappy with any of the default keyboard bindings you can override them using the `keyboard` config option: + +```javascript +Reveal.configure({ + keyboard: { + 13: 'next', // go to the next slide when the ENTER key is pressed + 27: function() {}, // do something custom when ESC is pressed + 32: null // don't do anything when SPACE is pressed (i.e. disable a reveal.js default binding) + } +}); +``` + +### Vertical Slide Navigation + +Slides can be nested within other slides to create vertical stacks (see [Markup](#markup)). When presenting, you use the left/right arrows to step through the main (horizontal) slides. When you arrive at a vertical stack you can optionally press the up/down arrows to view the vertical slides or skip past them by pressing the right arrow. Here's an example showing a bird's-eye view of what this looks like in action: + + + +#### Navigation Mode +You can finetune the reveal.js navigation behavior by using the `navigationMode` config option. Note that these options are only useful for presnetations that use a mix of horizontal and vertical slides. The following navigation modes are available: + +| Value | Description | +| :--------------------------- | :---------- | +| default | Left/right arrow keys step between horizontal slides. Up/down arrow keys step between vertical slides. Space key steps through all slides (both horizontal and vertical). | +| linear | Removes the up/down arrows. Left/right arrows step through all slides (both horizontal and vertical). | +| grid | When this is enabled, stepping left/right from a vertical stack to an adjacent vertical stack will land you at the same vertical index.

Consider a deck with six slides ordered in two vertical stacks:
`1.1`    `2.1`
`1.2`    `2.2`
`1.3`    `2.3`

If you're on slide 1.3 and navigate right, you will normally move from 1.3 -> 2.1. With navigationMode set to "grid" the same navigation takes you from 1.3 -> 2.3. | + +### Touch Navigation + +You can swipe to navigate through a presentation on any touch-enabled device. Horizontal swipes change between horizontal slides, vertical swipes change between vertical slides. If you wish to disable this you can set the `touch` config option to false when initializing reveal.js. + +If there's some part of your content that needs to remain accessible to touch events you'll need to highlight this by adding a `data-prevent-swipe` attribute to the element. One common example where this is useful is elements that need to be scrolled. + +### Lazy Loading + +When working on presentation with a lot of media or iframe content it's important to load lazily. Lazy loading means that reveal.js will only load content for the few slides nearest to the current slide. The number of slides that are preloaded is determined by the `viewDistance` configuration option. + +To enable lazy loading all you need to do is change your `src` attributes to `data-src` as shown below. This is supported for image, video, audio and iframe elements. + +```html +
+ + + +
+``` + +#### Lazy Loading Iframes + +Note that lazy loaded iframes ignore the `viewDistance` configuration and will only load when their containing slide becomes visible. Iframes are also unloaded as soon as the slide is hidden. + +When we lazy load a video or audio element, reveal.js won't start playing that content until the slide becomes visible. However there is no way to control this for an iframe since that could contain any kind of content. That means if we loaded an iframe before the slide is visible on screen it could begin playing media and sound in the background. + +You can override this behavior with the `data-preload` attribute. The iframe below will be loaded +according to the `viewDistance`. + +```html +
+ +
+``` + +You can also change the default globally with the `preloadIframes` configuration option. If set to +`true` ALL iframes with a `data-src` attribute will be preloaded when within the `viewDistance` +regardless of individual `data-preload` attributes. If set to `false`, all iframes will only be +loaded when they become visible. + +### API + +The `Reveal` object exposes a JavaScript API for controlling navigation and reading state: + +```javascript +// Navigation +Reveal.slide( indexh, indexv, indexf ); +Reveal.left(); +Reveal.right(); +Reveal.up(); +Reveal.down(); +Reveal.prev(); +Reveal.next(); +Reveal.prevFragment(); +Reveal.nextFragment(); + +// Randomize the order of slides +Reveal.shuffle(); + +// Toggle presentation states, optionally pass true/false to force on/off +Reveal.toggleOverview(); +Reveal.togglePause(); +Reveal.toggleAutoSlide(); + +// Shows a help overlay with keyboard shortcuts, optionally pass true/false +// to force on/off +Reveal.toggleHelp(); + +// Change a config value at runtime +Reveal.configure({ controls: true }); + +// Returns the present configuration options +Reveal.getConfig(); + +// Fetch the current scale of the presentation +Reveal.getScale(); + +// Retrieves the previous and current slide elements +Reveal.getPreviousSlide(); +Reveal.getCurrentSlide(); + +Reveal.getIndices(); // { h: 0, v: 0, f: 0 } +Reveal.getSlidePastCount(); +Reveal.getProgress(); // (0 == first slide, 1 == last slide) +Reveal.getSlides(); // Array of all slides +Reveal.getTotalSlides(); // Total number of slides + +// Returns the speaker notes for the current slide +Reveal.getSlideNotes(); + +// State checks +Reveal.isFirstSlide(); +Reveal.isLastSlide(); +Reveal.isOverview(); +Reveal.isPaused(); +Reveal.isAutoSliding(); + +// Returns the top-level DOM element +getRevealElement(); //
...
+``` + +### Custom Key Bindings + +Custom key bindings can be added and removed using the following Javascript API. Custom key bindings will override the default keyboard bindings, but will in turn be overridden by the user defined bindings in the ``keyboard`` config option. + +```javascript +Reveal.addKeyBinding( binding, callback ); +Reveal.removeKeyBinding( keyCode ); +``` + +For example + +```javascript +// The binding parameter provides the following properties +// keyCode: the keycode for binding to the callback +// key: the key label to show in the help overlay +// description: the description of the action to show in the help overlay +Reveal.addKeyBinding( { keyCode: 84, key: 'T', description: 'Start timer' }, function() { + // start timer +} ) + +// The binding parameter can also be a direct keycode without providing the help description +Reveal.addKeyBinding( 82, function() { + // reset timer +} ) +``` + +This allows plugins to add key bindings directly to Reveal so they can + +* make use of Reveal's pre-processing logic for key handling (for example, ignoring key presses when paused); and +* be included in the help overlay (optional) + +### Slide Changed Event + +A `slidechanged` event is fired each time the slide is changed (regardless of state). The event object holds the index values of the current slide as well as a reference to the previous and current slide HTML nodes. + +Some libraries, like MathJax (see [#226](https://github.com/hakimel/reveal.js/issues/226#issuecomment-10261609)), get confused by the transforms and display states of slides. Often times, this can be fixed by calling their update or render function from this callback. + +```javascript +Reveal.addEventListener( 'slidechanged', function( event ) { + // event.previousSlide, event.currentSlide, event.indexh, event.indexv +} ); +``` + +### Presentation State + +The presentation's current state can be fetched by using the `getState` method. A state object contains all of the information required to put the presentation back as it was when `getState` was first called. Sort of like a snapshot. It's a simple object that can easily be stringified and persisted or sent over the wire. + +```javascript +Reveal.slide( 1 ); +// we're on slide 1 + +var state = Reveal.getState(); + +Reveal.slide( 3 ); +// we're on slide 3 + +Reveal.setState( state ); +// we're back on slide 1 +``` + +### Slide States + +If you set `data-state="somestate"` on a slide `
`, "somestate" will be applied as a class on the document element when that slide is opened. This allows you to apply broad style changes to the page based on the active slide. + +Furthermore you can also listen to these changes in state via JavaScript: + +```javascript +Reveal.addEventListener( 'somestate', function() { + // TODO: Sprinkle magic +}, false ); +``` + +### Slide Backgrounds + +Slides are contained within a limited portion of the screen by default to allow them to fit any display and scale uniformly. You can apply full page backgrounds outside of the slide area by adding a `data-background` attribute to your `
` elements. Four different types of backgrounds are supported: color, image, video and iframe. + +#### Color Backgrounds + +All CSS color formats are supported, including hex values, keywords, `rgba()` or `hsl()`. + +```html +
+

Color

+
+``` + +#### Image Backgrounds + +By default, background images are resized to cover the full page. Available options: + +| Attribute | Default | Description | +| :------------------------------- | :--------- | :---------- | +| data-background-image | | URL of the image to show. GIFs restart when the slide opens. | +| data-background-size | cover | See [background-size](https://developer.mozilla.org/docs/Web/CSS/background-size) on MDN. | +| data-background-position | center | See [background-position](https://developer.mozilla.org/docs/Web/CSS/background-position) on MDN. | +| data-background-repeat | no-repeat | See [background-repeat](https://developer.mozilla.org/docs/Web/CSS/background-repeat) on MDN. | +| data-background-opacity | 1 | Opacity of the background image on a 0-1 scale. 0 is transparent and 1 is fully opaque. | + +```html +
+

Image

+
+
+

This background image will be sized to 100px and repeated

+
+``` + +#### Video Backgrounds + +Automatically plays a full size video behind the slide. + +| Attribute | Default | Description | +| :--------------------------- | :------ | :---------- | +| data-background-video | | A single video source, or a comma separated list of video sources. | +| data-background-video-loop | false | Flags if the video should play repeatedly. | +| data-background-video-muted | false | Flags if the audio should be muted. | +| data-background-size | cover | Use `cover` for full screen and some cropping or `contain` for letterboxing. | +| data-background-opacity | 1 | Opacity of the background video on a 0-1 scale. 0 is transparent and 1 is fully opaque. | + +```html +
+

Video

+
+``` + +#### Iframe Backgrounds + +Embeds a web page as a slide background that covers 100% of the reveal.js width and height. The iframe is in the background layer, behind your slides, and as such it's not possible to interact with it by default. To make your background interactive, you can add the `data-background-interactive` attribute. + +```html +
+

Iframe

+
+``` + +#### Background Transitions + +Backgrounds transition using a fade animation by default. This can be changed to a linear sliding transition by passing `backgroundTransition: 'slide'` to the `Reveal.initialize()` call. Alternatively you can set `data-background-transition` on any section with a background to override that specific transition. + + +### Parallax Background + +If you want to use a parallax scrolling background, set the first two properties below when initializing reveal.js (the other two are optional). + +```javascript +Reveal.initialize({ + + // Parallax background image + parallaxBackgroundImage: '', // e.g. "https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg" + + // Parallax background size + parallaxBackgroundSize: '', // CSS syntax, e.g. "2100px 900px" - currently only pixels are supported (don't use % or auto) + + // Number of pixels to move the parallax background per slide + // - Calculated automatically unless specified + // - Set to 0 to disable movement along an axis + parallaxBackgroundHorizontal: 200, + parallaxBackgroundVertical: 50 + +}); +``` + +Make sure that the background size is much bigger than screen size to allow for some scrolling. [View example](http://revealjs.com/?parallaxBackgroundImage=https%3A%2F%2Fs3.amazonaws.com%2Fhakim-static%2Freveal-js%2Freveal-parallax-1.jpg¶llaxBackgroundSize=2100px%20900px). + +### Slide Transitions + +The global presentation transition is set using the `transition` config value. You can override the global transition for a specific slide by using the `data-transition` attribute: + +```html +
+

This slide will override the presentation transition and zoom!

+
+ +
+

Choose from three transition speeds: default, fast or slow!

+
+``` + +You can also use different in and out transitions for the same slide: + +```html +
+ The train goes on … +
+
+ and on … +
+
+ and stops. +
+
+ (Passengers entering and leaving) +
+
+ And it starts again. +
+``` +You can choose from `none`, `fade`, `slide`, `convex`, `concave` and `zoom`. +### Internal links + +It's easy to link between slides. The first example below targets the index of another slide whereas the second targets a slide with an ID attribute (`
`): + +```html +Link +Link +``` + +You can also add relative navigation links, similar to the built in reveal.js controls, by appending one of the following classes on any element. Note that each element is automatically given an `enabled` class when it's a valid navigation route based on the current slide. + +```html + + + + + + +``` + +### Fragments + +Fragments are used to highlight individual elements on a slide. Every element with the class `fragment` will be stepped through before moving on to the next slide. Here's an example: http://revealjs.com/#/fragments + +The default fragment style is to start out invisible and fade in. This style can be changed by appending a different class to the fragment: + +```html +
+

grow

+

shrink

+

fade-out

+

fade-up (also down, left and right!)

+

fades in, then out when we move to the next step

+

fades in, then obfuscate when we move to the next step

+

blue only once

+

highlight-red

+

highlight-green

+

highlight-blue

+
+``` + +Multiple fragments can be applied to the same element sequentially by wrapping it, this will fade in the text on the first step and fade it back out on the second. + +```html +
+ + I'll fade in, then out + +
+``` + +The display order of fragments can be controlled using the `data-fragment-index` attribute. + +```html +
+

Appears last

+

Appears first

+

Appears second

+
+``` + +### Fragment events + +When a slide fragment is either shown or hidden reveal.js will dispatch an event. + +Some libraries, like MathJax (see #505), get confused by the initially hidden fragment elements. Often times this can be fixed by calling their update or render function from this callback. + +```javascript +Reveal.addEventListener( 'fragmentshown', function( event ) { + // event.fragment = the fragment DOM element +} ); +Reveal.addEventListener( 'fragmenthidden', function( event ) { + // event.fragment = the fragment DOM element +} ); +``` + +### Code Syntax Highlighting + +By default, Reveal is configured with [highlight.js](https://highlightjs.org/) for code syntax highlighting. To enable syntax highlighting, you'll have to load the highlight plugin ([plugin/highlight/highlight.js](plugin/highlight/highlight.js)) and a highlight.js CSS theme (Reveal comes packaged with the Monokai themes: [lib/css/monokai.css](lib/css/monokai.css)). + +```javascript +Reveal.initialize({ + // More info https://github.com/hakimel/reveal.js#dependencies + dependencies: [ + { src: 'plugin/highlight/highlight.js', async: true }, + ] +}); +``` + +Below is an example with clojure code that will be syntax highlighted. When the `data-trim` attribute is present, surrounding whitespace is automatically removed. HTML will be escaped by default. To avoid this, for example if you are using `` to call out a line of code, add the `data-noescape` attribute to the `` element. + +```html +
+

+(def lazy-fib
+  (concat
+   [0 1]
+   ((fn rfib [a b]
+        (lazy-cons (+ a b) (rfib b (+ a b)))) 0 1)))
+	
+
+``` + +#### Line Numbers & Highlights + +To enable line numbers, add `data-line-numbers` to your `` tags. If you want to highlight specific lines you can provide a comma separated list of line numbers using the same attribute. For example, in the following example lines 4 and 8-11 are highlighted: + +```html +

+import React, { useState } from 'react';
+ 
+function Example() {
+  const [count, setCount] = useState(0);
+ 
+  return (
+    
+

You clicked {count} times

+ +
+ ); +} +
+``` + +line-numbers + + + +### Slide number + +If you would like to display the page number of the current slide you can do so using the `slideNumber` and `showSlideNumber` configuration values. + +```javascript +// Shows the slide number using default formatting +Reveal.configure({ slideNumber: true }); + +// Slide number formatting can be configured using these variables: +// "h.v": horizontal . vertical slide number (default) +// "h/v": horizontal / vertical slide number +// "c": flattened slide number +// "c/t": flattened slide number / total slides +Reveal.configure({ slideNumber: 'c/t' }); + +// You can provide a function to fully customize the number: +Reveal.configure({ slideNumber: function() { + // Ignore numbering of vertical slides + return [ Reveal.getIndices().h ]; +}}); + +// Control which views the slide number displays on using the "showSlideNumber" value: +// "all": show on all views (default) +// "speaker": only show slide numbers on speaker notes view +// "print": only show slide numbers when printing to PDF +Reveal.configure({ showSlideNumber: 'speaker' }); +``` + +### Overview mode + +Press »ESC« or »O« keys to toggle the overview mode on and off. While you're in this mode, you can still navigate between slides, +as if you were at 1,000 feet above your presentation. The overview mode comes with a few API hooks: + +```javascript +Reveal.addEventListener( 'overviewshown', function( event ) { /* ... */ } ); +Reveal.addEventListener( 'overviewhidden', function( event ) { /* ... */ } ); + +// Toggle the overview mode programmatically +Reveal.toggleOverview(); +``` + +### Fullscreen mode + +Just press »F« on your keyboard to show your presentation in fullscreen mode. Press the »ESC« key to exit fullscreen mode. + +### Embedded media + +Add `data-autoplay` to your media element if you want it to automatically start playing when the slide is shown: + +```html + +``` + +If you want to enable or disable autoplay globally, for all embedded media, you can use the `autoPlayMedia` configuration option. If you set this to `true` ALL media will autoplay regardless of individual `data-autoplay` attributes. If you initialize with `autoPlayMedia: false` NO media will autoplay. + +Note that embedded HTML5 `