Browsers need to natively support image/audio/video codecs in order for content to be rendered. Polyfills are possible to work around lack of support but they require messing with the DOM e.g. to replace <img> tags.

A system enabling to write and distribute these algorithms in JavaScript and get them integrated in the overall media flow of user agents would provide much greater freedom to specialize and innovate in this space.

Performance would obviously be an issue although maybe not so much with the advent of WebAssembly, SIMD and Service Workers for background processing.

See the Custom image/audio/video codec APIs proposal in WICG.