I like the dat gui to tweak the value for the custom shader, that's great for a demo. This was working properly for me on Firefox and Chrome, but not Safari. But I couldn't test the networked part of it though.
Only GreenThresholdIn is really used, GreenThresholdOut is not used AFAICT.

I don't know much how aframe handle additional material, but I think you can create only once the custom shader, and reuse the same for every networked-video-source component instance, but using specific uniforms params?

I think for this PR to be merged and the new example to be maintained in the long term is just to keep your example advanced-video-green-screen.html and try to have only one networked-video-source component with necessary changes instead of duplicating it.

I see you committed the file basic-video-mediapiped.html. I think this was your previous example that was not using the new api. You probably want to remove the file from the PR.

If possible I think you should put an updated mediapipe example on glitch that is using the new networked-aframe api+this PR with changes in networked-video-source supporting the custom shader. The goal in the end is to have your mediapipe example running with minimal lines of code on a released version of networked-aframe, and not a custom build.
And we could put a link to your glitch in the examples.

Look at registerShader https://aframe.io/docs/1.2.0/components/material.html#registershader
and look a the Programs count of the stats component
https://aframe.io/docs/1.2.0/components/stats.html
I think with your current code, this Programs counter will keep increasing for each new participant (I can't confirm it right now), but if you properly reuse the shader you will only have one I think.

As regards the procedure of replacing pixels, it is as follows. Get my video stream and replace "green" pixels with flat RGBA = (0,1,0,1) pixels (flat green). The amount of "green" of the background is configured with GreenThresholdOut. This is due to the fact that there are always shadows on your background as no green is absolute flat green. For this procedure I have used an html canvas and traverse it pixel by pixel.

When you receive all the "Greenish" video streams from other clients, you should be able to remove the green pixels. This can be done by simply saying if RGBA == (0,1,0,1) then put transparent (0,0,0,0). But for experimentation reasons I have added also a GreenThresholdIn which is a threshold on the amount of green for the received streams (for all the clients that you receive a video stream). It is mainly for debugging. For this method I have used the shader language.

I am planning to do something for the white channel also as many have white background. It would be lossy however, as eye bulbs and reflections are white. A custom color picker would be also useful. Thank god there is dat.gui for that.

As regards Mediapipe for automatic selfie segmentation with AI, I will re-visit it when there will be an improvement by Google. The last version 0.6 was very problematic. Only 50% of the times worked and the error codes were not illuminating the problem.

Thank you for the explanation, it's clearer now what your loop does, replacing the greenish pixels with pure green pixels.
I completely skipped your loop when I read your code, I see now that GreenThresholdOut is used there.
You should put this explanation in comments just above your loop().

And as I understand it with your mediapipe version you did a similar thing, filling the detected background by pure green (canvasCtx.fillStyle = "#00FF00FF";). Cool.

Maybe I'm asking something obvious, is there a reason to compare the difference between red and green, instead of just checking green above some threshold?

For your loop, wouldn't it be better to use setInterval instead of setTimeout? I don't know if it makes a difference for performance.
https://developer.mozilla.org/en-US/docs/Web/API/setInterval

I reply to myself here, setInterval would potentially execute again the function even if the previous call wasn't finished I think.

For anyone interested in playing with the new mediapipe selfie segmentation:
https://twitter.com/TensorFlow/status/1488214410346872840?t=bEJ0d-1WnvU4lsLxhR-z5w
If you don't have a green background :)

As regards "Programs" in stats, you have right, it increases incrementally when each player enters. However, I do not know how to use shaders for video streams. There is no example anywhere.

I'll take a look for the shader code this weekend.

I have found how to make a green-screen shader for videos in AFrame without the need of a canvas and without the need to go back to Three.js.

Look at two examples at my aframe 1.3.0 fork
Shader based: https://github.com/jimver04/aframe/tree/master/examples/test/shader-texture-video
Canvas based: https://github.com/jimver04/aframe/tree/master/examples/test/canvas-texture-video

I don't know which is faster. My GPU is 3080 and doesn't have any problems showing both at 60fps. Is there any method to check it? Stats are showing about the same.

I will see to fix NAF now.
Best,
D.

The video dom example is not working properly in NAF as it shows always the same video for all players. I guess I have to use the embedded map of the shader

/* global AFRAME */
/**
 * The Green screen shader
 */
AFRAME.registerShader('green-screen-shader', {

    schema: {
        greenThreshold:{type: 'float', is: 'uniform', default:0.08}
    },

    vertexShader: `varying vec2 vUv;

                          void main() {
                              vec4 worldPosition = modelViewMatrix * vec4( position, 1.0 );
                              vec3 vWorldPosition = worldPosition.xyz;
                              vUv = uv;
                              gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
                          }`,

    fragmentShader: `varying vec2 vUv;
                     uniform sampler2D map;
                     uniform float greenThreshold;

                     void main() {
                                  vec2 uv = vUv;
                                  vec4 tex1 = texture2D(map, uv * 1.0);
                                   if (tex1.g - tex1.r > greenThreshold)
                                      gl_FragColor = vec4(0,0,0,0);
                                   else
                                      gl_FragColor = vec4(tex1.r,tex1.g,tex1.b,1.0);
                              }`

});

However the map corresponds to the whole level map but not the player map texture:

By Searching aframe.js, the only map declared for flat shader (the shader of videos) is src. I have replaced map with src but I get the same map, i.e. the skysphere map which is the first map loaded.

Finally I have done a simple solution. It is very robust in comparison to my other methods.

• No secondary stream
• No canvas for personal stream processing before send
• Just apply a green effect shader for all incoming video streams.
  Although it does not give the ability for each user to configure its booth setup, it is much better as regards speed and accessibility.
  More advanced-examples will follow in the future.

1. use canvas per user to configure personal stream locally before sending
2. Check new versions of MediaPipe (cross-fingers with WebAssembly)

I think we can not avoid Three.js as Aframe does not apply a shader with registerShader when no dom element for texture is defined. It applies only globaly in the scene to the first texture it finds, e.g. to the skysphere in my case.

I didn't look at the registerShader with custom shader with the map I talked about yet, but it's on my todo list.
I was busy with the other issues and doing the releases in the last weeks. ;-)

What are you up to in the latest commits?
FYI you're working on the master branch in your fork, so all the commits ends up in this PR.
For futures PR you may want to create a branch instead.

Hi Vincent,

my crazy idea is to do remote virtual productions for shooting movies whilst the actors are at their homes. It is very risky but I have managed to attract the interest of the local community of actors and musicians through a demo, as well as persuading the project managers. Actors will participate through their smartphones from home with a green screen background* and with a low rendering needs page. They only have to press 3 keys that will transfer them in their acting position, so as to avoid 3D navigational stress. I will be in the same room as a director with my RTX3080 card rendering the full scene:

Director High-end PC

then I stream back the overall scene with the "Screen sharing" feature that you have recently added in order for the actors to have a feedback:

Actor Smartphone

Most of the developments are on the director page. I have followed your advice to avoid using canvas on the sender side and use only shaders on the receiver side. I have changed the RGB algorithm to YUV space with double thresholds which is more commonly used in chroma key removal (see https://www.shadertoy.com/view/MlVXWD). I have also added these thresholds on the Director page together with the choice to control the chroma key (it can be also blue or any color). I have added dat.gui interfaces per actor so that each actor can have separate chroma key and different thresholds. I have also added the parameters to change the width and the height of each actor plane so that I can normalize the size of the actor in the scene.

I have followed your advice, to avoid doing processing on a canvas on the sender side and the code became more simple. I am working only with shaders and it works well. I am using a modified version of your adaptor, and I have delete mine.

Of course the nice Ship 3D model is excluded from all the commits as I have bought it from a well-known site.
The water is done with A-water component (https://github.com/Dante83/a-water) which is open source.

To summarize, ignore this pull request. I will prepare another branch depending on what you want from all these above.
Any comments ? ideas ? My future plan is to add some movement for the actors' planes to give them the ability to walk in the scene. It is not possible for them to touch their smartphone so it has to be done on the Director side.

Best,
Dimitrios

*Web AI for autoremoval (Mediapipe) is unstable and computational expensive for smartphones, plus it has artifacts of the background. We will share to actors some low cost "Creator studios" consisting of a green cloth sheet and a smartphone stand ~ 70euros. Of course some DIY solutions for green screen can work too as I found that green pieces of paper can do the job if the lighting is medium to good.

Nice project! Thanks for keeping us posted about the things you're trying, this is interesting.
I'll try to tinker a bit the custom shader on my side and let you know.