Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices #5137

mengyanshou · 2024-07-26T09:36:08Z

Environment

OS: macOS
Scrcpy version: 2.5
Installation method: brew
Device model: Xiaomi14 MEIZU 21
Android version: 14(both)

Describe the bug

I encountered a problem similar to one in another issue, related issue #4598

Some devices have virtual displays, such as Samsung's Dex, Smartisan's TNT. When using scrcpy to mirror these devices, the screen touch events become ineffective.

I have two devices, Xiaomi 14 and Meizu 21, both are Android 14, and the same issue exists.

If I cast to a display with an ID of 2 or 3,

when the target display is a VirtualDisplay (I haven't tested with a physical display yet), the screen control events do not work.

But if I direct all the events' display ID to the display ID created by the scrcpy-server's VirtualDisplay,

everything works fine.

The scrcpy-server created the virtual display here:

try {
    Rect videoRect = screenInfo.getVideoSize().toRect();
    virtualDisplay = ServiceManager.getDisplayManager()
            .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
    Ln.d("Display: using DisplayManager API");
} catch (Exception displayManagerException) {
    try {
        display = createDisplay();
        setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
        Ln.d("Display: using SurfaceControl API");
    } catch (Exception surfaceControlException) {
        Ln.e("Could not create display using DisplayManager", displayManagerException);
        Ln.e("Could not create display using SurfaceControl", surfaceControlException);
        throw new AssertionError("Could not create display");
    }
}

Here are my operations:

1. Create a virtual display using Android API

Assuming the obtained display ID is 2.

2. Run Task on the specified display

am display move-stack taskId $displayId

or

adb shell am start-activity -W -S --activityType 0 --activity-brought-to-front --display 2 -n com.nightmare.sula/.MainActivity
This will run the Android task on this virtual display.

3. Use scrcpy --display-id 2 to mirror

The screen can be rendered normally, but the events do not take effect.

I made the following modifications to the scrcpy server code:

Device.java

public final class Device {
    private final int displayId;
    private int inputDisplayId;
    // code

    public void setInputDisplayId(int id) {
        inputDisplayId = id;
    }
    // code

    public Device(Options options) throws ConfigurationException {
        displayId = options.getDisplayId();
        inputDisplayId = displayId;
    }

    public boolean injectEvent(InputEvent event, int injectMode) {
        return injectEvent(event, inputDisplayId, injectMode);
    }

    public static boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int displayId, int injectMode) {
        long now = SystemClock.uptimeMillis();
        KeyEvent event = new KeyEvent(now, now, action, keyCode, repeat, metaState, KeyCharacterMap.VIRTUAL_KEYBOARD, 0, 0,
                InputDevice.SOURCE_KEYBOARD);
        return injectEvent(event, displayId, injectMode);
    }

    public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
        // notice the inputDisplayId
        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
    }

    public boolean pressReleaseKeycode(int keyCode, int injectMode) {
        // notice the inputDisplayId
        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
    }
}

ScreenCapture.java

package com.genymobile.scrcpy;

import com.genymobile.scrcpy.wrappers.ServiceManager;
import com.genymobile.scrcpy.wrappers.SurfaceControl;

import android.graphics.Rect;
import android.hardware.display.VirtualDisplay;
import android.os.Build;
import android.os.IBinder;
import android.view.Surface;

public class ScreenCapture extends SurfaceCapture implements Device.RotationListener, Device.FoldListener {

    @Override
    public void start(Surface surface) {
        ScreenInfo screenInfo = device.getScreenInfo();
        Rect contentRect = screenInfo.getContentRect();

        // does not include the locked video orientation
        Rect unlockedVideoRect = screenInfo.getUnlockedVideoSize().toRect();
        int videoRotation = screenInfo.getVideoRotation();
        int layerStack = device.getLayerStack();

        if (display != null) {
            SurfaceControl.destroyDisplay(display);
            display = null;
        }
        if (virtualDisplay != null) {
            virtualDisplay.release();
            virtualDisplay = null;
        }

        try {
            Rect videoRect = screenInfo.getVideoSize().toRect();
            virtualDisplay = ServiceManager.getDisplayManager()
                    .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
            Ln.e("Display: using DisplayManager API virtualDisplay id -> " + virtualDisplay.getDisplay().getDisplayId());
            /// add line to set inputDisplayId
            device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
        } catch (Exception displayManagerException) {
            try {
                display = createDisplay();
                setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
                Ln.e("Display: using SurfaceControl API");
            } catch (Exception surfaceControlException) {
                Ln.e("Could not create display using DisplayManager", displayManagerException);
                Ln.e("Could not create display using SurfaceControl", surfaceControlException);
                throw new AssertionError("Could not create display");
            }
        }
    }
}

Now when I mirror the virtual display 2 again, the events work.

And again, it's very strange that

Using this scrcpy-server

SCRCPY_SERVER_PATH=server/build/scrcpy-server scrcpy --no-audio --display-id=0
It works normally when the display ID is 0, this has only been tested on Android 14, I will test more Android versions with a virtual machine later.

This is my description of this issue, I hope to get some response, if possible, I can submit a PR, thank you.

Other

Now, I can briefly introduce why I use a virtual display.

Virtual displays are now extensively used by me in Android app flow and a Launcher.

What is app flow?

An app is running on device A, now with the help of scrcpy-server, I can very smoothly run this app alone on device B.

In my own app called Uncon,

I used scrcpy-server, and then according to its protocol, used native Android code for rendering. After comparison, the efficiency is higher than SDL2 rendering.

Android Launcher?
Some Android devices (USB2.0) cannot output the screen through DP.

So I made such a try.

Create a virtual display.
Move the Launcher to the virtual display.
Through scrcpy --display-id $displayId.
This way, people can see this desktop launcher on the PC.

It looks like this.

2024-07-26.17.37.19_2.mp4

Ignore those errors.

Currently, this Launcher is running on this device.

And at this time, it does not interfere with the use of other functions of the device, which is the best.

Finally, thank you for your contribution to this project.

rom1v · 2024-07-29T22:02:17Z

Thank you for the post 👍

In the future, it might help to run scrcpy with a specific app (running on a virtual display). (I guess)

As you noticed, creating a new virtual display results in a new display id:

scrcpy/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java

Lines 52 to 53 in 071d459

    
           virtualDisplay = ServiceManager.getDisplayManager() 
        
                   .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);

So there is the displayId of the display to mirror, and the displayId of the virtual display.

In the scrcpy main use case (simple screen mirroring), input events must use the display id of the display to mirror (in practice it's 0). If I use the one returned by virtualDisplay.getDisplay().getDisplayId() to inject events, it does not work, they are not injected.

So IIUC, we should use the original displayId only if it's 0, and the one from the virtual display otherwise?

mengyanshou · 2024-07-30T17:46:08Z

I think it is feasible. I am about to submit a PR to use the virtual display's id for control events when the display id is not 0.

rom1v · 2024-07-30T18:00:34Z

I am about to submit a PR to use the virtual display's id for control events when the display id is not 0.

But is it the correct condition?

mengyanshou · 2024-07-31T12:18:55Z

In fact, in the actual test, when the displayId is 0, the event specified as the id of the virtual display created, it also works, and this has only been tested on Android 14
Or is it controlled by a switch? For example, --dispatch-to-virtual

rom1v · 2024-07-31T12:43:29Z

In fact, in the actual test, when the displayId is 0, the event specified as the id of the virtual display created, it also works, and this has only been tested on Android 14

It seems it does not work on a Pixel 8 with Android 14.

Here is the diff I tested:

diff

diff --git a/server/src/main/java/com/genymobile/scrcpy/device/Device.java b/server/src/main/java/com/genymobile/scrcpy/device/Device.java
index 5a1083fde..eee35254e 100644
--- a/server/src/main/java/com/genymobile/scrcpy/device/Device.java
+++ b/server/src/main/java/com/genymobile/scrcpy/device/Device.java
@@ -65,6 +65,8 @@ public final class Device {
      */
     private final int displayId;
 
+    private int inputDisplayId;
+
     /**
      * The surface flinger layer stack associated with this logical display
      */
@@ -246,7 +248,8 @@ public final class Device {
     }
 
     public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
-        return injectKeyEvent(action, keyCode, repeat, metaState, displayId, injectMode);
+        Ln.d("injectKeyEvent inputDisplayId=" + inputDisplayId);
+        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
     }
 
     public static boolean pressReleaseKeycode(int keyCode, int displayId, int injectMode) {
@@ -255,7 +258,7 @@ public final class Device {
     }
 
     public boolean pressReleaseKeycode(int keyCode, int injectMode) {
-        return pressReleaseKeycode(keyCode, displayId, injectMode);
+        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
     }
 
     public static boolean isScreenOn() {
@@ -401,4 +404,8 @@ public final class Device {
         DisplayInfo displayInfo = ServiceManager.getDisplayManager().getDisplayInfo(displayId);
         return displayInfo.getRotation();
     }
+
+    public void setInputDisplayId(int id) {
+        inputDisplayId = id;
+    }
 }
diff --git a/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
index fbeca2af0..d56d168aa 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
@@ -52,6 +52,9 @@ public class ScreenCapture extends SurfaceCapture implements Device.RotationList
             virtualDisplay = ServiceManager.getDisplayManager()
                     .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
             Ln.d("Display: using DisplayManager API");
+            int displayId = virtualDisplay.getDisplay().getDisplayId();
+            Ln.d("====" + displayId);
+            device.setInputDisplayId(displayId);
         } catch (Exception displayManagerException) {
             try {
                 display = createDisplay();

mengyanshou · 2024-07-31T14:41:35Z

I just noticed this code, so indeed, when displayId is 0, the events injected by injectEvent should also correspond to displayId 0

public static boolean injectEvent(InputEvent inputEvent, int displayId, int injectMode) {
    if (!supportsInputEvents(displayId)) {
        throw new AssertionError("Could not inject input event if !supportsInputEvents()");
    }
    if (displayId != 0 && !InputManager.setDisplayId(inputEvent, displayId)) {
        return false;
    }

    return ServiceManager.getInputManager().injectInputEvent(inputEvent, injectMode);
}

mengyanshou · 2024-07-31T15:29:54Z

I'll try not to affect the original functionality of scrcpy. First, I've added an option dispatch_to_vd to the scrcpy-server part. In the ScreenCapture.start function call:

if (device.isDispatchToVD()) {
    device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
}

Is this feasible?

This is the full diff

diff --git a/server/src/main/java/com/genymobile/scrcpy/Device.java b/server/src/main/java/com/genymobile/scrcpy/Device.java
index 8d0ee231..3969657f 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Device.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Device.java
@@ -60,6 +60,7 @@ public final class Device {
      * Logical display identifier
      */
     private final int displayId;
+    private int inputDisplayId;
 
     /**
      * The surface flinger layer stack associated with this logical display
@@ -68,8 +69,15 @@ public final class Device {
 
     private final boolean supportsInputEvents;
 
+    /**
+     * Whether to dispatch input events to the virtual display
+     */
+    private final boolean dispatchToVD;
+
     public Device(Options options) throws ConfigurationException {
         displayId = options.getDisplayId();
+        inputDisplayId = displayId;
+        dispatchToVD = options.getDispatchToVD();
         DisplayInfo displayInfo = ServiceManager.getDisplayManager().getDisplayInfo(displayId);
         if (displayInfo == null) {
             Ln.e("Display " + displayId + " not found\n" + LogUtils.buildDisplayListMessage());
@@ -168,6 +176,14 @@ public final class Device {
         return displayId;
     }
 
+    public int getInputDisplayId() {
+        return inputDisplayId;
+    }
+
+    public void setInputDisplayId(int id) {
+        inputDisplayId = id;
+    }
+
     public synchronized void setMaxSize(int newMaxSize) {
         maxSize = newMaxSize;
         screenInfo = ScreenInfo.computeScreenInfo(screenInfo.getReverseVideoRotation(), deviceSize, crop, newMaxSize, lockVideoOrientation);
@@ -218,11 +234,14 @@ public final class Device {
         return supportsInputEvents;
     }
 
+    public  boolean isDispatchToVD() {
+        return dispatchToVD;
+    }
+
     public static boolean injectEvent(InputEvent inputEvent, int displayId, int injectMode) {
         if (!supportsInputEvents(displayId)) {
             throw new AssertionError("Could not inject input event if !supportsInputEvents()");
         }
-
         if (displayId != 0 && !InputManager.setDisplayId(inputEvent, displayId)) {
             return false;
         }
@@ -231,7 +250,7 @@ public final class Device {
     }
 
     public boolean injectEvent(InputEvent event, int injectMode) {
-        return injectEvent(event, displayId, injectMode);
+        return injectEvent(event, inputDisplayId, injectMode);
     }
 
     public static boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int displayId, int injectMode) {
@@ -242,7 +261,7 @@ public final class Device {
     }
 
     public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
-        return injectKeyEvent(action, keyCode, repeat, metaState, displayId, injectMode);
+        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
     }
 
     public static boolean pressReleaseKeycode(int keyCode, int displayId, int injectMode) {
@@ -251,7 +270,7 @@ public final class Device {
     }
 
     public boolean pressReleaseKeycode(int keyCode, int injectMode) {
-        return pressReleaseKeycode(keyCode, displayId, injectMode);
+        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
     }
 
     public static boolean isScreenOn() {
diff --git a/server/src/main/java/com/genymobile/scrcpy/Options.java b/server/src/main/java/com/genymobile/scrcpy/Options.java
index 9b1d8d8d..b8242aab 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Options.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Options.java
@@ -47,6 +47,7 @@ public class Options {
     private boolean listDisplays;
     private boolean listCameras;
     private boolean listCameraSizes;
+    private boolean dispatchToVD = false;
 
     // Options not used by the scrcpy client, but useful to use scrcpy-server directly
     private boolean sendDeviceMeta = true; // send device name and size
@@ -194,6 +195,10 @@ public class Options {
         return listEncoders || listDisplays || listCameras || listCameraSizes;
     }
 
+    public boolean getDispatchToVD() {
+        return dispatchToVD;
+    }
+
     public boolean getListEncoders() {
         return listEncoders;
     }
@@ -370,6 +375,9 @@ public class Options {
                 case "list_camera_sizes":
                     options.listCameraSizes = Boolean.parseBoolean(value);
                     break;
+                case "dispatch_to_vd":
+                    options.dispatchToVD = Boolean.parseBoolean(value);
+                    break;
                 case "camera_id":
                     if (!value.isEmpty()) {
                         options.cameraId = value;
diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java b/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
index 090c96f0..d5222723 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
@@ -49,6 +49,9 @@ public class ScreenCapture extends SurfaceCapture implements Device.RotationList
             virtualDisplay = ServiceManager.getDisplayManager()
                     .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
             Ln.d("Display: using DisplayManager API");
+            if (device.isDispatchToVD()) {
+                device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
+            }
         } catch (Exception displayManagerException) {
             try {
                 display = createDisplay();

eiyooooo · 2024-08-05T03:14:22Z

Please note that only touch events or other screen control events confirmed to be non-functional need to redirect to mirrorVD.
pressReleaseKeycode or other normal key events should always be injected to the original DisplayId.

mengyanshou · 2024-08-09T13:05:21Z

Please note that only touch events or other screen control events confirmed to be non-functional need to redirect to mirrorVD. pressReleaseKeycode or other normal key events should always be injected to the original DisplayId.

Ok received, I have been busy these days, I will request PR later

mengyanshou · 2024-08-18T11:43:58Z

I have submitted the pull request, please check if there are any errors #5214

rom1v · 2024-10-12T07:34:01Z

Please test #5370.

mengyanshou · 2024-10-14T12:54:28Z

add dispatch_to_vd option to server #5214

see #5370 (comment)

Mouse and touch events must be sent to the virtual display id (used for mirroring), other events (like key events) must be sent to the original display id. Fixes #4598 <#4598> Fixes #5137 <#5137> PR #5370 <#5370> Co-authored-by: nightmare <mengyanshou@gmail.com>

When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, this mechanism has been found to cause some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370>

When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, it has been found to cause some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370>

When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, it causes some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Fixes #5616 <#5616> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370> PR #5614 <#5614>

mengyanshou mentioned this issue Aug 18, 2024

add dispatch_to_vd option to server #5214

Closed

rom1v mentioned this issue Oct 12, 2024

Add virtual display feature #5370

Merged

rom1v closed this as completed in d193967 Nov 24, 2024

This was referenced Dec 5, 2024

How to open only the mouse control thread? #5542

Closed

3.0.2 still has some bugs for just opening the mouse control thread #5611

Closed

rom1v mentioned this issue Dec 5, 2024

Inject main display events to the original display #5614

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices #5137

Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices #5137

mengyanshou commented Jul 26, 2024 •

edited

Loading

rom1v commented Jul 29, 2024 •

edited

Loading

mengyanshou commented Jul 30, 2024

rom1v commented Jul 30, 2024

mengyanshou commented Jul 31, 2024

rom1v commented Jul 31, 2024

mengyanshou commented Jul 31, 2024

mengyanshou commented Jul 31, 2024

eiyooooo commented Aug 5, 2024

mengyanshou commented Aug 9, 2024

mengyanshou commented Aug 18, 2024

rom1v commented Oct 12, 2024

mengyanshou commented Oct 14, 2024

Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices #5137

Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices #5137

Comments

mengyanshou commented Jul 26, 2024 • edited Loading

Environment

Describe the bug

Other

rom1v commented Jul 29, 2024 • edited Loading

mengyanshou commented Jul 30, 2024

rom1v commented Jul 30, 2024

mengyanshou commented Jul 31, 2024

rom1v commented Jul 31, 2024

mengyanshou commented Jul 31, 2024

mengyanshou commented Jul 31, 2024

eiyooooo commented Aug 5, 2024

mengyanshou commented Aug 9, 2024

mengyanshou commented Aug 18, 2024

rom1v commented Oct 12, 2024

mengyanshou commented Oct 14, 2024

mengyanshou commented Jul 26, 2024 •

edited

Loading

rom1v commented Jul 29, 2024 •

edited

Loading