Hi @innobead
I will verify this issue after @weizhe0422 fixes it.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Launch a large backing image and check the BackingImageDatasource POD should be launched
   • Create a large size volume, and create backing image via Export from a Longhorn volume
2. Block all ingress and outgress package with applying network policy, check the progress status should be paused after few seconds.

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-pod-all-packages
     namespace: longhorn-system
   spec:
     podSelector:
       matchLabels:
         longhorn.io/backing-image-data-source: <your-backing-image-name>
         longhorn.io/component: backing-image-data-source
     policyTypes:
     - Ingress
     - Egress

3. Check the logs of the longhorn manager pod, it should not show the error panic: close of closed channel. And after the monitoring is stopped because it cannot connect to the data source POD, the next round of monitoring will continue.
   

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:
  Since the lognhorn manager POD will panic and restart automatically, it will resume normal if the BackingImageDatasource POD is recovered.

• Does the PR include the explanation for the fix or the feature?
  fix: Check if the channel has been closed before closing longhorn-manager#1590 (comment)

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at fix: Check if the channel has been closed before closing longhorn-manager#1590

Update the e2e result for backingimage is pass

Verified on master-head 20221208

• longhorn master-head (13ac2a66)
• longhorn-manager master-head (5a07e68)

The test steps

Scenario 1
Ref #4865 (comment)

Scenario 2

1. Upload a local file as a backing image and check the BackingImageDatasource POD should be launched
2. Access to BackingImageDatasource POD's node
3. Find out the process ID and, freeze the process
   ps aux | grep 'backing-image-manager --debug data-source' |grep -v grep | awk '{print $2}' | xargs -i kill -STOP {}

Result
Scenario 1

1. Unblock BackingImageDatasource pod ingress and outgress package
2. The backing image downloading will be continued and marked as ready

Scenario 2

1. Unfreeze the process
   ps aux | grep 'backing-image-manager --debug data-source' |grep -v grep | awk '{print $2}' | xargs -i kill -CONT {}
2. The backing image downloading will be marked as failed without any panic
   

@roger-ryao @shuo-wu : what is the largest file ever tested?

users are reporting with a 75GB local file to upload, it failed.
harvester/harvester#3450

could it be related to POD resource restriction or disk space pressure ? thanks.
cc @chrisho

@w13915984028 What longhorn version is the user using? This fix is only going to 1.4.0 and 1.3.3.

@innobead
The issue is reported in Harvester v1.1.1, which has longhorn: helm.sh/chart: longhorn-1.3.2.

This fix, #4865, should not in Harvester v1.1.1.

And further, will big file causes potential issue to LH? That's the reason I want to know the ever tested largest single file size. thanks.

I see. If that's streaming, don't think that's a problem. Let's see the update from @roger-ryao / @shuo-wu .

@roger-ryao @shuo-wu : what is the largest file ever tested?

users are reporting with a 75GB local file to upload, it failed. harvester/harvester#3450

could it be related to POD resource restriction or disk space pressure ? thanks. cc @chrisho

Hi @w13915984028
I verified this issue with a 5GB local file to upload.

@roger-ryao @innobead

We suspect the large image file (e.g. 50Gb ~ 100Gb) may cause pressue to the PODs in the uploading path, when possible, please test with such big image files. And also add them into auto test.

LH will have a specification, saying which size of image can be handled smoothly.

thanks.

@w13915984028 Is the large file uploading failure caused by this kind of issue?
#4902

@shuo-wu
The straightforward issue is
harvester/harvester#3450, some POD crashed inbetween the uploading an 75Gb single-file image.

#4902 touchs more components, the vm-importer is an upper layer controller, which utilize the uploading functionality.

@w13915984028 Let's create another issue to track large file uploading instead to clarify this further.

OK, lets discuss further in #5395, thanks.