@mattiasbax hi, how did you get the onnx file? Could you provide a sample code?

Hi @RunningLeon ,

The onnx file was acquired by the provided deploy.py script, i.e:

python tools/deploy.py configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py ../mmdetection/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py ../mmdetection/checkpoints/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth ../mmdetection/demo/demo.jpg --work-dir ./work_dir/rtmdet-ins-ort-s --device cuda:0 --dump-info

The onnx file has been loaded into a python script using onnxruntime and executed inference on both image and video data successfully.

@mattiasbax mmdeploy provides torch2onnx with the awareness of the backend configured in deploy configs. So if you want to deploy onto TensorRT, you have to rerun tools/deploy.py with tensorrt deploy config such as configs/mmdet/instance-seg/instance-seg_tensorrt_static-800x1344.py

@RunningLeon So is there is a difference in the provided .onnx file if I run deploy.py with "instance-seg_tensorrt_static-800x1344.py" and "instance-seg_rtmdet-ins_onnxruntime_static-640x640"? The .engine file I received from running deploy.py with tensorrt_static_xyz.py was not compatible with my tensorRT version, which is why I wanted to build the engine file myself.

Yes, the onnx files are different. I just tested OK on torch1.10.0+trt8.4.1.5+cuda11.3.

@RunningLeon I see, thanks.

I just tried with the above script using the other .onnx file provided from using tensorrt_static_xyz.py file. I got these errors:

[07/03/2023-11:51:42] [TRT] [E] 3: getPluginCreator could not find plugin: TRTBatchedNMS version: 1
[07/03/2023-11:51:42] [TRT] [E] ModelImporter.cpp:771: While parsing node number 463 [TRTBatchedNMS -> "/TRTBatchedNMS_output_0"]:
[07/03/2023-11:51:42] [TRT] [E] ModelImporter.cpp:772: --- Begin node ---
[07/03/2023-11:51:42] [TRT] [E] ModelImporter.cpp:773: input: "/Unsqueeze_10_output_0"
input: "/Where_output_0"
output: "/TRTBatchedNMS_output_0"
output: "/TRTBatchedNMS_output_1"
output: "/TRTBatchedNMS_output_2"
name: "/TRTBatchedNMS"
op_type: "TRTBatchedNMS"
attribute {
  i: 0
  type: INT
}
attribute {
  name: "iou_threshold"
  f: 0.6
  type: FLOAT
}
attribute {
  name: "is_normalized"
  i: 0
  type: INT
}
attribute {
  name: "keep_topk"
  i: 100
  type: INT
}
attribute {
  name: "num_classes"
  i: 80
  type: INT
}
attribute {
  name: "return_index"
  i: 1
  type: INT
}
attribute {
  name: "score_threshold"
  f: 0.05
  type: FLOAT
}
attribute {
  name: "topk"
  i: 5000
  type: INT
}
domain: "mmdeploy"

[07/03/2023-11:51:42] [TRT] [E] ModelImporter.cpp:774: --- End node ---
[07/03/2023-11:51:42] [TRT] [E] ModelImporter.cpp:777: ERROR: builtin_op_importers.cpp:5404 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
In node 463 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"

I guess I'm missing some details on how to import the plugins correctly I guess

hi, pls. rebuild mmdeploy with tensorrt or use docker image https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/build_from_docker.html#use-docker-image

@RunningLeon Ok, I will try to rebuild mmdeploy. Do I have to build it with tensorRT 8.2.3 or can I build it with later versions?

Hi @RunningLeon ,

I rebuilt my mmdeploy in accordance to (https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/windows.html) and successfully built the mmdeploy_tensorrt_ops.dll.

I still get the above errors:

[07/04/2023-08:46:44] [TRT] [E] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
In node 463 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"

How do I use the mmdeploy_tensorrt_ops so that my application can find the operations?

hi, you could try add the dll to path env

@RunningLeon I already did that, without any success unfortunately. I even tried moving it directly to the tensorrt lib folder (which is in path env).

Is there anything else I need from the built custom ops? I did not build the SDKs, only the TensorRT custom ops

hi, pls. check cuda lib cudnn lib tensorrt lib and add them all to the path

@RunningLeon They are all correctly in the path. Finding cuda, cudnn or tensorrt is not the problem it seems. It seems that it is not loading the DLL that contains the custom ops for the RTMDet-inst model.

Should I try bumping to a later TensorRT version so that I can use load library (https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Plugin/IPluginRegistry.html#tensorrt.IPluginRegistry.load_library) to specify the path to the mmdeploy_tesonrrt_ops.dll?

Could you maybe send me your built mmdeploy_tesonrrt_ops.dll so I can test if it's the plugin or the environment that is faulty on my end?

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

I'm experiencing the same error, looking forward to a solution!

same for me : in my case the inference with python passes but with C++ I am getting the same error
ERROR] 4: [graphShapeAnalyzer.cpp::processCheck::862] Error Code 4: Internal Error (/model/proposal_generator/TopK: K exceeds the maximum value allowed (3840).)
look like a pattern

same for me : in my case the inference with python passes but with C++ I am getting the same error ERROR] 4: [graphShapeAnalyzer.cpp::processCheck::862] Error Code 4: Internal Error (/model/proposal_generator/TopK: K exceeds the maximum value allowed (3840).) look like a pattern

@datinje hi, could you try this PR #2343?

I have the same issue when I try to generate TensorRT engine file. I have used the tensorrt deploy config as suggested in the comment above : #2236 (comment)

Error[4]: [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace'::ShapeAnalyzerImpl::processCheck::862] 
Error Code 4: Internal Error (/TopK: K exceeds the maximum value allowed (3840).)

I also have the latest mmdeploy version that includes PR #2343, as suggested by @RunningLeon .