dnn: add more CANN operators to support SAM #23550

fengyuentau · 2023-04-26T10:32:50Z

This PR is based on #23491. Need to merge #23491 first then rebase this PR.

To-do list:

LayerNorm (completed via dnn cann backend: add hardswish, layernorm and instasnce norm for cann and bug fix #24462, need to remove code in this pr)
Gelu
Reduce
Sqrt
Fix NaryEltwise (Sub, Pow)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

…unsqueeze; support layernorm fusion variances; support getting 1d desc

modules/dnn/src/onnx/onnx_importer.cpp

dkurt · 2023-08-04T11:47:18Z

modules/dnn/src/layers/layer_norm.cpp

@@ -20,11 +21,13 @@ class LayerNormLayerImpl CV_FINAL : public LayerNormLayer

        // opencv attr
        hasBias = params.get<bool>("hasBias", false);
+        is1D = params.get<bool>("is1D", false);


If I'm not mistaken, in case of 1D cv::Mat it has N rows and 1 column (Nx1). So with axis = 0 it should be correct. I think using such shortcut from MVN is enough:

opencv/modules/dnn/src/layers/mvn_layer.cpp

Lines 331 to 346 in 2ff16d4

if ( inpBlob.total() == newRows )

{

// MVN is applied to single values at an every row.

if (shift.empty())

{

outBlob.setTo(0);

}

else

{

for ( i = 0; i < newRows; i++ )

{

outMat.row(i).setTo(((float*)shift.data)[i]);

}

}

return;

}

@dkurt, in 4.x there is no support for 1D. In 5.x I have added support for 1D and 0D matrices. 1D matrix of N elements may be interpreted (if you ignore Mat::dims) as 1xN 2D matrix, i.e. it's a single-row matrix, not single-column

My point is that instead of extra is1D flag there is a check that let skip layer compute because output values will be all zeros.

The 1d flag is introduced for other backends to avoid incorrect shape inference. Let me try to bypass this...

Lets say axis=-1 and the shape of scale mat is still 2d; that stands for a 1d scale and bias tensor.

fengyuentau · 2023-12-23T09:52:31Z

This PR is too out-of-date. Created a new PR #24756 in place of this one.

LaurentBerger · 2023-12-23T17:03:20Z

What's about Mod layer?

class MyModLayer : public cv::dnn::Layer
{
protected:
    Mat divisor;
public:
    MyModLayer(const cv::dnn::LayerParams& params);
    static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params)
    {
        return cv::Ptr<cv::dnn::Layer>(new MyModLayer(params));
    }
    virtual bool getMemoryShapes(const std::vector<MatShape >& inputs,
        const int requiredOutputs,
        std::vector<MatShape >& outputs,
        std::vector<MatShape>& internals) const CV_OVERRIDE
    {
        outputs.assign(1, inputs[0]);
        return true;
    }
    virtual void forward(cv::InputArrayOfArrays inputs_arr,
        cv::OutputArrayOfArrays outputs_arr,
        cv::OutputArrayOfArrays internals)
    {
        std::vector<cv::Mat> inputs, outputs, internal;
        inputs_arr.getMatVector(inputs);
        outputs_arr.getMatVector(outputs);
        internals.getMatVector(internal);
        Mat blob;
        if ((inputs.size() == 2 && inputs[1].total() == 1 ) || (inputs.size() == 1 && divisor.total()==1))
        {
            int valDivisor;
            if (inputs.size() == 2)
                valDivisor = inputs[1].at<int>(0);
            else
                valDivisor = divisor.at<int>(0);
            inputs[0].copyTo(blob);
            int x = 2;
            switch (blob.depth()) {
            case CV_8U:
                blob.forEach<uchar>([&](uchar elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_8S:
                blob.forEach<char>([&](char elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_16U:
                blob.forEach<ushort>([&](ushort elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_16S:
                blob.forEach<short>([&](short elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_32S:
                blob.forEach<int>([&](int elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_32F:
                blob.forEach<float>([&](float elem, const int* p) { elem = int(elem) % valDivisor; });
                break;
            default:
                CV_Error(-1, "Unimplemented type");
            }
        }
        else
            CV_Error(-1, "Cannot manage mod layer with more than one divisor");
        blob.copyTo(outputs[0]);
    }
    virtual void finalize(cv::InputArrayOfArrays inputs,
        cv::OutputArrayOfArrays outputs)
    {}

};

fengyuentau · 2023-12-25T08:53:49Z

What's about Mod layer?

class MyModLayer : public cv::dnn::Layer
{
protected:
    Mat divisor;
public:
    MyModLayer(const cv::dnn::LayerParams& params);
    static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params)
    {
        return cv::Ptr<cv::dnn::Layer>(new MyModLayer(params));
    }
    virtual bool getMemoryShapes(const std::vector<MatShape >& inputs,
        const int requiredOutputs,
        std::vector<MatShape >& outputs,
        std::vector<MatShape>& internals) const CV_OVERRIDE
    {
        outputs.assign(1, inputs[0]);
        return true;
    }
    virtual void forward(cv::InputArrayOfArrays inputs_arr,
        cv::OutputArrayOfArrays outputs_arr,
        cv::OutputArrayOfArrays internals)
    {
        std::vector<cv::Mat> inputs, outputs, internal;
        inputs_arr.getMatVector(inputs);
        outputs_arr.getMatVector(outputs);
        internals.getMatVector(internal);
        Mat blob;
        if ((inputs.size() == 2 && inputs[1].total() == 1 ) || (inputs.size() == 1 && divisor.total()==1))
        {
            int valDivisor;
            if (inputs.size() == 2)
                valDivisor = inputs[1].at<int>(0);
            else
                valDivisor = divisor.at<int>(0);
            inputs[0].copyTo(blob);
            int x = 2;
            switch (blob.depth()) {
            case CV_8U:
                blob.forEach<uchar>([&](uchar elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_8S:
                blob.forEach<char>([&](char elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_16U:
                blob.forEach<ushort>([&](ushort elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_16S:
                blob.forEach<short>([&](short elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_32S:
                blob.forEach<int>([&](int elem, const int* p) { elem = elem % valDivisor; });
                break;
            case CV_32F:
                blob.forEach<float>([&](float elem, const int* p) { elem = int(elem) % valDivisor; });
                break;
            default:
                CV_Error(-1, "Unimplemented type");
            }
        }
        else
            CV_Error(-1, "Cannot manage mod layer with more than one divisor");
        blob.copyTo(outputs[0]);
    }
    virtual void finalize(cv::InputArrayOfArrays inputs,
        cv::OutputArrayOfArrays outputs)
    {}

};

Sure, let me do it in a new PR.

fengyuentau and others added 6 commits April 26, 2023 16:43

support broadcast on axis > 1 for Expand

370b50c

allow null constant_value in Pad and ignore it when loading

381d2a5

add cann layernorm op

7cb5d46

add gelu

efa45b5

support pow

d4b43b1

add sqrt

1fddf4a

fengyuentau added this to the 4.8.0 milestone Apr 27, 2023

fengyuentau added 7 commits April 28, 2023 11:21

add reduce

1c070be

make error msg informative with dtype and node name

a949861

fix empty bias_mat when the node type is matmul

4452498

mat type should not have channel for CannConstOp

854c696

fix number of Split outputs

11cfffd

support batched matmul

0ae42db

support 1D const; fix desc_gamma and desc_beta in layernorm; support …

015c9c0

…unsqueeze; support layernorm fusion variances; support getting 1d desc

fengyuentau commented May 5, 2023

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

fengyuentau removed this from the 4.8.0 milestone May 31, 2023

fengyuentau mentioned this pull request Aug 4, 2023

Merge MVN and LayerNorm in one layer #24105

Closed

9 tasks

dkurt reviewed Aug 4, 2023

View reviewed changes

fengyuentau added the category:dnn_cann CANN backend related issues in DNN module label Sep 13, 2023

fengyuentau mentioned this pull request Oct 9, 2023

dnn onnx: add instance norm layer #24378

Merged

12 tasks

fengyuentau mentioned this pull request Oct 27, 2023

dnn cann backend: add hardswish, layernorm and instasnce norm for cann and bug fix #24462

Merged

9 tasks

fengyuentau added this to the 4.10.0 milestone Nov 10, 2023

revert some out-of-date changes

f38c4e8

fengyuentau mentioned this pull request Dec 23, 2023

dnn cann: support more operators for SAM #24756

Draft

6 tasks

fengyuentau closed this Dec 23, 2023

opencv-alalek removed this from the 4.10.0 milestone Dec 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dnn: add more CANN operators to support SAM #23550

dnn: add more CANN operators to support SAM #23550

fengyuentau commented Apr 26, 2023 •

edited

Loading

dkurt Aug 4, 2023

vpisarev Oct 9, 2023

dkurt Oct 9, 2023

fengyuentau Nov 10, 2023

fengyuentau commented Dec 23, 2023

LaurentBerger commented Dec 23, 2023 •

edited

Loading

fengyuentau commented Dec 25, 2023

	if ( inpBlob.total() == newRows )
	{
	// MVN is applied to single values at an every row.
	if (shift.empty())
	{
	outBlob.setTo(0);
	}
	else
	{
	for ( i = 0; i < newRows; i++ )
	{
	outMat.row(i).setTo(((float*)shift.data)[i]);
	}
	}
	return;
	}

dnn: add more CANN operators to support SAM #23550

dnn: add more CANN operators to support SAM #23550

Conversation

fengyuentau commented Apr 26, 2023 • edited Loading

Pull Request Readiness Checklist

dkurt Aug 4, 2023

Choose a reason for hiding this comment

vpisarev Oct 9, 2023

Choose a reason for hiding this comment

dkurt Oct 9, 2023

Choose a reason for hiding this comment

fengyuentau Nov 10, 2023

Choose a reason for hiding this comment

fengyuentau commented Dec 23, 2023

LaurentBerger commented Dec 23, 2023 • edited Loading

fengyuentau commented Dec 25, 2023

fengyuentau commented Apr 26, 2023 •

edited

Loading

LaurentBerger commented Dec 23, 2023 •

edited

Loading