With your changes in #2413 we should be able to accept targets that are not one-hot encoded (e.g., [1, 0, 2, 2, 0] instead of [[0, 1, 0], [1, 0, 0], [0, 0, 1], [0, 0, 1], [1, 0, 0]]) and have it configurable for the metric.

But this can be done in a follow-up PR 🙂

That's true, I've been looking at the existing code in burn-trai/src/learner/classification and I think it would be easier to just use the adaptor to convert between bin/multiclass/multilable outputs to the general one hot encoded metrics, non?

Yeah my comment was not meant to say that this is where is should be handled specifically 😄

Yeah of course, I just wanted to get your thoughts on what I had thought to do. Now we have the ClassificationOutput and MultiLabelClassificationOutput adapted for ClassificationInput. It works but I'm still not super happy about it since then the user is able to use, for example BinaryPrecisionMetric with a MultiLabelClassificationOutput. My idea for the future would be to have separated Inputs and Outputs for each of the classification types such that this would not be possible and would complain at compile time. Thoughts?

I understand where you're coming from! But I don't see a straightforward way to do this.. we would have to have a different implementation for the binary, multiclass and multilabel precision metrics because the input is an associated type for the Metric trait.

For now I think it's fine.

Yeah, I couldn't think of a way around it but maybe it'll come to me while working on other metrics. 🤞

Shouldn't this also transform the targets with .one_hot(...) but force the num classes to 2 (assuming binary classification)?

Hum, I think that would be covered by the case above since then num_classes == 2. This, I think, is slightly different from binary classification ( I'm thinking for example classifying binary: spam email vs not spam email and multiclass with 2 labels: trees or bushes).Finally, I don't think it would work as output and targets should have the same shape.

Spam vs not spam (i.e., positive vs negative) is still considered binary classification. I think what you are talking about is just that one uses sigmoid to model the positive output (so there is only one score) vs softmax where you have a score for each. But the targets are still represented as 0 or 1.

I'm not sure what self.targets.clone().unsqueeze_dim(1).bool() is supposed to represent 🤔 I could be missing something though

Sure, at the end of the day they all are the same thing. The point I was trying to make is that in binary classification tasks I expect the output of the model to be (batch_size x 1) while in multi class I would expect (batch_size x 2), thus the targets for the first would be transformed from (batch_size) -> (batch_size, 1) and the second (batch_size) -> (batch_size, 2) so they match the shapes of the outputs. Still this is a preference, if you think it doesn't make sense we can just assert that the second dim of output is not less than 2.

Ahhh ok nvm I thought this would lead to some issues because the targets are expected to be one-hot encoded, but in reality this is not entirely true for the binary case for a single scalar output and target. The operations performed will still be valid.

Spam vs not spam (i.e., positive vs negative) is still considered binary classification. I think what you are talking about is just that one uses sigmoid to model the positive output (so there is only one score) vs softmax where you have a score for each. But the targets are still represented as 0 or 1.

I'm not sure what self.targets.clone().unsqueeze_dim(1).bool() is supposed to represent 🤔 I could be missing something though

Ahhh ok nvm I thought this would lead to some issues because the targets are expected to be one-hot encoded, but in reality this is not entirely true for the binary case for a single scalar output and target. The operations performed will still be valid.