Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhance] Support non-scalar type metric value. #827

Merged
merged 4 commits into from
Jan 12, 2023
Merged

[Enhance] Support non-scalar type metric value. #827

merged 4 commits into from
Jan 12, 2023

Conversation

mzr1996
Copy link
Member

@mzr1996 mzr1996 commented Dec 15, 2022 •
edited
Loading

Motivation

In the original implementation, the RuntimeInfoHook will raise an error if the metric value is not a scalar. But some metric result is non-scalar type, like confusion-matrix.

Modification

If the metric value is not scalar, the value will be saved into the runtime_info. And in the LogProcessor, it will extract runtime info according to the mode.

BC-breaking (Optional)

No

Use cases (Optional)

01/10 10:59:49 - mmengine - INFO - Load checkpoint from /home/PJLAB/mazerun/ckpt/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth
01/10 10:59:51 - mmengine - INFO - Epoch(test) [100/625]    eta: 0:00:08  time: 0.0068  data_time: 0.0001  memory: 84  
01/10 10:59:51 - mmengine - INFO - Epoch(test) [200/625]    eta: 0:00:04  time: 0.0068  data_time: 0.0001  memory: 85  
01/10 10:59:52 - mmengine - INFO - Epoch(test) [300/625]    eta: 0:00:03  time: 0.0068  data_time: 0.0001  memory: 86  
01/10 10:59:53 - mmengine - INFO - Epoch(test) [400/625]    eta: 0:00:02  time: 0.0069  data_time: 0.0001  memory: 86  
01/10 10:59:54 - mmengine - INFO - Epoch(test) [500/625]    eta: 0:00:01  time: 0.0068  data_time: 0.0001  memory: 87  
01/10 10:59:54 - mmengine - INFO - Epoch(test) [600/625]    eta: 0:00:00  time: 0.0068  data_time: 0.0001  memory: 88  
01/10 10:59:55 - mmengine - INFO - Epoch(test) [625/625]  accuracy/top1: 94.8200  single-label/precision_classwise: 
tensor([94.7473, 97.4975, 94.1414, 89.7670, 94.1642, 90.6931, 95.9446, 98.0652,
        96.0474, 97.1689])
  single-label/recall_classwise: 
tensor([95.6000, 97.4000, 93.2000, 88.6000, 95.2000, 91.6000, 97.0000, 96.3000,
        97.2000, 96.1000])
  single-label/f1-score_classwise: 
tensor([95.1717, 97.4487, 93.6683, 89.1797, 94.6793, 91.1443, 96.4694, 97.1746,
        96.6203, 96.6315])
  confusion_matrix/result: 
tensor([[956,   2,  10,   4,   3,   1,   4,   0,  17,   3],
        [  4, 974,   0,   1,   1,   0,   0,   0,   3,  17],
        [  9,   0, 932,  20,  15,   5,  13,   3,   2,   1],
        [  6,   0,  14, 886,  12,  64,  10,   3,   5,   0],
        [  1,   0,  10,  11, 952,  11,   8,   7,   0,   0],
        [  5,   0,   8,  48,  10, 916,   6,   5,   1,   1],
        [  5,   0,  10,   6,   5,   3, 970,   0,   0,   1],
        [  3,   0,   4,   6,  13,  10,   0, 963,   1,   0],
        [ 15,   4,   1,   3,   0,   0,   0,   0, 972,   5],
        [  5,  19,   1,   2,   0,   0,   0,   1,  11, 961]])

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@CLAassistant
Copy link

CLAassistant commented Dec 15, 2022 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@HAOCHENYE HAOCHENYE added this to the 0.4.0 milestone Dec 16, 2022
@HAOCHENYE
Copy link
Collaborator

Please update the branch with the latest main

@mzr1996 mzr1996 requested a review from HAOCHENYE December 21, 2022 06:07
HAOCHENYE
HAOCHENYE previously approved these changes Dec 21, 2022
View reviewed changes
@mzr1996 mzr1996 force-pushed the non-scalar-metric branch from ce94627 to fae73fb Compare January 3, 2023 02:56
@mzr1996 mzr1996 force-pushed the non-scalar-metric branch from fae73fb to 110e364 Compare January 6, 2023 09:29
@HAOCHENYE HAOCHENYE modified the milestones: 0.4.0, 0.5.0 Jan 12, 2023
@HAOCHENYE HAOCHENYE added the P0 Highest priority label Jan 12, 2023
HAOCHENYE
HAOCHENYE approved these changes Jan 12, 2023
View reviewed changes
Copy link
Collaborator

@HAOCHENYE HAOCHENYE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved!!!
image

@zhouzaida zhouzaida merged commit fcd783f into open-mmlab:main Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RangiLyu RangiLyu RangiLyu left review comments

@zhouzaida zhouzaida zhouzaida approved these changes

@HAOCHENYE HAOCHENYE HAOCHENYE approved these changes

Assignees

@ice-tong ice-tong

Labels
P0 Highest priority
Projects
None yet
Milestone
0.5.0
Development

Successfully merging this pull request may close these issues.
6 participants
@mzr1996 @CLAassistant @HAOCHENYE @RangiLyu @zhouzaida @ice-tong