Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stat/running: Add running package for computing stats of a running st… #1067

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

btracey
Copy link
Member

@btracey btracey commented Aug 20, 2019

…ream of data

Fixes #1037.

Please take a look.

@btracey
Copy link
Member Author

btracey commented Aug 20, 2019

Currently a mostly complete sketch of how it could look (but needs better comments and tests).

I think a good idea would be to have a running.Stats that computes the kitchen sink (i.e. some decent set of things that can be computed online). Seems reasonable to also have a MeanStd . Not sure how much broader the scope is (or if we even need to think about it now).

// is multiplied by 0.99 for every added value. If Decay is zero, a default
// value of 1 is used (the count is not decayed).
Decay float64
InitCount float64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments for these fields.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

// Reset resets the counter. The next accumulate will use the
// initial mean and count.
func (m *Mean) Reset() {
m.count = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I understand how these are being used. I was going to suggest

m.count = m.InitCount
m.mean = m.InitMean

but then, the code in AccumWeighted doesn't make sense.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


import "math"

// Mean helps compute running mean
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// Mean is a running mean accumulator.? The "helps" seems odd.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@kortschak
Copy link
Member

ping

@kortschak
Copy link
Member

Is this still being worked on?

Copy link
Member

@kortschak kortschak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering about the need for the accessors and unexported fields. Would it be reasonable just to have Mean and Count? Then the API gets much smaller.

return
}
decay := m.decay()
decay = math.Pow(decay, weight)
m.count *= decay
m.mean = (m.mean*m.count + v*weight) / (m.count + weight)
//m.mean = (m.mean*m.count + v*weight) / (m.count + weight)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete this commented out code?

@@ -72,3 +81,11 @@ func (m *Mean) decay() float64 {
}
return m.Decay
}

func updateMean(mean, count, v, weight float64) float64 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this be used elsewhere?

}

// Stats is an accumulator for running statistics.
type Stats struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More to come?


set bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a comment.

@btracey
Copy link
Member Author

btracey commented Jan 28, 2020

I'm sorry, the last thing I pushed up was not what I meant to push. I pushed up what I intended now. Doing a refactor which will make it simpler.

@btracey
Copy link
Member Author

btracey commented Jan 28, 2020

Yep, this is way better, thanks for the comments.

@kortschak
Copy link
Member

What about exporting the fields?

Copy link
Member

@kortschak kortschak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests?


// Accum adds the value to the running total.
func (m *Mean) Accum(v float64) {
m.AccumWeighted(v, 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do this explicitly here. There are elidable checks and math.Pow is not cheap.

@vladimir-ch
Copy link
Member

@btracey Do you want to keep this open?

import "math"

// Mean is a running mean accumulator.
type Mean struct {
Copy link
Contributor

@soypat soypat Aug 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe name this type "MeanExpDecay" to leave the floor open for other running mean implementations?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stat: "running" subpackage for generator-like statistics?
4 participants