Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test_scores_for_identical_examples unit test fails #1079

Merged
merged 4 commits into from
Apr 6, 2024
Merged

Fix flaky test_scores_for_identical_examples unit test fails #1079

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4f31a8b
Fix: np.isclose atol parameter
gogetron Apr 1, 2024
d3833de
Add: correct_precision_errors
gogetron Apr 4, 2024
830338b
Add: test_scores_for_identical_examples_across_rows
gogetron Apr 4, 2024
1ca5ef6
Fix: include p when metric is minkowski
gogetron Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cleanlab/internal/outlier.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,6 @@ def transform_distances_to_scores(
ood_features_scores: np.ndarray = np.exp(-1 * avg_distances / scaling_factor * t)

# Set scores to 1 if the average distance is close to 0
inds = np.isclose(avg_distances, 0)
inds = np.isclose(avg_distances, 0, atol=5e-6)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be changing our source code to fix unit tests. Should instead fix the unit test. This atol value was added here for a reason, but will let @elisno share how to best proceed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out tthat he default atol was a bit too small for the euclidean distance, so we explicitly set the atol parameter based how large the floating-point error can get for the distance computations.

It's still not 100% verified how this happens on only a few platforms for the euclidean distances, but we're confident that we can put a MUCH lower tolerance for the cosine metric.
The takeaway is: A single atol value won't work for different types of datasets, we need to set it carefully when comparing values against 0.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not within the scope of this PR, but we may want to look at similar things for the near-duplicate check.

ood_features_scores[inds] = 1.0
return ood_features_scores
Loading