Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collinearity Speed Up v2 #421

Merged
merged 19 commits into from
Jul 15, 2022
Merged

Collinearity Speed Up v2 #421

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
93f3d06
Collinearit Speed Up v2
ChadGueli Jun 21, 2022
5ce5c7a
small fix
ChadGueli Jun 21, 2022
49deb77
added a dash of numerical stability
ChadGueli Jun 21, 2022
99b5e88
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 21, 2022
1b70947
added comment
ChadGueli Jun 21, 2022
5d02bc3
Merge branch 'mcollin_prob' of github.com:ChadGueli/mage-ai into mcol…
ChadGueli Jun 21, 2022
22e31d9
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 22, 2022
366bc60
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 22, 2022
3c9e626
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 22, 2022
ae06e6f
just faster now
ChadGueli Jun 22, 2022
785f61e
Merge branch 'mcollin_prob' of github.com:ChadGueli/mage-ai into mcol…
ChadGueli Jun 22, 2022
914b6d4
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 22, 2022
c52de2c
added safety
ChadGueli Jun 22, 2022
55c06f1
Merge branch 'mcollin_prob' of github.com:ChadGueli/mage-ai into mcol…
ChadGueli Jun 22, 2022
7bab946
issues resolved
ChadGueli Jun 27, 2022
aba3192
sync
ChadGueli Jun 27, 2022
7652a5b
Merge branch 'mage-ai-master' into mcollin_prob
ChadGueli Jun 27, 2022
6a8087e
name correction
ChadGueli Jun 27, 2022
f061d19
Merge branch 'mage-ai:master' into mcollin_prob
ChadGueli Jun 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
issues resolved
  • Loading branch information
@ChadGueli
ChadGueli committed Jun 27, 2022
commit 7bab946a5244ede9c25fa3f53a340490ab3ae0e6
7 changes: 3 additions & 4 deletions mage_ai/data_cleaner/cleaning_rules/remove_collinear_columns.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,8 @@ def evaluate(self):
collinear_columns = []
good_columns = self.numeric_columns.copy()
while True:
e_vals, e_vecs = np.linalg.eigh(C)
e_vals = np.sign(e_vals) * (e_vals.abs() + self.EPSILON)
vifs = np.sum(e_vecs*e_vecs, axis=0) / e_vals
e_vals = np.linalg.eigvalsh(C)
e_vals = np.sign(e_vals) / (abs(e_vals) + self.EPSILON)
collinearity = vifs >= self.VIF_UB

i = collinearity.argmax()
Expand All @@ -44,4 +43,4 @@ def evaluate(self):
axis=Axis.COLUMN,
)
)
return suggestions
return suggestions