Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimizer decomposing Neural Net matrices as sums of structured matrices #64

Draft
wants to merge 56 commits into
base: master
Choose a base branch
from
Draft

Optimizer decomposing Neural Net matrices as sums of structured matrices #64

Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
ef6e2e6
First draft Stochastic Hybrid Prox LMO
GeoffNN Mar 31, 2021
6afc510
Added stochastic Robust PCA example first draft
GeoffNN Mar 31, 2021
1d7b945
Stochastic Robust PCA example fixed
GeoffNN Apr 1, 2021
fa60642
Added layer nuclear norm constraints + tests
GeoffNN Apr 2, 2021
7d0005e
Bug fixes; slowly migrating to param_groups API for Optimizer
GeoffNN Apr 6, 2021
aa447e0
MNIST example LR+Sparse
GeoffNN Apr 6, 2021
6b53c43
Merge branch 'master' into geoff/stochastic_hybrid_prox_lmo
GeoffNN Apr 7, 2021
4f52df2
Merge branch 'master' into geoff/stochastic_hybrid_prox_lmo
GeoffNN Apr 7, 2021
578e8b7
Bug fix NuclearNorm prox
GeoffNN Apr 13, 2021
c921483
slowly migrating optimizers to correct **params API
GeoffNN Apr 13, 2021
a672e64
Stochastic Robust PCA example
GeoffNN Apr 13, 2021
3061108
Deleted obsolete LR + sparse example
GeoffNN Apr 13, 2021
2182446
Merge branch 'master' into geoff/stochastic_hybrid_prox_lmo
GeoffNN Apr 20, 2021
bbe9470
fixes to splitting method -- added gradient normalization
GeoffNN Apr 24, 2021
7671cf7
removed redundant code
GeoffNN Apr 28, 2021
b41a17b
Enforcing relationship between lr, lr_prox and lipschitz
GeoffNN May 1, 2021
227b627
Removed lr_prox parameter; using lipschitz estimate, consistently wit…
GeoffNN May 3, 2021
ccc4059
state is initialized at every step in hybrid optimizer
GeoffNN May 4, 2021
6bf2b13
Prox/LMO are now Modules, to ensure pickle-ability
GeoffNN May 7, 2021
346dd32
Fixed initialization bug: making y feasible w/ correct projection
GeoffNN May 7, 2021
4200710
Fixed stochastic robust PCA example + code snippet for extracting com…
GeoffNN May 8, 2021
22a6c19
Minor fixes
GeoffNN May 8, 2021
6cb9a32
Fix when prox2 is None
GeoffNN May 9, 2021
fb6ec7d
Fixed view/reshape issue
GeoffNN May 10, 2021
28eff81
Allowing 0 constraints
GeoffNN May 10, 2021
34b99a7
Changed dataloader + loss for efficiency
GeoffNN May 14, 2021
9afbd5d
Removed debug statements
GeoffNN May 14, 2021
e95dbed
L1 penalty prox w/ stochastic hybrid splitting
GeoffNN May 14, 2021
601b738
Penalized version of RobustPCA w/ generalized LMO
GeoffNN May 14, 2021
633eee6
Hybrid Prox method now works for penalty LMO
GeoffNN May 15, 2021
10b7c4e
Slight change
GeoffNN May 15, 2021
3c99631
Slight algo modification
GeoffNN May 18, 2021
5106cf3
Fixed penalty to use torch.svd instead of torch.linalg.svd + step siz…
GeoffNN May 18, 2021
2b2f669
Fixed svd calls
GeoffNN May 18, 2021
4291e1f
SVD call updated
GeoffNN May 18, 2021
e5671d0
Fixed penalized training + changes to svd calls
GeoffNN May 18, 2021
b75436b
took out print statement
utrerf May 18, 2021
5460b80
Merge pull request #67 from utrerf/patch-2
GeoffNN May 19, 2021
69bec20
Added penalty initialization
GeoffNN May 20, 2021
c0be0ff
removed todo
GeoffNN May 21, 2021
7028526
removed print, redundant computation
GeoffNN May 21, 2021
34409b1
removed comment
GeoffNN May 21, 2021
88d72b1
Updated CIFAR
utrerf May 23, 2021
24f2faa
Merge pull request #68 from utrerf/patch-3
GeoffNN May 23, 2021
9393c27
Minor rewrites
GeoffNN May 26, 2021
65da92f
Merge branch 'geoff/stochastic_hybrid_prox_lmo' of https://github.com…
GeoffNN May 26, 2021
14378dc
vectorized L1/Simplex projections
GeoffNN May 26, 2021
cd283ca
special case when nuclear norm is of diameter 0
GeoffNN May 27, 2021
429129b
Made Frank-Wolfe savable
GeoffNN May 27, 2021
d575a09
Updated ImageNet
utrerf Jun 1, 2021
8634690
Merge pull request #69 from utrerf/patch-4
GeoffNN Jun 3, 2021
68129af
took out the ch
utrerf Aug 6, 2021
1b9fd33
Merge pull request #70 from utrerf/patch-5
GeoffNN Aug 6, 2021
86ba4bd
Added tqdm for iters in optim
GeoffNN Aug 6, 2021
47aa745
Bug fix for penalized stochastic FW
GeoffNN Sep 17, 2021
ca1bb2b
Merge branch 'geoff/stochastic_hybrid_prox_lmo' of github.com:openopt…
GeoffNN Sep 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added tqdm for iters in optim
  • Loading branch information
@GeoffNN
GeoffNN authored Aug 6, 2021
commit 86ba4bdb307255788beabcf809dc6b22e5a95c33
11 changes: 7 additions & 4 deletions chop/optim.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@

from numbers import Number
import warnings
from tqdm import tqdm

import torch
import numpy as np

from scipy import optimize
from numbers import Number
from chop import utils
Expand Down Expand Up @@ -153,7 +156,7 @@ def prox2(x, s=None, *args):
x = prox1(z - utils.bmul(step_size, grad), step_size, *args_prox)
u = torch.zeros_like(x)

for it in range(max_iter):
for it in tqdm(range(max_iter)):
z.requires_grad_(True)
fval, grad = closure(z)
with torch.no_grad():
Expand Down Expand Up @@ -316,7 +319,7 @@ def prox(x, s=None):
else:
raise ValueError("step must be float or backtracking or None")

for it in range(max_iter):
for it in tqdm(range(max_iter)):

fval, grad = closure(x)
x_next = prox(x - utils.bmul(step_size, grad), step_size, *prox_args)
Expand Down Expand Up @@ -410,7 +413,7 @@ def minimize_frank_wolfe(closure, x0, lmo, step='sublinear',

cert = np.inf * torch.ones(batch_size, device=x.device)

for it in range(max_iter):
for it in tqdm(range(max_iter)):

x.requires_grad = True
fval, grad = closure(x)
Expand Down Expand Up @@ -511,7 +514,7 @@ def minimize_alternating_fw_prox(closure, x0, y0, prox=None, lmo=None, lipschitz

fval, grad = closure(x + y)

for it in range(max_iter):
for it in tqdm(range(max_iter)):

if step == 'sublinear':
step_size = 2. / (it + 2) * torch.ones(batch_size, device=x.device)
Expand Down