[examples] Add IterationStrategies to MultiDataLoader/Module #94
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
This was referenced Jun 15, 2022
facebook-github-bot
added
the
CLA Signed
IvanKobzarev
added a commit
that referenced
this pull request
Jun 15, 2022
ghstack-source-id: 4986d3c7f4e2b9ed4a1f0ed3c127e93531515111 Pull Request resolved: #94
[ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jun 15, 2022
ghstack-source-id: 4986d3c7f4e2b9ed4a1f0ed3c127e93531515111 Pull Request resolved: #94
Codecov Report
@@ Coverage Diff @@
## gh/ivankobzarev/6/base #94 +/- ##
=========================================================
Coverage ? 92.97%
=========================================================
Files ? 47
Lines ? 2805
Branches ? 0
=========================================================
Hits ? 2608
Misses ? 197
Partials ? 0 Continue to review full report at Codecov.
|
[ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jun 15, 2022
ghstack-source-id: 081b11e84bfa90725a0686fab58feb8d7da1d87f Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jun 16, 2022
ghstack-source-id: b20fe653e45ec5b65edb685220c4831051199d2e Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jun 16, 2022
ghstack-source-id: 845e5c829dd48ff6cf7b833a99e4354eb9fe6fa1 Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jun 17, 2022
ghstack-source-id: 8fae59f37ad38a9e6d96f28c3b2efa544cf750a0 Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jul 19, 2022
ghstack-source-id: 294c95e7a13214c4938ca0c8b05c3bbcbeeb411c Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jul 19, 2022
ghstack-source-id: 49429ccd45c7e7a0ba12b8b04a029f2bc1481fcc Pull Request resolved: #94
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jul 19, 2022
ghstack-source-id: deb50818d1eaa28543c0f05ba4fffab68c28a6d9 Pull Request resolved: #94
Introducing `common.data.IterationStrategy` interface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios' For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary. Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered) Testing: Intruducing tests test/common/data/test_multidata.py test/common/data/test_iteration_strategies.py ``` cd examples python -m unittest test/common/data/*.py ``` Differential Revision: [D37188386](https://our.internmc.facebook.com/intern/diff/D37188386) [ghstack-poisoned]
IvanKobzarev
added a commit
that referenced
this pull request
Jul 20, 2022
ghstack-source-id: ce664943fe80efe96cecb08634e45bccd9a0b0b0 Pull Request resolved: #94
|
@IvanKobzarev has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Introducing
common.data.IterationStrategyinterface and its implementations for 'constant', 'round_robin', 'random', 'size_proportional' and 'ratios'For ratios we can specify probabilities per each datamodule, for that we need to keep name <-> dataloader => Changing specification of loaders not as a list but by dictionary.
Using OrderedDict to keep the insertion order for the round_robin iteration strategy. (Since python 3.7 default dictionaries are insertion ordered)
Testing:
Intruducing tests
test/common/data/test_multidata.py
test/common/data/test_iteration_strategies.py
Differential Revision: D37188386