Add AWS Inf2 instances support for aws_batch scheduler #977
Conversation
|
Hi @shixianc! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
@d4l3k, @kiukchung could you help review? thanks! |
Signed a few mins ago, probably take some time to propagate. |
facebook-github-bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
Hi @d4l3k |
|
@kiukchung @d4l3k rebased. Could you help rerun the checks? |
torchx/specs/named_resources_aws.py
Outdated
| return Resource( | ||
| cpu=4, | ||
| gpu=0, | ||
| memMB=32 * GiB, |
There was a problem hiding this comment.
16GB of memory not 32? Pretty sure this refers to the CPU memory not the accelerator memory
There was a problem hiding this comment.
@ashvinnihalani Good catch, thanks for pointing this out.
I was referencing trn1 and thought the field is for GPU memory, but then realize trn1 has same GPU and CPU memory.
torchx/specs/named_resources_aws.py
Outdated
| return Resource( | ||
| cpu=32, | ||
| gpu=0, | ||
| memMB=32 * GiB, |
torchx/specs/named_resources_aws.py
Outdated
| return Resource( | ||
| cpu=96, | ||
| gpu=0, | ||
| memMB=192 * GiB, |
There was a problem hiding this comment.
Even though it's 384 according to the spec: https://aws.amazon.com/ec2/instance-types/inf2/
In reality we have to use smaller number or we will get this from Batch:
Invalid memory value. It should be within the range [1, 376832]
With the current MEM_TAX=0.96:
376832/(384*1024*0.96) = 0.99826388888 => thus it has to be at most 383 with the current mem tax
There was a problem hiding this comment.
@clumsy This is good insights - I gave it a bit more thoughts, I prefer to have this config to stick to actual values of the instances, and have this mem_tax logic somewhere else, so that it can apply to other instances as well.
Also, probably it depends on your setup, we did observe similar issues on inf2.48xlarge batch queue tho.
There was a problem hiding this comment.
I'm OK with any solution that works. This right now I'm not sure works...
| ) | ||
|
|
||
|
|
||
| def aws_inf2_48xlarge() -> Resource: |
|
@ashvinnihalani @d4l3k @kiukchung I made a small correction on the config, could you help merging the CR? (not sure if linting blocker still there) |
|
@kiukchung Small ping about this. |
Add AWS Inf2 instances support for aws_batch scheduler. There're usecases to use torchx to launch data parallel inference jobs on inf2 instances on AWS Batch.
Configurations are referencing https://aws.amazon.com/ec2/instance-types/inf2/
Test plan:
Updated unittest to cover new changes.
=== SCHEDULER REQUEST ===
command:
--nnodes 1 --nproc_per_node 1 --tee 3 --role '' -m abc
image: sha256:3f8f845e25030d9523bf299dee1c3ca6f2b008fc2bf0a2161ed949efe168a3e1
kwargs:
devices:
environment:
LOGLEVEL: WARNING
TORCHX_JOB_ID: local_docker://torchx/abc-kr12qr37093rz
TORCHX_RANK0_HOST: abc-kr12qr37093rz-abc-0
TORCHX_TRACKING_EXPERIMENT_NAME: default-experiment
hostname: abc-kr12qr37093rz-abc-0
labels:
torchx.pytorch.org/app-id: abc-kr12qr37093rz
torchx.pytorch.org/replica-id: '0'
torchx.pytorch.org/role-name: abc
torchx.pytorch.org/version: 0.8.0dev0
mem_limit: 377472m
mounts: []
name: abc-kr12qr37093rz-abc-0
nano_cpus: 192000000000
network: torchx
privileged: false
shm_size: 377472m