Update local_context.py/ssh_context.py #370
Conversation
To support wildcards for backward*files
for more information, see https://pre-commit.ci
Modify error file download.
To raise RuntimeError if no files found.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #370 +/- ##
==========================================
- Coverage 57.43% 57.38% -0.06%
==========================================
Files 31 31
Lines 3409 3466 +57
==========================================
+ Hits 1958 1989 +31
- Misses 1451 1477 +26
☔ View full report in Codecov by Sentry. |
To support wildcards for ssh_context
for more information, see https://pre-commit.ci
ZhouXY-PKU
changed the title
Fix error of using glob for remote root.
for more information, see https://pre-commit.ci
To deal with error of the unit tests.
dpdispatcher/ssh_context.py
Outdated
| @@ -647,30 +648,45 @@ def download( | |||
| file_list = [] | |||
| # for ii in job_dirs : | |||
| for task in submission.belonging_tasks: | |||
| try: | |||
| remote_file_list = self.sftp.listdir( | |||
There was a problem hiding this comment.
I think listdir only lists files under the root directory, doesn't it? If the file is aa/bb, remote_file_list only contains aa and cannot find aa/bb.
There was a problem hiding this comment.
Also, i/o operation in the remote server might be expensive, so I suggest only doing that when necessary (i.e. * is in the filenames).
There was a problem hiding this comment.
Sorry, it truly ignores aa/bb , and it has been fixed now.
And thanks for your suggestions, listdir now only do when wildcards are in the filenames.
There was a problem hiding this comment.
(Bohrium HDFS OpenAPI Context remain to be modified)
Search for files in dirs of task_work_path. Listdir only when wildcards appear.
for more information, see https://pre-commit.ci
Try to solve pyright problem of unit test: type checker
for more information, see https://pre-commit.ci
Try to pass unit test of TypeChecker again
|
I am going to add some tests to confirm the code is correct. |
Fix regression caused by deepmodeling#370. Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Fix regression caused by #370. Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
To support wildcards for backward_files and backward_common_files.
Now only support for local_context and ssh_context.
#371