[storage] Preview 2 (#6442)

* [storage] Add async APIs for Files SDK (#6405) * Refactored blob shared utils * Refactored file shared utils * Refactored queue shared utils * Refactored downloads * Refactored file downloads * Started async downloads * Async Files API * Flatten copy polling * Renamed uploads * Fixes samples based on vendor feedback (#6357) * fixed import statements for sync samples * fixed secrets readme samples * fixed keys readme samples * deduped secret names across samples * Upload refactor * Release approval docs (#6361) * updating dev documentation with approval details * Updated async pipeline * Avoid surprising aiohttp with unexpected kwargs (#6355) * Add challenge authentication to azure-keyvault-keys (#6244) * Add decorator (#6299) * add opencensus impl * dont need to check for noop tracer * get rid of span and trace id * fix span instance * added documentation * added documentation and refactor names of a few variables * write test for opencensus wrapper * put opencensus in the dev requirements * initial common * only import the wrapper when necessary * add check for the exporter * rework logic and fix some settings * added initial decorator * some mroe documentation * added decorators * small change * some minor fixes * test patch happening * fix space * share a function * clearer logic for setting span context * better logic * better logic * fix environ variable * test the way opencensus does it * middle of tests * only load if opencensus has already been imported * fix spelling mistake * temp * finish writing tests for common * charles fixes * fix tests * fix test settings * to header should not take a dict * from header should be class method * initial tests * dont create trace and get rid of end_tracer * dont need to save the trace * more little fixes * some intermediatary changes * fix type annotations * rst fix types * add :class:annotations * fix line wrapping * added tests for decorator * rename opencensus wrapper * intermediate changes * use spans the right way * some formatting * some grammar * restructure settings and make tests pass * rename get_parent * fix typings * use protocol and from_headers becomes links * ramifications of opencensus wrapper being a protocol * add tests for link * added async tests * delete the unit test thing * added add_attribute * added add_attribute * added tests for add attributes * remove unused import * added docstrings * minor docstring formatting * fix pylint errors * don't rely on opencensus children to check * use exporter to not rely on parent.children * test should only propagate * made test tracing helper * decrease flakiness of test * simplify get parent * calling a decorator decorator is redundant * fix settings * more elegant code * add await for async * should only have to wait a 1/1000 of a second * fix tests spans too short * accidentally deleted setup.cfg * Added async file tests * Consolidate Key Vault shared code (#6384) * Download tests * Service property tests * No recordings * Add credential wrapping MSAL ConfidentialClientApplication (#6358) * Add policy (#6379) * add opencensus impl * dont need to check for noop tracer * get rid of span and trace id * fix span instance * added documentation * added documentation and refactor names of a few variables * write test for opencensus wrapper * put opencensus in the dev requirements * initial common * only import the wrapper when necessary * add check for the exporter * rework logic and fix some settings * added initial decorator * some mroe documentation * added decorators * small change * some minor fixes * test patch happening * fix space * share a function * clearer logic for setting span context * better logic * better logic * fix environ variable * test the way opencensus does it * middle of tests * only load if opencensus has already been imported * fix spelling mistake * temp * finish writing tests for common * charles fixes * fix tests * fix test settings * to header should not take a dict * from header should be class method * initial tests * dont create trace and get rid of end_tracer * dont need to save the trace * more little fixes * some intermediatary changes * fix type annotations * rst fix types * add :class:annotations * fix line wrapping * added tests for decorator * rename opencensus wrapper * intermediate changes * use spans the right way * some formatting * some grammar * restructure settings and make tests pass * rename get_parent * fix typings * use protocol and from_headers becomes links * ramifications of opencensus wrapper being a protocol * add tests for link * added async tests * delete the unit test thing * added add_attribute * added add_attribute * added tests for add attributes * add initial policy * remove unused import * added docstrings * minor docstring formatting * fix pylint errors * don't rely on opencensus children to check * use exporter to not rely on parent.children * added documentation and span attributes * added test tracing policy * test should only propagate * made test tracing helper * decrease flakiness of test * simplify get parent * calling a decorator decorator is redundant * middle of writing tests * fix settings * add tests * test propogation also happens * more elegant code * await async stuff * add await for async * should only have to wait a 1/1000 of a second * fix tests spans too short * accidentally deleted setup.cfg * add component * add set_http_attributes * fix span network name * fix http request types * fix more types * bryan fixes * more efficient tests * non flakey tests * make tracing only use my context * test user agent on exception * pylint formatting * delete unused import * fix spelling * pylint * adding dockerfile (#6393) * Update cheatsheet.md * Async share tests * Async directory tests * Fixed some tests * aiohttp socket timeout * Patch azure core * CI fixes * Fix async tests for py35 * Python 3.5 support * Clean pylint * Async implementation for storage queues. (#6360) * Async Queues + tests port * More tests plus changes * pytest conf * Port shared folder * remove warnings from tests * Latest shared code * change destination * some more changes * ..and some more * fix tests * pylint + mypy * some more lint * A couple of test fixes * Fixed fake settings mode * Refactored queue utils * change conf ignore * minor change * Fix for urlencoding SAS tokens * Merge latest azure-core changes * Updated shared blob client * add decorator and policy to storage_files and propagate context for it (#6454) * add decorator and policty to files also propagated context! * added decorator to pollers * finished should also be traced * polling method does not need to be decorated * add tracing policies * propagate context * no decorator for generate_shared_access_signature * added decorators to async libs * Trace storage queue (#6449) * decorators and policy added * properly clear context * dont decorate private stuff * added policy * decorated async * get rid of those that dont make netowrk calls * propagate context * Trace storage blob (#6478) * properly clear context (#6448) * added decorators to keyvault-keys (#6381) * Remove default development settings (#6376) * Trace decorator optional arguments (#6459) * add optional parameters to decorators * added type hints * Use tracing policy (#6390) * Remove config public API for KV keys (#6418) * Remove config public API for KV keys * Make create_config private * Sync _shared keys/secrets * Type annotation fix * Adapt secrets tests for no config * Update README.md * decorate polling * decorate lease * decorate container client * trace blob service client * polling method does not need to be decorated * decorate blob client * add policy * added policy to the blob * Update root README.md Updating the Contributing section according to the guidelines * get rid of util and propagate context properly. * get rid of functions that do not do a network call * propagate context * Revert "propagate context" This reverts commit 1bd07ed. * New paging to storage preview2 branch (with async) (#6493) * Update root README.md Updating the Contributing section according to the guidelines * Update CODEOWNERS * more code owners * add policy and decorators for secrets (#6453) * Smoke test for Python (#6412) * Smoke Test Sample for Track 2 libraries Smoke Test for Indentity, Key Vault Secrets, Storage Blobs, Event Hubs and Cosmos DB * simpleQuery method added * Method's names updated * Create README.md * Update README.md * Commented lines deleted * README.md moved to correct folder * Create requirements.txt * Update README.md * Update README.md * Imports changed * Use of literals instead of append * Database Name variable to class level. * Use of Pythonic with statements * Update requirements.txt * Revert "Update requirements.txt" This reverts commit 4b79c6a. * Revert "Use of Pythonic with statements" This reverts commit 81adc4c. * Revert "Revert "Use of Pythonic with statements"" This reverts commit 27b2a2d. * requiriments.txt encoded as a txt file * requirements.txt as text file * Misspelling in "Key concepts" * Update .docsettings.yml to match the tittle of Smoke Test * Went trought Suyog comments * Revert "Went trought Suyog comments" This reverts commit 35c6223. * Gone trought Suyog comments * use of snake case in file names * Paging v2 (#6420) * First version of paging * Clean sync paging * Async paging * Name AsyncList without underscore * MyPy happyness * pylint * black * PageIterator as conf * Simplify ItemPaged * Use chain to link iterable * Make Paging more open * Docstring * pylint / mypy / black * Regenerate KV with latest Autorest + Paging branch (#6479) * Regenerate KV with latest Autorest + Paging branch * Remove unused import in tests * Make the async adapter more generic * Regeneration after operation mixin fix in Autorest * COmmit changes forgotten in #6418 * Fixing syntax for 2.7 * Move Storage Queue to new Paging (#6447) * Move Storage Queue to new Paging * Working Storage queue on new paging * Fix types * First shot of blob tests update * Fixing blob tests * File to new paging * Doc fix * Feedbacks from @annatisch * Fix last @annatisch comment * Simplify paging contract * Fix storage queue after new paging contract * Fixed incorrect auto-merge * Fix docstring * Port aio to new async paging * Adapt async tests * Missing by_page * Fix async tests * Fix continuation token bug * Support for aiohttp records from vcrpy (#6552) * Async recording for Storage (#6560) * Aiohttp is the only default for async clients (#6561) * seed tests.yml (#6645) * [storage] Blob async APIs (#6489) * Updated blobs shared code * Started blob refactor * Refactoring upload * Updated shared code * Started fixing tests * Refactored sync blobs * Added blob async APIs * Some fixes * Append blob async tests * blob access async tests * Blob client async tests * encryption async tests * Patch for azure core exception * blob retry async * Retry async tests * Get blob async tests * Bug fix for clear page operation * More async tests + upload fix * Merged blobs * Updated Files shared code * Updated queue shared code * async tests pass except 2 common blob tests * adds async paging to blobs and some async tests (not all pass) * initial commit * block_blob_tests * page blob tests * fix for special chars, some tests, and recordings * add to shared storage and fix import * adding more tests/recordings * more tests/recordings * rerecord tests, fix imports * fix import again * blacklist azure-servicemanagement-legacy * get CI to run * rerecord all async tests * testing * add variable indirection for storage live tests. this is a temporary measure to create live test pipelines for this particular framework * newline * print envar * remove testing * adjust pypy testing * Storage Recordings For Queues and Files (#6629) * initial commit * recordings * fix all tests * update queue recordings * file recordings * queue recordings * fix for encoding tests * small fix * change conftest * remove recordings * oops * new recordings * some fixes * more fixes * and more fixes * and one more fix * allowing specific project targeting for storage livetests * allowing BuildTargetingString to flow through for templates following 'archtype-sdk-tests' * passing service directory to setup task * Support for Live storage tests (#6663) * play * playing * some prints * some tweaks * try-345 * fix * tweak for queue and file * remove prints * [storage] Preview2 updates (#6658) * Updated change logs * Bumped versions * Some test fixes * Fix for file handle next marker * Fixed error in release notes * Updated core version * Updated azure-core shared dep * Updated release notes * fix test (#6674) * Fix for queue models (#6681) * [storage] Readme tweaks (#6697) * Fix for queue models * Docs cleanups * Some final tweaks (#6687) * Some final tweaks * oops * comments * little better * Pylint * [storage] Better async import error message (#6700) * Fix for queue models * Docs cleanups * Better async import error
Azure · Aug 6, 2019 · 7202249 · 7202249
1 parent 8de236a
commit 7202249
Show file tree

Hide file tree

Showing 1,003 changed files with 359,420 additions and 58,696 deletions.
diff --git a/.gitignore b/.gitignore
@@ -78,3 +78,4 @@ code_reports
 sdk/storage/azure-storage-blob/tests/settings_real.py
 sdk/storage/azure-storage-queue/tests/settings_real.py
 sdk/storage/azure-storage-file/tests/settings_real.py
+*.code-workspace
diff --git a/eng/pipelines/templates/jobs/archetype-sdk-tests.yml b/eng/pipelines/templates/jobs/archetype-sdk-tests.yml
@@ -3,6 +3,7 @@ parameters:
   EnvVars: {}
   MaxParallel: 0
   BeforeTestSteps: []
+  BuildTargetingString: 'azure-*'
   Matrix:
     Linux_Python35:
       OSName: 'Linux'
@@ -34,6 +35,7 @@ jobs:
     steps:
       - template: ../steps/build-test.yml
         parameters:
+          BuildTargetingString: ${{ parameters.BuildTargetingString }}
           ServiceDirectory: ${{ parameters.ServiceDirectory }}
           AdditionalTestArgs: '--disablecov'
           EnvVars: ${{ parameters.EnvVars }}

diff --git a/eng/pipelines/templates/steps/build-test.yml b/eng/pipelines/templates/steps/build-test.yml
@@ -37,7 +37,7 @@ steps:
     displayName: 'Install Packages'
     inputs:
       scriptPath: 'scripts/devops_tasks/setup_execute_tests.py'
-      arguments: '"${{ parameters.BuildTargetingString }}" --runtype=setup'
+      arguments: '"${{ parameters.BuildTargetingString }}" --runtype=setup --service="${{ parameters.ServiceDirectory }}"'
     env: ${{ parameters.EnvVars }}
 
   - task: PythonScript@0

diff --git a/scripts/devops_tasks/common_tasks.py b/scripts/devops_tasks/common_tasks.py
@@ -15,7 +15,7 @@
 import sys
 
 DEFAULT_BUILD_PACKAGES = ['azure-keyvault', 'azure-servicebus']
-OMITTED_CI_PACKAGES = ['azure-mgmt-documentdb']
+OMITTED_CI_PACKAGES = ['azure-mgmt-documentdb', 'azure-servicemanagement-legacy']
 
 # this function is where a glob string gets translated to a list of packages
 # It is called by both BUILD (package) and TEST. In the future, this function will be the central location
@@ -40,12 +40,13 @@ def process_glob_string(glob_string, target_root_dir):
         return collected_directories
     # however, if there are multiple packages being built, we should honor the omission list and NOT build the omitted
     # packages
-    else :
+    else:
         return remove_omitted_packages(collected_directories)
 
 def remove_omitted_packages(collected_directories):
-    return [package_dir for package_dir in collected_directories if os.path.basename(package_dir) not in OMITTED_CI_PACKAGES]
-
+    return [package_dir for package_dir in collected_directories if
+            os.path.basename(package_dir) not in OMITTED_CI_PACKAGES]
+
 
 def run_check_call(command_array, working_directory, acceptable_return_codes = [], run_as_shell = False):
     try:

diff --git a/sdk/core/azure-common/testutils/common_recordingtestcase.py b/sdk/core/azure-common/testutils/common_recordingtestcase.py
@@ -66,7 +66,7 @@ def init_test_mode(self):
             self.test_mode = test_settings['mode']
         except:
             pass
-        
+
         if getattr(self, 'test_mode', None) is None:
             self.test_mode = TestMode.playback
 
@@ -134,11 +134,18 @@ def _scrub_sensitive_response_info(self, response):
             response = decode_response(response)
             headers = response.get('headers')
             if headers:
+                def internal_scrub(key, val):
+                    if key.lower() == 'retry-after':
+                        return '0'
+                    return self._scrub(val)
+
                 for name, val in headers.items():
-                    for i, e in enumerate(val):
-                        val[i] = self._scrub(e)
-                    if name.lower() == 'retry-after':
-                        val[:] = ['0']
+                    if isinstance(val, list):
+                        for i, e in enumerate(val):
+                            val[i] = internal_scrub(name, e)
+                    else:
+                        headers[name] = internal_scrub(name, val)
+
             body = response.get('body')
             if body:
                 body_str = body.get('string')

diff --git a/sdk/storage/azure-storage-blob/HISTORY.md b/sdk/storage/azure-storage-blob/HISTORY.md
@@ -1,5 +1,37 @@
 # Change Log azure-storage-blob
 
+
+## Version 12.0.0b2:
+
+**Breaking changes**
+- Renamed `copy_blob_from_url` to `start_copy_from_url` and changed behaviour to return a dictionary of copy properties rather than a polling object. Status of the copy operation can be retrieved with the `get_blob_properties` operation.
+- Added `abort_copy` operation to the `BlobClient` class. This replaces the previous abort operation on the copy status polling operation.
+- The behavior of listing operations has been modified:
+    - The previous `marker` parameter has been removed.
+    - The iterable response object now supports a `by_page` function that will return a secondary iterator of batches of results. This function supports a `continuation_token` parameter to replace the previous `marker` parameter.
+- Some parameters have become keyword only, rather than positional. Some examples include:
+    - `timeout`
+    - `lease`
+    - `encoding`
+    - Modification conditions, e.g. `if_modified_since`, `if_match` , `maxsize_condition`, etc
+
+**New features**
+- Added async APIs to subnamespace `azure.storage.blob.aio`.
+- Distributed tracing framework OpenCensus is now supported.
+
+**Dependency updates**
+- Adopted [azure-core](https://pypi.org/project/azure-core/) 1.0.0b2
+  - If you later want to revert to azure-storage-blob 12.0.0b1, or another Azure SDK
+  library requiring azure-core 1.0.0b1, you must explicitly install azure-core
+  1.0.0b1 as well. For example:
+
+  `pip install azure-core==1.0.0b1 azure-storage-blob==12.0.0b1`
+
+**Fixes and improvements**
+- Fix for SAS URL encoding (#6500)
+- General refactor of duplicate and shared code.
+
+
 ## Version 12.0.0b1:
 
 Version 12.0.0b1 is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Storage Blobs. For more information about this, and preview releases of other Azure SDK libraries, please visit

diff --git a/sdk/storage/azure-storage-blob/README.md b/sdk/storage/azure-storage-blob/README.md
@@ -133,7 +133,8 @@ from azure.storage.blob import BlobClient
 blob = BlobClient.from_connection_string("my_connection_string", container="mycontainer", blob="my_blob")
 
 with open("./BlockDestination.txt", "wb") as my_blob:
-    my_blob.writelines(blob.download_blob())
+    blob_data = blob.download_blob()
+    my_blob.writelines(blob_data.content_as_bytes())
 ```
 
 ### Enumerating blobs

diff --git a/sdk/storage/azure-storage-blob/azure/storage/blob/__init__.py b/sdk/storage/azure-storage-blob/azure/storage/blob/__init__.py
@@ -12,15 +12,14 @@
 from .container_client import ContainerClient
 from .blob_service_client import BlobServiceClient
 from .lease import LeaseClient
-from .polling import CopyStatusPoller
 from ._shared.policies import ExponentialRetry, LinearRetry, NoRetry
+from ._shared.downloads import StorageStreamDownloader
 from ._shared.models import(
     LocationMode,
     ResourceTypes,
     AccountPermissions,
     StorageErrorCode
 )
-from ._blob_utils import StorageStreamDownloader
 from .models import (
     BlobType,
     BlockState,
@@ -87,7 +86,6 @@
     'BlobPermissions',
     'ResourceTypes',
     'AccountPermissions',
-    'CopyStatusPoller',
     'StorageStreamDownloader',
 ]