Skip to content

Commit

Permalink
Swagger to sdk - new features (Azure#240)
Browse files Browse the repository at this point in the history 
* Add support for Autorest version choices

* Configure the language using the 'language' meta parameter

* Documentation

* Log update

* Add autorest_options conf options
  • Loading branch information
@lmazuel @amarzavery
lmazuel authored and amarzavery committed Apr 16, 2016
1 parent af9b9c5 commit 5576409
Show file tree
Hide file tree
Showing 3 changed files with 198 additions and 27 deletions.
111 changes: 111 additions & 0 deletions SwaggerToSdk/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
The SwaggerToSDK script is the automation script launched at each commit on any Swagger files to provide:
- Testing against SDK language
- Automatic PR on each language SDK

This works is still in progress and move fast. We'll do our best to keep this page up-to-date.

# Configuration of the script

```bash
usage: SwaggerToSdk.py [-h] [--rest-folder RESTAPI_GIT_FOLDER]
[--pr-repo-id PR_REPO_ID] [--message MESSAGE]
[--base-branch BASE_BRANCH] [--branch BRANCH]
[--config CONFIG_PATH] [-v] [--debug]
sdk_git_id

Build SDK using Autorest and push to Github. The GH_TOKEN environment variable
needs to be set.

positional arguments:
sdk_git_id The SDK Github id. If a simple string, consider it
belongs to the GH_TOKEN owner repo. Otherwise, you can
use the syntax username/repoid

optional arguments:
-h, --help show this help message and exit
--rest-folder RESTAPI_GIT_FOLDER, -r RESTAPI_GIT_FOLDER
Rest API git folder. [default: .]
--pr-repo-id PR_REPO_ID
PR repo id. [default: None]
--message MESSAGE, -m MESSAGE
Force commit message. {hexsha} will be the current
REST SHA1 [default: Generated from {hexsha}]
--base-branch BASE_BRANCH, -o BASE_BRANCH
The base branch from where create the new branch.
[default: master]
--branch BRANCH, -b BRANCH
The SDK branch to commit. Default if not Travis:
autorest. If Travis is detected, see epilog for
details
--config CONFIG_PATH, -c CONFIG_PATH
The JSON configuration format path [default:
sdk_autogen_config.json]
-v, --verbose Verbosity in INFO mode
--debug Verbosity in DEBUG mode

If Travis is detected, --branch is setted by default to "RestAPI-PR{number}"
if triggered by a PR, "RestAPI-{branch}" otherwise
```
# Configuration file swagger_to_sdk.json
This is a configuration which MUST be at the root of the repository you wants to generate.
```json
{
"meta": {
"version":"0.1.0",
"language": "Python",
"autorest": "latest",
"autorest_options": {
"ft": 2,
"AddCredentials": true
}
},
"data": {
"arm-authorization/2015-07-01/swagger/authorization.json": {
"autorest_options": {
"Namespace" : "azure.mgmt.authorization"
},
"output_dir": "azure-mgmt-authorization/azure/mgmt/authorization"
},
}
}
```
## Meta
### version
The version is not read yet, since this works is in progress. This could be use in the future to enable breaking changes detection and handling. Don't rely on this one for now
### language
The language parameter configure the language you want Autorest to generate. This could be (case-sensitive):
- Python
- CSharp
- Java
- NodeJS
- Ruby
This will trigger the Azure.<language> autorest code generator. Note that you can override this behaviour by specifying the "CodeGenerator" option in any "autorest_options" field.
### autorest
This the version to use from Autorest. Could be "latest" to download the latest nightly build. Could be a string like '0.16.0-Nightly20160410' to download a specific version.
If node is not present, consider "latest".
## autorest_options
An optional dictionary of options you want to pass to Autorest. This will be passed in any call, but can be override by "autorest_options" in each data.
You can override the default `Azure.<language>` CodeGenerator parameter here (do NOT use "g" but "CodeGenerator").
Note that you CAN'T override "-i/-Input" and "-o/-Output" which are filled contextually.
## Data
It's a dict where keys are swagger path. Value are:
### autorest_options
A dictionary of options you want to pass to Autorest. This will override parameter from "autorest_options" in "meta" node.
You can override the default `Azure.<language>` CodeGenerator parameter here (do NOT use "g" but "CodeGenerator").
Note that you CAN'T override "-i/-Input" and "-o/-Output" which are filled contextually.
### output_dir
This is the folder in your SDK repository where you want to put the generated files.
77 changes: 51 additions & 26 deletions SwaggerToSdk/SwaggerToSdk.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@

_LOGGER = logging.getLogger(__name__)

LATEST_AUTOREST_DOWNLOAD_LINK = "https://www.myget.org/F/autorest/api/v2/package/autorest/"
LATEST_TAG = 'latest'
AUTOREST_BASE_DOWNLOAD_LINK = "https://www.myget.org/F/autorest/api/v2/package/autorest/"

CONFIG_FILE = 'swagger_to_sdk_config.json'
NEEDS_MONO = platform.system() != 'Windows'
Expand All @@ -39,27 +40,57 @@ def read_config(sdk_git_folder):
return json.loads(config_fd.read())


def download_install_autorest(output_dir):
def download_install_autorest(output_dir, autorest_version=LATEST_TAG):
"""Download and install Autorest in the given folder"""
_LOGGER.info("Download Autorest from: %s", LATEST_AUTOREST_DOWNLOAD_LINK)
downloaded_package = requests.get(LATEST_AUTOREST_DOWNLOAD_LINK)
download_link = AUTOREST_BASE_DOWNLOAD_LINK
if autorest_version != LATEST_TAG:
download_link += autorest_version

_LOGGER.info("Download Autorest from: %s", download_link)
try:
downloaded_package = requests.get(download_link)
except:
msg = "Unable to download Autorest for '{}', " \
"please check this link and/or version tag: {}".format(
autorest_version,
download_link
)
_LOGGER.critical(msg)
raise ValueError(msg)
if downloaded_package.status_code != 200:
raise ValueError(downloaded_package.content.decode())
_LOGGER.info("Downloaded")
with zipfile.ZipFile(BytesIO(downloaded_package.content)) as autorest_package:
autorest_package.extractall(output_dir)
return os.path.join(output_dir, 'tools', 'AutoRest.exe')

def build_autorest_options(language, global_autorest_conf=None, autorest_conf=None):
"""Build the string of the Autorest options"""
if global_autorest_conf is None:
global_autorest_conf = {}
if autorest_conf is None:
autorest_conf = {}

local_conf = dict(global_autorest_conf)
local_conf.update(autorest_conf)
if "CodeGenerator" not in local_conf:
local_conf["CodeGenerator"] = "Azure.{}".format(language)

def generate_code(language, swagger_file, output_dir, autorest_exe_path, autorest_conf=''):
sorted_keys = sorted(list(local_conf.keys())) # To be honest, just to help for tests...
return " ".join("-{} {}".format(key, str(local_conf[key])) for key in sorted_keys)

def generate_code(language, swagger_file, output_dir, autorest_exe_path, global_autorest_conf=None, autorest_conf=None):
"""Call the Autorest process with the given parameters"""
if NEEDS_MONO:
autorest_exe_path = 'mono ' + autorest_exe_path
cmd_line = "{} -AddCredentials true -ft 2 -g Azure.{} " \
"-i {} -o {} {}"

autorest_options = build_autorest_options(language, global_autorest_conf, autorest_conf=None)

cmd_line = "{} -i {} -o {} {}"
cmd_line = cmd_line.format(autorest_exe_path,
language,
swagger_file,
output_dir,
autorest_conf)
autorest_options)
_LOGGER.debug("Autorest cmd line:\n%s", cmd_line)

try:
Expand Down Expand Up @@ -93,10 +124,8 @@ def update(language, generated_folder, destination_folder):
"""Update data from generated to final folder"""
if language == 'Python':
update_python(generated_folder, destination_folder)
elif language == 'NodeJS':
update_node(generated_folder, destination_folder)
else:
raise ValueError('Unknow language: {}'.format(language))
update_generic(generated_folder, destination_folder)

def update_python(generated_folder, destination_folder):
"""Update data from generated to final folder, Python version"""
Expand All @@ -115,8 +144,9 @@ def update_python(generated_folder, destination_folder):
client_generated_path.replace(destination_folder)


def update_node(generated_folder, destination_folder):
"""Update data from generated to final folder, Python version"""
def update_generic(generated_folder, destination_folder):
"""Update data from generated to final folder.
Generic version which just copy the files"""
client_generated_path = Path(generated_folder)
shutil.rmtree(destination_folder)
client_generated_path.replace(destination_folder)
Expand Down Expand Up @@ -302,7 +332,7 @@ def get_full_sdk_id(gh_token, sdk_git_id):

def clone_to_path(gh_token, temp_dir, sdk_git_id):
"""Clone the given repo_id to the 'sdk' folder in given temp_dir"""
_LOGGER.info("Clone SDK repository")
_LOGGER.info("Clone SDK repository %s", sdk_git_id)

credentials_part = ''
if gh_token:
Expand All @@ -318,7 +348,6 @@ def clone_to_path(gh_token, temp_dir, sdk_git_id):
credentials=credentials_part,
sdk_git_id=sdk_git_id
)
_LOGGER.debug("Url: %s", https_authenticated_url)
sdk_path = os.path.join(temp_dir, 'sdk')
Repo.clone_from(https_authenticated_url, sdk_path)
_LOGGER.info("Clone success")
Expand Down Expand Up @@ -349,14 +378,6 @@ def build_libraries(gh_token, restapi_git_folder, sdk_git_id, pr_repo_id, messag
branch_name = compute_branch_name(branch_name)
_LOGGER.info('Destination branch for generated code is %s', branch_name)

# FIXME to be refine
if 'python' in sdk_git_id.lower():
language = 'Python'
elif 'node' in sdk_git_id.lower():
language = 'NodeJS'
else:
raise ValueError('Unable to determine language')

with tempfile.TemporaryDirectory() as temp_dir, \
manage_sdk_folder(gh_token, temp_dir, sdk_git_id) as sdk_folder:

Expand All @@ -371,12 +392,16 @@ def build_libraries(gh_token, restapi_git_folder, sdk_git_id, pr_repo_id, messag
sync_fork(gh_token, sdk_git_id, sdk_repo)
config = read_config(sdk_repo.working_tree_dir)

meta_conf = config["meta"]
language = meta_conf["language"]
hexsha = get_swagger_hexsha(restapi_git_folder)
global_autorest_conf = meta_conf["autorest_options"] if "autorest_options" in meta_conf else {}
autorest_version = meta_conf["autorest"] if "autorest" in meta_conf else LATEST_TAG

autorest_temp_dir = os.path.join(temp_dir, 'autorest')
os.mkdir(autorest_temp_dir)

autorest_exe_path = download_install_autorest(autorest_temp_dir)
autorest_exe_path = download_install_autorest(autorest_temp_dir, autorest_version)

for file, conf in config["data"].items():
_LOGGER.info("Working on %s", file)
Expand All @@ -400,7 +425,7 @@ def build_libraries(gh_token, restapi_git_folder, sdk_git_id, pr_repo_id, messag
generated_path = os.path.join(temp_dir, os.path.basename(file))
generate_code(language,
swagger_file, generated_path,
autorest_exe_path, autorest_conf)
autorest_exe_path, global_autorest_conf, autorest_conf)
update(language, generated_path, dest_folder)

if gh_token:
Expand Down
37 changes: 36 additions & 1 deletion SwaggerToSdk/testing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ def tearDown(self):
if key.startswith('TRAVIS'):
del os.environ[key]

@unittest.skip
def test_get_pr_from_travis_commit_sha(self):
os.environ['TRAVIS_REPO_SLUG'] = 'Azure/azure-sdk-for-python'
os.environ['TRAVIS_COMMIT'] = '497955507bc152c444bd1785f34cafefc7e4e8d9'
Expand All @@ -27,5 +28,39 @@ def test_get_pr_from_travis_commit_sha(self):
pr_obj = get_pr_from_travis_commit_sha(GH_TOKEN)
self.assertIsNone(pr_obj)

@unittest.skip
def test_download_autorest(self):
with tempfile.TemporaryDirectory() as temp_dir:
exe_path = download_install_autorest(temp_dir)
self.assertTrue(exe_path.lower().endswith("autorest.exe"))

with tempfile.TemporaryDirectory() as temp_dir:
exe_path = download_install_autorest(temp_dir, "0.16.0-Nightly20160410")
self.assertTrue(exe_path.lower().endswith("autorest.exe"))

with tempfile.TemporaryDirectory() as temp_dir:
with self.assertRaises(ValueError):
exe_path = download_install_autorest(temp_dir, "0.16.0-FakePackage")

def test_build_autorest_options(self):
line = build_autorest_options("Python", {"A": "value"}, {"B": "value"})
self.assertEqual(line, "-A value -B value -CodeGenerator Azure.Python")

line = build_autorest_options("Python", {"A": "value"}, {"A": "newvalue"})
self.assertEqual(line, "-A newvalue -CodeGenerator Azure.Python")

line = build_autorest_options("Python", {"CodeGenerator": "NodeJS"}, {})
self.assertEqual(line, "-CodeGenerator NodeJS")

line = build_autorest_options("Python", {"CodeGenerator": "NodeJS"}, {"CodeGenerator": "CSharp"})
self.assertEqual(line, "-CodeGenerator CSharp")

line = build_autorest_options("Python", {}, {})
self.assertEqual(line, "-CodeGenerator Azure.Python")

line = build_autorest_options("Python", {"A": 12, "B": True})
self.assertEqual(line, "-A 12 -B True -CodeGenerator Azure.Python")


if __name__ == '__main__':
unittest.main()
unittest.main()

0 comments on commit 5576409

Please sign in to comment.