Automatic Generation of Swagger Documentation in Types #10933

andronat · 2015-07-08T17:07:39Z

This is a proposal on how we can handle and export our documentation from api/*/types.go to Swagger API. Refs #3060

The basic idea

Export a description for the structs themselves
We have a lot of interesting information as comments in api/types.go, api/v1/types.go, api/v1beta3/types.go that we might want to export in Swagger API.
We make sure that we don't want to maintain documentation in multiple places.

Problems

Ideally we would like go-restful to be able to parse our source files and export struct and fields comments in the Swagger API. Unfortunately this is not possible (or just very hard) as Swagger's JSON is builded dynamically on runtime and parsing the source files requires 1) having the files in the deployment environment 2) pass the file paths to go-restful.

Currently go-restful is able to export field descriptions of a struct by parsing struct tags (using the description tag) and lately, struct description is also supported through the modelDescription tag. The issue with this method is how can we maintain a source-reader-friendly documentation in our files and keep swagger-friendly tags up to date (these are 1 line long descriptions).

Solution

In this PR: emicklei/go-restful#215 we suggest an alternative way of keeping struct documentation through special methods attached on structs.

This PR is about creating a generator that parses the documentation from the Type files (this is done by using go/doc and go/ast) and automatically generates the aforementioned special documentation methods for each struct.

A smaller issue on TODOs and various other comments

We want this parser to ignore TODO lists and various other comments in our source code.

The suggested solution is:

Ignore one line TODO comments
Use --- in a line and everything after it will be not placed in swagger descriptions

TODO:

~~wait for Models can generate documentation through a struct method emicklei/go-restful#215 to be merged~~
~~ignore TODOs (and maybe also other information) -> Use --- and ignore everything after it~~
~~polish output (keep paragraphs, keep indentation, keep newline chars in indented lines)~~
~~ignore json",inline" tags~~
~~Add a comment here that thoroughly explains what this file is, what it's for, how it's generated, etc~~
~~Generator should not print keys for empty documentation (empty values overwrite struct tags)~~
~~Refactor verify-descriptions~~
~~Fixed missing ptr description emicklei/go-restful#219 needs merging~~
~~fix description parsing on recursion emicklei/go-restful#222 needs merging~~
~~change QueryParameter description in api_installer.go~~
~~Move all description tags into comments~~
~~Depends on updated go-restful #11406~~

k8s-bot · 2015-07-08T17:25:54Z

GCE e2e build/test passed for commit 36055bd3287b31233611820cb572e9817d6716f8.

ghodss · 2015-07-08T19:30:41Z

pkg/api/types_swagger_doc.go

+
+package api
+
+// AUTO-GENERATED FUNCTIONS START HERE


Add a comment here that thoroughly explains what this file is, what it's for, how it's generated, etc.

k8s-bot · 2015-07-09T18:34:23Z

GCE e2e build/test passed for commit afb8f124a8a8c2e1cb723ea0100ea5649f74d333.

nikhiljindal · 2015-07-09T20:28:22Z

pkg/api/types_swagger_doc.go

+	return map[string]string{
+		"":             "",
+		"name":         "Each container in a pod must have a unique name.",
+		"state":        "TODO(dchen1107): Should we rename PodStatus to a more generic name or have a separate states defined for container?",


We will need a way to keep these TODOs out of here :)

Yep, I have it already in my todo list on my PR. Working on it

nikhiljindal · 2015-07-09T20:37:42Z

This is exciting.
@thockin is going to be very happy to see this.
We have discussed having such a pass over comments in types.go, before they are passed to go-restful.

ghodss · 2015-07-09T23:52:21Z

@bgrant0607 This is the method @andronat and I came up with to retrieve API documentation out of the comments and make it available to swagger, as per our earlier convo.

k8s-bot · 2015-07-10T19:09:54Z

GCE e2e build/test passed for commit 6f3d3828f62c0b4e37b9759006dddcdbf165ef9a.

bgrant0607 · 2015-07-14T21:28:51Z

I am opposed to any solution that moves documentation out of our interface-definition files, currently api/*/types.go.

andronat · 2015-07-14T21:32:18Z

@bgrant0607 it doesn't move any documentation from api/*/types.go. On the contrary we found a way to keep the documentation only in the comments of the api/*/types.go files and not comments + struct tags that currently is.

ghodss · 2015-07-14T21:33:18Z

@bgrant0607 It's clear that this PR does not move documentation out of the current interface-definition files, which is the express purpose of this design, yes?

bgrant0607 · 2015-07-14T21:56:00Z

Sorry, I don't have time to look at this in detail right now.

How would this support additional properties, such as those in the following PR? Or would those remain in tags?
emicklei/go-restful#198

andronat · 2015-07-14T22:27:14Z

The current PR is only for the descriptions. Most definitely, we could make the appropriate changes to support also additional properties but I am not sure if we really want to do that.

The reason that makes me reluctant for something like this (supporting model attributes as comments) is that when a user/developer reads the source code he/she expects to see a description as a comment but "functional properties" as part of the code (e.g. max number, min number etc). I don't know if that makes any sense to you also :/ (I don't have a strong opinion though on that)

thockin · 2015-07-15T04:39:58Z

I like the idea. I loathe the triple-line-wrapped description tags - I think they discourage good descriptions. Moving all of that into comments is a WAY better answer. Let's define a set of standard (for us) comment structures and enforce that every field and every struct in our API is documented fully.

k8s-bot · 2015-07-15T16:16:59Z

GCE e2e build/test failed for commit afff8f4cb82b00c555a42999c4bc3c91e9020bf7.

k8s-bot · 2015-07-15T16:28:12Z

GCE e2e build/test passed for commit 9823d915f4116e11e8b581c9636bb3228781a353.

k8s-bot · 2015-07-18T21:13:40Z

GCE e2e build/test passed for commit 443527908004385f71e4b8a4785ae87f5ef3fbd9.

k8s-bot · 2015-07-19T22:07:36Z

GCE e2e build/test passed for commit eef905a1b6493111ae099473dde9ef5c8d674cd5.

andronat · 2015-07-19T22:07:50Z

Can someone please take a look on that f51274ed591906371cee4698747aca2f26b7bfc7. I am not 100% sure that I am doing it correctly and it also takes a lot of time. Some input would be appreciated.

/cc @ghodss @bgrant0607 @thockin

bgrant0607 · 2015-07-20T16:02:02Z

Have you thought about how to implement verify-descriptions?

andronat · 2015-07-20T19:01:23Z

Hmm, I missed that part.

How about using my generator's code to find if there are missing docs from fields and structs? The generator is parsing the AST of the source code and returns pairs of (field, doc). I can count the number of returned pairs with the number of fields and structs and return OK if none is missing.

bgrant0607 · 2015-07-20T20:31:26Z

It would need to report which fields lack documentation

andronat · 2015-07-21T15:28:26Z

How about this?

Also, the command exits with the number of the missing doc definitions.

Missing documentation for the struct itself: ComponentStatusList
In struct: ComponentStatusList, field documentation is missing: metadata
In struct: ComponentStatusList, field documentation is missing: items
Missing documentation for the struct itself: ConditionStatus
In struct: Container, field documentation is missing: ports
In struct: Container, field documentation is missing: env

# echo $?
6

caesarxuchao · 2015-08-21T22:01:10Z

hack/verify-generated-swagger-docs.sh

+source "${KUBE_ROOT}/hack/lib/init.sh"
+
+kube::golang::setup_env
+"${KUBE_ROOT}/hack/build-go.sh" cmd/genswaggertypedocs


Similarly could you split this script as well? Thank you.

andronat · 2015-08-21T23:27:52Z

fixed omitempty. working on other issues.

k8s-bot · 2015-08-21T23:52:32Z

GCE e2e build/test passed for commit 6ed3ee3d11f178da1c1ff0f630127451e7758e3d.

k8s-bot · 2015-08-21T23:52:56Z

GCE e2e build/test failed for commit 9cb4c486b2aa8e2a04945810c85d3fae6e12a4d5.

caesarxuchao · 2015-08-21T23:57:37Z

Thank you @andronat. I don't have extra comments. Ping me when you address all of the comments.

andronat · 2015-08-22T00:00:41Z

ok just done with the after-build scripts

andronat · 2015-08-22T00:01:39Z

Tell me @caesarxuchao when to start squashing

bgrant0607 · 2015-08-22T00:15:41Z

Go ahead and squash, @andronat. It's already not easy to determine what you changed relative to the previous push.

andronat · 2015-08-22T00:18:23Z

Yea sorry for that. I should had put everything as a new commit at the end :/

k8s-bot · 2015-08-22T00:22:08Z

GCE e2e build/test failed for commit 04ed44abad7da107448e87e46a05e92a8029e6cd.

caesarxuchao · 2015-08-22T00:49:13Z

@andronat, sorry just saw your message. Please add back the "omitempty" for nodeport. #10933 (comment)

k8s-bot · 2015-08-22T00:52:30Z

GCE e2e build/test passed for commit a44b3b5d961efca18d45720e52e9254efb46d479.

andronat · 2015-08-22T01:21:41Z

found the problem: in test-cmd.sh we are grep-ing for sentences in the old documentation. fixing it right now. (also fixed the omitempty)

k8s-bot · 2015-08-22T01:46:05Z

GCE e2e build/test passed for commit 1074784.

andronat · 2015-08-22T01:47:31Z

It is all green!

bgrant0607 · 2015-08-22T04:08:07Z

Thanks for all the work on this. LGTM.

caesarxuchao · 2015-08-22T04:26:28Z

Thank you!

andronat · 2015-08-22T14:39:33Z

Thank you too for your time and effort!

Automatic Generation of Swagger Documentation in Types

googlebot added the cla: yes label Jul 8, 2015

ghodss reviewed Jul 8, 2015
View reviewed changes

nikhiljindal reviewed Jul 9, 2015
View reviewed changes

andronat mentioned this pull request Jul 14, 2015

Attach documentation to API types and/or resources #5331

Closed

bgrant0607 self-assigned this Jul 14, 2015

This was referenced Jul 14, 2015

Generate static, object-focused documentation of types #10966

Closed

[WIP]: Add a command to generate docs for types defined in pkg/api/v*/types.go #11253

Closed

andronat mentioned this pull request Jul 16, 2015

updated go-restful #11406

Merged

caesarxuchao reviewed Aug 21, 2015
View reviewed changes

andronat added 3 commits August 22, 2015 02:26

swagger type documentation generator

9f21ae2

Added SwaggerDoc() in api-server

ff2fcd4

hack and hooks scripts for generating swagger docs

e27a76a

move type documentation from tags to comments

1074784

bgrant0607 mentioned this pull request Aug 22, 2015

Adding LastTransitionTime to PodCondition #12894

Merged

bgrant0607 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2015

bgrant0607 added the ok-to-merge label Aug 22, 2015

nikhiljindal added a commit that referenced this pull request Aug 24, 2015

Merge pull request #10933 from andronat/swagger_auto_type_docs

a09425d

Automatic Generation of Swagger Documentation in Types

nikhiljindal merged commit a09425d into kubernetes:master Aug 24, 2015

andronat deleted the swagger_auto_type_docs branch August 25, 2015 00:55

caesarxuchao mentioned this pull request Sep 2, 2015

Genereate types_swagger_doc_generated.go for expapi #13487

Merged

Automatic Generation of Swagger Documentation in Types #10933

Automatic Generation of Swagger Documentation in Types #10933

Conversation

andronat commented Jul 8, 2015

The basic idea

Problems

Solution

A smaller issue on TODOs and various other comments

k8s-bot commented Jul 8, 2015

ghodss Jul 8, 2015

Choose a reason for hiding this comment

k8s-bot commented Jul 9, 2015

nikhiljindal Jul 9, 2015

Choose a reason for hiding this comment

andronat Jul 9, 2015

Choose a reason for hiding this comment

nikhiljindal commented Jul 9, 2015

ghodss commented Jul 9, 2015

k8s-bot commented Jul 10, 2015

bgrant0607 commented Jul 14, 2015

andronat commented Jul 14, 2015

ghodss commented Jul 14, 2015

bgrant0607 commented Jul 14, 2015

andronat commented Jul 14, 2015

thockin commented Jul 15, 2015

k8s-bot commented Jul 15, 2015

k8s-bot commented Jul 15, 2015

k8s-bot commented Jul 18, 2015

k8s-bot commented Jul 19, 2015

andronat commented Jul 19, 2015

bgrant0607 commented Jul 20, 2015

andronat commented Jul 20, 2015

bgrant0607 commented Jul 20, 2015

andronat commented Jul 21, 2015

caesarxuchao Aug 21, 2015

Choose a reason for hiding this comment

andronat commented Aug 21, 2015

k8s-bot commented Aug 21, 2015

k8s-bot commented Aug 21, 2015

caesarxuchao commented Aug 21, 2015

andronat commented Aug 22, 2015

andronat commented Aug 22, 2015

bgrant0607 commented Aug 22, 2015

andronat commented Aug 22, 2015

k8s-bot commented Aug 22, 2015

caesarxuchao commented Aug 22, 2015

k8s-bot commented Aug 22, 2015

andronat commented Aug 22, 2015

k8s-bot commented Aug 22, 2015

andronat commented Aug 22, 2015

bgrant0607 commented Aug 22, 2015

caesarxuchao commented Aug 22, 2015

andronat commented Aug 22, 2015