@smoser @TheRealFalcon @OddBloke - I have gotten this far with this PR. Need your help to solve the test failures.

I haven't done a full review because this PR is WIP, but as I was investigating the test issue, I noticed a couple things around networking.

You shouldn't need to provide a separate networkd renderer, as that is already provided by the netplan renderer. Is there something about the netplan renderer that doesn't work for your case? Additionally, you don't need to support both the v1 and the v2 config. If the v2 config works for you, then go with that.

@TheRealFalcon thanks a lot for the input. My tests are passing now.

About adding a separate network renderer:

1. netplan doesn't work for us because we don't have netplan package in our system, ours is a minimalist system and we add packages only when necessary.

2. We consume read yaml data of network config, parse and create network file, if some extra config is needed we will add it based on requirement which may not be supported in upstream systemd/netplan. This approach will scale better for us.

3. I understand your idea of having V2 net config support but we support both V1 & V2 as there are consumers for both and it's kinda tricky situation. So good for us to support both versions.

Hope this clarifies at least some of your questions.

Hi @TheRealFalcon - Can you please review my changes?

@sshedi , yep, will do! Though probably won't get to it till sometime next week.

@sshedi , yep, will do! Though probably won't get to it till sometime next week.

Hi @TheRealFalcon - if you find time, can you please review my changes?

@sshedi We have a good start here. I'll wait to review the test files until we change some of the datasource networking things.

I left some comments inline. In addition to those:

cc_resolv_conf.py:33 : Let's add photon to 'Supported distros' line

I noticed you're setting locale both in the distro and also using cc_resolv_conf.py. Is this intentional? cc_resolv_conf.py isn't run by default. Are the changes to this file needed? If not, ignore my inline comments in that file.

For each new file that has a VMWare copyright line, can you also add a line underneath that says

# This file is part of cloud-init.  See LICENSE file for license information.

Thanks for the review comments. I have added the license info to all new files.
I didn't understand your point setting locale both in the distro and also using cc_resolv_conf.py.

I didn't understand your point setting locale both in the distro and also using cc_resolv_conf.py.

Sorry, I mean resolve_conf, not locale. In your distro, you have code to set the resolve_conf. If the distro is handling that, why do we also need changes to cc_resolv_conf.py? Unless there's a specific reason to use it, it can just go unsupported in your distro.

I didn't understand your point setting locale both in the distro and also using cc_resolv_conf.py.

Sorry, I mean resolve_conf, not locale. In your distro, you have code to set the resolve_conf. If the distro is handling that, why do we also need changes to cc_resolv_conf.py? Unless there's a specific reason to use it, it can just go unsupported in your distro.

Yes but that's inside _write_network function which will never be called, but I will think about it. I will try adding it in networkd.py

diff --git a/cloudinit/cmd/devel/net_convert.py b/cloudinit/cmd/devel/net_convert.py
index 0668ffa3..5c649fd0 100755
--- a/cloudinit/cmd/devel/net_convert.py
+++ b/cloudinit/cmd/devel/net_convert.py
@@ -11,7 +11,7 @@ from cloudinit.sources import DataSourceAzure as azure
 from cloudinit.sources import DataSourceOVF as ovf
 
 from cloudinit import distros, safeyaml
-from cloudinit.net import eni, netplan, network_state, sysconfig
+from cloudinit.net import eni, netplan, networkd, network_state, sysconfig
 from cloudinit import log
 
 NAME = 'net-convert'
@@ -51,7 +51,7 @@ def get_parser(parser=None):
     parser.add_argument("--debug", action='store_true',
                         help='enable debug logging to stderr.')
     parser.add_argument("-O", "--output-kind",
-                        choices=['eni', 'netplan', 'sysconfig'],
+                        choices=['eni', 'netplan', 'networkd', 'sysconfig'],
                         required=True,
                         help="The network config format to emit")
     return parser
@@ -118,9 +118,14 @@ def handle_args(name, args):
         config['netplan_path'] = config['netplan_path'][1:]
         # enable some netplan features
         config['features'] = ['dhcp-use-domains', 'ipv6-mtu']
-    else:
+    elif args.output_kind == "networkd":
+        r_cls = networkd.Renderer
+        config = distro.renderer_configs.get('networkd')
+    elif args.output_kind == "sysconfig":
         r_cls = sysconfig.Renderer
         config = distro.renderer_configs.get('sysconfig')
+    else:
+        raise RuntimeError("Invalid output_kind")
 
     r = r_cls(config=config)
     sys.stderr.write(''.join([

It'd be helpful to apply a patch like this so I can more easily work/play with the networkd renderer. Even with this change, there will still be a few issues to fix before it works (I'll address those inline), but here is an example of how you could run it:
cloud-init devel net-convert -p /path/to/config.yaml -k yaml -d /tmp -D photon -O networkd --debug

I have no idea why this is failing https://travis-ci.com/github/canonical/cloud-init/jobs/513191790

E             Full diff:

E             - [call(<ANY>, "# Your system has been configured with 'manage-resolv-conf' set to true.\n# As a result, cloud-init has written this file with configuration data\n# that it has been provided. Cloud-init, by default, will write this file\n# a single time (PER_ONCE).\n#\n\noptions foo\n", mode=<ANY>)]

E             ?       ^^^ ^^^^^^^^^^                                                                                                                                                                                                                                                                               ^^^^^

E             + [call('/etc/resolv.conf', "# Your system has been configured with 'manage-resolv-conf' set to true.\n# As a result, cloud-init has written this file with configuration data\n# that it has been provided. Cloud-init, by default, will write this file\n# a single time (PER_ONCE).\n#\n\noptions foo \n", mode=420)]

E             ?       ^^^^^^^^^^^^^^^^^^^^^^^ ^^^        

The latter has a an extra space at the end of foo, once again this doesn't happen in my setup. Only on xenial test.

Hey @sshedi , I took the liberty of pushing a commit to this branch for the cc_resolv_conf.py. Does it work for you? If not, we can always undo it.

I also removed the

        if self.uses_systemd():
            subp.subp(['systemctl', 'restart', 'systemd-resolved'])

The standard systemd-resolved.service runs after systemd-networkd.service, which runs after cloud-init-local.service, which is when this cc_resolv_conf.py is running. Since the config is being generated before systemd-resolved is started, we shouldn't attempt to restart it here.

I still need to do the rest of the review.

No problem, thank you. My idea was giving a bit more freedom in distro specific function by adding new function in distros/init.py. Ultimately resolve conf will be processed in same way. I am okay with this approach and it works for me. Also, good call on restarting resolved service along with networkd.

I have fixed the cloudinit/config/tests/test_resolv_conf.py script and resolved the merge conflicts.

I put up a PR for the DNS changes here: #923 . I borrowed a little from this PR if that's ok. If that PR lands, you shouldn't need any changes to network_state.py in this PR.

Hi @sshedi , thanks for those updates.

#923 just landed. It did include a doc change to the v1 format. After this is rebased, I should be able to do one final review tomorrow and hopefully get this one done!

@sshedi , I think this looks good now.

Thanks for working with me to get everything done needed to get this across the finish line!

Pleasure was all mine. Thanks James for all your insightful comments and being patient with me all this while and making me understand things better. Enjoyed working with you.

Will be sending incremental fixes and improvements from now on. Have a good time ahead, thanks again.