DEV Community: Julien Tanay

Keep your dependencies up to date with Renovate and Github Actions

Julien Tanay — Tue, 25 Feb 2020 12:48:25 +0000

With a little help from Renovate and the powerful Github Actions, it's becoming really easy to keep your dependencies up-to-date automatically. Let's see how we can easily setup an automated workflow to handle this for us for free!

Renovate all the things

In a few words : Renovate automatically keeps your dependencies up to date by regularly checking for available updates and opening PRs on your repository, using all your already available CI mechanisms.

Seems great, right ? Let's see how it works :

The first time you run it, it will try to detect which language you use, or if you have a Dockerfile, and so on. It will then create a PR with an initial configuration (in a renovate.json) so you can get started quickly.

Then, it will open PRs with updated dependencies. If you have some CI system testing your PRs, you will quickly see if the proposed changes passes the tests or not. If everything is green, you can safely merge the PR ! You can even tell Renovate to automatically merge the PR, if all tests pass the next time it scans your repository.

Of course, you can fine tune Renovate's behavior, e.g. to auto-merge patch updates only, or to completely ignore major updates, etc... But we won't cover that now :).

Rolling... Actions!

GitHub Actions is a great tool to automate repetitive tasks on your repository, like PR triage or small review tasks -- it can also act like a complete CI solution (that's what we do at Dior.com!). Plus, it comes with a generous free plan!

First, we need to give Renovate some permissions to "scan" your repository.

Go to Settings > Developer settings > Personal access tokens and Create a Personal Acces Token (PAT) with the repo scope.

Now, add it to your repository's secrets.

Now, simply create a file in .github/workflows :

on:
  schedule:
    - cron: "0 8 * * *"
name: Daily Jobs
jobs:
  renovate:
    name: Renovate
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@master
      - name: Run Renovate
        uses: docker://renovate/renovate:19.133-slim
        env:
          RENOVATE_REPOSITORIES: <owner/repository-name>
          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
          RENOVATE_AUTOMERGE: true # optional, see below

Here we are defining a simple workflow triggered periodically by Github Actions. It contains one job which does the following actions :

Clone your repository
Run Renovate

Simple, right ? We are just passing your newly created PAT and your repository's name. Remember to always pin the versions (tags, actually) of the Docker images you use, specially if it's a public one. You really want to avoid bad surprises if something gets broken some day.

You can add as many environment variables as you want; check the Renovate Self-Hosted Configuration Docs for the complete variables list.

Now, Github Actions will take care of running this job every day at 08 AM (feel free to customize the schedule option at will).

That was quick, right ? What do you think ?

Bootstrapping a Twitch Extension with NextJS

Julien Tanay — Tue, 15 Oct 2019 07:47:52 +0000

Twitch Extension allow broadcasters to engage their audience in many different ways. It's bringing a whole new level of interactivity to your ordinary gaming channel. Under the hood, it is a collection of HTML/JS pages hosted by Twitch. As is it mainly tiny JS apps, you can bring your favorite tools with you.

One of the tool I use more and more these day is NextJS. It's a powerful framework for building producion-ready React Apps. When I started to develop a Twitch extension, I naturally wanted to start with a NextJS app structure. After a bit of tweaking, I was able to put together a familiar development setup.

Let's do this

I'll guide you through a couple steps to get you up and running. Starting from here, all you need is your usual dev machine and some basic knowledge of JS and React. Bonus points if you are already used to Twitch extensions, advanced React apps, and/or NextJS itself!

Let's scaffold our NextJS project with create-next-app, a very handy script which will setup your new project. Open a terminal and type :

npx create-next-app my-extension # or 'yarn create-next-app my-extension'

Allright ! Now, cd into your new project folder and open a text editor of your choice. In a NextJS project, each page of your app sits in the pages folder (sic).

Twitch extensions are divided into pages, so we will need to tell NextJS to handle these separates pages. It is a bit like Webpack's entrypoints. As a simple example, let's assume we are building a panel extension (see this page: "A panel extension appears in the panel area below the video player. Panel Extensions stay active even when the channel is not live."). Rename pages/index.html into pages/panel.html.

Create a new next.config.js file at the root of your project like this one :

const isProduction = process.env.NODE_ENV === "production";

module.exports = {
  assetPrefix: "./",
  env: {
    STATIC_PREFIX: isProduction ? "./static" : "/static"
  },
  exportPathMap: async (
    defaultPathMap,
    { dev, dir, outDir, distDir, buildId }
  ) => {
    return !dev
      ? {
          "/panel": { page: "/panel.html" },
          "/live_config": { page: "/live_config.html" },
          "/config": { page: "/config.html" }
        }
      : defaultPathMap;
  },
  webpack(config, options) {
    config.optimization.minimize = false;
    return config;
  }
};

Wait! What are we doing here?

assetsPrefix tells NextJS that we need to use a relative path to find our bundled assets (see this page)
exportPathMap lists the pages we want NestJS to export. in the example above, I'm assuming you are developing a "Panel"-only extension. You might want to adjust this based on your needs, re-using the same syntax ("/<page_name>": { page: "/<page_name>.html" }).
config.optimization.minimize = false; makes sure Webpack won't minimize your files when bundling them. You will need your file not to be minimized to submit your extension for review later.

Finally, add the following convenience scripts to your package.json :

{
  // (...)
  "scripts": {
    // (...)
    "prerelease": "rm -rf .next out",
    "release": "NODE_ENV=production next build && next export && cd out && zip -qr bundle.zip *"
  }
}

release builds your site, export it as static html and then zip for you so you can upload it to your Developer Console isntantly!
prerelease will be run just before release to ensure you don't bundle old, stale files.

"And voilà", you are all set ! Now, you can start developing your extension logic.

Developing your extension locally

Using the Twitch Developer Rig, you can easily use this NextJS setup on your local machine. I won't go through the Rig setup as it is straightforward, but be sure to configure your extension before using it in the Rig: go to your Twitch developer console and make sure that the "Testing Base URI" is set to http://localhost:3000 (the default for a NextJS project).

Once you have your project in your Rig, make sure you tells it the right command for Front End, i.e. npm run dev (or yarn dev).

You should be able to launch "views" inside the rig and play with your extension!

Once your done with your developments, just use the yarn release command and you'll be ready to upload your bundle.zip on the console, in the "Files" tab.

Easy, right ? You can now proceed with the final steps on your Twitch Dashboard (move to hosted test, submit for review... and release !)

One more thing

Last month we launched our latest Twitch Extension, "Nice Shot!", a viewer engagement tool for Rocket League streamers. It's already live here and part of the Twitch Dev Jam 2019 (drop us some likes there)! Give it a try and tell us what on think on Twitter.

Documenting your Flask-powered API like a boss

Julien Tanay — Tue, 05 Dec 2017 14:23:24 +0000

Flask is a very popular and powerful framework for building web applications. Over the last years, people used it to create REST API that work well with decoupled, modern front-end applications.

One challenge that backend development teams often face, is how to make it easy for front-end developers, wethere internal or with a distant community, to create API-compliant clients (web app, mobile app or even CLI tools...)

In the wild, they are many good examples of well-documented APIs... Take the Twitter API : the docs are great, user-friendly and cover all the available endpoint with tips and examples. Any fresh CS student could write a small Python tool using it, just by following the documentation and its examples.

At @Ooreka, we decided to follow the OpenAPI (fka Swagger 2.0) specification to build a solid documentation for our Flask-powered micro-services APIs. Let's dive in.

3..2..1.. Doc!

Thanks to the apispec lib, you can automagically generate a specification file (commonly named swagger.json) form your Flask code. Some other libraries can do a lot of magic for you, but apispec is really simple to use and can sit next to your code without interfering with it.

It supports Marshmallow and Flask, allowing you to re-use your code to generate a proper documentation !

Let's write our generation script, e.g. app/scripts/openapi.py :

from apispec import APISpec

# Create spec
spec = APISpec(
    title='My Awesome API',
    version='1.0.42',
    info=dict(
        description='You know, for devs'
    ),
    plugins=[
        'apispec.ext.flask',
        'apispec.ext.marshmallow'
    ]
)

# Reference your schemas definitions
from app.schemas import FooSchema

spec.definition('Foo', schema=FooSchema)
# ...

# Now, reference your routes.
from app.views import my_route

# We need a working context for apispec introspection.
app = create_app()

with app.test_request_context():
    spec.add_path(view=my_route)
    # ...

# We're good to go! Save this to a file for now.
with open('swagger.json', 'w') as f:
    json.dump(spec.to_dict(), f)

Here, we first create a new APISpec instance with some details about our API.
Then, we add our definitions (here, we are using Marshmallow to define how our API will serialize/deserialize data) with APISpec.definition().
Finally, we add our routes to our API specification using APISpec.add_path(). apispec will parse your route functions docstrings, so make sure your add some OpenAPI YaML stuff here, as in :

@app.route('/foo/<bar_id>')
def my_route(gist_id):
    """ Cool Foo-Bar route.
    - - - # dev.to editor dislike triple hyphen, be sure to remove spaces here.
    get:
        summary: Foo-Bar endpoint.
        description: Get a single foo with the bar ID.
        parameters:
            - name: bar_id
              in: path
              description: Bar ID
              type: integer
              required: true
        responses:
            200:
                description: Foo object to be returned.
                schema: FooSchema
            404:
                description: Foo not found.
    """
    # (...)
    return jsonify(foo)

You will end up with a valid JSON API specification. Now, let's see how to bootstrap an HTML version to show it to the world!

Browerify-ing all this

A really cool tool to do that is the ReDoc Javascript library from the guys at APIs.guru. We'll use it to present the generated JSON specification in a convenient way.

Redoc is basically a single, minified JS file you can include in a bare index.html file and tell it where your swagger.json is located. It uses a really neat 3 columns design : a navigation sidebar, a wide center section with your API endpoints definitions and a third column dedicated to requests or responses samples and examples.

<!DOCTYPE html>
<html>
  <head>
    <title>Cool API Documentation</title>
    <meta charset="utf-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style> body { margin: 0; padding: 0; } </style>
  </head>
  <body>
    <redoc spec-url='./swagger.json' hide-loading></redoc>
    <script src="https://rebilly.github.io/ReDoc/releases/latest/redoc.min.js"> </script>
  </body>
</html>

Yep, that was quick. Check out a real-world example here.

Wrapping it up

The OpenAPI offers many options I didn't cover here for brievity and simplification. You can add your server's real endpoints to the doc, add many details about the parameters and responses of your routes, provide example in your routes functions docstring that will be parsed and added to your spec, etc...

As a final tip, head to the Flask CLI documentation to see how easily you can hook your generation script into the command line interface of Flask (this will give you some badass command like FLASK_APP=main.py flask generate_doc). Oh, and be sure to put this into your Continuous Integration routine to keep your API documentation up-to-date with your API!

Cheers!

NOTE: This post was originally posted on Medium.

Pipfile and Pipenv : the future of Python dependencies management

Julien Tanay — Wed, 22 Nov 2017 10:54:44 +0000

Yes, I heared you. pip is a great tool and has been around for quite a long time. But for 3 years or so, people (contributors) have been looking for a way to enhance our packages management experience. Think about the superpowers of composer, npm (or better, yarn) in your favorite tool.

What they offer is (more or less) a replacement for the age-old requirements.txt file : the Pipfile.

How I learn to stop worrying and love pip (again)

The new Pipfile differs from the old-style requirements file in several ways :

It uses TOML syntax to allow more configuration;
It declares dependencies groups (i.e. a default one plus a development one for... development packages) so you don't have to split your requirements into several files.
It comes with a Pipfile.lock file which pins the versions of your packages, in addition to some security bonus.

Let's have a look at a minimal, simple Pipfile:

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true
name = 'pypi'

[requires]
python_version = '3.6'

[packages]
flask = '*'

[dev-packages]
pytest = '*'

What do we have here :

First we declare the source that pip is going to use. Typically, it will be pypi.
Then we declare our Python version requirement.
Finally, we declare our packages and development packages.

You got it ? Now here is the example Pipfile from the official repository on GitHub :

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true
name = 'pypi'

[requires]
python_version = '2.7'

[packages]
requests = { extras = ['socks'] }
records = '>0.5.0'
django = { git = 'https://github.com/django/django.git', ref = '1.11.4', editable = true }
"e682b37" = {file = "https://github.com/divio/django-cms/archive/release/3.4.x.zip"}
"e1839a8" = {path = ".", editable = true}
pywinusb = { version = "*", os_name = "=='nt'", index="pypi"}

[dev-packages]
nose = '*'
unittest2 = {version = ">=1.0,<3.0", markers="python_version < '2.7.9' or (python_version >= '3.0' and python_version < '3.4')"}

See ? With the TOML syntax, you can be very specific about the version, source or os-variant you want to use. Note that all packages versions are not pinned, even if it was the preferred way of declaring dependencies.

Pip is going to support this specification in the furure with a syntax like pip install -p Pipfile. Let's see how we can use all this magical stuff today.

Back to the Future

Meet Pipenv (yep, that's another project from Kenneth Reitz, the guy behind httpbin, pep8.org, and, yes, request).

Julien Tanay

@djiit

the fuck when you realize @kennethreitz 's pipenv just dropped a 🎃 in your shell. Made my day.

13:14 PM - 31 Oct 2017

Pipenv is the current preferred implementation of Pipfile and allows you to manage your virtualenvs like a boss. It aims to ease and enhance how we manage our python environments (a la virtualenvwrapper, but with pip skills).

Here is a simple workflow :

1- In your project directory, create your virtualenv. Pipenv will store it in a centralized place instead of your project root directory :

pipenv --three # Yeah, 3 is the way to go

2- Install some dependencies :

pipenv install flask
pipenv install --dev pytest # Notice the --dev flag

3- Use your virtualenv. It will open a new shell :

pipenv shell

Note that we were not in our virtualenv before this command. Everything is handled for us by Pipenv.

From this point, you are good to go. Pipenv will help you manage your environment throughout your development process.

Finally, you shouldn't be worried about compatibility issues with your deployment server (which is surely still using pip as you are reading this) : you can generate a good old requirements.txt by running :

pipenv lock -r > requirements.txt

Pipenv handles your dependencies and it even adds some security reinforcements by specifying each package's hash.

That's it. Go test Pipenv and tell me what you think !

Cheers

NOTE: This post was originally posted on Medium.