Guide to UI testing with Jest and Puppeteer #5119

rohanjr · 2020-03-21T00:30:54Z

Open to general feedback.

changelog_begin
changelog_end

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

nemanja-da · 2020-03-23T09:08:07Z

docs/source/getting-started/testing.rst

+As a bonus, if something goes wrong you can take a screenshot and see what state your test browser got into!
+
+Let's see how to use these tools to write some tests for our social network app.
+You can see the full suite in the file ``index.test.tsx``.


Would be good to write what tests are covered in a list (just naming them), so the user knows what he can see in there. This primes him for what s/he's getting into (lower cognitive load).

nemanja-da · 2020-03-23T09:10:04Z

docs/source/getting-started/testing.rst

+We can also control multiple browser pages at once and test the interactions between different users.
+As a bonus, if something goes wrong you can take a screenshot and see what state your test browser got into!
+
+Let's see how to use these tools to write some tests for our social network app.


Are the two tools "blended" in use, i.e., they are used in a single test? So I write the expected behavior in Jest and I perform the actions with Puppeteer, right?

That's right. Well I suppose Jest is the "framework" encompassing your tests and expectations, and Puppeteer is a library that you use during those tests to perform the necessary actions. Do you think this should be made clearer, or can the reader wait until we show an example?

I would spell it out really, as I wasn't 100% sure if this is how the two work together (lookup Steve Krug's "Don't make me think!", some excerpts can be found here ). Had to read back and forth.

nemanja-da · 2020-03-23T09:11:33Z

docs/source/getting-started/testing.rst

+We first have some global state that we will use throughout our tests.
+Specifically, we have child processes for the ``daml start`` and ``yarn start`` commands, which run for the duration of our tests.
+We also have a single Puppeteer browser that we share among tests, opening new pages in each.
+


Would be great to have a new subsection here "Initialising tests" or something like that. In general adding more structure.

nemanja-da · 2020-03-23T09:13:42Z

docs/source/getting-started/testing.rst

+This looks somewhat mysterious at first.
+We first get some sort of handle to the username input element, and then use it to click into it and type the party name.


Would write with more clarity and would avoid ambiguous terms like "somewhat mysterious", "some sort of a handle".

nemanja-da · 2020-03-23T09:15:02Z

docs/source/getting-started/testing.rst

+We use it to spawn the ``daml start`` and ``yarn start`` processes and launch the browser.
+On the other hand the ``afterAll()`` section is used to shut down these processes and close the browser.
+This step is important to prevent child processes persisting in the background after our program has finished.
+


New subsection "Writing a simple login test", "Staging a simple login test", or something like that.

nemanja-da · 2020-03-23T10:00:40Z

docs/source/getting-started/testing.rst

+    - The ``test`` syntax is provided by Jest to indicate a new test running the function given as an argument (along with a description and time limit).
+    - ``getParty()`` gives us a new party name. Right now it is just a string unique to this set of tests, but in the future we will use the Party Management Service to allocate parties.
+    - ``newUiPage()`` is a helper function that uses the Puppeteer browser to open a new page (we use one page per party in these tests), navigate to the app URL and return a ``Page`` object.


These are the basics for any test, right?

That's right, I'll try to mention that.

nemanja-da · 2020-03-23T10:01:24Z

docs/source/getting-started/testing.rst

+    - Next we ``login()`` using the new page and party name. This should take the user to the main screen. We'll show how the ``login()`` function does this shortly.
+    - We use the ``@daml/ledger`` library to check the ledger state. In this case, we want to ensure there is a single ``User`` contract created for the new party. Hence we create a new connection to the ``Ledger``, ``query()`` it and state what we ``expect`` of the result. When we run the tests, Jest will check these expectations and report any failures for us to fix.
+    - The test also simulates the new user logging out and then logging back in. We again check the state of the ledger and see that it's the same as before.
+    - Finally we must ``close()`` the browser page, which was opened in ``newUiPage()``, to avoid runaway Puppeteer processes after the tests finish.


This one as well is part of the basics, right? Would be great to provide an overview of what commands are needed for every test (at least in the GSG).

nemanja-da · 2020-03-23T10:13:50Z

docs/source/getting-started/testing.rst

+Note that you can use other features of the elements to select, such as their types and attributes.
+We just show simple class selectors here.
+
+When writing CSS selectors for your tests, you will likely need to check the structure of the rendered HTML in your app by running it manually and inspecting elements using your browser's developer tools.


Subsection, e.g., "Writing CSS selectors"

nemanja-da · 2020-03-23T10:15:15Z

docs/source/getting-started/testing.rst

+
+There is a subtlety to explain here due to the `Semantic UI <https://semantic-ui.com/>`_ framework we use for our app.
+Semantic UI provides a convenient set of UI elements which get translated to HTML.
+In the example of the username field above, the original Semantic UI ``Input`` is translated to nested ``div`` nodes with the ``input`` inside.


Add an image showing what the nested div looks like. Easier to understand.

It is shown in the screenshot, though a little hard to read. Added a sentence pointing out it's there.

changelog_begin changelog_end

hurryabit

@rohanjr Can you please run through the docs once more with the two general attitudes I've describen in my comments in mind?

hurryabit · 2020-03-23T16:13:51Z

docs/source/getting-started/testing.rst

+
+When developing your application, you will want to test certain workflows through the UI.
+In particular, you'll want to make sure that changes such as new features do not break existing functionality by mistake.
+It may be fine to do this testing manually at first, but it quickly becomes more tiresome and error prone as the app grows.


I don't think the last two sentences are necessary here. We don't want to tell people why they should test but rather how to do it.

hurryabit · 2020-03-23T16:14:46Z

docs/source/getting-started/testing.rst

+Testing Your App
+****************
+
+When developing your application, you will want to test certain workflows through the UI.


We should use the phrase "end-to-end tests" here. And maybe something like "all the way from the UI to the ledger and back".

hurryabit · 2020-03-23T16:25:49Z

docs/source/getting-started/testing.rst

+There are two tools that we choose to write end to end tests for our app.
+Of course there are more to choose from, but we show you one combination here.
+The first tool is a general JavaScript testing framework called
+`Jest <https://jestjs.io/>`_.
+Jest allows you to write test cases containing expectations about behaviour should occur.
+If the expectations for a test are not met, then you should see helpful messages about what went wrong.
+You can also write setup and cleanup procedures that run before and after each test.
+This will be very useful for us to manage services like the DAML sandbox and JSON API server.
+
+The second tool we use heavily is a library called
+`Puppeteer <https://pptr.dev/>`_.
+Puppeteer allows you to control a Chrome browser from a JavaScript (or TypeScript) program running on `Node.js <https://nodejs.org/en/about/>`_.
+In our case we use it to perform actions like logging in, following friends and sending messages.
+We can also control multiple browser pages at once and test the interactions between different users.
+As a bonus, if something goes wrong you can take a screenshot and see what state your test browser got into!
+To summarize, Jest is the framework encompassing your tests and expectations, and Puppeteer is a library that you use during those tests to perform actions in place of a real user.


I think we should write this under that assumption that the reader is already familiar with testing. After all, we don't want to teach them about testing in general but only about testing full-stack DAML applications. Thus I would heavily shorten this to:

Suggested change

There are two tools that we choose to write end to end tests for our app.

Of course there are more to choose from, but we show you one combination here.

The first tool is a general JavaScript testing framework called

`Jest <https://jestjs.io/>`_.

Jest allows you to write test cases containing expectations about behaviour should occur.

If the expectations for a test are not met, then you should see helpful messages about what went wrong.

You can also write setup and cleanup procedures that run before and after each test.

This will be very useful for us to manage services like the DAML sandbox and JSON API server.

The second tool we use heavily is a library called

`Puppeteer <https://pptr.dev/>`_.

Puppeteer allows you to control a Chrome browser from a JavaScript (or TypeScript) program running on `Node.js <https://nodejs.org/en/about/>`_.

In our case we use it to perform actions like logging in, following friends and sending messages.

We can also control multiple browser pages at once and test the interactions between different users.

As a bonus, if something goes wrong you can take a screenshot and see what state your test browser got into!

To summarize, Jest is the framework encompassing your tests and expectations, and Puppeteer is a library that you use during those tests to perform actions in place of a real user.

We've made the opinionated choice to use the following two tools the end-to-end test of our app:

Of course there are more to choose from, but we show you one combination here.

The first tool is a general JavaScript testing framework called

- `Jest <https://jestjs.io/>`_ is a general-purpose testing framework for JavaScript that works well with both React and TypeScript.

- `Puppeteer <https://pptr.dev/>`_ is a library that allows for scripting interaction with a Chrome browser in JavaScript.

rohanjr · 2020-03-23T19:37:25Z

@hurryabit @nemanja-da I've addressed both of your feedback. Let me know if there's anything else you'd like to change before we merge this.

nemanja-da

Thnx for the edits @rohanjr , all good from my end.

hurryabit

Awesome. Thanks.

hurryabit · 2020-03-25T08:24:31Z

docs/source/getting-started/testing.rst

+We'll walk though this step by step.
+
+    - The ``test`` syntax is provided by Jest to indicate a new test running the function given as an argument (along with a description and time limit).
+    - ``getParty()`` gives us a new party name. Right now it is just a string unique to this set of tests, but in the future we will use the Party Management Service to allocate parties.


Can we please have a ticket in the GSG milestone for this.

It's not actionable right now as I understand, but it's here #5185 so we don't forget.

hurryabit · 2020-03-25T09:14:45Z

docs/source/getting-started/testing.rst

+We've only used class selectors in these tests.
+
+
+Writing CSS Selectors


I'm not convinced this section adds more value than confusion. I think the issues we're facing here are pretty universal for any React UI framework. Since we're not trying to teach web development but rather want to focus on the issues related to DAML application, I'd prefer to entirely remove this section, particularly since there's no actual issue but only an anticipated potential one.

I'm not sure. Agree we're not trying to teach web dev in its entirety, but we're also not assuming the reader is familiar with these concepts. I don't think it's fair to expect the reader to automatically understand the fact that we are using Semantic UI (we actually don't mention that anywhere) which translates into HTML, and we are selecting the resulting HTML elements. This was an actual issue for me when I was writing selectors, until I realised this point. I take your point that it's in the weeds, but it might still be helpful.

Don't think this section hurts. It's short and points out a specific thing for Semantic UI when testing DAML applications built with create-daml-app (that uses the Semantic UI).

hurryabit · 2020-03-25T09:16:21Z

@shayne-fletcher Can you please read this as well and decide if that information would be enough to add E2E tests for DAVL. If not, could you please give some feedback what is missing.

shayne-fletcher · 2020-03-25T17:14:11Z

@shayne-fletcher Can you please read this as well and decide if that information would be enough to add E2E tests for DAVL. If not, could you please give some feedback what is missing.

@hurryabit @rohanjr Yes. I will try to use this guide to implement testing in DAVL as we've all discussed. Consequently, it will take some time to provide this feedback but I will do so!

rohanjr requested review from shayne-fletcher and hurryabit March 21, 2020 00:30

rohanjr requested review from bame-da and nemanja-da as code owners March 21, 2020 00:30

nemanja-da reviewed Mar 23, 2020

View reviewed changes

rohanjr added 2 commits March 23, 2020 10:24

First draft of guide to UI testing

1e358b0

changelog_begin changelog_end

Address Nemanja's feedback

578e5c8

rohanjr force-pushed the docs-testing branch from 4a37009 to 578e5c8 Compare March 23, 2020 15:58

Add copyright headers

a83817e

rohanjr changed the title ~~First draft of guide to UI testing~~ Guide to UI testing with Jest and Puppeteer Mar 23, 2020

hurryabit reviewed Mar 23, 2020

View reviewed changes

rohanjr added 2 commits March 23, 2020 15:28

Cut down intro and description of Jest/Puppeteer

2510ece

Shorten a bit more

00d9624

rohanjr requested review from hurryabit and nemanja-da March 23, 2020 19:35

nemanja-da approved these changes Mar 24, 2020

View reviewed changes

rohanjr merged commit c950bf9 into master Mar 24, 2020

rohanjr deleted the docs-testing branch March 24, 2020 15:17

hurryabit reviewed Mar 25, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Guide to UI testing with Jest and Puppeteer #5119

Guide to UI testing with Jest and Puppeteer #5119

rohanjr commented Mar 21, 2020 •

edited

Loading

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020

rohanjr Mar 23, 2020

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020 •

edited

Loading

nemanja-da Mar 23, 2020

rohanjr Mar 23, 2020

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020

nemanja-da Mar 23, 2020

rohanjr Mar 23, 2020

hurryabit left a comment

hurryabit Mar 23, 2020

hurryabit Mar 23, 2020

hurryabit Mar 23, 2020

rohanjr commented Mar 23, 2020

nemanja-da left a comment

hurryabit left a comment

hurryabit Mar 25, 2020

rohanjr Mar 25, 2020

hurryabit Mar 25, 2020

rohanjr Mar 25, 2020 •

edited

Loading

nemanja-da Mar 26, 2020

hurryabit commented Mar 25, 2020

shayne-fletcher commented Mar 25, 2020

		This looks somewhat mysterious at first.
		We first get some sort of handle to the username input element, and then use it to click into it and type the party name.

		We've only used class selectors in these tests.


		Writing CSS Selectors

Guide to UI testing with Jest and Puppeteer #5119

Guide to UI testing with Jest and Puppeteer #5119

Conversation

rohanjr commented Mar 21, 2020 • edited Loading

Pull Request Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nemanja-da Mar 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hurryabit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanjr commented Mar 23, 2020

nemanja-da left a comment

Choose a reason for hiding this comment

hurryabit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanjr Mar 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hurryabit commented Mar 25, 2020

shayne-fletcher commented Mar 25, 2020

rohanjr commented Mar 21, 2020 •

edited

Loading

nemanja-da Mar 23, 2020 •

edited

Loading

rohanjr Mar 25, 2020 •

edited

Loading