DEV Community: HaramChoi

Automating WebGL-/Canvas-based Website

HaramChoi — Tue, 18 Apr 2023 14:41:44 +0000

WebGL-based websites are becoming increasingly popular among web developers looking to create immersive and interactive user experiences. By drawing graphics and animations directly on the canvas, they offer developers a high degree of creative control over the user interface, providing a more flexible and dynamic platform for graphical compositions. They also offer a more interactive user experience, allowing users to interact with graphics and animations in real-time.

However, WebGL-based websites also present a unique challenge for automation, as they often lack DOM elements that conventional selector-based test frameworks rely on.

In this tutorial, we will demonstrate an automation case using AskUI to automate a WebGL-based website. Since AskUI relies solely on what is visible on the screen, you will discover how easy it is to automate even such a canvas-based website without relying on any DOM element.

Visit the Target Website

For the demonstration, we will use the website below. First, visit this website. If you inspect the source of this website, you will discover that it consists of a single canvas element without giving any clue on how to programmatically automate user interaction.

https://chartogne-taillet.com

Disclaimer: We use this website merely for our demonstration purpose and don't intend to promote any product or brand from this website.

Plan the Automation

Our final goal of this automation is to visit a page of this website that gives detailed information about a certain grapevine. Below are the steps we will automate:

Open the web browser and go to the website.
Click Accept for the cookie pop-up.
Click the text Enter on the first page.
Click the text that is a certain sort of grapevine.
Move the mouse cursor to the interaction point.
Scroll the mouse wheel.

1. Visiting the Website

The first step of the automation is to open the web browser and go to the website. Use the code below, and adjust the necessary part for your environment:

import { aui } from './helper/jest.setup';

describe('interact with WebGL', () => {
  it('opens browser and go to website', async () => {
    // Windows
    await aui.pressKey('command').exec(); 
    // macOS
    // await aui.pressTwoKeys('command', 'space').exec();
    await aui.type('chrome').exec();
    await aui.pressKey('enter').exec();
    // wait for the browser to open
    await aui.waitFor(1000).exec(); 

    // run this if the address bar is not focused after opening chrome
    await aui.click().text().withText('Search Google or type a URL').exec();
    await aui.type('https://chartogne-taillet.com').exec();
    await aui.pressKey('enter').exec();
  })

  it('click accept', async ()=>{
    await aui.click().text().withText('accept').exec();
  });
});

After successfully running the code, we will see this page:

2. Click Enter and go to Les Couarres

Nextly, we will click the text Enter and the page will transit to the next page. After that, we press the text Les Couarres on the bottom right part of the page. Note that these texts are not DOM elements, but merely texts drawn on the canvas as a bitmap. So there is no clue how to interact with these text-buttons if you inspect the source HTML of the page. Add this code block to the end of the code:

it('clicks enter', async ()=>{
  await aui.click().text().withText('Enter').exec();
   // wait for the page transition
  await aui.waitFor(5000).exec();
  await aui.click().text().withText('Les Couarres').exec();
});

After running the code, we will be on this page:

3. Scroll on the Text

Now we want to move on to the page where we can get detailed information about Les Couarres. To do so, we have to interact with the text DISCOVER that is placed at the mid-bottom of the screen:

it('scrolls on the text', async ()=>{
  await aui.moveMouseTo().text().withText('DISCOVER').exec();
   // negative direction on y axis will scroll down
  await aui.scroll(0, -2000).exec();
  await aui.scroll(0, -2000).exec();
  await aui.scroll(0, -2000).exec();
});

Try to adjust the amount of scrolling that suits your condition, for example, decrease it to -3000 if you have a high-resolution display. When scrolling on the text, you will see a smooth transition to another page. Clicking the text also triggers the same result, so give it a try if you want to!

There will be an animation until the page transition gets done. And after the transition, we will be on a page about a certain vineyard and grapes. Congratulations! You just successfully automated mouse interactions with WebGL contents!

Conclusion

In this article, we demonstrated how to automate a WebGL-based website using AskUI. By relying solely on what is visible on the screen, AskUI can interact with canvas-based websites without requiring any selectors or tags. We walked through a step-by-step guide to visit a certain page of a website, click on non-DOM text elements, move the mouse cursor toward a bitmap text, and scroll on a specific bitmap text, all of which are common challenges when automating WebGL-based websites. With AskUI, you can easily automate even the most complex canvas-based user interfaces, providing a more efficient way to test and validate the user experience of your WebGL applications.

Did you get any issues while using AskUI? Drop by our Discord if you are seeking help!

How AskUI Works

HaramChoi — Thu, 13 Apr 2023 07:54:46 +0000

Overview

In this article, we will give a detailed overview of AskUI's architecture and how it works under the hood.

AskUI is built on top of a number of components. We will cover what these components are and how they work together to provide a flexible and reliable way to automate interactions with UI elements of any operating system or platform.

By the end of this article, whether you're a software developer, QA engineer, or automation specialist, you'll have a solid understanding of how AskUI works, and be able to use this knowledge to build more efficient automation for your project.

AskUI consists of three building blocks:

AskUI Control Client
AskUI UI Controller
AskUI Inference Server

We will step through each of them and see how they work together to perform UI automation.

Glossary

Throughout this article, we will use some terms that describe certain parts of AskUI. Some of them are used only internally and not exposed by the AskUI Control Client API, but are important for understanding how AskUI works and what it can do. Please refer to this table while reading.

Term	Description
Element-description	A method in the AskUI Control Client API that searches for a specific type of UI element. For example `button()`, `textfield()`.
Action	A method in the AskUI Control Client API that describes an action to be taken against the operating system. For example `click()`, `type()`.
InputEvent (internal)	A specific type of action to be taken against the operating system. For example MouseInputEvent or KeyboardInputEvent.
ControlCommand (internal)	A command sent to the UI controller telling what to perform on the operating system. It consists of one or more InputEvents.

AskUI Control Client

The AskUI Control Client provides the API that tells AskUI what/how to automate. Once you start using AskUI, you will mostly interact with AskUI via the AskUI Control Client. In most of our tutorials and demonstrations, you will see let aui: UIControlClient is declared and combined with an Action and Element-descriptions which ends up forming an instruction, e.g:

await aui.click().button().withText('login').exec();

As shown above, you form an instruction by chaining an Action with Element-descriptions using the Fluent API of the AskUI Control Client. It is designed as a fluent interface to increase readability and make it more understandable.
The AskUI Control Client sends a request to the AskUI UI Controller:
- to take a screenshot.
- with a ControlCommand that tells what InputEvent to perform on the operating system.
The AskUI Control Client communicates with the AskUI Inference Server:
- to send a screenshot to be annotated with the instruction.
- to receive the annotation, e.g. detected elements.

To use the AskUI Control Client, user credentials are required. User credentials can be obtained via our User Portal.

See our API documentation for more information on this component.

AskUI UI Controller

The AskUI UI Controller is a binary that controls the operating system. This binary gets automatically downloaded when the UiController is initialized by calling UiController.start(). Once executed, it stays in the background and communicates with the AskUI Control Client on a specific port to receive the ControlCommand. Based on the given ControlCommand, it triggers InputEvents respectively.

The AskUI UI Controller is responsible for:
- Taking a screenshot.
- Triggering the InputEvent, i.e MouseInputEvent, KeyboardInputEvent, or shell execution.
- Running the interactive annotation.

See our API documentation for more information on this component.

AskUI Inference Server

The AskUI Inference Server is responsible for the prediction of UI elements within the given screenshot. As soon as it receives a request from the AskUI Control Client, it performs the prediction on the given image and returns the annotation to the AskUI Control Client.

For the inference, we use a machine-learning model that consists of several submodels:

Object Detector: Detects UI elements (e.g Button, Textfield).
Icon Classifier: Predicts the class of an icon based on the detected objects (e.g. a user icon 👤).
Optical Character Recognition (OCR): Converts the image of a text into text.
Custom Element Detector: Searches for an area in the given screenshot that matches the image given by the Element-description .customElement().

Them All in Action

Assuming that we run AskUI on the same device we want to automate, the simplest synopsis can be described as such:

When running AskUI,

The AskUI Control Client checks whether it is needed to be processed by the Inference Server.
If the code contains any of the Element-description or Getters, then the AskUI Control Client tells the AskUI UI Controller to take a screenshot of the given screen and sends it to the Inference Server.

After the AskUI Control Client has retrieved the annotation back from the server, it sends a ControlCommand to the AskUI UI Controller. Afterwards, the AskUI UI Controller triggers the InputEvent on the operating system.

// an example of AskUI code containing an element-identifier
await aui.click().button().withText('Confirm').exec();
// Here, the filter 'button()' is used together,
// so the client will fire a request to the server.

If the code contains an Action but no Element-description, then the AskUI Control Client sends the ControlCommand to the AskUI UI Controller to trigger the InputEvent directly.

// an example of AskUI code containing only a command
await aui.pressThreeKeys('control','alt','del').exec();
// It uses only a command 'pressThreeKeys()',
// so it will be executed by the UI Controller directly.

An Element-description represents a specific type of UI element that can be recognized by inference. Most of the commonly used UI elements such as Button, Textfield are supported and can be used.
An Action represents a specific type of action to be performed, i.e Mouse/Keyboard Input Event or Shell Command. This action can be performed on a specific element when combined with Element-descriptions or can be performed on its own as shown in the example right above.

Please visit our API Docs, if you want to learn more about different types of Element-description and Action.

Conclusion

Here we have seen the three core components of AskUI. If you aim to use AskUI in a more advanced way, e.g. integrating it into your CI/CD pipeline, it may be worthwhile to get an overview of how it is composed. For more practical examples, please refer to our Tutorials and API docs. And don't forget to come over to our Discord community, if you have any questions about AskUI!

Custom Elements in AskUI

HaramChoi — Mon, 13 Feb 2023 13:59:34 +0000

Overview

Custom Element Selection is a feature in AskUI that enables you to create custom selectors for elements on the screen, instead of relying on the standard selectors provided such as Button, Textfield, etc.

With this feature, you can define a custom selector based on how the element is displayed on the screen. This can be particularly useful in situations where standard selectors are unreliable due to the non-standard properties of the element. It provides greater flexibility and control, allowing you to tailor the automation to meet the specific needs of your application.

Here we will demonstrate how to use a custom element to explore Google Street View.

Demonstration

Requirements

AskUI - Follow this tutorial if you haven't installed it yet.
Web Browser - We use Safari in this demonstration, but you can use any web browser you have.

Understanding the `customElement()` in AskUI

customElement() is an element to look for on the screen that is defined by the user with a given image.

// Example of customElement()

await aui
    .click()
    .customElement({
        customImage: './logo.png', // required
        name: 'myLogo', // optional
        threshold: 0.9, // optional, defaults to 0.9
        rotationDegreePerStep: 0, // optional, defaults to 0
        imageCompareFormat: 'grayscale', // optional, defaults to 'grayscale'
        mask:{x:0, y:0}[] // optional, a polygon to match only a certain area of the custom element
    })
    .exec();

Arguments

customImage (string, required):
- A cropped image in the form of a base64 string or file path.
name (string, optional):
- A unique name that can be used for filtering for the custom element. If not given, any text inside the custom image will be detected via OCR.
threshold (number, optional):
- A threshold for how much a UI element needs to be similar to the custom element as defined. Takes values between 0.0 (== all elements are recognized as the custom element which is probably not what you want) and 1.0 (== elements need to look exactly like the customImage which is unlikely to be achieved as even minor differences count). Defaults to 0.9.
rotationDegreePerStep (number, optional):
- Step size in rotation degree. Rotates the custom image by this step size until 360° is exceeded. The range is from 0 to 360. Defaults to 0.
imageCompareFormat ('RGB' | 'grayscale', optional):
- The color compare style. grayscale compares the brightness of each pixel whereas RGB compares all three color. Defaults to grayscale.
mask ({x:number,y:number}[], optional):
- A polygon defined by an array of points to match only a certain area of the given custom image.

Two Things to be Aware of When Using `customElement()`

1) Create the Custom Image by Cropping it From The Actual Screen

To find a matching element from the screen, the custom image must be the same as it is displayed on the screen. By saying same in this sense, includes the size, rotation as well as the overlapping object, if there is any.

Note the left-bottom case of the illustration. A rotated element can be also matched, but only if everything else except the rotation are staying the same as it is displayed on the screen. If you can assure that your custom image is exactly the same as it is displayed on the screen + if you know the degree of the rotation, then you could consider using the rotationDegreePerStep parameter. And because AskUI will try to rotate the custom element for the whole revolution, a divisor of the rotated degree could be also used, e.g in the illustrated case, we can use not only 90 but also 45, 30, 15, etc. But since smaller degrees will require more iteration steps, it will increase the runtime by a notable amount.
The simplest way to accomplish it might be to screen capture and crop the desired image from your screen directly. In Windows and macOS, you can use the built-in screen capture tool:
- Windows: Press windows + shift + s (Windows 10 or higher)
- macOS: Press cmd + shift + 4
In both cases, you will be asked to select a certain portion of the screen. On Windows, the captured image will be stored in the clipboard, so you will need to save it to an image file. On macOS, the image will be saved in the ~/Desktop by default.

2) The Time of the Execution will Increase by a Notable Amount

To examine whether the custom image matches the given screen, AskUI iterates through the whole pixels of the given screen as well as the custom image. So it is likely to increase the runtime by a notable amount. Therefore, if the task could be accomplished with other filters such as icon(), button(), or text(), then it's maybe better to avoid using the customElement().

Capture the Custom Element

In this demonstration, we will search for a certain area in Google Street View. This can be enabled by pressing a button at the right corner of the Google Maps:

Can you see the yellow tiny human in the corner? We need an image of this human figure to interact with it.
Let's make a screen capture of it. It shall look like this:

Then save the image in your project's root directory with the name human-figure.png. The file tree of your project's root directory will be like this:

project_root/
├─ node_modules/
├─ test/
├─ package.json
├─ tsconfig.json
├─ human-figure.png

Write the AskUI Code

If you are prepared with the image above, let's jump into our code:

import { aui } from './helper/jest.setup';

describe('Explore the world in google maps', ()=>{

  it('open web browser and go to  google maps', async ()=>{

    // open the start menu/spotlight to search for the web browser
    await aui.pressTwoKeys('command', 'space').exec(); // for macOS
    // await aui.pressKey('command').exec(); // for Windows
    await aui.waitFor(250).exec(); // wait for the start menu to open

    await aui.type('safari').exec(); // type the name of the web browser
    // await aui.type('chrome').exec(); // if you are using another web browser, replace the name to it
    await aui.pressKey('enter').exec(); // open the web browser
    await aui.waitFor(1000).exec(); // wait for the web browser to open

    await aui.type('https://maps.google.com').exec(); // type the url of the website
    await aui.pressKey('enter').exec(); // open the website
    await aui.waitFor(1000).exec(); // wait for the website to load

  });

  it('search for a location', async ()=>{

    await aui.type('machu picchu').exec(); // type the name of the location
    await aui.pressKey('enter').exec(); // search for the location
    await aui.waitFor(2000).exec(); // wait for the map to load
    await aui.pressKey(',').exec(); // hide the side panel

  });

  it('enable street view', async ()=>{

    // now we look for our custom element on the map
    // move the mouse to the custom element
    await aui.moveMouseTo()
        .customElement({
            customImage: "./human-figure.png",
            name: "street-view-icon",
            threshold: 0.9,
        })
        .exec();

    // click and hold on the custom element
    await aui.mouseToggleDown().exec();

    // drag the custom element(our human) to the location we want to explore
    // note the offset of -50 pixels along the y axis
    // we drag the human 10 pixels higher than the location Aguas Calientes
    await aui.moveMouseRelativelyTo(0, -10).text().withText('Aguas Calientes').exec();

    // release the mouse button
    await aui.mouseToggleUp().exec();
  });
});

After successfully running the code, you will be able to see the landscape of Machu Picchu, the most iconic citadel of the lost empire Inca.
It is possible that you end up with a plain Google Map without having the Street View enabled. It might be caused by various reasons, but the most likely scenario is due to the different resolutions of the screen (your display can have a different resolution than mine). You could try to adjust the amount of the pixel offset that is given to the moveMouseRelativelyTo(), for example, try with moveMouseRelativelyTo(-5, -15).

Breaking Down the Code

1) Open the Web Browser and Go To the Desired Website

One notable part of this step is the waitFor() after each execution. We have used it in three different lines of this code block. Check out the respective parts and adjust the amount of time to wait until the process is finished. It may take more or less time depending on the condition of your device and internet connection:

it('open web browser and go to  google maps', async ()=>{
    // open the start menu/spotlight to search for the web browser
    await aui.pressTwoKeys('command', 'space').exec(); // for macOS
    // await aui.pressKey('command').exec(); // for Windows
    await aui.waitFor(250).exec(); // wait for the start menu to open

    await aui.type('safari').exec(); // type the name of the web browser
    await aui.pressKey('enter').exec(); // open the web browser
    await aui.waitFor(1000).exec(); // wait for the web browser to open

    await aui.type('https://maps.google.com').exec(); // type the url of the website
    await aui.pressKey('enter').exec(); // open the website
    await aui.waitFor(1000).exec(); // wait for the website to load
});

Also, don't forget to change the key to press and the name of the web browser based on your condition.

2) Search for the Location

Here we type our desired keyword into the textfield of Google Maps. As the textfield gets focused automatically, we can directly type in the keyword to the textfield:

it('search for a location', async ()=>{

    await aui.type('machu picchu').exec(); // type the name of the location
    await aui.pressKey('enter').exec(); // search for the location
    await aui.waitFor(2000).exec(); // wait for the map to load
    await aui.pressKey(',').exec(); // hide the side panel

});

Note that we also press the ,(comma) key to hide the side panel of Google Maps. This is for hiding unnecessary information from the screen.

3) Drag the Human Icon to the Desired Location

Finally, we drag our human, which we defined as our Custom Element, to the desired location.
Firstly, we move the mouse cursor to our custom element.
For dragging the mouse, we use the mouseToggleDown() to press-and-hold the mouse left button.
After that, we move the mouse to the desired location.
Thereafter, we use mouseToggleUp() to release the mouse button.

it('enable street view', async ()=>{

    // now we look for our custom element on the map
    // move the mouse to the custom element
    await aui.moveMouseTo()
        .customElement({
            customImage: "./human-figure.png",
            name: "maps",
            threshold: 0.9,
        })
        .exec();

    // click and hold on the custom element
    await aui.mouseToggleDown().exec();

    // drag the custom element(our human) to the location we want to explore
    // note the offset of -10 pixels in the y axis
    // we drag the human to 10 pixels higher than the location Aguas Calientes
    await aui.moveMouseRelativelyTo(0,-10).text().withText('Aguas Calientes').exec();

    // release the mouse button
    await aui.mouseToggleUp().exec();
});

Note the optional parameters for the customElement(), especially the threshold that is set to 0.9.
This parameter can be set from 0.0 up to 1.0.
- 0.0 will consider every element on the screen as matched with the given image.
- 1.0 will examine the given elements as strict as possible, so you might end up without any matching element found.
So, the best scenario to set the threshold might be:
- 1) Make the custom image to be as precise as possible (cropping from the screen).
- 2) Keep the threshold relatively higher, but below 1.0

Conclusion

If you plan to program an automation where you have elements with non-standard properties, you might want to consider using the custom element feature of AskUI. But as mentioned above, keep in mind that, as a trade-off, it consumes more time than other features. Taking it into account, using a custom element to interact with the given UI can be a huge help, especially if the element lacks standard properties such as tag or appearance.

If you got any issues while following this article, don't hesitate to ask for help in our Discord Community! We are more than glad to hear about your experience and help!

Using Reporters in Jest Framework

HaramChoi — Mon, 30 Jan 2023 08:25:13 +0000

Overview

When using jest as the framework for your test, it's often desired to have a report that summarises the test result neatly. Although jest comes with a default test reporter that prints out the report in the console directly, one could wish to have a discrete report, e.g as an XML or HTML, that can be stored and shared among team members.

This article covers the usage of a test reporter within Jest framework. Particularly, we will set up the jest-junit, jest-html-reporter, and jest-html-reporters.

Requirements

AskUI SDK installed (follow this tutorial)
jest (should be set up after following the above tutorial)
For the demonstration, we will use the website Authentication Test as an automation target.

1. Prepare the AskUI Test Suite within jest

Let's say that, we want to use the AskUI Library to automate the login procedure in the example website.

Go to the example website in your web browser, and run the code provided below to automate the login:

describe('Login',()=>{
  xit('annotates', async ()=>{
    await aui.annotateInteractively();
  });
  describe('Simple Form',()=>{

    it('should click the button',async ()=>{
      await aui.moveMouseTo().button().contains().text().withText('Simple Form Auth').exec();

      await aui.mouseLeftClick().exec(); // This first click is to get the focus on the test app
      await aui.mouseLeftClick().exec(); // Thereafter, we click the button
    });

    it('should type in the email address',async ()=>{
      await aui.typeIn('simpleForm@authenticationtest.com').textfield().contains().text().withText('E-mail Address').exec();
    });

    it('should type in the password',async ()=>{
      await aui.typeIn('pa$$wOrd').textfield().contains().text().withText('Password').exec();
    });

    it('should click the login button',async ()=>{
      await aui.click().button().contains().text().withText('Log In').exec();
    });
  });
});

If the test has run successfully, then you will see the default test report printed on your console:

Now let's try to use reporters other than the default one.

2.1. Using jest-junit with AskUI

jest-junit is an npm library that creates an XML report file per test run in the format of the JUnit XML that can be understood by other development automation tools such as the Jenkins JUnit plugin. As JUnit is one of those unit test frameworks which were initially used by many Java applications as a unit test framework, jest-junit reporter facilitates a smooth integration of your test suite.

Run this command to install the reporter within your project root directory:

npm i jest-junit

Then, go to test/jest.config.ts and change it as below:

import type { Config } from '@jest/types';

const config: Config.InitialOptions = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  setupFilesAfterEnv: ['./helper/jest.setup.ts'],
  sandboxInjectedGlobals: [
    'Math',
  ],
  reporters: [
    "default",
    "jest-junit",
  ],
};

export default config;

And run the test suite again:

npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

By default, jest-junit will save the .xml report in your project root directory, with the file name junit.xml:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="5" failures="0" errors="0" time="28.265">
  <testsuite name="Login" errors="0" failures="0" skipped="1" timestamp="2023-01-15T17:33:04" time="28.24" tests="5">
    <testcase classname="Login annotates" name="Login annotates" time="0">
      <skipped/>
    </testcase>
    <testcase classname="Login Simple Form should click the button" name="Login Simple Form should click the button" time="7.597">
    </testcase>
    <testcase classname="Login Simple Form should type in the email address" name="Login Simple Form should type in the email address" time="5.468">
    </testcase>
    <testcase classname="Login Simple Form should type in the password" name="Login Simple Form should type in the password" time="5.356">
    </testcase>
    <testcase classname="Login Simple Form should click the login button" name="Login Simple Form should click the login button" time="4.937">
    </testcase>
  </testsuite>
</testsuites>

The jest-junit can be configured to format the resulting .xml. Try to change the test/jest.config.ts, and run the test again:

import type { Config } from '@jest/types';

const config: Config.InitialOptions = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  setupFilesAfterEnv: ['./helper/jest.setup.ts'],
  sandboxInjectedGlobals: [
    'Math',
  ],
  reporters: [
    "default",
    ['jest-junit',{
      "suiteName": "login test suite",
      "outputDirectory": "reports",
      "outputName": "login-test",
      "uniqueOutputName":"true",
      "ancestorSeparator": " › "
    }]
  ],
};

export default config;

To see more options for the configuration, please refer to the official README of jest-junit.

2.2. Using jest-html-reporter with AskUI

If we want to have a report that is more friendly and readable than XML, then we could give jest-html-reporter a try. This reporter generates a HTML file that, if opened in a web browser, visualises the test result in a much more human-friendly way.

Run the command below to install the jest-html-reporter:

npm i jest-html-reporter

Then, go to test/jest.config.ts and change it as below:

import type { Config } from '@jest/types';

const config: Config.InitialOptions = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  setupFilesAfterEnv: ['./helper/jest.setup.ts'],
  sandboxInjectedGlobals: [
    'Math',
  ],
  reporters: [
    "default", [
      "jest-html-reporter", {
        "outputPath": "./reports/test-report.html",
        "pageTitle": "Automation Test with AskUI",
        "includeFailureMsg": true
      }
    ],
  ],
};

export default config;

Again, refer to the official README to see more options for the configuration.

Then let's try to run the test suite again, but for the sake of testing the new reporter, let's make a test step to fail.

Insert this line within any test step in the test code:

    //... some test code

    throw new Error("I'm an error!");

    //... some test code

After running the test suite again, you will have an HTML file reports/test-report.html in your project directory. Open the HTML file in your web browser:

2.3. Using jest-html-reporters with AskUI

jest-html-reporters is another reporter that generates an HTML that summarizes the test result run within Jest. It might be similar to the above-mentioned reporter. But this one, jest-html-reporters, has a feature that can attach an image to the report, thus can assist your debugging process, especially for UI tests.

Run the command below to install the jest-html-reporter:

npm i jest-html-reporters

Now let's configure the reporter in jest.config.ts:

import type { Config } from '@jest/types';

const config: Config.InitialOptions = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  setupFilesAfterEnv: ['./helper/jest.setup.ts'],
  sandboxInjectedGlobals: [
    'Math',
  ],
  "reporters": [
    "default",
    ["jest-html-reporters", {
      "publicPath": "./html-report",
      "filename": "report.html",
      "openReport": true,
      "expand": true,
      "pageTitle": "My Test Report",
    }]
  ]
};

export default config;

The options used in this configuration are:

"publicPath": The basepath for the saved report file as a string.
"filename": The file name for the saved report file as a string.
"openReport": If true, it will open the generated HTML report after the test run is finished.
"expand": If true, it will expand all the dropdown list in the table contained in the report
"pageTitle": The header title for the HTML report.

The options used in the above configuration are those I found useful for most of the test cases. But since there are more options available, check out the official doc if you feel keen to use it.

The feature of the jest-html-reporters that could be nicely combined with AskUI is the feature to attach images to the report, since AskUI, as a UI test automation tool, functions on top of the screenshot of the given display.

To use this feature, we import two helper functions from the reporter's package:

// Add this line to the top of the test code
import { addAttach, addMsg } from 'jest-html-reporters/helper';

Here is an example of how to use this feature with AskUI. We will attach a screenshot of the test step to the report:

import { aui } from './helper/jest.setup';
import { addAttach, addMsg } from 'jest-html-reporters/helper';

describe('Login',()=>{
  describe('Simple Form',()=>{
    it('should click the button',async ()=>{

      // Frist, get all the information from the annotation
      // This will also save an interactive HTML file to the 'report/' folder
      const annotation = await aui.annotate();

      // The screenshot is contained as a string in 'base64' format
      // Create a buffer with the base64 image
      let buf = Buffer.from(annotation.image.split('base64,')[1], 'base64');

      // Attach the image to the test report.
      await addAttach({
          attach: buf,
          description: '',
      });

      // We filter only the text elements, and add it to the reporter
      const textonly = annotation.detected_elements.filter((e) => e.name=="text");

      // Attach the detected text elements to the reporter
      await addMsg({
          message: JSON.stringify(textonly, undefined, '\t'),
          context:''
      });

      await aui.moveMouseTo().button().contains().text().withText('Simple Form Auth').exec();
      await aui.mouseLeftClick().exec(); // This first click is to get the focus on the test app
      await aui.mouseLeftClick().exec(); // Thereafter, we click the button
    });
  });
});

After running this test code, your web browser will open and show the generated report. Click on the Info button on the right end of the table:

It will show the attached image beside the text elements we've added:

3. Conclusion

Although we covered only two ready-made jest reporters in this article, reporter in jest is fully configurable and customizable. To properly configure the testing pipeline, you could wish to have a custom reporter that fits nicely into your pipeline. As AskUI is a well-suited Typescript Library, combining it with jest and proper reporters can become a huge benefit to scaffolding robust test automation.

If you got any issues while following this tutorial, don't hesitate to ask our Discord Community!

Understanding Text Filters in AskUI

HaramChoi — Fri, 20 Jan 2023 08:26:25 +0000

This tutorial demonstrates different methods to handle text elements with AskUI. AskUI offers four different methods:

containsText()
withExactText()
withText()
withTextRegex()

Requirements

AskUI installed and set up.

For a convenient demonstration, we will use a Flutter web demo provided by Flutter.

Basic Text Matching

The simplest way to interact with a text element is to use withText(). Go to the demo app page and run the code below:

await aui.moveMouseTo().text().withText('matrial').exec();
await aui.mouseLeftClick().exec();

withText() tries to find a text that matches the whole sequence. In most test cases, you will want to stick to this method, as it supports Fuzzy Matching and tolerates misspelled text. Note that the above example code has two typos. matrial doesn't match the text in the demo app, which is Material, although AskUI will find the text element that roughly matches the text on the screen.

Match a Substring within a Text

Even though the the method withText() is handy and quite reliable, you might face a test case where you know only a fraction of the text element that you want to interact with. In such a case, containsText() is the method you might want to use:

await aui.moveMouseTo().text().containsText('Bottom').exec();
await aui.mouseLeftClick().exec();

Be aware that even if the method containsText() also supports Fuzzy Matching, it won't match the whole sequence by just a few characters. Try to use this code with the given demo app:

// this will fail
await aui.moveMouseTo().text().containsText('Bottm').exec();

You will notice that AskUI fails to match the given text Bottm, whereas this code will work:

// this will success
await aui.moveMouseTo().text().containsText('Bottm appbar').exec(); 
// this will also success
await aui.moveMouseTo().text().containsText('Bottom').exec();

The biggest difference between withText() and containsText() is whether it matches the text as a whole sequence or not. Matching many texts with a repeating affix could be a practical use case for the containsText().

It is recommended to experiment enough with these methonds to find a better option that suits your specific case, since it's not easy to predict if the given text can be fuzzy-matched with target texts.

Match the Exact Text

If you already know what text you are looking for, or if there are too many similar text elements, we could use the method withExactText().

From the main page of the demo app, go to Material->Data tables. You will see a table with different foods given with nutrition factors for each of them.

Let's say that we want to click on the items that has exactly 25.0 gm of Fat. In our demo app, only the Doughnut is the matching item. Run this code and see how withText() matches the text:


// Use this helper function to calculate the centroid of the detected elements.
function getCentroid(element: any): any {
    let x = (element.bndbox.xmax - element.bndbox.xmin) / 2.0 + element.bndbox.xmin;
    let y = (element.bndbox.ymax - element.bndbox.ymin) / 2.0 + element.bndbox.ymin;
    return { x: x, y: y };
}

// Find all the text elements that matches '25.0'
const elts = await aui.get().text().withText('25.0').exec();

// Then, iterate through the found elements and click on them
for(let i=0; i<elts.length; ++i){
    const centroid = getCentroid(elts[i]);
    await aui.moveMouse(Math.round(centroid.x), Math.round(centroid.y)).exec();
    await aui.mouseLeftClick().exec();
}

You will see that AskUI clicks not only the 25.0 but also the 26.0, which is the fat of the Apple pie. The result of this test code may differ in your case, because of the different screen resolution and the rendered-size of the demo app.

It will give you a clear idea where you will need to use the method withExactText() instead of withText(). Try to run the same code after replacing the withText() to withExactText():

// Find all the text elements that matches '25.0' exactly
const elts = await aui.get().text().withExactText('25.0').exec();

// Then, iterate through the found elements and click on them
for(let i=0; i<elts.length; ++i){
    const centroid = getCentroid(elts[i]);
    await aui.moveMouse(Math.round(centroid.x), Math.round(centroid.y)).exec();
    await aui.mouseLeftClick().exec();
}

Match Text with Regular Expression

The method withTextRegex() supports Regular Expression to match any text in the most flexible way. Although it might be tricky to use regex due to its esoteric appearance, it is maybe one of the best solution when it comes to character matching.

On the same page of the demo app, let's say that we want to click on the items whoes Calorie is between 300 and 500 (cal>=300 && cal<500). Since regex doesn't support numeric comparison, we will try to match the digits in a sequence:

// Find all the text that matches the expression
const cals = await aui.get().text().withTextRegex('[3-4][0-9]{2}').exec();

// Then, iterate through the found elements and click on them
for(let i=0; i<cals.length; ++i){
    const coord = getCentroid(cals[i]);
    console.log(cals[i].text, coord);
    await aui.moveMouse(Math.round(coord.x), Math.round(coord.y)).exec();
    await aui.mouseLeftClick().exec();
}

The regular expression [3-4][0-9]{2} means,

[3-4]: Match one character between 3 and 4.
[0-9]: Match one character between 0 and 9.
{2}: Repeat the previous expression ([0-9]) two times.

As the result, it will try to match every text that has a sequence starting with the digit 3 or 4, and then has any two digits in a row.

Conclusion

When using AskUI for automated tests, text elements are playing a big role, since they typically appear more distinctively than other elements such as icons or textfields. Hence, knowing the benefits of using different text filters can become critical in scaffolding a robust test suite.

If you got any issue by following this tutorial, or if you want to share your AskUI use case, you can join our community on Discord!

Automate Multiple Devices with AskUI

HaramChoi — Mon, 16 Jan 2023 09:39:48 +0000

Overview

This tutorial shows how to automate multiple devices on the same network by using AskUI library. After following this tutorial, you will be able to automate more than one device across different platforms, whether Linux, macOS, Windows or Android, with a single setup with AskUI library.

In fact, there are many automation tools in the wild that you can use to automate different devices, although most of them require different configurations and different test codes for different platforms. By using AskUI library, an automation tool that operates on the OS level, you can simply use the same test code for any application running on different devices even without so much change. And this makes the AskUI a powerful Cross-platform automation tool.

Let's have a look at the AskUI library and see how we can accomplish a Cross-platform/-device automation test🔥

📌 The following tutorial assumes that you have already installed and set up the AskUI library on your local device. The code for the configuration is based on the generated test code: npx askui init. See Getting Started for more details.

1. Download and Prepare the AskUI Binary for Each Device

If you already used the AskUI library once, then the binary for your platform already exists in the node_modules/ directory, as the binary gets automatically downloaded if an instance of UiController gets initialized. See here for more details on the AskUI UI Controller.

Follow this path and confirm that the binary exists:

# Windows
node_modules/askui/dist/release/latest/win32/askui-ui-controller.exe

# macOS
node_modules/askui/dist/release/latest/darwin/askui-ui-controller.app/Contents/MacOS/askui-ui-controller

# Linux
node_modules/askui/dist/release/latest/linux/askui-ui-controller

If the remote device runs the same platform as the local device, simply copy the binary from the local device to the remote device.
If the remote device runs a different platform, then download the binary for the respective platform: Windows | macOS | Linux

Android Only:

There is no need to save the binary to Android devices. They are controlled by the AskUI UI Controller running on the local device (desktop).

Be sure that your Android device is discoverable by the adb from your local device:

# Run this command to confirm that your Android device is discoverable
adb devices

If you don't have the ADB installed on your local device, set it up by following this tutorial.

Use the commands below, if you want to connect your Android device via ADB wirelessly:

# Make sure that the `USB Debugging Mode` is enabled in the Android device.
# Connect the Android device with a USB cable, and run this command:

adb devices # will print the <device-id>

adb -s <device-id> tcpip 9000 # replace <device-id> with your device-id

adb -s <device-id> connect <local-ip-address>:9000 # replace the <local-ip-address>

# Run this command, if you want to check the local ip address of the android device
adb -s <device-id> shell ip -f inet addr show wlan0

# Now you can disconnect the USB cable from the Android device.
# Confirm that the Android device is recognised by adb wirelessly.
adb devices

2. Configure the `jest.setup.ts`

Figure out the local IP address of the remote device, and then change the <local-ip-address> of the jest.setup.ts:

import { UiControlClient, UiController } from 'askui';

let localDevice: UiControlClient;
let remoteDevice: UiControlClient;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {

    // Get your AskUI credentials from https://app.askui.com/workspaces
const credentials = {
    workspaceId: '<your-workspace-id>',
    token: '<your-token>',
}

    // This client will communicate with
    // the controller running on this local device.
localDevice = await UiControlClient.build({ 
    uiControllerUrl: "ws://127.0.0.1:6769",
    credentials: credentials,
});

await localDevice.connect();

    // This client will communicate with
    // the controller running on the remote device.
    // Replace the <local-ip-address>
    // In case of Android device, replace it with 127.0.0.1
remoteDevice = await UiControlClient.build({    
    uiControllerUrl: "ws://<local-ip-address>:6769", 
    credentials: credentials,
});

await remoteDevice.connect();
});

afterAll(async () => {
localDevice.close();
remoteDevice.close();
});

export { localDevice, remoteDevice };

3. Run the Controller on Each Device

Run the binary AskUI UI Controller on the local and remote devices with the following command:

# Windows powershell or cmd
askui-ui-controller.exe --host 0.0.0.0 -d 0 -m

# macOS / Linux terminal
askui-ui-controller --host 0.0.0.0 -d 0 -m

If running successfully, you should see the logs printed on the terminal, e.g:

[2023-01-02 17:31:19.634 +0100] DEBUG (AskuiUiController): Window is minimized.
[2023-01-02 17:31:19.639 +0100] INFO (AskuiUiController): Selecting display number 0.
[2023-01-02 17:31:19.641 +0100] INFO (AskuiUiController): Successfully started.

Android Only:

If your remote device is an Android device, run the AskUI UI Controller on the local device (desktop) with an extra option as below:

# Windows powershell or cmd
askui-ui-controller.exe --host 0.0.0.0 -d 0 -m -r android

# macOS / Linux terminal
askui-ui-controller --host 0.0.0.0 -d 0 -m -r android

Make sure that your local device (desktop) is running TWO DIFFERENT askui-ui-controller, if you want to control the local device and the Android device at the same time.

(option) Running Multiple Android Devices

The AskUI UI Controller considers all Android devices as a single device with multiple displays. See the screenshot below.

Set the -d 0 option of the binary differently for each Android device.

# Windows
# For the first Android device
askui-ui-controller.exe --host 0.0.0.0 -d 0 -m -r android
# For the second Android device
askui-ui-controller.exe --host 0.0.0.0 -d 1 -m -r android

# macOS / Linux
# For the first Android device
askui-ui-controller --host 0.0.0.0 -d 0 -m -r android
# For the second Android device
askui-ui-controller --host 0.0.0.0 -d 1 -m -r android

4. Write the Test Code

Write the test code in test/my-first-askui-test-suite.test.ts:

import { localDevice, remoteDevice } from './helper/jest.setup';

describe('jest with askui', () => {
    it('should work with multiple devices', async () => {
        const everyElement_remote = await remoteDevice.getAll().exec();
        console.log(everyElement_remote);

        await localDevice.moveMouse(500,500).exec();
        const everyElement_local = await localDevice.getAll().exec();
        console.log(everyElement_local);
    });
});

5. Run the Test Code

Run the command below to run the AskUI test code:

npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

6. Conclusion

Now you should be able to automate multiple devices in the network. If you got any issues while following this tutorial, don't hesitate to ask our Discord Community!

Tutorial: AskUI Automation Example with Flutter

HaramChoi — Sun, 01 Jan 2023 12:40:41 +0000

This tutorial shows how to use AskUI to automate an Android app built with Flutter. We provide the source code for the Flutter demo app used in this tutorial. You can find the source code on our GitHub repository. Set up the demo app by following the instructions below. This tutorial assumes that you already have your Android device prepared. It can be a real Android device or an Android Emulator.

If you haven't set up your Android device or Emulator yet, follow this post.

This tutorial includes:

Build and Run the Flutter Demo App
Set up the ADBKeyboard
Click/Touch Automation
Type Automation
Swipe Automation

Live Demo in Action (playback speed x3)

Setup

The source code for the Flutter demo app used in this tutorial is provided in this repository.

1. Build and Run the Flutter Demo App

Install Flutter

Clone this repository and run flutter create demo_app within the directory:

git clone https://github.com/askui/flutter-example-automation
cd flutter-example-automation
flutter create demo_app
cd demo_app

Install dependencies for the Flutter demo app:

# run this command inside the flutter project directory `demo_app/`
flutter pub add camera intl

To use the camera, we need to set the minSdkVersion in android/app/build.gradle:

// inside the 'android/app/build.gradle' set the 
// 'minSdkVersion' to 21
    default config {
        ...
        minSdkVersion 21
        ...
    }

(optional) The app is ready to be built but will throw deprecation warnings. If you want to clear the deprecation warnings, follow this step. See this issue

# change the respective part inside the 'pubspec.yaml'
dependencies:
camera:
git:
  url: https://github.com/flutter/plugins
  path: packages/camera/camera
  ref: 9e46048ad2e1f085c1e8f6c77391fa52025e681f

Run the demo app:

# Make sure that your Android device/Emulator is connected.
flutter run

Now you should see the demo app running on your Android device.

2. Set up the ADBKeyboard

In this example, we are going to automate the typing on the Android device. To let AskUI fluently type as desired, we will use a virtual keyboard that handles the keyboard input via adb: ADBKeyboard.apk

Download the ADB-Keyboard package (Important: Version 2.0): Link to GitHub Repo
Unzip it.

Find your device:

# make sure that your android device is connected, and the USB debugging mode is enabled.
adb devices

Install the ADBkeyboard on the device:

# inside ADBKeyBoard-2.0/
adb -s <your device id> install ADBKeyboard.apk

Configure the ADB Keyboard:

# inside ADBKeyBoard-2.0/
adb -s <your device id> shell settings put secure default_input_method com.android.adbkeyboard/.AdbIME

Enable the ADB Keyboard:

# inside ADBKeyBoard-2.0/
adb -s <your device id> shell ime enable com.android.adbkeyboard/.AdbIME

To check if it is enabled:
Click on a textfield in an app and see if the ADB Keyboard {ON} notification is shown at the bottom of the screen.

3. Setup AskUI

Setup AskUI by following the Getting Started Guide.

We need to run the UiController directly with an extra argument to specify the runtime mode, as the current version of AskUI(ver. 0.6) doesn't provide the API for running it with the runtime argument yet.
From within your npm project path, go to the directory that contains the askui-ui-controller binary, and run ./askui-ui-controller -r android

cd <YOUR_PROJECT_DIRECTORY>/node_modules/askui/dist/release/latest/<YOUR_PLATFORM>
./askui-ui-controller -r android
# for example, Mac OS
cd node_modules/askui/dist/release/latest/darwin/askui-ui-controller.app/Contents/MacOS/
./askui-ui-controller -r android
# If you can't find the binary as described above,
# then you might have askui freshly installed and haven't run it yet.
# The binary gets downloaded as the askui test code runs.
# Run the command below to run the askui test code:
npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

If you got them both(emulator and UiController) running, then we are ready to go for the UI automation.

If you are working with the test code from our official docs, then you need to deactivate a few lines of the code in test/helper/jest.setup.ts that is running the UiController, because we are already running it manually in the previous step:

// file location: test/helper/jest.setup.ts
// comment out every line that uses uiController

import { UiControlClient, UiController } from 'askui';

// uiController: UiController;

let aui: UiControlClient;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {
//   uiController = new UiController({
//     /**
//      * Select the display you want to run your tests on, display 0 is your main display;
//      * ignore if you have only one display
//      */
//     display: 0,
//   });

//   await uiController.start();

aui = await UiControlClient.build({
    credentials:{
        workspaceId: 'YOUR_WORKSPACEID_FROM_USER_PORTAL',
        token: 'YOUR_TOKEN_FROM_USER_PORTAL',
    }
});

await aui.connect();
});

afterAll(async () => {
//   await uiController.stop();

aui.close();
});

export { aui };

Breaking Down the Askui Test Code

This chapter will walk you through the provided askui-test/demo-automation.ts step by step.
The test is divided into three parts, and each test is run for each tab within the demo app:

Outline tab
- Find a textfield and type in characters.
- Find a button and press it.
Datepicker tab
- Select a desired date within the date picker widget.
Camera tab
- Open the camera and push the record button.

0. General Tips for Using AskUI as a More Friendly Tool:

Try to annotate : Use await aui.annotateInteractively(); or await aui.annotate(); in order to see how AskUI is understanding the visible elements on your screen. By using await aui.annotate(), the result of the annotation will be saved in report/ as an HTML file.
Be aware of the screen size of your device : AskUI understands your application based on the screen shown and captured. Therefore, on some occasions, you may want to know your screen size to e.g. properly scroll or swipe within your application. You may need to change the numbers for the input swipe command within the provided test code so that it suits the screen size of your device.
Try to select the elements by their text

tip: If you are using a device with a bigger screen e.g. Tablet, then the screen of your test device (real android device or emulator) might be big enough to see the whole page without scrolling.

1. Click and Type

The test code is within the askui-test/demo-automation.ts. Copy and paste the code into your AskUI test code.

We start the test automation from the very first tab of our demo app.

First tab of the demo app

To type into a textfield, we first need to get focus on the desired textfield. We can achieve it by running the code below:

// click on the textfield and type characters
// repeat this as many times as the textfields
await aui.click().text().withText('Enter your username').exec();
await aui.type('askui').exec();

As we have multiple of textfields in our demo app, we can iterate the same procedure for each of them:

// click on the textfield and type characters
// repeat this as many times as the textfields
await aui.click().text().withText('Enter your username').exec();
await aui.type('askui').exec();

// click and type the email address
await aui.click().text().withText('Enter your email').exec();
await aui.type('askui@askui.com').exec();

// Click and type the address
await aui.click().text().withText('Enter your address').exec();
await aui.type('Haid-und-Neu-Straße 18').exec();

// Pressing enter is the equivalent of pressing the return button on the on-screen-keyboard
// This gets rid of the focus from the textfield
await aui.pressAndroidKey('enter').exec();

After filling up the textfields, we can push the buttons at the bottom of the page:

// Press the 'Submit' button
await aui.click().text().withText('Submit').exec();

// We will have a popup window that has two buttons. Press the 'Refuse' button
await aui.click().text().withText('Refuse').exec();

// Here we press multiple toggle buttons one by one
await aui.click().text().withText('Banana').exec();
await aui.click().text().withText('Mango').exec();
await aui.click().text().withText('Sunny').exec();
await aui.click().text().withText('Rainy').exec();
await aui.click().text().withText('Windy').exec();

// Attention for swiping!
/*  Swipe/scroll within the page
  - execOnShell() can run shell commands within the device via adb.
  - Note that, you have to adjust the four numeric parameters,
          in order to make it fit to your device's screen.
  - The syntax is:
      input swipe <startX> <startY> <endX> <endY>
  - Depending on the screen size of your device,
          the coordinates should stay within the scrollable/swipeable area of the app.
          i.e. the 'Tabbar' at the top of the demo app is not scrollable.
*/

// Here we swipe the page two times in a row
await aui.execOnShell('input swipe 1000 1000 100 1000').exec();
await aui.execOnShell('input swipe 1000 1000 100 1000').exec();

2. Datepicker

After running the test code above, we should see the demo app swiped to the Datepicker tab.

Datepicker tab of the demo app

First, we scroll the page down in order to see the date picker widget:

// First, we type in the desired values into the textfields.
await aui.click().text().withText('Title').exec();
await aui.type('My vacation plan').exec();
await aui.click().text().withText('Description').exec();
await aui.type('0. Drink a lot of water').exec();
await aui.pressAndroidKey('tab').exec();

After running the test code above, we should see two different date picker widgets that are represented with edit buttons:

// Second, we select a desired date from the Datepicker widget.
// Notice how we select the icon 'chevron right/left' to shift the calendar month.
await aui.click().text().withText('edit').nearestTo().text().withText('Depature').exec(); // this will open up the calendar
await aui.click().icon().withText('chevron right').exec();

// within the calendar, we push the > icon on the top right corner
await aui.click().icon().withText('chevron right').exec();
await aui.click().text().withText('7').exec(); // select 7
await aui.click().text().withText('ok').exec(); // then, press OK

// Repeat the step for the next Datepicker widget.
await aui.click().text().withText('edit').nearestTo().text().withText('Return').exec();
await aui.click().icon().withText('chevron right').exec();
await aui.click().icon().withText('chevron right').exec();
await aui.click().icon().withText('chevron right').exec();
await aui.click().text().withText('5').exec();
await aui.click().text().withText('ok').exec();

Let's go further below to the bottom of the page, and then interact with more interfaces:

// click and check the checkbox
await aui.click().checkboxUnchecked().nearestTo().text().withText('Brushed Teeth').exec();

// finally, we turn on the switch
await aui.click().switchDisabled().nearestTo().text().withText('Enable feature').exec();

// Swipe the page to the Camera tab
await aui.execOnShell('input swipe 1000 1000 100 1000').exec();

3. Take a Picture with the Camera

In the final tab Camera, we can launch the device's camera and take a picture by pressing the record button.

Camera tab of the demo app

// Click on the button 'Take a Picture', then it will launch the camera
await aui.click().button().contains().text().withText('Take a Picture').exec();

// Notice how we select the record button.
// Our demo-app intends to have the record button in a circular shape.
// So we can look for an icon which is a 'circle'
// It might be different in other applications.
await aui.click().icon().containsText('circle').exec();

4. Complete Test Code

This is the complete test code that runs AskUI to automate our test:

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
    xit('annotate', async () => {
        await aui.annotateInteractively();
    });

    it('should fill up the textfields and push buttons', async () => {
        // click on the textfield and type characters
        // repeat this as many times as the textfields
        await aui.click().text().withText('Enter your username').exec();
        await aui.type('askui').exec();

        // click on the textfield and type the email        
        await aui.click().text().withText('Enter your email').exec();
        await aui.type('askui@askui.com').exec();

        // Click and type the address
        await aui.click().text().withText('Enter your address').exec();
        await aui.type('Haid-und-Neu-Straße 18').exec();

        // Pressing enter is the equivelant to pressing the return button on the on-screen-keyboard
        // This gets rid of the focus from the textfield
        await aui.pressAndroidKey('enter').exec();

        // Press the 'Submit' button
        await aui.click().text().withText('Submit').exec();

        // We will have a popup window that has two buttons. Press the 'Refuse' button
        await aui.click().text().withText('Refuse').exec();

        // Here we press multiple of toggle buttons one by one
        await aui.click().text().withText('Banana').exec();
        await aui.click().text().withText('Mango').exec();
        await aui.click().text().withText('Sunny').exec();
        await aui.click().text().withText('Rainy').exec();
        await aui.click().text().withText('Windy').exec();


        // Attention for swiping!
        /*  Swipe/scroll within the page
            - execOnShell() can run shell commands within the device via adb.
            - Note that, you have to adjust the four numeric parameters,
            in order to make it fit to your device's screen.
            - The syntax is:
                input swipe <startX> <startY> <endX> <endY>
            - Depending on the screen size of your device,
            the coordinates should stay within the scrollable/swipeable area of the app.
            i.e. the 'Tabbar' at the top of the demo app is not scrollable.
        */

        // Here we swipe the page two times in a row
        await aui.execOnShell('input swipe 1000 1000 100 1000').exec();
        await aui.execOnShell('input swipe 1000 1000 100 1000').exec();
    });


    it('should pick the dates', async () => {
        // First, we type in the desired values into the textfields.
        await aui.click().text().withText('Title').exec();
        await aui.type('My vacation plan').exec();
        await aui.click().text().withText('Description').exec();
        await aui.type('0. Drink a lot of water').exec();
        await aui.pressAndroidKey('tab').exec();

        // Second, we select a desired date from the Datepicker widget.
        // Notice how we select the icon 'chevron right/left' to shift the calendar month.
        await aui.click().text().withText('edit').nearestTo().text().withText('Depature').exec(); // this will open up the calendar
        await aui.click().icon().withText('chevron right').exec(); // within the calendar, we push the > icon on the top right corner
        await aui.click().icon().withText('chevron right').exec();
        await aui.click().text().withText('7').exec(); // select 7
        await aui.click().text().withText('ok').exec(); // then, press OK


        // Repeat the step for the next Datepicker widget.
        await aui.click().text().withText('edit').nearestTo().text().withText('Return').exec();
        await aui.click().icon().withText('chevron right').exec();
        await aui.click().icon().withText('chevron right').exec();
        await aui.click().icon().withText('chevron right').exec();
        await aui.click().text().withText('5').exec();
        await aui.click().text().withText('ok').exec();

        // click and check the checkbox
        await aui.click().checkboxUnchecked().nearestTo().text().withText('Brushed Teeth').exec();

        // finally, we turn on the switch
        await aui.click().switchDisabled().nearestTo().text().withText('Enable feature').exec();

        // Swipe the page to the Camera tab
        await aui.execOnShell('input swipe 1000 1000 100 1000').exec();

    });

    it('should take a picture', async ()=>{
        // Click on the button 'Take a Picture', then it will launch the camera
        await aui.click().button().contains().text().withText('Take a Picture').exec();

        // Notice how we select the record button.
        // Our demo-app intends to have the record button in a circular shape.
        // So we can look for an icon which is a 'circle'
        // It might be different in other applications.
        await aui.click().icon().containsText('circle').exec();
    });
});

5. Conclusion

After following through this example, you should be able to automate the interaction with the provided demo app. Although this example specifically provides a demo app built with Flutter, the overall method of using AskUI should also work with any mobile app running on an Android device.

If you got an issue while following this example, or in case you would like to share your test case, don't hesitate to join our community on Discord!

Tutorial: Automate Web Search on Android Devices with AskUI

HaramChoi — Tue, 15 Nov 2022 08:41:51 +0000

In this tutorial, we will automate web browser searching on Android devices. This tutorial assumes that you have already set up your Android test device, alongside the Android Development Environment. If you haven't set it up yet, you can check out our previous tutorial.

Live Demo in Action (playback speed x3)

You will also need the Node.js to follow this tutorial, as the automation tool, AskUI, will be installed as an npm package. In case you don't have it installed yet, go to https://nodejs.org/ and install it.

Requirements

Android Studio installed, or without it. See this post.
Node.js installed

1. Prepare the AskUI Test Environment

Install and initialize AskUI

First, go to the directory where you have your npm project. If you don't have one, you can create it with npm init

Then, use the commands below to install AskUI alongside a few additional tools:

npm i -D askui typescript ts-node @types/jest ts-jest jest
npx askui init # this generates a test suite within the project directory

After creating the AskUI test suite, add your credential in helper/jest.setup.ts:

 aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your workspace id>',
      token: '<your access token>',
    }
  });

💡 AskUI credentials

You can get your AskUI credential from the AskUI user portal for free.

If you have any issues while setting up AskUI, you can have a look at the more descriptive Getting Started tutorial, or just drop by our Discord and ask the community.

Configure AskUI for Android

We need to run the UiController manually with an extra argument to specify the runtime mode, as the current version of AskUI(ver. 0.4) doesn't provide the API for running it with the runtime option yet.

From within your npm project path, go to the directory that contains the askui-ui-controller binary, and run ./askui-ui-controller -r android

cd <YOUR_PROJECT_DIRECTORY>/node_modules/askui/dist/release/latest/<YOUR_PLATFORM>
./askui-ui-controller -r android

# for example, Mac OS
cd node_modules/askui/dist/release/latest/darwin/askui-ui-controller.app/Contents/MacOS/
./askui-ui-controller -r android

# If you can't find the binary as described above,
# then you might have AskUI freshly installed and haven't run it yet.
# The binary gets downloaded as the AskUI test code runs.
# Run the command below to run the AskUI test code:
npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

Thereafter, we have to change a few lines of the generated test code, as the code ships with the code that creates the UiController.

Go to helper/jest.setup.ts and deactivate every line that is using uiController:

import { UiControlClient, UiController } from 'askui';

// Server for controlling the operating system
// let uiController: UiController;

// Client is necessary to use the AskUI API
// eslint-disable-next-line import/no-mutable-exports
let aui: UiControlClient;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {
//   uiController = new UiController({
//     /**
//      * Select the display you want to run your tests on, display 0 is your main display;
//      * ignore if you have only one display
//      */
//     display: 0,
//   });

//   await uiController.start();

  aui = await UiControlClient.build({
    credentials: {
      workspaceId: myworkspaceid,
      token: mytoken,
    }
  });

  await aui.connect();
});

afterAll(async () => {
//   await uiController.stop();

  aui.close();
});

export { aui };

2. Try Annotating

Make sure that your Android device is connected, or if you are using the Android Emulator, make sure that it is open and running on your computer.

AskUI provides a feature where you can monitor how the visible elements are understood by AskUI.
Try to change the code within test/my-first-askui-test-suite.test.ts:

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('should show the annotation', async () => {
    await aui.annotateInteractively();
  });
});

and run npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

image: Annotated screenshot of the Android Emulator

💡 Annotation is Interactive
Try to hover your mouse on the red bounding box. It will let you know how to manipulate that element via AskUI

3. Automate Web Searching

Now we are good to go for the actual automation process.
The automation consist of three steps:

Open Chrome
Select the search bar and type 'spacecraft'
Click on the desired search result

1) Open Chrome

To open Chrome, we first have to figure out how we can let AskUI know where to click on.

image: Annotated chrome icon

As we can see in the annotated information, the Chrome icon is recognized as an icon: undo. Indeed, we could also tell AskUI to select the icon: undo, but we will try to do it in a more precise way.

What we gonna do is:

(1) Select the search bar
(2) Type 'chrome'
(3) Select the icon above the text 'chrome'

This sideway approach will give us a more consistent result because typing chrome in the search bar will give us a more clearly understandable visual element.

Try to change your code according to this:

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('should open chrome', async () => {
    // Here we try to avoid failing our test by using the try-catch phrase.
    // It is because, 'textfield' and 'textarea' are seeming quite the same, even if they are tagged with different names.
    try {
        await aui.click().textfield().exec();    
    } catch (error) {
        await aui.click().textarea().exec();
    }
    // Type the desired keyword into the search bar
    await aui.type('chrome').exec();
    // We wait for 1500 milliseconds, to make sure that the search result has been loaded before askui start to look for the search result. 
    await aui.waitFor(1500).exec(); 
    // Then click the icon that is above the text 'chrome'
    await aui.click().icon().above().text().withText('chrome').exec();
  });
});

and run npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

Now you will be able to see that Chrome has been opened.

2) Select the Search Bar and Type 'spacecraft'

Let's select the search bar of chrome, and type our desired keyword in there.

image: Main page of the Chrome app

Add this code block to the bottom of our code:

// We first look for the search bar. Depending on the system language of your device, the default text within the search bar may differ.
await aui.click().text().withText('search or type web address').exec();

// Type our desired keyword and hit enter
await aui.type('spacecraft').exec();
await aui.pressAndroidKey('enter').exec();

In some cases, when searching in google, you will be asked to give consent for the cookies.
To avoid our test from failing, we have to examine whether we got a pop-up for the cookie consent or not:

try {
    // The `expect()` examines whether a specific element is detected or not.
    // A command starting with `expect()` must always end with `exists()` or `notExists()`
    await aui.expect().text().containsText('cookies').notExists().exec();
} catch (error) {
    await aui.click().text().withText('read more').exec();
    await aui.waitFor(1000).exec(); // wait until the scrolling animation has been finished
    await aui.click().text().withText('accept all').exec();
}
  // From here, we can write our next test code

3) Click on the Desired Search Result

image: Search result of 'spacecraft'

After clearing the cookie consent pop-up, we can see and click our desired search result. In our test case, we will look for the result from Wikipedia:

// We ask the AskUI to click the text that contains 'wikipedia' which is the text that is nearest to the text containing 'wikipedia.org'
await aui.click().text().containsText('wikipedia').nearestTo().text().containsText('wikipedia.org').exec();

Pay attention to the command nearestTo() that is interconnecting two different text elements.

AskUI offers several Relational commands, which enable you to select the desired element in an intuitively understandable way:

above()
below()
contains()
in()
leftOf
rightOf
nearestTo

💡 About withText() and containtsText()

You might wonder how withText() and containsText() differ. withText() tries to match the given text as the whole sequence, whereas containtsText() tries to match the given text as a sub-text within the elements. Generally speaking, containsText() can be handier to match the text roughly, but you might face a test case where you want to find a specific text as a whole sequence.

4. Complete Test Code

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('should search spacecraft in chrome', async () => {
    // Here we try to avoid failing our test by using the try-catch phrase.
    // It is because, 'textfield' and 'textarea' are seeming quite the same, even if they are tagged with different names.
    try {
        await aui.click().textfield().exec();    
    } catch (error) {
        await aui.click().textarea().exec();
    }
    // Type the desired keyword into the search bar
    await aui.type('chrome').exec();
    // We wait for 1500 miliseconds, to make sure that the search result has been loaded before askui start to look for the search result. 
    await aui.waitFor(1500).exec(); 
    // Then click the icon that is above the text 'chrome'
    await aui.click().icon().above().text().withText('chrome').exec();

    // We wait the Chrome app to be launched
    await aui.waitFor(1500).exec();

    // We first look for the search bar. Depending on the system language of your device, the default text within the search bar may differ.
    await aui.click().text().withText('search or type web address').exec();

    // Type our desired keyword and hit enter
    await aui.type('spacecraft').exec();
    await aui.pressAndroidKey('enter');

    // We wait for the search result to be loaded
    await aui.waitFor(3000).exec();

    try {
        // The `expect()` examines whether a specific element is detected or not.
        // A command starting with `expect()` must always end with `exists()` or `notExists()`
        await aui.expect().text().containsText('cookies').notExists().exec();
    } catch (error) {
        await aui.click().text().withText('read more').exec();
        await aui.waitFor(1000).exec(); // wait until the scrolling animation has been finished
        await aui.click().text().withText('accept all').exec();
    }

    // We ask the AskUI to click the text that contains 'wikipedia' which is the text that is nearest to the text containing 'wikipedia.org'
    await aui.click().text().containsText('wikipedia').nearestTo().text().containsText('wikipedia.org').exec();
  });
});

5. Done

We have covered a use case of AskUI to automate Web searching in Android devices. If you got any issues while following the instruction, feel free to ask in our Discord!

Set Up Android Sdk Command Line Tools without Android Studio

HaramChoi — Wed, 09 Nov 2022 09:22:55 +0000

Case01: Set Up for the Android Emulator

Download the Android SDK Command-Line Tools

Install the necessary tools:

# create a directory for the cli-tools, and move the .zip file into the directory
mkdir android_sdk
cd android_sdk
mv <PATH_TO_YOUR_DOWNLOADED_ZIP_FILE> .
# unzip the downloaded file, and put the contents in the new directory 'latest/'
unzip commandlinetools-<YOUR_PLATFORM>-8512546_latest.zip
mkdir cmdline-tools/latest
cd cmdline-tools/latest
mv ../* .
# The hierarchy of the 'android_sdk/':
└- android_sdk/
 ├- cmdline-tools/
    ├- latest/
       ├- bin/
       ├- lib/
       ├- NOTICE.txt
       └- source.properties
# Install platform and build tools
# You can fetch the complete list of available packages via:
bin/sdkmanager --list
# Install the desired package and tools via:
bin/sdkmanager --install "system-images;android-33;google_apis;x86_64"
# The above command will download and install the tools in the sdk_root (android_sdk/)

Download and install the Emulator:
```
bin/sdkmanager --install "emulator" 
```

Create a new avd device:

bin/avdmanager create avd -n mytestdevice -k "system-images;android-33;google_apis;x86_64"

Check if the avd device is available:

# go to the 'emulator/' directory, and run the command
cd ../../emulator
./emulator -list-avds
# Now you should see your newly created avd device `mytestdevice`

Run the emulator
```
./emulator -avd mytestdevice
```

Case02: Set Up for a real Android Device

Go to the official site and download the SDK Platform-Tools
Unzip the .zip file. You can find the adb binary within the extracted directory.
```
cd platform-tools
# Show every available Android device
./adb devices
```

Tutorial: Setting up Android Devices for Testing Mobile Apps with AskUI

HaramChoi — Wed, 02 Nov 2022 12:08:01 +0000

In this tutorial, we will walk you through how to set up an Android device for developing and testing mobile apps running on Android devices. Depending on the testing environment, i.e. whether it is a real Android device or an emulator, the procedure might slightly differ. But the overall process of the setup will be more or less the same.

If you want to prepare your Android development environment without using the Android Studio, then follow this post.

Requirements

Android Studio installed
(optional) Android device, if you want to run your app on a real device

1. Install Android SDK Command-line Tools

If you have the Android Studio installed, you have to verify if you have the Android SDK Command-line Tools and the Android SDK Platform-Tools installed. They both are packages containing several command line tools that are useful for developing apps for Android devices.

To verify the installation of them,

Click on the More Actions button, and then
Open the SDK Manager

Then, select Android SDK from the list on the left side of the window.
Go to the SDK Tools tab, and check the Android SDK Command-line Tools and the Android SDK Platform-Tools. After clicking the OK button, it will start to install the tools we have selected.

2. Set up the Test Device

After installing the necessary tools in the Android Studio, we can start to set up the Android device that we want to use for the development.

Set up a Real Android Device

This part is for setting up a real Android device. If you want to use the Android Emulator instead, then you can skip it and go directly to the next step.

To use additional development features in an Android device, we need to enable the Developer Options within the device. To enable the Developer Options, tap the Build Number option 7 times. You can find this option in one of the following locations, depending on your Android version:

Android 9 (API level 28) and higher: Settings > About Phone > Build Number
Android 8.0.0 (API level 26) and Android 8.1.0 (API level 26): Settings > System > About Phone > Build Number
Android 7.1 (API level 25) and lower: Settings > About Phone > Build Number

After enabling the Developer Options, we can enable the USB debugging option. This option will allow the Android Studio and other SDK tools to recognize your Android device via USB. To enable USB debugging, toggle the USB debugging option in the Developer Options menu. You can find this option in one of the following locations, depending on your Android version:

Android 9 (API level 28) and higher: Settings > System > Advanced > Developer Options > USB debugging
Android 8.0.0 (API level 26) and Android 8.1.0 (API level 26): Settings > System > Developer Options > USB debugging
Android 7.1 (API level 25) and lower: Settings > Developer Options > USB debugging

Set up an Android Emulator

If you want to use the Android Emulator for testing purposes, we have to create a virtual device that runs in the emulator. After running the Android Studio,

Click on the More Actions button, and then
Open the Virtual Device Manager

If no virtual device is created, we can create one by clicking the Create device button on the top left corner of the Virtual Device Manager window.

You will now see the Virtual Device Configuration window. Within the configuration window, you will be asked to:

Choose a hardware profile
Choose a system image

The hardware profile represents the specification of the hardware that will be used as a virtual device. Various presets of Android devices e.g. Pixel or Nexus are provided. But you can also create your own hardware profile, after which you will be customizing parameters in such:

Screen size
Resolution
Using Hardware buttons (Back, Home, Menu, Volume, etc.)
Using On-device sensors (Accelerometer, Gyroscope, GPS)

After choosing a hardware profile, we will be choosing a system image for the Android device. Here we can choose out of a variety of Android images, from the latest Android 13 Tiramisu down to the Android 7 Nougat. If you have the Android studio freshly installed and haven't used it yet, then it will start to download the image after selecting one.

After finishing creating a new virtual device, we will be able to see our new device listed in the Device Manager. Click the device's play button(▶️) to run it.

3. Set up the ADBKeyboard

Until now, we have prepared our Android device and now we are ready to go for testing our Android app. But before we jump into the test phase, we will set up one more utility that will make the test procedure easier.

ADBKeyboard is a virtual keyboard that can be installed on Android devices. It enables us to type within the Android device by using command lines via adb.

One noticeable advantage of using ADBKeyboard is that it can also handle base64 encoding, which becomes handy if you want to type Unicode characters such as emojis🔥 For more details about this virtual keyboard, see here.

To install the ADBKeyboard on your device,

Download the ADB-Keyboard package from this GitHub Repo (Important: Version 2.0)
Unzip it.

Find your device:

# make sure that your Android device is connected.
# in case of using the emulator, it should be running
adb devices

Install the ADBkeyboard on the device:

# inside ADBKeyBoard-2.0/
adb -s <your device id> install ADBKeyboard.apk

Configure the ADB Keyboard:

adb -s <your device id> shell settings put secure default_input_method com.android.adbkeyboard/.AdbIME

Enable the ADB Keyboard:

adb -s <your device id> shell ime enable com.android.adbkeyboard/.AdbIME

To check if it is enabled:
Click on a textfield in an app and see if the ADB Keyboard {ON} notification is shown at the bottom of the screen.

4. Done

We are finally done with the preparation for testing apps running on Android devices. In the next post, we will cover a simple test automation case utilizing AskUI on the device we set up in this article.

If you have a recurring or persisting issue while following this tutorial, don’t hesitate to ask the Discord community for help!

DEV Community: HaramChoi

Automating WebGL-/Canvas-based Website

Visit the Target Website

Plan the Automation

1. Visiting the Website

2. Click Enter and go to Les Couarres

3. Scroll on the Text

Conclusion

How AskUI Works

Overview

Glossary

AskUI Control Client

AskUI UI Controller

AskUI Inference Server

Them All in Action

Conclusion

Custom Elements in AskUI

Overview

Demonstration

Requirements

Understanding the customElement() in AskUI

Arguments

Two Things to be Aware of When Using customElement()

Capture the Custom Element

Write the AskUI Code

Breaking Down the Code

1) Open the Web Browser and Go To the Desired Website

2) Search for the Location

3) Drag the Human Icon to the Desired Location

Conclusion

Using Reporters in Jest Framework

Overview

Requirements

1. Prepare the AskUI Test Suite within jest

2.1. Using jest-junit with AskUI

2.2. Using jest-html-reporter with AskUI

2.3. Using jest-html-reporters with AskUI

3. Conclusion

Understanding Text Filters in AskUI

Requirements

Basic Text Matching

Match a Substring within a Text

Match the Exact Text

Match Text with Regular Expression

Conclusion

Automate Multiple Devices with AskUI

Overview

1. Download and Prepare the AskUI Binary for Each Device

Android Only:

2. Configure the jest.setup.ts

3. Run the Controller on Each Device

Android Only:

(option) Running Multiple Android Devices

4. Write the Test Code

5. Run the Test Code

6. Conclusion

Tutorial: AskUI Automation Example with Flutter

Setup

1. Build and Run the Flutter Demo App

2. Set up the ADBKeyboard

3. Setup AskUI

Breaking Down the Askui Test Code

0. General Tips for Using AskUI as a More Friendly Tool:

1. Click and Type

2. Datepicker

3. Take a Picture with the Camera

4. Complete Test Code

5. Conclusion

Tutorial: Automate Web Search on Android Devices with AskUI

Live Demo in Action (playback speed x3)

Requirements

1. Prepare the AskUI Test Environment

Install and initialize AskUI

Configure AskUI for Android

2. Try Annotating

3. Automate Web Searching

1) Open Chrome

2) Select the Search Bar and Type 'spacecraft'

3) Click on the Desired Search Result

4. Complete Test Code

Understanding the `customElement()` in AskUI

Two Things to be Aware of When Using `customElement()`

2. Configure the `jest.setup.ts`