Merge pull request #1910 from magento/ds_MQE-887_2.1-updates

MFTF 2.1 update (content migration from wiki)
magento · Apr 23, 2018 · 58df20c · 58df20c
2 parents ae96b73 + ccad705
commit 58df20c
Show file tree

Hide file tree

Showing 7 changed files with 417 additions and 169 deletions.
diff --git a/_data/toc/magento-functional-testing-framework-guide.yml b/_data/toc/magento-functional-testing-framework-guide.yml
@@ -47,6 +47,9 @@ pages:
       - label: Merging
         url: /magento-functional-testing-framework/release-2/merging.html
 
+      - label: Configuration
+        url: /magento-functional-testing-framework/release-2/configuration.html
+
       - label: CLI Commands
         children:
 

diff --git a/guides/v2.2/magento-functional-testing-framework/release-2/best-practices.md b/guides/v2.2/magento-functional-testing-framework/release-2/best-practices.md
@@ -4,53 +4,136 @@ group: mftf
 title: Best practices
 version: 2.2
 github_link: magento-functional-testing-framework/release-2/best-practices.md
+mftf-release: 2.1.2
 functional_areas:
  - Testing
 ---
 
-Check out our best practices below to ensure you're getting the absolute most out of the MFTF.
+_This topic was updated due to the {{page.mftf-release}} MFTF release._
+{: style="text-align: right"}
 
-## Annotations
+Check out our best practices below to ensure you're getting the absolute most out of the Magento Functional Testing Framework.
 
- - Always use annotations in a test.
-
- - When updating tests, always make corresponding annotation updates.
-
- - Annotation types and recommendations are described as:
-    - **Feature** - Report grouping, a set of tests that verify a feature.
-    - **Story** - Report grouping, a set of tests that verify a story.
-    - **Group** - Module name.
-    - **Title** - Describes the purpose of the test.
-    - **Description** - Describes how the test achieves the purpose defined in the title.
-    - **Severity** - Allowed values are _BLOCKER_, _CRITICAL_, _MAJOR_, _AVERAGE_, and _MINOR_.
+## Naming conventions
+
+### File names
+
+Name files according to the following patterns to make searching in future more easy:
+
+#### Test file name
+
+Format: {_Admin_ or _Storefront_}{Functionality}_Test.xml_, where Functionality briefly describes the testing functionality.
+
+Example: _StorefrontCreateCustomerTest.xml_.
+
+#### Section file name
+
+Format: {_Admin_ or _Storefront_}{UI Description}_Section.xml_, where UI Description briefly describes the testing UI.
+
+Example: _AdminNavbarSection.xml_.
+
+#### Data file name
+
+Format: {Type}_Data.xml_, where Type represents the entity type.
+
+Example: _ProductData.xml_.
+
+### Object names
+
+Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _classProperties_ in PHP.
+
+#### Upper case
+
+Use an upper case first letter for:
+- File names. Example: _StorefrontCreateCustomerTest.xml_
+- Test name attributes. Example: `<test name="TestAllTheThingsTest">`.
+- Data entity names. Example: `<entity name="OutOfStockProduct">`.
+- Page name. Example: `<page name="AdminLoginPage">`.
+- Section name. Example: `<actionGroup name="DeleteCategory">`.
+
+#### Lower case
+
+Use a lower case first letter for:
+- Data keys. Example: `<data key="firstName">`.
+- Element names. Examples: `<element name="confirmDeleteButton"/>`.
+
+## Test
+
+1. Use actions such as [`<waitForElementVisible>`], [`<waitForLoadingMaskToDisappear>`], and [`<waitForElement>`] to wait the exact time required for the test step.
+ Try to avoid using the [`<wait>`] action, because it forces the test to wait for the time you specify. You may not need to wait so long to proceed.
+2. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution.
+ It also helps you to identify the cause of test failure.
+3. Use comments to keep tests readable and maintainable:
+  * Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
+  It helps to inform the reader of what you are testing and to yield a more descriptive Allure report.
+  * Explain in comments unclear or tricky test steps.
+4. Refer to [sections] instead of writing selectors.
+
+## Action group
+
+1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used.
+ Add additional explanation in comments if needed. 
+2. Provide default values for the arguments that apply to your most common case scenarios.
+
+## Annotation
+
+1. Use [annotations] in a test. 
+2. Update your annotations correspondingly when updating tests.
 
-## Data entities
+## Data entity
+
+1. Keep your testing instance clean.
+ Remove data after the test if the test required creating any data.
+ Use a corresponding [`<deleteData>`] test step in your [`<after>`] block when using a [`<createData>`] action in a [`<before>`] block.
+2. Make specific data entries under test to be unique.
+ Enable data uniqueness where data values are required to be unique in a database by test design. 
+ Use `unique=”suffix”` or `unique=”prefix”` to append or prepend a unique value to the [entity] attribute.
+ This ensures that tests using the entity can be repeated.
+3. Do not modify existing data entity fields or merge additional data fields without complete understanding and verifying the usage of existing data in tests.
+ Create a new data entity for your test if you are not sure.
 
-- When using a `<createData>` action in a `<before>` block, always use a corresponding `<deleteData>` in your `<after>` block.
+## Page object
 
-- Where data values are required to be unique in the database, enforce the uniqueness on the attribute of the data entity. Use `[unique=”suffix”]` or `[unique=”prefix”]` to append or prepend a unique value to the entity attribute. This ensures tests using the entity can be repeated.
+Use [parameterized selectors] for constructing a selector when test specific or runtime generated information is needed.
+Do not use them for static elements.
 
-- Do not modify existing data entity fields or add/merge additional data fields without fully understanding and verifying all existing data usages. We recommend that you create a new data entity for your test when you are not sure. 
+{:style="color:red"}
+BAD:
+``` xml
+<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='{{productType}}']" parameterized="true"/>
+```
 
-## Page objects
+{:style="color:green"}
+GOOD:
 
-Do not overuse parameterized selectors. 
+Define these three elements and reference them by name in the tests.
+``` xml
+<element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='related']"/>
+<element name="upSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='upsell']"/>
+<element name="crossSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='crosssell']"/>
+```
 
-Parameterized selectors should only be used when test-specific or runtime-generated information is needed to construct a selector. Do not use it for static elements.
+## Test step merging order
 
-For example, do not define a parameterized element like the following:
-  ``` xml
-  <element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='{{productType}}']" parameterized="true"/>
-  ```
-  Instead, define these three elements and reference them by name in the tests:
-  ``` xml
-  <element name="relatedProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='related']"/>
-  <element name="upSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='upsell']"/>
-  <element name="crossSellProductSectionText" type="text" selector=".fieldset-wrapper.admin__fieldset-section[data-index='crosssell']"/>
-  ```
+When setting a [merging] order for a test step, do not depend on steps from Magento modules that could be disabled by an application.
 
-## Test step merge orders
+For example, when you write a test step to create a gift card product, set your test step **after** simple product creation and let the MFTF handle the merge order.
+Since the configurable product module could be disabled, this approach is more reliable than setting the test step **before** creating a configurable product.
 
-When setting merge orders for a test step, do not depend on steps from Magento modules that could be disabled by an application.
+<!-- Link definitions -->
 
-For example, when you write a test step to create a gift card product, it's probably better to set your test step **after** simple product creation and let MFTF handle the merge order. This is better than setting the test step **before** creating a configurable product, because the configurable product module could be disabled.
+[Action group]: test/action-groups.html
+[`after`]: test/actions.html#before-and-after
+[annotations]: test/annotations.html
+[`before`]: test/actions.html#before-and-after
+[`<comment>`]: test/actions.html#comment
+[`<createData>`]: test/actions.html#createdata
+[`<deleteData>`]: test/actions.html#deletedata
+[entity]: data.html
+[merging]: merging.html
+[parameterized selectors]: section/parameterized-selectors.html
+[sections]: section.html
+[`<wait>`]: test/actions.html#wait
+[`<waitForElement>`]: test/actions.html#waitforelement
+[`<waitForElementVisible>`]: test/actions.html#waitforelementvisible
+[`<waitForLoadingMaskToDisappear>`]: test/actions.html#waitforloadingmasktodisappear
diff --git a/guides/v2.2/magento-functional-testing-framework/release-2/configuration.md b/guides/v2.2/magento-functional-testing-framework/release-2/configuration.md
@@ -0,0 +1,146 @@
+---
+layout: default
+group: mftf
+title: Configuration
+version: 2.2
+github_link: magento-functional-testing-framework/release-2/configuration.md
+functional_areas:
+ - Testing
+mftf-release: 2.1.2
+---
+
+_This topic was updated due to the {{page.mftf-release}} MFTF release._
+{: style="text-align: right"}
+
+The _*.env_ file provides additional configuration of the Magento Functional Testing Framework (MFTF).
+All MFTF users will need to set the basic configuration values.
+Advanced users may need to set additional values to work with their environment.
+
+## Basic Configuration
+
+These basic configuration values are required and must be set by the user before MFTF can function correctly.
+
+### MAGENTO_BASE_URL
+
+* Use: Required.
+* Description: The root URL of the Magento application under test.
+* Example: `http://magento2.vagrant251`
+
+### MAGENTO_BACKEND_NAME
+
+* Use: Required.
+* Description: The path to the Magento Admin page.
+* Example: `admin_12346`.
+
+### MAGENTO_ADMIN_USERNAME
+
+* Use: Required.
+* Description: The username that tests will use to log in to the Magento Admin page.
+* Example: `admin`.
+
+### MAGENTO_ADMIN_PASSWORD
+
+* Use: Required.
+* Description: The password that tests will use to log in to the Magento Admin page.
+* Example: `1234reTyt%$7`
+
+## Advanced Configuration
+
+In most cases, these values are not required.
+Sensible defaults are in place.
+But in case you do need to do some configuration, they are shown here for your reference.
+
+### SELENIUM
+
+The `SELENIUM_*` values are concatenated together to form the URL of a custom Selenium server to test against.
+An example use case for these would be if you wanted to run your tests against any other external Selenium source instead of your local machine.
+
+```config
+SELENIUM_HOST
+SELENIUM_PORT
+SELENIUM_PROTOCOL
+SELENIUM_PATH
+```
+
+#### SELENIUM_HOST
+
+* Use: Optional
+* Description: For overriding the default Selenium server URL.
+* Example: `user:pass@ondemand.saucelabs.com`
+
+#### SELENIUM_PORT
+
+* Use: Optional
+* Description: For overriding the default Selenium server URL.
+* Example: `443`
+
+#### SELENIUM_PROTOCOL
+
+* Use: Optional
+* Description: For overriding the default Selenium server URL.
+* Example: `https`
+
+#### SELENIUM_PATH
+
+* Use: Optional
+* Description: For overriding the default Selenium server URL.
+* Example: `/wd/hub`
+
+### MAGENTO_RESTAPI
+
+These `MAGENTO_RESTAPI_*` values are optional and can be used in cases where your Magento instance has a different API path than the standard `MAGENTO_BASE_URL`.
+
+```config
+MAGENTO_RESTAPI_SERVER_HOST
+MAGENTO_RESTAPI_SERVER_PORT
+```
+
+#### MAGENTO_RESTAPI_SERVER_HOST
+
+* Use: Optional
+* Description: The protocol and host part of the API path.
+* Example: `http://localhost`
+
+#### MAGENTO_RESTAPI_SERVER_PORT
+
+* Use: Optional
+* Description: The port part of the API path.
+* Example: `5000`
+
+### TESTS
+
+These values can be used in cases where you are working locally on both MFTF's implementation code and Magento's test case code.
+Use them if you have a more advanced local development setup involving symlinking MFTF into the `vendor` directory of `magento2`.
+
+```config
+TESTS_BP
+FW_BP
+TESTS_MODULES_PATH
+```
+
+#### TESTS_BP
+
+BP is an acronym for BasePath.
+
+* Use: Optional.
+* Description: The path to where MFTF supplementary files are located in the Magento2 codebase.
+* Example: `~/magento2ce/dev/tests/acceptance`
+
+#### FW_BP
+
+FW_BP is an acronym for FrameWork BasePath.
+
+* Use: Optional.
+* Description: The path to where MFTF exists and from which it is symlinked.
+* Example: `~/magento/magento2-functional-testing-framework`
+
+#### TESTS_MODULE_PATH
+
+* Use: Optional.
+* Description: The path to where MFTF modules mirror Magento's modules.
+* Example: `~/magento2ce/dev/tests/acceptance/tests/functional/Magento/FunctionalTest`
+
+### MODULE_WHITELIST
+
+Use the `MODULE_WHITELIST` environment variable if you are working on a new module.
+When adding a new directory under `Magento/FunctionalTest`, add the directory name under `MODULE_WHITELIST` to enable the MFTF to process it.
diff --git a/guides/v2.2/magento-functional-testing-framework/release-2/section.md b/guides/v2.2/magento-functional-testing-framework/release-2/section.md
@@ -6,7 +6,7 @@ version: 2.2
 github_link: magento-functional-testing-framework/release-2/section.md
 functional_areas:
  - Testing
-mftf-release: 2.0.2
+mftf-release: 2.1.2
 ---
 
 _This topic was updated due to the {{page.mftf-release}} MFTF release._
@@ -114,8 +114,42 @@ Attributes|Type|Use|Description
 `type`|string|required|The type of the element. Possible values: `text`, `textarea`, `input`, `button`, `checkbox`, `radio`, `checkboxset`, `radioset`, `date`, `file`, `select`, `multiselect`, `wysiwyg`, `iframe`.
 `selector`|string|optional|[XPath](https://www.w3schools.com/xml/xpath_nodes.asp) or [CSS](https://www.w3schools.com/cssref/css_selectors.asp) selector of the element.
 `locatorFunction`|string|optional|[Locator function](./section/locator-functions.html) declaration to be used in lieu of a selector.
-`timeout`|string|optional|The default is `-`; The optional timeout value in seconds to wait for the operation on the element.
+`timeout`|string|optional|The timeout after interaction with the element (in seconds). The default is _none_.
 `parameterized`|boolean|optional|Include and set to `true` if the `selector` for this element has parameters that need to be replaced for proper use. Learn more in [Parameterized selectors](./section/parameterized-selectors.html).
 `remove`|boolean|optional|The default is `false`. Set to `true` to remove this element during parsing.
 
+#### `timeout` attribute {#timeout-attribute}
+
+The attribute adds the [waitForPageLoad] action after a reference to the element in test.
+The most usual use case is a test step with a button click action.
+
+**Use case**: Set a timeout of 30 seconds after clicking the `signIn` button.
+
+The section element code declaration containing the timeout attribute:
+
+> StorefrontSigninSection.xml
+
+```xml
+...
+<element name="signIn" type="button" selector="#signIn" timeout="30"/>
+...
+```
+
+The test step that covers the use case:
+
+> StorefrontSigninTest.xml
+
+```xml
+...
+<click selector="{{StorefrontSigninSection.signIn}}" ../>
+...
+```
+
+Whenever the `signIn` button is used in a test, the MFTF will add a 30 second `waitForPageLoad` action immediately after the `click`.
+
 {% endraw %}
+
+
+<!-- Link definitioins -->
+
+[waitForPageLoad]: test/actions.html#waitforpageload