Skip to content
This repository has been archived by the owner on Jul 28, 2024. It is now read-only.

Make tests pass on the updated testing infrastructure #1400

Closed
2 tasks done
borekb opened this issue Mar 12, 2019 · 5 comments
Closed
2 tasks done

Make tests pass on the updated testing infrastructure #1400

borekb opened this issue Mar 12, 2019 · 5 comments
Assignees
@borekb
Labels
scope: tests Testing code. For infrastructure (CI, etc.), use "dev-infrastructure".
Milestone
4.0

Comments

@borekb
Copy link
Member

borekb commented Mar 12, 2019 •
edited
Loading

#1389 improves the testing infrastructure and it seems that with enough "luck", by which I mean a good order of tests or their combination, all tests even pass. However, in practice, it's not very repeatable or reliable.

This issue is about getting the tests run reasonably reliably on both Mac and Windows. There are still debt items mentioned in #1284 like the "temporarily" disabled ActivationDeactivationTest and ThemeCustomizerTest which are out of scope of this issue.
@borekb borekb added the scope: tests Testing code. For infrastructure (CI, etc.), use "dev-infrastructure". label Mar 12, 2019
@borekb borekb added this to the 4.0 milestone Mar 12, 2019
@borekb
Copy link
Member Author

borekb commented Mar 12, 2019 •
edited
Loading

My current experience (on commit b9e6cc3):

$ git checkout b9e6cc356d
$ npm run refresh-dev
$ npm run tests:cleanup
$ npm run tests:full

> versionpress-dev-workspace@1.0.0 tests:full /Users/borekb/dev/VersionPress/versionpress
> docker-compose -f docker-compose-tests.yml run --rm tests-with-wordpress ../vendor/bin/phpunit -c phpunit.xml; npm run tests:stop

Creating network "versionpress_default" with the default driver
Creating volume "versionpress_db-data-for-tests" with default driver
Creating volume "versionpress_wpcli-cache" with default driver
Creating volume "versionpress_wordpress-files" with default driver
Creating volume "versionpress_test-logs" with default driver
Creating versionpress_selenium-hub_1    ... done
Creating versionpress_mysql-for-tests_1 ... done
Creating versionpress_wordpress-for-tests_1 ... done
PHPUnit 5.7.20 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.2.16 with Xdebug 2.6.0
Configuration: /opt/versionpress/tests/phpunit.xml

...............................................................  63 / 738 (  8%)
............................................................... 126 / 738 ( 17%)
............................................................... 189 / 738 ( 25%)
............................................................... 252 / 738 ( 34%)
............................................................... 315 / 738 ( 42%)
............................................................... 378 / 738 ( 51%)
............................................................... 441 / 738 ( 59%)
............................................................... 504 / 738 ( 68%)
............................................................... 567 / 738 ( 76%)
............................................................... 630 / 738 ( 85%)
............S....SS....S..S....S................S....S.S.SSS... 693 / 738 ( 93%)
........S....................S.....FSS......Php Execution Time: 0.033014059066772 Sec
.                   738 / 738 (100%)Bash Execution Time: 0.035650968551636 Sec


Time: 29.63 minutes, Memory: 18.00MB

There was 1 failure:

1) VersionPress\Tests\Workflow\CloneMergeTest::cloneLooksExactlySameAsOriginal
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@

-		Posted on November 11, 2011Test post for revert
+		Posted on November 11, 2011March 11, 2019Test post for revert
@@ @@

-		Posted on November 11, 2011Test post for revert
+		Posted on November 11, 2011Test post
@@ @@

-		Posted on November 11, 2011March 11, 2019Test post for revert
+		Posted on November 11, 2011Test post for revert
@@ @@

-		Posted on November 11, 2011Test post for revert
+		Posted on November 11, 2011Test post for comments
@@ @@

-		Posted on November 11, 2011Test post
+		Posted on November 11, 2011March 11, 2019Test post

[... Lots of whitespace here, see the attached file for full output ...]

/opt/versionpress/tests/Workflow/CloneMergeTest.php:210
/opt/versionpress/tests/Workflow/CloneMergeTest.php:48

--

There were 16 skipped tests:

[... Omitted ...]

FAILURES!
Tests: 738, Assertions: 950, Failures: 1, Skipped: 16.

> versionpress-dev-workspace@1.0.0 tests:stop /Users/borekb/dev/VersionPress/versionpress
> docker-compose -f docker-compose-tests.yml stop

Stopping versionpress_wordpress-for-tests_1 ... done
Stopping versionpress_selenium-hub_1        ... done
Stopping versionpress_mysql-for-tests_1     ... done

Full test-output.txt.

When I move Workflow tests to run before End2End tests, everything passes:

Screenshot 2019-03-12 at 11 00 17

image

It's not "right" but I'm going to commit the change to move us forward.

Note: The tests are trying to find a balance between isolation and performance. We could restore the database and WordPress site before each test but it would make everything extremely slow. Maybe we could restore the DB & site at least before each test suite.

Update: this re-grouping of tests makes them much slower – from 10 minutes to 30 minutes on my machine. I think we should do some sort of cleanup between tests suites, for example, start End2End tests and Workflow tests each on a fresh WP installation.

Update 2: I no longer understand what's causing the slowdown. I'm trying again on commits 341fec0 and 29fa978 where I've definitely seen 10-minute long runs and they now both take 30 minutes.

@borekb borekb mentioned this issue Mar 13, 2019
Merged
10 tasks
@borekb
Copy link
Member Author

borekb commented Mar 17, 2019

I no longer understand what's causing the slowdown. I'm trying again on commits 341fec0 and 29fa978 where I've definitely seen 10-minute runs before and they now both take 30 minutes. 🤔

@borekb borekb mentioned this issue Mar 17, 2019
Merged
@borekb
Copy link
Member Author

borekb commented Mar 17, 2019

Good is that I'm seeing consistent results on Docker Toolbox on Windows: #1403 (comment).

@borekb
Copy link
Member Author

borekb commented Mar 17, 2019

I no longer understand what's causing the slowdown. I'm trying again on commits 341fec0 and 29fa978 where I've definitely seen 10-minute runs before and they now both take 30 minutes. 🤔

So I tried git clean -fxd && npm i, it didn't help.

Then I restarted Docker and voila, tests were back at 10 minutes again.

Who will give me those few hours of my life back? 😅

@borekb borekb mentioned this issue Mar 19, 2019
Closed
13 tasks
@borekb borekb mentioned this issue Mar 26, 2019
Open
2 tasks
@borekb
Copy link
Member Author

borekb commented Mar 26, 2019

With an improved test runner in #1401 and a workaround for a cloneLooksExactlySameAsOriginal implemented in #1410, all tests now pass 🎉.

@borekb borekb closed this as completed Mar 26, 2019
This was referenced Mar 26, 2019
Closed
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@borekb borekb

Labels
scope: tests Testing code. For infrastructure (CI, etc.), use "dev-infrastructure".
Projects
None yet
Milestone
4.0
Development

No branches or pull requests
2 participants
@borekb @pavelevap