Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up outline mechanism for testing Qt GUI components #4617

Open
claremacrae opened this issue Oct 28, 2019 · 11 comments
Open

Set up outline mechanism for testing Qt GUI components #4617

claremacrae opened this issue Oct 28, 2019 · 11 comments

Comments

@claremacrae
Copy link
Contributor

Motivation

I'm looking for a Qt desktop project that would like some help adding or improving its automated testing...

The immediate motivation is to help me have some examples to use in a talk I'm giving real soon now, at Meeting C++.

@brianlheim saw my mention of this, with a link to the following tweet, and suggested I look at this project...

https://twitter.com/ClareMacraeUK/status/1185887010554335232

Based on an initial conversation with Brian, I'd really like to contribute to its automated testing - my background is C++ and Qt.

Description of Proposed Feature

Well, that's up for discussion - with this ticket suggested as a place-holder for that...

Based on past experience of automated tests of Qt desktop applications, I'd mainly be wanting to test behaviour rather than appearance.

Plan for Implementation

Q: Who will implement this? How long will it take?

A: Me for setting up the initial outline - with a few initial passing tests, showing different types of testing approach... I would want to do some kind of handover to others, as I wouldn't want to be the single-point-of-future-maintenance!

I have about a working week to work on this, part-time... It will be my main focus.

I am hoping to be able to pair with someone early on, to get me up to speed with the code as soon as possible...

Re handover, one idea might be to do a run-through of my talk for any supercollider developers who were interested - to share how I had worked.. Feedback on the talk would be good too :-)

@mossheim
Copy link
Contributor

Thanks @claremacrae !

As I mentioned when we talked earlier, I'd love to work with you on this and integrate your work into the main repository. Based on the (relatively light) knowledge I have of the use of Qt in this codebase, the IDE ("scide") would be easier to test in isolation than the Qt-based objects and bindings in the supercollider language ("sclang"), as the latter involve an object proxy and metatype system for converting between C++ and supercollider types. (But perhaps that would be more interesting?)

To help you with some quick navigation:

  • editors/sc-ide: the source code for the IDE
  • QtCollider: the code for the Qt bindings in sclang
  • lang: the non-Qt part of sclang

Both the IDE and sclang include a few source files from other parts of the repository as well, it's not quite as tidy as it could be.

Any new tests for this project would probably go under testsuite/editors/sc-ide and/or testsuite/sclang/qt [or gui] respectively.

I hope you will be able to make it to the developer call this weekend! (date/time still TBD)

@claremacrae
Copy link
Contributor Author

In preparation for this, and after discussion with @brianlheim last night, I have tried to divide the side build up so that it creates a static library, and then links that and a main to create the SuperCollider exe.

It is sadly still a work in progress: On Mac, SuperCollider only links successfully when
CMake was run with -DSC_USE_QTWEBENGINE=OFF

The full diffs of what I've done can be seen at:

claremacrae/supercollider@develop...claremacrae:create_libscide

Essentially, it is:

  • Pull the main() from editors/sc-ide/core/main.cpp out in to a separate file, editors/sc-ide/core/main_function.cpp
  • Change the Cmake file for SuperCollider so that it:
    • First creates a static library called libscide containing everything but main_function.cpp
    • Then create the exe, linking main_function.cpp and the static library.

Please see the comments in the CMakeLists.txt change in claremacrae@434f7ea for the linker errors (missing symbols) when QtWebEngine is enabled.

I've been staring at this for some hours now, so am sharing progress in the hope that anyone might be able to spot a fix for the error.

@claremacrae
Copy link
Contributor Author

The missing symbols linker error is reproducible on Linux too:
https://travis-ci.com/claremacrae/supercollider/jobs/251148707

The Mac CI build fails to find Qt for me - this happened with a previous push - I have not changed any CI files:
https://travis-ci.com/claremacrae/supercollider/jobs/251148709

@claremacrae
Copy link
Contributor Author

I've found the cause of the linker error - it was my mistake:
claremacrae@a108269

Next thing to fix: When running from a distribution, the program now runs, but the Post window has this output:

compiling class library...
	Found 847 primitives.
	Compiling directory '/Users/clare/Documents/develop/Audio/SuperCollider/build/Install/SuperCollider/SuperCollider.app/Contents/MacOS/SCClassLibrary'
	Compiling directory '/Users/clare/Library/Application Support/SuperCollider/Extensions'
ERROR: There is a discrepancy.
numClassDeps 0   gNumClasses 82
Library has not been compiled successfully.
Library has not been compiled successfully.

@claremacrae
Copy link
Contributor Author

I fixed that "There is a discrepancy" message in claremacrae@f68d112

@claremacrae
Copy link
Contributor Author

claremacrae commented Oct 30, 2019

@brianlheim

Status and questions

I think that all the coding/testing that remains on the sc-ide static library change relates to this code:

# final builds of the IDE seem to be broken atm
if(0 AND FINAL_BUILD)
  CREATE_FINAL_FILE(scide_final.cpp ${ide_sources})
  add_library( libscide STATIC scide_final.cpp ${ide_rc_sources} ${RES_FILES})
else()
  add_library( libscide STATIC ${ide_sources} ${ide_rc_sources} ${RES_FILES})
endif()

add_executable( SuperCollider MACOSX_BUNDLE core/main_function.cpp  ${AdditionalBundleSources})

I need to know whether either of ${ide_rc_sources} or ${RES_FILES} needs to move from the add_library() lines to the add_executable() line.

RES_FILES is only used on Windows... I suspect it does need to move the exe, but I don't have time to set up a Windows dev environment for this to check...

ide_rc_sources contains these:

set(ide_rc_sources ${ide_moc_src} ${ide_forms_src} ${ide_rcc} ${translations})

I suspect that ${ide_moc_src} and ${ide_forms_src} need to go in the library, in order for any tests that link in the library to work. I don't

I suspect that ${ide_rcc} and ${translations} need to go in the executable.

Request

What would be really, really helpful would be for someone who knows how to build and smoke-test the IDE on Windows to test the behaviours of my fork for all the relevant areas to confirm...

This would be the code to checkout: https://github.com/claremacrae/supercollider/commits/create_libscide

@claremacrae
Copy link
Contributor Author

I can tell from a Windows build that I have downloaded that I have broken the icon of the exe in the Windows explorer - which confirms my guess about ide_rcc above, so I'll fix that:

https://ci.appveyor.com/project/claremacrae/supercollider/builds/28498680/job/k4k6ah970k0caf5u/artifacts

@claremacrae
Copy link
Contributor Author

Confirmed by downloading a build, the windows icon is fixed now - by this:
claremacrae@0dcada0

The main thing that still needs testing is the translations.

@claremacrae
Copy link
Contributor Author

All my attempts to test the translation mechanism in unmodified source code have failed - see #4619

@claremacrae
Copy link
Contributor Author

claremacrae commented Nov 10, 2019

An Update

For info, in case anyone would like to see progress so far... And mainly to get it out of my head...

I've just pushed the topic/scide-test-experiments branch I was using to experiment with setting up initial tests for a couple of widgets in sc-ide...

https://github.com/claremacrae/supercollider/tree/topic/scide-test-experiments

My current thinking is that I won't create a PR from this branch - it's mostly full of experiments I did to see what would work. Instead, I expect I'll create a new branch and PR later this month...

There is a new folder: testsuite/editors/sc-ide/

With two temporary sub-directories:

  • catch2_sc-ide_tests - uses Catch2 to drive the tests, and small amounts of Qt Test for testing, e.g. for QSignalSpy
    • I expect to rename this something like sc-ide_tests and keep much of its contents
  • qttest_sc-ide_tests - uses Qt Test to drive the tests, and for all asserts
    • I expect to delete this - Qt Test is not convenient enough, compared to Catch

These lines show the kind of expressiveness I would aspire to in any final tests - with all details hidden behind test fixtures and helper functions:

https://github.com/claremacrae/supercollider/blob/topic/scide-test-experiments/testsuite/editors/sc-ide/catch2_sc-ide_tests/widgets/goto_line_tool_tests.cpp#L61-L74

TEST_CASE_METHOD(GoToLineToolFixture, "GoToLineTool emits signal when Go button clicked") {
    // Arbitrary upper limit in number of lines.
    // When used in the application, this would be obtained from the open docunent
    setMaximumLineCount(27);

    // Type a number, one digit at a time
    typeCharacterIntoSpinner('1');
    typeCharacterIntoSpinner('7');
    checkActivatedSignalCount(0);

    clickGoButton();
    checkActivatedSignalCount(1);
    checkActivatedSignalValue(17);
}

Dependencies - Catch2

The only new dependency is a single header copied in to [external_libraries/catch](https://github.com/claremacrae/supercollider/tree/topic/scide-test-experiments/external_libraries/catch}

ApprovalTests.cpp.Qt

I am in the process of creating a new project, ApprovalTests.cpp.Qt, to hide a lot of the boiler-plate of creating Catch2 + Qt Test test suites...

https://github.com/approvals/ApprovalTests.cpp.Qt

It will end up being a single-header release that will depend on the similarly single-header https://github.com/approvals/ApprovalTests.cpp

After my Meeting C++ talk, I'll pick this up again.

@mossheim
Copy link
Contributor

@claremacrae thanks so much for the update and all the work! looking forward to more :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants