End to end tests
All end-to-end tests are written with Cypress.
Writing good tests
Cypress tests come in addition to manual tests defined in our TestManagement tool. The major advantage of cypress is that tests are run every night and warn us quickly if something bad happens.
- One of the main difficulties in our end-to-end workflow is having a relevant test suite:
Tests covered by Cypress should not aim to eliminate TestManagement test definitions. That means that when we are working on a subject, before starting to write any code, we must think about what test we want to cover and who will execute the test (is the feature for everybody, for project admin …). Once it’s done, we can check if we have differences with our TestManagement test suite and adapt the TestManagement suite with what has been identified.
When we add new tests in Cypress that don’t exist in TestManagement, we must add them as a new TestManagement definition.
- When TestManagement test definitions can’t fully be covered by Cypress, then we should split them:
one part covered by Cypress
one part which will be done manually during validation
Best practises for writing tests
It’s better to have tests split by user roles.
Test should be executed by the least-powered user (don’t use project administrator for everything).
itwill help us debug what’s going on when tests fail.
Multiple TestManagement test definitions can not be covered by a simple testdefinition (
ttmtool won’t do the report automatically). In that case you should split your Cypress test and link each TestManagement test definition to a smaller
Always put your Cypress tests in the relevant plugin. End-to-end tests should still respect the plugin architecture.
If you want to run all cypress tests locally just launch:
$> make tests_cypress
You will be able to see the results of the test execution in
If you want to add new tests, you should use the cypress dev image:
$> make tests_cypress_dev
It will launch a local container with a bunch of projects (defined in
Before launching the Cypress electron app, you must be able to reach the “test”
Tuleap container at
https://tuleap/. Cypress will run tests on this URI. Add
a new entry in the
/etc/hosts file, the IP should correspond to the IP of
your container named
$> sudo vi /etc/hosts 172.18.0.2 oauth2-server-rp-oidc 172.18.0.4 tuleap
The IP address should be written at the end of the output from
tests_cypress_dev. If you can’t find it, run this command:
$> docker inspect e2e-tests_tuleap_1 # At the end of the big JSON output, there is a section named "NetworkSettings". Inside it, there is "Networks" and then "IPAddress"
After saving the
/etc/hosts file, verify that you can reach a working Tuleap
at https://tuleap/. If you do not see a Tuleap Login screen, something is
broken in the Tuleap container and tests will all fail.
The IP address will change every time you run
make tests_cypress_dev. You
/etc/hosts each time.
Once the container has started, you should be able to launch the Cypress electron app. The test structure respects the Tuleap distinction between core and plugins.
To write tests in Tuleap core, go in core folder and run
$> cd tuleap/tests/e2e/full/ $> npx cypress open
To write tests in plugins, go in plugin folder and run
npx cypress open
$> cd tuleap/plugins/<plugin_name>/tests/e2e/cypress/ $> npx cypress open
The electron app will be able to run only when the Tuleap container is fully
started. If https://tuleap/ is unreachable make sure that container
initialisation has finished.
If it does not solve your issue, verify the IP in your
As a reminder, an architecture diagram is available:
Some of our validation tests are executed by the CI. To declare a test as automated, follow these steps:
On your dev platform
Checkout the patch introducing the new tests
test_results_e2e_fulllook for the
result<sha1>.xmlcorresponding to the tests you are automating
In TTM open the test Validation Execution tracker
Open the artifact(s) correponding to the test you have automated
You should find a field
testcasename into this field
In the field
Refactoringchoose the value
moved to cypress(in order to see which tests must be run by the CI)