feat: use the --expect parameter to specify HTTP response code #343

bahmutov · 2022-12-05T21:24:32Z

closes #339

github-actions · 2022-12-05T21:31:00Z

🎉 This PR is included in version 1.15.0 🎉

The release is available on:

waldemarennsaed · 2023-04-17T16:40:27Z

This seems like a breaking change to me. Just tried to upgrade from 1.14.0 to 1.15.x and specifically the step from 1.14.0 to 1.15.0 breaks for me.

I am wondering if anyone else faced this issue.

Specific issue

In my case, a local test-environment with a vitest web-server & cypress was working fine but the pipeline on GitLab broke because Cypress could not find the application with 1.15.0. Reverting back to 1.14.0 fixed it for me.

Anyways. Will post more information when I get more insights.

MikeMcC399 · 2023-04-17T16:48:11Z

@waldemarennsaed

Why did you pick 1.15.0 specifically? Did you try 1.15.5 or 2.0.0?
What Node.js version is your GitLab pipeline running?

Logs would be useful to understand your error.

waldemarennsaed · 2023-04-18T07:07:46Z

I first used 2.0.0 and got the error. Then I tracked it down to 1.15.0. In 1.14.0 it doesn't happen so my assumption is that 1.15.0 breaks our setup.

As said, locally everything works fine but as soon as I run it on the CI job, it fails. Running the same script locally as in the CI.

Logs

The logs are not quite interesting since we only see the Cypress tests fail:

$ npm run ci:test:e2e -- --env oauth_authority=${E2E_OAUTH_AUTHORITY},oauth_client_id=${E2E_OAUTH_CLIENT_ID}
> frontend@8.1.0 ci:test:e2e
> tests/e2e/scripts/start-headless-e2e-tests.sh --env oauth_authority=https://my-auth-provider.com,oauth_client_id=frontend
1: starting server using command "npm run preview:e2e"
and when url "[ 'http://localhost:7357' ]" is responding with HTTP status code 200
running tests using command "npm run cypress:headless --env oauth_authority=https://my-auth-provider.com,oauth_client_id=frontend"
> frontend@8.1.0 preview:e2e
> vite build --mode=test-e2e-headless && vite preview --port=7357
[log] vite v4.1.1 building for test-e2e-headless...
[log] transforming...
[log] ✓ 1620 modules transformed.
[log] rendering chunks...
[log] computing gzip size...
[log] build/manifest.webmanifest                          0.15 kB
[log] build/index.html                                    4.24 kB
[log] build/assets/fa-v4compatibility-aacd78cf.woff2      4.57 kB
[log] build/assets/fa-v4compatibility-cacf649a.ttf       10.14 kB
[log] build/assets/logo_white-15de20cf.png               26.09 kB
[log] build/assets/logo-f8f67db1.png                     62.71 kB
[log] build/assets/fa-brands-400-7b17fce6.woff2         108.80 kB
[log] build/assets/fa-brands-400-0726ff45.ttf           188.99 kB
[log] build/assets/TagGroup.worker-f4c99ad2.js          229.71 kB
[log] build/assets/fa-solid-900-c0fd7653.woff2          320.08 kB
[log] build/assets/fa-regular-400-4bb4c355.woff2        395.06 kB
[log] build/assets/fa-light-300-e1ad92ed.woff2          [43](https://gitlab.com/my-org/frontend/frontend/-/jobs/4127394753#L43)1.54 kB
[log] build/assets/fa-duotone-900-fc69a3f7.woff2        435.83 kB
[log] build/assets/fa-thin-100-64cbd3b0.woff2           471.19 kB
[log] build/assets/fa-solid-900-c52f0989.ttf            890.70 kB
[log] build/assets/fa-regular-400-c262547c.ttf          989.12 kB
[log] build/assets/fa-light-300-a518c2e1.ttf          1,114.68 kB
[log] build/assets/fa-thin-100-679c81cf.ttf           1,230.04 kB
[log] build/assets/fa-duotone-900-7d194c5b.ttf        1,249.98 kB
[log] build/assets/index-b21b0396.css                 1,011.89 kB │ gzip: 160.03 kB
[log] build/assets/index-a5ed7f5e.js                  2,495.69 kB │ gzip: 702.31 kB │ map: 8,275.15 kB
[warn] 
(!) Some chunks are larger than 500 kBs after minification. Consider:
- Using dynamic import() to code-split the application
- Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/configuration-options/#output-manualchunks
- Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.
[info] 
PWA v0.14.1
mode      injectManifest
precache  6 entries (3652.62 KiB)
files generated
  build/service-worker.js
  build/service-worker.js.map
[log]   ➜  Local:   http://localhost:7357/
[log]   ➜  Network: use --host to expose
> frontend@8.1.0 cypress:headless
> cypress run --browser chrome oauth_authority=https://my-auth-provider.com,oauth_client_id=frontend
[STARTED] Task without title.
[SUCCESS] Task without title.
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[565:0417/164505.971691:ERROR:gpu_memory_buffer_support_x11.cc([44](https://gitlab.com/my-org/frontend/frontend/-/jobs/4127394753#L44))] dri3 extension not supported.
====================================================================================================
  (Run Starting)
  
  CYPRESS LOGS HERE  - LET ME KNOW IF YOU NEED THEM

The only difference is that Cypress can resolve the running web-server and the application locally but not in the CI. As already said, 1.14.0 forks fine - locally and in the CI job.

MikeMcC399 · 2023-04-18T08:26:21Z

@waldemarennsaed

I don't see the failure in your logs. Perhaps you are getting something like:

Error: connect ECONNREFUSED 127.0.0.1:5173

How are you starting vite?

If you are getting the above error then

npx vite --host

may get you better results.

See Vite documentation Server Options server.host.

"Specify which IP addresses the server should listen on. Set this to 0.0.0.0 or true to listen on all addresses, including LAN and public addresses.

This can be set via the CLI using --host 0.0.0.0 or --host."

waldemarennsaed · 2023-09-06T12:37:44Z

@waldemarennsaed

I don't see the failure in your logs. Perhaps you are getting something like:

Error: connect ECONNREFUSED 127.0.0.1:5173

How are you starting vite?

If you are getting the above error then
npx vite --host
may get you better results.

See Vite documentation Server Options server.host.

"Specify which IP addresses the server should listen on. Set this to 0.0.0.0 or true to listen on all addresses, including LAN and public addresses.

This can be set via the CLI using --host 0.0.0.0 or --host."

Sorry for my late response! But yes, this fixed it. My bad, was too easy ... 😄

waldemarennsaed · 2023-09-26T08:51:16Z

Update

No, I have not fixed my issue that way (as I wrote in my previous comment). I still had the issue that cypress was suddenly not running correctly in the CI and noticed the following:

Local setup

In my local setup, I run cypress with an existing cypress.env.json file that contains environment variables for our OAUTH provider (authority & client-id) that we then are mocking (to not actually connect to the auth provider).

The mocking just sets a localStorage variable, containing the key as oauth.client:https://my-auth-provider.com;client-id;. That key contains the mock-values for auth-tokens etc.

Executing the script that we use in the CI (GitLab) locally, works fine. The storage variable is being set correctly.

However, the CI setup & execution is a bit different.

CI setup

For once, the repository does not include the cypress.env.json. Instead, we are passing the environment variables via the CLI command:

.gitlab-yml.ci file:

script:
    - npm run ci:test:e2e -- --env oauth_authority=${E2E_OAUTH_AUTHORITY},oauth_client_id=${E2E_OAUTH_CLIENT_ID}

while E2E_OAUTH_AUTHORITY and E2E_OAUTH_CLIENT_ID include the same data as we pass in our local cypress.env.json - containing the auth-provider data.

It appears, that this was working _before the upgrade to _2.0.0 from our previously used version 1.14.0 but after the upgrade, the CLI-commands are not being passed correctly.

The npm run ci:test:e2e command executes the following:

{
  "scripts": {
    "ci:test:e2e": "tests/e2e/scripts/start-headless-e2e-tests.sh"
  }
}

with the start-headless-e2e-tests.sh being:

#!/bin/bash

# the arguments passed to this script are handed to npm run cypress:headless
npx start-server-and-test preview:e2e http://localhost:7357 \'npm run cypress:headless -- "$@"\'

with npm run cypress:headless being:

{
  "scripts": {
    "cypress:headless": "cypress run --browser chrome",
  }
}

So far so good. But something must have happened between version 1.14.0 and 2.0.0 that the arguments of my .gitlab-ci.yml file were not passed down to cypress run anymore, so my env-variables could not been initialized and my auth-setup was breking - thus, my Cypress E2E tests were failing.

I fixed this by creating a cypress.env.json in my CI job which cypress then takes, so finally my issue is fixed.

I still wonder why the arguments could not been passed down anymore, only due to the upgrade of 1.14.0 to 2.0.0.
If there is an answer to this question, I would be very happy. Otherwise, my part is done :) Thanks for the replies!

bahmutov added 3 commits December 5, 2022 16:12

feat: add --expect argument parsing

1f1e77e

feat: specify the --expected http status code

2ade4e8

chore: update the README

b100c30

bahmutov merged commit 17a427c into master Dec 5, 2022

github-actions bot added the released label Dec 5, 2022

muratkeremozcan mentioned this pull request Dec 5, 2022

Enable testing serverless locally cypress-io/github-action#647

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use the --expect parameter to specify HTTP response code #343

feat: use the --expect parameter to specify HTTP response code #343

bahmutov commented Dec 5, 2022

github-actions bot commented Dec 5, 2022

waldemarennsaed commented Apr 17, 2023

MikeMcC399 commented Apr 17, 2023

waldemarennsaed commented Apr 18, 2023 •

edited

Loading

MikeMcC399 commented Apr 18, 2023

waldemarennsaed commented Sep 6, 2023

waldemarennsaed commented Sep 26, 2023

feat: use the --expect parameter to specify HTTP response code #343

feat: use the --expect parameter to specify HTTP response code #343

Conversation

bahmutov commented Dec 5, 2022

github-actions bot commented Dec 5, 2022

waldemarennsaed commented Apr 17, 2023

Specific issue

MikeMcC399 commented Apr 17, 2023

waldemarennsaed commented Apr 18, 2023 • edited Loading

Logs

MikeMcC399 commented Apr 18, 2023

waldemarennsaed commented Sep 6, 2023

waldemarennsaed commented Sep 26, 2023

Update

Local setup

CI setup

waldemarennsaed commented Apr 18, 2023 •

edited

Loading