I wrote some flaky frontend tests using Cypress and then learned how to make them deterministic. This is the tale of that adventure.

An application I am working has a video player that should continue playing when changing pages. Additionally the player should be visible on the dedicated page of the video it is playing and hidden on other pages.

The <video> element cannot move within the DOM, because it would be destroyed and recreated, which interrupts the playback. The trick is to only change part of the page. That particular application achieves that with pjax (which is also still used by Github at the time of writing -- search for "pjax" in their HTML responses). The player lies outside the part that gets swapped during navigation. It is placed in a predefined area using CSS positioning.

I wanted to replace a lot of the player code, but there was no test coverage of the aforementioned requirements. I decided to add some using Cypress. I ended up with a long-ish integration test that does the following:

  1. Visit a page where the player should be displayed and press play
  2. Assert the player is covering its dedicated placeholder
  3. Navigate to a another page and assert the player is hidden
  4. Navigate to a page with a placeholder/preview of a different video and assert the player remains hidden
  5. Go back to the page of the currently playing media and assert the player is placed correctly again

The first assertions was already causing me problems as a newcomer to Cypress.

Scroll, scroll, scroll

Cypress scrolls an element into view before clicking. The scrolling algorithm puts the element at the top of the page.

If the top of the page is covered by a floating header that always remains there, you will need to disable the checks before the click:

cy.get(selector).click({force: true})

Working directly with a DOM element

The test clicks the play button and some music plays. I don't want the hear the music when running the test suite, so I tried to mute or pause it.

Usually, you would call a function with invoke. This does not work though:

cy.get('[data-cy=player]')
  .should('be.visible')
  .invoke('pause')

It fails with the following error:

cy.invoke() waited for the specified property 'pause' to exist, but it never did.

That is because get() returns a list of elements even if there's only one.

My first instinct to unwrap it also did not work:

// Results in error
cy.get('@player').invoke('get', 0).invoke('pause')

This did:

cy.get('@player').then(element => element.get(0).pause())

Flaky behaviour

I then went on to test that the player covers its placeholder:

cy.get('[data-cy=player]').then((player) => {
  cy.get('[data-cy=player-placeholder]').then((placeholder) => {
    cy.wrap(player.offset()).should('deep.eq', placeholder.offset())
    cy.wrap(player.width()).should('eq', placeholder.width())
    cy.wrap(player.height()).should('eq', placeholder.height())
  })
})

That worked sometimes in the Cypress UI and never in headless mode. Because the player is only displayed once it starts playing, sometimes Cypress would check its position before it has moved.

CypressError: Timed out retrying: expected { top: -10000, left: 0 } to deeply equal { top: 138, left: 0 }

Cypress deals with race conditions by automatically retrying commands and assertions.

It does not retry every command, though, and I spent a long time searching for a way to retry getting the player position and dimensions. Some of it was reading the Cypress source code. One attempt even resulted in an infinite loop that kept retrieving the same DOM element!

I eventually found a comment describing a hack using the public API.

Make any function call retry-able

The trick involves a custom command.

function try(subject, fn) {
  return cy.wrap({ try: () => fn(subject) }, { log: false })
    .invoke('try')
}

(log is set to false to exclude these implementation details from the command trace in the Cypress log)

Finally, I had a reliable test.

cy.get('[data-cy=player-placeholder]').then(placeholder => {
  cy.get('@player').try(boundingBox).should(box => {
    assert.deepEqual(box, boundingBox(placeholder),
      'the player is covering its placeholder during playback')
  })
})

// ...

function boundingBox(element) {
  return element[0].getBoundingClientRect()
}

When the assertion in the callback to should() fails, Cypress retries the last command in the chain.

More flakiness

Funnily enough I had another race condition in the test that I did not understand until I started writing this post. The music did not pause during some test runs. The real test code that should pause it is as follows.

cy.get('[data-cy=player]')
  .should('be.visible')
  .then(container => container.find('audio').get(0).pause())

The culprit is the visibility assertion. The element is hidden by default using a large negative margin for historical reasons involving old Internet Explorer versions and a Flash fallback for the player. IE would completely disregard the Flash object if it had visiblity: hidden. The workaround that tricked IE into believing the off-screen player was visible also has an effect on Cypress. There is nothing in Cypress' definition of visibility saying an element which is ten thousand pixels above the viewport is hidden.

My assumption was that a "visible" player is playing, because in the implementation we change its coordinates after starting playback. However for Cypress the element is always visible, so it immediately tries to pause which causes the race condition.

I can think of two ways of fixing this. One is to mute the sound at the start of the test and let the audio/video play.

cy.get('[data-cy=player] audio')
  .then(player => player[0].muted = true)

The other is to explicitly wait for the playback to start before pausing:

cy.get('[data-cy=player] audio')
  .should($element => {
    const player = $element.get(0)
    assert.isTrue(isPlaying(player), "the media is playing")
    player.pause()
  })

// ...

function isPlaying(media) {
  return (media.duration > 0 &&
    !media.paused &&
    !media.ended &&
    media.readyState > 2);
}

should() is retried if the assertion in the callback fails. Pausing the player after the assertion ensures it will be playing by that time. (The isPlaying() helper is a bit awkward, because there is no playing property on HTMLMediaElement.)

How can we make sure this works? By introducing a delay in the production code.

new Promise(resolve => setTimeout(resolve, 2000))
  .then(() => play())