Testing frontend right: from unit to end-to-end

Introduction

When we last wrote about CI/CD, we briefly touched on how tests enrich your pipeline, but can also be the biggest source of its slowdown. There the advice was to optimise your tests and make sure they are fit for purpose. I thought I’d elaborate a bit more with a focus on frontend tests since that is my focus and in the past, they were one of the major bottlenecks in our CI/CD.

This post is about what we’ve learned about frontend testing, some of the pitfalls, and how to avoid them. Let’s dive into it.

Disclaimer

For this blog post, we will be talking mainly about React but the ideas should work with some thought for other frameworks like Vue, Stencil, and Angular. So when we talk about Components, Hooks, and other terms, we are talking about React.

What is Frontend Testing?

Frontend testing is the discipline of verifying the user-facing interface of a software application. It requires knowledge about requirements related to user experience and API integration but not about the database and the back-end mechanics.

It entails testing your application's appearance, whether everything is in the right place, and how it responds to user interactions. You would also test that it looks the same in the various browsers and on different devices.

But it is not just the user interface that needs testing, since frontend applications can also have business logic, you could also test that your interface displays the correct data for the situation and that it handles integration with the backend correctly. There are many parts that you test on the frontend, not just the UX which is why there are many different types of frontend tests.

Testing Philosophy

Ideally, when you test on the frontend you do not want to assume any underlying implementation details. This is because your tests should still pass when you change the underlying implementation but the functionality remains the same.

So for example you have a component that has some functional logic, ideally, you’d separate the render logic from the functional logic, maybe using a hook if you’re on React, and test each of those separately. Your tests should be concerned with the interface, i.e. the input and output, and not whatever state shenanigans happen inside them. That way if the implementation of the unit changes, your tests will still work provided the interface stays the same.

Why does frontend testing feel painful?

Frontend testing is a different beast from backend testing. You have to deal with too many interconnected moving parts and asynchronously updating components that make getting any idea of how things work a bit more complicated. To make things more interesting, end users tend to use the system in “creative” ways that make predicting how the system responds tricky at best. Add to this, situations like race conditions, and mocking global values like React providers, it’s a miracle that we even do proper testing at all.

The tools we use to do testing try to help but they can also be part of the problem, for example, the mocked elements for some libraries don’t support the full API for the real elements, e.g. I once had a problem that MockedHTMLElelemnt didn’t support the real element’s setValidity.

Another pain point is the landscape, frontend applications can be used on browsers, phones, tablets, and smart devices just to name a few and you need to test that they work properly on all relevant operating systems, browsers, and their respective versions.

But there is some light at the end of the tunnel, and there are ways you can make your life easier.

Types of Frontend Testing

Let’s first define the types of tests you will encounter and then talk about the different strategies you can use to work with them.

There are several types of tests: unit tests, integration tests, visual regression tests, cross-browser tests, accessibility tests, acceptance tests, and end-to-end tests.

Unit tests

A unit test tests the smallest functional unit of code. This could be a component, function, or hook on the frontend. The idea is that it’s easier and usually only reasonably possible to cover all possible cases with unit tests. Trying this with Integration tests and E2E tests quickly gets out of hand. I’ll get more into the why when I talk about which tests to write.

Integration Tests

Integration testing means combining two or more units or modules in your test. You might be testing several frontend components to see how they interact with each other. Ideally, you do not want unexpected side effects due to the combination of several units. By definition, integration tests take more time to run than unit tests and need to account for more combinations of outcomes.

End To End Tests (E2E Tests)

E2E testing is testing the whole application from start to finish as if you were a real-world user. This means that all parts of the system must exist, from the UI to the database. In other words, nothing internal should be mocked, you want to test that everything works as it would for an end user using your system.

The End to End in E2E testing can have two meanings, each equally valid. It might refer to the start of a workflow to its end and it might also refer to the architecture, i.e. from the UI to the database.

Either way, E2E tests must match what your users will use in the real world. Everything has to be there. Everything in this case means all units. For example, if you are making a shopping cart then an E2E test could be testing that the whole process of purchasing a product works. This doesn’t mean that your tests should be one giant test, you should still have them implemented in smaller more manageable steps that share a setup.

E2E tests are made from the user's point of view and are done against all parts of a system, preferably in a staging environment that is identical to the production system. This can be in a proper staging clone of production or on a headless instance that your CI/CD uses, the only requirement is that you have a full system with a working API, frontend, and backend.

E2E tests test real business case workflows, reloads, and all, ensuring that the overall results of a user workflow is as expected.

Others

The above are the three basic types. Other frontend-specific types can fall within unit and integration, and I am listing them separately here because they have some pitfalls you should look out for.

Visual tests

Unit tests do not have eyeballs, so they can not catch subtle changes to the UI like a change of a few pixels because of some weird CSS interaction. Visual tests are when you take a visual snapshot of your UI and compare that to some baseline image.

Usability tests

Usability tests check if your UI is easy to use. A special type of usability test is an accessibility test, which tests if people using things other than a computer can access your UI, like people using a screen reader. These days pretty much everything is automated so you have tools like lighthouse that will give your app a usability score and make suggestions on how to improve your UI.

Cross-browser tests

Since web applications can be accessed on several devices, ranging from proper browsers on a computer to the display on a smart fridge, you need to test that they work properly on all the relevant devices. Like we said, pretty much everything is automated, so there are tools to do this for you. And you don’t need physical access to each type of device since we’ve gotten pretty good at virtualization.

Which to write?

The different test types on the frontend each have their advantages and disadvantages. There are a lot of models out there that try to highlight these advantages but I won’t be using those here.

Let’s keep it simple and break it down by category.

Speed

How fast does a test take to run?

When we talk speed, unit tests win hands down. They test single units and require the least amount of setup.

Integration tests tend to be slower than unit tests and E2E tests run the slowest. This is going to be a common theme, unit tests and E2E tests are always at opposite ends of any spectrum, and integration tests are that awkward middle child.

Maintainability

How easy are the tests to maintain?

Here again, because of unit tests' small sizes, they are easier to maintain. Integration tests are slightly harder to maintain since they integrate several units and E2E tests are the worst to maintain.

Flakiness

How predictable is the test? As in, if you run it 50 times, do you get the same result 50 times?

This unpredictability is called flakiness. Flakiness becomes more likely the more moving parts you have in a test. So while unit tests can and often are flaky, you are more likely to have more flakiness in integration tests and E2E tests.

Fidelity

How close are the tests to real-life situations?

Let’s face it, the last two categories all painted unit tests like they did everything better right? But those advantages come from trading off fidelity. Unit tests can be those things because you are testing just a single unit, in isolation. But that’s not how they will be used. So in terms of fidelity, unit tests have the worst, integrations tests sit in the middle and E2E tests have the best.

Cost

How much does it cost to run the test?

“The cost of anything development-related is measured in dollars or hours.” I have used this phrase or a variation of it several times in the past and it’s valid here. In terms of costs, unit tests are the cheapest, integration tests, are still the middle child, and E2E tests are the most expensive. This can be the dollar value you have to pay to run your tests on your CI/CD or the time the test would take to run.

In fact, the difference between unit tests and E2E tests is so huge that in some cases you can easily afford to run hundreds of unit tests in the time and for a fraction of the cost it would require to run a single E2E test.

Verdict

Which tests should you write the most? My suggestion is because unit tests give the best trade-off, they should be the majority of tests you write. Integration tests should be fewer than unit tests, and E2E fewer than that.

Unit Tests

Unit tests present a few common pitfalls you should be aware of. Since this is the most common type of test we’ll spend the most time here.

Testing Implementation Details

The first pitfall is testing implementation details instead of testing behavior. This is a problem because it ties your tests to the particular implementation and if that implementation changes it invalidates the whole test. This isn’t unique to frontend tests so it’s a good thing to remember even if you don’t work on UI.

The tip here is to test the behavior of the unit, not how it's implemented. First, this means that you should write your code to be testable, this is not purely a testing topic but it applies so we’ll include it. Well-structured code is easier to test than messy code.

Second, try not to reference a component's internal state when testing. I’ve seen this referred to as black box testing. Imagine your unit or component as a black box, you don’t know what is inside so you test how the box interacts with the outside world. So for example, if you have a dropdown component you test what happens when the user clicks it, i.e. a dropdown is shown, instead of checking the state changes one way or another.

Testing behavior means that if the implementation details change, your tests are still relevant. So for example, if you create a hook, testing behavior would mean testing that if you give it x input, it returns y output. If the inner workings of the hook change, x should still give y.

Large Test

It’s very easy to fall into the trap of large tests that test many things. This is bad since it hides failures. Large tests are also hard to read and understand, if they fail it’s harder to figure out why and even harder to adjust.

So make your unit tests focused and small. Tests should test a single thing, not multiple things. It’s better to have a lot of small tests rather than one large test.

Non Descriptive name

Naming is one of the hardest things to do in software development and that holds true for tests as well. It’s easy to fall victim to bad or misleading names when writing tests. If you look at the list of tests and can not figure out easily what the application is doing then you have terrible names.

The trick is to use clear names, don’t get cute with your names, be clear. The names should tell you what is being tested and the expected result.

So this is good
it("clicking button navigates to the homepage");

And this is bad
it("homepage");

Only testing the happy path

We touched on this point when we talked about CI/CD, but one common pitfall is only considering the happy path. Tests that do this are misleading at best and dangerous at worst. You should never only test that something goes well, you should also test what happens when something goes wrong. You should test failure cases and you should test edge cases.

Test all cases that you can think of. If you find a bug in the code, then you should add a test for that case so that it does not regress. Testing the happy path is not enough.

Another part not often handled is errors and exceptions, if your component has some validation handling or error handling, then that should be tested.

Not having proper test isolation

Imagine this scenario, you write several tests for a component, your tests all pass, and everything is right in the world. Then you want to add a new case, you add that one test and 5 other unrelated tests suddenly fail. But if you remove that test or change the order they pass.

What is happening here is that your tests are not properly isolated from each other and are bleeding into each other. This can be due to mocks that aren’t reset, unhandled async operations, tests that modify some piece of state, etc. The list is surprisingly large.

Bleeding tests can give you a false sense of security and are a pain to debug. So if you are using mocks, clear those suckers, or better yet do it beforeEach test is run. Make sure to clear or restore mocks afterEach test. If you are testing async functions then waitFor them, if you are using timers then useFakeTimers and if you have the time, run each test by itself.

Integration tests

Moving on to integration tests, a few things are relevant here.

Complex setup and teardown

By their very nature integration tests require more setup and teardown than unit tests and this can present a pitfall when this setup starts to get complex.

The tip here is to automate where possible and mock where required. In principle, since integration tests need to be closer to reality you need to mock less than in unit testing. The idea is not to have fewer mocks, the number of objects mocked is not a great indicator, but rather that the units integrated are as close to realistic working scenarios as possible. So if you are testing that 3 units or components work together then it’s fine to mock 5 others that are not involved in the tested interaction

Also since integration tests can get complicated quickly, the rule on keeping them small gets amended to “Keep them small where practical to do so”. For example, if you have a long setup, then it might be better to do it once and test several cases, instead of redoing the setup. You could have several tests that use the same setup or run several assertions in one test. If you do run several assertions make sure they are related. You do not want a situation where a single test is testing two or more unrelated functionalities.

Also, be aware that shared setup can lead to one test bleeding into another, so sometimes you may have no choice but to re-setup, or if you are lucky resetting mocks may be quicker than full setups.

Over mocking

It might sound weird but there is such a thing as over mocking. Since integration tests test how well units or services play with each other we recommend using realistic data where possible. So while it is ok to mock unrelated units, try by all means to have the units under test to be as close to reality as possible.

Using Integration tests to test units

Integration tests tend to be larger and take longer to run than unit tests. They are also harder to maintain so testing things that can be done at the unit level here might not be the best approach. For example, when running an integration test for a section you could give some valid input and some invalid input but testing how units respond to all edge cases might be better done at the unit level. You will have overlap in scenarios but I believe that is fine.

Unstable environments

This sounds obvious but it bears saying. The environment you run your tests on should be identical locally, on your CI/CD pipeline, AND in production. There is nothing more annoying than a test that passes locally but fails on CI/CD. You do not want tests that only pass on one developer's machine but fail on another.

Docker is your friend here, use docker locally and on your CI/CD. We suggest docker because we love docker but in all honesty, any tool that allows you to create reproducible development environments will also work. So if you prefer some other tool that gets the job done then go for it.

Visual Tests

Checking very minor changes

One common pitfall of visual tests is that they can appear flaky because of very minor changes. Let’s say you update your CSS and the layout shifts by one pixel. If your threshold for changes is low enough your tests will fail. Personally, I’m not strict about 1-pixel changes because I believe that users can not tell the difference and they aren’t big enough of a change to warrant the time spent to fix them.

So the tip here is to set your threshold for what is defined as a change to a reasonable level. Of course, this is subjective, and if you need to be pixel-perfect then a very low threshold for change might be a good thing.

Not checking changes

The number of visual tests tends to rise very quickly. For example, if you are testing 20 components you may end up having separate tests for the happy path, edge cases, and errors. That is a lot of snapshots to check. The pitfall here is that if a developer makes too large a change it may end up breaking a lot of tests and then checking all of them could be time-consuming. Developers are people and what may end up happening is they regenerate the comparisons without really checking.

The way around this is to always try to keep changes small. This doesn’t mean that all your PRs should be tiny, it means when you make a change, you should check your snapshots then and there and update if necessary. Basically, you should be more religious about the development/test iteration cycle.

Cross-browser Testing

Browsers handle CSS differently, it’s a fact of life and one you’d need to accept. The same page will look slightly different from browser to browser. The pitfall here is not including different browsers in your visual tests.

There is an argument to be had about whether this is a bad thing and a point against cross-browser testing in general since the only thing we are doing is testing the various browsers for their quirks but I think that’s an argument best left undebated for now.

The advice here is to run your visual tests on multiple browsers and devices. Thankfully a lot of the test runners support this feature so make sure to have your target browsers and devices configured.

Ignoring orientation

Mobile tests often focus on one orientation and ignore the concept of users flipping their phones. This doesn’t usually have dire consequences, but it is good practice to include other orientations in your tests.

One stumbling block to note here is that this, like browser or resolution testing, can cause the number of tests you will run to increase dramatically. Consider 10 components, over 5 resolutions over 2 orientations and suddenly you have 100 test scenarios.

Dynamic content

You should also take into consideration any dynamic content. If your UI has dynamic content and if you do not take that into account you will have flaky visual tests. Here is a trivial example, a component that shows the current time and date. It will fail whenever you run the test.

The trick here is to mock such data such that they are always the same. I’d also advocate for mocking external images and any external content like cms text.

Cost

If you’ve done all the above then you will have a ton of tests, and if you are using an external CI/CD, then costs will rise significantly. It can get so bad that honestly, visual testing can be the most expensive type of test to run. And considering the speed aspect it can be the greatest contributor to your CI/CD pipeline's slowdown.

I know it may sound weird after I spent the last few minutes talking about how to get the most out of your visual tests to pivot suddenly but here we are. You should use your best judgment on what to test. Visually testing every component in multiple ways may not be the best approach, there is no simple solution here since each case may be different, you need to consider it on a case-by-case basis. So for example, you could have visual tests for your home page but not for the admin page. Again, only you can make that call.

Accessibility Tests

To my mind, most of the pitfalls of accessibility testing come from not having an accessibility plan in the first place. If you don’t have it as a requirement from the get-go, it is all too easy to ignore it.

For example, one problem is missing alt texts on images, but if you have a plan that specifies this as a requirement you can configure your linter to flag images that do not have an alt tag.

Automated accessibility tests also do not get as much attention as other tests, and get neglected as a result.

So my advice, have a plan and use lighthouse or something similar.

End to End tests

Time to Run

By definition, E2E tests are large and can take a very long time to run. Some people prefer to avoid them altogether but I feel if you must have them then be strategic about when you run them. It might be more useful to run them only when deploying to a test server rather than whenever a PR is created or updated.

Focus

Designing E2E tests can be complicated. One pitfall is focusing on the wrong things. E2E tests should not focus on your app, i.e., on implementation but instead on the end user. Ideally, you want to only include high-value paths in your tests, i.e., focusing on how end-users will go through the system to accomplish a goal.

If you recall, we said you should write more units than integration tests and more integration than E2E tests. So do not go crazy and write a million E2E tests, they should be focused on what brings the best value.

Conclusion

Testing is a vital part of development, and it’s a huge topic even if we are just focusing on the frontend. I am honestly just scratching the surface on what you need to be careful of and I’ll probably write more about the topic later with more focus on each aspect. If you’d be interested in that you can follow our LinkedIn page not to miss an update and if you would like to work with a partner that knows all about testing, you can contact us.