GraphQL As A Platform For Testing
By Mike Harris
A Quick Overview Of GraphQL
GraphQL was created by Facebook and was first released in 2015. It is now widely used in web applications. GraphQL is a way of building and querying APIs. GraphQL creates a specification for client-server communication. It can also be seen as a query language that is used to describe an API.
Each implementation of GraphQL has a schema. The core objects in your application are defined in the GraphQL schema as types. Interactions using GraphQL are made via operations, which are also defined in a schema. There are three operations in GraphQL:
- Queries, which are used to read information from GraphQL
- Mutations which are used for create, update and delete actions
- Subscriptions which are used to listen for updates
The book “Learning GraphQL” by Eve Porcello and Alex Banks is a great resource to learn more about GraphQL.
Exploratory Testing Of GraphQL With An IDE
Exploratory testing is often talked about as being executed via the UI, but we can also execute exploratory testing through GraphQL. Exploratory testing is defined by Cem Kaner as:
"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."
In her book “Explore It!,” Elizabeth Hendrickson has a section about exploring an API, and this is very similar to how we can explore with GraphQL.
The GraphQL schema is a useful place to start exploratory testing as it defines your implementation of GraphQL. The operations in the implementation of GraphQL are all defined in the schema. Analysing the schema is a good way to start testing so you can understand the GraphQL operations. Below is a mock-up of part of a schema:
By exploring the GraphQL operations, such as queries and mutations, you can discover useful information about the arguments developers have created. This approach proved useful, as there were several which would need additional arguments to be able to be used in a way which is expected. When discussed with the developers, they asked that cards be raised for the arguments to be added to the operations.
You can also test each of the operations in a schema. If you test each operation you are able to analyse each part of each operation and see which parts of each operation you can change and test them. An example of this is that some of the queries may contain SQL and you are able to test different pieces of SQL for the supported database systems. If you consult the documentation for each of the supported database systems and find the pieces of SQL that caused unexpected results when included in a query you can check that the unexpected result is not due to you writing the SQL incorrectly.
There are IDE’s that can be used to explore and test via GraphQL. These IDE’s enable you to test via GraphQL using the operations that are part of the application that you are testing and enable you to view the results of these queries. GraphQL operations run against an API and return a response as JSON.
GraphiQL is an IDE that you can use for testing. It allows you to interactively explore a GraphQL interface by writing and running queries and mutations. GraphiQL can be used to test a GraphQL layer in a similar way that Postman can be used to test a REST API layer. You can send HTTP requests to the REST API via Postman and with GraphiQL you can send GraphQL operations to the GraphQL interface. Both tools also enable you to test the response that you receive.
GitHub's Explorer, which is based on GraphiQL and is for a public API, is shown below. The query is on the left and the response is on the right of the screen:
A GraphQL IDE can be used to look at the results that were returned and use the results of one test to inform the next. This can enable you to explore the GraphQL interface that the developers have written. These exploratory tests can find issues that can be raised. You may also find gaps in the automated tests, which you can then fill. The IDE can be used to investigate the operations and analyse how they could be used, and that way find issues that need to be resolved.
Some black box testing techniques such as equivalence partitioning and boundary value analysis are helpful in designing tests. These are techniques for designing tests for inputs via a UI and can also be used for designing tests to be run via GraphQL. Use of these techniques can give you test cases that provide good test coverage, find issues and can be automated. Boris Beizer’s book “Black-box testing” describes more techniques that may be used to derive test cases.
Advantages Of Exploratory Testing Via GraphQL
A Deeper View Of The Application
Through exploring the GraphQL interface you can get a deeper view of the application that you are testing as you can see the objects and data structures that are returned by the queries, and are used by the web pages that we are developing. Being able to see the results of the queries gives you a better understanding of the functionality that your team is developing, and so helps with analysing the application and the testing that needs to be done. You can experiment with passing various values as arguments to queries, and see what results the queries return. As a result of this better understanding, you are more able to have useful conversations with the developers about how the functionality should work, which will help you to identify bugs.
A Better Understanding Of Test Coverage
GraphQL also enables you to judge the level of test coverage you are getting by using the schema. The schema details all the operations that are in the implementation of GraphQL that you are testing, so you can ensure that you have tested all the operations in the implementation if you are testing by working through all the operations in the schema. You are also able to decide, within the time box that you have, on which operations to get a deeper test coverage. It would have been difficult to have done this while testing via the UI, this visibility of test coverage is an advantage of testing via GraphQL.
Testing via GraphQL means that testing can start before the UI is complete. You are able to test the GraphQL queries that the UI will use before the UI is written, and are able to identify issues and gain familiarity with the functionality early in the development of the new functionality. It is also interesting to see how the objects in the UI are represented in the query results returned by GraphQL, these can give you ideas for new angles for testing the application and you can test them now without waiting for the UI. As well as executing functional tests you can also do some security testing via GraphQL, for example, by submitting operations that include strings to test SQL injection, this is useful in gaining confidence early in the development process.
Providing Help For Your Team
The GraphQL schema details the types, and operations that the GraphQL implementation supports, however even with the schema GraphQL queries can appear hard to write. A ‘help page’ called Documentation Explorer is generated from the schema but it does not contain examples. Examples are useful in gaining an understanding of any new area of functionality, and examples of GraphQL queries help when writing queries for the first time. As you test you can create a library of example queries using the queries that you had used for testing, and this can be shared with your team.This helps the wider team use the GraphQL layer, and will help you when you return to testing the GraphQL layer. The front page of the Document Explorer is shown below.
GraphQL As A Test Automation Platform
Tests that are automated using GraphQL are integration tests, as a number of services and modules are integrated and are being tested at the GraphQL layer. They sit in a layer between unit tests and UI/Selenium/Cucumber tests.
The “Test Pyramid” as defined by Mike Cohn in his book “Succeeding with Agile” shows that there should be a large number of unit tests, a smaller number of “service” tests, to test each service and an even smaller number of tests running through the UI. This enables automated testing to be carried out at the lowest level possible in the application, and so minimise the number of tests run through the UI.
GraphQL enables you to run a layer of integration tests in what Cohn described as the service layer. The service or middle layer of tests often lack automated tests, as teams create unit tests and automated tests via the UI but rarely create many integration tests. This can create pressure to automate an increased number of tests via the UI in order to cover a wide range of paths through the application under test. Integration tests at the GraphQL layer can test a wide range of paths through the application that would otherwise need to be tested via the UI, and so can remove the pressure to create a large test pack that runs through the UI.
Advantages Of Automated Testing Via GraphQL
Automating tests via GraphQL avoids having a large number of tests automated through the UI. Tests that are automated via the UI can be cumbersome as they can be flaky, take a long time to run, and need maintaining each time the UI changes.
Through GraphQL you can test functionality without having to deploy the UI. You can also test the same functionality as you would via the UI but do so via GraphQL. This means that the tests are less flaky so less time is lost re-running tests that have failed and less time is spent on maintaining the flaky tests. As the automated GraphQL tests are less flaky than UI tests they also increase confidence in the automated testing process and avoid those conversations about the tests falling over or returning spurious errors. Tests running in GraphQL also run quickly as the tests do not have to spend time navigating around the UI.
It is common for many parts of UI’s to be updated regularly, while the underlying functionality does not change, and if you are running automated tests via the UI you then have to update the automated tests each time the UI changes. GraphQL tests do not need to be maintained each time the UI changes as they do not use the UI. These factors enable the testers to increase code coverage of automated tests.
We created two automated test packs using GraphQL. The first was created by my colleague Jon Normington who created a pack that used fuzzing to run a large number of tests with random values. This pack gives us broad and deep test coverage. It also finds issues that would take time for us to find manually. We also created an automated pack of functional tests that runs against a list of supported types of input to our application, and so guarantees that every deployment supports the input types tested in the pack. The test pack of functional tests is easy to understand as the tests do not rely on finding a way through the UI for each test, you just have to make the appropriate call to GraphQL. Both of these packs run in CI on every merge and so help give developers confidence to deploy to production frequently.
Leveraging The Benefits Of GraphQL
Developing applications with GraphQL helps to create a layer of integration tests and has a number of benefits. GraphQL can be viewed as a platform for testing, as it enables testing of the service layer of the application. It provides a useful platform for testing as it enables a number of manual and automated approaches to be taken for testing. The range of testing and speed in which automated tests run against GraphQL means that developing applications with GraphQL can help teams gain confidence in the application and enables software to be released to customers quickly.
- A Tutorial in Exploratory Testing - Cem Kaner
Mike has been a tester for 20 years and is currently the tester for Geckoboard. He has been a Tester and a Test Lead and worked as a part of waterfall, lean and agile teams. Mike has a B.Sc.(HONS) from Middlesex University and is an Associate of the University of Hertfordshire. He has also set up and led a Testing Community of Practice and been part of a successful agile transition.
In his spare time, Mike coaches his daughter's cricket team.
Follow Mike on Twitter: @TestAndAnalysis