Automated Testing in the Cloud with TestChameleon

As is generally known, quality assurance in software development cannot exist without software testing. Developers and dev teams often deal with incompatibility issues, because of the fact that they do not have access to every single environment to test their software on.

In line with what Anubha Jain and Vipin Jain described in their article Offering Performance Testing as a Cloud Service, the cloud can also offer a solution in this respect. In this article Georg Hansbauer wants to share some insights on automated testing via virtualization. Specifically, with TestChameleon, an infrastructure we have developed at Testbirds over the past few years.

Let’s say your organization developed an application for Windows 8. It could be part of your test plan to see how it runs on Windows 8.1, Windows 7, or even on Windows 10. Or maybe you need to try the latest build of your web app specifically on Chrome 38.0. and Firefox 33.0. With TestChameleon (TeCh), it is possible to test how an application functions on different operating systems containing, for example, specific browser versions, without the need to actually acquire those environments and devices. Better yet, you can personally compose any combination of needed software and, in addition, you can install your own applications. This makes it a lot easier to test software for incompatibility and inconsistencies.

On-demand test infrastructure

TestChameleon is an on-demand infrastructure in which disposable virtual machines (VMs) are constructed. Accordingly, these VMs can be set up with almost every possible software combination. Testers receive unrestricted administrator rights to their VMs and are able to install any kind of software on it. Whether it’s a single application that needs to be installed or an entire .NET Framework, the tester can arrange the VM as he or she wishes. In approximately one minute, the VM will be up and running. Testers then have the option to run a manual test via the web front end. They can click around and try their scripts on the software they’ve installed. Or, alternatively, the extensive API allows testers to execute automated tests, for example by means of Selenium or SikuliX.

Even though TeCh was mainly developed as a cloud solution, it’s also possible to offer the infrastructure as an on-premises solution. In principle, there are three components that make up TestChameleon: a controller with a central database, at least one VM-host and a web interface. Fig. 1 gives an overview of the architecture behind TeCh. The infrastructure is highly scalable and management completely takes place within the controller. Allocation of the VMs on the hosts happens with our own unique algorithm.

Fig. 1. The TestChameleon Architecture.

Fig. 1. The TestChameleon Architecture.

TestChameleon Dashboard

To give an impression of how TeCh works in practice, it might be good to view the ways in which tests can be executed. First, I’ll share some insights on how to create and operate a VM via our Dashboard. Second, I’ll go into more depth about creating automated tests by using the TestChameleon API.

Fig. 2. A new configuration of a virtual machine within TestChameleon. Testers can initiate a VM, customized to their own needs.

Fig. 2. A new configuration of a virtual machine within TestChameleon. Testers can initiate a VM, customized to their own needs.

After the tester logs in, he or she is taken to the TeCh Dashboard, where a new VM can be ordered. In Fig. 2 we can see how that looks like. The user can submit a name for the VM and decides which operating system should run on it. Accordingly, the architecture and build version are selected. And fortunately, it doesn’t stop there. TestChameleon offers several other software packages, ranging from browsers and plugins to archivers and Java, for example.

The tester can choose to save this specific configuration as a template, in case a similar set-up is needed in the future. Subsequently, TeCh will process the request and very soon after will present the desired VM. In Fig. 3 you can see it takes about a minute to have the VM ready for use. Next, by selecting the WebVNC tab, one can immediately begin testing in the newly created environment.

Not only do we offer pre-defined software, testers receive full administrator rights to install any kind of software they need to test.

3_TeCh_screenshot7b

Fig. 3. Dashboard view and WebVNC: After approximately one minute (see the red outlined box), the VM is up and running. With WebVNC, testers can directly operate the VM.

Fig. 3. Dashboard view and WebVNC: After approximately one minute (see the red outlined box), the VM is up and running. With WebVNC, testers can directly operate the VM.

TestChameleon API

The TestChameleon API is an automated remote testing API. If you want to test your web application on every device, and with every browser, our API can help to ease the testing process. In the simple demonstration below a version of the Selenium grid is being used. So, in this case, to work with the system you have to get an instance of the SeleniumTestcase.class. The singleton can be accessed and used to control your nodes. This class provides you with all the needed methods to execute your functional tests in the cloud. Here, a Windows 7 VM is started with Firefox 27 as a Selenium Node:

sc = SeleniumTestcase.getInst();
sc.loadIni("example.ini");
sc.startGrid();
ArrayList<SoftwarePackage> software = new ArrayList<SoftwarePackage>();
software.add(new SoftwarePackage("firefox", "27", "X86"));
OS os = new OS("WIN", "6.1", "X86");
singleVM = sc.startNode(os, software);
sc.waitAllReady();

If we look at this step by step, you can see that we start with communicating with the core of TeCh’s API (1). Then, we need to verify ourselves by providing the path to our .ini-file. This file contains all the necessary information to our key store and password (2). After, we can start our grid (3) and declare a simple ArrayList for the software we want to install on our node (4). We make sure we add the correct browser that is needed for our test case (5), next to the desired operating system (6). Now that we’ve got our operating system and software packages in the command, we can order and start the new node (7). Finally, we have to wait until all our nodes are ready (8). This can take some time, depending on the number of nodes that are being created.

In addition, this is an example of a command that initiates a Selenium WebDriver and enables the execution of the test case on the created node:

driver = sc.getWebDriver(singleVM);
driver.get("https://nest.testbirds.com");
assertEquals(driver.getTitle(), "Testbirds.de");

However, TestChameleon also provides simple usage of the API, without using a Selenium hub. This method can be applied for different kinds of scenarios, such as testing on Jenkins nodes, Sikuli test cases and so on. In this instance, lastly, the code would look like this:

import java.io.IOException;
import java.util.UUID;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.slf4j.LoggerFactory;
import de.testbirds.tech.data.OS;
import de.testbirds.tech.data.SoftwarePackage;
import de.testbirds.tech.testcase.SimpleTestcase;
import de.testbirds.tech.tools.TeChIni;

public class SimpleExample {

    /**
     * the uuid of the vm started by the testcase.
     */

    private static UUID firstVM;
    private static SimpleTestcase sc;
    private static final org.slf4j.Logger LOGGER = LoggerFactory.getLogger(SimpleExample.class); 

    /**
     * setup the test
     *
     * @throws Exception
     */

    @BeforeClass
    public static void setUp() throws Exception {
        TeChIni.getInst().loadIni("example.ini");
        sc = SimpleTestcase.getInst();
        firstVM = sc.startNode(new OS("WIN", "6.1 SP1", "X86"), new SoftwarePackage[] { new SoftwarePackage("firefox", "28.0", "X86") });
        sc.waitReady(firstVM);
    }

    /**
     * Executed the test
     *
     * @throws InterruptedException
     * @throws IOException
     */

    @Test
    public final void testSomething() throws InterruptedException, IOException {
        LOGGER.info("Running the test");
        Thread.sleep(1000*200);
        LOGGER.info("End the test");
    }

    /**
     * shut down the whole testcase.
     *
     * @throws Exception when shutdown fails
     */

    @AfterClass
    public static void tearDownAfterClass() throws Exception {
        SimpleTestcase.getInst().stopNode(firstVM);
    }

}

Flexible and powerful testing solution

We have developed TestChameleon in order to fulfill our clients’ need for automated testing. It’s a cost-efficient way to extensively test for incompatibility and other programmable issues, on a wide range of software environments. Whether it’s implemented on-premises or accessed through the cloud, TestChameleon is most powerful when it’s continuously integrated into the build process. This way, developers and dev teams can easily ensure consistent UX’s and achieve bug free software on all operating systems, browsers, and associated software. We have done quite a few successful implementations and are ready to open it up to a larger audience. And since we are eager to improve our TestChameleon API every single day, we consider this a great development.

About the author

Georg Hansbauer is founder and CEO of Munich-based Testbirds. He is responsible for finance and controlling, the strategic business development as well as the cooperation with major customers. Hansbauer gathered extensive experience in the field of enterprise testing – from automated tests for complete IT service desks to load testing – and has been in charge of various IT projects to international corporate groups. He graduated in the elite network master‘s program Finance and Information Management at the University of Augsburg and the TU Munich and carries a Bachelor’s degree in Business Informatics.

You might be interested in... The Dojo

Dojo Adverts-05

Tags:

3 Responses to “Automated Testing in the Cloud with TestChameleon”

  1. Joel BarhMarch 2, 2015 at 11:59 am #

    Thanks a lot Georg, that sounds very interesting.

    How do you folks compare to let’s say a Saucelabs or BugBuster for functional web testing?

    Cheers,

    Joe

    • GeorgMarch 3, 2015 at 5:52 pm #

      Hi Joe,

      Great question! We think BugBuster and Saucelabs are both great tools! We share a lot of cool features like an API for Selenium or integration with tools like Jenkins etc.

      For web testing we also add the possibility to not only test against different browsers but also against further software/plugins like Flash, Java or Silverlight.

      The main difference though is that with TestChameleon you get full access and administration rights to the VMs. That gives you a huge variety of test infrastructure scenarios since we have added a second layer of infrastructure (operating systems) to the browser layer. This means that with TestChameleon you can not only test web applications on a browser level but also every packable piece of software (like rich client software or .exe files) against millions of combinations of OS, browser and other software.

      Hope that answers your question.

      Cheers,

      Georg

  2. GeorgMarch 2, 2015 at 5:06 pm #

    Hi Joe,

    Great question! We think BugBuster and Saucelabs are both great tools! We share a lot of cool features like an API for Selenium or integration with tools like Jenkins etc.

    For web testing we also add the possibility to not only test against browsers but also against further plugins / software like Flash, Java or Silverlight.

    The main difference though is that with TestChameleon you get full access and administration rights to the VMs. That gives you a huge variety of test infrastructure scenarios since we have added a second layer of infrastructure (operating systems) to the browser layer. This means that with TestChameleon you can not only test web applications on a browser level but also every packable piece of software (like rich client software or .exe files) against millions of combinations of OS, browser and other software.

    Hope that answers your question 🙂