Let’s face it you can never have enough money and your tests never run fast enough. I can’t help you with your money, but if you follow the steps in this post I can help you speed up your Cucumber tests. The improvement process that I’ll walk you through is known commonly as DMAIC (Define, Measure, Analyze, Improve, Control). For those not familiar with DMAIC, Google can tell you everything you need to know. But, basically DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimise business processes. We’ll use DMAIC in this post to improve the speed of your Cucumber tests.
DefineYou’ve probably already defined the problem, your tests are too slow. What you haven’t done is define success. You will define success in terms of execution time. For instance, if your Cukes are taking 30 mins you might define success as, “Make all Cukes run in 20 mins or less”, others might define success as, “Cukes running in 5 mins or less”. This is up to you and your team to decide. Just remember, the amount of time it takes to run your tests will be inversely proportional to number of times you will run them. So the faster they are the more often you will run them.
MeasureYou know how long it takes to run the suite. You might even know how long it takes to run each scenario or sets of scenarios. What you need to know now is how long it takes each step to run. Wait! Before you go diving back into your steps to instrument them with timers and such, let’s see what Cucumber can do to help us. Fortunately for us Cucumber offers a number of helpful options for formatting the output of our test runs. One of those options is --format usage. Using the ‘usage’ format will show us which steps take the longest to run, how many times each step executed and if any steps are unused. I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.
AnalyzeWith our new data in hand we can begin to zero in on the trouble spots. The usage report displays the steps ordered from slowest to fastest. The report also shows the number of times each step executed. Your slowest step might not be the biggest problem. You should focus on the steps that run the slowest and the most often. Also, consider why the step executed so often. If you wrote the scenarios following a behavior driven development (BDD) cycle then the scenario was written to define behavior in the system. As a result you may find many scenarios exercising the same part of the system in only slightly different ways. Viewing the scenarios in hindsight you should consider if you can consolidate or eliminate some of the scenarios.
ImproveNow that we know where to focus our efforts it’s time to make our improvements.
ConsolidateIn the ‘Analyze’ phase you considered if you could remove or consolidate scenarios. Now is the time to do that work. Deleting scenarios is fun and easy. Consolidating scenarios takes more effort and thought. Look for recurring patterns such as screen navigation and data setup. If you notice that you repeatedly setup the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation. From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing while maintaining the ability to independently execute the scenario.
Learn to waitOne of the most common causes of slow downs in cukes is ‘sleep’ing. People often use sleep to wait for AJAX activity to complete. This is the most unpredictable and wasteful way to control test execution. Imagine that an AJAX action normally takes between .5 and 2 seconds to complete. If we sleep to wait for the action to complete how long should we sleep? Let’s say we decide to sleep for 3 seconds. If that action ever takes more than 3 seconds the test fails. Also, every time the action takes less than 3 seconds the difference is wasted doing nothing and if this is one of your most executed steps those seconds quickly turn into minutes. Instead of sleeping you must learn to ‘wait’. The Selenium client includes a number of ‘wait_for’ methods that will improve the predictability of your steps and will remove the waste caused by sleeping.
wait_for_text(value)-- Simply waits for the text you specify to appear in the DOM. Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.
wait_for_element(xpathLocator)-- Allows you to specify an exact xpath locator to wait for before continuing.
wait_for_no_text(value)-- Allows you to wait for the removal of an xpath locator or text value from the DOM before continuing.