![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/ultimate-square.png\")
Automation testing takes a manual test case and applies a tool or programming language to automate its execution. As more teams shift left, there is a need for tests to run earlier and faster in the development lifecycle.<\/p>\nAutomation testing takes a manual test case and applies a tool or programming language to automate its execution. As more teams shift left, there is a need for tests to run earlier and faster in the development lifecycle.<\/p>\n
Traditionally, during regression testing, a manual tester will take an existing test case procedure and execute it step by step. This can be time-consuming and also error-prone, since it is done by hand.<\/p>\n
Because of the reasons above, to save time, many companies try to take their manual test cases and convert them to an automated test case. An automated test tool then executes the test steps automatically without human intervention.<\/p>\n
Sounds easy, but there are many pitfalls teams encounter when starting their test automation journey. In this tutorial I will address the most common automation issues folks face and hopefully help you begin your automation project off right.<\/p>\n
This is a long guide, so here’s an overview of what we’re going to cover:<\/p>\n
Let\u2019s start at the beginning by defining automation testing.<\/p>\n
Functional automation testing uses tools designed specifically for automation to emulate a user interacting with an application and verifying test steps using programming assertions.<\/p>\n
Many folks also call these tests automation \u201cchecks\u201d or automated checking<\/a>.<\/p>\n This distinction is made to remind testers that automation and the implementation of automation is a checker\u2014it doesn\u2019t replace your testing strategy. An automated test is also, in a sense, dumb in that it can test only what you tell it to test; if you don\u2019t assert it, it doesn\u2019t get checked.<\/p>\n Also, it\u2019s important to remember that \u201cautomation\u201d does not apply to just user interface (UI) end-to-end tests. In fact, I would say that you get more benefit from a lower-level automated test, like a unit test, than you do from a big bloated test suite of end-to-end tests.<\/p>\n Before we take a more in-depth look at automated testing, let\u2019s touch on some problems with manual testing.<\/p>\n There are many reasons why having a testing strategy that relies heavily on just manual testing causes issues. Here are a few:<\/p>\n Automation testing can help.<\/p>\n But one of the first hurdles you might face when introducing automated testing to your organization is the false belief that automation can replace all your testers and tests with automation.<\/p>\n Here\u2019s the deal:<\/p>\n Automation Testing Does Not Replace Testers.<\/strong><\/p>\n Some people assume because a test activity is automated, that means it replaces human testers. In fact, however, it\u2019s the opposite. Automated tests are great for running tests precisely and quickly, but they in no way replace human testers.<\/p>\n Automation tests are also great for running the same steps over and over again, but they don\u2019t think.<\/p>\n I like to think of automation in the way that Peter Thiel explains it in the \u201cMan and Machine\u201d chapter of his book Zero to One<\/a><\/em>:<\/p>\n \u201cComputers are complements to humans, not substitutes. The most valuable businesses of the coming decades will be built by entrepreneurs who seek to empower people rather than try to make them obsolete.\u201d<\/p><\/blockquote>\n Although we can agree that automation testing does not replace other testing activities, with today\u2019s software development environment and continuous integration practices, it is critical and cannot be ignored.<\/p>\n So what are some reasons for using automated tests?<\/p>\n Practices like continuous integration and delivery require tests that run quickly and reliably. Lots of manual tests will stop your ability to achieve velocity with your software development.<\/p>\n I\u2019d go so far as to say that in today\u2019s modern development environment, we couldn\u2019t succeed without automation.<\/p>\n Although the main reason teams try to create automated tests is to save the company both time and money, it\u2019s also important to give developers quick feedback so that when they check in code, they are alerted as soon as possible that the change they checked in broke something.<\/p>\n Some other reasons for automated testing are:<\/p>\n While these are good reasons for automation, many folks fail to factor in the amount of time and money it takes to maintain sizeable automated test suites.<\/p>\n So are there any other downsides to creating automated tests? What\u2019s the real story?<\/p>\n Since automated tests usually rely on programming languages for their creation, automation becomes a full-blown development effort. What you are doing is developing a piece of software to test another piece of software.<\/p>\n Automation testing is difficult and complicated, just like most other development software projects. It also presents many of the same issues other software programs do. Treating your automated code just like your development code is essential. Follow the same processes and best practices you would use for any other software development project.<\/p>\n To learn more, John Sonmez covers many of these best practices in his awesome Pluralsight course Creating an Automated Testing Framework With Selenium<\/a>.<\/p>\n Teams often claim that automation testing \u201cdoesn\u2019t work.\u201d But this attitude is usually caused by poorly designed test automation more than anything else.<\/p>\n If you keep the issues listed below in mind as you create your test automation framework, you can avoid many of these automation pitfalls ahead of time.<\/p>\n Many issues are caused by setting unrealistic goals, like, for example, having a goal of reaching 100% UI automated testing coverage. Teams often believe that automation tests will find more new defects, so they have a false sense of security. Your automation is only as good as your tests.<\/p>\n Teams also underestimate the amount of time it takes to maintain automation. They\u2019ll often create large, end-to-end tests, but tests should be atomic so that when they fail, you know why.<\/p>\n Several other common issues that teams face are:<\/p>\n For automation to be successful in an organization, you need to educate everyone on the team as to what the expectation should be for your testing. It\u2019s also critical that you create a whole team approach to your automation efforts, meaning regardless of a person\u2019s role on the team, development and testing (and ultimately, delivering a feature) requires a total team effort. Testing shouldn\u2019t be an activity that is done only at the end of a sprint by a designated tester.<\/p>\n Quality needs to be baked into the software from the beginning, not after the fact. The only way to do this is to have everyone working toward making the application under development as testable as possible. This takes the whole team working together to deliver a quality product.<\/p>\n Many failures I see with automation are not caused by technical issues, but rather by a company\u2019s cultural issues.<\/p>\n Remember\u2014automation is a time-consuming test activity. You want to use it only when it makes sense. Other testing activities, like exploratory-type work, should be encouraged.<\/p>\n Automation is just one of many types of test activities that can be used by testers.<\/p>\n At this point, another common question I\u2019m frequently asked is, \u201cWhich tests should be automated?\u201d<\/p>\n The biggest problem I usually see is that teams start off trying to automate everything. The problem is that not everything is automatable. When planning which test cases to automate, you should look for tests that are deterministic, don\u2019t need human interaction, are hard to test manually, and need to run more than once.<\/p>\n You should also seek to automate any manual process that will save engineers time (not necessarily an official \u201ctesting\u201d process), along with tests that focus on the money or risk areas of your application.<\/p>\n Other tests that are useful to automate are unit tests, as well as tests that run against different data sets, focus on critical paths of your application, need to run against multiple builds and browsers, and are used for load\/stress testing.<\/p>\n The more repetitive the execution is, the better candidate a test is for automation testing. However, every situation is different.<\/p>\n Ultimately, you should consider using automation for any activity that saves your team time. It doesn\u2019t have to be a pure testing activity; you can leverage automation to help reduce any time-consuming activity found anywhere in the software development lifecycle.<\/p>\n At this point, some of you may be asking, \u201cWhat is the return on investment (ROI) of test automation?\u201d<\/p>\n Determining the ROI of your automation testing efforts can be tricky. Here is a common calculation some folks use to get a rough estimate of their test automation costs. This can also help you decide whether a test case is even worth automating as opposed to testing it manually.<\/p>\n Automation Cost = how much the tools cost + how much the labor costs to create an automated test + how much it costs to maintain the automated tests<\/strong><\/p>\n Consequently, if your automation cost calculation is lower than the manual execution cost of the test, it\u2019s an indicator that automation is a good choice.<\/p>\n Moreover, ROI quickly adds up with each re-run of your automated test suite.<\/p>\n Because it\u2019s critical that you get a good return on your test automation investment, there are some things you shouldn\u2019t automate.<\/p>\n There are exceptions to everything, of course, but in general, you may not want to automate the following test case scenarios:<\/p>\n In addition to what not to automate, another element of a successful automation project is having an automation framework.<\/p>\n I like to break down an automation testing framework into specific areas of concern, or what I call the \u201cfour Ps\u201d of an automation framework: people, planning, process, and performance.<\/p>\n The first one is the people aspect of a test automation framework.<\/p>\n People<\/strong><\/p>\n As we already covered earlier, you want to make sure that you have set the expectations of your managers and team about your automation strategy. To help ensure that your automation is collaborative and a whole team effort, I recommend that you include automation on your sprint team\u2019s definition of done.<\/p>\n The next stage is the planning piece of your automation framework.<\/p>\n Planning<\/strong><\/p>\n Before writing one line of code, always check to see if there is an existing library or tool you can use before inventing your own. Break your automation framework into abstraction layers so that if anything changes, you just need to make the change in one place. Using established automation testing design patterns like the ones we cover later in this post should be part of the planning stage.<\/p>\n Separating your tests from your framework will also help when you have to make changes to your framework.<\/p>\n Also, plan on making your methods and utilities reusable to avoid code duplication. Making your test and code as readable as possible (they should read like English) will go a long way to prevent confusion and code duplication.<\/p>\n Finally, when planning your test, it’s important to be aware of what test data your tests need. Many times, tests run against different environments that might not have the data you expect, so make sure to have a test data management strategy in place. Including support for mocking and stubbing in your framework can also help with some test data issues.<\/p>\n Process<\/strong><\/p>\n Having a process in place that holds team members accountable for automation is another vital piece of a framework. Since automation is just like any other development project, make sure to use the same process and best practices that developers already follow, like using version control and performing code reviews on all automated tests.<\/p>\n Performance<\/strong><\/p>\n Always start your automation framework with the end in mind. Tests need to be reliable and maintainable. They also should run as fast as possible.<\/p>\n Along these lines, make sure that your developers are creating unique IDs for each element that you will have to interact with within your tests. Doing this will help you avoid resorting to lousy automation practices like relying on a coordinate-based way to communicate with an element.<\/p>\n Another top killer of test automation script performance is the failure to use proper synchronization\/wait points in your tests. Too many hard-coded waits will slow down your test suite. Use the preferred explicit wait<\/a> method for synchronization.<\/p>\n As you write your test scripts, think about how they would perform if you had to run in parallel. Thinking about possible parallel issues beforehand will avoid problems when you start to scale the running of your test suite again on a grid or a cloud-based service like Sauce Labs<\/a>.<\/p>\n To ensure that your tests are as preformatted as possible, refactor slow or poorly written code whenever possible. Including reporting and logging in your framework will help you quickly identify poorly running code.<\/p>\n To make sure that teams follow all these guidelines, determine a strategy for training and retraining your framework users.<\/p>\n That\u2019s not all \u2026<\/p>\n It\u2019s a given that your applications are going to change over time. And since you know change is going to happen, you should start off right from the beginning using best practices or design patterns. Doing so will make your automation more repeatable and maintainable.<\/p>\n Here are some common automation testing design patterns that many teams use to help them create more reliable test automation.<\/p>\n One popular strategy to use when creating your test automation is to model the behavior on your application. Creating simple page objects<\/a> that model the pieces of your software that you are testing against can do this.<\/p>\n So, for example, you would write a page object for login or a page object for a homepage. Following this approach correctly makes use of the single responsibility principle.<\/p>\n If anything changes\u2014say, an element ID\u2014you just need to go to one place to make the change and all of the tests that use the page object will automatically pick up the changes without you doing anything else. The test code needs to be updated in only one place.<\/p>\n Page objects also hide the technical details about HTML fields and CSS classes behind methods that have easy-to-understand names. Being mindful when naming your methods has the extra benefit of helping to create a nice readable test application programming interface (API) that a less technical automation engineer can quickly start using for automation.<\/p>\n Page objects are a good place to start making your test maintainable, but if you\u2019re not careful, they can grow out of control over time. The Screenplay pattern<\/a> takes page objects and chops them down into really tiny pieces. Some testers tell me that this has made their tests much more maintainable and reliable.<\/p>\n Another significant benefit is that it makes test scripts more readable.<\/p>\n The ports and adapters<\/a> design strives to make sure you are using the single responsibility principle so that an object should do only one thing and have one reason to change.<\/p>\n When you apply this to test automation, make sure to decouple your test code to allow you to swap slow components with fast simulators so that you can run your test and the app you\u2019re testing in the same process.<\/p>\n Remove all networking and input\/output so nothing is slowing down the test suite. Of course, this is not easy to do, but the more you try to do this when creating UI automation, the better off you will be.<\/p>\n Presenter First<\/a> is a modification of the model-view-controller<\/a> (MVC) way of organizing code and development behaviors to create completely tested software using a test-driven development (TDD) approach.<\/p>\n I first heard of this pattern when interviewing Seb Rose<\/a>, one of the contributors to the Cucumber project and author of The Cucumber for Java Book<\/a>.<\/em><\/p>\n He mentioned that if you draw out the MVC pattern as blocks and arrows, you can see that the view, which is your UI, has well-defined channels of communication with the model and the controller. If you can replace those during runtime with models and controllers that your test creates and controls, then there is no reason why you can\u2019t just test that the UI behaves in the way that you want.<\/p>\n You can also set your model and controller to mimic all sorts of odd behaviors, like a network going down.<\/p>\n Here are some high-level good automation practices you should follow in more detail:<\/p>\n An atomic test is a strategy for ensuring that each test is entirely self-contained. That is, it should not depend on the outcome of other tests to establish its state, and another test should not affect its success or failure in any way.<\/p>\n Also, when an automated test fails, you need to know why. Having a well-named atomic test that tests only one thing will help you quickly identify what broke if your test fails.<\/p>\n Furthermore, you should endeavor to get feedback to your developers as quickly as possible, and the best way to do that is with a fast, well-named test.<\/p>\n This is also critical if you plan on running your automation test in parallel in a Selenium Grid<\/a>.<\/p>\n Test size matters because tests need to run quickly.<\/p>\n At this point, many people visualize a traditional test pyramid<\/a>, which has unit tests as its base, integration tests in the middle, and graphical user interface (GUI) tests at the top.<\/p>\n But I think more in terms of test size. By test size, I\u2019m referring to tests that are faster than others.<\/p>\n While I understand the need to run UI tests if you have to create one, make it as fast as possible.<\/p>\n A quick point on test code readability\u2014did you know that developers spend more time reading code than actually writing it?<\/p>\n It is rare that the person who wrote code will also be the one that needs to modify it. Even worse, how many times have you written code only to come back to it months later and have no idea what it is doing?<\/p>\n Since, as we mentioned, automation code is software development, you should create your test code with the reader of the code in mind\u2014not the computer.<\/p>\n This will help not only to make your test more maintainable, but also will help ensure that you do not duplicate code because you didn\u2019t know what an existing piece of code was doing.<\/p>\n This might seem like a minor issue, but ignore readability of your automation test at your peril.<\/p>\n The importance of code readability really hit home for me after watching Cory House\u2019s Pluralsight course called\u00a0 Clean Code: Writing Code for Humans<\/a>.<\/p>\n Testability needs to be baked into our applications right from the start. As a regular part of sprint planning, developers should be thinking about how they can make their application code more testable. They can do this by providing things like unique element IDs for their application fields and APIs to help create hooks into their application(s) that can be used in their automated tests.<\/p>\n They should also be thinking about how any code changes they make to the application are going to impact existing automated tests, and plan accordingly.<\/p>\n If you don\u2019t do this, you\u2019re not going to be successful with automation for very long.<\/p>\n Remember, you can\u2019t automate that which is not testable.<\/p>\n This one is a really common hindrance for many teams with their automation efforts.<\/p>\n Without a stable test environment that is always in a known state, it will be tough for your teams to make progress with their automation efforts.<\/p>\n Tests failing due to environmental issues rather than actual application issues will cause your teams to lose confidence in your test feedback quickly.<\/p>\n Once teams start ignoring automation results, your test efforts become useless.<\/p>\n Some tools that can help create more stable and repeatable environments are Docker<\/a>, Vagrant<\/a>, and Skytap<\/a>.<\/p>\n There is no \u201ccorrect\u201d test tool for automation testing. Ultimately, it all depends on your team\u2019s unique needs and skill set.<\/p>\n I always recommend that you run a two-week proof of concept (POC) for each tool that you are considering and include your team\u2019s feedback in the process before committing to a tool.<\/p>\n The first place to start is to look at the product roadmap and make sure the tools you select will handle future features and technologies. To avoid future compatibility issues and tedious framework refactoring due to false notions, don\u2019t skip this step.<\/p>\n Next, you should evaluate the cost, including maintenance. If you plan on having your whole team help out with the automation effort, make sure to use a tool that leverages the same tools and languages your developers use.<\/p>\n Don\u2019t just assume a tool will work for you. Create a small POC for each tool and get team feedback before committing to anything. Ask the team: Is the tool extensible? How easy is it to use and get started? Does it provide reporting and debugging capabilities? Does it recognize all the objects in your application? Can it integrate with other tools like version control, test management tools, and continuous integration tools?<\/p>\n Most importantly, find out if the tool has an active user base and select tools that other companies are using. You don\u2019t want to select an open-source solution that is not actively maintained by the community. You may need to ask how much training it will take to get your teams up to speed with the tool. Finally, determine how easy is it to hire folks that have the skills needed to create your automated tests.<\/p>\n Coming up with metrics that teams can use to make sure they are on track is tough. These should be used just as a quick way to monitor your team\u2019s progress and shouldn\u2019t be used as hard and fast rules.<\/p>\n Mean Time to Diagnosis (MTD)<\/strong>\u2014How long does it take you to debug a failing automated test? A high MTD is an indicator that your automated test code is not high-quality.<\/p>\n Bugs Found by Automation<\/strong>\u2014This can be helpful to determine how much value your automation efforts are bringing.<\/p>\n Flaky Rate<\/strong>\u2014Ideally, this should be zero, but this is a good indicator to know if your automated tests are reliable or not.<\/p>\n Automated to Manual Ratio<\/strong>\u2014The more manual tests you have, the longer it will take to tell if your application is ready for release. This helps you keep a pulse on how long your release efforts will take. It also enables you to gain visibility into whether teams are automating (or not automating) the right things.<\/p>\n Here are some vendors that allow you to save a ton of time by running your test in the cloud and on multiple OS, devices, and configurations. Get rid of the headache of having to maintain your own in-house lab\/grid.<\/p>\n Finally, here\u2019s a list of my favorite automation testing courses:<\/p>\n Automation testing takes a manual test case and applies a tool or programming language to automate its execution. As more teams shift left, there is a need for tests to run earlier and faster in the development lifecycle. Traditionally, during regression testing, a manual tester will take an existing test case procedure and execute it…<\/p>\n","protected":false},"author":1113,"featured_media":27795,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[44070,2185,415,162228543,162229840,12],"tags":[],"class_list":["post-27687","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-automation","category-best-practices","category-guest-post","category-pluralsight-courses","category-skills","category-testing"],"yoast_head":"\nIssues With Manual Testing<\/h2>\n
\n
Why Is Automation Testing Necessary?<\/h2>\n
![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/Why-Is-Automation-Testing-Necessary.png\")
With the increased speed in which we develop software, we need automation testing. As more companies move toward Agile<\/a> and DevOps<\/a>, automation is more important than ever before.<\/p>\n\n
Automation Testing Considerations<\/h2>\n
Automation Testing Pitfalls<\/h2>\n
\n
Automation Is a Team Effort<\/h2>\n
![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/Once-you-have-the-whole-team-on-board-with-automation.png\")
Once you have the whole team on board with automation, and your manager\u2019s expectations have been correctly set, it\u2019s time to write your automated tests.<\/p>\nTests That Should Be Automated<\/h2>\n
ROI: The Cost of Test Automation<\/h2>\n
What Shouldn\u2019t Be Automated?<\/h2>\n
\n
What Is an Automation Testing Framework?<\/h2>\n
![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/What-Is-an-Automation-Testing-Framework.png\")
An automation framework is a common set of tools, guidelines, and principles for your tests. Having a framework helps to minimize test script maintenance.<\/p>\nAutomation Testing Design Patterns<\/h2>\n
Page Objects<\/h3>\n
Screenplay Pattern<\/h3>\n
Ports and Adapters<\/h3>\n
Presenter First<\/h3>\n
Test Automation Process<\/h2>\n
![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/Test-Automation-Process.png\")
I like to follow a six-step cyclical process when planning my test automation efforts that have the following phases: Analyze, Write, Execute, Evaluate, Communicate, and Repeat\/Refactor.<\/p>\n\n
Test Automation Best Practices<\/h2>\n
Atomic Test<\/h3>\n
Test Sizing<\/h3>\n
Readability<\/h3>\n
Testability<\/h3>\n
Stable Environment<\/h3>\n
How To Pick an Automation Testing Tool<\/h2>\n
Test Automation Metrics<\/h2>\n
Automation Testing Frameworks and Tools<\/h2>\n
![\"\"](\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2018\/02\/Automation-Testing-Frameworks-and-Tools.png\")
This is not an exhaustive list, but rather a quick summary of some of the more popular test tools that I\u2019m aware of.<\/p>\nOpen-Source Tools<\/h3>\n
\n
Vendor-based Automation Test tools<\/h3>\n
\n
API Automation Test Tools<\/h2>\n
Open-source API tools<\/h3>\n
\n
Vendor API Tools<\/h3>\n
\n
Test Execution Report Tools<\/h3>\n
\n
Run Your Automated Test in the Cloud or on Mobile Devices<\/h2>\n
\n
Automation Test Management Tools<\/h2>\n
\n
Automation Testing Courses<\/h2>\n
\n
Test Automation Conferences<\/h2>\n
\n