{"id":1295,"date":"2011-04-05T08:48:16","date_gmt":"2011-04-05T15:48:16","guid":{"rendered":"https:\/\/simpleprogrammer.com\/?p=1295"},"modified":"2018-04-13T16:28:33","modified_gmt":"2018-04-13T20:28:33","slug":"getting-up-to-bat-adding-smoke-tests-to-your-build","status":"publish","type":"post","link":"https:\/\/simpleprogrammer.com\/getting-up-to-bat-adding-smoke-tests-to-your-build\/","title":{"rendered":"Getting up to BAT: Adding Smoke Tests to Your Build"},"content":{"rendered":"<p>Once you\u2019ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible\u2026\\n\\nBut!\u00a0 You might want to consider holding off for a second and reading this post!\\n\\nIt is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.\\n\\nThere are several things we are going to want to discuss and think about before adding automated tests to a build.\\n\\n<a href=\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2.jpg\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0; padding-right: 0; display: inline; padding-top: 0; border: 0;\" title=\"build a site2\" src=\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg\" alt=\"build a site2\" width=\"497\" height=\"335\" border=\"0\" \/><\/a>\\n\\n<\/p>\n<h2>Why are we adding the smoke tests to the build?<\/h2>\n<p>\\n\\nI think it is always important that before we do something we ask ourselves why we are doing it.\u00a0 Doing things, because someone said it is good, or because it seems right is not a very good reason.\\n\\nThe primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.\\n\\nWe want to be able to get this value as soon as possible.\u00a0 We don\u2019t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.\\n\\nThe other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken.\u00a0 By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.\\n\\nThis concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.\\n\\nPerhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies.\u00a0 Better to start small with a small number of tests and get everything working.\\n\\nWith those ideas in mind, we can move onto some of the more important considerations.\\n\\n<\/p>\n<h2>Don\u2019t ever add failing tests to the build<\/h2>\n<p>\\n\\nWhen you add your smoke test suite to the automated build, all of the tests should pass!\\n\\nI will say it again, because it is really important and ignoring this advice may doom your entire BAT project.\\n\\n<\/p>\n<blockquote><p><strong>Do not ever add failing tests to the build!<\/strong><\/p><\/blockquote>\n<p>\\n\\nIt seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf.\u00a0 Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.\\n\\nFirst thing that will happen is that our tests will run during the next build and 3 will fail.\u00a0 This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of \u201cignore that fail, those are real defects, but they are known issues.\u201d\\n\\nNext, let\u2019s assume a developer checks in some bad code and it causes a 4th test to fail.\\n\\nDo you think most people will notice this 4th failing test?\\n\\nEven if they do, do you think they will just assume it is another known issue?\\n\\nWorse yet, they may think your BATs are meaningless and just fail randomly.\\n\\nDo you see where I am going with this?\u00a0 We want to start off with the correct expectations and we want test failures to be meaningful.\u00a0 To ALWAYS be meaningful.\u00a0 The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.\\n\\nThis is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails.\u00a0 When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.\\n\\nMake sure your tests never undermine this message.\\n\\n<\/p>\n<h2>What do I do with failing tests then?<\/h2>\n<p>\\n\\nIf they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.\\n\\nYour automated tests should be acting as regression tests for things that are already working.\u00a0 It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!\\n\\nIt is the process of writing the automated test that should uncover defects.\u00a0 Running of automated tests should uncover new defects where functionality has regressed from working to not working.\\n\\n<\/p>\n<h2>Other considerations<\/h2>\n<p>\\n\\nHere are a list of some other considerations you will want to think about before embarking on this journey.\\n\\n<\/p>\n<ul>\\n    <\/p>\n<li><strong>How will you actually run the tests?<\/strong> (Will you use the build script to kick it off, some 3rd party plugin, etc)<\/li>\n<p>\\n    <\/p>\n<li><strong>Where will you run the tests?<\/strong> (Will you use the build server itself, or another machine.\u00a0 You probably don\u2019t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)<\/li>\n<p>\\n    <\/p>\n<li><strong>How long will it take to run the smoke tests?<\/strong> (Fast is better, because feedback cycle is reduced.\u00a0 If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)<\/li>\n<p>\\n    <\/p>\n<li><strong>Will developers be able to run the tests locally?<\/strong> (Make sure you consider how to do this before putting the tests into the build.\u00a0 Nothing more frustrating than being measured against a bar you can\u2019t yourself see.)<\/li>\n<p>\\n    <\/p>\n<li><strong>How will you notify and report results from the test failures?<\/strong> (We don\u2019t care about passes, only failures, but you need a good way to report the result that prompts action.)<\/li>\n<p>\\n    <\/p>\n<li><strong>What is the policy for test failures? <\/strong> (Make sure there is some agreement on what kind of action will be taken when tests fail.\u00a0 A great place to start, if you can get management to buy in on it, is all work stops until all tests pass.\u00a0 Extreme, but extremely effective.)<\/li>\n<p>\\n    <\/p>\n<li><strong>How will you prevent false fails and identify them? <\/strong>(Don\u2019t get stuck with egg on your face.\u00a0 If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem.\u00a0 There are a few strategies for doing this, but that is for another post.)<\/li>\n<p>\\n    <\/p>\n<li><strong>How do new tests get added and commissioned?<\/strong> (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)<\/li>\n<p>\\n<\/ul>\n<p>\\n\\nRemember, you only get one chance at a first impression!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Once you\u2019ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible\u2026\\n\\nBut!\u00a0 You might want to consider holding off for a second and reading this post!\\n\\nIt is worth taking a bit of time and thinking a bit about&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[44070,50917,2185,32,12],"tags":[],"class_list":["post-1295","post","type-post","status-publish","format-standard","hentry","category-automation","category-bat","category-best-practices","category-infrastructure","category-testing"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/simpleprogrammer.com\/products\/careerguide\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer\" \/>\n<meta property=\"og:description\" content=\"Once you\u2019ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible\u2026nnBut!\u00a0 You might want to consider holding off for a second and reading this post!nnIt is worth taking a bit of time and thinking a bit about...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/simpleprogrammer.com\/products\/careerguide\" \/>\n<meta property=\"og:site_name\" content=\"Simple Programmer\" \/>\n<meta property=\"article:published_time\" content=\"2011-04-05T15:48:16+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-04-13T20:28:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg\" \/>\n<meta name=\"author\" content=\"John Sonmez\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"John Sonmez\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/getting-up-to-bat-adding-smoke-tests-to-your-build\\\/\"},\"author\":{\"name\":\"John Sonmez\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/#\\\/schema\\\/person\\\/1abc70edfb189184827740310672de67\"},\"headline\":\"Getting up to BAT: Adding Smoke Tests to Your Build\",\"datePublished\":\"2011-04-05T15:48:16+00:00\",\"dateModified\":\"2018-04-13T20:28:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/getting-up-to-bat-adding-smoke-tests-to-your-build\\\/\"},\"wordCount\":1249,\"commentCount\":3,\"image\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/simpleprogrammer.com\\\/wp-content\\\/uploads\\\/2011\\\/04\\\/build-a-site2_thumb.jpg\",\"articleSection\":[\"Automation\",\"BAT\",\"Best Practices\",\"Infrastructure\",\"Testing\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/getting-up-to-bat-adding-smoke-tests-to-your-build\\\/\",\"url\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide\",\"name\":\"Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/simpleprogrammer.com\\\/wp-content\\\/uploads\\\/2011\\\/04\\\/build-a-site2_thumb.jpg\",\"datePublished\":\"2011-04-05T15:48:16+00:00\",\"dateModified\":\"2018-04-13T20:28:33+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/#\\\/schema\\\/person\\\/1abc70edfb189184827740310672de67\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#primaryimage\",\"url\":\"https:\\\/\\\/simpleprogrammer.com\\\/wp-content\\\/uploads\\\/2011\\\/04\\\/build-a-site2_thumb.jpg\",\"contentUrl\":\"https:\\\/\\\/simpleprogrammer.com\\\/wp-content\\\/uploads\\\/2011\\\/04\\\/build-a-site2_thumb.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/products\\\/careerguide#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/simpleprogrammer.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Getting up to BAT: Adding Smoke Tests to Your Build\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/#website\",\"url\":\"https:\\\/\\\/simpleprogrammer.com\\\/\",\"name\":\"Simple Programmer\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/simpleprogrammer.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/simpleprogrammer.com\\\/#\\\/schema\\\/person\\\/1abc70edfb189184827740310672de67\",\"name\":\"John Sonmez\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g\",\"caption\":\"John Sonmez\"},\"url\":\"https:\\\/\\\/simpleprogrammer.com\\\/author\\\/jsonmez\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/simpleprogrammer.com\/products\/careerguide","og_locale":"en_US","og_type":"article","og_title":"Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer","og_description":"Once you\u2019ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible\u2026nnBut!\u00a0 You might want to consider holding off for a second and reading this post!nnIt is worth taking a bit of time and thinking a bit about...","og_url":"https:\/\/simpleprogrammer.com\/products\/careerguide","og_site_name":"Simple Programmer","article_published_time":"2011-04-05T15:48:16+00:00","article_modified_time":"2018-04-13T20:28:33+00:00","og_image":[{"url":"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg","type":"","width":"","height":""}],"author":"John Sonmez","twitter_card":"summary_large_image","twitter_misc":{"Written by":"John Sonmez","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#article","isPartOf":{"@id":"https:\/\/simpleprogrammer.com\/getting-up-to-bat-adding-smoke-tests-to-your-build\/"},"author":{"name":"John Sonmez","@id":"https:\/\/simpleprogrammer.com\/#\/schema\/person\/1abc70edfb189184827740310672de67"},"headline":"Getting up to BAT: Adding Smoke Tests to Your Build","datePublished":"2011-04-05T15:48:16+00:00","dateModified":"2018-04-13T20:28:33+00:00","mainEntityOfPage":{"@id":"https:\/\/simpleprogrammer.com\/getting-up-to-bat-adding-smoke-tests-to-your-build\/"},"wordCount":1249,"commentCount":3,"image":{"@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#primaryimage"},"thumbnailUrl":"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg","articleSection":["Automation","BAT","Best Practices","Infrastructure","Testing"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/simpleprogrammer.com\/products\/careerguide#respond"]}]},{"@type":"WebPage","@id":"https:\/\/simpleprogrammer.com\/getting-up-to-bat-adding-smoke-tests-to-your-build\/","url":"https:\/\/simpleprogrammer.com\/products\/careerguide","name":"Getting up to BAT: Adding Smoke Tests to Your Build - Simple Programmer","isPartOf":{"@id":"https:\/\/simpleprogrammer.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#primaryimage"},"image":{"@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#primaryimage"},"thumbnailUrl":"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg","datePublished":"2011-04-05T15:48:16+00:00","dateModified":"2018-04-13T20:28:33+00:00","author":{"@id":"https:\/\/simpleprogrammer.com\/#\/schema\/person\/1abc70edfb189184827740310672de67"},"breadcrumb":{"@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/simpleprogrammer.com\/products\/careerguide"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#primaryimage","url":"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg","contentUrl":"https:\/\/simpleprogrammer.com\/wp-content\/uploads\/2011\/04\/build-a-site2_thumb.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/simpleprogrammer.com\/products\/careerguide#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/simpleprogrammer.com\/"},{"@type":"ListItem","position":2,"name":"Getting up to BAT: Adding Smoke Tests to Your Build"}]},{"@type":"WebSite","@id":"https:\/\/simpleprogrammer.com\/#website","url":"https:\/\/simpleprogrammer.com\/","name":"Simple Programmer","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/simpleprogrammer.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/simpleprogrammer.com\/#\/schema\/person\/1abc70edfb189184827740310672de67","name":"John Sonmez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dc2f18dfa76e017566ce607de002d7abe4acfd5ec19a6db82c180b00867b8d94?s=96&d=mm&r=g","caption":"John Sonmez"},"url":"https:\/\/simpleprogrammer.com\/author\/jsonmez\/"}]}},"_links":{"self":[{"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/posts\/1295","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/comments?post=1295"}],"version-history":[{"count":0,"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/posts\/1295\/revisions"}],"wp:attachment":[{"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/media?parent=1295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/categories?post=1295"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/simpleprogrammer.com\/wp-json\/wp\/v2\/tags?post=1295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}