Running DevTest from CA

Link = https://support.ca.com/cadocs/0/CA%20DevTest%20Solutions%208%200-ENU/Bookshelf.html

That’s cool, thanks for explaining that Curt. So is this automated testing that’s being done? You said that the “tool runs”.

Yes, you have the basic concept of what the requirements were…. and yes, I could have added one more to each of the test cases to test 1 more higher than the cutoff point.

The actual testing limits that we have are a lot different and higher than my example. I just tried to show in easy terms what the higher bounds looked like. There is no limit, but the chances of going a lot higher than the higher limit is very minimal.

If everything would have passed the first run, I would have done another round of testing to test inside the boundaries at different intervals. Because the tool (DevTest 8, formally Lisa) uses an Excel or .csv files, its simple to create another round of testing with expected limits, then past the calculated values into the input spreadsheet (or save as, but I find it simpler to past the values) and let the tool run on its own in the background. This project testing took a couple of hours to run through, so I work on another project while I am waiting for results (I sometimes have 4 active projects at any one time). It keeps my whole department busy.

Ah, nice, I think I get it. Let me reiterate to make sure that we’re on the same page haha.

Input Expected
20000 50
22000 51
24000 52

Did I get that right? If so, would you want to try 19999, 20000, and 20001 for the first boundary of 20000?

That’s great that it caught the bugs though. Awesome job Curt!

To better explain the upper bounds (which I did a poor job of explaining), the requirements gave me a table for limits for each symbol. There was an upper limit to the value/symbol pair. After that upper limit, there is a formula for determining the symbol

Lets say the upper limit of the table is 20,000 with a symbol of 50. For every 2,000 above 20,000, add 1 to the largest symbol.

1st test case – sent a value of 20,001, looking for a return of 50
2nd test case – sent a value of 21,999, looking for a return of 50
3rd test case – sent a value of 22,000, looking for a return of 51
4th test case – sent a value of 23,999, looking for a return of 51
5th test case – sent a value of 24,000, looking for a return of 52

all test cases above returned a symbol of 0, thus the three edges were failures, which produced the bug reports.

Hey Curt, thanks for all the positive feedback. I am glad that you found this helpful and good. I also really appreciate you explaining a real life scenario that you faced. However, I was a bit unclear on the bugs that you found. Did you increment the upper most boundary by 1 and that caused the bug? So if the uppermost boundary was 10, you tested with 11,12,13? And those 3 found bugs? Sorry, just trying to understand 🙂

I also love how you stated, “every programmer should have a basic knowledge of testing to do unit testing on their code”. It’s so true. I still struggle to this day to understand why developers wont test their code. It doesn’t make sense to me. I see nothing but benefits from that approach.

I’m sure that this will change with time. But for now, we need to keep fighting the battle of convincing the developers to test. And sometimes, that yields absolutely no results haha

Boundary testing is very valuable and needed tool in ones testing toolbox. Not only can you catch simple coding mistakes, but you can also catch requirement mistakes, like your first example in the article. If I was testing, I would file 2 bugs, one for never seeing 5 monkeys, and the second would be never seeing 9 bananas. (actually I would catch that before even writing test cases, but I have been testing for almost 10 years). This is a case of bad requirements.

Here is a real world example for the readers, and a test I just completed a couple of weeks back. We are coding a new service (Black box for me) that takes a value and returns a symbol. I was given the range of value, symbol pairs. At the top end of the values, for each value of so much, we add one to the symbol and return it. I started out just testing the boundary for each symbol. the end of the test, I testing 3 steps past the largest value in the chart.

Thankfully, we have automation tools that help with this testing. Initial results were 3 boundaries failed, and the three past the largest value failed. I then manually tested the three boundaries to verify the failures (never file a bug without recreating the bug reliably). I then focused on the range of failures inside the boundaries and found a range of values sending an error message, even though they were valid values. This turned out to be a database bug that is behind the service. The three failures past the largest value is a coding failure(no code to check the value and then add to the symbol). All this testing took less than one day to complete, thanks to the automation tools that use spreadsheets for input. All I had to do to was to change the values of the input in the spreadsheet.

I do like your article though. Every programmer should have a basic knowledge of testing to do unit testing on their code. At my company, we require programmers to unit test their code before submitting, but with some of the test results I get, either the programmer doesn’t know basic testing, or they just didn’t test their code.

Have a good day