Test Reporting and what charts in the Trends display would help you most?

Instead of guessing or choosing what works for me, lets have this discussion open with the community on what each individual finds valuable when it comes to the charts within the Trends section of a Cycle Cloud Project.
As a refresher, these are the 3 charts we currently have but want to improve:



When it comes to charts, I need them to provide a quick answer to what used to be a manual process of collecting and counting.
I want to know how many tests are passing and how many are failing.
I want to know which tests are flaky, as in they pass today and tomorrow they fail then pass again after immediately being rerun.
I want to know my pass fail percentage so I can identify how well my tests are executing. However I also don’t want tests I’m working on to be included in that count. Tests being written will fail until the test is completed and considered “done”.
So lets discuss! What would make the Trends charts work for you?

3 Likes

This is a really good topic @SethTarrants . I think we have a great opportunity to pull meaningful, actionable results into these charts. Instead of spending all of our time manually collecting and counting results from various tests, the charts give us a chance to see that information right away, identify if there is an issue, and get started on resolving that issue.

In addition to everything you listed, I would also like to know which tests over a given period of time have failed the most frequently.

You bring up a very good point about excluding tests in the process of being written. I think it’s important to exclude any tests that aren’t considered “done” from the trends. Including those results would skew the data and make it difficult to isolate issues with the tests that are “done”. Then you will end up in a manual process of determining which results to ignore and which results to take action on.

2 Likes

That is good mention @Art.Smith . Providing the ability to quickly see tests which fail frequently is quite valuable. Identify hotspots or trouble areas within the system under test so it can be improved or tested a different way.

I’m curious if others have some input, feedback or even opposite opinions?

Man, for reporting a few things come to mind. I’m sure I’ll come back later with more.

  • Test statuses (in progress, development, “production” wrong word probably, etc) and being able to filter
  • Test case tags and being able to filter reports on them.
  • Test case unique identifiers - being able to find the unique id instead of filtering on maybe similar-sounding test cases.
  • Pass vs fail rate of course. Pass with exceptions/errors as a status.
  • Associating tests with a defect (maybe I’m dipping too much into a test management or defect management tool but still).
  • Pass vs fail trends by day, week, month, etc
  • Being able to identify failures or comment on them. We see all the time where a failure is because of the system under test and it’s a real system defect but we also have to diagnose if it was the automation itself first.
  • Being able to clearly identify Cycle or system under test defect for a failure would be a great reporting metric.
  • In an enterprise there might be test case for region “midwest” or region “midwest location ABC” vs region “Northeast location XYZ”. If they have good standards in place they’re sharing a lot of the same test cases and it’s great to filter by those regions, sites, etc.

I might think of a few later @JonathanYiv I’m sure you’ve thought about this before.

1 Like

@magowan
Test Statuses: When you say “in progress” and “development” what does that mean to you?

In development and in progress could probably be interchangeable.
Or in progress could be even higher level of simply not approved for testing and under that umbrella it might be in development, in review by a stakeholder, etc.
In dev would be just that, the test is being written.

Thank you for that detail @magowan and I now have another question.
Are you asking for reporting which outputs which tests are in a state of still being worked on or awaiting approval before they are added to a recurring test execution plan?
Or just a way to remove filter them out from a report?

Any other feedback from interested parties as to what type of charts or reports would be ideal in order to help them gain confidence in how their Tests and System under Test are functioning?

Before I delve into what I believe would be most valuable, I think it’s important to distinguish between reporting and analytics.

By definition:

  • A report is a document that presents information in an organized format for a specific audience and purpose.
  • Analytics is the systematical computational analysis of data or statistics used for the discovery, interpretation, and communication of meaningful patterns in data.

I like to sum it up as presenting data (report) and analyzing patterns in data (analytics).
In Cycle Cloud, we have two available menu options:
image
I see Results as representing reporting and Trends as representing Analytics.

With respect to reporting, here would be my ideal requirements:

Each Playlist (a group of tests run in sequence as one unit) should have the compiled results in a single high-level section:

  • Total Pass (by Test Case and Example)
  • Total Failure (by Test case and Example)
  • Pass Rate (by Test Case and Example)
  • Total Pass by Functional Area designated in the Library Shortcode (by Test Case and Example)
  • Total Failure by Functional Area designated in the Library Shortcode (by Test Case and Example)

Note that Functional Areas may not apply to other systems under test beyond BY WMS, in which case, those metrics can be excluded.
Playlists should be drilled into to view Test Cases as line items.

Test Cases should be categorized by Functional Area designed in the Library Shortcode.
Each Test should have these attributes tracked on a single line:

  • Library Shortcode or Test Case Name
  • Example Number
  • Status
  • Failure Type
  • Error

Failure Type should be populated by a human and should be restricted to a set of configurable Failure Types, including:

  • Configuration
  • CycleScript
  • Dataset
  • Data Creation
  • Defect
  • Environment CSV
  • False Negative
  • False Positive
  • Latency
  • Test Data
  • Unknown

Test Cases can be drilled into to view the detailed execution in a paginated tree view such that context of a given step can be related to its ancestors and descendants without changing the view.
The paginated tree view should have line items for each step including the following attributes:

  • Step
  • Status
  • Screenshot, if applicable

These summarize what I believe would represent a minimum viable reporting mechanism within Cycle Cloud for regression testing.
I can provide thoughts on performance testing and analytics at a later time, but that may make this post too long.

This is a good breakdown for reporting @JonathanYiv . Thank you for taking the time to articulate to this extent.
I am curious about your comments on the charts and analytics aspect as soon as you have the time.

1 Like

I can certainly do so!
As full disclosure, I have not put nearly as much thought into analytics as I have with reporting.
I personally believe that reporting is the first rational step towards actionable data and that skipping over it could lead to effectively useless analytics.
This is especially the case since in my ideal world, I have allotted for user-defined Failure Types integrated with the reporting solution and that would be taken into account for any subsequent analytics.

Finally, before I dive in, I think we should draw the line of what sort of tool Cycle is today.
Today, in practicality, Cycle is a test automation tool and BDA tool but is not a test management tool nor a defect tracking system.

@SethTarrants
Do we envision ourselves as moving forward into either of those domains or any others one day - with Cycle being a holistic solution?
Or do we envision Cycle as being a test automation/BDA tool?
The potential scope of analytics, and therefore my reply, changes dramatically depending on this answer.

@JonathanYiv

That’s a good point you bring up about Cycle being a test automation tool/BDA tool and not a test management tool/defect tracking system. Perhaps in the future Cycle can broaden out into test management and defect tracking.

As Cycle evolves and potentially delivers functionality to facilitate test management and defect tracking, those new areas will most certainly be incorporated into the Trends section and the analytics charts.

Having said that Cycle may broaden scope in the future, I’m interesting in hearing your input on analytics based on the scope of Cycle as a test automation/BDA tool as it is today.

This will give us an opportunity to start iterating and improving how we gather and display analytical data and prepares us with more experience/knowledge in analytics when Cycle does broaden into a more holistic solution.

2 Likes

I would like to add a novel qualifier to the above reporting requirements suggestion.
Everywhere I have outlined a distinction by Functional Area, I also propose a distinction by Interface (i.e. Windows Native Apps vs VT220 Terminal Emulation).
That said…

I would absolutely love this.
I think so many of the enterprise testing tools today are an amalgamation of tools and integrations due to acquisitions and varying product strategy.
I think one testing tool to rule them all by broadening into test management, defect tracking, etc could develop an unshakeable foundation in the market.
We could then work with the Tolkien Estate on a cobranding initiative. Replace the ring below with our logo. @courtneyshaw Here’s a swag idea for you - cycle rings.

Absolutely, sir!
I hereby set four axioms before proceeding:

  • It is my belief that skipping reporting for analytics is a misstep.
  • I am limited by the breadth of my experience and what cursory research seems to indicate is important.
  • I am suggesting analytics under the assumption that the above reporting requirements have been fully implemented.
  • This does not address performance testing.

Here is what I have historically manually compiled:

  • Test Execution - Being able to filter test execution by time, functional area, interface, and so on and so forth allow one to cut up the underlying data in a variety of ways for custom purposes.
  • Failure Percentages - What is the recurrence rate of failure types in a given test suite execution? What is the recurrence rate of failure types overall or during a specified time period? What is the recurrence rate of failure types in a given functional area in a given test suite execution? What is the recurrence rate of failure types in a given functional area overall or during a specified time period? Seeing failure percentages with these filters could provide numerous valuable insights with respect to the impact of a given changeset.
  • Test History - See the failure history of a given Test Case, while excluding certain executions that were due to development or known external factors, to see if any patterns emerge and inform actionable steps such as collaboration with product development. This may also inform Test Case Volatility in the sense that test cases that fail regularly with each release require maintenance., but volatility is more of a test management feature.

Here is what I have historically manually compiled, but appear to be relative to test management and defect tracking:

  • Test Flakiness - See the reliability of a given Test Case over time, while excluding certain executions that were due to development or known external factors, to identify actionable steps for removing or improving unreliable tests. However. this analytic begins to blend into test management and therefore may be out of scope.
  • Defect Mapping - If a Test Case execution eventually leads to the discovery of a defect, then this can be logged such that defect data can be exposed to assess which tests are most effective in discovering defects. However, this clearly maps to defect tracking.
  • Release Mapping - If a Test Suite execution is mapped to a given release, hotfix, patch, etc, then mapping these executions to them will allow us to later expose data on how certain releases fared with test automation.

Overall, I do believe that once a set of vanilla analytics have been developed, offering extensibility in our analytics solution is vital as every organization may have differing needs to address their business needs.
For example, some organizations may be interested in mapping test cases and test suites to certain geographical regions or physical locations and exposing performance based on those filters.

1 Like

Hey @JonathanYiv I really appreciate the passion you brought to this discussion topic. I’ve logged your responses in relation to the analytics we are researching currently.

I hear your concern that we should focus on reporting over analytics and that is a fair callout. Please continue pushing for areas where you feel we should be focusing on in order to help guide the community, discussions and work to be done.

1 Like