Cycle Cloud - External Feedback

I was able to get Bas Dijkstra to sign up for Cycle Cloud last week. Bas is a very well respected name in the test automation space with some particular expertise around BDD. I’m meeting with Bas tomorrow to review his feedback in more detail but here are his initial thoughts:

  • The ‘in web browser’ part of the predefined steps gets really boring really quickly, lots of repetition throughout the scenario
  • There’s a heavy focus on UI interaction, are you going to add other steps as well at some point (for APIs, for example)?
  • Is there a way to create reusable blocks and thereby make the scripts easier to read and maintain?
  • Please don’t market this as ‘doing BDD’, even when you’re using the Gherkin scenario, and be really clear about this when people ask :slight_smile:
  • What happens (or should happen) when I click ‘Queue’? I expect a way to run a scenario, but nothing seems to happen…
  • Why does everything happen in the same tab yet when I click to open a project it opens a new tab?
  • The ability to drag and drop steps / snippets would be great, would save me a lot of typing…
2 Likes

Some notes for further conversation for Bas

  • In Cycle 2 - this manages context switching between various technologies. We will bring all of these other technology interactions forward into Cycle Cloud and this was our MVP attempt at managing context switching in Cycle Cloud - open to thoughts.
  • See follow up above ^^. As we think about bringing additional support in, we are curious what your thoughts might be regarding the the line between paid/free regarding the proper amount of value in the free version but also leaving enough encouragement to upgrade for additional value.
    • What other technologies do you believe are important to pull forward? (REST vs JSON/XML etc…)
  • Can we get Bas to expand on this a bit? Potentially provide some examples.
  • Let’s talk BDD methodology and go from there…

Can someone from the Product team product guidance on these last 3?

I also received some feedback from Heath Howe who is a local test automation expert here in Raleigh. Heath is collecting further feedback but his initial feedback (which I also got from Bas) was that our signup/login process is to convoluted.

So I don’t have an answer to all of these, but as a member of the delivery team, I can speak to my experience with some of these -

  • The ‘in web browser’ part of the predefined steps gets really boring really quickly, lots of repetition throughout the scenario

This is a good call-out but is simply part of the Early Access lifestyle of Cycle Cloud. Anyone who is familiar with Cycle 2x knows that we need to have the various “in browser,” “in terminal,” “in app,” etc. steps to tell the software where to look. AFAIK Cycle Cloud just uses web steps at the moment, but should be expanding into the different contexts over time. Hopefully with this context, these steps make more sense

  • There’s a heavy focus on UI interaction, are you going to add other steps as well at some point (for APIs, for example)?

Is Bas familiar with the Cycle Client? I don’t know if Cycle Cloud is set up to use that yet, but it sounds like something he’d be interested in. Pinging someone from the Product team may offer more insight.

  • Is there a way to create reusable blocks and thereby make the scripts easier to read and maintain?

Ah - sounds like we need to introduce this good man to wonders of Utility Files! If he hasn’t gotten a chance to look at the Test Automation Framework, I would encourage you to introduce him to it. The TAS’ utility scenario focussed design - breaking everything into modular pieces that can be managed and called - sounds like exactly what he’s looking for!

  • Please don’t market this as ‘doing BDD’, even when you’re using the Gherkin scenario, and be really clear about this when people ask :slight_smile:

I’m not sure I understand this point. What does market ‘as doing BDD’ mean. I see you’ve linked another discussion on this though so I’ll probably head over there for my answer XP

This is a good call-out but is simply part of the Early Access lifestyle of Cycle Cloud. Anyone who is familiar with Cycle 2x knows that we need to have the various “in browser,” “in terminal,” “in app,” etc. steps to tell the software where to look. AFAIK Cycle Cloud just uses web steps at the moment, but should be expanding into the different contexts over time. Hopefully with this context, these steps make more sense

I totally agree @Micah.Jones - and that context makes a ton of sense for those of us familiar with Cycle 2x. But as we gain traction with new users in this new cloud context I wonder if it’s worth challenging ourselves to think of other ways to manage context switching that isn’t part of the Steps themselves?

Is Bas familiar with the Cycle Client? I don’t know if Cycle Cloud is set up to use that yet, but it sounds like something he’d be interested in. Pinging someone from the Product team may offer more insight.

No - he is aware of our organization but hasn’t been able to get his hands into the application until now with Cycle Cloud. I was able to spend some time talking with Bas this morning though and it was able to explain more of our path. We talked about maybe having web and API steps as part of the free tier and that could be valuable to the broader community.

Ah - sounds like we need to introduce this good man to wonders of Utility Files! If he hasn’t gotten a chance to look at the Test Automation Framework, I would encourage you to introduce him to it. The TAS’ utility scenario focussed design - breaking everything into modular pieces that can be managed and called - sounds like exactly what he’s looking for!

Totally agree! And we talked about some of Cycle’s capabilities there. I don’t know how much of that is functional in the Cloud version yet. Perhaps someone from product can let us know?

This has been a point of contention, the app is in a bit of a mid-way state between our experimenting with a system like Google Drive (where there is a centralized ‘asset’ view that spawns additional tabs for each ‘asset’ that is open) and the virtualized tab system we were using in Cycle 2 (where there is a single tab in the browser that contains an infinite number of tabs that only exist within a webapp).

We were unsure which approach was best, so we chose the one we hadn’t tried before and gave it a shot. What we discovered is the data system we use behind the scenes requires each new tab to phone home and reauthorize (you can see this when the Home view opens a new tab, the screen goes white and says “Authorizing” up in the left corner) before displaying the rendered view. It’s not great and the user feedback we’re getting supports that notion.

We’re beginning to look into virtualized tabs again to resolve the issue.

2 Likes

Thanks @mitch - that makes sense!

@basdijkstra - Hopefully that answers your question on that. Let us know if you have any additional thoughts on user experience considering those points.

That’s a really good point about the tedium of always typing “in web browser”. The more re-usable blocks fits in well with what was discussed in demos last week w/ some of the language change proposals. A couple of interesting questions related to that would be:

  • In Cycle 2, those reusable blocks are implemented as “scenarios” with “wip” tags. Using a different key word than “scenario” would be good to differentiate a reusable block of steps from an actual test case.
  • I’d be interested in getting more feedback on scoping rules. In Cycle 2, for each scenario, everything becomes a global variable, all your test data/inputs and any result data that’s produced while the test runs, e.g., some text scraped from an html element. As a result, there are a lot of steps in people’s features that really have nothing to do with what it is you’re actually testing. They’re just there to manage test input and result data - has this value already been defined somewhere else or explicitly clean up this value so it doesn’t leak out to the rest of my steps. Being that they’re interleaved with the actual steps of the process being tested, it makes it more difficult to read and easily understand what a test is actually doing. I’ve had some mixed feedback from people on making inputs/outputs for a block like that required. Some felt the more structure that’s required, the more friction there is in getting people started. We could make the inputs/outputs optional. Example of using defined inputs - if your block requires a URL to be supplied and that value is missing, fail immediately rather than waiting until half the block executed before it fails. Example of using defined outputs - if you create a variable in the block that’s just used to iterate over rows in a csv file or whatever, unless that variable is explicitly defined as output of that block, don’t propagate that value out to the rest of the test, and don’t make the user have to explicitly clean up every value that they don’t want leaking out to the rest of the test.
  • If we’re going to add something like a formal input/output declaration for these reusable blocks, perhaps an indicator of the resource/environment used by that block might really help with readability and maintenance as well. E.g., define that a given block requires a browser, so within that block all the “in web browser” text can be omitted, as it is inferred by the context.

This is all really good feedback, and we are chewing on it as we prioritize updates to Cycle Cloud. Some of these are really shaking up some of our assumptions about the product (particularly around how the language works and the structure/scope of tests). Good stuff!

1 Like

That’s awesome @jillian.ketterer - I love to hear that! :grinning:

@brian – one thing Bas had mentioned was that Cycle’s implementation of Steps reminded him very much of Robot Framework’s Keywords. I’m not sure how exactly translatable keywords are to our steps but it may be worth understanding how those work if you don’t already know.

1 Like

Very good points, thanks for the elaboration. That makes things a lot more clear!

I think the last point Josh mentioned with regards to the similarities between what I’ve see from Cycle cloud so far (which arguably isn’t all that much) really reminded me of Robot Framework.

A platform that has the flexibility of Robot Framework, including:

  • Libraries that plug in new functionality, for example for different types of applications
  • The ability (for the open source community?) to build their own libraries
  • Providing a programming language-like syntax, with support for variables, constants, building functions/methods (in the form of higher-level reusable building blocks) etc.

but as a SaaS solution, with nice reports, good CI integration, etc., might be a really useful platform to use. I’ve seen a lot more people using Robot Framework in the last couple of years, but it’s still very much something you host yourself, you’ve got to deal with all the dependencies yourself, the reporting is fairly standard, etc. If Cycle can offer that out of the box, might be interesting for many.

5 Likes

As for the whole ‘do not market this as doing BDD’ thing, that’s a bit of a story. There’s too many people claiming they’re doing BDD when all they do is writing test scripts using Given/When/Then. That’s not BDD, that’s sticking an abstraction layer on top of your test code. Often an unnecessary layer.

If you’re using the Given/When/Then Gherkin syntax, people will associate that with BDD, even though they’re two different (although related) things. There’s nothing wrong with using the syntax (I think it does make sense) but BDD is much more. It’s a process, a way of creating software, much more than just a technique or a tool.

And that’s the story you need to tell in your marketing, especially if you continue using the Gherkin syntax. Otherwise people will get confused, start associating this with doing BDD, then the purists will barge in and slag you off for it, a discussion you don’t want to be in, I think.

Happy to tell you more about this if needed, just ping me.

6 Likes

This is really valuable feedback - thank you, Bas!

@basdijkstra, based on this response from you, I think we are pretty aligned in our thinking. I generally feel that people focus on tools instead of process and people - then, when the tool doesn’t magically fix their problem, they will often blame the tool. Sometimes the right tool wasn’t selected for the process, and sometimes the right process isn’t being used with the tool - and all of those issues boil down to PEOPLE.

I have certainly seen what you described - folks saying they are doing BDD when they are simply using Gherkin. It is akin to folks saying they are doing Agile when they are simply using Jira or a Kanban board to manage work. I appreciate that you called this tendency out as I think it’s an important consideration in the story we tell with our marketing and our product!

2 Likes

Valuable guidance @basdijkstra - thank you!

Hey @jillian.ketterer, that’s exactly what I’d like you to prevent from happening with Cycle. Too many people get this wrong, then the whole BDD community, including the purists, get annoyed, doesn’t do wonders for your product and the way it’s regarded :slight_smile:

You’re absolutely right in saying it’s almost always a people problem, not a tool problem. Jerry Weinberg got it right: Gerald Weinberg quote: No matter what the problem is, it's a people problem.

3 Likes

Great points @jillian.ketterer and @basdijkstra! As we think about the growth of Cycle in the community we’ll put extra focus on ensuring our messaging takes a people-first approach.

@courtneyshaw - let’s think about this part of the conversation as we go forward with some of our messaging

1 Like

You’re welcome, Josh. Let me know if there’s something I can do to help.

2 Likes

I’m certain we’ll come up with something :grinning: