Is Cycle "doing BDD"?

Continuing the discussion from Cycle Cloud - External Feedback:

One of Bas’ talking points was that we shouldn’t market Cycle as ‘doing BDD’. I think it’s difficult to say any tool (Cycle or otherwise) is “doing BDD” without an in depth understanding of the methodology that is associated with implementation/usage of the tool.

What do you guys think? Is Cycle doing BDD?

So this is an interesting question - if you needed me to answer this without prep time, I’d say it is designed to work atop a BDD framework, not that it’s “doing BDD.” Let me explain my logic and then we can really dissect it.

As I understand it, BDD (Business Driven Design) is less a thing you do and more a process of thinking by moving the planning left and including the business partners in on the test development process. Cycle doesn’t do this for you. Rather, it’s designed to be used for this development style. The best way I can illustrate this is by describing how we use it here in Cycle Labs -

  • Before even opening Cycle, we have a conversation with our partners
    • What are their goals?
    • What are they trying to test?
    • What are they really trying to test?
    • We explain we aren’t the experts of their system and they we succeed together or we don’t succeed - this makes a clear that we’re going to rely on their business knowledge
  • After this initial conversation, we move on to a series of “BDD Sessions.”
    • This puts the partner in the position of authority, hopefully making them feel more empowered
    • It lets us take a guided tour through their process to ask questions and better understand the various fail states
  • Once we’ve captured information from the BDD sessions, we draft up Test Specs to basically show that we know that processes and pass them to the partner for approval
  • Finally we open Cycle and begin development

So, why did I write all this and then barely describe the Cycle portion of this? As I said at the top, BDD isn’t something you do. It’s a mode of thinking. It’s a design process. Everything I have described is just as important if not more important than Cycle’s abilities, as it ensures our partners are involved with development and have a stake in its success.

But didn’t I also say Cycle was designed to work with BDD? Good catch. So what do I mean by that?

Cycle is a Gherkin language. We write in plain text and can describe our process as part of the development. When using more traditional coding languages, developers have to write in comments to describe what’s happening. In Cycle, that’s just part of the code itself.

So when we take notes in our BDD session, we’re already laying out the framework for the test. When we draft up the Test Specs, we’re fleshing out that framework and noting areas we can create utilities for better reuse and DRYer code. When we have that initial sit-down meeting and determine what the client’s needs are, we are already designing, through conversation, what test style we’re going to utilize.

So, does Cycle “Do BDD?” I don’t think so. I propose it is designed to excel within a BDD framework.

6 Likes

@Micah.Jones - well said! I think that’s exactly it. Theoretically Behaviour-Driven Development or BDD is a methodology that technically isn’t designed to automate tests, but to automate specifications. And because the existing tools (think SpecFlow, Cucumber, Cycle, Karate and more…) don’t necessarily enforce those ceremonies and practices there are tons of folks who use Gherkin-style testing to simply build test automation suites – and this is a hotly debated topic because many folks would say if you aren’t using Steps to follow the BDD methodology then you aren’t getting the value of those tools and that likely there are better automation tools for unit/regression testing that are designed for those purposes.

So I agree with you Cycle itself isn’t doing BDD – but I would say neither are many of the other tools. I do think though, that Cycle facilitates the BDD process if the practitioners using Cycle believe in and follow the methodology.

And furthermore, I think Cycle can actually dramatically expedite BDD practices with our predefined Steps that we use to map to those higher-level specifications in what I believe is the only BDD-focused tool that allows for authoring/execution with or without parallelization all in a simple-easy-to-use platform. Bas had some interesting thoughts on some value in the BDD space that could be achieved if Cycle is positioned appropriately.

2 Likes

I agree with what other folks said, for the most part!

I don’t believe Cycle or any tool can force a team to “do BDD”. The requirements and development process is a human process. Tools are designed to support this process in different ways. Cycle supports it by enabling users with different technical and/or business backgrounds to gather around the same document and speak the same language.

But I believe we can do better than that!

Behavior-driven development (BDD) starts with generating intended behaviors of the process or system. There are many processes for generating these behaviors: brainstorm meetings, example mapping, user story mapping (depending on what level of behavior we are talking about), talk aloud exercises, recording actions, etc.

My observation over time has been that the right approach is whatever gets the job done, and teams don’t all work the same way.

I think if we understood at a much more detailed level the variety of ways teams generate intended behaviors, we could provide a better Cycle user experience to support it. Kind of like what @Micah.Jones described above - that is our process, and it works! Are there other approaches we can learn from too? Would the ability to gather requirements via a diagram help? Are there integrations that could help?

Similarly, I’d be interested in the variety of real-life ways this list of intended behaviors feeds into the creation of tests (whether manual or automated) and development/configuration of the system under test. I don’t mean at the theoretical level, but literally who does what? What meetings happen and who attends? How are tests reviewed? What gaps exist or what struggles are encountered? What works well? I want all the messy human details! We ask these questions often, but I think getting answers from a broader community would be so helpful to us.

3 Likes

Yeah - this is a good point I know we can sometimes slip on: We have a methodology that works for our team and so we try to share it with others, but just because it works for us doesn’t mean it’s a one-size-fits-all. I don’t know how we capture those external processes or how we implement the knowledge gained there, but it would be cool if, long term, Cycle had some built in BDD tools - though that may be asking for feature creep in a software that already has a pretty tight and well-defined focus.

2 Likes

I know this has been close to @Evan’s heart here recently as well! Can we make Cycle more valuable and grow our user base by creating tools more conducive to the involvement of all three amigos?

1 Like

@Micah.Jones I think Cycle 2.x has a pretty tight and well-defined focus, for sure! And we definitely want to do the same for Cycle Cloud. But we have opportunities to redefine some of the focus in Cycle Cloud. Are predefined Steps necessarily the best thing for our users, or should we have other options? Do we want to play with variable scope? I don’t know that it is scope creep to consider the broader BDD story, here, and the various ways we could tackle it. We are figuring out the scope to some extent, based on feedback from here, discussions with users, data from the platform, etc. We don’t necessarily have to take Cycle 2x and put it on the cloud - maybe we end up doing that because it makes sense, but also, maybe we prioritize other things that add value to our users. It’s an exciting time!

2 Likes

I think the problem is associating Cycle with BDD at all. Cycle uses Gherkin syntax. Calling Gherkin syntax “BDD” is a misnomer (as clearly discussed above) and reinforces, even lends credence to, a lot of bad BDD habits in the industry. I recommend eliminating BDD from any Cycle references other than general discussions.

Hello @heathh and welcome to the community!

The position in which we find ourselves in is one where most folks wanting to use Cycle come to the table with with a legacy regression test suite of 1000’s of manual tests they run and rerun week over week, month over month. The immediate ask, of course, is to estimate the automation of those existing test suites. A practice that I know you, @heathh , have a strong opinion about.

What I would love to do is tee up an entry point with that part of the enterprise market to get them thinking differently about approaching the automation of the validation of their most core and sensitive requirements. In this industry that’s likely requirements in the area of Primary Picking method and Primary Allocation method. But we need to walk down the path together and go on a journey to achieve this. Taking all advice on how to start that conversation. I’m happy for that process or method that leads into automation to be BDD, but I’m just as happy with any method that leads to reduced timelines and risk and higher quality for these enterprise folks.

What would this group suggest?

By “Primary Picking method” and “Primary Allocation method” are you meaning how they pick which tests to automate/automate first? If that’s the case, that has nothing to do with BDD and imo will not be helped by implementing BDD for their development process. Determining what to automate, writing those automated tests, and implementing BDD as a way to work are each big asks and separate processes, and I don’t believe they should be undertaken at the same time.

Gherkin is a syntax used for writing tests and may or may not align directly with BDD outputs (e.g. story acceptance criteria). The reverse is also true – BDD can be used very successfully with automated tests without a single line of code written in Gherkin syntax.

Primary pick and allocation methods are the most complex and notorious areas of troubleshooting and costly failures common in supply chain execution systems. Its short hand for a subset of a specific industry’s critical functions.

I totally agree that determining what to automate, the automation itself, and the facilitation of the upstream conversation that has led to those outputs is a lot to juggle. I further totally agree that those upstream methods like BDD are commonly very useful and have an ROI without any automation at all. So my question –

Say, for example, we have an industry that is engaging in a conversation around modernizing. This is an industry that still in large part makes use of systems from the 1970’s and is replacing them slowly with packaged on-prem and cloud software solutions which are customized and contain configuration that’s as complicated as code if not more so. So far the engagement with such an industry has largely gravitated toward to automation of legacy test cases. I think many of us understand the fallacy there.

Coming around to my question, as an entry point for each new conversation with each new player in industry that is attempting to modernize, where would you start? Would you begin where there is some energy and desire (in my above example that’s with automation of less than idea tests) or would you begin upstream of that with methods like story or example mapping with the understanding that this is not what your customer has budget or energy to entertain?

In short can we entertain an original sin in the form of a late entry point into test automation skipping some upstream details if that sin allows an opening for a conversation to occur around the more holistic approach with an industry titan that would have otherwise never considered such modern means?

Looking for all opinions here. This is something I struggle with.

Thank you for the clarifications. I feel resetting expectations is part of a being a responsible authority in the space. That said, I understand the need to maintain the energy (and the customer). If there were only energy/money for one significant undertaking around test automation, and I were doing consulting, I would probably present the idea of filtering the tests and getting the automation ball rolling as the same effort – filtering for critical path (core) and highest risk (sensitive requirements) to implement those first. As the automation effort moves forward, they’ll get a feel for how much appetite they have for it, how much benefit they feel it’s providing, and how much maintenance it requires.

It’s also worth asking the question for the less-core, lower-risk tests something like…how many times over the last XX runs has this test failed? If the answer is “none” then… how critical is it if this were to fail? If the answer is “not very” then the bottom line is…it probably doesn’t need to be automated. It just becomes an opportunity for longer test runs, false positives, and more maintenance.

That’s my 2¢…not sure if this gets to your point, but :crossed_fingers:

3 Likes

This is great conversation everyone - it seems like in many ways we are all in some level of agreement that BDD is bigger than any particular tool and that it’s often misused and conflated. Related, I was having a really good conversation last night with @branko (of Gwen) and we were talking about this very topic. @branko even suggested that perhaps it’s time we start to think of alternative to BDD that might not strike such a nerve with the existing community.

It reminded me of a post he had made suggesting that perhaps BDA was a reasonable alternative that captures a different and perhaps more automation-targeted purpose.

Related, and internally at Cycle we used to refer to what we did as BD3 where the idea was that Behavior-Driven Design and Deployment (BD3) is an extension of BDD that enables teams to take advantage of the benefits of BDD when deploying and extending packaged applications rather than developing the solutions themselves. It’s similar concept but focuses on deployment rather than development of solutions.

BD3 applies the principles of BDD when project/deployment/implementation teams, rather than purely developing software, are tasked with installing, configuring, extending, and deploying packaged software solutions.

With either of these options BDA or BD3 it seems the goal is to somewhat segment the activities from the purist view of BDD to try and take the best parts of it and apply it to a different context.

What do you think?

1 Like

Related when John Ferguson Smart reviewed our example Feature file, he pointed out that perhaps it was still at too much of an imperative level, describing implementation rather than intent.

For comparison, here is John’s feedback and an example:

Compare with this one (which came from a recent Serenity Dojo session): each step expresses the intent, not the implementation, and leaves the details free to be implemented in the most appropriate way. (Also, we generally avoid using “I” in scenarios, as it doesn’t identify the WHO very well).

So to me, this begs an interesting question. When it comes to testing an existing solution, like most Cycle customers – and existing processes with defined UI interactions. If we are constructing a Feature file after the fact – where is the line between declarative and imperative with regards to BDD value?

I’m curious what your thoughts are @heathh, @basdijkstra, @AutomationPanda and @branko

I’m thinking that features should always be declarative and never be concerned with automation or implementation details. A separate mapping layer that acts as the automation glue is where all the imperative stuff belongs. I haven’t had a close look at how Cycle labs maps features to automation instructions, but would imagine that’s where all the imperative steps would go (and not in the feature).

Recently this is the approach I prefer and in my recent work in a similar space I’m introducing rules to enforce it by default so that all imperative automation steps are confined to the mapping layer. Even if writing features after the fact, it’s a cleaner approach. If a client however still wishes to write imperative features that describe automation behavior only, then that should be a conscious decision for them to make. You could perhaps provide a configurable option so they can turn off the declarative rules.

So that’s my opinion. Encourage declarative by default but give clients the option to override at their own choosing.

1 Like

That’s a really interesting thought @branko. In today’s world, even the implementation of the more declarative steps are constructed in the form of a Feature file. Often, we’ll find that with an existing application our users will want the Feature to exist more imperatively to almost act as a standard operation procedure while we can also map that to a more declarative version that doesn’t necessarily express implementation details.

It may be worth considering alternative ways of doing the mapping though so there are more mediums than Feature files themselves. ← @Evan

I like the idea of an on/off rule set like you’re working on in Gwen - sounds like a great way to enforce and encourage best practices.

@branko - would you mind providing a link to your rules engine? I’d love for our team to see what you’ve done

I’ve written an article about it here: Strictly Good Gherkin with Gwen – Gwen Interpreter

The code is open sourced. The rules code is here essentially: gwen/src/main/scala/gwen/core/behavior at master · gwen-interpreter/gwen · GitHub

In early versions of Gwen, imperative was the default (what the article was based on), but we’re going to reverse that in the upcoming major release to make declarative the default instead. Note there are two layers of rules.

2 Likes

Fantastic! Thanks for sharing!

@brian - would love your thoughts on this.

I think the declarative approach reads really well. I like how it succinctly describes the expected behavior. I see a lot of feature files in Cycle written more imperatively that immediately deep dive into such detail about every UI interaction that it takes too much time to really reason about what a particular test is actually supposed to be validating.

1 Like