Interview with an Expert

ALM Works: Hey Erez! Thanks for chatting with us about your approach to improving QA processes.

EM: Of course! It’s my pleasure.

ALM Works: How long have you been doing R&D and QA using SDLC and/or ALM solutions?

EM: About four years. In fact, three years ago I pushed for Amdocs to move away from an internally developed tool toward something off the shelf and broadly accepted. I believed it was the best approach for us then, and I still do.

ALM Works: So tell us, what pain points were you hoping to address with the move to something off the shelf?

EM: One of the biggest drivers for us was full test coverage. Our major challenge was that we knew we lacked full traceability between requirements and testing. If teams can’t easily create clear alignment between requirements and tests, new functionality may be released without being fully tested. When that happens, you have no warning about potential issues in the code until customers start complaining.

For us, Atlassian Jira was the best fit but the essential point here is that an issue tracking system is key. When one is properly implemented and managed, it can be a terrific platform for achieving full traceability.

ALM Works: Based on your experience, how would you advise others to get started building traceability into their SDLC or ALM platforms? What’s step one?

EM: First, get all your stakeholders to the table. This is crucial to getting buy-in and having a successful implementation. You’ll want to be sure to involve all the testing managers, testing engineers, as well as the development stakeholders like Scrum masters and dev managers, in some way.

Next, you should strive to have all related artifacts in one place. You need to have a single source of truth. Using two entirely separate systems — one for requirements management and one for test management — can be a nightmare (it was for us), so getting all the data together will help tremendously.

A common problem is this: Testers might know their way around the QC system, but they can’t provide clear and accurate testing statuses for the requirements in the upcoming version if that information is in a different system. That’s why consolidation is an important first step.

ALM Works: What’s the next step?

EM: Once all the data is together, you’ve got to get the tooling right. The main challenge for us was combining testing levels with our requirements hierarchy.

As I mentioned, we chose Jira as our single source of truth. Then we added on a couple of key plugins to extend it for requirements traceability. We used Structure for Jira to manage all the Jira issue types needed to plan, design, execute and track all testing activities. You need a tool that can work at scale, show the overall status, and allow users to easily drill down into detailed requirements. Structure can do that.

Xray Test Management for Jira is a key plugin for us as well. It is a complementary solution for designing multiple, complex tests as Jira issues. We use it to establish multiple levels of testing (e.g., component, integration and regression). Here again, everything is stored in Jira — our single source of truth.

Structure helps us keep all of this content manageable and visible within the hierarchy of requirements and tests. Xray helps us ensure we have complete test coverage.

ALM Works: Any nitty-gritty details about implementation you can share?

EM: Sure. We always have a test set created for every user story the Scrum team is developing. That test set is linked to the requirements with a Jira link that we can see in Structure and/or Xray. We use a custom “test execution status” field in Jira to ensure and track that every test gets executed.

Our vision is to ultimately have regression tests created and categorized automatically once the functional tests have passed.

Whatever you decide to use for tooling, I’m confident this approach will work.

ALM Works: What unexpected challenges do you think others might face in this type of project?

EM: Terminology differences can be tough. Switching tools can cause major pain if people aren’t speaking the same language. It can cause a lot of doubt and discontent. We struggled a bit with this early on but quickly realized what was happening and took steps to get everyone on the same page. Don’t let that happen to you.

ALM Works: Finally, what do you do when you’re not designing awesome traceability systems?

EM: I love to swim about 10 km every week. In fact, this project was a bit like a long swim. It requires stamina and endurance. Just like a relay race, ultimately it’s worth it when your teammates can complete their sprints with confidence and consistent results.

Erez was a featured guest speaker for our September 2019 webinar on the topic of "Requirements Traceability with Jira." He was joined by Yaniv Shoshani, CEO of Atlassian Platinum Solution Partner, Methoda. Check out the webinar recording to learn how they used Jira, Structure for Jira, and Xray to tackle requirements traceability.

Update

During our webinar with Erez Marcovich, a number of questions were asked and answered. Here’s a recap.

Q1: Manjunatha

Can you quickly describe how a typical test planning is done in the Agile world using issues types test, test set, execution and plan? I would like to understand how you typically structure and organize in the agile world when you are doing sprints?

A: First of all, Agile is a mindset; there are a set of values and principles that are important for Agile. For example, people are at the center of Agile; a tool may help by providing transparency, collaboration, flexibility, automation of certain tasks and more. We invite everyone to have a look at the Xray docs concerning Agile and Agile Testing.

You can use Xray entities in an Agile context, since Xray entities are implemented as Jira issue types.That way your agile team members can easily collaborate on them, and you can have easy status visibility.

From a test planning perspective, you may have, for example, Test Plans per sprint and track their consolidated progress on your Agile Boards (e.g. Scrum Boards) in Jira; thus, you don’t need to go to a specific report to track testing progress.

Q2: Yve

Can this traceability tool link to design outputs (ex. drawing)?

A: The simple answer is “Yes.” You can create a link to anything that is in or accessible from Jira. You can even link to external assets in a “loosely coupled” fashion. If you’d like to explore the details for your specific use case more fully, please feel free to contact the Structure folks at ALM Works or Xray folks at Xpand IT.

Q3: Yve

Can Structure show the relationships between issues that are spread across different projects? e.g. Requirements or tests in a different project as user stories.

A: Yes! This is one of the primary use cases / benefits of Structure. You may have items from different projects in the same “structure.” And therefore you may have User Stories in one Jira project while Requirements and Tests are in a second project.

Q4: Yve

Can a structure be exported to a table (Excel)?

A: Yes, there is an Export button in the top of the Structure screen. You can export to PDF, SVG or Excel.

Q5: Francois

Have you looked into using the R4J plugin for requirements management and coverage/traceability reports? It works well with XRay test issue types.

A: Thanks for that feedback, Francois. We’ll take a closer look.

Q6: Ahmed

Have you migrated all HP QC objects to Xray? Any advice on how to do it?

A: Xray app provides a built-in HP QC/ALM migration utility that migrates QC test cases (only).

To fully migrate, we’d suggest you work with an expert partner, the way Amdocs did with Methoda (as outlined in this webinar).

Q7: Ahmed

How can we manage requirement versions in Jira (like DOORS)?

A: Jira and DOORS are two very different systems. For one thing, Jira issues do not have Version Control (like DOORS Requirements).

If you need a version control for your requirements, you may want to consider RJ4, a requirements management app built on/for Jira.

Q8: Anonymous Attendee

Can you give a comparison view of Jira's native hierarchy structure and reports vs Structure, to get a better idea of structure's power?

A: That’s not really the right way to think about it…. Jira is built around an assumption that everyone will only use the Epic-Story-Subtask hierarchy that is native to Jira. But people use other issue types in Jira all the time. Then they use Jira links to create parent-child relationships between them.

The problem is, Jira only recognizes the Epic-Story-Subtask hierarchy, so there’s no way to visualize these other relationships.

Structure enables you to arbitrarily build any hierarchy you may need, and it makes it easy for you to visualize them. You can create parents of Epics, and parents of those parents, or Subtasks of Subtasks. Whatever you need. Moreover, Structure can also be used as a live reporting tool — you can see changing Jira data in real time. You can also group and sort your issues for these reports and even calculate your own KPIs via Formula columns.

Q9: Reagan

Does Structure provide the ability to create Baselines i.e., a snapshot in time of the state of all requirement and test artifacts at their current state?

A: Sorry. No. Think of Structure as a window into your Jira data. If there were a Jira baselining/snapshotting tool then Structure would work with that Jira data like it does all Jira data.

Q10: Manjunath

In Xray Test creation, we see only a few test types. However, many organizations use different tools like Python, Selenium, etc. How can someone manage these?

A: In Xray, there are currently three different test types: “manual”, “cucumber” and “generic”. These are hardcoded into the system; however, you can use names/aliases that map to other test types.

Also, note that a “generic” test is an abstraction of an automated test (not Gherkin / Cucumber / SpecFlow based) in Jira, so you can report results against it. It can be used to map results from JUnit, Nunit, TestNG or Robot framework tests. Thus, you may choose whatever automation library you want, including Selenium, and whatever language.

What matters is the report format produced by the test runner. Xray supports many formats.

You may also want to have a look at around 50 tutorials we have for different programming languages and testing frameworks: https://confluence.xpand-it.com/display/XRAY/TTT%3A+Automation.

Q11: Manjunath

How can you generate different kinds of Reports related to a. Sprint Level b. Overall Project status health with respect to all the Requirements/EPIC/Stories etc. Is this reporting tool available? Or do we need to get them enabled.

A: There are many ways of doing reporting, either using gadgets, built-in reports, directly in the Scrum Board, or even by using a dedicated app for that (eazyBI).

More info about all these possibilities here: https://confluence.xpand-it.com/display/XRAY/Tracking%2C+Measuring+and+Analyzing.

You can also track testing progress in Structure as shown here: https://confluence.xpand-it.com/display/XRAY/Integration+with+Structure#IntegrationwithStructure-ShowprogressinformationofTestExecutionsandTestPlans.

Q12: Rupali

What is the review mechanism in XRay? How are the review comments are captured?

A: Xray entities are Jira issues. You may take advantage of the built-in Jira workflow mechanism. You may see some concrete examples and some recommendations here: https://confluence.xpand-it.com/display/XRAY/Using+Jira+workflows+for+testing+purposes.

Q13: Rudolf

I'm interested in how IBM DOORS requirement can be integrated.

A: As far we we know, there is no off-the-shelf solution for this; however, we have seen a number of customers do this using custom-built integration. They convert each requirement to a Jira Issue, or a Confluence Page. Alternatively, you can convert DOOR requirements to REQIF documents and then take it from there to Jira or Confluence.