Trying to compare Azure Devops vs Jira? We’re here to help. Atlassian offers one of the most comprehensive suites on the market today, but it never hurts to take a look at the competition. To that end, we’ll be looking at Microsoft’s issue tracking and development offering, Azure DevOps (previously Visual Studio Team Services) and comparing it against Atlassian’s flagship Jira product (read about Jira vs Jira Align).
In this article we’ll be primarily concentrating on the differences in terms of Azure DevOps vs Jira, rather than the similarities – so there’ll be no comparisons between Issues and Issues! This also means that the Bitbucket analogous functionality of Azure DevOps will be left to another post.
Azure DevOps at a Glance
Azure DevOps has a similar construction to Jira. Every instance is a collection of projects and users, with users
assigned to projects and security groups defining what each user can see and do in their assigned projects.
Rather than a workflow, every project in an instance is assigned a ‘process’. A process defines which Work Item Types are available to that project and how they can be interacted with.
Work Items are the building blocks of Azure DevOps. It’s a catch-all term for Issues, Bugs, Epics, Features, User Stories and Tasks (each of which is a Work Item Type). A Work Item is a collection of information in the form of values held in the fields, as well as links to other Work Items and attachments (it’s equivalent to an Issue in Jira).
Every project contains a number of Teams, Areas and Iterations. Teams (as expected) are just a collection of project users. Areas are locations where Work Items live (more on that later). An Iteration largely adheres to the common Agile definition.
Projects are divided into 5 sections: Overview, Boards, Pipelines, Repos and Test Plans. As mentioned above, we’ll be leaving out the Bitbucket comparisons, so Pipelines and Repos won’t be looked at here.
What we’re left with is the management of Work Items within a project (Boards and Overview) and the real USP of Azure DevOps, the Test Management functionality (Test Plans).
1.1 Overview
The Overview section contains the project summary, wiki and all the dashboards. The wiki on an Azure DevOps project is not overly impressive. It’s just a tree-like structure of text only pages, no more no less.
1.2 Boards
All the heavy lifting in terms of Work Item management is done in the Boards section. As you might expect, this is the section that contains the Scrum and Kanban boards available in Azure DevOps.
Every Team has one Scrum board and any number of Kanban boards. These (surprisingly) live in the Boards section. The Work Items that appear on these boards is determined by the Area of the Work Item and the process assigned to the project.
The Area of a Work Item is just another field. Every Team in a project has a Team Area. If the Area field of a Work Item is assigned to a Team Area it will appear on that Team’s boards
Finally, we have Queries. A Query is very similar to a JQL query. An Azure DevOps Query is defined by a series of clauses contructed using drop-downs.
If a Work Item meets the conditions defined in the clauses of a Query, it will be displayed in the results of that Query. Once a Query has been created it can be used to generate reports, populate test suites and as well any number of important applications. Like Jira, a lot of the work carried out is off the back of queries. Read more on Jira Align vs Azure Boards.
1.3 Test Plans
The Test Management functionality provided by Azure DevOps is it’s USP. It is the most widely used extension and has now been partially incorporated into the core product. To get the full range of functionality Test Manager does still need to be purchased as an extension. This article assumes (and frankly, recommends) this purchase.
In this extension live three additional Work Items: the ‘Test Case’, ‘Test Suite’ and ‘Test Plan’. A Test Case is similar to a standard Work Item with the addition of a ‘Steps’ section and the ability to be associated with an automated test. The steps section houses a manual test script, a series of actions and expected outcomes. The Test Plan and Test Suite Work Items are essentially just folders arranged in a tree-like structure, which hold Test Cases, with the Test Plan always being at the highest level.
On the manual execution side, once a Test Case has test steps and has been allocated to a Test Plan, it can be executed. This can be done either from the Test Plan itself or from the Requirement/User Story Kanban board (if that test has been earmarked as testing a specific User Story). Test Cases associated with an automated test are collected in a Test Suite, then run as a build.
The overall results will be stored as run data and can be displayed on any dashboard as a widget. Test Suites are used to collect run data and are generally used to power said widgets.
The real advantage here is the flexibility. You can design Test Plans to be automated or manual regression packs, end to end test phases, system test phases or even just to guide manual exploratory testing. It’s very easy to allocate manual tests out to individual users and then monitor their progress. This extension can be used to support pretty much any kind of testing you can think of.
Azure DevOps vs Jira Software Comparison
So, is it any good and if so, how does Azure DevOps vs Jira measure up? Well, what I’ve always said is this: If it worked, it would be an excellent low-cost, simple alternative to Jira (with the added bonus of an excellent test management extension). Unfortunately, it doesn’t always work.
Let’s unpack that.
2.1 Pricing
Azure DevOps comes with 5 ‘Basic’ users and unlimited number of ‘Stakeholder’ users for free. ‘Basic’ level is required for any kind of team or system administration, but your standard user should be fine with a free licence. The Test Manager licence is pricier but importantly only needs to be bought by users that will be scripting or managing test plans – not by users executing tests. So, if you’re smart about the kinds of licence you allocate, a very large instance with full testing capabilities can be run for under $300 a month (link to a full breakdown of pricing).
2.2 Simplicity
This is a bit of a double-edged sword. Azure DevOps does not have the same level of customisation available as Jira. However, what customisation there is, is very simple to set up and therefore it does not require the same level of technical expertise to administer an Azure DevOps instance as a Jira instance.
2.3 It Doesn’t Always Work
The main problem with Azure DevOps is that it feels like a very rushed, young piece of software – which given the length of time it’s been around in various incarnations (TFS, VSTS, VSO…) is not really acceptable. The number of bugs we’ve encountered – even on core processes, is astounding. Most of these are cosmetic, like dropdown lists not rendering correctly or free text stylings disappearing, but some have been more detrimental.
The real source of frustration with Azure DevOps is the number of bugs and functionality limitations in the Test Management module. The details belong more in a test report rather than a blog post, but some examples include: performance of query-based suites, requirement-based suites duplicating when generating across multiple team areas, the usability and organisation of shared parameter sets – trust us, this list goes on for a while.
2.4 The Verdict
Even with the problems laid out above, Azure DevOps can be a good fit if you’re short on time and money. It is simpler but therefore more user-friendly than Jira and most of the bugs are cosmetic in nature.
Given the amount of development Microsoft is currently sinking into Azure DevOps, there’s hope that this will become a serious product and Atlassian should definitely be looking over their shoulder. However, for now at least, stick with Jira.
