Building an enterprise software usage report in the age of SaaS

“Just get me a software usage report”

It’s a phrase that most IT professionals dread – and that was before the average enterprise had hundreds of SaaS applications. For the person asking, the request is simple: She just wants to know how much software is being used, and probably wants the usage compared to how much software is licensed. Simple, right?

Not so fast. It turns out that software usage reports are incredibly hard to create — even more so if you expect to meaningfully track application usage. But they don’t have to be! This post will walk through modern software usage reporting with a focus on SaaS. More importantly, this guide will help you understand:

• Why might someone want a usage report? What are the key use cases for it?
• What are the common problems for saas usage tracking?
• How does one monitor usage, including usage monitoring software options?

Let’s dig in.

What is software usage reporting? How does it apply to SaaS?

First of all, let’s clarify what a software usage report actually is. There are three ways to think of application usage tracking, depending on who you are. Here’s a quick summary of the most common patterns:

• Product managers look at software usage analytics to understand the adoption of their application by individuals who are likely outside your company. They may possibly be internal users if you are building a custom application for internal workflow or similar processes.
• Developers and DevOps may look at usage analytics combined with application performance to understand areas of stress within their application or areas for performance improvement.
• CIOs, IT administrators, and Procurement look at SaaS and software usage reports (or the related usage monitoring software) to understand the broad portfolio of applications used across the employee base, and how well those applications are being adopted. This can include license usage and cost, but ideally includes deeper analysis of what employees are actually doing with those applications.

So, while Product Managers and Developers are typically looking for depth in a single application (or the specific areas of code that they’ve personally built), CIOs need breadth as well as depth across a large number of applications which they most likely didn’t build. In fact, in the case of software as a service, they may not even run those applications themselves.

Now we have more context around usage reporting, depending on who’s asking. And while there are many resources for developers and product managers to track what they need to do their jobs, it’s less true for IT professionals. So let’s go into more detail about the CIO and IT use case around software usage reporting.

Use Cases for Software Usage Reporting

IT usually has more than one purpose for tracking SaaS application usage. So, when considering a reporting project like this, it’s helpful to understand exactly why someone needs this data. There are a half dozen common reasons for usage tracking:

1. License Management – Can we report on how many licenses per application are being consumed? For SaaS applications, can we report per tier (basic/advanced/enterprise etc)?
2. Auditing – Can we report for audit and compliance purposes on who is using particular applications, and what corporate information is flowing into particular applications?
3. Software Adoption – We’re paying for many applications. Can we track application usage to see if these investments are being adopted? And specifically, can we track if key features within those applications are actually being used?
4. Renewal process – We’re coming up on a renewal for a particular application. How are we doing with it? Is it being used? Are employees happy with it? Are we using the product at the right packaging tiers? Are there alternatives we should be considering?
5. Mergers and acquisitions – We just acquired another company. What does their application portfolio look like? What applications have been adopted significantly? Where do we have overlap? What investments should we double down on, and which should be sunset?
6. Future software investments – Are users adopting the applications we have invested in, or are similar applications appearing as shadow IT in our portfolio? For applications that are being adopted, do our usage reports show where we should be proactively increasing the available functionality of those products?

In an ideal world, any reporting on SaaS application usage can answer all of these questions at once, but this context is helpful for anyone doing the analysis so that she can provide the appropriate depth of information.

Now, in a data reporting exercise, asking the right questions is only about a third of the challenge. How you capture usage information, normalize it, and report back is a real mountain to climb. Let’s talk through some of the common software usage reporting challenges that IT faces.

Common problems for enterprise SaaS usage reporting:

1. Knowing what software to report on. In the age of user-driven acquisition of applications, it’s actually pretty hard to know what software is being used. Sure, you’re likely to know that your company has a company-wide contract with Salesforce or Microsoft, but what about the viral adoption of Slack within Marketing, or the fact that Sales has standardized on Zoom for customer communications? Some of these dynamics will be clear to IT, others are waiting to be discovered. 
2. Getting feature-level software and SaaS usage reporting. Even within a known application like Salesforce, how can you accurately get a picture of what people are actually using within the application? That is, are the features they are using appropriate for the tier of license you’ve given them? And how often do they use those features? What if you have multiple instances of the application for different business units or geographies?
3. Real-time data reporting on an individual’s usage. While executives most frequently think of reporting on an application’s usage, they are actually asking about the aggregate usage of many individuals. To the data gatherer, this is a really important distinction: if the approach to SaaS application monitoring starts at the user level, most questions around feature usage, license usage, audit and compliance, and future investments can be effectively derived from the information at hand. If, however, the starting point is the application or the license, then application tracking is much more limited in its scope and impact.

How to monitor software usage: three different options

Surveys

This is the most low-tech option available to you. The pros for using surveys as the basis for your software usage report are:

• It’s low cost.
• You can run them when you like, across a specific set of applications
• You can customize your questions per application
  
  There are however, a number of cons which might make surveys far less effective than software or SaaS usage monitoring applications:
  
• It’s high effort on the part of the administrator
• Results can vary depending on response rates and bias based on who responds. For example, only the people who love an application might fill out a survey, even if they are the minority.
• Survey respondents will often tell you what they feel or what they intend, but it’s harder for them to share what they have actually done with an application.
• They are point-in-time, not continuous data collection
• How many surveys can your employees handle before they go dark on you?
• Significant post-survey data analysis efforts on the administrative side, including correlating data with contractual and license information.

Application administration dashboards

Individual applications may provide some amount of usage reporting within their product. For example, here is an application usage report from Microsoft Office 365:

At first glance, it seems like a great high level view of a company’s use of Microsoft products.

The pros of a dashboard like this:

• It is included with your purchase of the application
• Typically no effort to set up

The cons of this approach:

• This data is primarily accessible to the administrator of the application, making it more challenging for others in the organization to access it
• It’s hard to normalize this data across applications – different vendors will report on different statistics according to what they feel like they want to share with their customers
• If you have more than a handful of applications, using each one can be a significant burden to gather the necessary usage tracking data
• Application vendors’ own software usage monitoring rarely correlates this information with licensing information to optimize usage of the application

SaaS Management Platforms

Usage monitoring software has now evolved to address the challenge of application tracking in a world where “your” apps are really vendors running their software for you in disparate locations across the globe. These tools are called SaaS Management Platforms, or SMPs, and are a third approach to how you can monitor software usage.

SMPs connect directly to SaaS applications you use, as well as through layers like cloud identity management, in order to provide you a complete view of your application portfolio.

The pros of this approach:

• See all of your contracted applications, as well as discover shadow IT that users are adopting
• Some SaaS management tools provide normalized, deep feature-level insight on an application by application basis, essentially replacing the need of dozens of vendor-specific dashboards or employee surveys
• Optimize renewals by combining the usage data with contractual data
• Minimize ongoing efforts by implementing workflows to provision and deprovision licenses automatically
• See security and compliance details for each application, across each user and every team
• Easily give access to all the data to IT, Procurement, Security, Finance, and business leaders

The cons of this approach:

• You will need to administer another piece of monitoring software
• There will be a one-time setup effort to get up and running
• Depending on the platform you choose, you may have to pay for the monitoring software

There are any ways to build a software usage report

Tracking application usage is important, but the effort that you put into the challenge should be driven by the questions you are trying to answer, the breadth of the applications you’re trying to answer them for, and the frequency at which you need to answer them. 

Over time most organizations need to automate and centralize their application usage tracking, so SaaS Management Platforms will likely make sense to drive real-time collaboration, better software utilization tracking, automation, and ultimately right-sized application investment. In the short term, however, you might be able to squeeze by with another method if you have limited time and limited needs for the data at hand.