We Can Improve What We Measure
We know metrics are often misleading, yet we are still seduced by the seeming authority of numbers to measure technical content effectiveness. Instead, apply a mix of quantitative and qualitative metrics to get a more complete picture of your content campaign’s support of your DevRel initiatives.
Hello, Data — Our Old Friend
It’s so tempting to just jump into the data (traffic, time on page, etc). These are all very helpful metrics when understanding how content is performing, particularly in a marketing context. But those numbers alone aren’t helpful in assessing how well tech content supports a developer community. It’s better to define metrics for technical content after defining the key performance indicators (KPIs) for your developer community.
Typical DevRel Program Goals and Metrics
There are different types of developer communities (and therefore, many different useful sets of KPIs). However, we have found that these three high-level DevRel goals crosscut many flavors of developer community:
- Engagement––Measures the relationships between community members and between the brand and the developer community. This includes specific KPIs, like developer growth by data source (all the touch points that your developer interacts with from the time they start adding value to the community); developer interest by product or project (how many products or projects a single developer interacts with), and developer contributions to the community (asking/answering questions in forums, contributing code, etc.)
- Acquisition––Tallies the increase the total size of (or segment size within) a developer community. KPIs could include number of signups, downloads, and project starts or stops for a particular time period.
- Satisfaction––Indicates the positive experience for the members of the developer community. Example KPIs include time-to-response when a member has a question, time to close an issue, as well as the sentiment of comments within and outside the community. Other related KPIs could be diversity of members as measured by who comments, contributes, recruits or invites others, and who stays/leaves, or contributes/stops contributing.
Measuring and Trouble-ShootingTechnical Content
First, evaluate the success of specific DevRel community projects, initiatives or behaviors, then collect information about the content that supports them. For example, if a particular project is doing well in terms of acquisition, engagement, and satisfaction, check out the traffic flow from project to content — like documentation, technical articles, blog posts, case studies, tutorials, and how-to’s. This gives you insight into the content mix and content use flow that truly supports an initiative so you can replicate it for other initiatives. If a project, program, or initiative is flailing, observe the traffic flow to the related content, then experiment with corrections.
Where Do They Give Up?
If there is a drop in interest at a particular part of a project, ask where they go next. Search for where traffic stops. If traffic flow dies within a specific piece of content, review that content, and look for road blocks. Maybe links don’t work, updates are needed after a version release, or there is another technical, structural or editorial problem. If there is no other explanation, it’s possible developers are finding the wrong content to keep them going.
If developers give up in a project without looking for a solution, this can point to problems like poor content organization, searchability, or indexing. Just as likely –– the project just isn’t interesting enough for a developer to troubleshoot.
What next? If you are in a dialogue with your developer community, you can credibly ask questions. (But only ask if you are prepared to listen and field a solution.)
Serve Up Stories From Your Data
Look for the stories your portal data tells — not traffic data, but the data you can collect from logs, forums, and even support requests. Did an enterprising developer try and apply your product in an unexpected way, giving you a trail of support requests that you can turn into a case study? (Same with API logs, forum exchanges, and even project contributions.) The information you can extract from your community’s activities is solid gold, and it points to what content helps, what content is missing, and/or what content could be done better.
Monitor Your Search Data
Do you know what your developer community is searching for? Look at the searches within your developer portal as well as the Google searches people are performing when they search for your company. This can point both to knowledge gaps and information (content) gaps.
Number-crunching alone is dangerous. Simply pointing to traffic increases, or number of comments, or other so-called vanity metrics feels good, but reveals very little about how well your technical content supports your DevRel goals. A combination of quantitative and qualitative metrics, plus a healthy portion of assumption testing will give you more useful information about the performance of your technical content. Even better, when you examine these metrics with an eye to storytelling, they will help you tell the next technical story your members need.