Measuring Developer Relations

swyx 2021-08-16

“Games are won by players who focus on the playing field –- not by those whose eyes are glued to the scoreboard.” - Warren Buffett

Special thanks to Matt Asay, Chris Aniszczyk, Gwen Shapira, the Software Defined Talk crew, and many others for their discussion of this post!

Image description

A few years ago the key theme of one of the DevRelCons was “metrics”. People breathlessly tweeted about how this was the most important topic in DevRel, to general agreement. The 2 day schedule was packed with 48 speakers with impressive titles stack ranked by follower count. Hundreds of DevRels flew in to hear DevRels thoughtlead DevRel. Drinks were drunk, hands were shook, empty promises made. Then they went home.

Nothing changed. Today in 2021 people continue to talk about how hard it is to measure DevRel. The full videos of talks to this highly anticipated paid event worth hundreds of dollars per ticket were released on YouTube, with a total view count of 601 (not a typo). The organizers did an incredible job garnering 18 sponsors, all duly ignored. My company was one of them, spending thousands on a Gold slot for which we got 4 tickets, presumably to learn DevRel best practices and hire great talent. We sent none.

I did cherry pick an egregious anecdote, but it helps illustrate the typical state of DevRel accountability today. Lots of sizzle, questionable steak. Loudly performing “DevRel”, yet indistinguishable from ”Con”. Yet despite lack of visibility, companies continue to invest! Because we do know that through all the noise and gladhanding, some value does get created and it is unique to all the other user acquisition channels available.

I of course don’t have the perfect answer. But I’ve lived it for a while and wanted to write down my evolving thoughts for others who are going down this same path. Instead of solutions, I’ll offer structure. Instead of answering your questions, I hope to offer you better ones.

Note: I write for new DevRels as well as people running DevRel programs. This is a “201” blogpost rather than a “101” — I’m aiming to cover what introductory blogposts don’t say. I will also have inconsistent capitalization and voice because I am condensing a massive amount of thoughts and trust you are smart enough to figure it out. Sometimes “DevRel” is one person (who actually holds the title “Developer Advocate” or “DX Engineer”), sometimes “Devrel” is an industry. If this upsets you, please read something else.

Bottom Line Up Front

Alt Text

Stop looking: Your North Star metric is “monthly active developers”. If growth is accelerating, good. If growth is constant, fine. If growth is 0% or worse then whatever you are doing isn’t working.

  • Every other company measuring devrel eventually settles on this, so just cut right to the chase. You could try “monthly active clusters” but you’ll want each cluster’s usage to grow as well so you end up indexing on “developers” anyway.
  • However MAD is multi-causal and a lagging indicator so you need leading indicators which you have more direct control over.

What kind of DevRel are you?

Image description

Unhappiness arises when there is an expectations mismatch between what the company wants out of DevRel and what each DevRel is naturally good at. People have natural affinities — you are more likely to do well with community if you already organize meetups, you are more likely to be great at content if you are a regular on the speaker circuit, you are more likely to understand how to plug user gaps with product feedback if you already built one, etc. Don’t judge a fish by its ability to climb trees.

There are three emerging sub-specialities of developer relations: community-focused, content-focused, and product-focused. How you are measured depends on what your strengths are and what your company actually thinks devrel is. We’ll explore each in turn, but first some context:

  • A lot of companies will proclaim that their approach to developer relations is different from developer evangelism because it is a “two way street”, meaning you put product in front of people but also bring people’s opinions back into product.
    • The reality is often the ratio of this two way street traffic is 99% outbound and 1% inbound, because the product/engineering orgs haven’t set aside any bandwidth for “shadow PM-ing” from devrel and all of devrel’s metrics are outbound focused. Totally understandable, but could be better.
  • If you are in “devrel” but you walk like marketing, talk like marketing, and are measured like marketing, you are in marketing. In fact, you are very expensive affiliate marketing.
    • If you measure devrel by views, the established industry metric is “cost per mille” aka cost per thousand people reached. Indicative numbers: a typical developer content creator’s CPM is $5, a normal devrel’s CPM is $50-$100. In other words you are paying 10-20x more to make content “in-house” than just paying a “professional” to make something about you (often of higher quality, to a bigger audience).
    • Dev content agencies like Draft and Ironhorse exist but are an awkward half solution where they don’t bring audience and don’t have a deep relationship with your product.
  • So for a company devrel investment to make sense, you have to provide some other form of value than raw anonymous reach. This can be: depth, breadth, consistency, access, insight, community, email list building, etc.
    • Devrel specifically coexists as a separate discipline from marketing because developers hate marketing fluff and bottom-up + open source sales is so important for devtool adoption.
    • Career marketers have trouble relating to developers because so much of the conventional wisdom is inverted - you have to sell features, not benefits, and be ready to dive into just-enough-technical-detail over using vague superlatives. However, marketers can still be very useful selling to other parts of the org.

Bad Metrics

I’ve been asked to measure all these in my work: GitHub stars on my demos (yuck), traffic attributed to my Google Analytics UTM tag (yuck yuck), number of badges I could scan at a conference (yuck yuck yuck). All well intentioned but ultimately not meaningful, because they value quantity over quality, breadth over depth, free-and-superficial over paid-and-indicating-serious-interest.

  • Sometimes these are justified as “something is better than nothing”, but once in place, metrics have a strange hold on the imagination: I’ve seriously had a CTO carelessly reject my genuine idea out of hand because “it doesn’t help OKRs”, the same OKRs we previously agreed should not describe all that we do.

I agree with Amir Shevat that we should ”do the right things over the easy to measure things“:

  • People are too flippant about using NPS. NPS is easily gamed and rating product on a 1-10 scale is meaningless to developers who know how this works. Try the Sean Ellis question instead: “How would you feel if you could no longer [use the product/be part of the community/read this content]?” and use it as a way to segment/understand your audience rather than being satisfied just keeping the NPS number forever high.

  • You are hereby banned from suggesting to A/B test anything if you do not have the traffic or the infrastructure to easily implement an A/B test like 90% of devtools startups. A tight user feedback loop beats anonymous data. You find out much more just showing a preview to users 1:1 or in small groups than keeping them at arms length.

  • Will Klein suggests “How quickly new releases are adopted” for OSS devrel. This sounds good until you realize that upgrade adoption is just a proxy for active usage and the curves pretty much all look like this and no matter what you do, you can only budge them a few %:

    Alt Text

Community-focused Developer Advocacy

People are waking up to the lasting power of community and now Technical Community Builder is the Hottest New Job in Tech.

Community metrics that I like (pick 1-3):

  • Number of active members in Slack/Discord
  • Number of weekly new topics in Discourse/StackOverflow/GitHub Discussions
  • Number of user contributions (whether it is PRs, questions, answers, blogposts, meetup talks, etc)
  • Number of Orbit level 1 users
  • Number of events and number of attendees
  • Number of engaged “superusers”

Things to look out for:

  • Beware the optics of regularly asking your users to do free work for you, and measuring yourself by metrics that are at best only loosely correlated with your community building work.
    • For example, most company “communities” are really just support forums, and increased support load is not always good nor does it have anything to do with your stewardship of it. Also beware of using devrel in place of a proper support team.
    • You need to give people a cause to rally around other than getting help. Having great content is often the “minimal viable community”.
    • A “big tent” serves your community over the company — invite your competitors to your event or conference, present together or let them do friendly trash talk, whatever makes your community an actual place where people get together to learn about your topic is good for you. This is especially effective when attempting Category Creation.
  • “True” community metrics are extremely hard to measure. For example, one definition of a good community is that people expect their association with your project/product to outlast their current employment. The natural frequency for measuring this spans years but no metrics system is this patient.
  • Events are underrated for fostering company community. To paraphrase Marshall McLuhan, forums and chats are “cold”, events are “hot” (EDIT: I wrote more on this idea in Community Heat). From company office hours, to social hours, to full-on conferences and hackathons, to user-organized meetups. Measure attendance and frequency of these events. I am dead serious about this — we had a Principal DevRel whose primary job was to help organize our 3 conferences (together with Marketing) and their associated community. If he did nothing else but that, he would already have been worth his weight in gold to us.
    • Live hackathons are appropriate if you have good docs, short time to value, fast iteration times (<1min to change and redeploy) and people can get to a tangible wow moment that wins them the hackathon (particularly in a multi-vendor hackathon). So you should develop a sense of what kinds of things win hackathons — anything realtime (because a live demo can include the audience), anything phone based (everyone has a phone), anything AI based (people like seeing machines act like people), anything visual design. Remote/async hackathons may be a better option if your product has a longer time to value. Work with Major League Hacking, DevTo, or FreeCodeCamp, or collab/sponsor with YouTubers in your niche.
  • Having an engaged and growing superuser community, e.g. AWS Heroes, Java Champions, GitHub Stars and Stripe Community Experts, can help reward your most ardent advocates with access and status. A small group of highly engaged fans can make a community feel more alive than a large group of passive users.
  • If your core product is open source, how much community involvement do you really want? Runa Capital has a definition of Active Contributors, and the big dogs go up to 200 regular contributors a year. This is a lot of external community to manage - is this something your engineering/product team is equipped/prepared/willing to handle?
  • Bonus Community questions:
    • Does devrel or marketing handle the swag program? Do you tie it to open source like Gatsby? Do you build your swag/CRM with your product as a demo or is that Not-Invented-Here syndrome?
    • How can you measure community-generated content and community-organized meetups? Devrel scales by how it enables users to tell (even brag to) each other about their usage. You can borrow ideas from non-devtool “user community” efforts like Notion.
    • How do you bring your community closer in touch with your engineers? Kelsey Hightower ran empathy sessions where the Kubernetes team were just expected to use Kubernetes and failed. Obviously these interactions are valuable but how do you measure?
    • How can you help your users hire each other? At Temporal we set up a careers page for our users. Unfortunately most of these attempts will be too sporadic to seriously measure.

Content-focused Developer Advocacy

Content is the bread and butter of DevRel. If you have no idea where to start I recommend producing at least 1 piece of content on your company a week and just experimenting until you find your groove. Rewriting your blog once a year is fun, but not everyone has put in the reps to learn to regularly produce interesting content.

Content metrics that I like (in rough order):

  • Number of Newsletter subscribers
  • Number of YouTube subscribers
  • Number of Twitter follows
  • Number of Workshop completions
  • Number of Conference/meetup appearances
  • SEO Domain Authority

Nuances to consider:

  • As a content creator you have the choice between TOFU, MOFU, and BOFU content:
    • Top of Funnel - never heard of you (Awareness)
    • Middle of Funnel - comparing you to others and learning basic features/concepts (Evaluation)
    • Bottom of Funnel - deciding to buy and put you into production (Conversion)
    • If you only measure website traffic then you naturally incentivize DevRel to create clickbait TOFU that doesn’t have to convert at all and alienate your biggest fans.
  • Most companies create and measure the blogposts as their main metric and the newsletter is the afterthought. If you are building a company you should see this is ass backwards. Newsletter signup (incentivized by great company blogposts and updates) keeps you honest. Most marketers and professional creators view 1 email subscriber to be worth between 100-1000 social media followers. Even though many developers don’t like giving their email and you’ll need other ways to reach them, you are not exempt from universal rules of media.
  • In fact, the more seriously you view yourself as ”building a media-company-within-a-company” rather than “doing some content marketing”, the better you will do.
  • Keep in mind the half life of your content — how long until a piece of content gets half of the views it will ever get in its lifetime. Twitter’s half-life is hours, YouTube’s half-life is months, blog is years. Create accordingly.
    • You can test things on more ephemeral media before developing them further on more permanent media.
    • Most managers think they are helping by establishing content calendars. I have never seen a DevRel operation stick to these beyond month 1. Turns out it is much easier to say you will do things than to actually do things. Ultimately you will create content because you are inspired to, not because some calendar said so. Keep a list of topics that you want to cover and reguarly pick them off based on subjective interest vs user need. Exception for YouTube - the recommendation algorithm does reward weekly output. See Dan Luu on best practices for engineering blogs.
  • Community sizing: There are between 5-10 million developers on YouTube (my estimate). There are between 1-2 million developers on Twitter (my estimate). There are about 6 million developers on Hacker News. Each have their own pet topics and sub communities. Consider/gauge/prioritize accordingly.
  • Default to Consistency: If you are new to professional content creator life, best way to get started is just default to consistency. That said, 10x content has incredible half-life when you have the right idea.
  • Workshops are extremely underrated forms of content for achieving depth. Instead of 5,000 people watching your 5 minute video and then disappearing forever, you could have 50 people go through your 3-5 hour workshop and put it in production at work.
    • Record it and put it online - you’re not going to get huge views on a 3-5 hour video that is so company specific, but for the people who prefer fastforwarding thru video or desperately need to solve some problem, it is hugely appreciated.
    • Most workshops can also be converted into self guided text based versions: see AWS Workshops like Amplify, ECS, and EKS, Google Colabs, GitHub Learning Labs.
  • Conference appearances are probably overrated by most companies, by the simple fact that most developers don’t go to conferences. Companies have historically overspent on devrel travel budgets if you just take into account attendee reach. However, millions of developers do watch good conference videos after the fact. You can abstractly view conference expenses as production costs for social proof and a really good YouTube video (on a channel you don’t own). It’s well worth producing 1-2 great conference talks a year at high profile events, and sometimes to get there, you need to practice and iterate at 5-10 smaller venues, but anything beyond that probably has diminished returns compared to anything else you could be doing.
  • Meetups are a different matter: high touch, low intensity. Being able to give the same elevator pitch over and over again at different meetups can be great for capturing the attention of a small set of developers. Use their questions and feedback to improve your elevator pitch (2 word, 1 sentence, 2 minute, 7 minute, 25 minute, 55 minute versions) as an infinite game. Raw # of meetup appearances is probably a good enough metric here.
  • I agree with Steph Smith that most companies focus too much on social channels, with the twist that most developers discover solutions more by hearing about them from friends and thoughtleaders than by searching for them. Auth0 and Digital Ocean famously prioritized an SEO based approach, however your devtool/brand should be general enough for this to work.
  • Bonus “Content” things to consider:
    • who owns example repos and code samples and keeps them up to date?
    • are you creating content because your docs aren’t clear enough? are you creating docs because your product isn’t intuitive? All content ultimately has a half-life — sometimes you have to go downstream to fix root cause instead of measuring someone on how well they apply bandaids.
    • how can you encourage real-engineer-content and user-generated content? Devrel scales by how it enables others to create content, not just by having a sole monopoly on content.

Product-focused Developer Advocacy

Product metrics that I like:

  • Number of launch day users, and positive launch day mentions
  • Number of prioritized user issues from DevRel
  • Number of monthly users of integrations/tooling
  • the Sean Ellis question (over NPS) for integrations/tooling managed by devrel
  • Gustaf Alstromer’s PMF measure: Value metric, and % retention at ideal recurring frequency
  • Something that measures Developer Exceptions, I haven’t worked it out yet

Devrels can provide tremendous value in the lead up to any product launch by beta testing (either personally or with users), or creating eyecatching/inspiring demos, blogposts, and videos for launch day/annual conference.

  • Sometimes the best product feedback can be negative: “Hey, you need to postpone this launch, you aren’t ready.” Negative feedback is better coming from within than without, but the company culture needs to be receptive to criticism.
  • If you habitually shoot the messenger, however mistaken they are, don’t be surprised that you stop receiving bad news.
  • Yes, people really do hire DevRels, tell them to give product feedback, and then ignore that feedback. This is the norm.

Devrel are also often responsible for maintaining non-core integrations and helper tooling.

  • Examples: Netlify has an entire Integrations Engineering team. Currently it just works on Next.js integrations, but it could also own, for example, the VS Code extension. In the past I helped build out Netlify Dev and react-netlify-identity as part of this function. Sourcegraph’s Integrations are not core product, but help adoption under their 7 stage SDLC framework. Popular quick start tooling like Docker Compose and Helm Charts also fall under this function.
  • A more involved version of this bleeds into “Solutions Engineering” where you help out with customer-specific custom demos and integration work. For large enough customers and small enough startups this is fine, but the bulk of the DevRel effort should focus on work that scales.

Most devrels spend most of their time advocating TO developers than FOR them. The user-to-product feedback loop is the most underdeveloped element of developer advocacy right now basically because it is unclear who owns this function between Devrel and PM.

  • It can be like grooming a “top 3 user asks” list like Amir Shevat did at Slack and Twitch
  • or it can be building prototypes and “hacky MVPs” to validate ideas that you then hand off to engineering
  • or it can be collecting data on what messaging resonates with developers based on how they explain it to others
  • all of which is really hard to measure

Conclusion

Ultimately I hope for more intellectual honesty within the industry that acknowledges the complex tradeoffs:

  • between the need for communicating value and quantifying progress vs the dehumanizing effect of metrics on what is ultimately an extremely human endeavor
  • between lagging metrics that are more meaningful but where you have less control, and leading metrics where you have more control but little accountability, and the primary link between leading and lagging is “merely” anecdotal and qualitative but ultimately your job
  • where content is a personal, creative process which needs a culture that accepts risk taking and failure, hot streaks and lulls, where consistency, quality, and scope are valued differently by different segments of your audience, and too many backseat drivers guarantees death of personality and ownership.

I’ll leave you with one anecdote: When I first hear about a technology I keep it on a mental backburner simply to see if it sticks around and makes progress. I typically end up only trying out a technology at least one year after first hearing about it and seeing consistent progress. It’s not just me; many CTOs and VP Engs recommend a multiyear adoption path especially for core tech. Traditional marketing advice says people have to hear about you at least 7 times before they decide to buy; for developer tools, Matt Biilmann from Netlify says it’s more like 14.

Try fitting that into your quarterly OKR review cycles.

If you are interested in more DevRel thoughts, podcasts, and books, check the DX Circle guide for more!

Addendum

Tagged in: #dx #devrel #content

Leave a reaction if you liked this post! 🧡
Loading comments...
Webmentions
Loading...

Subscribe to the newsletter

Join >10,000 subscribers getting occasional updates on new posts and projects!

I also write an AI newsletter and a DevRel/DevTools newsletter.

Latest Posts

Search and see all content