What’s Going On With Language Rankings?

Summarizing a report on the top programming languages of 2023, and the impact LLMs have had:

We’ve been internally discussing how we’re going to address the impact of AI-based code assistants on our language rankings since GitHub released Copilot in October 2021. However, it was when ChatGPT hit the market on November 30, 2022 and went from 0 to 100M users in two months that we started seeing undeniable impacts on our source data.

The first chart is worth the click on its own. StackOverflow hasn't seen so few new questions asked since 2012. The decline looks less sudden than it is, because the number of questions being asked started declining in mid-2021, which is incidentally when GitHub Copilot was first released.

You can argue all you want about the quality of the code that AI tools produce (the results I've seen from a year of using them in earnest has been middling at best), but I can think of no better metric for GitHub to hang its hat on than questions no longer asked to other humans. That seems like incontrovertible evidence people are asking AI tools those questions instead and that they're getting good enough answers that they ultimately decide not to go ask a human.

This decline is dramatic, but let's play the AI skeptic for a moment: it would be interesting to learn what nature of questions Copilot is taking from StackOverflow's "marketshare." Countless questions asked on StackOverflow represent rubber duck pairing, in which the very act of articulating a problem carefully makes the answer become apparent. If 30% of StackOverflow questions are effectively solved in the act of the author being forced to cogently describe a problem in writing, then a 30% decline in StackOverflow questions isn't necessarily evidence that AI is providing good answers—only that people are now using LLM chatbots as their rubber duck pair partners instead of the text area of that particular web forum. (No matter how you slice it, though, less user-generated content and engagement is bad news for StackOverflow's business model, it would seem.)