How not to use OpenStack Community Metrics tools

I have noticed this morning another article/blog post mistakenly trying to extrapolate hard facts about a company’s involvement in OpenStack using one of the reporting tools built for the community. The reporter went to Stackalytics (but it could have gone to Activity Board, it would have been the same) to check if Oracle had made any contribution to the OpenStack codebase. That’s the wrong way to use these tools: numbers regarding companies contributing to OpenStack published daily cannot be trusted.

Both Stackalytics and Activity Board depend on data that is entered voluntarily by the contributors to OpenStack, therefore they cannot be trusted. Stackalytics has a mapping file in its repository that is kept up to date by developers themselves (those that know its existence). Activity Board pulls data straight from the OpenStack Foundation Members database: when you sign up as Member of the Foundation (a precondition to become a developer) you’re asked to enter data about who pays for your contribution. The bylaws of the Foundation also require that you keep that info up to date, but we know as a fact that few people log back in their member profile and even fewer update their affiliation. Therefore we know that the data about the affiliation in all reporting tools is not 100% reliable at any point in time. It’s good enough if you’re looking at the top contributing companies where the volumes are high enough to remain fairly valid despite small percentage errors. But when a reporter goes to check if a total newcomer to the community has submitted any code, that number is very likely to be wrong (and close to zero): the new developers may have not understood what the Affiliation field is and not filled it out (I see a lot of those on a weekly basis) and they’re very unlikely to know about the mapping file in git for Stackalytics.

The data that I trust most (but still not 100%, especially for ‘long tail’ contributors) are the reports published with Bitergia at release time: every OpenStack release we do a lot of manual cleanup of the data in the Foundation database, ask people to update their affiliation, normalize names of companies and circulate the report for comments before making them public. Still, those may contain errors which we track on Launchpad.

As far as I know the reporter didn’t ask the Foundation nor Oracle if anybody could point at actual commits done by Oracle employees and that’s what he should have done.

OpenStack prouds itself for being an open community and I’ve been the first proposer of having a public way to see the various activities inside the project, in real time and including the information about companies, not just individuals. I think we need to discuss how we can provide better data and avoid giving false illusion of precision to casual visitors to these sites.