How I evaluate submissions for talks at OpenStack Summits

As a track chair, I look for content that will be informative for participants live at the event. I also look for entertaining content, well delivered, clear and usable also as a complement to written documentation, something to be enjoyed also after the event on the video channel.

I’ve been a track chair for OpenStack Summit tracks many times and recent discussions on the community mailing list made me decide to publish the principle that have guided my decisions.

When I’m evaluating a talk I look at:

  • the title
  • the abstract
  • the bio of the speaker

A good title conveys immediately what the talk is about and some idea of the argument or area of discussion. A good abstract expands on the title describing the thesis, the argument and the conclusion, includes also a reason for an identified audience to go see it at the conference, or later on youtube. The bio of the speaker needs to reinforce all of the above, explain why the speaker is the best person to deliver the talk.

By looking only at title, abstract and bio, I discard bad titles, bad abstracts and speakers with a bad bio and who are not known to me, to linkedin, to slideshare and to google in general. Usually that leaves me with twice as many talks as slots available in the track.

The next criteria I use to discard other talks are: does this talk fit with the overall objective of the Summit? Does it fit with the specific objective of the track? For example, the objective of the Tokyo summit is to focus on application developers and containers. And for the How To Contribute track, the colleagues track chairs decided to focus on content that we didn’t hear before or needed update and to give precedence to more regional-specific content. This pass usually identifies a couple of clear winners and 2-3 tracks with very little debate.

Now it’s time for group deliberation to finish the selection, starting from those that gained the majority of support and asking for a dissenting opinion. All of our arguments were around title, abstract and bio of the presenters, those provided more than enough information to make a decision and we never had to use anything else.

We never looked at the public votes because those are easily gamed and I think it would be unfair at this point to give priority to someone only because they work for an organization that can promote talks during the voting process. Each candidate needs to be evaluated based on what they bring to the Summits, not on their marketing teams.

My argument against using the results votes of the public voting process to judge proposals is exactly that, at best, those votes don’t provide any value: bad titles and bad abstracts proposed by non-involved individuals will gather very few votes anyway, but those are very easy to discard for track chairs anyway. So no value here. But once the decent tracks remain under consideration, looking at the public votes result may skew track chair decisions towards the usual people that speak every time, with lots of twitter followers or the ones working for companies with well organized marketing departments.

To me the votes are the result of a popularity contest and if used for anything, they dramatically damage the minorities that are not on twitter, the people who are shy by nature and those working for companies that don’t have a strong social media presence (or don’t use it at all). In fact, I’d argue that the results of the votes should be even hidden in the track chair UI.

I always considered the voting process as a marketing tool for the event, a community ritual, a celebration of the OpenStack community as a whole and not something that the selection committee should use. I find looking at votes extremely unfair to the submitters and diminishing of the selection committee’s role, too. IMO a good committee should evaluate based on quality of content relative to the objectives for that specific summit (overall focus, location), and totally ignore the popularity of their proposers (or their employees).

I wish we could have the data to analyze and demonstrate if the votes are the expression of a larger community or, as I suspect, are just the result of twitter reactions and push from marketing efforts of few companies. My gut feeling is that with thousands of proposals nobody has the possibility to read and vote them all. So the proposals voted are necessarily only a fraction.  Also, of all the people rotating around OpenStack, few of them vote. This means that
few people find some talk proposals. Which talks are more popular? I notice how well organized are companies like Rackspace, Red Hat, Mirantis, IBM, Tesora and few others blogging about their proposals when time comes. Maybe if the data show that the talks from employees of those companies are the most voted probably we can infer that you either push your talks or your talk doesn’t get voted.

Am I suggesting getting rid of voting all together?  No, that’s not what I’m advocating. I think the voting process is valuable for the summit as a whole and for the community as a whole. It’s a ritual, it’s a celebration, it’s a preparation to the event, it’s a collective, fun activity that we repeat every six months. The voting process is not broken and needs no fixing: it’s great. Only the results IMO are useless for selecting good content at the Summit.

Post-summit summary of OpenStack Paris

Seven straight full time working days, starting from 9am Saturday and Sunday for the second edition of OpenStack Upstream Training to the feedback session on Friday evening at 6pm are finally over and I’m starting to catch up.

I spoke on Monday, highlighting some of the findings of my research on how to change organizations so they can better contribute to OpenStack, later lead with Rob, Sean and Allison the creation of the Product working group (more on this later). The double-length session dedicated to addressing the growth pain in Neutron has removed the “if” and left the question to “how and when do we empower drivers and plugin developers to take their fate in their own hands”.

I think this has been one of the most productive summit, I’m leaving Paris with the feeling we keep on improving our governance models and processes to better serve our ecosystem. Despite the criticisms, this community keeps on changing and adapts to new opportunities and threats like nothing I’ve seen before. I’m all charged up and ready for the six months of intense work leading up to Vancouver!

Wrap-up post OpenStack Summit in Hong Kong

Back from busy days in one of the most exciting cities I’ve ever visited, I needed some time to put thougths back in order, recover from jet lag and deal with my irremediably broken WordPress installation (will probably blog about this later, too). Hong Kong was a blast in many aspects. The Summit itself started with lions dancing at the sound of drums in front of over 3,000 people:

And then we saw that Bejing is the first city by total number of contributors to OpenStack in the whole world:

There has been an incredible growth inside and around OpenStack, the project is growing fast. Growth is good, it’s what we wanted so we have plenty of reasons to celebrate. The second edition of the User Survey brought us more insights about usage of OpenStack around the world. We announced our first OpenStack Ambassadors, people who will help the OpenStack Community team get closer to many communities around the world. It was great to meet personally more women from the Outreach Program for Women of OpenStack. we’ve been doing this for over a year now, it’s a thing.

IMG_20131111_204451Growth also brings challenges and some of them were evident in some of the conversations had at the Summit, around it and after. A few signals I caught during and around the Design Summit sessions highlighted that we may need to start taking new steps to reinforce the culture of collaboration inside the project. The challenges highlighted go from lack of reviewers (not core reviewers, just developers who pay attention and help others), PTLs getting overloaded, the high traffic on the Development mailing list (which leads to loss of information), the increasing number of questions on Ask OpenStack with no interactions (no up/down votes, comments, etc) and little engagement in its Chinese version, the challenges inside the Internationalization Team with processes and tools. We’ve also heard of a very few Design sessions where it was too hard to have a productive discussion because of one or two uncollaborative people.

Since we’re getting so many new developers in the project we’re probably getting to the point where we can’t assume they are accustomed to contributing upstream first. The founders and first members of OpenStack all had a brilliant pedigree of open source contributions and collaboration. New members of the OpenStack Foundation may need some help to succeed. I enjoyed the session Getting Your Blueprint Accepted Quicker: the VPNaaS Use Case so much that I’m proposing the Upstream University training as an official program at the OpenStack Foundation to help new members. I’ll write more about this in the future.

The next six months will continue to be super exciting and full of things to do. If you have missed Hong Kong go watch the recordings of the sessions  keep watching this space for more news.


Back from San Diego OpenStack Summit

My experience at the OpenStack Summit in San Diego has been really good. I have received lots of positive comments about this configuration, merging the Design Summit with the Conference. Despite the high amount of people it seems that things went well. I’m waiting to gather more details about the remote participation to the Design Summit with WebEx, I’ll report more about that soon.

I presented one session at the conference, lead two sessions on the Design Summit track and facilitated the meeting of the APEC group. Especially on the last day I missed the integration of the summit’s agenda from into my personal calendar applications (phone and desktop) because I ended up overbooking myself a couple of times. Thankfully Monty Taylor covered for me.

The ‘Community Dashboard‘ that I presented with zAgile was received with enthusiastic comments: the crowd cheered ‘ship it’ when I asked them what they thought of the demo. I’ve talked with Sanjiva and Andrew after the presentation, we should have an early beta out there by the end of the year.

More needs to be done in order to improve the community resources: IRC channels are not owned by the Foundation, some services depend by one person only (the main website and etherpad service, just to make an example). The forums need some love and probably we should have a Q&A system in place. During the sessions Atul Jha from India showed an askbot-powered system that he volunteered to run. In the next weeks I’ll help him go live with it. We discussed also the migration of the General mailing list out of Launchpad: unfortunately I have no news since my last update. The planet needs a better look, if nothing else. And the OpenStack blog needs a better content policy: some people in the room raised some concerns over the abuse of corporate posts on it.

During the discussion on how to track OpenStack’s adoption I was suggested to focus on users’ survey instead of proposing to add some ‘telephone home’ capability to OpenStack’s code (like Mozilla Foundation does with Firefox). I think this is a good idea and I’ll make sure this will become a project of the Foundation in the next months, once we’ll be fully staffed (we’re hiring).

Monty lead the session I proposed about an Integrated Identity System among all OpenStack tools. We went through the improvements we’re working on regarding the CLA workflow and its integration into Gerrit and the Foundation’s membership database. Todd Morey was in the session and we had more ideas on how to make things progress a bit faster (we agreed that hanging out on IRC is a prerequisite for make things happen faster). The future of Launchpad as the main ID system for OpenStack will be decided after Ubuntu Developer Summit: Thierry will spend time with Canonical’s folks there to understand if it will still be able to serve our purposes in the future.

My list of things to do has increased, as it’s expected after the Summit. Thanks everybody for joining.

OpenStack at UDS

What a week! Ubuntu Developer Summit is one of the best meetings I have attended to. Here are a few things that impressed me most.

The infrastructure is amazing! The networking is astonishingly good. I learned that the Ubuntu team bypasses the usually lame the Internet connections provided by the hotels and puts down their own. Ubuntu’s wifi was gratis, easy to join and always on! By comparison, the access provided by the hotel was mostly down and costed $10 per day.

The participation is insane! Not only Canonical employees participate to the summit but also volunteers from around the world. In Orlando there were around 700 people all interested in making Ubuntu the greatest operating system in the world.

OpenStack is everywhere! Three plenary sessions were dedicated to OpenStack and many sessions of the summit had to do with it. Canonical is putting lots of energy into making OpenStack its cloud. Mark Shuttleworth in one of these sessions made it also clear that he wants to provide resources for OpenStack to maintain compatibility Amazon’s API.

Rackspace distributed the coolest t-shirts at the event: we ran out of three full boxes in a few minutes.

The pace of the summit was not as mad as I imagined after looking at the schedule. Even if there are many parallel tracks from 9am to 6pm for five full days, I ended up with plenty of time free to meet people and talk with them. Sharing the same hotel with a big swimming pool and very nice weather probably helped the conversations. I enjoyed also the free buses available to go out in large groups.

All in all, it was great to be there since OpenStack Developer Summit is modelled after UDS.