Meet OpenStack at FOSDEM 2016

The most important gathering of free software and open source enthusiasts in Europe is coming on Jan 30-31, in Brussels and OpenStack will have a table booth there, plus many talks.

I look forward to go there to get the pulse on the open source community regarding the abolition of Safe Harbor provision, and how that impacts users of US-based public clouds, like DreamHost. It’s a complicated issue that I’ve just started looking into. One of the talks I’ll make sure to attend to is Rosario Di Somma’s talk about Magnum in production: he’s been evaluating Magnum for DreamHost and he’ll share the first results of his investigations.

On the OpenStack side of things, Adrien Cunin is the main point of contact and he’s looking for volunteers to preside at the official OpenStack table at FOSDEM. If you have holes in your agenda, please consider adding your name to the etherpad.

Developing apps on OpenStack is still too complicated

Yesterday’s meeting with the OpenStack App Developer Working Group proved that new developers approaching OpenStack enter a system designed to make them fail.

The community seems to be distracted to uncover new problems while the old and known problems are not being addressed. I’m running for a seat on OpenStack Foundation Board: if you care about the developer’s experience, consider voting for me (search your inbox for OpenStack Foundation – 2016 Individual Director Election, you can also change your vote.)

The App Developer Working Group drafted a detailed analysis of the app developer experience with major cloud providers AWS, Azure and Google Compute and compared such experience with Rackspace Cloud. It’s a great piece of work.

The first thing you notice in the report? The team at Intel who ran the analysis chose Rackspace as an ‘OpenStack reference cloud’.  That choice is debatable but during the meeting when we discussed alternatives, it became clear that there is no good choice! There is no vanilla OpenStack implementation when it comes to application developers, they’re all snowflakes (as Randy Bias put it)… All of the public clouds in OpenStack Marketplace have made choices that affect, one way or another, app developers. If we want to assess the whole development experience on OpenStack, we need a different framework.

As an open source community we can’t compare AWS, Azure and Google Compute to either all of OpenStack public clouds or only one. Powering up TryStack to be an app developers playground wouldn’t really work either.

This is a much larger conversation: we need to discuss more on the User Committee mailing list, do less in-person meetings, share intentions online with others before turning them into actions and waste time.

I’m concerned by the lack of focus within the community: when operating OpenStack became a visible issue, the whole community focused on helping operators out. We need to do the same for app developers.

What is an OpenStack Powered Compute?

The OpenStack Technical Committee voted a resolution suggesting to the OpenStack Board to modify the definition of “OpenStack Powered Compute” to include statements such as

An `OpenStack Powered Compute`_ Cloud MUST be able to boot a Linux Guest

This is quite a change in OpenStack DefCore efforts, since as Rob says

The fundamental premise of DefCore is that we can use the development tests for API and behavior validation

DefCore has always been about the OpenStack API and carefully avoided checking the implementation of clouds, leaving enough space for vendors to differentiate their products without harming consumers. An ironic twist of fate is now forcing the whole program to take a stand on implementation, too.

Stating that an OpenStack cloud MUST be able to boot a Linux Guest  is the most controversial and I see why the TC is going into this direction: as a OpenStack user, I expect to be able to upload my images in any OpenStack cloud. Given that Linux is the OS of choice for the vast majority of today’s cloud workloads, booting my Linux image is a must have. It’s a practical choice and one that makes a lot of sense.

The problem is that forcing implementation details by the TC is a slippery slope. Will the TC suggest also that any OpenStack cloud must offer a routable IP or boot a VM in less than 5 seconds or that flavor names are always the same across all OpenStack clouds? Granted, all these requests make sense from the perspective of at least some users: there are already lots of unnecessary complications for putting workloads on OpenStack right now.

The question is if those mandates make sense for OpenStack as a whole and I’m not convinced they do. I tend to lean towards two complimentary positions:

  1. make sure that OpenStack Compute Powered clouds are transparent and their behavior is discoverable. Like the OpenStack Foundation exposes the results of the DefCore API compatibility on the Marketplace, I think DefCore should test the implementation of the clouds, public clouds especially and expose the results. This way a user would know that a cloud passes DefCore tests, runs OpenStack upstream code (as done now) and allows uploading/booting Linux guests, offer IPv6 by default, boot in x seconds, has XYZ flavors, allows custom flavors, etc.
  2. DefCore should consider public clouds, hosted private clouds and distributions as different beasts. To me it makes more sense to expect a public cloud to allow uploading Linux guests than mandate the same for a private clod… even less sense for a distribution. The buying process for these is different and the needs for users are different, too.

Thoughts?

I’m a candidate for OpenStack Foundation Board of Directors: if you like what you read, remember to vote for me at the 2016 elections

Balancing IRC and email, the secret of success for open collaboration

Are seasoned contributors to OpenStack giving too much importance to IRC, and is that creating bad side effects in other areas of the community?

Since I started contributing more to working groups outside of the OpenStack upstream contributor community, I noticed that email is used in a limited way, with a couple of lists having either very low traffic or low signal/noise ratio. The email lists are often used to share calendar notifications for meetings and their agendas with often little to no discussions. Have we insisted so much in holding weekly meetings “like developers do” that now people think that email discussions are less important?

This weekend a couple of posts on Planet OpenStack talk about IRC: one very good source of information on how to use IRC better from Chris Aedo who suggests to stay more on IRC. He argues:

If you are interested in becoming an OpenStack contributor, having a persistent IRC presence might be one of the most important secrets to success. Anyone who spends time in one of the more active project channels will immediately see the value of synchronous communication here where problems are sometimes solved in minutes.

On the other hand of the spectrum Chris Dent who highlights the chaotic nature of IRC. He mentions that too  much IRC leads to:

no opportunity to be truly away, frequent interruptions, an inflated sense of urgency, and a powerful sense of “oh noes, I’m missing what’s happening on IRC!!!!”—but the killer is the way in which it allows, encourages and even enforces the creation and expression of project knowledge within IRC, leaving out people who want or need to participate but are not synchronous with the discussion.

This last point is crucial: not everybody can be online all the time, some of us have to sleep or go see a movie at times. IRC is often mentioned as a blocker to new community members. I advocate using a bouncer only to get personal messages, not to read the backlogs.

While synchronous communication is crucial for distributed collaboration, a sane habit of using email is as important. The OpenStack community has put so much emphasis on using IRC to hold weekly meeting and to resolve the most controversial conversations in real-time that some new comers now think that synchronous communication is more important than async email.

A sane balance of sync-async communication is more crucial to success of open source collaboration, mixing email and IRC (which is what most of OpenStack upstream developers do, by the way.)

All in all, I think there is also a need to teach people how to hold conversations via email, with proper quoting, no top posting and other nice things to do… but also email sorting and filtering client-side, what not to say (hint: “me too” messages are generally considered noise)… a lost art, as much as using IRC.

Why OpenStack should stop tracking its Net Promoter Score

TL;DR because it makes no sense for OpenStack and it provides too much distraction at a time when the Foundation should focus on addressing more specific and actionable concerns tied to its mission “to inform the roadmap based on user’s opinions and deployment choices.

I’d like the OpenStack community to focus on the fundamental issues of OpenStack adoption, ease and speed of development and ease of use by application developers. There is plenty of evidence that such areas need attention and there are already metrics tracking them: adding Net Promoter Score (NPS) is a distraction.

This is part of the reason why I’m running for Individual Director of the OpenStack Board. Read the rest of my candidacy pitch and hit me with any questions, comments or concerns you have over email, twitter, IRC (@reed), phone call, smoke signals … whatever works for you!

So, there are many strong reasons for removing the Net Promoter Score from the survey altogether. The main objection: NPS is used to track customer loyalty to a brand. For typical corporations, they can identify with precision customers and brand, therefore effectively measure such loyalty.

On the other hand, the OpenStack Foundation has many types of customers, the concept of its products is not exactly defined and ultimately OpenStack can’t be considered a brand in the sense of the original HBR article that launched NPS. The User Survey is answered by everyone from cloud operators or sys-admins to OpenStack upstream developers,  app developers / deployers and even a mystery blob called “Other” in a multiple choice answer. The question, “How likely are you to recommend OpenStack to a friend or colleague?” can be interpreted in too many ways to make any sense. Who exactly is the [Promoter|Detractor], who are their friends and colleagues, what exactly is “OpenStack” to them? Did they get some pieces (which ones?) of OpenStack from a vendor? Did they get the tarballs? Are they skilled to use whatever they have purchased or downloaded for free?

I’d argue that the NPS collected in the Survey in its current form has no value whatsoever.

 - Dilbert by Scott Adams
The OpenStack NPS went from #$ to *&

I asked the User Committee to answer these questions: Is the User Committee convinced that an open source project like OpenStack gets any value from tracking this score? Why exactly is that number tracked in the survey, what exactly does the UC want to get from that number, what actionable results are expected from it?

Stay tuned: I will update this blog post as the conversation continues.

Frank Day added to the list:

NPS is an industry standard measure.

He missed my point unfortunately. I didn’t start this conversation to debate if NPS is an industry standard measure. My question is whether it makes sense to track it for OpenStack when: a- OpenStack is not an industry but it’s an open source collaboration, it has no exact definition of its product (what is OpenStack in the context of the user survey? Only the core? the whole big tent? the tarball from upstream? a distribution from $VENDOR?) and finally, it lacks a definition of who its customers are (the survey has only a vague idea).

If we are to drop NPS, it should only be in exchange for some other measure of satisfaction.

He fails to explain why that should be important for a huge open source collaborative project like OpenStack. And also satisfaction of who, doing what is not clear.

Lauren Sell and Heidi Joy Tretheway gave a more thoughtful answer that I suggest to read in full. Some excerpts:

When we analyzed the latest user survey data, we looked at a demographic variable (user role, e.g. app developer), a firmographic variable (e.g. company size), and deployment stage. We learned that overall, there was no significant difference in NPS scores for people who identified with a specific function, such as app developers, or for companies by size. As a result, we didn’t do further data cuts on demographic/firmographic variables. We did learn that people with deployments in production tended to rate OpenStack more highly (NPS of 43 for production, vs 24 for dev/qa and 20 for POC).

One cause for variance is that unfortunately we’re not comparing apples to apples with the trending data in the latest survey report.

Going forward, I think we should focus on deployments as our trend line.

As a next stepthe independent analyst plans to draw up correlations (particularly for low scores) associated with particular technology decisions (e.g. projects or tools) and attitudinal data from the “your thoughts” section (e.g. we might find that firms that value X highly tend to rate OpenStack lowest).

I replied on the list saying that ultimately all this slicing and dicing is not going to tell us more than what we already know anecdotally and by looking at other data (that the community used to collect), suggesting that resources would better be allocated towards 1-1 interviews with survey respondents and other actions towards the community.

Roland Chan sent an update to the list after the meeting and the committee decided to keep analyzing this score. Heidi Joy will work with the data scientist, which means resources that could be better spent are being used to serve a corporate number.

The hard life of OpenStack application developers

I’m starting to feel the pain of developers attempting to write applications on top of OpenStack clouds: it hurts, and I haven’t come anywhere close to doing anything special.

I started following the getting started tutorial for First App Application For OpenStack (faafo) and I gut stuck on step 1, authentication. There are way too many things that are convoluted, confusing and sometimes carefully hidden (or ridiculously displayed) on Horizon that make it too hard to get started developing apps on OpenStack. I started following the Get Started document for python-libcloud:

You need the following information that you can obtain from your cloud provider:

auth URL
user name
password
project ID or name (projects are also known as tenants)
cloud region

Looks simple but it’s not. The auth URL doesn’t include the path say libcloud documentation but in practice every cloud I tried behaves differently. DreamHost and Runabove require not only v2.0/ but also /tokens after the base URL (https://keystone.dream.io/v2.0/tokens) while CityCloud and HPCloud seem to work according to the documentation. Some will throw weird errors if you add /tokens. Libcloud is also confusing regarding support for Keystone API v3 (which Citycloud uses): the official docs don’t mention v3 (bug), but a blog post from libcloud maintainer provides examples on how to use it.

Finding the right projectID or name is also challenging because some clouds don’t show the project ID immediately in the web control panel (and not every cloud I tested runs OpenStack Horizon). Chances are you’ll have to find the RC file, which is fairly easy if the cloud you’re targeting is using Horizon.

Even after I found all the pieces, I haven’t managed to authenticate using libcloud on CityCloud, Rackspace (using openstack provider) and HP. Only with OVH I managed to authenticate and get a list of images with my stock Ubuntu 15.04. I managed to authenticate on DreamHost only on Ubuntu 14.04 and on a clean, updated virtualenv.

It took a lot of trials and errors to get rid of the Method Not Allowed errors and get libcloud.common.types.InvalidCredsError: ‘Invalid credentials with the provider’, which at least is hinting that I have guessed the auth_url for Citycloud and HP. But I have no idea why credentials are not accepted, since the username and password are those I use on the Horizon panel on HP and on on the custom Citycloud panel. My guess is that I haven’t guessed correctly the project_name or tenant_id or whatever it’s called. This is too confusing.

OVH seems to works, but it requires the full url:

auth_url = ‘https://auth.runabove.io/v2.0/tokens

it won’t work without /v2.0/tokens.  So, after spending one day reading docs, trying to guess permutations I started thinking that probably libcloud is not a feasible approach.

I’ve started looking at OpenStack Infra homegrown interoperability library shade instead: authentication went smoothly immediately with DreamHost and HP on my machine. At least that part was easy and I managed to make some progress in my test with that, finally. Hopefully by the end of the day I’ll have the FAAFO running on

The end result is that it shouldn’t be this hard, even for a very modest developer like me, following a well written, copy-and-paste tutorial should not be as confusing. A lot of work needs to be done to make OpenStack clouds friendly to app developers.

So long OpenStack community, see you soon

Let’s call it a “ciao” and not an “addio.”

After almost four years (un)managing the OpenStack community, I have decided to move on and join DreamHost’s team to lead the marketing efforts of the DreamCloud. The past three years and 10 months have been amazing: I’ve joined a community of about 300 upstream contributors and saw it grow under my watch to over 3,600. I knew OpenStack would become huge and influence IT as much as the Linux kernel did and I still think it’s true. I’m really proud to have done my part to make OpenStack as big as it is now.

During these years, I’ve focused on making open source community management more like a system, with proper measurement in place and onboarding programs for new contributors. I believe that open source collaboration is just a different form of engineering and as such it should be measured correctly in order to be better managed. I am particularly proud of the Activity Board, one of the most sophisticated systems to keep track of open collaboration. When I started fiddling with data from the developers’ community, there were only rudimentary examples of open source metrics published by Eclipse Foundation and Symbian Foundation. With the help of researchers from University of Madrid, we built a comprehensive dashboard for weekly tracking of raw numbers and quarterly reports with sophisticated, in-depth analysis. The OpenStack Activity Board may not look as pretty as Stackalytics, but the partnership with its developers makes it possible to tap into the best practices of software engineering metrics. I was lucky enough to find good partners in this journey, and to provide an example that other open source communities have followed, from Wikimedia to Eclipse, Apache and others.

OpenStack Upstream Training is another example of a great partner: I was looking into a training program to teach developers about open source collaboration when I spoke in Hong Kong to an old friend, Loic Dachary. He told me about his experiment and I was immediately sold on the idea. After a trial run in Atlanta, we scaled the training up for Paris, Vancouver, plus community members repeated it in Japan already twice. I’m sure OpenStack Upstream Training will be offered also in Tokyo.

It’s not a secret that I can’t stand online forums and that I consider mailing lists a necessary evil. I setup Ask OpenStack for users hoping to provide a place for them to find answers. It’s working well, with a lot of traffic in the English version and a lot less traffic in Chinese. My original roadmap was to provide more languages but we hit some issues with the software powering it (Askbot) that I hope the infra team and the excellent Marton Kiss can solve rapidly.

On the issue of diversity, both gender and geographic, I’m quite satisfied with the results. I admit that these are hard problems that no single community can solve but each can put a drop in the bucket. I believe the Travel Support Program and constant participation in Outreachy are two such drops that help OpenStack be a welcoming place for people from all over the world and regardless of gender. The Board has also recently formalized a Diversity working group.

Of course I wish I did some things better, faster. I’m sorry I didn’t make the CLA easier for casual and independent contributors: I’m glad to see the Board finally taking steps to improve the situation. I wish also I delivered the OpenStack Groups portal earlier and with more features but the dependency on OpenStackID and other projects with higher priorities delayed it a lot. Hopefully that portal will catch up.

I will miss the people at the OpenStack Foundation: I’ve rarely worked with such a selection of smart, hard workers and fun to be around, too. It’s a huge privilege to work with people you actually want to go out with, talk about life, fun, travel, beers, wines and not work.

When we Italians say “ciao,” it means we’re not saying good bye for long.

So long, OpenStack community, see you around the corner.

OpenStack is as start-up friendly as anything

I read someone complaining about OpenStack being not friendly enough to startups one time too many. Latest post by Rob Hirschfeld 10 ways to make OpenStack more Start-up Friendly made me want to respond. I just can’t stand the Greek chorus of complaints, especially from someone who sits on the board and can actually push for changes where necessary. I promised I’d spend time on his 10 points and here is my take on each of them:

Accept companies will have some closed tech – Many investors believe that companies need proprietary IP. An “open all things” company will have more trouble with investors.

Why is this an OpenStack problem? If the VC industry have a problem with open source then the problem is of open source as a whole or of the VC (depending on your point of view). Maybe customers prefer to buy open source based solutions instead of buying proprietary code from small startups. This is not an OpenStack issue.

Stop scoring commits as community currency – Small companies don’t show up in the OpenStack committer economy because they are 1) small and 2) working on their product upstream ahead of OpenStack upstream code.

Code is the most valuable currency in an open source project, there is absolutely no way OpenStack should be any different. Companies small and big will be contributing compared to their size and code is how a small company doing great work can gain more influence than a large company doing little stuff. If you’re saying that the community shouldn’t put first and foremost the “top ten” charts of companies contributing (like stackalytics does), then I agree with you. The Foundation celebrates individual contributors to the release and only counts company. This is not an OpenStack problem. Maybe it is a stackalytics problem and one of the reasons I prefer to partner with Bitergia for the community dashboard.

Have start-up travel assistance – OpenStack demands a lot of travel and start-ups don’t have the funds to chase the world-wide summits and mid-cycles.

OpenStack has already has addressed this problem with the Travel Support Program. In Vancouver the Foundation spent around $50k to send about 30 people from all over the world, from startups and students. If that’s not enough, I’m sure more money can be raised for that. This is not an issue, there is a solution in place already.

Embrace open projects outside of OpenStack governance – Not all companies want or need that type of governance for their start-up code base.  That does not make them less valuable, it just makes them not ready yet.

I don’t even know where this comes from: is anybody forcing anyone to host code on git.openstack.org/openstack namespace? And isn’t the OpenStack project offering for free its resources to host code in our systems, without imposing governance or rules with git.openstack.org/stackforge? If there are companies interested in getting under the OpenStack governance, it’s their choice and based on my knowledge they choose because they get business value off of it. This is a non-issue.

Stop anointing ecosystem projects as OpenStack projects – Projects that are allowed into OpenStack get to grab to a megaphone even if they have minimal feature sets.

Even if this was a problem, and I don’t think it is, it should work in favor of startups: many of them have nothing to show but good intentions, for which a megaphone is exactly what they want and use. This is a non-issue.

Be language neutral – Python is not the only language and start-ups need to make practical choices based on their objectives, staff and architecture.

Nobody forces anyone to become an OpenStack official project (which requires some level of standardization). It’s a choice. And, in any case, there is a lot of javascript and ruby in OpenStack, with pieces in Go also coming. This is a non-issue.

Have a stable base – start-ups don’t have time to troubleshoot both their own product and OpenStack.  Without core stability, it’s risky to add OpenStack as a product requirement.

This is a tautology also it’s something that is being constantly advocated and keeps on improving.

Focus on interoperability – Start-ups don’t have time evangelize OpenStack.  They need OpenStack to have large base of public and private installs because that creates an addressable market.

So let me get this straight: IBM, EMC, Cisco are scooping up the first waves of startup that tried to build a product on the rudimentary OpenStack. Such big guns have the clout to create that large addressable market, lending their credibility to OpenStack as a whole. They also pay good checks to the OpenStack Foundation to power its awesome marketing machine.  This is good for startups: they ride on a wave created with someone else’s money.

Limit big companies from making big pre-announcements – Start-ups primary advantage is being a first/fast mover.  When OpenStack members make announcements of intention (generally without substance) it damages the market for start-ups.  Normally corporate announcements are just noise but they are given credibility when they appear to come from the community.

Yeah, right, like you can really put on a rule against vaporware. It’s the way this market has worked and will continue to work this way. Startups, in any field have to learn how to live with it. This is not an OpenStack issue.

Reduce the contribution tax and patch backlog – Start-ups must seek the path of least friction.  If needed OpenStack code changes require a lot of work and time then they are unlikely to look for less expensive alternatives.

Here I guess you’re talking about contributions to existing OpenStack projects, like Nova, Neutron and the like. If you think that some company can innovate fast on Nova while keeping the interoperability and stable release you talk about above then you’ve managed to confuse me. How realistically can you have some startup rip Nova apart and replace parts of it (all of it?) with the greatest big thing and keep the thousands of users out there with a happy upgradeable path, interoperability and stability? This is just impossible to reconcile. Startups cannot innovate on something that is mature and in production. It would be like asking Apache HTTPD to be something else. Guess what: nginx happened outside of Apache and it’s only natural. As a parallel to OpenStack, if someone comes up with a better Neutron, written in Go or Rust, and wide support I’m ready to bet it will be admitted in the big tent rapidly.

Let me tell you why I think that OpenStack is at least as friendly as any other business environment, and maybe more:

  1. Corporate sponsorship for startups is low, a lot lower than for big corporations. And there are ways to lower the admission price even further (just ask).
  2. The ecosystem is now so huge that cool startups innovating on the edges can get exposed to potential customers, investors and buyers very quickly (the megaphone works).
  3. The ‘cloud’ space is so rapidly changing that the big guys cannot keep up and count on startups to do the risky experimentation. There are lots of big companies in OpenStack, I suppose there are opportunities to find good contacts.
  4. The OpenStack Summits have a whole track dedicated to startups, with talks about funding, business, strategies, acquisitions and more.

And finally a reminder: all startups operate in extremely unfriendly environments, most startups fail for various reasons.

The recent exits of companies that innovated on the edges when OpenStack was held together with shoestring and spit as glue are to me a confirmation that the edges where innovation can happen are just moving outside of Nova and Neutron. It’s just normal and to be expected with the maturity of the project.

Let’s celebrate and be happy for Piston and Blue Box and the others. Startups will always be welcome and will always find a good home in and around OpenStack.

How is your OpenStack team organized?

I’ve been collecting a lot of good insights talking to directors and managers about how their companies are organized to contribute to OpenStack. For geographic reasons I have mostly gathered comments from people between San Francisco and Silicon Valley and I’d like to expand the research.

I’m especially interested in learning about internal processes, system of incentives, things that impact performance evaluation for engineers contributing to OpenStack.

To expand the research I’m asking the OpenStack community to fill in this brief survey or contact me directly (stefano openstack.org) for a quick interview.