Monday, August 20, 2012

The grand "Cadence" experiment

It all started innocently enough. A simple idea, turned into a simple post on the mailing list. This idea eventually led the ubuntu QA community to perform an experiment for this cycle, which has been dubbed "cadence testing".

Now, before I manage to confuse everyone with this "cadence testing" term, let's define cadence testing. Scratch that, let's just give a simple definition of what was intended by original idea. If you want the whole story read the thread. heh. I'll be waiting (hint it's LONG!).

Cadence testing was intended to introduce regularity into testing. If the development release could be "stable" everyday (which was the grand experiment during the precise cycle), could we not also test to ensure that things were good all throughout the release? If the everyday images and archive were now to the quality of previous release's milestones, could we just eliminate the milestone idea and go with a calendar schedule for testing? Thus, a proposal was made test every 2 weeks, whether or not a milestone had been planned, and report the results.

Fast forward 2 months to today. So what happened? Well, I'm happy to report that the QA community despite the confusion more or less met the goal of testing the desktop images every 2 weeks (milestone or not). But what did this achieve? And where are the results?

Let's step back a moment and talk about what we learned by doing this. My comments are specific to the non-milestone cadence testing weeks. First, the development process inside ubuntu is still built around milestones. The daily images during cadence testing weeks were sometimes stable, and sometimes flat out broken by a new change landing from a development team. Second, the tools we used are built around milestone testing as well. The qatracker as well as any qa dashboard or report available doesn't have a good way to track and image health across the cadence week. This meant it was both difficult to test and difficult to see the results of the testing. Finally, the development teams were not expecting results against the daily images, and couldn't follow-up well on any bugs we reported, nor where we able to coordinate well with the release team, as the bugs reported were not available in a summarized or meaningful way.

Now, I'll save the discussion on my ideas of a healthy QA workflow for a later post, but I think we can agree that testing without good result reporting, and without developer follow-up has a limited impact. So does this mean "cadence testing" was a bad idea? No, it was simply poorly executed. The trouble comes in the assumptions listed above.

The archive has not been "stable" everyday, and development teams have have continued development, pausing only as required by the current milestones. In addition, changes, even major ones (like the ubiquity changes landing a few weeks ago, or the nvidia change just this past weekend), are not well communicated. Since they land without little or no warning, we as a QA community are left to react to them, instead of planning and executing them. In this environment, cadence testing makes little sense.

So was the experiment a failure then? In my mind, not at all! In fact, I think the future of ubuntu and QA is to push for complete adoption of this idea, and this experiment confirms the obstacles we will face in getting there. I'll be posting more about what this vision for QA looks like, but I'll leave you with a few thoughts until then.

In my mind, QA should enhance and improve developers, testers, and users lives and workflows. Our work is critical to the success of ubuntu. I would like to see a future where users receive regular, timely scheduled updates that the folks in the QA community have vetted by working with the development and release teams to deliver focused quality updates. The ideal workflow is more fluid, more agile and yes, it has a cadence.


  1. Hello,

    What needs to be done to include mod_wsgi 3.4 into Ubuntu 12.10?!topic/modwsgi/GN1NTuPu8Mw
    "The intent here is to get this out before feature freeze for Ubuntu
    12.10 which is in a few days. Ubuntu 12.10 will as I understand it
    default to using Python 3.2, but existing mod_wsgi 3.3 will not work
    with Python 3.2, so could end up with situation that there is no
    mod_wsgi on the system by default. "

    1. Rene, this is best worked as a bug report. I would encourage you to use launchpad to do so. If upstream debian has already packaged the newer version, note that in the report. That will make it easier for anyone who is trying to do the work. For more realtime help, see #ubuntu-devel on freenode.

  2. Great article Nicholas. I finally understand the meaning of cadence testing.

    We had a grand idea going, but now we need greater interaction with the developers more than ever. As you have mentioned, our bug reports require more developer attention. I too believe in the potential of the qa team.

    I would like to ask: where can I find a timetable for the cadence testing day?

    1. John, the release interlock has the full schedule including our "cadence days".

      The "community testing" column is intended to incorporate all the cadence testing days. I know that calendar might be a bit hard to understand (there's A LOT going on in that page!), but our next date is September 6th at beta. Do you see the Q-B1? It's intending to mean the community will test quantal beta1 that week. As we move forward with idea, communicating this schedule is also going to be important. Excellent point.