Friday, November 21, 2014

Virtual Hugs of appreciation!

Because I was asleep at the wheel (err, keyboard) yesterday I failed to express my appreciation for some folks. It's a day for hugging! And I missed it!

I gave everyone a shoutout on social media, but since planet looks best overrun with thank you posts, I shall blog it as well!

Thank you to:

David Planella for being the rock that has anchored the team.
Leo Arias for being super awesome and making testing what it is today on all the core apps.
Carla Sella for working tirelessly on many many different things in the years I've known her. She never gives up (even when I've tried too!), and has many successes to her name for that reason.
Nekhelesh Ramananthan for always being willing to let clock app be the guinea pig
Elfy, for rocking the manual tests project. Seriously awesome work. Everytime you use the tracker, just know elfy has been a part of making that testcase happen.
Jean-Baptiste Lallement and Martin Pitt for making some of my many wishes come true over the years with quality community efforts. Autopkgtest is but one of these.

And many more. Plus some I've forgotten. I can't give hugs to everyone, but I'm willing to try!

To everyone in the ubuntu community, thanks for making ubuntu the wonderful community it is!

Wednesday, November 5, 2014

Ubuntu Online Summit: Vivid Edition

Ubuntu Online Summit is once again upon us. This is a community event by and for the community. It's all encompassing and intends to cover a wide range of topics. You don't need to be a developer, project lead, member of a team, or even a member of ubuntu to join and participate. The only requirement is your passion for ubuntu and desire to discuss about it's future with others.

The dates are set as November 12-November 14th from 1400 UTC to 2000 UTC. I am once again privileged to be a track lead for the users track. In my opinion, this is the best track as it's the one the largest number of us within the community can easily feel a part of (just don't like Michael, David, Daniel or Alan know I said that). Do you use ubuntu? Awesome, this is the track for you.

What I'm asking for is sessions. Have an idea for a session? Please propose it! Everything you need to know about participating can be found here. If you've attended things like ubuntu open week or a classroom session in the past, all of those types of sessions are welcome and encouraged.

"The focus of the Users track is to highlight ways to get the most out of Ubuntu, on your laptop, your phone or your server. From detailed how-to sessions, to tips and tricks, and more, this track can provide something for everybody, regardless of skill level."

Regardless of your desire to contribute a session, I would encourage everyone to take a look at the schedule as it evolves and considering joining in sessions they find interesting. In addition, it's not yet too late to offer up ideas for sessions (though I would encourage you to find a way to host the session).

Ready to propose a session? Checkout this page and feel free to ping me or any track lead for help. Don't forget to register to attend and check out the currently scheduled sessions!

Tuesday, November 4, 2014

Autopilot Feature Primer

Autopilot celebrated it's 2 year anniversary as an independent project this summer. During that time it has developed into a useful tool for testing application UI's for gtk and qt toolkits. Support has also been extended to MIR as well as phablet devices.

With this in mind, I thought it would be useful to bring attention to some new and under-used features of autopilot, along with providing a brief explanation of some companion tools you might find useful. Thus I present to you, an autopilot primer. Let's talk through some new features shall we?

Python3 Support
Autopilot started as a python2 tool but has since migrated to python3 and you should too! For now the entire source tree remains python2 compatible, but you really should migrate your tests to python3. You'll notice the autopilot3 binary in newer releases which should be used to run autopilot with python3.

Scenario Support
Scenarios are a wonderful way to keep your tests simple and easy to read while allowing you to test with many different inputs. In short, you might need to test several edge cases as part of your acceptance testing. This is most easily accomplished by keeping the test itself generic and utilizing a scenario to vary your inputs. You can check out more information on scenarios specific to autopilot in the autopilot documentation.

Screenshots / Video
Autopilot allows you to get a video recording of a test failure. To make sure autopilot records failures, install recordmydesktop and pass the -r argument to your autopilot3 run command. However, at the moment this requires X so for now it doesn't work with the MIR backend (which things like the ubuntu phone utilize). Fortunately a screenshot at the point of failure when combined with the log is generally sufficient to solving your issue. Getting those screenshots brings us to subunit support.

Subunit Support
By default autopilot generates the test output and logs straight to your console in a text format. However autopilot also supports outputting to xml and subunit. Subunit support is what I would like to highlight for a few reasons. When you set the output format as subunit, you get a few niceties. One of which is an easier to grok format for tools, and the other is screenshots of the application when failures occur. To get a subunit stream, pass the -f subunit argument to your autopilot3 run command. You will want to also pass -O with a filename to save the output to a file as the subunit stream contains binary data.

Test Result Viewer
So, with this subunit test results file, how can you enjoy all of it's goodness? Enter trv, a simple python ui that will let organize the test run in an easy to view manner, including screenshots. The tool is the creation of Thomi Richards who describes it as a quick hack (:p), and has a youtube video demonstrating it's use. It's perfect for viewing the subunit stream and visualizing your test results. For now, it's not packaged but can be easily obtained via launchpad. Grab it with bzr branch lp:trv.

autopilot3 vis
The vis tool allows you to visually interact with the introspection tree after launching an application using autopilot launch. What this means is you can visualize the application in the same way autopilot does at runtime, with live tree updates. It lets you see what autopilot sees, allowing you to interactively build your testcase.

I'll refer you the official tutorial for more information, as well as a youtube video by yours truly. It's from a livestream, but covers what you need to know. autopilot3 vis also contains a search box, and a highlight tool which didn't exist in the orignal version, so it's even nicer now than before. Give it a whirl!

I talked about this utility when I covered the test runners for autopilot. Still I would be remiss if I didn't mention it again. Everything I said in the test runners for autopilot post still applies, so go have a quick read about how to use the tool if you need more information. This tool enables you to easily run autopilot tests on the desktop in a nested xserver. What that means to you as a test author is that you can run tests without giving up your desktop session. No more waiting for autopilot to hand back control of your mouse after a test. If you are writing tests, you should be using this tool along with autopilot vis mentioned above during your test writing process.

Per test timeout
Although we all only write "good tests", sometimes you may find your test misbehaves. When this happens the test may even not exit cleanly or get stuck in a loop. The result is autopilot and the system under test to wait forever for the test to finish. To prevent a rouge test from killing a test suite run, autopilot is introducing support for per-test timeouts. This has landed in vivid; you'll need version 1.5.0+15.04.20141031-0ubuntu1 or later. To use the feature, add the --test-timeout argument to autopilot run and give is a timeout in seconds.

In conclusion
Autopilot has gotten many new features along the way, and these are but a few of the most recent and important ones. I hope this helps you take another look at what autopilot might be able to help you test. Happy Testing!

Monday, October 27, 2014

Sprinting in DC: Friday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

Friday brings an end to an exciting week, and the faces of myself and those around me reflect the discussions, excitement, fun and lack of sleep this week has entailed.

The first session of the day involved hanging out with the QA team while they heard feedback from various teams on issues with quality and process within there project. Always fun to hear about what causes different teams the most issues when it comes to testing.

Next I spent some time interviewing a couple folks for publishing later. In my case I interviewed Thomi from the QA team and Zoltan from the SDK team about the work going on within there teams and how the last cycle went. The team as a whole has been conducting interviews all week. Look for these interviews to appear on youtube in the coming weeks.

Thursday night while having a look through a book store, I came across an ad for ubuntu in Linux Voice magazine. It made me smile. The dream of running ubuntu on all my devices is becoming closer every day.

I'd like to thank all the community core app developers who joined us this week. Thanks for hanging out with us, providing feedback, and most of all for the creating the wonderful apps we have for the ubuntu phone. Your work has helped shaped the device and turn it into what it is today.

Looking back over the schedule there were sessions I wish I had been able to attend, and it was wonderful catching up with everyone. Sadly my flight home prevented me from attending the closing session and presumably getting a summary of some of these sessions. I can say I was delighted to talk and interact with the unity8 team on the next steps for unity8 on the desktop. I trust next cycle we as a community can do more around testing there work.

As I head to the airport for home, it's time to celebrate the release of utopic unicorn!

Friday, October 24, 2014

Sprinting in DC: Thursday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

Today started with some UOS planning which is happening in a couple short weeks. If you haven't yet put it on your calendar, please do so! And plan to not only attend, but consider submitting a session as well. The users track might be just the place for your session. Session topics can be about anything ubuntu related you might want to share or discuss with others.

As the week has progressed I've enjoyed getting to know the core apps developers better. Today we met with all of them to hear feedback on how the projects have been going. Lots of good discussion was had discussing how things like meetings and reviews work, individual project needs and actions that could be taken to improve all of the projects. It's wonderful to have everyone in the same place and able to talk.

After lunch the QA team discussed manual testing and proposed utilizing moztrap for some of the manual testing they are undertaking as part of the CI process for ubuntu touch images. While it is too early to say what implications this will have on manual testing from a community perspective, I'm happy to see the conversation has begun around the current issues facing manual tests. I'm also happy someone else is willing to be a guinea pig for changes like this! For image testing, the qatracker has served us well and will continue to do so, but I hope in the future we can improve the experience. In fact, we have done work in this area recently, and would love to hear from anyone who wants to help improve the qatracker experience. So, whether or not a migration to moztrap occurs at some point, the future looks bright.

The core app developers also got a chance to both get and receive feedback from the SDK and design teams. The deep dives into applications like calendar were very much appreciated and I expect those suggestions will filter into the applications in the near future. As usual the core apps developers came prepared with suggestions and grievances for the SDK team, as well as praises for things done well.

Finally to end the day, we discussed developer mode on the device. Rather than talk about the history of how it was implemented, let me share with you the future. Rather than locking adb access via a password, we'll utilize certificates. The password based solution already will ensure your locked device isn't vulnerable to nefarious humans who might want to connect and steal your data or reflash your phone. However, things like passwordless sudo will be possible with using certificates. In addition if security is the bane of your existence, you will be able to enable developer mode without setting a password at all.

Whew, today was very full!

Thursday, October 23, 2014

Sprinting in DC: Wednesday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

To kick off the day, I led a session on something that has been wreaking havoc for application test writers within the core apps -- environment setup. In theory, setting up the environment to run your test should be easy. In practice, I've found it increasingly difficult. The music, calendar, clock, reminders, file manager and other teams have all been quite affected by this and the canonical QA team and myself have all pitched in to help, but struggled as well. In short, a test should be easy to launch, be well behaved and not delete any user data, and be easy to setup and feed test data into for the test process. I'm happy to report that the idea of a permanent solution has been reached. Now we must implement it of course, but the result should be drastically easier and more reliable test setup for you the test author.

I also had the chance to list some grievances for application developers with the QA team. We spoke about wanting to expand the documentation on testing and specifically targeted the need to create better templates in the ubuntu sdk for new projects. When you start a new project you should have well functioning tests, and we should teach you about how to run them too!

Just before lunch the community core app developers were able to discuss post-RTM plans and features. A review of the apps was undertaken and some desire for new designs or features were discussed. Terminal is being rebuilt to be more aligned with upstream. Music is currently undergoing a re-design which is coming along great. Calculator is anxious to get some design love. Reminders potential for offline notetaking as well as potential name changes were all discussed. Overall, an amazing accomplishment by all the developers!

After lunch, I spent time confirming the fix for a longstanding bug within autopilot. The merge proposal for fixing this bug has been simmering all summer and it's time to get it fixed. The current test suites for calendar and clock have been impacted by this and have already had regressions occur that could have been caught had tests been able to be written for this area. Having myself, the autopilot team, and the calendar developers in one place made fixing this possible.

To end the day, I spent some time attending sessions for changes to CI and learning more about the coming changes to CI within ubuntu. In summary the news is wonderful. CI will test using autopkgtest, and all of ubuntu will come under this umbrella -- phone, desktop, everything. If it's a package and it has tests, we will do all of the autopkgtest goodness currently being done for the distro.

The evening closed with a bit of fun provided by a game making hackathon using bacon2d and the hilariously horrible "Turkish Star Wars". We could always use more games in the ubuntu app store, and I hear there might even still be a pioneers t-shirt or two left if you get it in early!

Wednesday, October 22, 2014

Sprinting in DC: Tuesday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

On Tuesday I was finally able to sit down with the team and plan our week. In addition I was able to plan some of the work I had in mind with the community folks working on the core apps. Being obsessed with testing, my primary goals this week are centered around quality. Namely I want to make it easier for developers to write tests. Asking them to write tests is much easier when it's easy to do so. Fortunately, I think (hope?) all of the community core apps developers recognize the benefits to tests and thus are motivated to drive maturity into the testing story.

I'm also keen to work on the manual testing story. The community is imperative in helping test images for not only ubuntu, but also all of it's flavors. Seriously, you should say thank you to those folks helping make sure your install of ubuntu works well. They are busy this week helping make sure utopic is as good as it can be. Rock on image testers! But the tools and process used weigh on my mind, and I'm keen to chat later in the week with the canonical QA team and get there feedback.

During the day I attended sessions regarding changes and tweaks to the CI process. For core apps developers, errors in jenkins should be easier to replicate after these changes. CI will be moving to utilizing adt-run (autopkgtest) for there test execution (and you should too!). They will also provide the exact commands used to run the test. That means you can easily duplicate the results on the dashboard locally and fix the issues found. No more works on my box excuses!

I also met the team responsible for the application store and gave them feedback on the application submission process. Submitting apps is already so simple, but even more cool things are happening on this front.

The end of the evening found us shuffling into cab's for a team dinner. We had a long table of folks eating Italian food and getting to know each other better.

After dinner, I pressured a few folks into having some dessert and ordered a sorbet for myself. After receiving no less than 4 fruit sorbets due to a misunderstanding, I began carving the fruits and sending plates of sorbet down the table. My testcase failed however when the plates all came back :-(