Friday, October 24, 2014

Sprinting in DC: Thursday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

Today started with some UOS planning which is happening in a couple short weeks. If you haven't yet put it on your calendar, please do so! And plan to not only attend, but consider submitting a session as well. The users track might be just the place for your session. Session topics can be about anything ubuntu related you might want to share or discuss with others.

As the week has progressed I've enjoyed getting to know the core apps developers better. Today we met with all of them to hear feedback on how the projects have been going. Lots of good discussion was had discussing how things like meetings and reviews work, individual project needs and actions that could be taken to improve all of the projects. It's wonderful to have everyone in the same place and able to talk.

After lunch the QA team discussed manual testing and proposed utilizing moztrap for some of the manual testing they are undertaking as part of the CI process for ubuntu touch images. While it is too early to say what implications this will have on manual testing from a community perspective, I'm happy to see the conversation has begun around the current issues facing manual tests. I'm also happy someone else is willing to be a guinea pig for changes like this! For image testing, the qatracker has served us well and will continue to do so, but I hope in the future we can improve the experience. In fact, we have done work in this area recently, and would love to hear from anyone who wants to help improve the qatracker experience. So, whether or not a migration to moztrap occurs at some point, the future looks bright.

The core app developers also got a chance to both get and receive feedback from the SDK and design teams. The deep dives into applications like calendar were very much appreciated and I expect those suggestions will filter into the applications in the near future. As usual the core apps developers came prepared with suggestions and grievances for the SDK team, as well as praises for things done well.

Finally to end the day, we discussed developer mode on the device. Rather than talk about the history of how it was implemented, let me share with you the future. Rather than locking adb access via a password, we'll utilize certificates. The password based solution already will ensure your locked device isn't vulnerable to nefarious humans who might want to connect and steal your data or reflash your phone. However, things like passwordless sudo will be possible with using certificates. In addition if security is the bane of your existence, you will be able to enable developer mode without setting a password at all.

Whew, today was very full!

Thursday, October 23, 2014

Sprinting in DC: Wednesday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

To kick off the day, I led a session on something that has been wreaking havoc for application test writers within the core apps -- environment setup. In theory, setting up the environment to run your test should be easy. In practice, I've found it increasingly difficult. The music, calendar, clock, reminders, file manager and other teams have all been quite affected by this and the canonical QA team and myself have all pitched in to help, but struggled as well. In short, a test should be easy to launch, be well behaved and not delete any user data, and be easy to setup and feed test data into for the test process. I'm happy to report that the idea of a permanent solution has been reached. Now we must implement it of course, but the result should be drastically easier and more reliable test setup for you the test author.

I also had the chance to list some grievances for application developers with the QA team. We spoke about wanting to expand the documentation on testing and specifically targeted the need to create better templates in the ubuntu sdk for new projects. When you start a new project you should have well functioning tests, and we should teach you about how to run them too!



Just before lunch the community core app developers were able to discuss post-RTM plans and features. A review of the apps was undertaken and some desire for new designs or features were discussed. Terminal is being rebuilt to be more aligned with upstream. Music is currently undergoing a re-design which is coming along great. Calculator is anxious to get some design love. Reminders potential for offline notetaking as well as potential name changes were all discussed. Overall, an amazing accomplishment by all the developers!

After lunch, I spent time confirming the fix for a longstanding bug within autopilot. The merge proposal for fixing this bug has been simmering all summer and it's time to get it fixed. The current test suites for calendar and clock have been impacted by this and have already had regressions occur that could have been caught had tests been able to be written for this area. Having myself, the autopilot team, and the calendar developers in one place made fixing this possible.

To end the day, I spent some time attending sessions for changes to CI and learning more about the coming changes to CI within ubuntu. In summary the news is wonderful. CI will test using autopkgtest, and all of ubuntu will come under this umbrella -- phone, desktop, everything. If it's a package and it has tests, we will do all of the autopkgtest goodness currently being done for the distro.

The evening closed with a bit of fun provided by a game making hackathon using bacon2d and the hilariously horrible "Turkish Star Wars". We could always use more games in the ubuntu app store, and I hear there might even still be a pioneers t-shirt or two left if you get it in early!

Wednesday, October 22, 2014

Sprinting in DC: Tuesday

This week, my team and I are sprinting with many of the core app developers and other folks inside of Ubuntu Engineering. Each day I'm attempting to give you a glimpse of what's happening.

On Tuesday I was finally able to sit down with the team and plan our week. In addition I was able to plan some of the work I had in mind with the community folks working on the core apps. Being obsessed with testing, my primary goals this week are centered around quality. Namely I want to make it easier for developers to write tests. Asking them to write tests is much easier when it's easy to do so. Fortunately, I think (hope?) all of the community core apps developers recognize the benefits to tests and thus are motivated to drive maturity into the testing story.

I'm also keen to work on the manual testing story. The community is imperative in helping test images for not only ubuntu, but also all of it's flavors. Seriously, you should say thank you to those folks helping make sure your install of ubuntu works well. They are busy this week helping make sure utopic is as good as it can be. Rock on image testers! But the tools and process used weigh on my mind, and I'm keen to chat later in the week with the canonical QA team and get there feedback.

During the day I attended sessions regarding changes and tweaks to the CI process. For core apps developers, errors in jenkins should be easier to replicate after these changes. CI will be moving to utilizing adt-run (autopkgtest) for there test execution (and you should too!). They will also provide the exact commands used to run the test. That means you can easily duplicate the results on the dashboard locally and fix the issues found. No more works on my box excuses!

I also met the team responsible for the application store and gave them feedback on the application submission process. Submitting apps is already so simple, but even more cool things are happening on this front.

The end of the evening found us shuffling into cab's for a team dinner. We had a long table of folks eating Italian food and getting to know each other better.


After dinner, I pressured a few folks into having some dessert and ordered a sorbet for myself. After receiving no less than 4 fruit sorbets due to a misunderstanding, I began carving the fruits and sending plates of sorbet down the table. My testcase failed however when the plates all came back :-(



Sprinting in DC: Monday

This week, my team and I are sprinting in Washington DC with many of the core app developers and other folks inside of Ubuntu Engineering. Sprints are always busy, but the work tends to be a mix of social and technical. I get to assign names (IRC nicknames mostly) to faces as well as get to know my co-workers and other community members better.

I thought it might be useful to give writeups each day of what's going on, at least from my perspective during the sprint. I won't yammer on too much about quality and instead bring you pictures of what you really want. And some of this too. Whoops, here's one.

Pictures of people taking pictures . . .
Monday was the first day of the sprint, and also the day of my arrival! Personally I'm busy at home during this week, so it's tough to get away. That said, I can't imagine being anywhere else for the week. The sprints are a wonderful source of respite for everyone.

Monday itself consisted of making sure everything is ready for the week, planning events, and icebreakers. In typical fashion, an opening plenary set the bar for the week with notes about the progress being made on the phone as well as the future of the desktop. Lots of meetings and a few blurry jet lagged hours later, everyone was ready to sit for a bit and have some non-technical conversation!

Fortunately for us there was an event planned to meet both our social and hunger needs. After being split randomly into teams of bugs (love the play on quality), we played a bit of trivia. After each round teams were scored not only on the correct response, but also how quickly they responded. The questions varied from the obscure to fun bits about ubuntu. The final round centered around Canonical itself which was fun trip down memory lane to remember.

As I crawled into bed I still had the wonderfully cheesy announcer playing trivia questions in my head.


Thursday, October 16, 2014

Final testing for Utopic

The final images of what will become utopic are here! Yes, in just one short week utopic unicorn will be released into the world. Celebrate this exciting release and be among the first to run utopic by helping us test!

We need your help and test results, both positive and negative. Please head over to the milestone on the isotracker, select your favorite flavor, and perform the needed tests against the images.

If you've never submitted test results for the iso tracker, check out the handy links on top of the isotracker page detailing how to perform an image test, as well as a little about how the qatracker itself works. If you still aren't sure or get stuck, feel free to contact the qa community or myself for help.

Thank you for helping to make ubuntu better! Happy Testing!

Tuesday, September 23, 2014

Final Beta testing for Utopic

Can you believe final beta is here for utopic already? Where has the summer gone? The milestone and images are already prepared for the final beta testing. This is the first round of image testing for ubuntu this cycle. A final image will also be tested next month, but now is the time to try out the image on your system. Be sure to report any bugs you may find. This will help ensure there is time to fix them before the release images.

To help make sure the final utopic image is in good shape, we need your help and test results! Please, head over to the milestone on the isotracker, select your favorite flavor and perform the needed tests against the images.

If you've never submitted test results for the iso tracker, check out the handy links on top of the isotracker page detailing how to perform an image test, as well as a little about how the qatracker itself works. If you still aren't sure or get stuck, feel free to contact the qa community or myself for help. Happy Testing!

Thursday, September 4, 2014

Autopilot Test Runners

In my last next post, I discussed will discuss notable autopilot features and talk about how autopilot has matured since it became an independent project.

In the meantime I would be remiss if I didn't also talk about the different test runners commonly used with autopilot tests. In addition to the autopilot binary which can be executed to run the tests, different tools have cropped up to make running tests easier.

autopilot-sandbox-run
This tool ships with autopilot itself and was developed as a way to run autopilot test suites on your desktop in a sane manner. Run the autopilot3-sandbox-run command with --help to see all the options available. By default, the tests will run in an Xvfb server, all completely behind the scenes with the results being reported to you upon completion. This is a great way to run tests with no interference on your desktop. If you are a visual person like me, you may instead wish to pass -X to enable the test runs to occur in a Xephyr window allowing you to see what's happening, but still retaining control of your mouse and keyboard.

I need this tool!
sudo apt-get install python3-autopilot

I want to run tests on my desktop without losing control of my mouse!
autopilot3-sandbox-run my_testsuite_name

I want to run tests on my desktop without losing control of my mouse, but I still want to see what's happening!
autopilot3-sandbox-run -X my_testsuite_name

Autopkgtest
Autopkgtest was developed as a means to automatically test Debian packages, "as-installed". Recently support was added to also test click packages and to run on phablet devices. Autopkgtest will take care of dependencies, setting up autopilot, and unlocking the device. You can literally plug in a device and wait for the results. You should really checkout the README pages, including those on running tests. That said, here's a quick primer on running tests using autopkgtest.

I need this tool!
sudo apt-get install autopkgtest
If you are on trusty, grab and install the utopic deb from here.

I want to run tests for a click package installed on my device!
Awesome. This one is simple. Connect the device and then run:
adt-run --click my.click.name --- ssh -s adb

For example,
adt-run --click com.ubuntu.music --- ssh -s adb

will run the tests for the installed version of the music app on your device. You don't need to do anything else. For the curious, this works by reading the manifest file all click packages have. Read more here.

I want to run the tests I wrote/modified against an installed click package!
For this you need to also pass your local folder containing the tests. You will also want to make sure you installed the new version of the click package if needed.

adt-run my-folder/ --click my.click.name --- ssh -s adb

Autopkgtest can also run in a lxc container, QEMU, a chroot, and other fun targets. In the examples above, I passed --- ssh -s adb as the target, instructing autopkgtest to use ssh and adb and thus run the tests on a connected phablet device. If you want to run autopilot tests on a phablet device, I recommend using autopkgtest as it handles everything for you.

phablet-test-run
This tool is part of the greater phablet-tools package. It was originally developed as an easy way to execute tests on your phablet device. Note however that copying the tests and any dependencies to the phablet device is left to you. The phablet-tools package provides some other useful utilities to help you with this (checkout phablet-click-test-setup for example).

I need this tool!
sudo apt-get install phablet-tools

I want to run the tests I wrote/modified against an installed click package!
First copy the tests to the device. You can use the ubuntu sdk or click-buddy for this, or even do it manually via adb. Then run phablet-test-run. It takes the same arguments as autopilot itself.

phablet-test-run -v my_testsuite

Note the tools looks for the testsuite and any dependencies of the testsuite inside the /home/phablet/autopilot folder. It's up to you to make sure everything that is needed to run your tests are located there or else it will fail.

other ways
There are of course other possible test runners that wrap around autopilot to make executing tests easier. Perhaps you've written a script yourself. Just remember at the end of the day the autopilot binary will be running the tests. It simply needs to be able to find the testsuite and all of it's dependencies in order to run. For this reason, don't be afraid to execute autopilot3 and run the tests yourself. Happy test runs!