C&I WIki Portal/OCP Checkbox

IMPORTANT ANNOUNCEMENT

ANNOUNCEMENT There are now two versions of this tool available. The old version, known as OCP Checkbox will sunset at some point in the future. The new version, based on a new harness called Plainbox will take it's place. In the interim, both versions will be available to allow for continued Certification Testing activities while the new version is beta tested, tweaked and made ready for General availability. The following is an estimated outline of the transition process. There are no dates attached to these as each stage could take a variable amount of time.

1. opencompute-certification available in the ppa [DONE]
2. opencompute-certification tested [IN PROGRESS]
3. Option to use either OCP Checkbox or opencompute-certification for official testing [TO DO]
4. Require opencompute-certification for official testing (OCP Checkbox still available) [TO DO]
5. OCP Checkbox Sunests, opencpmpute-certification the only available toolset [TO DO]

OCP Checkbox

Checkbox is a test harness used to execute test cases for OCP Ready and OCP Certification testing. It's a command line driven tool written in Python. OCP checkbox is capable of executing test scripts written in a variety of languages (Python, Bash, Perl, C, etc). As of OCP Summit 2015 in San Jose, OCP Checkbox has reached End of Life. It is currently in essentially non-maintained mode though bug fixes will be applied in emergencies. It is being replaced by a new tool based on Plainbox, which will be described below. At some point in the future, OCP Checkbox will be removed entirely from the Open Compute Tools PPA.

OCP Certification

opencompute-certification is the package name of the next generation Open Compute Certification Test tool. It is based on a tool called Plainbox, developed by the Hardware Certification team at Canonical. Essentially, it is a drop-in replacement for OCP Checkbox and the end user should not notice much change.

End User Differences

Mostly, the end user will notice that the menuing system is better, the options for picking tests and suites are improved, as is job handling, runtime output, logging and results output.

Developer Differences

The Test Developer will see some more difference, but again, the difference should not be overwhelming.

• Plainbox honors the same job definition format that Checkbox does. It adds several new definition items that are entirely optional
• Plainbox honors the same whitelist format. It also includes better parsing and handling of regexs in whitelists. Additionally, the long list of "resource" jobs has been replaced with a single job, making whitelist development a bit easier.
• Plainbox still runs anything you ask it too, be it Python, C, C++, Bash, Perl, Go, etc). Anything that needs to be compiled is compiled per architecture during packaging so the end user will never see a difference in that respect.
• Plainbox provides better output. XML, HTML and XLSX is available
• Plainbox adds better built in unit testing ability (This is not currently active in the codebase, but can be added in)
• The package hierarchy is different and will be explained below in more detail.

Origins

Checkbox began as a script called hwtest and was written by people at Canonical. As the initial script grew into a more functional test harness, it was renamed "Checkbox" and has been the testing tool used for Ubuntu certification for many years. The upstream checkbox project is fully open sourced as is the OCP branch of Checkbox. Anyone can contribute code to the tool at any time.

Checkbox (and it's next generation) is an extensible test harness that is capable of running tests that are in pretty much any language, interpreted or compiled, and even running external test suites like Phoronix, NGINX, etc.

opencompute-certification is a new package that is responsible for providing the test launcher and whitelists and certification specific job files and scripts that are not already a part of the plainbox toolset from Canonical. This next generation test harness is a complete, ground-up replacement for the old Checkbox, designed from the ground up to be modular, extensible, useful, and adaptable. It is under very active development and because of the new process for maintaining packaging for OCP Certification, testers should always have the latest bug fixes made available.

About the Tools

As of Ubuntu 14.04 LTS, Checkbox has been deprecated in favor of a new generation test harness called Plainbox. Because of this, the last version of OCP Checkbox is 1.17.4. Going forward, this code is in Maintenance/Bug Fix only mode and will not receive any major updates. OCP Checkbox is still perfectly valid for testing use but will be replaced in the future by a plainbox based test tool. The difference to the tester will be minor, though for the developer, the difference will be great.

The development focus going forward is on the new Plainbox based test tool also stored on Launchpad. There are currently two packages necessary:

opencompute-certification

Provides the launcher and ensures the necessary package dependencies are installed

plainbox-provider-opencompute-testing

Provides OCP Specific whitelists and test scripts, job definitions and other data for testing

plainbox-provider-checkbox

Provides the generic test scripts and job descriptions. This is the exact same package used by Canonical Hardware Certification, and thus is considered an upstream package for opencompute-certification.

plainbox-provider-resource-generic

Provides resource jobs and scripts and libraries necessary to ascertain system hardware inventory and system specific data. This too is the exact same package used by Canonical Hardware Certification and thus is considered an upstream package.

checkbox-ng

The underlying test harness based on Plainbox. This is also the exact package used by Canonical Certification.

The design relies heavily on the existing, current Canonical packages because it is far easier to maintain just the provider and launcher packages necessary for OCP while reusing existing, well maintained tools underneath. It avoids having to maintain an active fork and further avoids issues with forking the fork causing an irreconcilable delta when updates and bug fixes are necessary.

For rare occasions where a fork IS necessary, it can be more gracefully handled.

Additionally, the design allows the easy ability to spin off new providers or launchers as necessary to create custom tailored test packages for each OCP Project or Platform. (Imagine opencompute-certification-winterfell and opencompute-certification-leopard and opencompute-storage-certification-knox)

This will change slightly as development progresses.

Developer level documentation can be found here:

CheckBox-NG

PlainBox

Keep in mind that the documentation is developer focused, not user focused, and is also actively developed and thus could change in the future.

Testers

Installation

Installing is pretty easy. You will need a couple things to get started:

• A SUT (Server Under Test). This is the system you are going to be testing.
• A second machine on the same LAN segment running iperf in server mode
• The SUT will need a method to access the internet for package installation

The OCP Certification packages are built for Ubuntu 14.04, 14.10 and 15.04. They are based on the new OCP Certification Suite tool which is in turn based on the new Checkbox-NG tool from Canonical.

To install:

• Install Ubuntu 14.04, 14.10 or 15.04 to the SUT
• Once Installation is complete and the SUT is rebooted, log in as the non-privileged user.
• Once logged in:
  

$ sudo add-apt-repository ppa:opencompute-developers/ocp-certification-tools-ppa
$ sudo add-apt-repository ppa:hardware-certification/public
$ sudo apt-get update
$ sudo apt-get install opencompute-certification

And that will install all the necessary packages.

Configuring and running the Tool

Edit the file /etc/xdg/opencompute-testing.conf

Uncomment the following four lines:

#TEST_TARGET_FTP = your-ftp-server.example.com
#TEST_USER = anonymous
#TEST_PASS = password
#TEST_TARGET_IPERF =  your-iperf-server.example.com

Edit the TEST_TARGET_IPERF line replacing your-iperf-server.example.com with the IP address (Or FQDN if applicable) of your iperf server

Now simply issue the command:

$ opencompute-certification

and follow the prompts.

Suite Selection

After the welcome screen, you will see the Suite Selection screen. This will list all available whitelists for testing. You must select at least one (The "local" whitelist is selected by default) but you may also select more than one.

Selecting multiple whitelists will cause opencompute-certification to create a superset of the tests in all selected whitelists, properly ordered according to dependency, job type and whitelist order.

Test Selection

Once you have selected the suite or suites, you will see a test selection screen where you can review or edit the tests to be run. For Certification, you should not de-select any tests.

After Testing

Once testing is complete, you will see a test summary on the console as well as indications on where to find the result output files. You will be provided output files in HTML, XML and XLSX.

Developers

Developers are welcome to participate. The test tools, both the OCP Specific packages and the Canonical Certification packages are completely Open Source software under the GPL v3.

How to Participate

There are a number of ways you can participate in making this tool better for OCP.

Join the dev team

There is a team set up on Launchpad for people who wish to contribute:

https://launchpad.net/~opencompute-developers

We hang out on Freenode IRC in the #opencompute-ci channel

Code!

You can always submit code to the project.

The OCP Certification source code can be found here: https://code.launchpad.net/~opencompute-developers/opencompute/opencompute-testing

The Canonical Certification source code here: https://code.launchpad.net/checkbox

It is stored on launchpad and uses bzr as it's control system: http://bazaar.canonical.com/en/

Coding can be anything from fixing bugs (if any) to assisting with the plainbox port to writing new test scripts and jobs. The latter is most important as we want to ensure that the OCP Ready and OCP Certified specs are thoroughly tested and test cases are represented in the test tools.

Specifically, help is appreciated in upstreaming individual jobs and whitelists and other files stored in ITRI's github location. The goal is to move all that code upstream into the OCP Certification code base and maintain it there. This will provide a single location for ALL OCP Certification testing tools, and eliminate some of the current complexity of installing tools from one place then overlaying additional tools from other places.

Docs

We can always use new documentation, be it user guides, development guides, wiki pages, etc. Currently, there is limited documentation available, and this is more akin to generic documentation and development documentation. We could really use user guides from an OCP point of view that explains how to perform testing, how to install the tools, how to USE the tools as well as documentation on how to write test cases and mapping existing test cases to OCP Ready and OCP Certified test cases in the specs.

Additionally, documentation on building alternate packaging (RPM) and installing/running the tools on OSs other than Ubuntu Server would be useful.

Testing

Feel free to install and run Checkbox. It runs well on Ubuntu and should run on Debian. It will also run on CentOS, RHEL and other Linuxes, but the test scripts themselves will need porting as they rely on Ubuntuisms in some cases. It is packaged in .deb only so any additional formats will need to be created (.rpm, for instance).

Bug Reporting

Please feel free to file bugs on problems encountered or requests for enhancements:

https://bugs.launchpad.net/opencompute/checkbox

Questions?

Primarily, we hang out on the opencompute-ci mailing list and IRC Channel #ocp-ci on Freenode.

Feel free to ping Jeff Lane (bladernr, bladernr_ on freenode) in #ocp-ci or via E-Mail