C&I Wiki Portal/ToolsDevelopmentGuideline

Overview

The official repository for the source code to OCP Certification and OCP Ready is Launchpad. There are a number of projects hosted there but the most important one is opencompute-testing.

The opencompute-testing project contains all the code for generating the OCP Certification and OCP Ready testing tools. It is the code that creates the Plainbox Provider and Launcher Packages, hosting the job descriptions that are OCP Specific, scripts and whitelists. Package builds occur automatically once per day if the code base has been updated.

Contacts

If you have ANY questions at all about the code base, contributing, or anything else related, contact Jeff Lane For anything related to Open Compute Certification (including code questions):

• Mailing List: http://lists.opencompute.org/mailman/listinfo/opencompute-ci
• IRC: Freenode IRC Servers #ocp-ci

Basics

This section will cover some of the basics for contributing to the code for these tools. This is a general overview, more specific information can be found on line at the links below.

Bazaar

About Bazaar

Bazaar is a version control system that helps you track project history over time and to collaborate easily with others. Whether you're a single developer, a co-located team or a community of developers scattered across the world, Bazaar scales and adapts to meet your needs. Part of the GNU Project, Bazaar is free software sponsored by Canonical

You can learn more at http://bazaar.canonical.com/en/

Obtaining Bazaar

To install Bazaar on a debian or ubuntu system: sudo apt-get install bazaar

For other Operating Systems, including Windows, Red Hat, SuSE, OSX and more, please see http://wiki.bazaar.canonical.com/Download

Learning Bazaar

Bazaar is easy to learn. There is an excellent 5 Minute Tutorial that should get you started. Beyond that, bzr has a fairly well developed man page and in-program help system.

Please read and do the 5 Minute Tutorial now, which will teach you the basics like creating branches, logging into LP, pushing and merging.

Trees, Branches, Trunks?

The basic top level of a codebase is called the Trunk. The Trunk is the mainline code that all branches are a part of. Branches are localized copies of Trunk. Branches are where you make changes for submission and merge consideration. Once accepted, Branches are merged into Trunk and become part of the mainline codebase.

Basic Commands

• The most basic command is: bzr
• To get a local copy of a project: bzr branch lp:opencompute/opencompute-testing
• To review a change summary: bzr status
• To add new files that are not yet managed: bzr add path/to/filename
  • Note that /path/to/filename needs to be a path INSIDE your branch. When adding files not in the branch, you need to copy those files into the branch before adding them.
• To see a diff of changes: bzr diff path/to/filename
• To commit your changes to a new revision bzr commit -m "Some descriptive summary of your changes"
• To push your changes to launchpad in preparation for a Merge Request: bzr push lp:~<Your Launchpad Username>/opencompute/<Your Branch Name>
  • for example: bzr push lp:~bladernr/opencompute/ocp-testing-job-updates

There are many MANY more commands but those are the ones you'll likely use most frequently.

Learning More

• http://doc.bazaar.canonical.com/en/
• http://wiki.bazaar.canonical.com/Tutorials
• https://yade-dem.org/wiki/Quick_Bazaar_tutorial
• http://wiki.bazaar.canonical.com/Tutorials

Practices

Languages to use

Main Tooling

Plainbox, Checkbox, the providers and such are all written in Python3.

Job Files and Whitelists

Job Files are the same as they were in Checkbox, that is RFC822 compliant text files. See any file in the jobs/ directory for examples. Whitelists are also simply text files, HOWEVER, you can also use regular expressions in job naming to pattern match jobs.

Test Scripts

These are scripts that perform actual testing. For example, a storage IO test, or a network performance test. Scripts can be written in pretty much anything that is compiled or has a valid interpreter installed on the System Under Test. The following are just a few examples of the languages you can write test scripts in:

• Python3
• Bash
• Ruby
• Perl

And some more complex ones

• C/C++
• Go

'Please note that compiled languages will need special consideration and handling in order to get the packages to build properly. Packages are built for AMD64 and i386 for now. We have the capability to also compile them for OpenPower, ARM and ARM64, but that will require some porting work.

In general, we advise the following for test scripts:

• Python 3 for pretty much everything
• BASH for one-liners and short simple scripts. Generally, if it starts feeling complex, it's time to re-write in Python3
• If you MUST use compiled languages, C but this is exceedingly rare and should only be done when something easier to maintain is not appropriate

Translations

We currently do not have any Translations for the tooling, nor are there plans to enable this.

Style Guide

The general rule is Python code should adhere to Python Foundation's PEP-8. There are several tools to help you check your code for adherence such as pep8 and Pylint and many others.

Project Hierarchy

opencompute-certification

What this provides

This is the trunk for the Open Compute Project's certification tooling. It can be found on Launchpad: https://code.launchpad.net/opencompute/opencompute-testing

This code is used to build the launcher packages:

• opencompute-ready
• opencompute-certification

and the provider package:

• plainbox-provider-opencompute-certification

What's a Provider?

A Provider is a package that provides the bits necessary for Plainbox/Checkbox-NG to perform testing. These files include job descriptions, whitelists, test scripts and wrappers, helpers, custom libraries and anything else that you may need.

What's a Launcher?

A Launcher is a package that does two things.

1. It provides the actual program launcher (the command you issue to start the testing) such as opencompute-certification
2. It ensures the lower-level packages and dependencies are installed

checkbox

What this provides

This is the main project for Canonical's certification tool. This provides the back-end and underlying support structure as well as some of the general test scripts and jobs. The following packages make up our use of Checkbox, in order from top to bottom.

plainbox-provider-checkbox

This package provides the actual checkbox test tools used by Canonical Certification. Many tests in OCP Certification and OCP Ready are based on the jobs, scripts and data in this provider.

plainbox-provider-resource-generic

This package provides system resource jobs and libraries. These are used at run-time to gather system information and provide that data to CheckboxNG for job generation, to the results files for system hardware inventory, and to provide the validation that allows certain jobs to run.

To explain that last statement, one job of this is to determine if certain hardware exists, and if that hardware exists, then the jobs that test that hardware are configured and executed. For example, there is a USB resource that determines if a given motherboard has a USB controller. If the USB Controller is detected by the USB Resource, then the USB test jobs will be run. If the USB Resource does NOT detect a USB controller, then the USB jobs will be skipped.

python3-checkbox-support

This package is an infrastructure package that provides several python3 support libraries that Checkbox needs to run.

checkbox-ng

This package is the back-end. It provides Checkbox-NG, the Plainbox based test harness that replaces the old Checkbox tool. It provides the internals that organize and run all the various test cases, performs all the parsing and heavy work of Certification Testing.

Documentation?

For the most part, Documentation for the OCP Tools can be found here on the OCP Certification Wiki. For Checkbox, documentation can be found:

• http://checkbox.readthedocs.org/en/latest/
• http://plainbox.readthedocs.org/en/latest/

As CheckboxNG and Plainbox are under active development, this documentation is constantly evolving and improving.

How can I contribute to this?

There are several ways to contribute. For the most part, you'll likely want to contribute to test scripts, jobs and whitelist changes, so you will mostly be interested in the opencompute-testing project.

To get started, follow the bits above to branch the project (lp:opencompute/opencompute-testing) and start hacking!

If you'd like to contribute to the Checkbox project, please see the Questions? section below.

Putting it all together

So now that you know about the various packages that make up the OCP Certification and OCP Ready test tools, let's install them so you can try them out to see what they do.

Installation of Packages

First, we need to install the PPAs. Because this is based on Canonical's actual certification tooling instead of a fork, we need to add two PPAs:

sudo apt-add-repository ppa:opencompute-developers/ocp-certification-tools-ppa
sudo apt-add-repository ppa:hardware-certification/public

Next, we need to update the apt cache:

sudo apt update

And finally, install the packages:

sudo apt-get install opencompute-certification

Interaction

After you have done the above, you will see a long list of packages that need to be installed. These all provide testing tools, libraries and other things necessary for Certification testing. The number of packages will vary depending on how you installed the OS onto the SUT, what version is being used, what tool is used (Ready vs Certification) and so forth.

Hit Y and the installation will begin.

Once installation is complete, to check it out qickly, simply run the command opencompute-certification or opencompute-ready

To do proper testing, there is some configuration and other things that need to be performed first, but this will get you an idea of how the tool works and what it looks like.

Obtaining the Code

Locations

For now, there are multiple locations to find testing tools. In the near future, these will be merged into the Launchpad branches so that all code is easily found and maintained.

Launchpad

The main project code is housed in the following projects on Launchpad:

• lp:opencompute/opencompute-testing
• lp:checkbox

GitHub

ITRI has placed the majority of the Certification test scripts, job files and such on GitHub:

• https://github.com/opencomputeproject/ocp_ci_server_certification_test_cases

Pulling the code

See above about using bzr to obtain local branches of the Checkbox or OpenCompute Testing trunks.

For the ITRI github location, you'll need to use git.

Adding new code

The following is meant to help you understand where to add things in the code.

opencompute-certification

Trunk Contents

The trunk contains the following directories:

bin

This directory contains all the test scripts that are not part of the lp:checkbox branch. These are the OCP specific test scripts.

COPYING

This is a copy of the license. All code is Open Source under the GPLv3. Proprietary code is not allowed in the certification suite.

data

This directory is used in case sample test data needs to be installed along with the test suite.

debian

This directory contains all the necessary Debian packaging information. It also contains the Changelog. Likely, you will only ever need to edit the changelog in this directory.

examples

This directory contains sample config files that are installed when a given package is installed.

jobs

This directory contains the job description files. Jobs are the units of Test that are performed.

launcher

This directory contains the launcher file that creates the command used to execute the test tool on the SUT

manage.py

This is a helper program for testing and building packages.

po

UNUSED CURRENTLY, but this would house translation data if implemented.

README.development

A README file that gives you basic development information. Please read this file.

README.rst

A description of the trunk contents (the same thing you're reading right now)

requirements

Package building requirements.

whitelists

The whitelists that define each test suite.

Job Files

As discussed before, job files contain the descriptions that define each test case or unit of test.

Basic Job Description

An example job description looks like this:

plugin: shell
name: memory/ocp/stress_30min
user: root
command: stressapptest -s 1800
_description:
 Test and exercise memory.

This is a very basic example that contains most of the common KEY:VALUE pairs that are used to describe a test.

plugin

Plugins are the type of job to perform.

shell is a fully automated job that is executed by the test harness automatically with no user intervention.

manual is a test/job that requires the user to perform some task and verify the results.

user-interact which also requires the user to initiate the test with the PASS or FAIL results being automatically determined by Checkbox

user-interact-verify requires the user to initiate the test AND validate the results, choosing PASS or FAIL afterwards.

name

This is a descriptive name of the test. In this case, the hierarchy means that it's a Memory test that is part of the OCP Specific tests and the test case is 'stress_30min'

user

Some jobs can be run as an alternate user (in fact, some require this). So specifying root here will tell Checkbox to run the command as the root user (essentially using sudo to elevate privileges for this one test only).

command

This is the command to perform. This item can be in-line code (like a bash oneliner or test) or the name of a command that is in the bin/ directory.

_description

This is a text description of the job. For a Manual job, description would list the steps necessary to perform the test. For an automated test, this is just a summary of what the test does.

Meta-Jobs

There are also things we refer to as "Metajobs". These jobs are actually run early on and are simply used to generate jobs on the fly. In general, you would use a metajob to create separate jobs for multiple pieces of the same hardware. In the following example, we have a metajob that creates a separate storage test for any disk device (NON-USB) discovered.

plugin: local
name: disk/ocp/io_stress
requires:
 device.category == 'DISK'
 package.name == 'stressapptest'
_description: Verify that storage devices, such as Fibre Channel and RAID can be detected and perform under stress.
command:
 cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
 plugin: shell
 name: disk/ocp/io_stress_`ls /sys$path/block`
 user: root
 requires:
  device.path == "$path"
  block_device.`ls /sys$path/block`_state != 'removable'
 description: Disk I/O stress test for $product
 command: disk_stress `ls /sys$path/block | sed 's|!|/|'`
 EOF

Note the differences between a Metajob and a normal Job.

Plugin

local is a special plugin that tells Checkbox to run this job as soon as the tool starts. This means this job is executed well before the user ever sees a Suite or Test Selection screen.

requires

This job also introduces the requires directive. This means that the following conditions must be met before the job will be run.

Also note that you can specify multiple requirements by placing each one on a new line with a single-space indention.

Command

In the basic example, I mentioned that a Command can be embedded shell code. This is an example of that embedded shell code. This runs the shell code that follows. In this example, it cats all the text between the 'cat' line and the 'EOF' marker (which looks exactly like the Basic Job Description) and pipes all that through the 'run-templates' helper tool which creates on-the-fly jobs. In this case, 'run-templates' also executes the udev-resource resource gathering script and ensures that child jobs are ONLY created for hardware devices that fit into the DISK category (so we don't create these Disk jobs for network devices or CPUs).

Scripts

Scritps are found in the bin/ directory. These can be anything from bash to python to ruby to anything you feel like coding in. Scripts can be actual tests themselves, or wrappers that call other tools or test suites. In the past, we have had wrapper scripts that ran things like Phoronix Test Suite and Piglit (for desktop testing).

Whitelists

Whitelists are text lists that define the test suite. This is a list of tests to run. Below is a sample whitelist:

# Resource Jobs
miscellanea/submission-resources
#Info attachment jobs
__info__
cpuinfo_attachment
dmesg_attachment
dmi_attachment
dmidecode_attachment
efi_attachment
lspci_attachment
lshw_attachment
mcelog_attachment
meminfo_attachment
modprobe_attachment
modules_attachment
sysctl_attachment
sysfs_attachment
udev_attachment
lsmod_attachment
acpi_sleep_attachment
info/hdparm
info/hdparm_.*.txt
installer_debug.gz
info/disk_partitions
# Actual test cases
__TC-003__
TC-003-0003-001/Network_performance
TC-003-0003-001/Network_performance_.*

The first job, miscellanea/submission-resources is always required in a whitelist. It is what triggers all the various resource jobs to run as well as jobs necessary to create a full submission file. After that, add whatever jobs you need. The first group in the example above are jobs that attach files containing the output of various commands.

In the next section is an example of two things. The job TC-003-0003-001/Network_performance is a "metajob" that generates other jobs. IN this case, it's a job that creates a separate network test for each NIC discovered on the SUT. The next job, TC-003-0003-001/Network_performance_.* is an example of using a regular expression. In this case, this line item tells Checkbox to run any job that starts with the text 'TC-003-0003-001/Network_performance_' (which when run would be jobs like 'TC-003-0003-001/Network_performance_eth0' and 'TC-003-0003-001/Network_performance_eth1'.

Submitting Code to the Project

The following will cover how to submit code once you've made changes to your branch. The examples here assume you have already performed the following steps:

1. Branched lp:opencompute/opencompute-testing.
2. Made changes or additions
3. Tested your changes or additions
4. Committed your changes or additions

Launchpad

As noted early on, you will need to be logged in via bzr to submit code. This is accomplished using the bzr launchpad-login command:

bzr launchpad-login <YOUR_LAUNCHPAD_USERNAME>

You will likely also need to have created and submitted an SSH key to Launchpad to use for submissions. This is explained here: https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair

Pushing Changes

Once you have committed your changes, you need to push them to YOUR OWN branch on Launchpad. To do so, cd into your branch and do the following:

bzr push lp:~<YOUR LP USERNAME>/opencompute/<YOUR BRANCH NAME>

Example:

bzr push lp:~bladernr/opencompute/ocp-cert-cleanup

Merge Requests

Once you have pushed your branch to Launchpad, you need to make a merge request. To do so you need to do the following things:

1. In a web browser, go to http://code.launchpad.net/~YOUR_LAUNCHPAD_USERNAME
2. Click on the name of the branch you just pushed
3. Under Branch Merges click on Propose for merging
4. Under Target Branch select Other and enter '~opencompute-developers/opencompute/opencompute-testing'
5. In the Description text box, enter a summary of the changes you made
6. Click the Propose Merge button at the bottom of that page

This is submit your merge request. At this point, it will be reviewed, commented on, approved or sent back for fixing.

Questions?

OCP Ready or OCP Certification

• Certification Project Team Mailing List: http://lists.opencompute.org/mailman/listinfo/opencompute-ci
• Freenode IRC Servers #ocp-ci

CheckboxNG or Plainbox

• Freenode IRC Servers #checkbox
• http://launchpad.net/checkbox

Further Reading, Resources, Etc

• Launchpad Help
• Bazaar (bzr) Help
• Bazaar (bzr) Documentation