Editing C&I Wiki Portal/ToolsDevelopmentGuideline (section)

=== opencompute-certification ===
==== Trunk Contents ====
The trunk contains the following directories:
;bin
:This directory contains all the test scripts that are not part of the lp:checkbox branch.  These are the OCP specific test scripts.
;COPYING
:This is a copy of the license.  All code is Open Source under the GPLv3.  Proprietary code is not allowed in the certification suite.
;data
:This directory is used in case sample test data needs to be installed along with the test suite.
;debian
:This directory contains all the necessary Debian packaging information.  It also contains the Changelog.  Likely, you will only ever need to edit the changelog in this directory.
;examples
:This directory contains sample config files that are installed when a given package is installed.
;jobs
:This directory contains the job description files.  Jobs are the units of Test that are performed.
;launcher
:This directory contains the launcher file that creates the command used to execute the test tool on the SUT
;manage.py
:This is a helper program for testing and building packages.
;po
:UNUSED CURRENTLY, but this would house translation data if implemented.
;README.development
:A README file that gives you basic development information.  Please read this file.
;README.rst 
:A description of the trunk contents (the same thing you're reading right now)
;requirements 
:Package building requirements.
;whitelists
:The whitelists that define each test suite.

==== Job Files ====
As discussed before, job files contain the descriptions that define each test case or unit of test.  
===== Basic Job Description =====
An example job description looks like this:

<pre>
plugin: shell
name: memory/ocp/stress_30min
user: root
command: stressapptest -s 1800
_description:
 Test and exercise memory.
</pre>

This is a very basic example that contains most of the common KEY:VALUE pairs that are used to describe a test.
;plugin
:Plugins are the type of job to perform.  
:'''shell''' is a fully automated job that is executed by the test harness automatically with no user intervention.  
:'''manual''' is a test/job that requires the user to perform some task and verify the results.
:'''user-interact''' which also requires the user to initiate the test with the PASS or FAIL results being automatically determined by Checkbox
:'''user-interact-verify''' requires the user to initiate the test AND validate the results, choosing PASS or FAIL afterwards.
;name
:This is a descriptive name of the test.  In this case, the hierarchy means that it's a Memory test that is part of the OCP Specific tests and the test case is 'stress_30min'
;user 
:Some jobs can be run as an alternate user (in fact, some require this).  So specifying '''root''' here will tell Checkbox to run the '''command''' as the root user (essentially using sudo to elevate privileges for this one test only).
;command
:This is the command to perform.  This item can be in-line code (like a bash oneliner or test) or the name of a command that is in the bin/ directory.
;_description
:This is a text description of the job.  For a Manual job, '''description''' would list the steps necessary to perform the test.  For an automated test, this is just a summary of what the test does.

===== Meta-Jobs =====
There are also things we refer to as "Metajobs".  These jobs are actually run early on and are simply used to generate jobs on the fly.  In general, you would use a metajob to create separate jobs for multiple pieces of the same hardware.  In the following example, we have a metajob that creates a separate storage test for any disk device (NON-USB) discovered.

<pre>
plugin: local
name: disk/ocp/io_stress
requires:
 device.category == 'DISK'
 package.name == 'stressapptest'
_description: Verify that storage devices, such as Fibre Channel and RAID can be detected and perform under stress.
command:
 cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
 plugin: shell
 name: disk/ocp/io_stress_`ls /sys$path/block`
 user: root
 requires:
  device.path == "$path"
  block_device.`ls /sys$path/block`_state != 'removable'
 description: Disk I/O stress test for $product
 command: disk_stress `ls /sys$path/block | sed 's|!|/|'`
 EOF
</pre>

Note the differences between a Metajob and a normal Job.
;Plugin
:'''local''' is a special plugin that tells Checkbox to run this job as soon as the tool starts. This means this job is executed well before the user ever sees a Suite or Test Selection screen.
;requires
:This job also introduces the '''requires''' directive.  This means that the following conditions must be met before the job will be run.  
:Also note that you can specify multiple requirements by placing each one on a new line with a single-space indention.
;Command
:In the basic example, I mentioned that a '''Command''' can be embedded shell code.  This is an example of that embedded shell code.  This runs the shell code that follows.  In this example, it cats all the text between the 'cat' line and the 'EOF' marker (which looks exactly like the Basic Job Description) and pipes all that through the 'run-templates' helper tool which creates on-the-fly jobs.  In this case, 'run-templates' also executes the udev-resource resource gathering script and ensures that child jobs are ONLY created for hardware devices that fit into the DISK category (so we don't create these Disk jobs for network devices or CPUs).

==== Scripts ====
Scritps are found in the bin/ directory.  These can be anything from bash to python to ruby to anything you feel like coding in.  Scripts can be actual tests themselves, or wrappers that call other tools or test suites.  In the past, we have had wrapper scripts that ran things like Phoronix Test Suite and Piglit (for desktop testing).

==== Whitelists ====
Whitelists are text lists that define the test suite.  This is a list of tests to run.  Below is a sample whitelist:
<pre>
# Resource Jobs
miscellanea/submission-resources
#Info attachment jobs
__info__
cpuinfo_attachment
dmesg_attachment
dmi_attachment
dmidecode_attachment
efi_attachment
lspci_attachment
lshw_attachment
mcelog_attachment
meminfo_attachment
modprobe_attachment
modules_attachment
sysctl_attachment
sysfs_attachment
udev_attachment
lsmod_attachment
acpi_sleep_attachment
info/hdparm
info/hdparm_.*.txt
installer_debug.gz
info/disk_partitions
# Actual test cases
__TC-003__
TC-003-0003-001/Network_performance
TC-003-0003-001/Network_performance_.*
</pre>

The first job, '''miscellanea/submission-resources''' is always required in a whitelist.  It is what triggers all the various resource jobs to run as well as jobs necessary to create a full submission file.  After that, add whatever jobs you need.  The first group in the example above are jobs that attach files containing the output of various commands. 

In the next section is an example of two things.  The job '''TC-003-0003-001/Network_performance''' is a "metajob" that generates other jobs.  IN this case, it's a job that creates a separate network test for each NIC discovered on the SUT.  The next job, '''TC-003-0003-001/Network_performance_.*''' is an example of using a regular expression.  In this case, this line item tells Checkbox to run any job that starts with the text 'TC-003-0003-001/Network_performance_' (which when run would be jobs like 'TC-003-0003-001/Network_performance_eth0' and 'TC-003-0003-001/Network_performance_eth1'.