RackHD OSE Test Architecture

This page contains an overview of the OSE Test architecture and a description of its components.

RackHD is developed and tested under a 'Continuous Integration' model. All PRs and builds are automatically 'smoke-tested' for basic functionality.

Automated testing utilizes a custom Test Framework to integrate test operations, resource scheduling, and data collection which is implemented as a common python import module.

RackHD Test Terminology

Continuous Integration - the methodology that provides for continuous automated testing integrated to the peer review and build process.

RackHD Test Framework - the set of common routines that are located in the 'rackhd/test' repository under the common and modules directory which support common functions, reporting, and test execution.

Test Harness - the set of test scripts using the 'RackHD Test Framework' located in the github.com rackhd/rackhd/test directory that are included in triggered runs by the Jenkins build server.  (CIT and FIT envrionment scripts)

Smoke Tests - a set of test cases that are run to verify basic high level functionality.  These get run  on an individual PR (code peer review) prior to merge and on RackHD code builds.

Regression Tests -  a larger suite of tests that are run against the RackHD code base on an intermittent basis.  The tests can run scripts that contains the full set of approved and qualified tests to be run on a schedule.

Extended Tests - any valid tests that do not meet the criteria for inclusion into Smoke or Regression tests.

CI Stack - the hardware reserved to run the triggered and scheduled testing

 Wrapper Script - a script that executes all of the test scripts in a specific test suite. Wrapper scripts contain no tests, just execute other scripts that do.

RackHD Test Architecture

 See CI test architecture.

RackHD Test Script Design Requirements

• A 'smoke test' subset of the regression test shall be run on every PR prior to merge and on every development build via the build server to verify basic system functions
• The 'smoke test' shall be a minimal set of regression tests which will run for less than 30 minutes to an 1 hour
• A 'regression test' consisting of a comprehensive non-destructive test of system functions shall be run on a scheduled basis
• Test scripts can be run optionally from a build server, test server, or manually
• Test scripts shall be able to run on a variety of test configurations.  Note that some test scripts may be very hardware or test bed specific, making it not a smoke or regression candidate.
• Test results shall be reported in human-readable and optionally in machine-readable form
• Test cases shall be organized in 'suites' which can have common setup/teardown routines
• Test scripts shall use the Python 'unittest' module
• Test scripts shall employ a common library of reusable functions (the 'framework')
• Test framework shall be configurable to allow operating conditions to be altered at run time
• Test framework shall capture and store system logs, standard output, debugging output, etc. onto a log server for later retrieval
• Test framework shall be able to push results to a test case management server for reporting
• Test framework shall be able to define software release and target test servers at run time

RackHD Test Design Principles

RackHD CIT and FIT test framework and scripts are coded using Python 2.7.

All RackHD test scripts are intended to be launched either by a build server (Jenkins) or manually.

The RackHD Test (FIT) framework adheres to the following principles:

• Use the Python 'unittest' module to organize and report testing
• Use 'unittest' asserts to verify test conditions whenever possible
• After the test, leave the System Under Test in the same state that it was in the beginning
• Bundle similar tests together in scripts (testsuites)
• Utilize common library functions for repetitive tasks (tests/common)
• Use command line arguments to configure test parameters

NOTE: There is some confusion over the term 'unit test' as it applies to RackHD Test framework because it utilizes the Python 'unittest' module. The 'unittest' module is a software component utilized by both Unit Tests and Feature/Functional Tests. Unit Tests are typically created by the software developer and checked in with the product code. RackHD  functional test suites also use the 'unittest' module but is typically created by SW engineers wokring in a testing mode and checked into a test repository separate from the product code.

Links to Test Toolchain servers