RackHD CI Test Conventions - Smoke, Regression, Extended test requirements
This page describes how to make test code submissions to the RackHD test suites.
The 'Continuous Integration Test Harness' is defined as a set of RackHD test scripts that use the 'Test Framework' which are executed automatically via Jenkins using the Continuous Integration methodology.
Automatic execution of RackHD tests requires that these scripts meet some well-defined standards and processes for inclusion.
The table below outlines the scope and criteria for each component of the Harness.
.
RackHD Test Components
| Type | Description | Submission Criteria | Exclusions |
|---|---|---|---|
| Smoke Test | A broad subset of Regression Tests that run automatically when a PR is submitted for review to the prodcut repository in github. Tests in the Smoke test are intended to cover core functionality but are not comprehensive. The Smoke Test is used as a determinant for permitting of a code merge and is intended to be 'green' (no errors) at all times. The Smoke Test is to be generalized and portable so that it can be run on Vagrant Box installations and all certified hardware and configurations. The Smoke Test is time-bounded to under one hour and is non-destructive. |
|
|
| Regression Test | A broad and deep non-destructive test suite that is intended to be a comprehensive test of all product functions that is executed on a regular schedule. The Regression Test is to be generalized and portable so that it can be run on all certified hardware and configurations. The Regression Test is not time-bounded but will not include very time-consuming tests such as load and scaling. |
|
|
| Extended Tests | Any valid tests that do not meet criteria for Smoke or Regression are here. These tests may be separated into more than group responding to the test categories below. Runs are scheduled when time permits on relevant CI stacks and testbeds. Extended Test categories:
|
|
|
| Framework | Common routines, libraries, logging, etc. that are used by test scripts. No tests (or anything with an assert) should be included in framework code. |
|
|
RackHD Test Directory Structure
RackHD/RackHD/test/...
/common (Test Framework and library scripts)
/templates (sample scripts)
/util (helful test utilities and tools, no tests)
/tests/... (All test scripts for any rackhd testing in functional area subdirs)
/config (configuration files for the running test environment, both CIT and FIT)
/deploy (installation, deployment, and initializatoin scripts for the test environment)
/modules (CIT supported library modules, some being deprecated)
/stream-monitor (logging plugin, used within the framework and tests)
/test-opts (internal installed directory, used to hold special files needed by some specific test scripts. Test scripts can use this directory for their own use, for files that aren't opensource, for hw specific config, ip address, etc.
Test Submissions to RackHD Test Harness
RackHD Engineers are encouraged to contribute new tests to the Harness under the following conditions:
- A change in product behavior that requires a revision in a Harness script that would otherwise fail
- New functions and features that do not have existing tests
- Manual tests that have been automated
RackHD Engineers have a responsibility to maintain product code so that new commits do not break any existing non-failing regression and smoke tests.
If tests do fail after a commit and the failing tests are deemed to be valid, the Engineer is required to revise the product code to remove the test failure. Otherwise, if the test is deemed invalid, either fix the test code or write a bug to have someone else fix it.
Submissions and revisions to RacHD Test code will be processed like any other code change with a github pull Request and code review.
Note that each PR into the Rackhd/RackHD repository triggers a Smoke Test run on the Jenkins server.
Regression and Smoke Test Submissions
Submissions to the Regression and Smoke test scripts must meet all of the criteria in the table above subject to code review and approval by Core Committers.
Typically, a test will start life in the Functional test area, then be promoted to Regression, and then Smoke if it meets criteria.
Exceptions will be revisions to existing tests and functional enhancements to existing Regression and Smoke scripts if needed.
Anyone can submit a test script into the 'feature' test area by creating a Pull request and then having the Core Committers review and merge the code to Master. At that point the script will be available for integration.
The test framework team is responsible for 'curating' the contents of the smoke, regression, and extended test runs which are executed via Jenkins. To have scripts included in regular automated test runs. If the scripts meet criteria, they will be added to the appropriate test runs.
Running flake8 Python style guide enforcement.
We are using the flake8 tool and Hound to check the python scripts before merge for style. It is expected that all code remove all hound violations before it can be merged.
When the code is submitted in a PR, the code is automatically checked via Hound guidelines and a report is generated via Hound. Before submitting your PR, please run a check against your code to find the style errors and fix them first. This eliminates doing a lot of extra commits in your PR to simply fix style issues.
Unfortunately, while we run Python 2.7, Hound currently checks to Python 3 standards, so you may want to run a separate check using python3 in your environment.
How to run the checks:
# Note: https://github.com/RackHD/RackHD - the flake8 configuration file is contained in this directory. Examples of how to use flake8. virtualenv -p /usr/bin/python2.7 p27 source p27/bin/activate cd rackhd flake8 --config=.flake8 test/tools/save-logs/save_logs.py virtualenv -p /usr/bin/python3 py3 source py3/bin/activate cd rackhd flake8 --config=.flake8 test/tools/save-logs/save_logs.py