This page contains the test environment overview and instructions on how to run the RackHD Tests from the command line.
RackHD Test CI is the automated testing harness that is run via Jenkins whenever new code is checked in.
The 'Jenkins' build server kicks off the RackHD 'Smoke Test' suites on each PR and Code checkin to the main branch. The Smoke Test is a system test suite that verifies basic functionality.
RackHD Test Environment Overview
The RackHD Test repo has a good README file that lays out the basics for get the test environment running. https://github.com/RackHD/RackHD/blob/master/test/README.md
Test Environment Setup
Repositories: https://github.com/RackHD/RackHD RackHD test repository in open source https://eos2git.cec.lab.emc.com/OnRack/dellemc-test Internal reposity for local configs, scripts, miscellaneous - that don't meet open source criteria or sensitive info Plymouth lab configuration is contained in config-mn.
Download Rackhd: This gets the latest master branch from github. git clone https://github.com/RackHD/RackHD rackhd
Download the internal repository: This gets the latest master branch from internal github. git clone https://eos2git.cec.lab.emc.com/OnRack/dellemc-test dellemc-test
If desired, local lab configs and internal tests can be copied into your working directory. cd rackhd/test cp -Rp dellemc-test/* .
Setup the virtual environment: Note a separate environment is needed to run on Ubuntu14 verses Ubuntu16. In the MN lab, test servers lab03-lab15 run Ubuntu14, lab15-lab24 run ubuntu16. Name your environment to help yourself out, name it so that you know which OS you ran it on, if you run on both. cd rackhd/test ./mkenv.sh eh_ubuntu14 source myenv_eh_ubuntu14
Test Beds supported Physical stacks - Reserve hardware stack via booked for MN, physical stacks exist in Hopkington, Austin, and Shanghai as well. Vagrant virtual stacks - setup a virtual stack via virtrual box on your local laptop or virtual machine of choice - default stack is vagrant and localhost
Test Code
RackHD Test framework and test scripts are located under the RackHD/RackHD code repository, under directory test.
https://github.com/RackHD/RackHD/tree/master/testThe test tree has the following structure:
rackhd/test/
- deploy (RackHD deployment and stack initialization scripts)
- common (shared framework and library import files)
- modules (miscellaneous library modules, logger and amqp being deprecated)
- templates (sample scripts)
- util (non-test utilities and helper scripts)
- tests (test scripts not included in the harness)
- Vagrantfile - vagrant installation configuration file
All testcode is written in Python 2.7x and conforms to Python unittest standards as defined here:
https://docs.python.org/2/library/unittest.html
Test Runners
Currently we support both CIT and FIT test runners to execute the functional test scripts under rackhd/test/tests.
When developing NEW test scripts, they should be developed to be run by the FIT run_tests.py test runner and scripts may not be checked into tests/api or tests/api-cit.
run_tests.py
This runner is used to run FIT and converted CIT scripts
The converted cit to fit style scripts are located under test/tests/api-cit
The FIT test scripts are located under the other directories in test/tests
run.py
This is used to run CIT scripts in their current form and being deprecated.
CIT scripts are located under test/tests/api
Configuration
When running CIT and FIT, the default configuration directory is rackhd/test/config.
The files under this directory are used by both run.py and run_tests.py and are defaulted for a Vagrant RackHD installation and for running tests using 'off-box' or 'remote' access.
Default configuration files for the MN lab are in the internal repository, dellemc-test, under the directory config-mn. Default conifguration files for Shanghai are in config-sh.
Using run_tests.py allows the user to override the default directory by using the -config parameter to point to another directory.
ie. run_tests.py -stack 1 -config config-mn -test tests -group smoke
For run_tests.py, the config files used are:
install_default.json - this file includes parameters needed for your local environment installation. Default is for vagrant install, modify as needed for your lab. rackhd_default.json - this file is the rackhd config.json that gets installed on your rackhd appliance. /opt/monorail/config.json Default is for vagrant install. It should be updated based on your lab environment. stack_config.json - this file contains lab specific stack configurations, default is for vagrant stack. Used for communication to your test bed. credentials_default.json - this file contains any username/password credentials for your lab, your hosts, nodes, switches, etc.
<my-config>.json - this file can contain any special parameters needed for a specific script, added via command line option -extra
The following is the order of precedence in parameter processing:
- command line parameters are top precedence
- extra 'test specific` parameters contained in test config file (added on command line via -extra <myconfig>.json)
- stack_config.json
- install_default.json
- credentials.json
- rackhd_default.json
The configuration files are read in at the start of the run and all parameters then can be accessed via fitcfg() functions during the test.
For run.py, the config files used are:
config.ini - this file contains the rackhd defaults and test bed parameters settings.py - this file reads the config.ini file to populate parameters. Overrides can be done in here for certain parameters.
Note: In run_tests.py, each time a test run occurs, a snapshot of the configuration is created and saved into a file.
That file can be used as input to a subsequent test run (overriding the config directory), allowing for debugging and recreation of a specific test run. This will be helpful when field issues are found and need to be debugged.
This will let us see exactly the configuration that was being run and how it was setup or how to recreate the setup.
Installing RackHD
There are multiple ways to install RackHD on your test bed for performing your own testing. Here are some methods:
1. Install via Vagrant Follow the directions in the RackHD/RackHD/test/ readme file
https://github.com/RackHD/RackHD/blob/master/test/README.md
2. Install from Jenkins job (MN Lab) https://roebling.hwimo.lab.emc.com/jenkins/job/Running-Tests/job/Install-RackHD-on-any-Stack/ - this job will: run the deploy script run_rackhd_installer.py - install the ubuntu16 ova onto the stack via deploy script os_ova_install.py - install latest rackhd code via deploy rackhd_source_install.py - initialize the stack via deploy rackhd_stack_init.py
3. Installing via rackhd/test/deploy scripts (MN Lab) This allows you to deploy the latest rackhd code by hand as needed.
Run the full install via:
run_tests.py -stack NN -config config-mn -test deploy/run_rackhd_installer.py
Run each step via:
run_tests.py -stack NN -config config-mn -test deploy/os_ova_install.py
run_tests.py -stack NN -config config-mn -test deploy/rackhd_source_installer.py
or
run_tests -stack NN -config config-mn -test deploy/rackhd_package_installer.py
run_tests -stack NN -config config-mn -test deploy/rackhd_stack_init.pyRunning the Tests
Running the Tests
Manual Test Runs via Jenkins
By default, Jenkins initiates an 'Smoke Test' run after each code checkin from github. The MN Jenkins can also execute a manual run with custom parameters. Here's is a link to Installing RackHD via Jenkins in the MN lab.
https://roebling.hwimo.lab.emc.com/jenkins/job/Running-Tests/job/Install-RackHD-on-any-Stack/
Running tests locally
RackHD run_tests.py scripts have a common set of arguments that can be used at runtime:
run_tests.py -h
usage: run_tests.py [-h] [-test TEST] [-config CONFIG] [-extra EXTRA]
[-group GROUP] [-stack STACK] [-rackhd_host RACKHD_HOST]
[-template TEMPLATE] [-xunit] [-numvms NUMVMS] [-list]
[-sku SKU] [-obmmac OBMMAC | -nodeid NODEID]
[-http | -https] [-port PORT] [-v V] [-nose-help]
Command Help
optional arguments:
-h, --help show this help message and exit
-test TEST test to execute, default: tests/
-config CONFIG config file location, default: config
-extra EXTRA comma separated list of extra config files (found in
'config' directory)
-group GROUP test group to execute: 'smoke', 'regression',
'extended', default: 'all'
-stack STACK stack label (test bed)
-rackhd_host RACKHD_HOST
RackHD appliance IP address or hostname, default:
localhost
-template TEMPLATE path or URL link to OVA template or RackHD OVA
-xunit generates xUnit XML report files
-numvms NUMVMS number of virtual machines for deployment on specified
stack
-list generates test list only
-sku SKU node SKU name, example: Quanta-T41, default=all
-obmmac OBMMAC node OBM MAC address, example:00:1e:67:b1:d5:64
-nodeid NODEID node identifier string of a discovered node, example:
56ddcf9a8eff16614e79ec74
-http forces the tests to utilize the http API protocol
-https forces the tests to utilize the https API protocol
-port PORT API port number override, default from
install_config.json
-v V Verbosity level of console and log output (see -nose-
help for more options), Built Ins: 0: Minimal logging,
1: Display ERROR and CRITICAL to console and to files,
3: Display INFO to console and to files, 4: (default)
Display INFO to console, and DEBUG to files, 5:
Display infra.run and test.run DEBUG to both, 6: Add
display of test.data (rest calls and status) DEBUG to
both, 7: Add display of infra.data (ipmi, ssh) DEBUG
to both, 9: Display infra.* and test.* at DEBUG_9 (max
output)
-nose-help display help from underlying nosetests command,
including additional log options
To run FIT, use run_tests.py
python run_tests.py -stack vagrant -test tests -group smoke python run_tests.py -stack vagrant -test tests/rackhd20/test_rackhd20_api_nodes.py -list python run_tests.py -stack vagrant -test tests/rackhd20/test_rackhd20_api_nodes.py -v 2 python run_tests.py -stack 1 -config config-mn -test util/display_rackhd_nodes.py -v 2 python run_tests.py -stack 3 -config config-mn -test deploy/rackhd_stack_init.py -list python run_tests.py -stack 10 -config config-mn -test deploy/rackhd_stack_init.py:rackhd_stack_init.test10_apply_obm_settings -v 2 python run_tests.py -stack 5 -config config-sh -test tests -group smoke -v 4 python run_tests.py -stack sh10 -config config-sh -extra imageservice.json -test tests/imagesevice/test_image_service.py
To run CIT tests, use run.py
python run.py --show-plan - will display a list of all tests and groups in the overall plan python run.py --group="pollers_api2.tests" - will run a single group of tests
An example to run the full 'smoke test' against stack 1, use the following commands:
git clone https://github.com/hohene/RackHD rackhd git clone https://eos2git.cec.lab.emc.com/hohene/dellemc-test dellemc-test cd rackhd/test cp -Rp ../../dellemc-test/* . run_tests.py -stack 1 -config config-mn -test deploy/run_rackhd_installer.py -v 2 run_tests.py -stack 1 -config config-mn -test tests -group smoke
Interrupting or Cancelling Test Execution
Ctrl-C will exit the current test and flag an error.
Ctrl-Z will exit the test script and return control to the shell. Note that there may still be tests running in background.
Output Formats
Test output can be exported to jUnit/xUnit format for import into Jenkins or for another utiiltyl by using the -xunit optional argument.
Sample XML output file:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="test script template" tests="4" errors="1" failures="1" time="4.01295185089" timestamp="2015-06-02T15:26:06" hostname="0.0.0.0" package="autotest" >
<testcase name="Successful test" time="1.0016348362">
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Failed test" time="1.00292992592">
<failure>
Traceback (most recent call last):
File "./script_template.py", line 66, in test1
self.assertEqual(0,1, 'Dummy failed assert')
AssertionError: Dummy failed assert
</failure>
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Erred test" time="1.00141096115">
<error>
Traceback (most recent call last):
File "./script_template.py", line 78, in test1
x = y[0] # assign an undefined variable
NameError: global name 'y' is not defined
</error>
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Successful test" time="1.00119400024">
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
</testsuite>
</testsuites>
Test Logging
The test framework provides for logging of the test activity using the logging plugin stream-monitor.
Logging is displayed in a number of ways:
- Test case activity logging on Jenkins - each test case executed via Jenkins will generate an entry under the "Test Result" link in the test results page that is sorted alphabetically by test case name
- Test console log - verbose output form the test script is stored under the Jenkins link "test_console.log" in the test results page
- log_output directory - a directory for each run is created in the running directory
- Appliance logs - system logs and system state will be stored on a log server after each test run.