This page contains the test environment overview and instructions on how to run the RackHD Tests from the command line.

RackHD Test CI is the automated testing harness that is run via Jenkins whenever new code is checked in.

The 'Jenkins' build server kicks off the RackHD 'Smoke Test' suites on each PR and Code checkin to the main branch. The Smoke Test is a system test suite that verifies basic functionality.

RackHD Test Environment Overview

The RackHD Test repo has a good README file that lays out the basics for get the test environment running. https://github.com/RackHD/RackHD/blob/master/test/README.md

Test Environment Setup

Repositories:

https://github.com/RackHD/RackHD RackHD test repository in open source

https://eos2git.cec.lab.emc.com/OnRack/dellemc-test Internal reposity for local configs, scripts, miscellaneous - that don't meet open source criteria or sensitive info Plymouth lab configuration is contained in config-mn.

Download Rackhd:

This gets the latest master branch from github.

git clone https://github.com/RackHD/RackHD rackhd

Download the internal repository:

This gets the latest master branch from internal github.

git clone https://eos2git.cec.lab.emc.com/OnRack/dellemc-test dellemc-test

If desired, local lab configs and internal tests can be copied into your working directory.

    cd rackhd/test

    cp -Rp dellemc-test/* .

Setup the virtual environment:

Note a separate environment is needed to run on Ubuntu14 verses Ubuntu16.

In the MN lab, test servers lab03-lab15 run Ubuntu14, lab15-lab24 run ubuntu16.

Name your environment to help yourself out, name it so that you know which OS you ran it on, if you run on both.

    cd rackhd/test

    ./mkenv.sh eh_ubuntu14

    source myenv_eh_ubuntu14

Test Beds supported Physical stacks - Reserve hardware stack via booked for MN, physical stacks exist in Hopkington, Austin, and Shanghai as well.

Vagrant virtual stacks - setup a virtual stack via virtrual box on your local laptop or virtual machine of choice - default stack is vagrant and localhost

Test Code

RackHD Test framework and test scripts are located under the RackHD/RackHD code repository, under directory test.

https://github.com/RackHD/RackHD/tree/master/test

The test tree has the following structure:

rackhd/test/

deploy (RackHD deployment and stack initialization scripts)
common (shared framework and library import files)
modules (miscellaneous library modules, logger and amqp being deprecated)
templates (sample scripts)
util (non-test utilities and helper scripts)
tests (test scripts not included in the harness)
Vagrantfile - vagrant installation configuration file

All testcode is written in Python 2.7x and conforms to Python unittest standards as defined here:

https://docs.python.org/2/library/unittest.html

Test Runners

Currently we support both CIT and FIT test runners to execute the functional test scripts under rackhd/test/tests.

When developing NEW test scripts, they should be developed to be run by the FIT run_tests.py test runner and scripts may not be checked into tests/api or tests/api-cit.

run_tests.py

This runner is used to run FIT and converted CIT scripts

The converted cit to fit style scripts are located under test/tests/api-cit

The FIT test scripts are located under the other directories in test/tests

run.py

This is used to run CIT scripts in their current form and being deprecated.

CIT scripts are located under test/tests/api

Configuration

When running CIT and FIT, the default configuration directory is rackhd/test/config.

The files under this directory are used by both run.py and run_tests.py and are defaulted for a Vagrant RackHD installation and for running tests using 'off-box' or 'remote' access.

Default configuration files for the MN lab are in the internal repository, dellemc-test, under the directory config-mn. Default conifguration files for Shanghai are in config-sh.

Using run_tests.py allows the user to override the default directory by using the -config parameter to point to another directory.

  ie. run_tests.py -stack 1 -config config-mn -test tests -group smoke

For run_tests.py, the config files used are:

install_default.json      - this file includes parameters needed for your local environment installation.                                
                            Default is for vagrant install, modify as needed for your lab.  
rackhd_default.json       - this file is the rackhd config.json that gets installed on your rackhd appliance.  /opt/monorail/config.json                              
                            Default is for vagrant install.  It should be updated based on your lab environment.  
stack_config.json         - this file contains lab specific stack configurations, default is for vagrant stack.                                
                            Used for communication to your test bed.  
credentials_default.json  - this file contains any username/password credentials for your lab, your hosts, nodes, switches, etc.

<my-config>.json          - this file can contain any special parameters needed for a specific script, added via command line option -extra

The following is the order of precedence in parameter processing:

command line parameters are top precedence
extra 'test specific` parameters contained in test config file (added on command line via -extra <myconfig>.json)
stack_config.json
install_default.json
credentials.json
rackhd_default.json

The configuration files are read in at the start of the run and all parameters then can be accessed via fitcfg() functions during the test.

For run.py, the config files used are:

  config.ini    - this file contains the rackhd defaults and test bed parameters    
  settings.py   - this file reads the config.ini file to populate parameters.  
                  Overrides can be done in here for certain parameters.

Note: In run_tests.py, each time a test run occurs, a snapshot of the configuration is created and saved into a file.

That file can be used as input to a subsequent test run (overriding the config directory), allowing for debugging and recreation of a specific test run. This will be helpful when field issues are found and need to be debugged.

This will let us see exactly the configuration that was being run and how it was setup or how to recreate the setup.

Installing RackHD

There are multiple ways to install RackHD on your test bed for performing your own testing. Here are some methods:

1. Install via Vagrant

Follow the directions in the RackHD/RackHD/test/ readme file

https://github.com/RackHD/RackHD/blob/master/test/README.md

2. Install from Jenkins job (MN Lab)

https://roebling.hwimo.lab.emc.com/jenkins/job/Running-Tests/job/Install-RackHD-on-any-Stack/

This job will run the deploy script run_rackhd_installer.py which does the following:

- install the ubuntu16 ova onto the stack via deploy script os_ova_install.py

- install latest rackhd code via deploy rackhd_source_install.py

- initialize the stack via deploy rackhd_stack_init.py

3. Installing via rackhd/test/deploy scripts (MN Lab) This allows you to deploy the latest rackhd code by hand as needed.

run fit installation examples

Run the full install via:
  run_tests.py -stack NN -config config-mn -test deploy/run_rackhd_installer.py

Run each step via:
  run_tests.py -stack NN -config config-mn -test deploy/os_ova_install.py
  run_tests.py -stack NN -config config-mn -test deploy/rackhd_source_installer.py
        or
  run_tests -stack NN -config config-mn -test deploy/rackhd_package_installer.py
  run_tests -stack NN -config config-mn -test deploy/rackhd_stack_init.pyRunning the Tests

Running the Tests

Manual Test Runs via Jenkins

By default, Jenkins initiates an 'Smoke Test' run after each code checkin from github. The MN Jenkins can also execute a manual run with custom parameters. Here's is a link to Installing RackHD via Jenkins in the MN lab.

https://roebling.hwimo.lab.emc.com/jenkins/job/Running-Tests/job/Install-RackHD-on-any-Stack/

Running tests locally

RackHD run_tests.py scripts have a common set of arguments that can be used at runtime:

run_tests.py command line options

run_tests.py -h
usage: run_tests.py [-h] [-test TEST] [-config CONFIG] [-extra EXTRA]
                    [-group GROUP] [-stack STACK] [-rackhd_host RACKHD_HOST]
                    [-template TEMPLATE] [-xunit] [-numvms NUMVMS] [-list]
                    [-sku SKU] [-obmmac OBMMAC | -nodeid NODEID]
                    [-http | -https] [-port PORT] [-v V] [-nose-help]

Command Help

optional arguments:
  -h, --help            show this help message and exit
  -test TEST            test to execute, default: tests/
  -config CONFIG        config file location, default: config
  -extra EXTRA          comma separated list of extra config files (found in
                        'config' directory)
  -group GROUP          test group to execute: 'smoke', 'regression',
                        'extended', default: 'all'
  -stack STACK          stack label (test bed)
  -rackhd_host RACKHD_HOST
                        RackHD appliance IP address or hostname, default:
                        localhost
  -template TEMPLATE    path or URL link to OVA template or RackHD OVA
  -xunit                generates xUnit XML report files
  -numvms NUMVMS        number of virtual machines for deployment on specified
                        stack
  -list                 generates test list only
  -sku SKU              node SKU name, example: Quanta-T41, default=all
  -obmmac OBMMAC        node OBM MAC address, example:00:1e:67:b1:d5:64
  -nodeid NODEID        node identifier string of a discovered node, example:
                        56ddcf9a8eff16614e79ec74
  -http                 forces the tests to utilize the http API protocol
  -https                forces the tests to utilize the https API protocol
  -port PORT            API port number override, default from
                        install_config.json
  -v V                  Verbosity level of console and log output (see -nose-
                        help for more options), Built Ins: 0: Minimal logging,
                        1: Display ERROR and CRITICAL to console and to files,
                        3: Display INFO to console and to files, 4: (default)
                        Display INFO to console, and DEBUG to files, 5:
                        Display infra.run and test.run DEBUG to both, 6: Add
                        display of test.data (rest calls and status) DEBUG to
                        both, 7: Add display of infra.data (ipmi, ssh) DEBUG
                        to both, 9: Display infra.* and test.* at DEBUG_9 (max
                        output)
  -nose-help            display help from underlying nosetests command,
                        including additional log options

To run FIT, use run_tests.py

run fit examples

python run_tests.py -stack vagrant -test tests -group smoke
python run_tests.py -stack vagrant -test tests/rackhd20/test_rackhd20_api_nodes.py -list
python run_tests.py -stack vagrant -test tests/rackhd20/test_rackhd20_api_nodes.py -v 2

python run_tests.py -stack 1 -config config-mn -test util/display_rackhd_nodes.py -v 2
python run_tests.py -stack 3 -config config-mn -test deploy/rackhd_stack_init.py -list
python run_tests.py -stack 10 -config config-mn -test deploy/rackhd_stack_init.py:rackhd_stack_init.test10_apply_obm_settings -v 2

python run_tests.py -stack 5 -config config-sh -test tests -group smoke -v 4
python run_tests.py -stack sh10 -config config-sh -extra imageservice.json -test tests/imagesevice/test_image_service.py

To run CIT tests, use run.py

run cit examples

python run.py --show-plan                    - will display a list of all tests and groups in the overall plan
python run.py --group="pollers_api2.tests"  - will run a single group of tests

An example to run the full 'smoke test' against stack 1, use the following commands:

git clone https://github.com/hohene/RackHD rackhd
git clone https://eos2git.cec.lab.emc.com/hohene/dellemc-test dellemc-test
cd rackhd/test
cp -Rp ../../dellemc-test/* .
run_tests.py -stack 1 -config config-mn -test deploy/run_rackhd_installer.py -v 2
run_tests.py -stack 1 -config config-mn -test tests -group smoke

Interrupting or Cancelling Test Execution

Ctrl-C will exit the current test and flag an error.

Ctrl-Z will exit the test script and return control to the shell. Note that there may still be tests running in background.

Output Formats

Test output can be exported to jUnit/xUnit format for import into Jenkins or for another utiiltyl by using the -xunit optional argument.

Sample XML output file:

xUnit XML

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="test script template" tests="4" errors="1" failures="1" time="4.01295185089" timestamp="2015-06-02T15:26:06" hostname="0.0.0.0" package="autotest" >
<testcase name="Successful test" time="1.0016348362">
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Failed test" time="1.00292992592">
<failure>
Traceback (most recent call last):
  File "./script_template.py", line 66, in test1
    self.assertEqual(0,1, 'Dummy failed assert')
AssertionError: Dummy failed assert
</failure>
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Erred test" time="1.00141096115">
<error>
Traceback (most recent call last):
  File "./script_template.py", line 78, in test1
    x = y[0] # assign an undefined variable
NameError: global name 'y' is not defined
</error>
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
<testcase name="Successful test" time="1.00119400024">
<system-out>
</system-out>
<system-err>
</system-err>
</testcase>
</testsuite>
</testsuites>

Test Logging

The test framework provides for logging of the test activity using the logging plugin stream-monitor.

Logging is displayed in a number of ways:

Test case activity logging on Jenkins - each test case executed via Jenkins will generate an entry under the "Test Result" link in the test results page that is sorted alphabetically by test case name
Test console log - verbose output form the test script is stored under the Jenkins link "test_console.log" in the test results page
log_output directory - a directory for each run is created in the running directory
Appliance logs - system logs and system state will be stored on a log server after each test run.

BrownBags

Browser not supported