Software testing is an integral part of the software development life cycle (SDLC). Simply put, the primary goal of software testing is to find bugs and fix them as fast as possible, preferable at the early stages of the SDLC, and ensure bug-free software or software products. There are many different types of testing, but for this article's purpose, we will focus on functional testing.
By reading this article, you'll learn about functional testing in terms of network-related projects, what it is and whether it is actually needed.
We will also introduce you to the tools that are commonly used to automate functional testing in the QA (Quality Assurance) process.
What is functional testing—a brief introduction
Functional testing refers to the type of tests which are performed to verify that a product's functionalities work as expected, that is, they give the expected outputs from specified inputs.
Functionalities can be described in product specifications, epics, user stories, etc. but some functions may also be undocumented. Therefore, the tester should understand the product being tested and work closely with the developers to properly decide which functionalities should be tested, how, and at what stage.
According to ISTQB (International Software Testing Qualifications Board), functional testing can be executed at any test level, i.e. unit, component/module, integration, system and acceptance tests.
In creating test cases, black box testing techniques can be used, which focus on the inputs and outputs of the tested objects, rather than focusing on their internal structures.
Why do we need functional testing?
Functional testing is crucial for a product's success because it ensures that the product behaves as expected (i.e. as described in product specification) and is working properly from the user's perspective.
Imagine a situation in which a product is tested only for non-functionality (for instance, only performance, stress, stability and usability tests are conducted), you would be sure that the product behaves in the "right way" (the product is fast, stable over the long term, user-friendly, etc.) but you would not know whether it does the “right thing“.
Therefore, functional testing should be conducted prior to non-functional testing so that you can be sure that you are checking, for example, the performance of a properly-functioning product.
Tools for automating network functional testing
The following solutions may be beneficial if you want to create automated functional tests for a network project (as well as for some other types of tests).
This list shows two full solutions, namely toolboxes for creating, running and analyzing tests (pyATS, NITA).
If you want more flexibility and don't want to use full "end-to-end" frameworks, you can also consider the other tools mentioned later to create your own toolbox. Such tools can be helpful in automating network-related stuff (Nornir, Ansible, NAPALM, Netmiko), creating test cases (PyTest, Robot Framework, PTF) or testing configuration of the network (Batfish).
Toolboxes
As mentioned earlier, the two open-source toolboxes that will be briefly described below are complete solutions that can be easily integrated with a CI/CD (Continuous Integration/Continuous Deployment or Continuous Delivery) process for your software.
pyATS
PyATS - Python Automation Test System - is a test framework based on Python and supports Python 3.4+.
This tool was created not only for functional network tests but also for executing sanity, solution, system, and scale tests for network solutions consisting of products ranging from routers/switches, to access points, firewalls and cable CPEs.
It was originally created and used internally by Cisco, but has been available to the public since 2017.
PyATS is the foundation of Cisco's Test Automation Solution.
Installing pyATS with PyPI (Python Package Index) will also install, among other elements:
- Genie - the standard pyATS library that provides methods for device interaction - connecting to a device, executing commands and getting parsed output from a device. It ensures platform-agnostic Python object models for features such as VRF, VXLAN, BGP, etc. (they are all listed in the Genie Docs) and pre-built reusable test cases.
- XPRESSO - a web UI dashboard allowing you to manage the test environment, run tests with both Jenkins and Docker-based execution back-ends, live-view of tests while they are running, and analyze the results.
The open-source pyATS is vendor, platform, feature and protocol independent, so it is used by many QA/DevOps/Network Engineers. Thus, the community is working on improvement and delivery of new functionalities by providing integration to other frameworks, for example, Robot Framework or Ansible in the form of libraries, extensions, plugins and APIs.
A great tool, pyATS allows preemptive and proactive network failure detection. But since it is a full framework, it can be difficult and time-consuming to use libraries other than those supported by developers and the pyATS community.
NITA
NITA - Network Implementation and Test Automation - is a containerized application used for automating the building and testing of networks. It depends on Docker CE and Docker Compose.
NITA can be seen as a toolbox that consists of five basic tools:
- NITA webapp - a Django-based web application which orchestrates Jenkins, Ansible and Robot Framework,
- Ansible - used for configuration change management,
- Jenkins server - responsible for automation and job control,
- Robot Framework - allows you to write different types of tests,
- Juniper's network automation libraries (PyEZ - Python library to remotely manage Junos devices) and others (Netbox).
NITA has been open source since 2020, but originally was developed in 2015 and used internally by Juniper Networks.
Similar to pyATS, NITA is vendor neutral and allows you to include other tools that you would like to use in testing or deployment of the network. Further, it also supports some traffic generators, such as IXIA or Spirent.
NITA is not provided as a package on PyPI and you have to install all components independently.
NITA is not as popular as pyATS. There are no much additional resources provided by the community and the only training that is available is a paid-for Juniper course Advanced Junos Platform Automation and DevOps (AJAUT) (as for August 2022).
Tools used for environment setup
The following open-source tools provide similar functionalities. In short, they allow you to interact with network devices, configure them, etc.
If you want to create your own toolbox (similar to the ones listed above), you will probably need to include some of these tools to prepare an environment for performing functional network testing.
Ansible
Ansible is an automation framework that can manage network devices in a multi-vendor environment. It allows you to configure the network stack, test and validate the existing network state, as well as discover and correct network configuration drift.
But Ansible can do much more. For example, according to the GitHub README it "handles configuration management, application deployment, cloud provisioning, ad-hoc task execution, network automation, and multi-node orchestration".
Ansible uses inventories and playbooks, which are defined in YAML files with Ansible DSL (domain specific language). Inventories contain information about network devices, while playbooks consist of tasks that will be executed (synchronously or asynchronously) on hosts defined in inventories.
Ansible comes with straightforward documentation which allows you to learn it quickly.
Tasks run sequentially and stop when an error is encountered. This makes troubleshooting easier.
Another advantage is that Ansible is agentless, so there is no need to install any agent on a remote system. It handles all master/agent communications with standard SSH or the Paramiko module.
For some people, the disadvantage of Ansible may be that it uses its own DSL, but some think that this makes it easier to learn.
There are also some known drawbacks to Ansible, such as: insufficient UI and enterprise support experience, no notion of state, limited Windows support, or that it's new to the market [^5](https://www.upguard.com/blog/top-5-best-and-worst-attributes-of-ansible) [^6](https://www.whizlabs.com/blog/ansible-advantages-and-disadvantages/).
Nornir
Nornir - the name of this tool was borrowed from Norse mythology, Nornir means the three goddesses ruling people’s fate.
Nornir is a Python automation framework that supports Python version 3.6.2+.
It can be easily installed from PyPI along with other useful plugins like: nornir-salt, nornir-netmiko and many others. Additionally, you can create your own plugin to satisfy your needs.
Nornir is a multithreaded framework that works on inventories, just like Ansible. Inventories are collections of data stored in a dictionary or separate YAML file. In network-related projects, an inventory contains information about network devices.
Nornir allows you to define and run tasks on all or a subset of hosts, defined in the inventory, and keeps track of all the threads.
Nornir can also be used in conjunction with pyATS, as shown in Tutorial: Combining Nornir with pyATS.
The previously mentioned nornir-salt plugin is worth checking out. Among other elements, it is comprised of TestsProcessor, which is a plugin that allows you to run tests and process their results, etc. For more information, see the article Testing your network with Nornir TestsProcessor.
One of the advantages for coding members of the team is that Nornir can be directly used in Python scripts to define the actions that will be performed on hosts. This makes Nornir easier to debug. It isn't another DSL, like Ansible, and this is the main difference between these two frameworks. If you are curious about more differences between the two, check out this article: Ansible vs. Nornir.
NAPALM
NAPALM - Network Automation and Programmability Abstraction Layer with Multi-vendor support - is a Python library that supports Python version 3.6+ (or 3.7+ for NAPALM version 4.0.0) and can be easily installed from PyPI.
This library was created by the author of Nornir, the tool described above.
NAPALM is a tool that provides vendor-neutral functions, used in Python script, to interact with network devices, allowing you to connect to a device, manipulate its configurations and receive data from it.
All this is delivered in a unified, vendor-neutral API, so it can be used in a single project that consists of various OS devices to configure all devices. The same method can be used to interact with devices with a supported OS, namely: Arista EOS, Cisco IOS, Cisco IOS-XR, Cisco NX-OS, Juniper’s Junos.
Another advantage of this tool is that it can be integrated with some automation tools like Ansible, by using an additional napalm-ansible plugin. This plugin contains three Ansible modules that use NAPALM to retrieve data or modify configuration on networking devices.
NAPALM can be combined with Nornir in Python scripts by using a nornir-napalm plugin, which provides five NAPALM methods (napalm_cli, napalm_configure, napalm_get, napalm_ping, napalm_validate) that could be used in Nornir tasks.
It also allows easy validation of deployment. All you need to do is create a YAML file containing information about the expected state of the device (for example, with information about BGP neighbors or the expected result of running a ping), and then run a method called compliance_report with that YAML file passed as an argument. After executing this method, you will receive a JSON-based report with detailed results [^7](https://napalm.readthedocs.io/en/latest/validate/index.html).
The downside is that it doesn't support all commands that can be executed on every supported OS. Before using NAPALM, it is advisable to examine the official documentation with information about supported devices and their supported commands.
Netmiko
Netmiko is a Python library that provides a simple way of interacting with multi-vendor platforms. It is based on the Paramiko library, which is a Python implementation of the SSHv2 protocol.
Netmiko provides SSH connections to the network device, but it also supports SCP (secure copy protocol), TELNET and serial connections.
Netmiko offers methods for setting configurations of the tested device by entering config mode or switching to privileged mode. You can execute commands on the device and gather output from executed commands in a string format.
Since the output is in string format, it can be difficult to work with such unstructured data. Fortunately, the community has enabled the use of Netmiko with TextFSM to get structured data in the form of a list or dictionary. For more information, check out this article: Netmiko and TextFSM.
For some examples of usage visit Netmiko's GitHub page.
In functional network testing Netmiko is sometimes combined with Nornir and a community-provided plugin nornir-netmiko to make it easier.
If you are not satisfied with this solution maybe try to use Paramiko itself or Scrapli (which could be combined with Nornir by using the plugin nornir-scrapli) or Plumbum.
There are other tools for automating the preparation of the network environment for functional testing, but we will not present all of them in this article. Another article detailing this topic will soon follow.
Tools used for testing
These three open-source tools are commonly used to create automated tests in networking projects. They are all based on the Python language.
For other interesting tools, refer to: The best QA automation tools that you should know.
PTF
PTF - Packet Testing Framework - a library based on OFTest - Open Flow Testing Framework. Both frameworks can be useful for writing dataplane tests.
PTF provides an easy way of defining the methods setUP and tearDown for tests that will be executed before and after each runTest method, because it is also based on Python unittest - the Python standard library. Thus, it also provides many other unittest functionalities, for instance assertion methods.
PTF contains a packet manipulator (by default it is a Scapy, but you can always change it), which allows you to easily define the packets you want to test, as well as receive, filter and analyze them. For example, you can easily sniff only selected packets and check the value of the chosen header fields of those packets.
Pytest
Pytest is one of the most common automation frameworks used for writing test cases. Pytest itself offers many useful functionalities, like parametrization, fixtures and assertions for writing tests (as mentioned in the article about automation tools). Pytest comes with good documentation and a large number of plugins. It is very popular among QA engineers, so there are many external resources and tutorials.
It is not designed only for network testing, as PTF is, so pytest doesn't have features related to networks by default. But, since it's a library used in Python, and Python has many libraries useful for interacting with network devices, tests written with pytest can be easily extended with extra functions to check network functionality.
Consider an example of using pytest, with additional Python libraries. First, use an automation tool like Ansible with the napalm-ansible plugin, which primarily configures the network environment and runs Python scripts containing tests. The test could use some supplementary libraries, such as Paramiko or Netmiko, that will connect to the device via SSH and transfer files to it via SFTP. The whole solution could use RPyC (Remote Python Call) library to execute commands on a remote host, and a packet manipulator such as Scapy. Finally tighten everything up with Jenkins.
It is not an easy tool for non-coding members of the team, but it gives you a lot of flexibility. You decide what library suits your personal preferences regarding setting up the environment and interacting with network devices.
Robot Framework
Robot Framework - is a KDT (keyword driven testing) framework that allows definition of test cases in a human-readable way by using special keywords, which is useful for non-coding members of the team to easily understand what each test does.
In terms of network testing there are some useful libraries:
- SSH Library - provides keywords that allows you to connect via SSH to network devices (e.g. a switch), execute a command on it, and receive output and transfer files and directories over SFTP. It could be useful for simple ping tests [^3](https://packetpushers.net/exploring-robot-framework-for-automated-testing/).
- Telnet - enables interaction with devices via a telnet connection.
- Arista Library - if you are working on a project with Arista EOS devices this library is perfect for creating simple tests. It is a wrapper for pyeapi - the Python library for eAPI - which enables interaction with the Arista EOS. Unfortunately it is available only for Python 2.7.
Other libraries that can be helpful in writing network tests using Robot are: Netmiko wrapper and ncclient library which is a wrapper for the Python library ncclient and can be used to make NETCONF connections to a network device. But both of these libraries are not mentioned in the Robot Framework documentation (also, the latest release of ncclient library was May 2017, as of August 2022). Thus, it could be difficult to troubleshoot problems encountered in using these plugins.
Robot Framework is so popular that it comes with many other solutions, for instance, it is a part of the NITA solution as a NITA Robot and is an extension plugin in pyATS.
But it's ready to use only for simple network testing. If you want to write a test for verifying, for example, the correctness of handling TTL values of IPv4 or the hop limit of IPv6 on each node of your network, you will probably need to add a binding to a packet manipulator or other tool responsible for network configuration. Such bindings are available, but they may not be enough and you can end up writing your own extension. This can be a problem for people inexperienced in programming, yet the Robot Framework is aimed at just such users.
Tools used for configuration verification
Perhaps, for some network projects, you might be interested in an open-source tool that can be used for testing the correctness of network configuration.
Batfish
Batfish is a Docker service tool which uses vendor-independent models created from snapshots to verify configuration even before deployment, and also after it.
From the official documentation: "A snapshot is the state of the network at a given time. A network may contain many snapshots, allowing you to understand the evolution of your network".
Snapshot is a folder consisting of two sub-folders:
- configs - folder with the configuration files of network devices,
- batfish - supplemental information.
After installing Batfish, it is necessary to download a Python client or Ansible roles for Batfish. These allow you to interact with Batfish in order to verify models.
Using Batfish questions you can not only validate the configuration of the network, but also, for example, perform packet forwarding, ACL, firewall, VXLAN and EVPN configuration and behavior analysis.
Batfish supports various vendors: A10, Arista, AWS, Cisco, Check Point, Cumulus Linux, F5 Big IP, Fortinet, Juniper, Palo Alto Networks, and SONiC.
Conclusion
Functional tests are crucial for the success of a software product, that is, for the quality of service perceived by end-users.
It is extremely important to properly identify the functionalities of the product to be tested and create a test plan. It is also vital to decide which tests should be automated and which tools should be used for this.
In this article, we have presented the most common automation tools used for testing networking solutions, including both ready-to-use toolkits and standalone tools that can be included in your toolkit.
