Building an automation module for a network observability solution

Automation

Cloud

Monitoring & alerting

Full stack development

Business background

A provider of a network monitoring and observability platform

Automate the onboarding process of virtual private clouds to the client’s network observability platform

Provide a team of developers and DevOps engineers with expertise in network monitoring and observability, cloud and automation
Design and build an entire solution from the ground up: architecture, technology stack, execution, testing
Enable configuration of customer's observability platform using Infrastructure-as-Code (IaC) approach
Ensure close cooperation with the client and a fast feedback loop
Organize regular, recorded demos of new features
Ensure the client’s feedback is promptly implemented

Reduced number of errors: the error-prone manual process of integrating a virtual private cloud with the client’s platform was fully automated.
Saved time and labor: integrating new cloud deployment with the client’s network observability platform is faster and simpler.
Reproducibility ensured: the IaC approach allowed for an easily reproducible and trackable configuration.
Ensured adherence to industry standards: the integration was implemented utilizing widely adopted technologies.

Prepared a configuration of AWS Cloud to be consumed by the client’s platform:
- IAM roles and policies that provide access to the client’s platform to EC2 and S3 resources
- S3 bucket to store flow logs for the client’s platform
- Flow Log configuration
Prepared configuration of GCP Cloud to be consumed by the client’s platform:
- Enabled flow logs for required subnets
- Flow log sink to filter logs from required subnets to PUB/SUB topic
- Created PUB/SUB topic
- Created PUB/SUB topic subscription
- Assigned roles to allow the client’s platform to pull logs from PUB/SUB
Prepared configuration of Azure Cloud to be consumed by the client’s platform:
- Turned on appropriate providers to enable flow logs and network watchers
- Network watcher and flow logs to get logs from resource group
- Granted Azure roles access to the client's platform
- Configured storage account
Prepared Terraform/Ansible modules to automatically integrate new virtual private clouds with the monitoring and observability platform.
Implemented Terraform provider (in Golang) allowing to automate configuration of flow log export from cloud deployment to the observability platform. The Terraform provider utilizes observability platform's API to implement changes.
Ensured that Terraform modules support automated export of AWS and GCP clouds to the client’s platform.
Implemented Golang SDK for the platform API.
Implemented unit tests.