Unlocking the power of GitOps for network automation

In rapidly developing technologies, especially cloud technologies, the flexibility and at the same time reliability of the tools and techniques we use to manage infrastructure, including networks, become extremely important.

One approach that is becoming more and more popular is GitOps. Why should you consider this technique? How is it best applied in the context of network automation?

Definitions

Before providing detailed explanations of why and how GitOps can be used for network automation, let’s define the basic terms.

What is GitOps

Version control and declarative configuration are the foundations of GitOps, which can be used not only in software engineering, but also in infrastructure as a code or automated network management. In short, we can define GitOps as an approach, in which:

• Git repositories are a single Source of Truth (sometimes also called Source of Intent). 
• Code for infrastructure or configuration is expressed declaratively.

Besides these basic principles, there are two more in OpenGitOps      , in which GitOps is described as a:

• Desired state declared by code pulled automatically by software agents (controllers)
• Software agents (controllers) continuously reconcile system state

Both these rules may be debatable as there are two possible approaches - push and pull      . In typical CI/CD pipelines, the push style is used because it’s simple, flexible and efficient. In the pull style approach there is a need to have a controller, which continually polls Git repositories and applies changes in the environment. A huge advantage of the pull style is that controllers continuously reconcile state, so even if something is alternated manually, the changes should be reversed according to what is defined in the Git repository.

The differences between both approaches are presented in the following diagram.

More detailed information about GitOps, and  comparison to DevOps, can be found in our blog article GitOps vs. DevOps — what is the difference?. In another article GitOps — which tools should you choose?) a selection of tools are presented for application in everyday work. 

Network automation - a brief overview

Network automation is a process in which software is used to automate network management , configuration and security provisioning. It aims to optimize network-related tasks, reduce manual activities and enhance quality. Network automation involves the use of scripting and programming languages as well as dedicated tools to perform repetitive networking tasks, such as:

• network device configuration management, 
• provisioning and decommissioning of virtual network resources,
• security policy enforcement,
• backup,
• monitoring and alerting.

There are several benefits of network automation, including:

• consistency, as usage of configuration templates reduces human errors,
• scalability, as using scripts we can easily scale out/in the environment,
• efficiency, as manual tasks are automated,
• cost-savings, as operational costs for non-automated tasks are reduced.

More information about network automation with Python can be found in the article Python Nornir for simplifying network automation: code examples and Network automation – why does it matter?.

Network automation with GitOps

While GitOps is most often used with Kubernetes, its principles can be adapted to other environments. The following sections, present three approaches for implementing GitOps for network automation:

• in an environment with K8s,
• without a K8s cluster,
• in hybrid environments with a K8s cluster and additional services / infrastructure working on-prem or in cloud.

For every kind of approach, besides high-level explanation, in the chapter with examples there will be listed tools, which can be used, and sample commands and configuration files.

GitOps in Kubernetes environments 

In the diagram there is a simplified GitOps flow, beginning when the developer / user changes the source code and does a Git commit. The repository contains the K8s configuration files related to application settings and Kubernetes networking preferences. After pushing changes to the remote Git repository, GitOps controller, working in pull mode, detects the differences between what is declared and what is deployed and applies the changes:

• in every application working in the cluster, which is specified in Git repository,
• in network settings for the CNI plugin used in K8s.

GitOps in non-Kubernetes environments

In environments based on virtual machines or on-premises hardware there are limited opportunities to apply GitOps as it is defined by OpenGitOps. Nevertheless, in the push style, there is a possibility to successfully follow basic GitOps principles - version control and declarative configuration - but it requires more work to implement, especially at the beginning when custom scripts need to be prepared.

The CI pipeline is responsible for validation and testing code before executing the CD pipeline, which needs to be developed in such a way to allow execution of tools such as:

• Ansible playbooks,
• Terraform code,
• Any other tools used as IaC (infrastructure as code) or CaC (configuration as code).

GitOps in hybrid environments

In environments which contain Kubernetes clusters and other resources or services, e.g. from a public cloud, there is the possibility of using the pull style for GitOps and at the same time define infrastructure as code, e.g. using Terraform and HCL language. Weave GitOps Terraform Controller (TF-controller      ) is an example of such a solution, where the controller is running inside the K8s cluster, but resources created by Terraform can be defined outside the cluster. 

GitOps best practices

Regardless of the approach (push vs. pull), while preparing GitOps setup we need to remember a few best practices:

• Testing code (e.g. before merging to the main branch, depending on the tool for IaC / CaC, we can use native commands like terraform validate or additional tools like Terratest).
• Code review for all changes made in pull requests.
• Secrets management (internally in CI/CD platforms like Azure DevOps or Jenkins, or externally like Vault from Hashicorp).
• Monitoring configuration (like Prometheus or Grafana).
• Automatic validation steps or tests (if possible in OPA tools and the Rego language).
• Documentation in README.

GitOps tools and setup examples

GitOps in Kubernetes environments

The first step is to set up version control configuration and create Git repositories, e.g. on GitHub. 

If you don’t have an existing Kubernetes cluster, then it has to be provisioned and ready before going to the next stage. 

The third step is to install the CNI plugin. From a network point of view, specific commands and configuration depend on the networking solution, e.g.:

• Calico      ,
• Flannel      , 
• Weave Net      .

As an example below there are presented commands required to setup Calico as a CNI plugin:

1. (optionally) Install calicoctl on your local machine if you want to check, for example, deployed network policies or verify other Calico settings.

2. Install the Tigera operator, which is responsible for managing the life cycle of Calico (installation, upgrades, etc.)

   https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml      

3. Create custom resource to configure Calico:

   https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml      

The next step is to choose the GitOps tool. There are several possibilities designed specifically for Kubernetes, such as:

• ArgoCD      , 
• Flux      , 
• Jenkins X      . 

Depending on the GitOps tool there are different ways to integrate with the Kubernetes cluster, nevertheless all of them pull the desired state from the Git repository and apply it to the K8s cluster.

As an example below there are listed the actions needed to set up Flux for GitHub       :

1. Install

   Flux CLI

   on your local machine.
2. Prepare GitHub personal access token (PAT) with administration permissions and export PAT as an environment variable.
3. Check Kubernetes cluster by command (run pre-installation checks): 
4. Install Flux into cluster, e.g. 
5. Check Flux installation:

When we have the Kubernetes cluster with Calico as a CNI plugin and Flux as GitOps tools installed, the next step is to start configuring the network settings. As this article is not about networking in Kubernetes, the below example lists only options possible in Calico:

• BGP peering      ,
• IP address management (IPAM),      
• Network policies      .  

As an example below there are the activities required to deploy network policies:

1. Organize the structure of your Git repository (you can reuse the document “Ways of structuring your repositories”      ) and prepare a dedicated folder for network settings.
2. Prepare a simple global network policy, which will allow DNS (UDP protocol, port 53) to block traffic to a specific IP address 1.1.1.1 and allow all other TCP traffic:
3. Commit changes and push them into the Git repository.
4. After Flux reconciliation, changes should be deployed into your Kubernetes cluster. If not, you can force reconciliation and check logs using the commands:

GitOps in non-Kubernetes environments

Multiple tools exist to provision infrastructure as code or do configuration as code in a cloud environment or on-prem. Some of the most popular are:

• Terraform      ,
• Pulumi      ,
• AWS CloudFormation      , 
• Ansible      ,  
• Chef      ,
• Puppet      . 

For every tool listed above there are a plethora of extensions available. Only Terraform delivers providers for multiple vendors of network devices, e.g.:

• Cisco      ,
• CheckPoint      ,
• FortiGate      ,
• Palo Alto Networks      . 

As an example to demonstrate the GitOps approach in a non-Kubernetes environment, we can set up Jenkins as a CI/CD tool and use Terraform to configure a Palo Alto Networks firewall:

1. Install and configure Jenkins      
2. Prepare firewall configuration, e.g. simple address objects        defined in main.tffile e.g.:
3. Store Terraform state remotely, e.g. in an S3 bucket (if it doesn’t exist, please create it) by defining it in versions.tf:
4. Create secrets, e.g. pano_hostname and pano_api_key, which contains credentials       for the Palo Alto Networks device. 
5. Create pipeline      , which can be based on the below Jenkinsfile :
6. Commit changes and push them to the Git repository

In the above example, a single CI/CD pipeline was created. This article is not focused on testing CaC or IaC but in a real environment splitting into two pipelines is recommended - the first responsible for testing and validation of changes, preparing packages to be deployed (CI), the second responsible for deployment (CD).

GitOps in hybrid environments

As an example of using this approach in a hybrid environment, below there are presented commands to deploy the TF-controller and then configure the infrastructure in AWS cloud (as a non-Kubernetes environment):

1. Install TF-controller:

   https://raw.githubusercontent.com/weaveworks/tf-controller/main/docs/release.yaml      

2. Configure AWS credentials:

3. Define infrastructure in HCL language, e.g IAM role and policy:

4. Commit changes and push them to the Git repository.

Summary

The GitOps approach is mainly used in Kubernetes environments, but there are no contraindications to use it on different platforms, as was presented in the examples. The two major styles (push and pull) give you flexibility with designing your GitOps platform. Critically, in the context of network automation, GitOps helps to achieve:

• Consistency of network configurations by deploying changes by code and avoiding manual errors.
• Collaboration as all network engineers can use Git repositories as a source of truth.
• Traceability and auditing as all changes are in Git history.
• Efficiency as CI and CD pipelines automate repetitive jobs. 

If you are still hesitating about using the GitOps approach in your work, maybe Kelsey Hightower’s quote       will convince you: 

“GitOps is the best thing since configuration as code. Git changed how we collaborate, but declarative configuration is the key to dealing with infrastructure at scale, and sets the stage for the next generation of management tools.”