Recently, I was working with an Azure design for a customer, and I needed to compare what my terraform scripts had planned for against my customer's Excel template.

GitHub Repository

Table of Contents

• Terraform Prep
• Python Script
  • Parsing arguments
  • Executing Terraform
  • Parsing the Terraform plan
  • Final Script
• Usage
• Final thoughts

If we were talking about a few resources with static names and properties, it wouldn't be too hard to comb through the terraform scripts and compare them against the Excel template. However, the customer's design had about 60+ resources, distributed in 15 Resource Groups.

So, I decided to do everything harder (now) so that it was easier (later), and started writing a Python script that would create a Terraform Plan, parse it and then generate an Excel file with all the resources grouped by type in different worksheets.

Terraform Prep

First of all, since my customer was using Azure as the Terraform backend and deploying via Azure DevOps agent, I had to do some prepping in the form of a Terraform override file so that I could execute my plan locally and get a full rundown as if there were no deployed resources.

Since it was mainly backend and provider, what I had to override, I named my file backend_override.tf, which I placed in the same folder where the rest of my Terraform scripts were.

The file ended looking something like this:

terraform {
  backend "local" {
    path = "./.local-state"
  }
}

provider "azurerm" {
  features {}
  client_id       = "00000000-0000-0000-0000-000000000000"
  client_secret   = "MySuperSecretPassword"
  tenant_id       = "10000000-0000-0000-0000-000000000000"
  subscription_id = "20000000-0000-0000-0000-000000000000"
}

Python Script

The script has two main parts: – Terraform planning – Plan flattener and Excel file creation

Parsing arguments

First, I had to write the base for the script. The script would receive a couple of arguments:

This arguments are parsed using argparse and saving the Terraform path to tfpath and the variables as a dict in vars. > I based this part on Sam Starkman's article and Laurent Franceschetti's gist

import argparse

def parse_var(s):
    items = s.split('=')
    key = items[0].strip()
    if len(items) > 1:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

vars = {}
parser = argparse.ArgumentParser(description="...")
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+')
parser.add_argument("--tfpath",
                    type=str,
                    required=True)
args = parser.parse_args()
vars = parse_vars(args.set)

tfpath = args.tfpath

Executing Terraform

To execute Terraform, I used the python_terraform library. I generated the plan, passing vars as an argument value for var, saved it first to plan.tfplan and the read it as a JSON into the variable plan. The code to generate the Terraform plan is really simple:

from python_terraform import *
import json

tf = Terraform(working_dir=tfpath)
tf.init()
tf.plan(out="plan.tfplan",var=vars)
json_data = tf.show("plan.tfplan",json=IsFlagged)

plan = json.loads(json_data[1])

Parsing the Terraform plan

Finally, I needed to parse the plan, especifically resource_changes. Since it contained everything from null and false, all the way to list and dict values, I decided to do a recursive function (flattener) that would iterate through all the resources.

The bit where I get the current directory, for the Excel file name, is based on vinithravit's answer over at StackOverflow.

import json
import xlsxwriter

def ofname(tfpath=".",extension=".xlsx"):
    os.chdir(tfpath)
    str1=os.getcwd()
    str2=str1.split('/')
    n=len(str2)
    name = str2[n-1] + extension
    return name

def flattener(jdata,row,column,worksheet,itemc):
    if isinstance(jdata,dict):
        for k,v in jdata.items():
            if isinstance(v,dict) or isinstance(v,list):
                worksheet.write(row,column,k)
                if isinstance(v,list) and len(v) == 1 :
                    row = flattener(v[0],row,column+1,worksheet,len(v))
                else:
                    row = flattener(v,row,column+1,worksheet,len(v))
            else:
                worksheet.write(row,column,k)
                worksheet.write(row,column+1,v)
                row = row + 1
    else:
        for v in jdata:
            if isinstance(v,dict) or isinstance(v,list):
                row = flattener(v,row,column,worksheet,len(v))
            else:
                worksheet.write(row,column,v)
                row = row + 1
    return row

classed = {}

for rc in plan['resource_changes']:
    classed[rc["type"]] = {}

for rc in plan['resource_changes']:
    rc_dict = rc['change']['after']
    rc_dict['address'] = rc['address']
    rc_dict['type'] = rc['type']
    classed[rc["type"]].update({rc['address']: rc_dict})

workbook = xlsxwriter.Workbook(ofname(tfpath))
cell_format = workbook.add_format()
cell_format.set_text_wrap()
cell_format.set_align("vcenter")
for type,data in classed.items():
    sheet = type[:31]
    worksheet = workbook.add_worksheet(sheet)
    worksheet.set_column(0,1000,42,cell_format)
    flattener(data,1,0,worksheet,0)
workbook.close()

Final Script

Putting everything together, plus a couple of minor adjustments (like the addition of tfcheck [a small bash script that I wrote to validate Terraform scripts] and rm plan.tfplan for cleaning up), the script ends up as follows:

from python_terraform import *
import json
import os
import xlsxwriter
import argparse

def parse_var(s):
    items = s.split('=')
    key = items[0].strip()
    if len(items) > 1:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

def ofname(tfpath=".",extension=".xlsx"):
    os.chdir(tfpath)
    str1=os.getcwd()
    str2=str1.split('/')
    n=len(str2)
    name = str2[n-1] + extension
    return name

def flattener(jdata,row,column,worksheet,itemc):
    if isinstance(jdata,dict):
        for k,v in jdata.items():
            if isinstance(v,dict) or isinstance(v,list):
                worksheet.write(row,column,k)
                if isinstance(v,list) and len(v) == 1 :
                    row = flattener(v[0],row,column+1,worksheet,len(v))
                else:
                    row = flattener(v,row,column+1,worksheet,len(v))
            else:
                worksheet.write(row,column,k)
                worksheet.write(row,column+1,v)
                row = row + 1
    else:
        for v in jdata:
            if isinstance(v,dict) or isinstance(v,list):
                row = flattener(v,row,column,worksheet,len(v))
            else:
                worksheet.write(row,column,v)
                row = row + 1
    return row

vars = {}
parser = argparse.ArgumentParser(description="...")
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+')
parser.add_argument("--tfpath",
                    type=str,
                    required=True)
args = parser.parse_args()
vars = parse_vars(args.set)

tfpath = args.tfpath

tf = Terraform(working_dir=tfpath)
tf.init()
tf.plan(out="plan.tfplan",var=vars)
json_data = tf.show("plan.tfplan",json=IsFlagged)

plan = json.loads(json_data[1])

classed = {}

for rc in plan['resource_changes']:
    classed[rc["type"]] = {}

for rc in plan['resource_changes']:
    rc_dict = rc['change']['after']
    rc_dict['address'] = rc['address']
    rc_dict['type'] = rc['type']
    classed[rc["type"]].update({rc['address']: rc_dict})

workbook = xlsxwriter.Workbook(ofname(tfpath))
cell_format = workbook.add_format()
cell_format.set_text_wrap()
cell_format.set_align("vcenter")
for type,data in classed.items():
    sheet = type[:31]
    worksheet = workbook.add_worksheet(sheet)
    worksheet.set_column(0,1000,42,cell_format)
    flattener(data,1,0,worksheet,0)
workbook.close()

os.system("tfcheck")
os.system("rm plan.tfplan")

I'm no Python expert, by any means, and I'm sure that this script can be improved and optimized.

Usage

Now, how do we use this script? Pretty easily. Once we've created our override file for Terraform, we simply run the script, passing the arguments we require:

python3 main.py --tfpath terraform/ --set location="Central US" testing=true

Final thoughts

It's been over a year since my last post... And what a year it has been! My baby daughter was born recently, I got a new job, my grandma died...

Anyway, I got several ideas I'd like to share with you, so I'll try to post more frequently.

See you soon! (Hopefully)

To complete our router, we need a DHCP server and DNS server. The DHCP server will assign IPs to our internal network, while the DNS server will resolve our queries to their corresponding IPs.

Since I like to have a private and ad-free experience when I surf, I'm running Pi-hole as a container. So the first thing I need to do is setup Docker using the official instructions:

Docker installation

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Use the following command to set up the stable repository.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update the apt package index, and install the latest version of Docker Engine and containerd, or go to the next step to install a specific version:

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io

Add user to the docker group.

sudo usermod -aG docker $USER

Logout and log in again to reload the permissions, so that we can run docker commands as the logged in user, and not as sudo.

Now, we'll install Docker Compose by running this command to download the current stable release of Docker Compose:

 sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

And apply executable permissions to the binary:

sudo chmod +x /usr/local/bin/docker-compose

Pi-hole container

We want our Pi-hole container to serve both DHCP and DNS. The following command will create a container named pihole using the latest Pi-hole image. It will use our internal interface lan0 (with its static IP address, 192.168.1.254) and will enable DHCP, using an IP range between 192.168.1.1 and 192.168.1.100:

docker run -d \
    --net=host \
    -e TZ='America/Chicago' \
    -e WEBPASSWORD: '{mysupersecretpassword}' \
    -e ADMIN_EMAIL='{email@domain.com}' \
    -e ServerIP='192.168.1.254' \
    -e INTERFACE='lan0' \
    -e DHCP_ACTIVE='true' \
    -e DHCP_START='192.168.1.1' \
    -e DHCP_END='192.168.1.100' \
    -e DHCP_ROUTER='192.168.1.254' \
    -e PIHOLE_DOMAIN='{homedomain.local}' \
    -v ./pihole/etc-pihole/:/etc/pihole/ \
    -v ./pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/ \
    --cap-add=NET_ADMIN \
    --restart=unless-stopped \
    --name pihole \
    pihole/pihole:latest

After setting up the container, since I'm using Ubuntu, I need to disable the caching DNS stub resolver that it Ubuntu has, since it will prevent Pi-hole from listening on port 53 (the port used by DNS requests). The stub resolver can be disabled with:

sudo sed -r -i.orig 's/#?DNSStubListener=yes/DNSStubListener=no/g' /etc/systemd/resolved.conf

After this, I need to change the nameserver settings, which currently point to the stub resolver (which we have just disabled). I need to point the /etc/resolv.conf symlink to /run/systemd/resolve/resolv.conf by running the following command:

sudo sh -c 'rm /etc/resolv.conf && ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf'

Finally, I need to restart systemd-resolved so that our changes are applied:

sudo systemctl restart systemd-resolved

Having done this, our Pi-hole container should start working as both a DHCP and DNS server. All we have to do now is disable our previous DHCP server, in my case my ISP modem.

With this, we have a fully working router with the added benefit of having a privacy and ad-blocking solution incorporated.

Stay tuned!

Other posts in this series:

• Introduction
• Part 1 – Network description
• Part 2 – IPv4 forwarding

One of the fundamental functions of a router is to forward traffic between our internal network and the internet.

In order to use the Raspberry Pi, or any Linux machine, as a router, the first thing we need to do is to enable packet forwarding.

Since I'm using Ubuntu 20.04 on my PiRouter, I can enable IPv4 forwarding instantly by executing, as sudo, the following command:

sysctl -w net.ipv4.ip_forward=1

| Remember to run these commands as root, or with sudo.

To make the change permanent, I need to modify /etc/sysctl.conf, adding the following line at the end of the file:

net.ipv4.ip_forward=1

This will ensure that IPv4 forwarding will be enabled on boot.

Now, I need to setup iptables rules in my firewall so that the PiRouter accepts and forwards the traffic it receives from my internal network/interface (lan0) to my external interface/internet (wan0).

This can be accomplished by executing the following commands:

iptables -A FORWARD -i lan0 -j ACCEPT

iptables -A POSTROUTING -o wan0 -j MASQUERADE

These rules will only be active until we reboot, so it's important to save them. I use iptables-persistent, which can be installed using the following command:

apt install iptables-persistent

After the installation, the setup will ask if you wish to save the current rules. By clicking Yes, we save our forwarding rules to /etc/iptables/rules.v4, and they'll be loaded every time our router boots.

Finally, we can ensure that the IPTables Persistent service is enabled and running by executing the following commands:

systemctl enable netfilter-persistent.service

systemctl start netfilter-persistent.service

Having done all of these steps, we now have a Raspberry Pi that will forward traffic between interfaces, however it's still not ready to be used as a router, since we're still missing a DHCP server and a DNS server. I'll be covering these key pieces in the next post.

Other posts in this series:

• Introduction
• Part 1 – Network description
• Part 3 – DHCP and DNS with PiHole

In this first part, I'll be going into a general description of my home network, as well as the services I'm running in my PiRouter.

Network description

Figure 1 – General network diagram

As you can see in the previous image, I've got my ISP modem, whose function is to connect with my ISP. From there, the PiRouter is setup in the modem's DMZ, so that it receives all traffic.

The PiRouter has 2 ethernet interfaces: 1. The Raspberry Pi integrated Gigabit Ethernet, which connects to the internal network (lan0). 2. A USB 3.0 Gigabit Ethernet adapter, which connects directly to the ISP modem (wan0).

Internally, I've got a server running Unraid on an old Supermicro server, which sports an Intel Xeon E5620 with 24GB of RAM. This server runs a couple of VMs and several containers.

I also have a Raspberry Pi 3B running Raspberry Pi OS with some containers used in home automation.

For WiFi, I use 2 TP-Link Deco M4 antennas. I need the power since I live in a two-story flat with a total surface of 175m² (1,880 sqft), with thick reinforced concrete walls.

My home automation runs on Home Assistant, using both WiFi and Zigbee. I'll probably go into this topic on another series of posts.

Other posts in this series:

• Introduction
• Part 2 – IPv4 forwarding
• Part 3 – DHCP and DNS with PiHole

A while back, I started using a Raspberry Pi 4B with 2GB of RAM as my home router.

The reason for doing this is because my ISP modem is pretty basic and limited. I also didn't want to have to reconfigure my network each time I changed modem or ISP.

Some benefits I got from using the RPI4 as a router were that I could setup an ad blocker, Pi-hole in my case, as well as adding a small UPS, for power outages, and LTE modem, for ISP outages (I haven't configured this last one).

After using my PiRouter (that's what I like calling it) for a while, I decided to clean everything up and standarize it, so that I could replicate it if/when I had to replace my Raspberry or the microSD card, as painlessly as possible.

In these series of articles I'll be detailing how I configured it using Git, Docker and Ansible, having learned the basics of Ansible from the great guides published by Jeff Geerling.

Other posts in this series:

• Part 1 – Network description
• Part 2 – IPv4 forwarding
• Part 3 – DHCP and DNS with PiHole

The idea for this blog is to share my ideas and projects on sustainability, cloud, electronics and automation.

I'll be sharing guides and projects, which I'll (probably) be uploading to GitHub.