...
Terraform takes two input files to automate the infrastructure provisioning and produces a state file at the end of the automation.
Variables file (variable.tf) - Input file : Run time tunables that helps to customize the infrastructure configuration. The provider and the resource blocks in the main.tf file can be configured by changing the values in variables.tf file.For example, if you want to change the aws_instace type from t2.small to t2.micro, set the TF_VAR with the appropriate values. Other resource-specific values like aws_region, aws_ami, vpc_id and the subnet can also be changed the same way by editing the respective TF_VAR environment variables. List of variables taken from the variable.tf file and the list of ENVs read are listed below:
- TF_VAR_aws_region - AWS Region where the cluster needs to be provisioned
- TF_VAR_aws_ami - Image to bring up the master and worker node EC2 instances. Based on the region, the AMI corresponding to Ubuntu Server 18.04 LTS needs to be configured.
- TF_VAR_aws_instance - AWS Instance type eg. t4g.medium
- TF_VAR_vpc_id - VPC ID of a pre-existing VPC
- TF_VAR_aws_subnet_id - Subnet ID of a pre-existing subnet.
- TF_VAR_access_key - AWS IAM User Access Key
- TF_VAR_secret_key - AWS IAM User Secret Key
- TF_LOG - Log level while execute terraform templates. Supported values - TRACE, DEBUG, INFO, WARN, or ERROR.
- TF_LOG_PATH - Path of the file to redirect the terraform execution logs.
- terraform configuration file (main.tf) - Input file : The Terraform configuration file contains the workflow and automation scripts to create the microk8s cluster.
- worker_user_data.tmpl - user data for the worker nodes : This file maintained by the blueprint internally to dynamically configure the join token on the worker nodes.
- terraform state file: Terraform maintains the current state of the infrastructure in the state file. The state file remains empty until the first terraform initialization. The stateful is used for further updates or tear down of the cluster.
...
- The user provides the VPC ID, region, instance type, ami, subnet ID and the IAM access/secret as the input in the variable.tf file. The values are configurable at the time of applying the template.
- The variable ‘count’ indicates the number of worker nodes to be provisioned. A count of 2 brings up a 2 node cluster.
- We also need the PEM private key file to remote SSH into the EC2 instances. This file needs to be present in the same location as the main.tf and vairable.tf files.
- The security group is automatically provisioned based on the VPC and the Subnet ID provided.
- Currently, load balancer is not provisioned. Any workloads deployed in the cluster can be accessed via the Kubernetes Node port.
...
- Provision master node. The template executes EC2 user_data on the master node that uses snap package manager to install microk8s.
The user_data in the main.tf file installs the microk8s inside the EC2 instance.
Code Block language java #!/bin/bash sudo su apt update -y >> microk8s_install.log apt install snapd -y >> microk8s_install.log snap install core >> microk8s_install.log export PATH=$PATH:/snap/bin snap install microk8s --classic --channel=1.20/stable >> microk8s_install.log microk8s status --wait-ready microk8s enable dns >> microk8s_install.log microk8s add-node > microk8s.join_token microk8s config > configFile-master
Once the microk8s master is installed on the first node, the template then does a remote SSH command to the master node and generates a token by executing the command - ‘microk8s.add-node’ This makes use of the private key file in the local directory to execute the remote SSH command. The token generated in this step is used to join the remaining nodes to the cluster. The following describes how a connection block is configured in main.tf file to perform a remote exec to the master node. terrform.pem is the private file in the local client system to remote exec to the master or worker nodes.
Code Block language java connection { host = self.public_ip type = "ssh" user = "ubuntu" password = "" private_key = "${file("terraform.pem")}" } .... .... provisioner "remote-exec" { inline = ["until [ -f /microk8s.join_token ]; do sleep 5; done; cat /microk8s.join_token", "sudo sed -i 's/#MOREIPS/IP.7 = ${self.public_ip}\\n#MOREIPS/g' /var/snap/microk8s/current/certs/csr.conf.template", "sudo sleep 1m", "sudo microk8s stop", "sudo microk8s start" ] }- Copy the generated token on the remote machine to the local machine using the terraform ‘datasource’ plugin.
- Now provision the worker nodes and install microk8s on the remaining nodes
Use the local datasource to read the join token and add the worker nodes to the master node using the command ''microk8s.join_token". Following code block explains how a worker node is added to the cluster using remote exec.
Code Block language java provisioner "remote-exec" { inline = ["until [ -f /microk8s.join_token ]; do sleep 5; done; cat /microk8s.join_token"] }
...