Create a Kubernetes Cluster

Use Terraform to set up a Kubernetes cluster in your Oracle Cloud Infrastructure account.

Key tasks include how to:

Copy your existing scripts from Terraform tutorials.
Edit existing scripts for reuse.
Write new scripts for a Kubernetes cluster.

A diagram of the components needed to create a Kubernetes cluster with Terraform. From a local Linux environment, the user connects to an Oracle Cloud Infrastructure account and creates a cluster. The cluster creation includes creating network resources. The network resources include one private regional subnet for worker nodes. These nodes are compute nodes. There is also a public regional subnet created to use for load balancers. In this example, each regional subnet spans three availability domains: AD1, AD2, and AD3. The public subnet is connected to the internet with a two-way arrow. The private subnet has two one-directional, outgoing arrows, one to the internet and the other to Oracle services network.

For more information, see:

Before You Begin

To successfully perform this tutorial, you must have the following:

Requirements

A paid Oracle Cloud Infrastructure account. See Request and Manage Free Oracle Cloud Promotions.
A MacOS, Linux, or Windows computer.
Terraform tutorial resources:
- Go through all the steps in:
  - Set Up a Simple Infrastructure with OCI Terraform
- Keep the scripts you created in the following directory:
  - $HOME/tf-simple-infrastructure/

1. Gather Required Information

Gather information for the compute instances in the node pool.

Get Node Shape

This tutorial uses VM.Standard2.1 for the compute instances in the node pool..

Save the <node-shape> VM.Standard2.1 in your notepad.

To learn more about the shape, go to Standard Shapes .

Get Image ID

In the Console navigation bar, find your region.
See Working in Regions.
Go to Image Release Notes.
Select Oracle Linux 7.x.
Select the latest Oracle Linux 7.x-<date>. Don't select any images labeled Gen2-GPU.
Copy the Image OCID for your region.
Save the <image-ocid> in your notepad.

Note

Ensure that you select a commercial OCID without gov in its OCID.

2. Copy Existing Scripts

Copy scripts created at Set Up a Simple Infrastructure with OCI Terraform. Then, remove the scripts and outputs related to the compute instance. In the next section, you declare a node pool with compute instances.

Copy the Scripts

In your $HOME directory, create a directory called tf-cluster and change to that directory.
```
mkdir tf-cluster
```
```
cd tf-cluster
```
Copy all the files ending in .tf from the tf-simple-infrastructure directory.
```
cp ../tf-simple-infrastructure/*.tf .
```
Confirm that you have the following files in your directory.
```
ls
```
```
availability-domains.tf
compartment.tf
compute.tf
outputs.tf
private-security-list.tf
private-subnet.tf
provider.tf
public-security-list.tf
public-subnet.tf
vcn-module.tf
```
Note

Don't copy the state files (terraform.tfstate or terraform.tfstate.backup). These files contain the state of resources for their current directory. After you run the scripts in this new directory, you get a new state file.

Remove Irrelevant Scripts

Remove the compute.tf file from the tf-cluster directory.
```
rm compute.tf
```

In the outputs.tf file, remove all the outputs for the compute instance.



# Outputs for compute instance

output "public-ip-for-compute-instance" {
  value = oci_core_instance.ubuntu_instance.public_ip
}
output "instance-name" {
  value = oci_core_instance.ubuntu_instance.display_name
}

output "instance-OCID" {
  value = oci_core_instance.ubuntu_instance.id
}

output "instance-region" {
  value = oci_core_instance.ubuntu_instance.region
}

output "instance-shape" {
  value = oci_core_instance.ubuntu_instance.shape
}

output "instance-state" {
  value = oci_core_instance.ubuntu_instance.state
}

output "instance-OCPUs" {
  value = oci_core_instance.ubuntu_instance.shape_config[0].ocpus
}

output "instance-memory-in-GBs" {
  value = oci_core_instance.ubuntu_instance.shape_config[0].memory_in_gbs
}

output "time-created" {
  value = oci_core_instance.ubuntu_instance.time_created
}

3. Create Scripts

Create scripts for a cluster, a node pool, and to print outputs.

Declare a Cluster

Create a file called cluster.tf.

Add the following code to cluster.tf.

Replace <your-cluster-name> with a name of your choice. Example: tf-cluster.

# Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_cluster

resource "oci_containerengine_cluster" "oke-cluster" {
    # Required
    compartment_id = oci_identity_compartment.tf-compartment.id
    kubernetes_version = "v1.21.5"
    name = "<your-cluster-name>"
    vcn_id = module.vcn.vcn_id

    # Optional
    options {
        add_ons{
            is_kubernetes_dashboard_enabled = false
            is_tiller_enabled = false
        }
        kubernetes_network_config {
            pods_cidr = "10.244.0.0/16"
            services_cidr = "10.96.0.0/16"
        }
        service_lb_subnet_ids = [oci_core_subnet.vcn-public-subnet.id]
    }  
}

Save the cluster.tf file.

Explanation

At Argument Reference (oci_containerengine_cluster), find all required arguments:

compartment_id
kubernetes_version
name
vcn_id

To navigate to this URL

To navigate to Argument Reference (oci_containerengine_cluster):

Go to Oracle Cloud Infrastructure Provider.
In the Filter box on the upper left, enter container engine.
Results are returned for both data sources and resources.
Under Container Engine, go to Resources and select oci_containerengine_cluster.
Select Argument Reference.
Argument Reference opens.

Construct a resource block:

Declare a resource block with the keyword: resource
Add a label for resource type: "oci_containerengine_cluster"
Add a label for a local name (your choice):
- The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit.
- Example: "oke-cluster"
Inside the code block, provide a value for the required arguments:
- compartment_id: Point to the compartment declared in compartment.tf: oci_identity_compartment.tf-compartment.id
- kubernetes_version: This tutorial uses version v1.21.5. You can check the Quick Create option in the Console for the latest version.
- name: Assign a name of your choice.
- vcn_id: Point to the compartment declared in vcn-module.tf: module.vcn.vcn_id
A required argument doesn't have a default value.
Provide values for the following optional arguments to override their default values.
- add_ons: Assign true or false for the following arguments:
  - is_kubernetes_dashboard_enabled
  - is_tiller_enabled
- kubernetes_network_config: Assign a CIDR block as a string for the following arguments:
  - pods_cidr
  - services_cidr
  Note
  
  The CIDR block for the pods must not overlap with the worker node and load balancer subnet CIDR blocks.
  
  The CIDR block for the Kubernetes service must not overlap with the VCN CIDR block.
  
  The example code in this tutorial uses the same CIDR blocks as the Quick Create option in the Console.
  
  For more explanation, see CIDR Blocks and Kubernetes Engine (OKE).
- service_lb_subnet_ids: Assign the public subnet you declared in public-subnet.tf
  Note
  
  The argument, service_lb_subnet_ids accepts a list of subnet ids:
  - Even if you have one subnet, use square brackets to denote a list.
  - Example: [oci_core_subnet.vcn-public-subnet.id]

Declare a Node Pool

Create a file called node-pool.tf.

Add the following code to node-pool.tf.

Replace the following fields with the information you gathered in section one:
- <node-shape> with VM.Standard2.1
- <image-ocid>
Replace the following field with the name you chose when you declared a cluster:
- <your-cluster-name>

# Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_node_pool

resource "oci_containerengine_node_pool" "oke-node-pool" {
    # Required
    cluster_id = oci_containerengine_cluster.oke-cluster.id
    compartment_id = oci_identity_compartment.tf-compartment.id
    kubernetes_version = "v1.21.5"
    name = "pool1"
    node_config_details{
        placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        } 
        placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[1].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        }
         placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[2].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        }
        size = 3
    }
    node_shape = "<node-shape>"

    # Using image Oracle-Linux-7.x-<date>
    # Find image OCID for your region from https://docs.oracle.com/iaas/images/ 
    node_source_details {
         image_id = "<image-ocid>"
         source_type = "image"
    }
 
    # Optional
    initial_node_labels {
        key = "name"
        value = "<your-cluster-name>"
    }    
}

Save the node-pool.tf file.

Explanation

At Alarm Reference (oci_containerengine_node_pool), find all required arguments:

cluster_id
compartment_id
kubernetes_version
name
node_config_details
- placement_configs
  - availability_domain
  - subnet_id
node_shape
node_source_details
- image_id
- source_type

To navigate to this URL

To navigate to Alarm Reference (oci_containerengine_node_pool):

Go to Oracle Cloud Infrastructure Provider.
In the Filter box on the upper left, enter container engine.
Results are returned for both data sources and resources.
Under Container Engine, go to Resources and select oci_containerengine_node_pool.
Select Argument Reference.
Argument Reference opens.

Construct a resource block:

Declare a resource block with the keyword: resource
Add a label for resource type:
```
"oci_containerengine_node_pool"
```
Add a label for a local name (your choice):
- The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit.
- Example:"oke-node-pool"
Inside the code block, provide a value for the required arguments:
- cluster_id: Point to the cluster declared in cluster.tf:
```
oci_containerengine_cluster.oke-cluster.id
```
- compartment_id Point to the compartment declared in compartment.tf:
```
oci_identity_compartment.tf-compartment.id
```
- kubernetes_version: This tutorial uses the same version as the Console Create Cluster wizard.
- name: Assign a name of your choice. The Console Create Cluster wizard uses the name pool1.
- node_shape: Enter information you gathered in section one.
- node_source_details:
  - image_id: Enter information you gathered in section one.
  - source_type: Set to "image".
Provide values for the following optional arguments to override their default values.
- initial_node_labels: Assign key/value pairs for the nodes.
  - key: Assign a key of your choice. The Console Quick Create option creates the key "name".
  - value: Assign a value for the key. The Console Quick Create option assigns "<your-cluster-name>" to the "name" key.

Add Outputs

In this section, you declare outputs for the cluster and the node pool.

Add the following code to outputs.tf.

# Outputs for k8s cluster

output "cluster-name" {
  value = oci_containerengine_cluster.oke-cluster.name
}
output "cluster-OCID" {
  value = oci_containerengine_cluster.oke-cluster.id
}
output "cluster-kubernetes-version" {
  value = oci_containerengine_cluster.oke-cluster.kubernetes_version
}
output "cluster-state" {
  value = oci_containerengine_cluster.oke-cluster.state
}

# Outputs for k8s node pool

output "node-pool-name" {
  value = oci_containerengine_node_pool.oke-node-pool.name
}
output "node-pool-OCID" {
  value = oci_containerengine_node_pool.oke-node-pool.id
}
output "node-pool-kubernetes-version" {
  value = oci_containerengine_node_pool.oke-node-pool.kubernetes_version
}
output "node-size" {
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
}
output "node-shape" {
  value = oci_containerengine_node_pool.oke-node-pool.node_shape
}

Save the outputs.tf file.

Explanation

For cluster outputs, see Attribute Reference at oci_containerengine_cluster.
For node pool outputs, see Attribute Reference at oci_containerengine_node_pool page.

Outputs for List Items

Usually list attributes are plural (end in s).
List attribute example for node pool:
- node_config_details
To output all the attributes in a list, use the list attribute by itself, without any brackets.

Example:

output "node-configuration-details"{
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details
}

Sample output:

node-configuration-details = [
  {
    "placement_configs" = [
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-1"
        "subnet_id" = "ocid1.subnet.xxx"
      },
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-2"
        "subnet_id" = "ocid1.subnet.xxx"
      },
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-3"
        "subnet_id" = "ocid1.subnet.xxx"
      },
    ]
    "size" = 3
  },
]

To output or call an item from a list:
- Use the following format:
  <list-attribute-name>[index].<attribute-from-list>
- Replace [index] with:
  - [0] for the first item.
  - [1] for the second item.
  - ...
  - [n] for the (n+1)^th item.

Example:

Value for the size attribute:

output "node-size" {
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
}

4. Run Scripts

Run your Terraform scripts to create a compartment, a virtual cloud network, a Kubernetes cluster, and a node pool.

Initialize

Initialize a working directory in the tf-cluster directory.

terraform init

Example output:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/oci versions matching ">= 4.41.0"...
- Installing hashicorp/oci v4.59.0...
- Installed hashicorp/oci v4.59.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Check the contents of the tf-cluster directory.
```
ls -a
```
Note

Troubleshooting:
- After running terraform init
- error message: Failed to query available provider packages:
  - If you are on a VPN, check your proxy settings.

You now have a folder called .terraform that includes the plugins for the oci provider.

Plan

Create an execution plan:
```
terraform plan
```

Review the changes that Terraform plans to make to your account.

Example output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

Plan: 14 to add, 0 to change, 0 to destroy.

Apply

Run your Terraform scripts:

terraform apply

When prompted for confirmation, enter yes, for your resources to be created.

It might take 15 minutes or more for the cluster to be created. After Terraform creates the resources, review the output in the terminal.

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Outputs:
...
cluster-OCID = ocid1.cluster.xxx
cluster-kubernetes-version = "v1.21.5"
cluster-name = <your-cluster-name>
cluster-state = ACTIVE
...
node-pool-OCID = ocid1.nodepool.xxx
node-pool-kubernetes-version = "v1.21.5"
node-pool-name = "pool1"
node-shape = "VM.Standard2.1"
node-size = 3
...

Troubleshooting

401 errors - (Service error:NotAuthenticated):
- You have an incorrect value for one of the following:
  - tenancy OCID
  - user OCID
  - fingerprint
  - RSA private key (the path or the key)
no such host:
- You have an incorrect value for the following:
  - region identifier

References:

What's Next

Congratulations! You have created a Kubernetes cluster using Terraform, in your Oracle Cloud Infrastructure account.

Now that you have a Kubernetes cluster, try Kubernetes tutorials at Developer Tutorials.

To explore more information about development with Oracle products, check out these sites:

Oracle Cloud Infrastructure Documentation

Create a Kubernetes Cluster

Before You Begin

1. Gather Required Information

2. Copy Existing Scripts

3. Create Scripts

Outputs for List Items

4. Run Scripts

What's Next