ACS Engine — How to customize a Kubernetes Deployment On Azure
Introduction
There are many ways to provisioning Kubernetes in Azure. The most modern approach and the Microsoft recomended approach is to leverage AKS — the Azure Kubernetes Service. The goal of AKS is to remove the complexity of implementing, installing, and maintaining Kubernetes in Azure. It eliminates the burden of ongoing operations and maintenance. AKS assists with the provisioning, upgrading, and scaling of resources on demand, while keeping your applications online.
ACS Engine — Build your cluster the way you want
Oftentimes, developers and businesses want to fully manage their cluster. There are many reasons why you wish to manage Kubernetes yourself. It often comes down to the fact that developers and system administrators want absolute and full control over their cluster. That is what ACS Engine does.
ACS Engine — The Proper And Flexible Way To Deploy Self-Managed Kubernetes Into Azure
The most flexible way to deploy unmanaged Kubernetes into Azure is to use acs-engine. The Azure Container Service Engine (acs-engine) generates ARM (Azure Resource Manager) templates that are the fundamental declarative syntax Azure uses to provision everything. ACS Engine is a template generating tool, meaning that it outputs JSON-formatted declarations that conform to the Azure Resource Manager specification. Building these ARM Templates are quite tedious and greatly simplify the ability to deploy a Kubernetes cluster.
The process begins by the cluster administrator defining a cluster definition file that gets fed into the ACS engine. The ACS engine takes this file, reads it, and builds and ARM template file, which is a declarative syntax that the provisioning engines in Azure use to deploy infrastructure. Those provisioning engines are called the Azure Resource Manager.
The Azure Resource Manager enables you to work with the resources in your solution as a group. You can deploy, update, or delete all the resources for your solution in a single, coordinated operation. You use a template for deployment and that template can work for different environments such as testing, staging, and production. Resource Manager provides security, auditing, and tagging features to help you manage your resources after deployment.
The Azure command line tooling can take these ARM template files and build out a Kubernetes cluster based on the elements in the original cluster definition file.
You can read more about ACS engine here at the Github Repo — https://github.com/Azure/acs-engine.
Capabilities of ACS Engine
The purpose of ACS Engine is to support highly customizable Kubernetes clusters. Other cluster types are supported but for this post we are focused on Kubernetes. You can also define additional agent pools using ACS engine. An agent pool allows you to have a separate set of agent pods running in parallel in the same cluster as your original pods. An agent pool can be used to in-place upgrades both at the hardware and software levels.
ACS engine allows you to customize other aspects of your Kubernetes cluster, such as the VM type. Azure supports standard or premium VM’s, in addition to GPU and other VM types that are optimized for memory, compute, or disk. Another capability of ACS engine is to indicate the need for ScaleSets, which is in Azure technology that allows you to scale new agent nodes in the Kubernetes cluster, based on CPU, memory or by manual means. In addition, the ACS engine allows you to specify the operating system type or software version. You can also customize networking, such as the ability to make a virtual network that is bridged to on premise networks.
For the full set of capabilities, see the github repo mentioned above.
High level walk-through with ACS engine
For this example we are going to use a Linux workstation that is running in the Azure cloud. This jump box will be used to issue commands against the Azure resource manager so that we could build out our Kubernetes cluster. You can just as easily use your own Mac or Windows PC to do the same thing.
The overall steps include:
1. Provision an Ubuntu VM in Azure (not necessary if you already have a Mac, Windows, or Linux VM)
2. Install the Azure Command Line tooling on the VM Jump Box
3. Install acs-engine on the VM Jump Box
4. Use Azure Command Line tooling to provision your cluster
Provision a Jump Box VM in Azure
Before you can even work with ACS engine you will need a computer that has the Azure command line tooling installed.
Installing Azure Command Line Tools
You will need to install the Azure Command Line tooling. The Azure CLI is a command-line tool providing a great experience for managing Azure resources. The Azure CLI (Azure Command Line Interface) is designed to make scripting easy, query data, support long-running operations, and more.
Testing the Azure CLI Tools Listing your subscriptions
After installing the Azure command line tools, it makes sense to validate that they work correctly.
Before moving forward, let’s make sure the Azure command line tooling is working but addressing the tools to list the available subscriptions.
azureuser@btubuntu1:~$ az account list -o tableName CloudName SubscriptionId State IsDefault — — — — — — — — — — — -Windows Azure Internal Consumption AzureCloud d3fsdfsafc1-c74f-dddd-9e40-dsfsadf Enabled True
Get ACS Engine onto your Jump Box VM
The git command below will clone the entire ACS engine code base onto your jump box VM. A lot of the examples and helper files are included in this download.
$ git clone https://github.com/Azure/acs-engine.gitCloning into ‘acs-engine’…
remote: Enumerating objects: 104, done.
remote: Counting objects: 100% (104/104), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 58992 (delta 43), reused 41 (delta 12), pack-reused 58888
Receiving objects: 100% (58992/58992), 78.65 MiB | 23.08 MiB/s, done.
Resolving deltas: 100% (35458/35458), done.
Checking connectivity… done.
View kubernetes.json — the cluster definition file
There are a variety of examples for the cluster definition file. Kubernetes.json is the basic file, but you can view other examples that incorporate other features. We will return this file in a moment. First, we need to deal with SSH keys and Service Principals.
# From [/home/azureuser/acs-engine/examples]
$ lt k*.json-rw-r — r — 1 root root 696 Nov 12 16:48 kubernetes-D2.json-rw-r — r — 1 root root 1089 Nov 12 16:48 kubernetes-clear-containers.json-rw-r — r — 1 root root 824 Nov 12 16:48 kubernetes-kata-containers.json-rw-r — r — 1 root root 649 Nov 12 16:48 kubernetes.json-rw-r — r — 1 root root 1083 Nov 12 16:48 kubernetes-containerd.json
ACS Engine the Long Way — High Level (details below)
Before jumping into all of the details, there are five general steps to take when working with ACS engine.
Step 1: Generate an SSH Key
You may already have one. If not, there are a variety of ways to generate one. The SSH key will enable you to easily remote into the master node from your jump box VM.
Step 2: Create a Service Principal
There are a number of ways to get a Service Principal created. We will use the Azure command line tooling to do so.
Step 3: Edit your Cluster Definition
The Cluster Definition file allows you to customize your deployment. This is where you make the modifications that you need in your cluster, including everything from the number of nodes, the size of the VM, the operating system — to name a few.
Step 4: Generate the Templates
This is really what ACS Engine does by generating ARM template files, that are json-formatted. These ARM template files will be used at the command line with the Azure command line tooling to actually perform the deployment of the Kubernetes cluster.
Step 5: Submit your Templates to Azure Resource Manager (ARM)
Take the templates and build your K8s cluster. This is where you will use the Azure command line tooling to actually provision the cluster.
Detailed Instructions
Now that we spoke to the steps that a high level, let’s dive in deeper now and walk through each step in detail.
ACS Engine the Long Way — Step 1: Generate an SSH Key
It is likely that in your ~/.ssh folder you already have a public SSH key. If not, you can use ssh-kegen to generate your own ssh key.
$ cd ~/.ssh$ cat id_rsa.pubssh-rsa afdsrVpXUwFcfwe6gZ/6POHX6B9r8/qewrewrewq3hjr324jBFjLn/2Zn0dT4GnsS+1YUQlxjTgG2EBt root@your-vm
ACS Engine the Long Way — Step 2: Create a Service Principal
The command below is how you generate a service principal. Notice that you can choose “contributor” or “owner” and that you should pass in the https://github.com/Azure/acs-engine.git.
$ az ad sp create-for-rbac — role=”Contributor” — scopes=”/subscriptions/${SUBSCRIPTION_ID}”Create a service principal and configure its access to Azure resources.
az ad sp create-for-rbac [ — cert][ — create-cert][ — keyvault][ — name][ — password]
[ — role][ — scopes][ — sdk-auth {false, true}][ — skip-assignment {false, true}][ — years]
Service Principal Roles — Different roles, different capabilities
Owner — Lets you manage everything, including access to resources.
Contributor — Lets you manage everything except access to resources.
Reader — Lets you view everything, but not make any changes.
Create Service Principal
Note that in the command below I’ve indicated a few details. The first thing you should note is that it reads, “create-for-rbac,” which is a way to tell the Kubernetes cluster to install with role-based access control (RBAC). Also notice that the subscription ID has been indicated, meaning that the service principal will be relevant only within that subscription.
$ az ad sp create-for-rbac --role="Owner" --scopes="/subscriptions/d3faf3c1-c74f-dddd-9e40-fjdsafd7asd"{
"appId": "a3cb2f26-5f51-4490-a6d1-bdc1dee8e7a5",
"displayName": "azure-cli-2018-11-12-18-02-41",
"name": "http://azure-cli-2018-11-12-18-02-41",
"password": "2sw5kd13-97f5-4eb1-b8f1-dk3459adjfas",
"tenant": "72fd88dd-83d6-41af-91ab-2d7cd011db47"
}
Test the Service Principal
A good way to test the service principal is to simply login using it.
$ az login --service-principal -u a3cb2f26-5f51-4490-a6d1-bdc1dee8e7a5 -p 2sw5kd13-97f5-4eb1-b8f1-dk3459adjfas --tenant 72fd88dd-83d6-41af-91ab-2d7cd011db47[
{
"cloudName": "AzureCloud",
"id": "d3faf3c1-c74f-dddd-9e40-fjdsafd7asd",
"isDefault": true,
"name": "Windows Azure Internal Consumption",
"state": "Enabled",
"tenantId": "72fd88dd-83d6-41af-91ab-2d7cd011db47",
"user": {
"name": "a3cb2f26-5f51-4490-a6d1-bdc1dee8e7a5",
"type": "servicePrincipal"
}
]
The JSON below is the output that results after creating a service principal. There are some important mappings, such as the App ID being equal to that client ID. Also notice that the password is equal to the service principal secret.
ACS Engine the Long Way — Step 3: Edit your Cluster Definition
ACS Engine consumes a cluster definition file which outlines the desired shape, size, and configuration of Kubernetes. There are a number of features that can be enabled through the cluster definition: check the examples directory of the acs-engine git download.
In this next section we will actually modify the cluster definition file and make entries into it that indicate the way we wish our Kubernetes cluster to be deployed. Here are some sample elements:
dnsPrefix — must be a region-unique name and will form part of the hostname (e.g. myprod1, staging, leapingllama) — be unique!
keyData — must contain the public portion of an SSH key — this will be associated with the adminUsername value found in the same section of the cluster definition (e.g. ‘ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA….’)
clientId — this is the service principal’s appId uuid or name from step 2
secret — this is the service principal’s password or randomly-generated password from step 2
Optional : attach to an existing virtual network (VNET)
Here is a simple one
Notice the cluster definition file below has been filled in. Notice that we plugged in our SSH key, the service principal information (appid, and password). Also notice we’ve indicated the VM size in addition to the number of agent and master nodes we want in our cluster. This is the place where you really indicate exactly what you want. The acs-engine repo that we cloned where you can find many examples of how you might modify this file.
Pause for a moment. This is JSON document below (Cluster Definition File) is key — it is how you specify the details of your Kubernetes cluster.
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "your-company",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 2,
"vmSize": "Standard_D2_v2"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa wlUTeiJBFJiajk31YUQlxjTgG2EBt root@bt-ubuntu-vm"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "a3cb2f26-5f51-4490-a6d1-bdc1dee8e7a5",
"secret": "2sw5kd13-97f5-4eb1-b8f1-dk3459adjfas"
}
}
}ACS Engine the Long Way — Step 4: Generate the Templates
In order to be able to run the acs-engine binary, we will first need to download it based on the operating system. In my case, I have a Linux VM so I will make the appropriate download, as seen below. If you have a Windows or Mac or even another flavor of Linux, you will need to download the appropriate version.
Here is the Ubuntu based version for acs-engine.
Notice that I also un-tarred the binary.
# Get the binary
$ wget https://github.com/Azure/acs-engine/releases/download/v0.25.2/acs-engine-v0.25.2-linux-amd64.tar.gz# Untar the binary
$ tar -xvzf acs-engine-v0.25.2-linux-amd64.tar.gz
Verifying ACS Engine
Now that we’ve downloaded and untarred the binary, let’s go ahead and run the command to see that it functions correctly, in addition to seeing some of the options that are available.
$ cd acs-engine-v0.25.2-linux-amd64/[/home/azureuser/acs-engine/acs-engine-v0.25.2-linux-amd64]$ ./acs-engineUsage:acs-engine [flags]acs-engine [command]Available Commands:completion Generates bash completion scriptsdcos-upgrade Upgrade an existing DC/OS clusterdeploy Deploy an Azure Resource Manager templategenerate Generate an Azure Resource Manager templatehelp Help about any commandorchestrators Display info about supported orchestratorsscale Scale an existing Kubernetes or OpenShift clusterupgrade Upgrade an existing Kubernetes clusterversion Print the version of ACS-Engine
Use ACS engine to generate the ARM template files
It’s at this point that we generate the ARM template files based on the cluster definition files. There will be two important output files that we will use later when we execute the Azure command line tooling.
# Generate azuredeploy.json and azuredeploy.parameters.json$ ./acs-engine generate ../kubernetes.jsonINFO[0000] Generating assets into _output/your-company..
Look at all generated assets
A number of files were generated by ACS engine, the list of which can be seen below. The two most important files are: (1) azuredeploy.json and (2) azuredeploy.parameters.json.
# Folder [/home/azureuser/acs-engine/acs-engine-v0.25.2-linux-amd64/_output/applied-intuition]$ ls
apimodel.json ca.key etcdpeer0.keyapiserver.crt client.crt etcdserver.crtapiserver.key client.key etcdserver.keyazuredeploy.json etcdclient.crt kubeconfigazuredeploy.parameters.json etcdclient.key kubectlClient.crtca.crt etcdpeer0.crt kubectlClient.key
ACS Engine the Long Way — Step 5: Submit your Templates to Azure Resource Manager (ARM)
The next step is to use the Azure command line tooling to actually provision the cluster. ACS engine has already created the necessary text files that the Azure command line tooling needs to execute (azuredeploy.json and azuredeploy.parameters.json).
Use the az commands to build the k8s cluster
There are two commands below. The first command creates the resource group (rg-applied-intuition). The second command deploys a new Kubernetes cluster into the resource group specified (rg-applied-intuition).
Here are the commands:
# Create a resource group for the kubernetes cluster
$ az group create \--name "rg-your-company" \--location "westus2"# Do the actual provisioning of the Kubernetes cluster
$ az group deployment create \--name "YourCompanyDeployment" \--resource-group "rg-your-company" \--template-file "azuredeploy.json" \--parameters "azuredeploy.parameters.json"
Validate with portal
You can navigate back to the Azure portal to take a view of your Kubernetes cluster.
You can use the Azure command line tooling to view your current cluster.
The command below will list out the master nodes of your Kubernetes cluster.
# List master nodes
$ az vm list -g rg-your-company -o table
Name ResourceGroup Location— — — — — — — — — — — — — — — — — — — — — — — — — —k8s-master-38483117–0 rg-your-compnay westus2
You can also view the VM scale sets for the various agent notes that are running in your cluster.
Install kubectl
While you may have a provisioned a cluster, you do not necessarily have access to manipulating the cluster. The utility that is needed to do that is called, kubectl.
Kubectl is used to deploy and manage applications on Kubernetes. Using kubectl, you can inspect cluster resources; create, delete, and update components; and look at your new cluster and bring up example apps.
You can find detailed instructions here for installing kubectl here:
https://kubernetes.io/docs/tasks/tools/install-kubectl/
To install kubectl on Ubuntu, follow the commands below:
$ sudo apt-get update && sudo apt-get install -y apt-transport-https$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" \ | sudo tee -a /etc/apt/sources.list.d/kubernetes.list$ sudo apt-get update$ sudo apt-get install -y kubectl
Validate kubectl
Once you have completed the installation of kubectl, you can test it as follows:
kubectl versionGet fqdn so we can copy “config” from master node
Kubectl requires a configuration valve with credentials that will enable you to interact with your Kubernetes cluster. You can get those credentials by remolding into the master noted the clot start. To be able to remote into the master node, you will need to get its public IP address using the command below from the Azure command line tooling.
$ az network public-ip list — resource-group rg-your-companyYou will see fqdn:
“your-company.westus2.cloudapp.azure.com”You can also go to the Azure portal to see the URL to the master node of your Kubernetes cluster:
Copy the config file from the master node to the local folder of your jumpbox VM. Make sure you are in the ~/.kube directory of your jumpbox.
$ scp “azureuser@your-company.westus2.cloudapp.azure.com:/home/azureuser/.kube/config” .Put config into your local K8s ~/.kube folder.
What you have copied is the config file from the master node of the Kubernetes cluster over to your jump box VM. Now you should copy it into a special folder called, “.kube.” This is the default location where kubectl looks for configuration information.
Once you successfully copy the config file over to .kube folder, you can test connectivity to the Kubernetes cluster by requesting cluster information, as seen by the second command below.
$ cp config ~/.kube/# The second command$ kubectl cluster-infoKubernetes master is running at https://your-company.westus2.cloudapp.azure.comHeapster is running at https://your-company.westus2.cloudapp.azure.com/api/v1/namespaces/kube-system/services/heapster/proxyKubeDNS is running at https://your-company.westus2.cloudapp.azure.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxykubernetes-dashboard is running at https://your-company.westus2.cloudapp.azure.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxyMetrics-server is running at https://your-company.westus2.cloudapp.azure.com/api/v1/namespaces/kube-system/services/https:metrics-server:/proxytiller-deploy is running at https://your-company.westus2.cloudapp.azure.com/api/v1/namespaces/kube-system/services/tiller-deploy:tiller/proxyTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The Kubernetes Dashboard
The Kubernetes Dashboard is a web-based user interface that you can use to deploy containerized applications to a Kubernetes cluster, troubleshoot your containerized application, and manage the cluster itself along with its attendant resources. You can use Dashboard to get an overview of applications running on your cluster, as well as for creating or modifying individual Kubernetes resources (such as Deployments, Jobs, DaemonSets, etc).
To enable it you will need to define a ClusterRoleBinding. Create the YAML file dashboard_crb.yaml as follows:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-systemNext, issue the kubectl create -f dashboard_crb.yaml command.
kubectl create -f dashboard_crb.yamlYou are ready to navigate to the Kubernetes Dashboard using the following URL.
Start kubectl proxy so your jumpbox can point to the K8s cluster.
kubectl proxyNow use the browser to see the dashboard
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=defaultConclusion
This concludes our little journey to using ACS engine to configure and unmanaged Kubernetes cluster. As was mentioned at the beginning, it is possible to leverage AKS, which is a managed Azure Kubernetes service. However, if ultimate flexibility and control over the cluster is desired, the steps in this post will get you there efficiently.
Because this is now on unmanaged cluster, you can do anything you want to it, including provisioning a cluster with GPU enabled agent nodes, or leveraging a 32 core CPU, or leveraging the Calico network stack.
