Use node taints in an Azure Kubernetes Service (AKS) cluster

This article describes how to use node taints in an Azure Kubernetes Service (AKS) cluster.

Overview

The AKS scheduling mechanism is responsible for placing pods onto nodes and is based upon the upstream Kubernetes scheduler, kube-scheduler. You can constrain a pod to run on particular nodes by attaching the pods to a set of nodes using node affinity or by instructing the node to repel a set of pods using node taints, which interact with the AKS scheduler.

Node taints work by marking a node so that the scheduler avoids placing certain pods on the marked nodes. You can place tolerations on a pod to allow the scheduler to schedule that pod on a node with a matching taint. Taints and tolerations work together to help you control how the scheduler places pods onto nodes. For more information, see example use cases of taints and tolerations.

Taints are key-value pairs with an effect. There are three values for the effect field when using node taints: NoExecute, NoSchedule, and PreferNoSchedule.

• NoExecute: Pods already running on the node are immediately evicted if they don't have a matching toleration. If a pod has a matching toleration, it might be evicted if tolerationSeconds are specified.
• NoSchedule: Only pods with a matching toleration are placed on this node. Existing pods aren't evicted.
• PreferNoSchedule: The scheduler avoids placing any pods that don't have a matching toleration.

Node taint options

There are two types of node taints that can be applied to your AKS nodes: node taints and node initialization taints.

• Node taints are meant to remain permanently on the node for scheduling pods with node affinity. Node taints can only be added, updated, or removed completely using the AKS API.
• Node initialization taints are placed on the node at boot time and are meant to be used temporarily, such as in scenarios where you might need extra time to set up your nodes. You can remove node initialization taint using the Kubernetes API and aren't guaranteed during the node lifecycle. They appear only after a node is scaled up or upgraded/reimaged. New nodes still have the node initialization taint after scaling. Node initialization taints appear on all nodes after upgrading. If you want to remove the initialization taints completely, you can remove them using the AKS API after untainting the nodes using the Kubernetes API. Once you remove the initialization taints from the cluster spec using the AKS API, newly created nodes don't come up with those initialization taints. If the initialization taint is still present on existing nodes, you can permanently remove it by performing a node image upgrade operation.

Use node taints

Prerequisites

This article assumes you have an existing AKS cluster. If you need an AKS cluster, you can create one using Azure CLI, Azure PowerShell, or the Azure portal.

Create a node pool with a node taint

1. Create a node pool with a taint using the az aks nodepool add command and use the --node-taints parameter to specify sku=gpu:NoSchedule for the taint.

   az aks nodepool add \
       --resource-group $RESOURCE_GROUP_NAME \
       --cluster-name $CLUSTER_NAME \
       --name $NODE_POOL_NAME \
       --node-count 1 \
       --node-taints "sku=gpu:NoSchedule" \
       --no-wait
   

2. Check the status of the node pool.

3. Check that the taint is set on the node.

Update a node pool to add a node taint

1. Update a node pool to add a node taint using the az aks nodepool update command and use the --node-taints parameter to specify sku=gpu:NoSchedule for the taint.

   az aks nodepool update \
       --resource-group $RESOURCE_GROUP_NAME \
       --cluster-name $CLUSTER_NAME \
       --name $NODE_POOL_NAME \
       --node-taints "sku=gpu:NoSchedule" \
       --no-wait
   

2. Check the status of the node pool.

3. Check that the taint has been set on the node.

Use node initialization taints (preview)

Prerequisites and limitations

• You need the Azure CLI version 3.0.0b3 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.
• You can only apply initialization taints via cluster create or upgrade when using the AKS API. If using ARM templates, you can specify node initialization taints during node pool creation and update.
• You can't apply initialization taints to Windows node pools using the Azure CLI.

Get the credentials for your cluster

• Get the credentials for your AKS cluster using the az aks get-credentials command.

  az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME
  

Install the aks-preview Azure CLI extension

• Register or update the aks-preview extension using the az extension add or az extension update command.

  # Register the aks-preview extension
  az extension add --name aks-preview
  
  # Update the aks-preview extension
  az extension update --name aks-preview
  

Register the NodeInitializationTaintsPreview feature flag

1. Register the NodeInitializationTaintsPreview feature flag using the az feature register command.

   az feature register --namespace "Microsoft.ContainerService" --name "NodeInitializationTaintsPreview"
   

   It takes a few minutes for the status to show Registered.

2. Verify the registration status using the az feature show command.

   az feature show --namespace "Microsoft.ContainerService" --name "NodeInitializationTaintsPreview"
   

3. When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider using the az provider register command.

   az provider register --namespace Microsoft.ContainerService
   

Create a cluster with a node initialization taint

1. Create a cluster with a node initialization taint using the az aks create command and the --node-initialization-taints parameter to specify sku=gpu:NoSchedule for the taint.

   Important

   The node initialization taints you specify apply to all of the node pools in the cluster. To apply the initialization taint to a specific node, you can use an ARM template instead of the CLI.

   az aks create \
   --resource-group $RESOURCE_GROUP_NAME \
   --name $CLUSTER_NAME \
   --node-count 1 \
   --node-init-taints "sku=gpu:NoSchedule"
   

2. Check the status of the node pool.

3. Check that the taint is set on the node.

Update a cluster to add a node initialization taint

1. Update a cluster to add a node initialization taint using the az aks update command and the --node-initialization-taints parameter to specify sku=gpu:NoSchedule for the taint.

   Important

   When updating a cluster with a node initialization taint, the taints apply to all node pools in the cluster. You can view updates to node initialization taints on the node after a reimage operation.

   az aks update \
   --resource-group $RESOURCE_GROUP_NAME \
   --name $CLUSTER_NAME \
   --node-init-taints "sku=gpu:NoSchedule"
   

2. Check the status of the node pool.

3. Check that the taint is set on the node.

Check the status of the node pool

• After applying the node taint or initialization taint, check the status of the node pool using the az aks nodepool list command.

  az aks nodepool list --resource-group $RESOURCE_GROUP_NAME --cluster-name $CLUSTER_NAME
  

  If you applied node taints, the following example output shows that the <node-pool-name> node pool is Creating nodes with the specified nodeTaints:

  [
    {
      ...
      "count": 1,
      ...
      "name": "<node-pool-name>",
      "orchestratorVersion": "1.15.7",
      ...
      "provisioningState": "Creating",
      ...
      "nodeTaints":  [
        "sku=gpu:NoSchedule"
      ],
      ...
    },
   ...
  ]
  

  If you applied node initialization taints, the following example output shows that the <node-pool-name> node pool is Creating nodes with the specified nodeInitializationTaints:

  [
    {
      ...
      "count": 1,
      ...
      "name": "<node-pool-name>",
      "orchestratorVersion": "1.15.7",
      ...
      "provisioningState": "Creating",
      ...
      "nodeInitializationTaints":  [
        "sku=gpu:NoSchedule"
      ],
      ...
    },
   ...
  ]
  

Check that the taint is set on the node

• Check the node taints and node initialization taints in the node configuration using the kubectl describe node command.

  kubectl describe node $NODE_NAME
  

  If you applied node taints, the following example output shows that the <node-pool-name> node pool has the specified Taints:

  [
      ...
      Name: <node-pool-name>
      ...
      Taints: sku=gpu:NoSchedule
      ...
      ],
      ...
   ...
  ]
  

Remove node taints

Remove a specific node taint

• Remove node taints using the az aks nodepool update command. The following example command removes the "sku=gpu:NoSchedule" node taint from the node pool.

  az aks nodepool update \
  --cluster-name $CLUSTER_NAME \
  --name $NODE_POOL_NAME \
  --node-taints "sku=gpu:NoSchedule"
  

Remove all node taints

• Remove all node taints from a node pool using the az aks nodepool update command. The following example command removes all node taints from the node pool.

  az aks nodepool update \
  --cluster-name $CLUSTER_NAME \
  --name $NODE_POOL_NAME \
  --node-taints ""
  

Remove node initialization taints

You have the following options to remove node initialization taints from the node:

• Remove node initialization taints temporarily using the Kubernetes API. If you remove them this way, the taints reappear after node scaling or upgrade occurs. New nodes still have the node initialization taint after scaling. Node initialization taints appear on all nodes after upgrading.
• Remove node initialization taints permanently by untainting the node using the Kubernetes API, and then removing the taint using the AKS API. Once the initialization taints are removed from cluster spec using AKS API, newly created nodes after reimage operations no longer have initialization taints.

When you remove all initialization taint occurrences from node pool replicas, the existing initialization taint might reappear after an upgrade with any new initialization taints.

Remove node initialization taints temporarily

• Remove node initialization taints temporarily using the kubectl taint nodes command.

  This command removes the taint from only the specified node. If you want to remove the taint from every node in the node pool, you need to run the command for every node that you want the taint removed from.

  kubectl taint nodes $NODE_POOL_NAME sku=gpu:NoSchedule-
  

  Once removed, node initialization taints reappear after node scaling or upgrading occurs.

Remove node initialization taints permanently

1. Follow steps in Remove node initialization taints temporarily to remove the node initialization taint using the Kubernetes API.

2. Remove the taint from the node using the AKS API using the az aks update command. This command removes the node initialization taint from every node in the cluster.

   az aks update \
   --resource-group $RESOURCE_GROUP_NAME \
   --name $CLUSTER_NAME \
   --node-init-taints "sku=gpu:NoSchedule"
   

Check that the taint has been removed from the node

• Check the node taints and node initialization taints in the node configuration using the kubectl describe node command.

  kubectl describe node $NODE_NAME
  

  If you removed a node taint, the following example output shows that the <node-pool-name> node pool doesn't have the removed taint under Taints:

  [
      ...
      Name: <node-pool-name>
      ...
      Taints: 
      ...
      ],
      ...
   ...
  ]
  

Next steps

• Learn more about example use cases for taints and tolerations.
• Learn more about best practices for advanced AKS scheduler features.
• Learn more about Kubernetes labels in the Kubernetes labels documentation.