Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.
Here are the edits we need to make to the cluster config in kubelet.
kubelet:
victionHard: memory.available<500Mi
systemReserved:
memory: "256Mi"
My default cluster config:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
creationTimestamp: "2021-09-07T23:03:47Z"
name: k8s.myezbrew.com
spec:
api:
dns: {}
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://my-ezbrew-state-store/k8s.myezbrew.com
dnsZone: myezbrew.com
etcdClusters:
- cpuRequest: 200m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: main
- cpuRequest: 100m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: events
iam:
allowContainerRegistry: true
legacy: false
kubelet:
anonymousAuth: false
kubernetesApiAccess:
- 0.0.0.0/0
kubernetesVersion: 1.21.4
masterInternalName: api.internal.k8s.myezbrew.com
masterPublicName: api.k8s.myezbrew.com
networkCIDR: 172.20.0.0/16
networking:
kubenet: {}
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 0.0.0.0/0
subnets:
- cidr: 172.20.32.0/19
name: us-west-2a
type: Public
zone: us-west-2a
topology:
dns:
type: Public
masters: public
nodes: public
My edited cluster config:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
creationTimestamp: "2021-09-07T23:03:47Z"
generation: 1
name: k8s.myezbrew.com
spec:
api:
dns: {}
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://my-ezbrew-state-store/k8s.myezbrew.com
dnsZone: myezbrew.com
etcdClusters:
- cpuRequest: 200m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: main
- cpuRequest: 100m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: events
iam:
allowContainerRegistry: true
legacy: false
kubelet:
anonymousAuth: false
evictionHard: memory.available<500Mi
systemReserved:
memory: 256Mi
kubernetesApiAccess:
- 0.0.0.0/0
kubernetesVersion: 1.21.4
masterInternalName: api.internal.k8s.myezbrew.com
masterPublicName: api.k8s.myezbrew.com
networkCIDR: 172.20.0.0/16
networking:
kubenet: {}
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 0.0.0.0/0
subnets:
- cidr: 172.20.32.0/19
name: us-west-2a
type: Public
zone: us-west-2a
topology:
dns:
type: Public
masters: public
nodes: public
To review the changes this will make. I will first look at the output from running kops update cluster. I find the output to contain more informational in some cases than what terraform provides. In this output you will see what the original settings are in place for the cluster. The terraform output will only show the setting being put in place as it is a server default.
Here is the change of our memory setting:
+ evictionHard: memory.available<500Mi
– evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
Notice that we have actually changed a string list and unset some values!
bitnami@ip-172-26-15-161:~/repo/terraform$ kops update cluster --name=$NAME
Will modify resources:
LaunchTemplate/master-us-west-2a.masters.k8s.myezbrew.com
UserData
...
clusterDomain: cluster.local
enableDebuggingHandlers: true
+ evictionHard: memory.available<500Mi
- evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
hostnameOverride: '@aws'
kubeconfigPath: /var/lib/kubelet/kubeconfig
...
nonMasqueradeCIDR: 100.64.0.0/10
podManifestPath: /etc/kubernetes/manifests
+ systemReserved:
+ memory: 256Mi
masterKubelet:
anonymousAuth: false
...
clusterDomain: cluster.local
enableDebuggingHandlers: true
+ evictionHard: memory.available<500Mi
- evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
hostnameOverride: '@aws'
kubeconfigPath: /var/lib/kubelet/kubeconfig
...
podManifestPath: /etc/kubernetes/manifests
registerSchedulable: false
+ systemReserved:
+ memory: 256Mi
__EOF_CLUSTER_SPEC
...
clusterDomain: cluster.local
enableDebuggingHandlers: true
+ evictionHard: memory.available<500Mi
- evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
hostnameOverride: '@aws'
kubeconfigPath: /var/lib/kubelet/kubeconfig
...
podManifestPath: /etc/kubernetes/manifests
registerSchedulable: false
+ systemReserved:
+ memory: 256Mi
UpdatePolicy: automatic
channels:
...
LaunchTemplate/nodes-us-west-2a.k8s.myezbrew.com
UserData
...
clusterDomain: cluster.local
enableDebuggingHandlers: true
+ evictionHard: memory.available<500Mi
- evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
hostnameOverride: '@aws'
kubeconfigPath: /var/lib/kubelet/kubeconfig
...
nonMasqueradeCIDR: 100.64.0.0/10
podManifestPath: /etc/kubernetes/manifests
+ systemReserved:
+ memory: 256Mi
__EOF_CLUSTER_SPEC
...
clusterDomain: cluster.local
enableDebuggingHandlers: true
+ evictionHard: memory.available<500Mi
- evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
hostnameOverride: '@aws'
kubeconfigPath: /var/lib/kubelet/kubeconfig
...
nonMasqueradeCIDR: 100.64.0.0/10
podManifestPath: /etc/kubernetes/manifests
+ systemReserved:
+ memory: 256Mi
UpdatePolicy: automatic
channels:
...
Must specify --yes to apply changes
bitnami@ip-172-26-15-161:~/repo/terraform$
When we review the terraform plan output you will notice the previous values and the current setting are not available. They are part of the user_data that is passed in from kOps as a single block. I have truncated the user_data value to “…” so you will have to run this yourself to see it. The page would be a scroll fest to include it.
We can however take note the following resources will be updated
resource “aws_autoscaling_group” “master-us-west-2a-masters-k8s-myezbrew-com”
resource “aws_autoscaling_group” “nodes-us-west-2a-k8s-myezbrew-com”
resource “aws_launch_template” “master-us-west-2a-masters-k8s-myezbrew-com”
resource “aws_launch_template” “nodes-us-west-2a-k8s-myezbrew-com”
Terraform will perform the following actions:
# aws_autoscaling_group.master-us-west-2a-masters-k8s-myezbrew-com will be updated in-place
~ resource "aws_autoscaling_group" "master-us-west-2a-masters-k8s-myezbrew-com" {
id = "master-us-west-2a.masters.k8s.myezbrew.com"
name = "master-us-west-2a.masters.k8s.myezbrew.com"
# (22 unchanged attributes hidden)
~ launch_template {
id = "lt-01a299d21352fe0bc"
name = "master-us-west-2a.masters.k8s.myezbrew.com"
~ version = "1" -> (known after apply)
}
# (11 unchanged blocks hidden)
}
# aws_autoscaling_group.nodes-us-west-2a-k8s-myezbrew-com will be updated in-place
~ resource "aws_autoscaling_group" "nodes-us-west-2a-k8s-myezbrew-com" {
id = "nodes-us-west-2a.k8s.myezbrew.com"
name = "nodes-us-west-2a.k8s.myezbrew.com"
# (22 unchanged attributes hidden)
~ launch_template {
id = "lt-027306dba611aa56d"
name = "nodes-us-west-2a.k8s.myezbrew.com"
~ version = "1" -> (known after apply)
}
# (8 unchanged blocks hidden)
}
# aws_launch_template.master-us-west-2a-masters-k8s-myezbrew-com will be updated in-place
~ resource "aws_launch_template" "master-us-west-2a-masters-k8s-myezbrew-com" {
id = "lt-01a299d21352fe0bc"
~ latest_version = 1 -> (known after apply)
name = "master-us-west-2a.masters.k8s.myezbrew.com"
tags = {
"KubernetesCluster" = "k8s.myezbrew.com"
"Name" = "master-us-west-2a.masters.k8s.myezbrew.com"
"k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup" = "master-us-west-2a"
"k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/kops-controller-pki" = ""
"k8s.io/cluster-autoscaler/node-template/label/kubernetes.io/role" = "master"
"k8s.io/cluster-autoscaler/node-template/label/node-role.kubernetes.io/control-plane" = ""
"k8s.io/cluster-autoscaler/node-template/label/node-role.kubernetes.io/master" = ""
"k8s.io/cluster-autoscaler/node-template/label/node.kubernetes.io/exclude-from-external-load-balancers" = ""
"k8s.io/role/master" = "1"
"kops.k8s.io/instancegroup" = "master-us-west-2a"
"kubernetes.io/cluster/k8s.myezbrew.com" = "owned"
}
~ user_data = "..."
# (9 unchanged attributes hidden)
# (7 unchanged blocks hidden)
}
# aws_launch_template.nodes-us-west-2a-k8s-myezbrew-com will be updated in-place
~ resource "aws_launch_template" "nodes-us-west-2a-k8s-myezbrew-com" {
id = "lt-027306dba611aa56d"
~ latest_version = 1 -> (known after apply)
name = "nodes-us-west-2a.k8s.myezbrew.com"
tags = {
"KubernetesCluster" = "k8s.myezbrew.com"
"Name" = "nodes-us-west-2a.k8s.myezbrew.com"
"k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup" = "nodes-us-west-2a"
"k8s.io/cluster-autoscaler/node-template/label/kubernetes.io/role" = "node"
"k8s.io/cluster-autoscaler/node-template/label/node-role.kubernetes.io/node" = ""
"k8s.io/role/node" = "1"
"kops.k8s.io/instancegroup" = "nodes-us-west-2a"
"kubernetes.io/cluster/k8s.myezbrew.com" = "owned"
}
~ user_data = "..."
# (9 unchanged attributes hidden)
# (7 unchanged blocks hidden)
}
Plan: 0 to add, 4 to change, 0 to destroy.
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
creationTimestamp: "2021-09-07T23:03:47Z"
name: k8s.myezbrew.com
spec:
api:
dns: {}
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://my-ezbrew-state-store/k8s.myezbrew.com
dnsZone: myezbrew.com
etcdClusters:
- cpuRequest: 200m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: main
- cpuRequest: 100m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: events
iam:
allowContainerRegistry: true
legacy: false
kubelet:
anonymousAuth: false
kubernetesApiAccess:
- 0.0.0.0/0
kubernetesVersion: 1.21.4
masterInternalName: api.internal.k8s.myezbrew.com
masterPublicName: api.k8s.myezbrew.com
networkCIDR: 172.20.0.0/16
networking:
kubenet: {}
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 0.0.0.0/0
subnets:
- cidr: 172.20.32.0/19
name: us-west-2a
type: Public
zone: us-west-2a
topology:
dns:
type: Public
masters: public
nodes: public
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
creationTimestamp: "2021-09-05T22:47:10Z"
generation: 1
name: uswest2.k8s.myezbrew.com
spec:
api:
dns: {}
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://my-ezbrew-state-store/uswest2.k8s.myezbrew.com
etcdClusters:
- cpuRequest: 200m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: main
- cpuRequest: 100m
etcdMembers:
- encryptedVolume: true
instanceGroup: master-us-west-2a
name: a
memoryRequest: 100Mi
name: events
iam:
allowContainerRegistry: true
legacy: false
kubelet:
anonymousAuth: false
evictionHard: memory.available<500Mi
kubernetesApiAccess:
- 0.0.0.0/0
kubernetesVersion: 1.21.4
masterInternalName: api.internal.uswest2.k8s.myezbrew.com
masterPublicName: api.uswest2.k8s.myezbrew.com
networkCIDR: 172.20.0.0/16
networking:
kubenet: {}
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 0.0.0.0/0
subnets:
- cidr: 172.20.32.0/19
name: us-west-2a
type: Public
zone: us-west-2a
topology:
dns:
type: Public
masters: public
nodes: public