About our setup:
- We use AWS, where we have a system called Kubernetes to manage our servers.
- Our servers are called nodes. We have 7 nodes for our important production work.
- These servers are special AWS servers called EC2 instances, and we use something called "spot instances." Spot instances are cheaper, but sometimes AWS can "take them back". This happens all the time, but today we had an issue with this setup.
What happened today:
- AWS took back 5 of our spot instance servers.
- 2 of them got replaced automatically in kubernetes, but 3 others didn't, even though the servers were replaced in AWS.
- Because 3 nodes were missing, we didn't have enough resources to run all of the services.
- We fixed it by manually deleting the “hanging” servers in AWS. So the auto scaling group could create 3 new servers and connect them to Kubernetes.
Downtime:
Around 1 hour from 09.50-10.50 spread across some, and not all the services.
Kubernetes.
Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications.
EC2.
Amazon Web Service EC2 (Amazon Elastic Compute Cloud). An EC2 instance is simply a virtual server in Amazon Web Services terminology.
Nodes.
Kubernetes runs your workload by placing containers into Pods to run on Nodes. A node may be a virtual or physical machine, depending on the cluster.
Pods.
Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. I.e. a pod in Mentor Medier can be the vipps login service.