Securing Kubernetes Deployments on AWS – Guide

Kubernetes Deployment on AWS

Kubernetes is open-source software for deploying and managing containerized applications at scale. Kubernetes can manage clusters on Amazon EC2 instances, run containers on those instances, and perform deployment, maintenance, and scaling processes.

Kubernetes lets you run containerized applications on-premises and in the cloud using the same set of tools.

AWS offers Amazon Elastic Kubernetes Service (EKS), a managed, certified Kubernetes-compatible service for running Kubernetes on AWS and on-premises, with community-supported service integrations.

Kubernetes is used as an open-source project. Kubernetes allows you to run containerized applications anywhere without changing operational tools. Kubernetes is maintained and improved regularly by a large community of volunteers.

The large Kubernetes community builds and maintains Kubernetes-compatible software that can be used to enhance and extend application architectures.

Kubernetes Security Best Practices on AWS

Understanding the Shared Responsibility Model

Security and compliance are considered shared responsibilities when using managed services like EKS. In general, AWS is responsible for security “inside” the cloud, and the cloud customer is responsible for security “inside” the cloud.

With EKS, AWS manages the Kubernetes control plane. This includes the Kubernetes master server, etcd database, and other infrastructure required by AWS to provide secure and reliable services.

EKS customers are primarily responsible for identity and access management (IAM), pod security, runtime security, and network security.

AWS is also responsible for keeping EKS-optimized Amazon Machine Images (AMIs) up to date with Kubernetes patch releases and security patches. Customers using managed node groups (MNGs) must upgrade their node groups to the latest AMI via the EKS API, CLI, Cloudformation, or AWS console.

Practice Red/Blue Team and Penetration Testing

Divide the security staff into two teams: a red team and a blue team. The red team focuses on investigating vulnerabilities in various systems, while the blue team is in charge of vulnerability defense.

If you do not have enough security staff to form a separate team, consider hiring an outside organization with knowledge of Kubernetes exploits.

Kubesploit is a penetration testing framework that you can use for tests. It can simulate real attacks against Kubernetes clusters, allowing the blue team to practice responding to attacks and evaluate their effectiveness. You can regularly attack your cluster to discover vulnerabilities and misconfigurations.

Auditing and Logging

Collecting and analyzing audit logs can be useful for a number of reasons.

Logs are useful for root cause analysis of production issues. When enough logs are collected, they can also be used to detect anomalous behavior. EKS sends audit logs to the Amazon Cloudwatch service.

Audit logs are managed by the EKS-managed Kubernetes control plane. Amazon provides instructions for enabling/disabling control plane logs, including the Kubernetes API server, controller manager, and scheduler.

Encryption at Rest

Kubernetes provides three AWS native storage options for Luster: EBS, EFS, and FSx. All three provide data-at-rest encryption using either a service-managed key or a customer master key (CMK).

For EBS, you can use either an in-tree storage driver or an EBS CSI driver. Both contain parameters to encrypt the volume and provide a CMK.

For EFS, you can use the EFS CSI driver, but unlike EBS, this driver does not support dynamic configuration. If you are using EFS with EKS, you must provision and configure at-rest filesystem encryption before creating persistent volumes (PVs).

Network Policy

In a Kubernetes cluster, pod-to-pod communication is allowed by default. Although this flexibility is useful in development stages, it is not considered safe for production.

Kubernetes network policies provide a mechanism to restrict network traffic between pods and between pods and external services. Kubernetes network policies apply to layers 3 and 4 of the OSI model.

Network policies use pod selectors and labels to identify source and destination pods, but can also include IP addresses, port numbers, protocol numbers, or a combination thereof.

Scan Images for Vulnerabilities Regularly

Like virtual machines, container images can contain vulnerable binaries and application libraries. The best way to avoid threats is to scan images regularly with an automated scanner.

Images stored in Amazon Elastic Container Registry (ECR) can be scanned by automated trigger or on demand (every 24 hours). ECR currently leverages Clair, an open source image scanning solution.

After the image is scanned, the results are written to EventBridge’s ECR event stream. You can also view the scan results in the ECR console. Images with HIGH or CRITICAL vulnerabilities should be deleted or rebuilt. When a deployed image becomes vulnerable, it should be replaced as soon as possible.

Policy as Code

A policy can be thought of as a set of rules that govern behavior. Finding and enforcing policies consistently across Kubernetes clusters can be difficult.

They also need to be flexible, because requirements change. Policy-as-code (PaC) solutions can help detect, prevent, and respond to known persistent threats by automating security, compliance, and privacy controls. A number of third-party solutions are available which can help configure and manage policies in EKS clusters.

Conclusion

In this article, I explained the basics of Kubernetes deployment and how to secure it on AWS:

Understanding the shared responsibility model—security and compliance are considered shared responsibilities when using managed services like EKS.
Practice red/blue team and penetration testing—the red team focuses on investigating vulnerabilities in various systems, while the blue team is in charge of vulnerability defense.
Auditing and logging—collecting and analyzing audit logs can be useful for root cause analysis of production issues.
Encryption at rest—Kubernetes provides three AWS native storage options that provide data-at-rest encryption.
Network policy—network policies use pod selectors and labels to identify the source and destination pods.
Scan images for vulnerabilities regularly—container images can contain vulnerable binaries and application libraries. The best way to avoid threats is to scan images regularly with an automated scanner.
Policy as code—policy-as-code solutions can help detect, prevent, and respond to known persistent threats by automating security, compliance, and privacy controls.