Apply Configurations across Hybrid Cloud Environment with Anthos Config Management
Written by Mustafa Ali
Data EngineerWith the expansion of organizations, there is an increase in the development and production of clusters which involves creating and executing stable configuration and security policies across multiple environments, resulting in increased complexity and confusion. With Anthos Config Management (ACM), companies can automate and configure deployment in the shared environment and thus ensure approved security policies across Kubernetes clusters, on Google Kubernetes Engine (GKE), on-prem, and across other public cloud services providers too.
One of the critical challenges that we face in managing Kubernetes workloads running on a Hybrid set-up is applying configurations across multiple clusters and avoiding errors that affect all the environments simultaneously. A traditional approach is to set up a Github repository explicitly for Kubernetes configuration changes which are synced with multiple CI/CD pipelines. But the error rate and likelihood of things getting broken during the deployment cycle are significant.
Introducing Anthos Configuration Management
Anthos Config Management (ACM) is a solution, which addresses the management of Kubernetes clusters across hybrid cloud environments and helps manage resources in a configuration-as-code manner that combines three components:
Policy Controller
Config Sync
Config Controller
Anthos Config Management continuously protects and configures the Google Cloud and Kubernetes resources through these components.
Simply by interacting with your Anthos Config Management Git repository, you can deploy and manage Kubernetes objects across multiple clusters. It's essential to secure your ACM repository, which might get attacked because it contains your credentials.
ACM allows you to deploy your shared environment configurations automatically. It also enforces a platform to apply approved security policies across GKE, on on-premises Kubernetes clusters and other public cloud platforms. It also lets platform admins configure Google Cloud services using the same resource model. It's handy to enforce policies such as:
- PodSecurityPolicies: Prevent Pods from using the root Linux user.
- NetworkPolicies: Control the network traffic inside your clusters.
- ClusterRoles and ClusterRoleBindings: Control permissions within a cluster .
ACM Architecture
Anthos Config Management uses a Git repository to store configuration(s) and its operators, which apply configs to selected clusters.
Apply Configurations across Hybrid Environment: GKE & OnPrem
We can also use Anthos Config Management with GitLab to manage multiple Kubernetes clusters in a production environment. Securing the ACM repository is a vital deployment step.
The following diagram shows how we can apply ACM configurations with GitLab to manage multiple Kubernetes clusters, one in GKE, one in Anthos clusters on on-prem, and one in another cloud provider.
This diagram illustrates the following steps in the pipeline:
- The user submits a modification that calls a merge request (MR); after the merge request is validated in GitLab, it changes to one or all clusters.
- The MR triggers a GitLab CI/CD automated pipeline which tests and validates the new configuration.
- An administrator can approve or reject the MR. If the MR is admin-approved, the change is merged into the Git repository.
- After the approval, the ACM agents running in each cluster read this modification from GitLab and apply it to their cluster.
Manage Who can Approve Merge Requests
By default, GitLab uses roles to grant permissions on GitLab projects. For example, a user with the Developer role can open a merge request, and a user with the Maintainer role can approve the request. While this permission system can work well at first for Anthos Config Management, you might run into problems as your Kubernetes and ACM footprint grows. For example, the maintainers of the Anthos Config Management repository might get overwhelmed by the number of merge requests they have to process and approve.
Configuration Steps: Multi-Cluster Anthos Config Management Setup
Deploy multiple GKE clusters running in a shared project on a shared network in two regions and install Anthos Config Management. Follow the detailed steps for the configuration:
- Enable required GCP services: You can use the following gcloud commands to enable the respective Google APIs that are required for ACM setup:
- gcloud services enable
- container.googleapis.com \
- anthos.googleapis.com \
- gkeconnect.googleapis.com \
- gkehub.googleapis.com \
- cloudresourcemanager.googleapis.com
- Create or select a network: Create a network if compute.skipDefaultNetworkCreation organization policy constraint is enabled, else just set the NETWORK variable
- NETWORK="< NETWORK >"
- gcloud compute networks create "${NETWORK}."
- Configure firewalls to allow unrestricted internal network traffic: Configure the firewall if compute.skipDefaultNetworkCreation organization policy constraint is enabled, or else an equivalent set of firewalls should already be configured on the default network
- gcloud compute firewall-rules create allow-all-internal \
- network ${NETWORK} \
- allow tcp,udp,icmp \
- source-ranges 10.0.0.0/8
- Deploy Cloud NAT to allow egress from private GKE nodes
- Deploy the GKE clusters
- Authenticate with cluster
- Register the GKE clusters with Hub
- Register a GKE cluster using Workload Identity (recommended)
- Enable Anthos Config Management: Anthos Config Management (ACM) includes an operator that manages the lifecycle of other operators
- Config Sync
- Policy Controller
- Binary Authorization
- Hierarchy Controller
These operators are all managed as features using Hub.
- Enable ACM on the Hub
- gcloud alpha container hub config-management enable
How can we at Royal Cyber help?
When deploying a configuration set within a hybrid cloud environment, companies need to rely on a certified expert team that can help operationalize business needs in the cloud environment. As partners of Google Cloud, we have individuals who can collaborate with the clients, develop and manage the advanced services and products offered by Google Cloud.
To learn more about what we can offer, contact us at [email protected] or visit us at www.royalcyber.com
