Skip to main content

Enterprise Patterns in Terraform

What are Modules? 
- self contained pieces of IAC that abstract the infrastructure deployments 
- use clear organization and DRY (Dont Repeat Yourself) 
- helps in writing composable, shareable and reusable infrastructures 

 Scope the requirements into appropriate modules: - When building a module, consider 3 areas: 
 1. Encapsulation - Group infrastructure that is always deployed together 
 2. Privileges - Restrict modules to privilege boundaries 
 3. Voltatility - Separate long lived infrastructure from short-lived (Ex: Database-static vs Application Servers-dynamic) 

Create the module MVP :
* Always aim to deliver a module that works for 80% of usecases 
* Never code for edge cases. A module should be a reusable block of code. 
* Avoid conditional expressions in MVP 
* Module should only expose the most commonly modified arguments as variables. 

 Scoping Example - A team wants to provision their infrasturucture, web tier application, and app tier using Terraform 
- web application requires autoscaling group 
- app tier also requires autoscaling group, an S3 and a database.

So the modules for the above requirement could be as: 
Module 1: Network: [VPC, NACL, NAT Gateway] 
- responsible for infrastructure networking 
 - contains network ACLs and NAT gateway 
 - also includes VPC, subnets, peering and direct connect 

Module 2: Web: [Load Balancer, Auto Scaling Group] 
- creates and manages the infrastructure needed to run the web application 
 - contains load balancer and auto scaling group 
 - could also include EC2 instances, S3 buckets, security groups inside the application and logging 

Module 3: App: [Load Balancer, Auto Scaling Group, S3 bucket] 
- creates and manages the infrastructure needed to run the app tier application 
 - contains the load balancer, auto scaling group, and s3 buckets 
 - can also include EC2 instances, S3 buckets, security groups inside the application and logging 

Module 4: Database: [Database] 
 - creates and manages the infrastructure needed to run the database 
 - contains the RDS instance used by the application 
 - can also include all associated storage, all backup data and logging 

Module 5: Routing: [Hosted Zone, Route 53, Route Table] 
- creates and manages the infrastructure needed for any network routing 
 - contains hosted zones, Route 53, Route Tables 

Module 6: Security: [IAM- Identity And Access Management] 
- creates and manages the infrastructure needed for security 
 - contains IAM resources, also include security groups and MFA 

 After we are done writing modules: 
- we import them into the private module registry 
- advertise their availability to the respective team members for consumption


Define and use a consistent module structure:
- Define list of .tf files that must be in the module and what they should contain
- Define a .gitignore for modules
- Create a standard way of providing examples (terraform.tfvars.example)
- Use a consistent directory structure with a defined set of directories, even if they may be empty
- All module directories should have a README detailing the purpose and use of files within it


Use source control to track modules:
- Place modules ini source control to manage versions, collaboration, and audit trail of changes
- Tag and document all releases to master (use CHANGELOG and README as a minimum)
- Code review all changes to the master
- Encourage your module users to reference by tag
- Assign each module an owner
- Use only one module per repository
 

Documentation source:



Labels:

Comments

Post a Comment

Popular posts from this blog

AWS Connect: Reporting and Visualizations

Amazon connect offers: - built in reports i.e., historical and real-time reports.  We can customize these reports, schedule them and can integrate with any BI tool of our requirement to query and view the connect data.  Sample solution provided by AWS: 1. Make sure Connect is exporting the CTR data using Kinesis Data Stream 2. Use Kinesis Firehose to deliver the CTR that are in KDS to S3. (CTR's can be delivered as batch of records, so one s3 object might have multiple CTR's). AWS Lambda is used to add a new line character to each record, which makes object easier to parse.  3. s3 Event Notifications are used to send an event to modify the CTR record and saves it in S3. 4. Athena queries the modified CTR's using SQL. Use partitions to restrict the amount of data scanned by each query, improving performance and reducing cost. Lambda function is used to maintain the partitions.  5. Quicksight is used to visualize the modified CTRs.  Solution variations: Convert re...
Post a Comment
Read more

Databricks: Job aborted due to stage failure. Total size of serialized results is bigger that spark driver memory.

  While running a databricks job, especially running a job with large datasets and longer running queries that creates a lot of temp space - we might be facing below issue if we have a minimal configuration set to the cluster.  The simple way to fix this would be changing the spark driver config in the databricks cluster tab spark.driver.maxResultSize = 100G (change the GB based on your cluster size)
Post a Comment
Read more

Terraform lifecycle

 If we are using terraform, terraform state file is the heart of all the infrastructure that we spin up using terraform templates.  There are several ways to deploy the infrastructure using terraform: 1. Using CLI (setup terraform and then run terraform commands) 2. Automated Build (terraform scripts integrated as part of your jenkins pipeline) No matter of the way we chose, we must make sure that we are using the same terraform state file, so that we are having a sync and proper checklists of the resources that we used.  I would like to share the terraform commands that we do on a daily basis: terraform init = the basic/starting command which initializes the terraform (make sure the proper provider is provided. In my case, I use AWS).  terraform workspace select <workspace name > (creates a new workspace, useful in scenarios where we have different terraform modules - database, servers, logs, storage) terraform state list = shows the list of terraform resour...
Post a Comment
Read more